Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24449

New Support for dependencies between multiple repositories



    • Improvement
    • Status: Accepted
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Code Generator
    • Point
    • 2020q3
    • Support for better dependencies between multiple repositories


      Currently it is hard to represent the ECL for a query in more than repository.  This can either lead to lots of teams developing in the same repository - with independent branches that don't tend to get merged, or the code being cloned into another repository, with even bigger problems about keeping in sync.

      One of the reasons it is a problem is the way imports are implemented.  The code generator assumes all the import locations are merged into a single module hierarchy.  This can cause problems with symbols from multiple repository clashing.  Alternatively the different repositories can be treated as if they are in different sub directories, but then there are problems with relative v absolute imports.
      As a consequence, I started thinking about the current processing model and wondering if we could change it.  In particular, allow the different repositories to live in different global scopes, so the symbols could not clash, and each repository could continue to use absolute imports. 

      After various iterations, a model based on the npm package manager has been implemented. The aim is to allow the npm package manager to be used if desired, but for it to not be necessry.

      A Packages is a set of source code that is compiled independently of the code that uses it i.e. names will not clash. Packages appear as normal modules within the repository that use them.

      Normal process

      The normal process for defining dependencies on other packages is as follows:

      Define dependencies in a package.json file in the root of the repository:

        "dependencies": {
          "demoRepoD": "ghalliday/gch-ecldemo-d#version3"

      Where ghalliday/gch-ecldemo-d is the name of the git repository containing the code, and version3 is a branch or tag for the version of the code that. (A SHA can also be used). This package.json should be versioned in the source control system along with the rest of the code.

      The attributes within the package can then be used adding "import demoRepoD;" to an ecl definition.

      If eclcc is used with the --fetchrepos and --updaterepos options eclcc will take care of fetching the code from the remote git repositories.

      npm package manager

      It is possible to use the npm package manager to ensure that the labels or branches are mapped to a specific version. If the following command is used:

      npm install --package-lock-only

      The --package-lock-only indicates that npm will create a package-lock.json file in the same location as package.json, but not check out the associated versions of the code. The package-lock.json file will contain something similar to the following:

        "packages": {
          "node_modules/demoRepoD": {
            "resolved": "https://github.com/ghalliday/gch-ecldemo-d.git#644c4585221f4dd80ca1e8f05974983455a244e5",

      This package-lock.json file will take precedence over the package.json file. This can be compiled in the same way with eclcc automatically downloading the dependencies.

      This option also allows you to use npm's semantic versioning (#semver) which is not currently supported by eclcc.

      (NOTE: Currently the node_modules/ prefix will need to be deleted from the entry once it is generated. That is likely to be fixed before 8.4.0 is released.)

      Finally npm install can be used without the --package-lock-only file. This will check out the appropriate version of the code into the node_modules subdirectory of the current project. You can then compile the code using eclcc completely independently from the source control system.

      Package precedence

      As a result of the above, packages are defined in one of the following 3 ways - and have the following precedence:

      1. Contents of the node_modules/ directory.
      Any directories within this directory are treated as packages, and compiled separately. Although normally created by an npm install, the node_modules directory could also contain git submodules, each of which would be compiled as a separate package. This has not been tested, and there may be issues (e.g. whether submodules work with githooks.)

      2. Entries in a package-lock.json file.

      3. Entries in a package.json file.

      Other considerations

      • Different repositories are allowed to use different versions of packages


        Issue Links

          Add an option to suppress plugins (and std) from archives Sub-task Resolved Gavin Halliday
          Implement dependency support using package-lock.json Sub-task Resolved Gavin Halliday
          Add archive support for compiling from multiple repositories Sub-task Resolved Gavin Halliday
          Add an option to directly compile a query from a repository Sub-task Resolved Unassigned
          Add support for package.json Sub-task Resolved Gavin Halliday
          Allow the default repo directory to be configured for eclccserver Sub-task New Unassigned
          Document and release test cases. Sub-task New Unassigned
          Generate a warning if multiple repository versions are used by a query. Sub-task New Unassigned
          Allow default git repo to be configured Sub-task Resolved Gavin Halliday
          review git authentication options Sub-task Resolved Gavin Halliday
          Allow source code dependencies to be resolved before compiling Sub-task Resolved Gavin Halliday
          Report a warning if new archive format is used in old systems. Sub-task New Unassigned
          Option to exclude auto-fetched repos from an archive. Sub-task New Unassigned
          Document multi repository support Sub-task Resolved Greg Panagiotatos
          Generate an error if #semver used Sub-task Resolved Unassigned
          Support semantic versioning for repository dependencies Sub-task New Unassigned
          Add support for new archive format to EclWatch Sub-task Accepted Jaman Brundage
          Allow archives and links within node_modules directory Sub-task New Unassigned
          Support node_modules/module in package-lock.json Sub-task Resolved Unassigned
          package.json should not be processed in package-lock.json is present Sub-task Resolved Gavin Halliday
          Ensure git fetching does not have multi-threading issues Sub-task New Unassigned
          Ensure that git local storage can be configured cleanly on the cloud Sub-task Resolved Gavin Halliday
          Allow git passwords to be supplied from a vault as well as a k8s secret Sub-task Resolved Gavin Halliday
          Default --fetchrepos and --updaterepos on Sub-task Resolved Gavin Halliday
          Delay processing package dependencies until the module is used Sub-task Resolved Gavin Halliday
          Add --cleanrepos and --nodeimport options Sub-task New Unassigned
          Support https:// as well as git+https:// as a git repo locator Sub-task Resolved Gavin Halliday
          Investigate using App authentication for git Sub-task New Unassigned



              ghalliday Gavin Halliday
              ghalliday Gavin Halliday
              0 Vote for this issue
              13 Start watching this issue