Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24449

New Support for dependencies between multiple repositories

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Accepted
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Code Generator
    • Labels:
    • Compatibility:
      Point
    • Roadmap:
      2020q3
    • Roadmap Summary:
      Support for better dependencies between multiple repositories

      Description

       
      Currently it is hard to represent the ECL for a query in more than repository.  This can either lead to lots of teams developing in the same repository - with independent branches that don't tend to get merged, or the code being cloned into another repository, with even bigger problems about keeping in sync.

      One of the reasons it is a problem is the way imports are implemented.  The code generator assumes all the import locations are merged into a single module hierarchy.  This can cause problems with symbols from multiple repository clashing.  Alternatively the different repositories can be treated as if they are in different sub directories, but then there are problems with relative v absolute imports.
       
       
       
      As a consequence, I started thinking about the current processing model and wondering if we could change it.  In particular, allow the different repositories to live in different global scopes, so the symbols could not clash, and each repository could continue to use absolute imports. 

      After various iterations, a model based on the npm package manager has been implemented. The aim is to allow the npm package manager to be used if desired, but for it to not be necessry.

      A Packages is a set of source code that is compiled independently of the code that uses it i.e. names will not clash. Packages appear as normal modules within the repository that use them.

      Normal process

      The normal process for defining dependencies on other packages is as follows:

      Define dependencies in a package.json file in the root of the repository:

      {
        "dependencies": {
          "demoRepoD": "ghalliday/gch-ecldemo-d#version3"
      }
      

      Where ghalliday/gch-ecldemo-d is the name of the git repository containing the code, and version3 is a branch or tag for the version of the code that. (A SHA can also be used). This package.json should be versioned in the source control system along with the rest of the code.

      The attributes within the package can then be used adding "import demoRepoD;" to an ecl definition.

      If eclcc is used with the --fetchrepos and --updaterepos options eclcc will take care of fetching the code from the remote git repositories.

      npm package manager

      It is possible to use the npm package manager to ensure that the labels or branches are mapped to a specific version. If the following command is used:

      npm install --package-lock-only
      

      The --package-lock-only indicates that npm will create a package-lock.json file in the same location as package.json, but not check out the associated versions of the code. The package-lock.json file will contain something similar to the following:

      {
        "packages": {
          "node_modules/demoRepoD": {
            "resolved": "git+ssh://github.com/ghalliday/gch-ecldemo-d.git#644c4585221f4dd80ca1e8f05974983455a244e5",
          }
        }
      }
      

      This package-lock.json file will take precedence over the package.json file. This can be compiled in the same way with eclcc automatically downloading the dependencies.

      This option also allows you to use npm's semantic versioning (#semver) which is not currently supported by eclcc.

      (NOTE: Currently the node_modules/ prefix will need to be deleted from the entry once it is generated. That is likely to be fixed before 8.4.0 is released.)

      Finally npm install can be used without the --package-lock-only file. This will check out the appropriate version of the code into the node_modules subdirectory of the current project. You can then compile the code using eclcc completely independently from the source control system.

      Package precedence

      As a result of the above, packages are defined in one of the following 3 ways - and have the following precedence:

      1. Contents of the node_modules/ directory.
      Any directories within this directory are treated as packages, and compiled separately. Although normally created by an npm install, the node_modules directory could also contain git submodules, each of which would be compiled as a separate package. This has not been tested, and there may be issues (e.g. whether submodules work with githooks.)

      2. Entries in a package-lock.json file.

      3. Entries in a package-lock.json file.

      Other considerations

      • Different repositories are allowed to use different versions of packages
         

        Attachments

          Issue Links

          1.
          Add an option to suppress plugins (and std) from archives Sub-task Resolved Gavin Halliday
          2.
          Implement dependency support using package-lock.json Sub-task Resolved Gavin Halliday
          3.
          Add archive support for compiling from multiple repositories Sub-task Resolved Gavin Halliday
          4.
          Add an option to directly compile a query from a repository Sub-task Resolved Unassigned
          5.
          Add support for package.json Sub-task Resolved Gavin Halliday
          6.
          Allow the default repo directory to be configured for eclccserver Sub-task New Unassigned
          7.
          Document and release test cases. Sub-task New Unassigned
          8.
          Generate a warning if multiple repository versions are used by a query. Sub-task New Unassigned
          9.
          Allow default git repo to be configured Sub-task Resolved Gavin Halliday
          10.
          review git authentication options Sub-task New Unassigned
          11.
          Allow source code dependencies to be resolved before compiling Sub-task Resolved Gavin Halliday
          12.
          Report a warning if new archive format is used in old systems. Sub-task New Unassigned
          13.
          Option to exclude auto-fetched repos from an archive. Sub-task New Unassigned
          14.
          Document multi repository support Sub-task Accepted Jim DeFabia
          15.
          Generate an error if #semver used Sub-task Resolved Unassigned
          16.
          Support semantic versioning for repository dependencies Sub-task New Unassigned
          17.
          Add support for new archive format to EclWatch Sub-task Accepted Jaman Brundage
          18.
          Allow archives and links within node_modules directory Sub-task New Unassigned
          19.
          Support node_modules/module in package-lock.json Sub-task New Unassigned
          20.
          package.json should not be processed in package-lock.json is present Sub-task Resolved Gavin Halliday
          21.
          Ensure git fetching does not have multi-threading issues Sub-task New Unassigned

            Activity

              People

              Assignee:
              ghalliday Gavin Halliday
              Reporter:
              ghalliday Gavin Halliday
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated: