New Support for dependencies between multiple repositories

 
Currently it is hard to represent the ECL for a query in more than repository.  This can either lead to lots of teams developing in the same repository - with independent branches that don't tend to get merged, or the code being cloned into another repository, with even bigger problems about keeping in sync.

One of the reasons it is a problem is the way imports are implemented.  The code generator assumes all the import locations are merged into a single module hierarchy.  This can cause problems with symbols from multiple repository clashing.  Alternatively the different repositories can be treated as if they are in different sub directories, but then there are problems with relative v absolute imports.
 
 
 
As a consequence, I started thinking about the current processing model and wondering if we could change it.  In particular, allow the different repositories to live in different global scopes, so the symbols could not clash, and each repository could continue to use absolute imports. 

After various iterations, a model based on the npm package manager has been implemented. The aim is to allow the npm package manager to be used if desired, but for it to not be necessry.

A Packages is a set of source code that is compiled independently of the code that uses it i.e. names will not clash. Packages appear as normal modules within the repository that use them.

Normal process

The normal process for defining dependencies on other packages is as follows:

Define dependencies in a package.json file in the root of the repository:

{
  "dependencies": {
    "demoRepoD": "ghalliday/gch-ecldemo-d#version3"
}

Where ghalliday/gch-ecldemo-d is the name of the git repository containing the code, and version3 is a branch or tag for the version of the code that. (A SHA can also be used). This package.json should be versioned in the source control system along with the rest of the code.

The attributes within the package can then be used adding "import demoRepoD;" to an ecl definition.

If eclcc is used with the --fetchrepos and --updaterepos options eclcc will take care of fetching the code from the remote git repositories.

npm package manager

It is possible to use the npm package manager to ensure that the labels or branches are mapped to a specific version. If the following command is used:

npm install --package-lock-only

The --package-lock-only indicates that npm will create a package-lock.json file in the same location as package.json, but not check out the associated versions of the code. The package-lock.json file will contain something similar to the following:

{
  "packages": {
    "node_modules/demoRepoD": {
      "resolved": "https://github.com/ghalliday/gch-ecldemo-d.git#644c4585221f4dd80ca1e8f05974983455a244e5",
    }
  }
}

This package-lock.json file will take precedence over the package.json file. This can be compiled in the same way with eclcc automatically downloading the dependencies.

This option also allows you to use npm's semantic versioning (#semver) which is not currently supported by eclcc.

(NOTE: Currently the node_modules/ prefix will need to be deleted from the entry once it is generated. That is likely to be fixed before 8.4.0 is released.)

Finally npm install can be used without the --package-lock-only file. This will check out the appropriate version of the code into the node_modules subdirectory of the current project. You can then compile the code using eclcc completely independently from the source control system.

Package precedence

As a result of the above, packages are defined in one of the following 3 ways - and have the following precedence:

1. Contents of the node_modules/ directory.
Any directories within this directory are treated as packages, and compiled separately. Although normally created by an npm install, the node_modules directory could also contain git submodules, each of which would be compiled as a separate package. This has not been tested, and there may be issues (e.g. whether submodules work with githooks.)

2. Entries in a package-lock.json file.

3. Entries in a package.json file.

Other considerations

• Different repositories are allowed to use different versions of packages