Support the option to sign code

There are several situations where it would improve security if access to different features could be controlled.

• Which functions are constant folded at compile time
• Which uses of pipe can be supported.
• Which uses of embedded C++ and other language are allowed.

At the moment we have all or nothing support for some of these, but it would be better to have a secure fine-grained approach.

This could be supported by extending the system in the following ways

• Adding new attributes onto service definitions and other activities (e.g., PIPE, EMBEDDED, ...) to indicate that the feature is allowed (possibly only if the user has a specific right')

• Allowing ecl definitions to be signed, so they can be verified that they come from a particular source and haven't been modified.

• Adding options to the system to requiring that some features (e.g., constant folding/external C++) are only allowed in a query if they have the appropriate attributes, defined in a signed module, and the user has the appropriate rights to use the attribute.

• Add an option to restrict deploying from dlls/other non controlled sources. Preventing for any other ways of deploying code that wouldn't have been checked.

Some complications

• What is the signature derived from. The source of the attribute would be simple, but if the attribute has dependencies then they wouldn't be included.
  richardkchapman has suggested using the digest that is generated for the expression which is used for determining when persists need rebuilding. That has disadvantages that it does sometime change from release to release - e.g., if the expressions that are generated by the parser change - either because of a new optimization, or a change in the representation.

• ecl ide, the eclipse plugin and the build system need to be able to sign an ecl attribute using a private key. If it is based on the digest of the code then they will need to call eclcc to generate it.

• eclcc that builds archives may not have access to the public keys, and will not be able to check ldap permissions. That should be fine since it will not be generating code that runs on the platform.

• There would need to be multiple public/private pairs.
  HPCC systems would use one to sign attributes for the binary builds it produces.
  Users would be free to use their own when building their own copies of the system.
  Users would define their own to allow access to embedded c++ etc. if that feature is locked down.
  Multiple keys will allow different groups to independent sign.
  ?Should the keys have an associated name which is used when signing the definitions, or should each key be tried in turn. (I think I prefer named - it may also simplify the user interface for next option..)

Valid keys should be configurable per-cluster - since development may be less restrictive than a production system.

Some Questions:

• What is the full list of rights to be controlled?
• What control do we need?
• What should the signature be based on?
• What ECL syntax is used to sign a definition (?A special format comment, if so what format?)
• What attributes need to be added to control the rights.

• Should we also revisit the encryption support, and reimplement that with configurable keys?

jakesmith gordonsmith mckellyln please read and comment.

Related to HPCC-13173