Details
Description
Break the reliance on direct Dali connectivity for logical files.
Currently the DFS interfaces connect directly to Dali and pull meta data on demand.
This means that for inter-environment file access the Dali in external environments would need to be open. This is how foreign files work, but it is inherently inseucre, and must be secure with whitelists and firewalls at a bare minimum.
In the cloud, Dali is locked down, as an internal service only. Nothing from the outside can connect to it.
A Esp DFS service will provide interfaces to clients, including from other environments, such that they can pull DFS logical file meta data and read logical files, without relying on Dali connectivity.
Steps:
+ Introduce a 'remote' keyword for logical files, e.g. ~remote::otherenv::somescope::somefile
+ Implement Esp DFS service and interfaces to pull existing meta data
+ For ease of transition/compatibiltiy etc., reimplement the existing DFS interfaces that utilize this meta data. i.e. so existing engines/usages are transparent to the different implementations.
+ Allow remote mappings to be defined, that will define the remote service (it's protocol, host/ip/port, etc) and the local plane details that mount the remote storage.
+ Implement a locking protocol. This will require a new non-session based locking mechanism that is currently ubiquous and provided by Dali. Instead a named lock will be created with a lease and a keep alive timeout. Clients must close or keep the lock alive otherwise it will self-close. This needs to be inter-operate with existing locks, since both new and old DFS interfaces will co-operate.
+ Implement write/create functionality.
Phase 1 will implement reads from exteranl (remote) files, without locking, which is akin to existing ~foreign file access, which is also read-only without locking.
Attachments
Issue Links
- relates to
-
HPCC-27294 Enable remote dfs client access from hthor and roxie
-
- Resolved
-
-
HPCC-26994 Document remote storage plane configuration
-
- Resolved
-