The issue that prompted this discussion in the TUX meeting is that we would like to add a restricted source to the UK Header. The restriction on the source is that it cannot be viewed from outside the US. We use US developers on the UK Header. So, if we add this source, then we either need to discontinue US development on the UK Header or we need to implement new security features to prevent US access to the header records from just this source.
The purpose of this ticket is to discuss this particular UK Header need along with related/future needs in order to come up with a vision/specification.
Potential points of access to the data include:
- eclwatch access to datasets
- eclwatch access to WU outputs
- thor access to datasets through ECL
- roxie access to indexes through a service
Potential approaches to the UK Header need include:
- encapsulation of the data with ECL enforcement of permissions
- monitoring and punishing unauthorized access
- using subfiles with different file scopes
- discontinuing development from US
Some of these can be combined (such as 2 and 1). 3 is very promising, but also comes with concerns about whether to use subfiles within indexes (performance cost) and how builds/research might suffer if records were silently dropped for some developers.
An additional concern, obvious at the index level, is that we often do propagation and rolling up and do not track the original sources of each data point.
Unfortunately, a magical solution, one which somehow guaranteed that US developers could only ever see data with a field called SOURCE containing a value in ['AA','BB','CC'], would be crippling to our current style of header development (because so much of the data within the header process comes from combined sources).
Flavio Villanustre has related literature to share.
John Holt has related needs/requirements.
Russ Whitehead has an intern working on related features.
Please correct me if I have misrepresented you