There are several places in the system which use different formats for serializing information about fields:
a) text of the record structure
b) fileview2 internal format (derived from ( a )
c) IndexFileMapping (field translation) using a format generated for Pete
d) field information output in the generated code used by toXML etc.
e) filtering inside the roxie debugger
It would be much more sensible if these all used the same format. The proposal is to use the structures that are generated in the generated code, make them serializable, and provide functions that
a) convert an IHqlExpression to this form
b) prefetch a record
c) calculate the record size.
d) Any other useful functions.
e) Provide efficient mappings from one format to another (e.g., field projection)
f) prefetch a record and project at the same time.
g) filter a row against a set of constant field conditions.
If the prefetch helper was very efficient then we could consider removing the specially generated prefetch functions from the generated code.
That serialized format would then be used in all the places that currently process fields. It could also be use to serialize a filter/project to a remote location/process. We could also use it to recover from situations where the disk file doesn't match the expected format because extra fields have been added. (Apply the index file mapping to standard files as well.)