Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-13990

CSV Parsing limited to ~10M bytes ... regardless of MAXLENGTH() setting

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Not specified
    • Resolution: Duplicate
    • Affects Version/s: 5.2.4
    • Fix Version/s: 5.2.0
    • Component/s: None
    • Labels:
      None
    • Environment:
      Elsevier Disambiguation AWS development ... but this issue occurs in older platform versions as well.

      Description

      While importing CSV files (originating from hadoop) containing very long fields the following error was encountered:

      'System error: 99: Graph[1], pipethrough[7]: SLAVE #11 [172.31.23.63:20300]: CSV file contained a line of length greater than 10485760 bytes., '

      The file does indeed contain very long lines exceeding this limit (up to about 20M unicode characters in this case), but setting MAXLENGTH() did not allow processing to complete nor change the limit displayed in the error message. Because I happen to control the end-to-end environment I was able to work around the issue this time.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                brianb644 Brian Bounds
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: