Uploaded image for project: 'JAPI'
  1. JAPI
  2. JAPI-359

dfsclient should utilize dafilesrv commcompression

    XMLWordPrintable

Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • 7.12.x
    • None
    • dfsclient
    • None

    Description

      /* Notes on protocol:
      *

      • A JSON request with these top-level fields:
      • "format" - the format of the reply. Supported formats = "binary", "xml", "json"
      • "handle" - the handle of for a file session that was previously open (for continuation)
      • "commCompression" - compression format of the communication protocol. Supports "LZ4", "LZW", "FLZ" (TBD: "ZLIB")
      • "replyLimit" - Number of K to limit each reply size to. (default 1024)
      • "node" - contains all 'activity' properties below:
        *
      • For a secured dafilesrv (streaming protocol), requests will only be accepted if the meta blob ("metaInfo") has a matching signature.
      • The request must specify "filePart" (1 based) to denote the partition # being read from or written to.
        *
      • "filePartCopy" (1 based) defaults to 1
        *
      • "kind" - supported kinds = "diskread", "diskwrite", "indexread", "indexcount" (TBD: "diskcount", "indexwrite", "disklookup")
      • NB: disk vs index will be auto detected if "kind" is absent.
        *
      • "action" - supported actions = "count" (used if "kind" is auto-detected to specify count should be performed instead of read)
        *
      • "keyFilter" - filter the results by this expression (See: HPCC-18474 for more details).
        *
      • "chooseN" - maximum # of results to return
        *
      • "compressed" - specifies whether input file is compressed. NB: not relevant to "index" types. Default = false. Auto-detected.
        *
      • "input" - specifies layout on disk of the file being read.
        *
      • "output" - where relavant, specifies the output format to be returned
        *
      • "fileName" is only used for unsecured non signed connections (normally forbidden), and specifies the fully qualified path to a physical file.
        *
        */
        void cmdStreamCommon(IPropertyTree *requestTree, MemoryBuffer &rest, MemoryBuffer &reply, CRemoteClientHandler &client, CClientStats &stats)
        {
        /* Example JSON request:
        *
      • {
      • "format" : "binary",
      • "handle" : "1234",
      • "replyLimit" : "64",
      • "commCompression" : "LZ4",
      • "node" : {
      • "metaInfo" : "",
      • "filePart" : 2,
      • "filePartCopy" : 1,
      • "kind" : "diskread",
      • "fileName": "examplefilename",
      • "keyFilter" : "f1='1 '",
      • "chooseN" : 5,
      • "compressed" : "false"
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }

        ,

      • "output" : { * "f2" : "string", * "f1" : "real" * }
      • }
      • }
      • OR
      • {
      • "format" : "binary",
      • "handle" : "1234",
      • "replyLimit" : "64",
      • "commCompression" : "LZ4",
      • "node" : {
      • "kind" : "diskread",
      • "fileName": "examplefilename",
      • "keyFilter" : "f1='1 '",
      • "chooseN" : 5,
      • "compressed" : "false"
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }

        ,

      • "output" : { * "f2" : "string", * "f1" : "real" * }
      • }
      • }
      • OR
      • {
      • "format" : "xml",
      • "handle" : "1234",
      • "replyLimit" : "64",
      • "node" : {
      • "kind" : "diskread",
      • "fileName": "examplefilename",
      • "keyFilter" : "f1='1 '",
      • "chooseN" : 5,
      • "compressed" : "false"
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }

        ,

      • "output" : { * "f2" : "string", * "f1" : "real" * }
      • }
      • }
      • OR
      • {
      • "format" : "xml",
      • "handle" : "1234",
      • "node" : {
      • "kind" : "indexread",
      • "fileName": "examplefilename",
      • "keyFilter" : "f1='1 '",
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }

        ,

      • "output" : { * "f2" : "string", * "f1" : "real" * }
      • }
      • OR
      • {
      • "format" : "xml",
      • "node" : {
      • "kind" : "xmlread",
      • "fileName": "examplefilename",
      • "keyFilter" : "f1='1 '",
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }

        ,

      • "output" : { * "f2" : "string", * "f1" : "real" * }
      • "ActivityOptions" : { // usually not required, options here may override file meta info. * "rowTag" : "/Dataset/OtherRow" * }
      • }
      • OR
      • {
      • "format" : "xml",
      • "node" : {
      • "kind" : "csvread",
      • "fileName": "examplefilename",
      • "keyFilter" : "f1='1 '",
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }

        ,

      • "output" : { * "f2" : "string", * "f1" : "real" * }
      • "ActivityOptions" : { // usually not required, options here may override file meta info. * "csvQuote" : "\"", * "csvSeparate" : "," * "csvTerminate" : "\\n,\\r\\n", * }
      • }
      • OR
      • {
      • "format" : "xml",
      • "node" : {
      • "action" : "count", // if present performs count with/without filter and returns count
      • "fileName": "examplefilename", // can be either index or flat file
      • "keyFilter" : "f1='1 '",
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }

        ,

      • }
      • }
      • OR
      • {
      • "format" : "binary",
      • "handle" : "1234",
      • "replyLimit" : "64",
      • "commCompression" : "LZ4",
      • "node" : {
      • "kind" : "diskwrite",
      • "fileName": "examplefilename",
      • "compressed" : "false" (or "LZ4", "FLZ", "LZW")
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }
      • }
      • }
      • OR
      • {
      • "format" : "binary",
      • "handle" : "1234",
      • "replyLimit" : "64",
      • "node" : {
      • "kind" : "indexwrite",
      • "fileName": "examplefilename",
      • "input" : { * "f1" : "string5", * "f2" : "string5" * }
      • }
      • }
        *
        */

      Attachments

        Issue Links

          Activity

            People

              mcmuja01 James McMullan
              rpastrana Rodrigo Pastrana
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: