Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-19155

Thor's lazy MRU file cache can cause failures accessing closed files

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.2.0
    • Component/s: Thor
    • Labels:
      None

      Description

      Several bugs in Thor's MRU file cache that could cause files to be accessed after having been closed by the MRU cache.
      As a result queries could fail with checked_pread failures on local files or RFSERR_InvalidFileIOHandle errors if dealing with a remote file on dafilesrv.

      The main problem was that the implementation incorrectly allowed the client of a cached file to retrieve the underlying IFileIO, allowing the cache to close the file whilst still in use.

      Original report:

      Dafilesrv can report RFSERR_InvalidFileIOHandle, as in:

      0005C453 2018-02-21 14:34:50.852 1547513 1675693 "Connect from 10.144.190.71:51670"
      0005C454 2018-02-21 14:34:50.856 1547513 1675704 "ERR(8002) 10.144.190.124:55652 : RFSERR_InvalidFileIOHandle"
      0005C455 2018-02-21 14:34:50.857 1547513 1675706 "Connect from 10.144.190.1:26986"
      

      and clients fail:

      0066838E 2018-02-21 09:58:44.940 1548475 1393796 " - graph(graph1, 4) : Graph graph1[4], keyedjoin[6]: SLAVE #1 [10.144.190.6:17000]: /var/lib/HPCCSystems/hpcc-data/thor/xcr_cert/enrichmentmaster/level-4/w20180213-203541/dmr/cross_link/defs_keyed_linkvalue._32_of_201[10.144.190.32:7100] RFSERR_InvalidFileIOHandle,  - caused by (8002, /var/lib/HPCCSystems/hpcc-data/thor/xcr_cert/enrichmentmaster/level-4/w20180213-203541/dmr/cross_link/defs_keye
      d_linkvalue._32_of_201[10.144.190.32:7100] RFSERR_InvalidFileIOHandle)"
      

      From debugging it appears client has sent a handle value of 0 to dafilesrv which is why it cannot be found and produces the error.
      Client should not send handle==0 to dafilesrv but either throw an exception or re-establish a file handle/connection to dafilesrv.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jakesmith Jake Smith
                Reporter:
                mckellyln Mark Kelly
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: