Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-13966

Error when deleting multiple files

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 5.2.4
    • None
    • DFU Server

    Description

      I have been trying to delete some files on this cluster using ECL Watch but they won’t delete. The current version of ECL Watch just continues to refresh. Switched to the legacy version and attempted again but I got this message:

      Proxy Error
      The proxy server received an invalid response from an upstream server.
      The proxy server could not handle the request POST /WsDFU/DFUArrayAction.
      Reason: Error reading from remote server

      This has happened to me multiple times. Fernando at RIS-BCT Supercomputer Ops - Boca said couple of weeks ago that:

      It looks like the esp OOM the box.

ISO 9660 Extensions: RRIP_1991A
eth3: no IPv6 routers present
type=1305 audit(1408739774.478:29649): auid=4294967295 ses=4294967295 op="remove rule" key=(null) list=4 res=1
type=1305 audit(1408739774.478:29650): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
readahead-collector: starting delayed service auditd
readahead-collector: sorting
readahead-collector: finished
hrtimer: interrupt took 6377604 ns
NOHZ: local_softirq_pending 100
NOHZ: local_softirq_pending 100
esp[10347]: segfault at 40 ip 00007f06e26bf59c sp 00007f0647ffcf30 error 4 in libdalibase.so[7f06e257b000+18d000]
esp invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
esp cpuset=/ mems_allowed=0
Pid: 26274, comm: esp Not tainted 2.6.32-279.5.2.el6.x86_64 #1


Looks like it went off track yesterday…


esp_main.2015_07_05.log:00018928 2015-07-05 20:34:49.640  3615 25781 "Time taken for WUDELETE cleanupAndDelete total: 8478900 cycles (8M) = 2 msec"
esp_main.2015_07_05.log:00018B6D 2015-07-05 20:43:01.737  3615 26186 "Time taken for WUDELETE cleanupAndDelete total: 7978651 cycles (7M) = 2 msec"
esp_main.2015_07_05.log:00018CE5 2015-07-05 20:47:55.023  3615 26378 "Time taken for WUDELETE cleanupAndDelete total: 10043183 cycles (10M) = 3 msec"
esp_main.2015_07_05.log:00018D58 2015-07-05 20:49:32.186  3615 26476 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::prim_name on thor: file not found"
esp_main.2015_07_05.log:00018D8A 2015-07-05 20:49:46.294  3615 26497 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::prim_name on thor: file not found"
esp_main.2015_07_05.log:00018D8B 2015-07-05 20:49:46.298  3615 26497 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::state on thor: file not found"
esp_main.2015_07_05.log:000193A2 2015-07-05 20:55:41.341  3615 26476 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::datafile::patched_candidates on thor: file not found"
esp_main.2015_07_05.log:000193A3 2015-07-05 20:55:41.949  3615 26497 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::property on thor: file not found"
esp_main.2015_07_05.log:000193A6 2015-07-05 20:55:42.231  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::zip4 on thor: file not found"
esp_main.2015_07_05.log:000193A7 2015-07-05 20:55:42.234  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::fein on thor: file not found"
esp_main.2015_07_05.log:000193A8 2015-07-05 20:55:42.236  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::unit_desig on thor: file not found"
esp_main.2015_07_05.log:000193AC 2015-07-05 20:55:42.781  3615 26476 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::unit_desig on thor: file not found"
esp_main.2015_07_05.log:000193AD 2015-07-05 20:55:42.803  3615 26476 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:000193AE 2015-07-05 20:55:42.910  3615 26497 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:000193AF 2015-07-05 20:55:43.024  3615 26631 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::unit_desig on thor: file not found"
esp_main.2015_07_05.log:000193B0 2015-07-05 20:55:43.149  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::addr_suffix on thor: file not found"
esp_main.2015_07_05.log:000193B1 2015-07-05 20:55:43.152  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::prim_range on thor: file not found"
esp_main.2015_07_05.log:000193B2 2015-07-05 20:55:43.265  3615 26414 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::state on thor: file not found"
esp_main.2015_07_05.log:000193B3 2015-07-05 20:55:43.267  3615 26414 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::specificities on thor: file not found"
esp_main.2015_07_05.log:000193B4 2015-07-05 20:55:44.082  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::city on thor: file not found"
esp_main.2015_07_05.log:000193B5 2015-07-05 20:55:44.197  3615 26414 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::property on thor: file not found"
esp_main.2015_07_05.log:000193B6 2015-07-05 20:55:44.199  3615 26414 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::unit_desig on thor: file not found"
esp_main.2015_07_05.log:000193B7 2015-07-05 20:55:44.201  3615 26414 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::sec_range on thor: file not found"
esp_main.2015_07_05.log:000193B8 2015-07-05 20:55:44.203  3615 26414 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::zip on thor: file not found"
esp_main.2015_07_05.log:000193BB 2015-07-05 20:55:44.320  3615 26631 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:000193BC 2015-07-05 20:55:44.698  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::bankruptcy on thor: file not found"
esp_main.2015_07_05.log:000193BD 2015-07-05 20:55:44.828  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::vehicles on thor: file not found"
esp_main.2015_07_05.log:000193BE 2015-07-05 20:55:45.099  3615 27080 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::property on thor: file not found"
esp_main.2015_07_05.log:000193C9 2015-07-05 20:56:16.081  3615 29420 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::county on thor: file not found"
esp_main.2015_07_05.log:000193CA 2015-07-05 20:56:16.084  3615 29420 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::company_name on thor: file not found"
esp_main.2015_07_05.log:000193CB 2015-07-05 20:56:16.087  3615 29420 "CWsDfuEx::DFUDeleteFiles: Cannot delete key::trainingadvsalt::bdid::word::city on thor: file not found"
esp_main.2015_07_05.log:000193E1 2015-07-05 20:56:18.014  3615 26497 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:000193FA 2015-07-05 20:56:24.945  3615 26476 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:0001940B 2015-07-05 20:56:41.718  3615 26631 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019436 2015-07-05 20:57:30.380  3615 26497 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019437 2015-07-05 20:57:35.016  3615 26476 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:0001943A 2015-07-05 20:57:48.153  3615 29471 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019442 2015-07-05 20:58:01.232  3615 26631 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019443 2015-07-05 20:58:14.436  3615 26497 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:0001944B 2015-07-05 20:58:36.887  3615 26631 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:0001944C 2015-07-05 20:58:38.937  3615 26476 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019470 2015-07-05 20:58:50.184  3615 26497 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019476 2015-07-05 20:59:15.163  3615 29471 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019479 2015-07-05 20:59:16.211  3615 26476 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:0001947D 2015-07-05 20:59:21.000  3615 26631 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019486 2015-07-05 20:59:51.240  3615 29471 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:00019487 2015-07-05 20:59:56.940  3615 26497 "CDelayedDelete: pausing"
esp_main.2015_07_05.log:000194AA 2015-07-05 21:00:05.915  3615 26476 "CDelayedDelete: pausing"

…

esp_main.2015_07_06.log:00025BDB 2015-07-06 14:31:36.459  3615 30430 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BDC 2015-07-06 14:31:39.073  3615  1245 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BDD 2015-07-06 14:31:41.814  3615 29969 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BDE 2015-07-06 14:32:08.689  3615 29811 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BDF 2015-07-06 14:32:08.776  3615 30430 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BE0 2015-07-06 14:32:09.124  3615 26497 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BE1 2015-07-06 14:32:10.725  3615 26476 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BE2 2015-07-06 14:32:10.966  3615 30285 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BE3 2015-07-06 14:32:11.599  3615   842 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BE4 2015-07-06 14:32:11.728  3615  1245 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BE5 2015-07-06 14:32:12.929  3615 19817 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BE6 2015-07-06 14:32:14.224  3615 26631 "CDelayedDelete: pausing"
esp_main.2015_07_06.log:00025BE9 2015-07-06 14:32:20.472  3615 29471 "CDelayedDelete: pausing"

      Attachments

        Activity

          People

            miguelvazq Miguel Vazquez (Inactive)
            micevepay Michael Payne
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: