Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24407

Regression tests that fail should be meaningfully compared

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.10.6
    • Component/s: Regression Test Engine
    • Labels:
      None

      Description

      The output for a successful test is like the following:

      <Dataset name='logging'>
      <Row><value>5</value></Row>
      </Dataset>

      The output for a failed test is like the following:

      Error (1001): Command return code non-zero.
      retCode: 4
      'W20200710-092435 failed

      Deploying ECL Archive /home/nathan/HPCCSystems-regression/archives/archive_workflow_failure_1.xml

      Deployed
      wuid: W20200710-092435
      state: compiled

      Running deployed workunit W20200710-092435
      <Warning><Source>eclagent</Source><Message>Unknown clause failed (execution will continue): 0: </Message></Warning>
      <Exception><Code>1000</Code><Source></Source><Message>Workunit terminated unexpectedly</Message></Exception>
      <Dataset name='logging'>
      <Row><value>1</value></Row>
      <Row><value>2</value></Row>
      <Row><value>3</value></Row>
      </Dataset>

      It would be useful if there was a way to compare the outputs even though the workunit fails

      If //fail is used, then the regression test will always pass, even if the program doesn't produce the desired output.

      If //fail is not used, then the regression test will fail, because each workunit has a unique WUID. Furthermore, it appears as though the output is not compared like a typical test, since the display is missing (+/-) to show differences between lines.

      Here is the full display after a workunit fails

      [Action] Suite: roxie
      [Action] Queries: 1
      [Action] 1. Test: workflow_failure_1.ecl
      [Failure] 1. Fail workflow_failure_1.ecl - W20200710-093720 (0 sec)
      [Failure] 1. URL http://127.0.0.1:8010/?Widget=WUDetailsWidget&Wuid=W20200710-093720
      [Failure] 1. Zipped Analysis Package: ZAP file written to /home/nathan/HPCCSystems-regression/zap/ZAPReport_W20200710-093720_regress.zip.

      [Action]
      -------------------------------------------------
      Result:
      Passing: 0
      Failure: 1
      -------------------------------------------------
      Error:
      1. Test: workflow_failure_1.ecl
      Error (1001): Command return code non-zero.
      retCode: 4
      'W20200710-093720 failed

      Deploying ECL Archive /home/nathan/HPCCSystems-regression/archives/archive_workflow_failure_1.xml

      Deployed
      wuid: W20200710-093720
      state: compiled

      Running deployed workunit W20200710-093720
      <Warning><Source>user</Source><Message>FAILURE clause failed (execution will continue): 0: </Message></Warning>
      <Exception><Source>Roxie</Source><Message></Message></Exception>
      <Dataset name='logging'>
      <Row><value>1</value></Row>
      <Row><value>2</value></Row>
      <Row><value>3</value></Row>
      </Dataset>

      -------------------------------------------------
      Log: /home/nathan/HPCCSystems-regression/log/roxie.20-07-10-09-37-19.log
      -------------------------------------------------
      Elapsed time: 3 sec (00:00:03)
      -------------------------------------------------

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                attilavamos Attila Vamos
                Reporter:
                Nathan Halliday Nathan Halliday
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: