-
Type:
Improvement
-
Status: Accepted
-
Priority:
Minor
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Labels:
-
Compatibility:Minor
-
Roadmap:Future
-
Roadmap Summary:Reduce the latency for executing complex queries.
It looks like a Jira was never created for this issue, so creating one now (cc David Bayliss)
Currently the workflow engine executes a single item at a time, and waits for that workflow item to complete before continuing. Some jobs would benefit significantly if separate persists or independent actions were executed in parallel.
The workflow information already contains all the dependencies, and information about items that need to be executed sequentially. What would be required would be
i) Restructure the workflow engine to create a graph of tasks that can be use to track which tasks have been executed, and which tasks should be executed next.
ii) Ensure that there are no multi threading issues in the workflow engine (e.g. the way persist information is calculated).
iii) Check eclagent for any multi threading issues
It may be sensible to initially only support this for roxie/thor, and then revisit hthor.
This possibly becomes even more significant on cloud environments since it would be quite possible to spin up extra thors on demand, so allow multiple graphs to be processed in parallel.