Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
None
-
None
Description
The reported error is:
- Other problem(s) code: 0 ( 1 item(s) )
Test case Error message WUID (link to ZAP file) 1. SVTest2 Result doesn't match to key file W20221209-073348
From the test log:
Error: 1. Test: SVTest2.ecl '--- SVTest2.xml +++ SVTest2.xml @@ -2,7 +2,7 @@ <Row><Test1>Pass</Test1></Row> </Dataset> <Dataset name=Test2> - <Row><Test2>Pass</Test2></Row> + <Row><Test2>This word is not present but the vec field is not empty</Test2></Row> </Dataset> <Dataset name=Test3> <Row><Test3>Pass</Test3></Row>
The releated code fragment is:
Sentences := DATASET([{1, 'Greetings everyone'},
{2, 'The [sky] is blue'},
{3, 'The house has four ^rooms'},
{4, 'HPCC-Systems programming uses ecl'}], Sentence);
Model := SV.GetModel(sentences);
Words := DATASET([{1, 'The'}, {2, 'house'}, {3, 'has'}, {4, 'rooms'}, {5, 'zoo'}], Word);
OUTPUT(Words, NAMED('Words'));
Vectors := SV.GetWordVectors(Model, Words);
OUTPUT(Vectors, {wordid}, NAMED('Vectors'));
// The last vector word is not present in the model so it should have an empty vec field
OUTPUT(IF(Vectors[5].vec = [], 'Pass', 'This word is not present but the vec field is not empty'), NAMED('Test2'));
The test code implementation had been tested in an environment with 1 or 4 Thor slaves and it passed. However, in OBT the bundle executes in 6 Thor slaves environment and always fails.
I ran some tests with different numbers of Thor slaves, and it revealed the word order in Vectors may be different depending on the slaves.
| Vectors' record index | wordid for 1 slave | wordid for 2 slaves | wordid for 4 slaves | wordid for 6 slaves | wordid for 8 slaves | wordid for 10 slaves |
|---|---|---|---|---|---|---|
| 1 | 3 | 1 | 1 | 3 | 2 | 1 |
| 2 | 2 | 3 | 3 | 1 | 4 | 3 |
| 3 | 4 | 2 | 2 | 5 | 1 | 2 |
| 4 | 1 | 4 | 4 | 2 | 3 | 4 |
| 5 | 5 | 5 | 5 | 4 | 5 | 5 |
Related information from Lili:
I believe what’s happening here is that the model is distributed to each node. Thus as the number of nodes changes, the vector id for same word can change as well. That’s why the same word has different vector ID when the cluster nodes increase or decrease.
and Roger:
Word vectors start out with random values, and need a lot of data to converge. You won’t get any consistent results with such a small corpus. Also, the vectors need to be trained to a low loss (which you won’t be able to do with that corpus), but training on multiple nodes will require more iterations to reach the same loss, though they will get there faster. So you should expect random results, even on the same number of nodes.
So, it was a bit naïve approach or misleading result of the test in 1 or 4 slaves Thor environment to expect the 5th element of the Vectors always refer to the 5th element of the Words dataset.
