If a subfile is added to an empty superfile, then a superfile with subfiles is added to that same empty superfile within a transaction an error occurs:
" addSubFile: <file1>'s record layout (fixed) is different than <file2>'s (dynamic) "
It's caused because the empty superfile which has had a subfile added inside the transaction, has not updated by the time the nested superfile is added.
When the nested superfile is added, it validates that it's subfiles are consistent with the outer super and checks it's attributes.
Unfortunately as it's within a transaction those attributes are out of date and in this case the record layout attribute is blank.