Using Set::merge() on an empty array causes issues with out of order numeric keys. Only merge if necessary. Fixes #2595