Well, look at it this way. The Yuuzhan Vong were originally a Force sensitive species that existed on Yuuzhan'tar in another galaxy.
Circa 15000 BBY the Cremlevian War ends, so we know the Yuuzhan Vong had definitely been stripped of the Force by then. We know their technology was based on gravity and their darkspace drives relied upon locking upon planets to jump.
We understand that the Silentium and Abominor originated in their galaxy and their conflict is what radicalised the Yuuzhan Vong, the original Yuuzhan'tar producing weapons which fended them off, and they went on a mission of conquest. This results in them depopulating their galaxy and moving by sublight towards our galaxy. Smaller vessels could advance much quicker, and one scout reached our galaxy circa 3960 BBY, the latter of which assisted in the secession of the Kanz Sector and added to the chaos of the era, after fending off Mandalorian's who attacked her ship in the Crispin system. Eventually Yuuzhan'tar's child, Zonama Sekot, arrives in our galaxy, in an unknown date. The first scouts hit our galaxy in 32 BBY and the armada in 25 ABY.
This is information we all (and both) know.
We also know that the Yuuzhan Vong language was impossibly linked to a deceased species here. We also know that Sekot moved to our galaxy without the aid of hyperdrives in the sense of mechanical ones.
So this raises questions.
1. Why did all three species flee here?
2. Why did the Yuuzhan Vong have the same language as a species from here?
3. How did the Yuuzhan'tar produce weapons do quickly?
4. What was the mechanic for the removal of the Force? Normal stripping is not to the extent that it is genetic and then hereditary.
So, let's assume that humans were the medium in which the Infinite Sith ruled the galaxy, and, to a greater of lesser extent, all species acknowledged that humanity represented the future of the galaxy. Let's assume that the Infinite Sith, having anchored a Sith God to Yuuzhan'tar, withdraw their thralls from the galaxy using the Force and then prepare for a counter-invasion when the Celestials make their vast move.
Let's also assume that the Infinite Sith, having secured this galaxy by 100,000 BBY, turned to further galaxies and gathered that the only way they could assemble a reasonable force of strength that would not tear itself apart in the travel (as is the Sith way) was to make it mechanical. The Abominor are sent to conquer another galaxy, but in the mainstream galaxy the Celestials also follow the selfsame logic and create and dispatch the Silentium, what with the need to focus their efforts upon the Infinite Sith here.
So as time passes and the Infinite Sith do not recover in this galaxy, Yuuzhan'tar heads to the Vong galaxy and the species devolves into primitivism. However, the stated objective of creating a species which can also move planets, undo gravitational binds and so forth, is completed by way of advanced alchemy, and the technology is such that it would be considered superior to any in the Sith galaxy.
Yuuzhan'tar settles to watch out the Silentium and Abominor War, before being dragged in and required to destroy them or lose the real prize; the ur-Yuuzhan Vong, alchemically modified to be longer lived, be stronger, and more near human than human. This sees the Yuuzhan Vong seize their galaxy and war upon themselves, with Yuuzhan'tar reaching the critical mass, feeding from all the death and so forth, to split his consciousness into a second planet sized avatar. However, he loses control of the Yuuzhan Vong at the critical moment and puts into place the last element to make the Yuuzhan Vong truly a terrifying threat to the Celestials; the ability to be immune to the Force. Having modified the technology which itself would eventually strip the Rakata of the Force, Yuuzhan'tar seeks to make it's thralls a lesser threat but they instead destroy him. His second avatar, not fully formed, is scarred by the blow, and flees the galaxy as Zonama Sekot, before descending into madness in the Void and wiping it's own memory clean to cope, returning home, much as the Silentium and Abominor fled back here to find their creators, long forgotten, arguably, deceased. Or perhaps not, what with the Silentium decision to attack Pakrith with Lando... Taking down Andeddu's legacy, perhaps, as an Infinite Sith? (incidentally I assume Andeddu succeeds Pall and Hord drives him off on his return).
As such the Yuuzhan Vong War is an actual invasion by the Infinite Sith, as demonstrated by the darkness they brought with them into this galaxy that Jacen sent away.