Now it did took me quite awhile to 'host' said party at those enclaves base, lots of noisemaker and horn going out as well, but the moment 4 of those enclaves finally joined along with the rest of the invited neighbours, the last mission for Warlord's Legacy popped-up to me, and I can see why it didn't do that before. There's an alternative way to make them leave the map which is to ignore their request for help the next time it popped up so that they will leave the map, but then again it's pretty long gap to wait for each enclaves to request some half-hearted quest just to remove them from map, which lead to my next Warlord-like decision: Simply invite few 'roaming neighbour' to the base of said enclaves that you happen to decide not worth your time (The more the merrier).
Community Driven World: Each survivor in the game is their own person, with a unique mix of background traits. Players can build a lasting community working with other players or solo to overcome challenges while exploring a world filled with blood thirsty zombies. Now the problem for me is that all enclaves in my map is currently allied status and that I'm at maxed community members. State of Decay 2 is the sequel to the survival fantasy franchise by Undead Labs. The solution is to disband the enclaves via recruiting member (if said enclave are allied) or threaten them (if said enclave relationship is lower than allied).
I found this from reddit that when you have more than 11 enclaves in your map (15 enclaves in my case during my playthrough) the final mission for Warlord Legacy refuse to start.
Not sure if the info worth much by now, but might just put this as reference.