The Vishanti, or in this case a third of them, specifically Agamotto, although extremely powerful cannot match the might of the spectre.
Together the Vishanti might pose a threat but not individually. The Spectres power and durability is too great. By himself he can effect massive planet wide changes. Agamotto does not possess this scope of power.
Spectre has shown to be able to defeat and destroy hugely powerful mystical beings like Shazam, demons/devils of name and easily defeat beings like Dr Fate (a Lord of Order).