So King Kai simply waited years and years whilst watching it happen to make his statement about it being destroyed? That sounds reasonable... or maybe Broly destroyed it all at once which was visibly shown?
The argument that "there were still stars so the galaxy wasn't destroyed" is a complete fallacy and here's why. There is no logical way for a single blast to destroy every bit of matter in the galaxy (unless of course that blast were multi-galaxy busting in power but condensed, in which case it'd easily vaporize everything in the area. Evidently Broly's blast was not multi-galaxy busting).Think about it: you fire a sawed-off shotgun at two people standing next to each other and it will hit them both, killing them; then you take two others, and position one behind the other and fire, and the person in the front will be hit by the bullets whilst the other remains relatively unharmed.