Probability plays the greatest part in it. If anyone could positively identify a world as their own, it would seem much more likely, but as it is... well, I suppose it's really a moot point. If these are real worlds, the evidence suggests the City's moving to places no Citizens have come from, meaning it's unlikely that this will prove to be a way home.