Would the white walkers have been able to cross the wall without the dragon?

Rewatching game of thrones here. I feel like Jon was panicking for nothing about the white walkers. They have been beyond the wall for many years.

His panic brought him beyond the wall again, and the that’s how the walkers got the dragon.

So I feel like if Jon would just shut up, be king of the north, mind his business, there wouldn’t have been no walkers crossing in the first place.

Please tell me if my thinking is correct.