Prestige Victories seem to come fast?

Is it just me? 

First game I played, another empire came really close before I'd explored half available sectors, and I ended up rushing a prestige victory to win. Second, I got distracted by a war and another empire got a prestige victory. Both games were under 200 turns.

I like prestige victories but it feels like I need to disable them to get the full benefit of playing on larger maps with lots of opponents.