Is it worth swapping out the factory exhaust system for a performance upgrade?

Considering upgrading to a high-performance exhaust system on my Lexus, seeking input on potential benefits and drawbacks.