They are easy enough to remove. Why don't you try it 1 day with out and see how you like it. Each boat reacts differently due to size, weight, water conditions, etc, etc. If you notice a difference and don't like it, then put it back on.
I've spun boats out with rudders installed so don't think it's impossible to spin out. It may help prevent it, but it's still possible.
In most boats I have driven, I do notice an improvement in slow speed maneuvering with a rudder installed.