Hi, I tend to run as low as possible, for no reason other than managing the amount of work the voltage regulator on the arduino has to do. I think it reduces the heat that the bits and pieces generate too.
Like you, I've never got to the bottom of this mysterious slowdown issue, if it's the same as other similar reports. I've had it sporadically myself, but not been in a position to isolate and test it, and it's "fixed itself" after lots of reloads, reboots etc. I assume the folk who have reported it have had the same experience.
I'm hoping to have a closer look at the code this week, and I have a new release coming up shortly that rolls up the live webcam drawing in the controller (that I was using at the maker faire last weekend), and the configurable pen lift height that has been requested. I need to test that code on some other machines before it's good.
I don't think any of the current changes will fix the issues, because they are nowhere near that stuff, but if I (or anyone else) comes across a solution, I'll try to get it out to everyone.
One thing to note is that unless you tell me (us) what hardware and firmware combination you're using, I (we) can't really help. There's a few platforms out there, and while in principle, the parts of the code that do this basic stuff are shared between all platforms, sensible debugging is only really made possible by replicating the faulty situation in as many ways as possible.