Ianthan, this is badass, thanks for taking the time to give it a shot. I have been wanting to get away from depending on the motorshield because of the pins it uses makes it difficult to use SPI at the same time.
I'm definitely going to buy a couple of serial stepper boards and try this out. A slightly reduced-feature RAMPS shield would do the business.
When you say that you haven't found the right settings for 3200 / 40, I guess that means that 3200 and 40 don't work? What goes wrong with those? The controller gets very confused if you change stepsPerRev and mmPerRev, best off doing a save+restart after that.
Ok, doing my own trials with microstepping, a couple of things jump out as things that need fixing:
1. In the controller, the Upload machine spec function adds an extra 0.5 onto the stepsPerRev and the mmPerRev. Fix it by casting to an int for those commands:
// ask for input to get the new machine size
String command = CMD_CHANGEMACHINENAME
command = CMD_CHANGEMACHINESIZE
command = CMD_CHANGEMACHINEMMPERREV+
command = CMD_CHANGEMACHINESTEPSPERREV
2. More significantly, with so many microsteps, ints no longer have enough space to hold most position values. I should have guessed this was going to come up eventually, sorry folks. Int max value is 32,767, so it might move ok at the top, in the centre, but any further down than that, then the position rolls over and the gondola heads off to wobble around angrily at the top of the machine, presumeably looking for -32000 or something. Try changing all your positional ints to longs, and use asLong() to extract the params rather than asInt().
After that, C17 works ok for me. Very slow, but que sera. Accelstepper isn't known to work that well at high speeds (1000+), so you might want to get rid of it and use Stepper library directly, but I like acceleration, I think it makes for smoother drawings and a nicer noise!