Wrist movements for the most part, for me. I sometimes did larger arm movements for times when I really wanted to make it more accurate. But since I couldn't be bothered waving my arms about most of the time, it was mostly wrist.
The game doing the opposite might just be it picking up on a slight jolt in your movement just before you make your swing. When you go to make a swing from right to left, your starting position usually means you naturally move your hand to the right first, then swing to the left. The game picks up on that sometimes, requiring you to be more calm in your movements before making a swing.
This might not even be what the problem is for you, but that's the closest relation I can think of, since it happened with me. It was just a case of learning how much it picked up on your movements and then adapting to that.
( Edited 02.01.2012 12:41 by Azuardo )