Our horses can be very good at re-positioning us.

Try using a marker on the ground when handling your horse to help you keep track of where you are and see who is moving who.