Sadly, this will likely be an issue with your video. Most likely what you see initially is correct, but, for whatever reason, the manual tool is having difficulty keeping the video frames and the alignment information synced up.
This will most likely be fixable by re-encoding the video file.