See below links for reasoning, but in short, avoid sqrt and division of small values. It's also possible to normalize the quaternion after the conversion, but better precision is preferable. http://www.euclideanspace.com/maths/geometry/rotations/conversions/matrixToQuaternion/index.htm http://www.euclideanspace.com/maths/geometry/rotations/conversions/matrixToQuaternion/ethan.htm