How Meta Quest tracking works

When you move your head using a Meta Quest, the virtual image accompanies the gesture almost without giving away the work behind it. The same happens with the controls: the hand can accelerate, rotate, cross in front of the body and disappear for an instant from the field of vision, but the virtual object tries to remain exactly where it should be.

This sensation is not born from a single sensor and does not depend only on light. It arises from the combination of cameras, infrared points, gyroscopes, accelerometers, geometry, statistics, map construction and machine learning. In a few milliseconds, the system needs to find out where the headset, controls and, in some modes, up to 26 joints of each hand are.

This article explains how this evolution happened, why early systems seemed more limited, and what is the science that allows Quest to track movements in space. The exact details of the current software are proprietary. When Meta does not publish a step, the text identifies the general scientific principle instead of presenting an assumption as fact.

First, what does tracking mean in 6DoF

A rigid object in space can change position on three axes and rotate around those same three axes. This is where the expression comes from Six degrees of freedom, or 6DoF.

advance and retreat;
up and down;
move left and right;
tilt up and down;
rotate sideways;
tilt sideways.

Tracking rotation alone allows you to look around a virtual scene. Tracking position and rotation at the same time allows you to duck, dodge, walk, and bring your hands to an object. For the illusion to work, estimates need to be fast, stable, and consistent across headset and controllers.

The first phase: cameras looking from the outside in

The commercial Oculus Rift, launched in 2016, used the Constellation system. External sensors positioned on the table observed arrays of infrared LEDs in the headset and Touch controls. The camera saw luminous dots arranged in a known pattern, and the computer calculated the position of that pattern in space.

The principle is similar to recognizing a constellation in the sky. An isolated point says little. Several points at known distances form a geometric signature. When this signature appears larger, smaller, tilted, or displaced in the camera image, an algorithm can estimate the object's rotation and translation.

The method was accurate, but it brought friction. The user had to install sensors, find USB ports, position the equipment, and maintain a visible area. Blind spots could arise when the body blocked vision or when few sensors covered the environment.

2019: Quest reverses camera direction

With the Oculus Quest and Rift S, launched in 2019, Meta took the cameras to the headset itself. Instead of external sensors looking at the user, the device started to look at the environment and controls. The company called the system Oculus Insight.

This change is known as tracking Inside-out. It eliminated outdoor stations and made room-scale virtual reality much simpler to install. It also shifted a huge responsibility to the mobile computer inside the display: understanding the room, locating the headset itself, and finding the controls at the same time, within a tight power and processing budget.

The official video above shows the current result of this evolution. The form factor has become lighter and the Touch Plus controls have lost the large ring visible in previous generations. The science that made this possible begins in the formation of the image.

How a camera turns light into position

A camera does not directly receive the coordinate of a control. It receives a grid of pixels with different intensities. The three-dimensional world was projected onto a two-dimensional surface, so some of the depth was lost.

To retrieve the pose, the system looks for matches between known points on the object and observed points in the image. In computer vision, this type of problem is known as Perspective-n-Point, or PnP. Rotation and translation are chosen in such a way as to reduce the error between the position at which each point should appear and the position at which it actually appeared.

In simplified terms:

the control has a known pattern of luminous dots;
the camera detects some of these spots as small light spots;
calibration informs focal length, optical center, and lens distortion;
geometry projects three-dimensional hypotheses back into the pixels;
The hypothesis with the lowest reprojection error helps to estimate the pose.

More than one camera observes space from different angles. When the same point appears in two views, triangulation provides depth information. Even when not all points are visible, recent history and other sensors narrow down the possibilities.

Scientific illustration showing image formation, inertial sensors, three-dimensional map, and the fusion that accompanies a controller and a hand — The illustration summarizes the scientific pipeline. The exact implementation used by Meta is proprietary and may vary between models and system versions.

The IMU senses the movement before the camera finishes seeing it

Inside the headset and controls is an inertial measurement unit, the IMU. Two components are key:

o gyro measures angular velocity;
o accelerometer measures specific acceleration along the axes.

These sensors respond very fast. By integrating angular velocity over a small interval, the system predicts how orientation has changed. By integrating acceleration, it estimates changes in velocity and position.

The problem is that integration accumulates error. A small calibration deviation added thousands of times becomes a noticeable drift. If tracking depended only on the IMU, a stopped control could seem to slide little by little.

The camera has the complementary behavior. It is slower, but it can observe external references and correct for drift. The combination is often referred to as Visual-inertial odometry. The principle is simple: the IMU predicts what happened between the frames, and the vision compares that prediction to the observed world.

SLAM: Build the map while searching for yourself

To keep track of the headset, it's not enough to locate the controls. The device needs to figure out how it itself moved within an environment it's still getting to know.

SLAM stands for simultaneous localization and mapping. The system identifies points of interest at edges, corners, and contrasting regions, tries to rediscover them in the following frames, and builds a sparse map of three-dimensional references. At the same time, it estimates the trajectory of the cameras within this map.

It's a circular problem: to map the room you need to know where the camera is, but to know where the camera is it's useful to have a map of the room. SLAM algorithms solve the two estimates together and refine the result as new observations come in.

Meta described a mobile pipeline capable of capturing images, finding hundreds of points, associating them with previous frames, and using trigonometry to position them in 3D. It also highlighted the need to calibrate lenses, IMUs, and clocks. If the camera and inertial sensor disagree about the instant at which a measurement occurred, even a good measurement can be applied to the wrong time.

Sensor fusion: no measurement needs to be right alone

A useful way to imagine fusion is to think of two responses accompanied by confidence levels.

The IMU says, "By acceleration and rotation, the control probably got here." The camera replies, "The bright spots appeared in these pixels, so it's probably there." An estimator combines the answers by considering noise, delay, and uncertainty.

Kalman filters and optimization methods are classic tools for this type of problem, but Meta does not publish the entire drawing of the current Quest estimator. The important thing is the principle: each sensor corrects the limitations of the other and uncertainty is part of the calculation.

Then comes the prediction. The pose used to render an image can't just represent the instant when the last camera captured a frame. By the time the panel lights up, the head will have moved a bit. The system projects the pose to the future moment when the image will be displayed. This prediction helps reduce the feeling of lag and discomfort.

The part of the light that feels like magic

In the Touch controls with ring, small LEDs emit infrared light, invisible to human eyes and detectable by tracking cameras. The apparent shape, the distance between the points and the way the assembly moves help determine where the control is.

But the camera doesn't automatically know which point came from the control. A Christmas tree, a chandelier, reflections, and other sources can produce similar light spots. In 2019, Meta engineers published a case where hundreds of decorative lights increased the cost of processing and could lead to false associations.

The solution shows well the mixture of science and engineering. The system triangulated stationary lights, saved their positions on a 3D map and associated each reference with a covariance, a representation of uncertainty. After observing a still light enough times, the software began to treat it as part of the environment and exclude it from the search for control.

There were still false candidates. So the team trained a small convolutional neural network to see if the size and appearance of a light spot made sense for the suggested pose. Camera distance, angle of incidence, position in the frame, and ellipse orientation helped separate LEDs from ambient noise control.

This example corrects a common idea. Machine learning hasn't replaced all geometry. It was used alongside triangulation, projection, history, maps, and statistics to solve a difficult part of the problem.

Why too dark and too much light get in the way

The headset needs visual references in the environment. A completely dark room reduces the corners and textures that SLAM can follow. A smooth wall, without contrast, also offers few stable points.

At the opposite extreme, direct sunlight and intense infrared sources can saturate or confuse the cameras. Reflections change with angle, while moving objects do not serve as reliable fixed references. The system must continually decide which observations belong to the static world, which belong to the control, and which are just noise.

In practice, uniform internal lighting and an environment with visual contrast usually help. Covering the cameras, using the device in very strong direct sun, or playing in the dark increases the chance of temporary loss of tracking.

From Ring to Touch Plus: Where Machine Learning Comes in

The Quest 3 and Quest 3S use the Touch Plus controls without the outer ring of previous generations. Meta's own comparison page describes the current tracking as Hybrid of computer vision and machine learning.

Removing the ring improves the approximation between the hands and reduces collisions between the controllers, but also decreases the large structure that left the LEDs well distributed. To compensate, the system can combine the visible signals, the IMU, the recent pose, and models capable of estimating likely movements of the hands and controls. Meta does not publicly detail all the weights, frequencies, and internal states of this pipeline, so it is not correct to turn this general description into an exact recipe for the Quest 3.

When a control briefly moves out of the field of view, the IMU and the prediction maintain an estimate. The longer the occlusion time, the greater the uncertainty tends to be. When cameras find reliable evidence again, the pose is corrected. The challenge is to make the correction without a sudden visual jump.

And when there is no control at all?

In 2019, Meta introduced hand tracking on the Quest without gloves or external sensors. Cameras observe hands, and computer vision models estimate a skeleton. Current documentation reports 26 joints per hand, with position, rotation, and confidence level.

The video also functions as a practical animation of the concept: the image of the hand is converted into an articulated structure that the software can use. Gestures such as pinching, touching, and grasping arise from the relationships between these joints.

The method has physical limits. One hand can hide the other, fingers can overlap, and very fast movements generate blur. Meta's documentation recommends keeping gestures facing the cameras and recognizes that tracking can have more delay and less precision in fast situations. Therefore, controllers are still preferred in games that require aiming and precise haptic feedback.

A timeline of evolution

2016: Rift and Touch popularize tracking with external sensors and infrared constellations.
2019: Quest and Rift S bring the cameras to the headset with Oculus Insight and inside-out tracking.
2019: The Quest gets hand tracking without additional hardware.
2020: Quest 2 improves performance, control autonomy and Insight system stability.
2022: Touch Pro controls are now located with their own cameras, reducing dependence on the headset's field of view.
2023: Quest 3 debuts Touch Plus without the big rings and expands the mix of computer vision, sensors and machine learning models.
2024 onwards: Meta opens Horizon OS to a larger ecosystem and continues to develop hands, body, environment, and hybrid interactions.

What the system really knows about the environment

To track, the device needs to produce internal representations of points, surfaces, and motion. This does not mean that every tracking camera functions as a common recording available to any application. Developers' access to raw images is controlled by specific APIs and permissions, and varies between models and versions of Horizon OS.

There is also a difference between locating and understanding. A point map can be enough to estimate movement without recognizing that an object is a sofa. Mixed reality features add depth, planes, boundaries, and semantic information to allow digital objects to interact with walls, floors, and furniture.

Why it all seems instant

The short answer is parallelism. While the camera captures a new frame, the IMU keeps sending measurements. While the view looks for points, the estimator predicts the pose. While the GPU renders the scene, the compositor prepares final corrections for the latest orientation.

Each step works with an imperfect version of the world, but it works fast and updates the answer continuously. The result is not a definitive measurement. It is the best estimate available for that instant, accompanied by uncertainty and corrected in the next table.

This is the most fascinating part of modern tracking: the Quest does not see space as a person. It reconstructs movement from light projected in pixels, accelerations, rotations, geometric relationships, and probabilities. The sense of presence appears when all these approximations agree well enough to disappear before our eyes.

How Meta Quest sees your movements: the science of tracking

Listen to this article