The physical AI glossary
Definitions for the terms used across robot policy evaluation, synthetic data, and autonomy.
Robot learning and evaluation
VLA (vision-language-action model)
A neural network that takes camera images and a natural-language instruction as input and outputs robot actions directly, such as joint or end-effector commands. VLAs extend vision-language models with an action decoder so one model can follow varied instructions across many tasks. RT-2, OpenVLA and π0 are examples.
Embodied AI
AI that perceives and acts inside a physical or simulated environment, rather than only processing static datasets. An embodied agent takes an action, observes the consequence, and updates its next action in a closed perception-action loop. Robots, autonomous vehicles and simulated agents all count as embodied AI.
Robot policy
The function, typically a neural network, that maps a robot’s observations to actions. A policy can be a low-level controller or a high-level VLA model, and is usually trained by imitation learning, reinforcement learning, or a combination of both.
Policy evaluation
Measuring how well a trained robot policy performs a task, usually by running many trials (rollouts) in simulation or on hardware and scoring the outcomes. Rigorous policy evaluation needs reproducible environments, fixed initial conditions and consistent metrics so scores stay comparable across teams and model versions.
Rollout
A single execution of a policy from an initial state to termination, whether success, failure or timeout, producing a trajectory of observations, actions and rewards. Aggregating many rollouts under varied initial conditions is what turns a single run into a statistically meaningful result.
Success rate
The fraction of rollouts in which a policy meets a task’s defined goal condition, usually reported as a percentage over a fixed number of episodes. Because policies and environments are stochastic, success rate is only meaningful when the trial count and success criteria are stated alongside it.
Benchmark
A standardized task suite, environment and scoring method used to compare different policies or models under identical conditions. A benchmark is only useful for comparison if the simulator, task definitions and metrics are fixed and disclosed.
LIBERO
An open-source benchmark suite for lifelong robot learning that tests whether a policy can acquire new manipulation skills while retaining old ones, across tasks that share and vary objects, layouts and instructions. It is widely used to evaluate generalization and forgetting in imitation-learned and VLA policies.
RoboCasa
A large-scale simulated benchmark and asset library of household manipulation tasks, with diverse kitchen scenes, everyday objects and long-horizon activities generated at scale. Teams run it, alongside LIBERO and other suites, as one of the standard harnesses Manifold reports verified baselines for.
Imitation learning
Training a policy to reproduce actions from expert demonstrations, typically human teleoperation, rather than learning purely from trial and error. Behavior cloning is the simplest form; more advanced variants correct for the drift between the states seen in demonstrations and the states the policy actually reaches at test time.
World model
A learned model that predicts how an environment will change as a function of an agent’s actions, letting a system simulate consequences internally, plan ahead, or generate additional training rollouts without acting in the real world.
Sim-to-real transfer
Deploying a policy trained in simulation onto physical hardware so that it performs comparably despite differences in dynamics, sensors and appearance, often called the reality gap. Domain randomization, system identification and photorealistic rendering are the main techniques used to narrow that gap.
Domain randomization
Randomizing simulation parameters, such as textures, lighting, physics and object properties, during training so a policy learns features that generalize instead of overfitting to one simulated configuration. It is one of the primary techniques for closing the sim-to-real gap.
CI for robot policies
Continuous integration applied to robot policies: automatically re-running a fixed suite of simulated benchmarks whenever a policy, dataset or codebase changes, so a regression in success rate is caught before it reaches hardware or production.
Leaderboard / verified baseline
A public ranking of results on a shared benchmark, where a verified baseline means the number was reproduced independently under the benchmark’s stated protocol rather than self-reported, giving researchers and buyers confidence the score is comparable to others on the board.
6-DoF pose
The full six degrees of freedom describing an object’s or end-effector’s position (x, y, z) and orientation (roll, pitch, yaw) in 3D space. Manipulation and grasping typically require estimating or controlling 6-DoF pose, since a 2D detection alone doesn’t capture the orientation needed to grasp an object correctly.
WAM (world action model)
An embodied foundation model that predicts how the world will change and generates actions in one model, learning a joint distribution over future states and actions rather than mapping observations straight to actions. Where a VLA is reactive, a WAM makes future prediction an explicit part of control. VLAs and WAMs are complementary lines of research, and both are evaluated the same way, by running the policy and scoring outcomes.
VLM (vision-language model)
A model trained to jointly understand images and text, such as answering questions about a scene or grounding a phrase to a region. VLAs are built by extending a VLM with an action output, which is why VLM capabilities like spatial reasoning carry over into robot policies.
Foundation model
A large model pretrained on broad data that can be adapted to many downstream tasks with little or no task-specific training. In robotics, foundation models for perception, language and action are the base that generalist policies are fine-tuned from.
Generalist policy
A single policy trained to perform many tasks across varied objects, scenes and instructions, rather than one narrow skill. Generalist policies are the reason evaluation now has to cover a wide task distribution instead of a single benchmark.
Reinforcement learning (RL)
Training a policy by trial and error against a reward signal, so it discovers behavior that maximizes reward rather than copying demonstrations. RL can exceed the demonstrator but needs many interactions, which is why it usually runs in simulation before touching hardware.
Behavior cloning
The simplest form of imitation learning, where a policy is trained by supervised learning to output the action a human demonstrator took in each observed state. It is easy to train but drifts once the policy reaches states the demonstrations never covered.
Teleoperation
A human operating a robot in real time, often with a controller, VR rig or leader arm, to collect demonstration data or complete a task directly. Most manipulation datasets are built from teleoperated demonstrations.
Diffusion policy
A policy that generates actions with a diffusion model, denoising from noise into an action sequence conditioned on observations. Diffusion policies capture multi-modal behavior well, which makes them a common architecture for modern manipulation and VLA systems.
Action chunking
Predicting a short sequence of future actions at once instead of one step at a time, then executing the chunk before replanning. Chunking improves temporal consistency and reduces the compounding error that single-step policies accumulate.
Cross-embodiment
Training or evaluating a policy across different robot bodies, such as varied arms, grippers or humanoids, so skills transfer instead of being tied to one platform. Cross-embodiment datasets pool many robots into a shared format.
End-effector
The tool at the end of a robot arm that interacts with the world, most often a gripper or hand. Policies are frequently commanded in end-effector space, moving the tool to a target pose rather than setting each joint directly.
Dexterous manipulation
Manipulation that requires fine, coordinated control of many degrees of freedom, such as in-hand reorientation or multi-finger grasping. It is one of the hardest capabilities to evaluate because small control errors compound quickly.
Bimanual manipulation
Manipulation using two arms in coordination, needed for tasks such as handovers, folding, or holding one object while working on another. Bimanual policies are harder to train and evaluate because the arms must stay coordinated in time.
Humanoid
A robot with a broadly human body plan, typically two legs, a torso and two arms, meant to operate in spaces and with tools designed for people. Humanoid evaluation is an active frontier as embodiments and datasets scale.
Quadruped
A four-legged robot, used for locomotion over rough or unstructured terrain where wheels struggle. Quadruped policies are usually trained in simulation with reinforcement learning and domain randomization before deployment.
Proprioception
A robot’s sense of its own body, such as joint angles, velocities and end-effector pose, as opposed to external sensing like cameras. Policies typically combine proprioception with vision to act.
Long-horizon task
A task that requires many steps over an extended time, such as cooking or assembly, where an early mistake cascades into later failure. Long-horizon tasks stress a policy’s ability to chain sub-behaviors and are a core target of benchmarks like CALVIN.
Zero-shot and few-shot generalization
Performing a task with no task-specific examples (zero-shot) or only a handful (few-shot). Strong zero-shot and few-shot behavior is the main promise of foundation-model policies, and it is only credible when measured on tasks held out from training.
Simulation and synthetic data
Synthetic data
Artificially generated sensor data, such as images, video or point clouds, produced by rendering simulated scenes instead of capturing the real world. It comes with automatic, pixel-perfect ground truth and can cover rare or dangerous scenarios that are impractical to collect physically; Stardust generates it for perception and autonomy teams.
Photorealistic rendering
Generating synthetic imagery that closely matches the visual statistics of real camera output, using techniques like path tracing, physically based materials and sensor-accurate noise. Photorealism narrows the visual gap between synthetic and real data so perception models trained on one transfer to the other.
Digital twin
A simulated representation of a specific real-world object, environment or system, built to match its geometry, physics and behavior closely enough to run experiments, generate data, or test policies as a stand-in for the physical original.
Closed-loop simulation
A simulation in which an agent’s actions change the environment state and the resulting new observations feed back to the agent in real time, as opposed to open-loop replay of a fixed log. It is required to see how a policy behaves once its own actions diverge from any recorded trajectory.
Procedural generation
Algorithmically generating simulation content, such as scene layouts, object placement, terrain or task variations, from rules and parameters instead of hand-authoring every scene. It turns a limited set of assets into a large, diverse distribution of training and evaluation scenarios.
Segmentation mask
A pixel-level label that assigns every pixel in an image to a class or instance, such as cup, floor or robot arm, used to train and evaluate perception models. Synthetic pipelines can produce exact segmentation masks automatically, since the renderer already knows which object occupies each pixel.
Ground-truth label
The correct answer paired with a piece of data for training or evaluation, such as an object’s class, 6-DoF pose, depth or segmentation. In simulation, ground truth comes directly from the scene description, so it is exact and free of the cost and error of manual annotation.
Reality gap
The set of differences between simulation and the real world, in dynamics, sensor appearance and unmodeled physics, that can make a policy trained in sim fail on hardware. Closing the reality gap is the central challenge of sim-to-real transfer.
Neural rendering
Using learned models, such as NeRFs or Gaussian splatting, to synthesize images of a scene from new viewpoints rather than rendering from hand-built geometry alone. It can produce highly realistic imagery reconstructed from captured real scenes.
Ray tracing and path tracing
Rendering techniques that simulate how light travels and bounces to compute each pixel, producing physically accurate shadows, reflections and global illumination. They are slower than rasterization but far closer to how a real camera sees, which matters for training perception.
Physically based rendering (PBR)
A rendering approach where materials and lights are defined by real physical properties, so surfaces respond to light consistently across scenes and viewpoints. PBR is what keeps synthetic imagery looking correct as conditions change.
3D asset
A reusable digital model of an object or environment, with geometry, materials and often real-world scale, used to compose simulated scenes. The size and realism of an asset library sets the ceiling on the scenarios a synthetic pipeline can generate.
Point cloud
A set of 3D points sampling the surfaces of a scene, produced by LiDAR, depth sensors or reconstruction. Point clouds are a common input for mapping, obstacle detection and 3D perception.
Depth map
An image where each pixel stores distance from the sensor rather than color, capturing the geometry of a scene. Synthetic pipelines produce exact depth for free, since the renderer already knows scene geometry.
Multimodal data (multi-sensor)
Data that pairs several sensor streams of the same scene, such as RGB, depth, IR and LiDAR, time-aligned so a model can learn across modalities. Generating perfectly aligned multi-sensor data is far easier in simulation than in the field.
Long-tail data
The rare, dangerous or unusual cases, such as a near-collision or an odd lighting condition, that seldom appear in collected data but cause most real-world failures. Synthetic generation targets the long tail directly, since these cases are impractical to stage physically.
Auto-labeling
Generating annotations automatically rather than by hand. In simulation, labels such as segmentation, depth and pose come directly from the scene description, so every frame is labeled exactly and at no marginal cost.
Sensors and modalities
EO (electro-optical)
Sensors that capture imagery in the visible-light spectrum, similar to a standard camera, used for detection, tracking and identification in daylight. EO is usually paired with IR sensors to cover both day and night operation.
IR / thermal
Infrared sensors that capture the heat an object emits rather than the visible light it reflects, making it possible to detect people, vehicles and other warm objects in darkness, fog or smoke where EO sensors fail.
SWIR (short-wave infrared)
The infrared band from roughly 1 to 2.5 micrometers, past visible light and the near-infrared. SWIR penetrates haze and some obscurants better than visible imaging, reveals material differences invisible to standard cameras, and unlike thermal bands can use conventional glass optics.
LiDAR
Light Detection and Ranging: a sensor that measures distance by timing laser pulses reflected off surfaces, producing a 3D point cloud of the environment. It is a staple of autonomous vehicles and robots that need precise geometric mapping and obstacle detection.
Radar (range-Doppler)
A sensor that transmits radio waves and measures the reflected signal’s time delay, giving range, and frequency shift, giving velocity via the Doppler effect. Range-Doppler radar keeps working in weather and darkness that degrade optical and infrared sensors, which is why it anchors maritime, aerial and automotive autonomy stacks.
SAR (synthetic aperture radar)
A radar imaging technique that uses the motion of a moving platform, such as a satellite or aircraft, to synthesize a much larger virtual antenna, producing high-resolution ground imagery regardless of cloud cover or lighting. SAR is widely used for earth observation and maritime monitoring.
RGB
Standard three-channel color imagery in the visible spectrum, the default input for most vision models. RGB is cheap and information-rich but degrades in darkness, glare and bad weather, which is why it is often fused with IR, depth or radar.
RGB-D
An RGB image paired with a per-pixel depth channel from a depth sensor, giving both appearance and geometry in a single frame. RGB-D is widely used for indoor manipulation and mapping.
MWIR (mid-wave infrared)
A thermal infrared band, roughly 3 to 5 micrometers, that senses emitted heat and gives strong contrast on warm objects against cool backgrounds. MWIR is common in high-end thermal imaging but usually needs a cooled sensor.
LWIR (long-wave infrared)
The long-wave thermal band, roughly 8 to 14 micrometers, that images emitted heat and works in total darkness. LWIR sensors can be uncooled, which makes them common for night vision, security cameras and vehicle autonomy.
Multispectral and hyperspectral
Imaging that captures many spectral bands per pixel, from a handful (multispectral) to hundreds (hyperspectral), well beyond the three RGB channels. The extra bands reveal material and chemical differences invisible to a normal camera, used heavily in geospatial and remote sensing.
IMU (inertial measurement unit)
A sensor that measures acceleration and angular rate, used to track a platform’s motion and orientation between other sensor updates. IMUs are core to drones, vehicles and any system doing state estimation.
GNSS / GPS
Satellite positioning that provides global location and timing. GNSS is the backbone of outdoor navigation, but it can be denied, jammed or spoofed, which is why autonomy stacks fuse it with inertial and visual sensing.
FMV (full-motion video)
Continuous video from an airborne or mobile platform, as opposed to still frames, used to track and monitor objects over time. Training detection and track on FMV requires temporal consistency, not just per-frame accuracy.
WAMI (wide-area motion imagery)
Persistent, very large-footprint aerial imagery that covers a whole area at once at lower frame rates, letting analysts track many movers across a city-scale scene. WAMI supports wide-area monitoring and tracking across a scene.
Platforms and autonomy
UAV (uncrewed aerial vehicle)
An aircraft that flies without a pilot aboard, from small quadcopters to fixed-wing platforms, used for mapping, inspection, monitoring and delivery. UAV perception has to handle altitude, slant range and fast-changing look angles.
UGV (uncrewed ground vehicle)
A ground robot or vehicle that operates without a human aboard, used for logistics, inspection and mapping across roads and terrain. UGV autonomy has to handle unstructured ground, obstacles and GPS-denied navigation, which is central to off-road evaluation.
USV (uncrewed surface vessel)
A boat or ship that operates on the water’s surface without a human crew aboard, using autonomy software to navigate, avoid collisions and complete missions such as monitoring, mapping or logistics. Testing a USV’s autonomy stack across the sensor and sea-state range it will face is central to maritime evaluation.
UUV / AUV (uncrewed or autonomous underwater vehicle)
A submersible robot that operates below the surface without a crew, used for survey, inspection and mapping. Underwater autonomy is hard because GPS and radio do not penetrate water, so navigation leans on inertial and acoustic sensing.
UAS (uncrewed aircraft system)
The full system around a UAV, including the air vehicle, ground control station, data links and payloads. The term is used when the mission depends on the whole system, not just the airframe.
RPO (rendezvous and proximity operations)
Maneuvering one spacecraft close to another to inspect, dock with or service it. RPO autonomy depends on relative pose estimation from EO and other sensors, often trained on synthetic space imagery because real data barely exists.
SLAM (simultaneous localization and mapping)
Building a map of an unknown environment while tracking the platform’s own position within it, from camera, LiDAR or radar data. SLAM is foundational to navigation for vehicles and robots that cannot rely on GPS.
Autonomy stack
The full software pipeline that turns raw sensor input into action, typically perception, localization, planning and control. Evaluating an autonomy stack means testing these layers together in closed loop, not just a model in isolation.
Perception
The part of an autonomy stack that turns raw sensor data into a structured understanding of the scene, such as detections, tracks, depth and segmentation. Perception is where synthetic data has the most direct impact, since it can generate labeled examples of the exact conditions a system will face.
Path planning
Computing a route or trajectory from where a system is to where it needs to be, while avoiding obstacles and respecting constraints such as vehicle dynamics or COLREGS. Planning sits downstream of perception and upstream of control in the autonomy stack.
COLREGS
The International Regulations for Preventing Collisions at Sea: the maritime rules of the road governing vessel right-of-way, signaling and collision-avoidance behavior. Any autonomous vessel operating alongside crewed traffic has to encode COLREGS-compliant behavior to navigate legally and safely.