Synthetic Data For Off-Road Autonomy Perception: What Works

The Short Answer

Synthetic data for off-road autonomy perception works best in two roles, as a pretraining set before fine-tuning on real off-road images, and as targeted cover for the terrain, weather, dust and obstacles that fleets rarely capture. The published evidence is consistent on that. OffRoadSynth, a 6,170-image synthetic set rendered in a Unity-based simulator, gave better segmentation after fine-tuning than random initialization or larger generic datasets (Małek et al., 2024). Applied Intuition reported improvements of up to 40 percent in classes like grass when it mixed synthetic off-road data with the real RUGD and RELLIS-3D sets, with almost no degradation in other classes (Applied Intuition, May 2024). What no published result shows, as of October 2026, is synthetic-only training matching a model trained on in-domain real data.

This post covers why off-road perception is a harder target than on-road, which real and synthetic datasets exist and how big they are, what the synthetic-data studies found, which simulators teams use, and a working procedure. The general case for synthetic data in detection is covered in does synthetic data work for object detection, and the transfer techniques in how to close the sim-to-real gap for perception. Neither is repeated here.

Why Is Off-Road Perception Harder Than On-Road?

Urban driving datasets have structure a model can lean on: lane markings, curbs, signs, buildings with straight edges, and a road surface that is drivable by definition. The RUGD authors built their dataset because none of that holds off-road. Their sequences contain many more terrain types, irregular class boundaries, few structured markings, and motion-blurred frames from a small ground robot (Wigness et al., IROS 2019).

Two things follow. First, the labels themselves are ambiguous. Where grass ends and bush begins, or whether a rut is mud or a puddle, is a judgment call, and the baseline numbers show it. On RELLIS-3D, HRNet with OCR reaches 48.83 mean IoU on images and GSCNN 50.13, with SalsaNext at 40.20 on LiDAR (RELLIS-3D benchmark). On GOOSE, DDRNet scores 46.53 class-level mIoU and 70.23 at the coarser category level, and the point-cloud baselines are lower, 34.32 for PVKD and 17.32 for SPVNAS (Mortimer et al., ICRA 2024). Second, traversability depends on the vehicle. The CaT dataset labels the same trail images three ways, for a sedan, a pickup and an off-road vehicle, because a rut that stops one is fine for another (Sharma et al., IEEE Access 2022).

Collection is the other constraint. DARPA's RACER fleet vehicles, Polaris RZR S4 1000 Turbo platforms with multiple LiDARs, cameras and radar, record four terabytes of sensor data per hour, and by January 2022 the program had gathered more than 100 terabytes over more than 500 kilometers of terrain (DARPA, January 2022). Recording at that rate is the easy part. Labeling it is the cost, the terrain that matters most for a UGV program is remote, and the conditions that break perception (dust, rain, low sun, night) are the ones a test crew avoids.

Which Off-Road Datasets Exist, Real And Synthetic?

The table lists the public datasets most cited in off-road perception work, with sizes taken from each project's own page or paper. Semantic sets and traversability sets answer different questions, and two of the real sets (TartanDrive and Rally Estonia) list no pixel labels at all.

Two things stand out. The real sets are small by on-road standards, and the labeled ones top out at 10,000 to 15,000 frames. And the public synthetic sets are smaller still, at 2,000 to 6,000 images, which says something about where the published work is: proof that synthetic pretraining helps, not large-scale synthetic benchmarks.

What Does Synthetic Data Change For Off-Road Segmentation?

The three studies that compare training recipes on real off-road test data reach the same conclusion.

Pretraining on a small, domain-specific synthetic set beats a large generic one. OffRoadSynth tested how weight initialization affects a segmentation CNN that is then trained on real off-road images. Pretraining on the 6,170 synthetic images gave higher accuracy than random initialization and than initializing from larger generic datasets. The authors' conclusion is that a smaller synthetic set in the right domain is an effective way to initialize weights before real-world off-road training (Małek et al., 2024).

Mixing synthetic with real lifts the classes that real data under-represents. Applied Intuition, in work with partners including USC and the Army Research Laboratory, generated a fully annotated synthetic off-road set to supplement RUGD and RELLIS-3D and presented the result at SPIE Defense and Commercial Sensing. Mixing it with the real data gave improvements of up to 40 percent in classes like grass, with almost no degradation elsewhere (Applied Intuition, 2024). The post does not give dataset sizes or mIoU, so treat the figure as a per-class gain on a benchmark, not a headline number.

Paired real and synthetic sets make the gap measurable. CaSSeD from Mississippi State ships 1,077 real images from the CAVS proving ground alongside 1,947 synthetic image pairs rendered in MAVS with the same six classes, with time of day, lighting, plant density, cloud cover and landscape varied (CAVS). That pairing is what lets you train on synthetic, test on real, and read the gap directly, which is the measurement the rest of this post depends on.

Two 2026 papers extend the idea rather than overturn it. ForestSim is a 2,094-image synthetic benchmark for forest environments with 20 classes, built in Unreal Engine with AirSim (Wagle, Chen and Liu, 2026). Trinity extends the OAISYS simulator to produce RUGDSynth and uses it to train a joint class-agnostic terrain and semantic segmentation model, alongside the real EXTerra dataset, which carries both label types (Müller et al., 2026). Neither abstract reports mIoU figures, so neither is in the table of effects above.

The most recent large real-data paper is also a useful control. Dror, Aviv and Hadar trained SegFormer MiT-B3 on a 14,879-image off-road set from 16 locations in four countries and raised test mIoU from 68.66 to 70.06 with a loss that penalizes systematic class confusions (Sensors, March 2026). No synthetic data was used. The gain from a better loss on a large real set was 1.4 points, which is the scale of improvement you should expect from any single change once the real set is big and well matched.

Which Simulators Generate Off-Road Synthetic Data?

The published synthetic sets above came from four kinds of tool, and each has a different trade.

• MAVS (Mississippi State Autonomous Vehicle Simulator). An MIT-licensed, standalone simulator with ROS wrappers, physics-based LiDAR, GPS and camera sensors, a validated tire-soil model for sand and clay, a validated LiDAR-vegetation interaction model, automatic off-road terrain generation, and snow, fog, dust, rain, haze and night conditions (CAVS). It is the only off-road simulator on this list that is free, open and built for the domain. It generated the CaSSeD synthetic set.
• Duality AI Falcon. Falcon is a digital-twin simulator built on Unreal Engine. Duality was one of two prime contractors on DARPA's RACER-Sim effort alongside Intel Federal (DARPA, January 2022), and in September 2024 NASA JPL contracted Duality to use Falcon for mapping, perception and planning work on RACER (The Robot Report). If your program is DARPA-adjacent and needs closed-loop autonomy testing as well as data, it is the obvious reference point.
• NVIDIA Isaac Sim with Omniverse Replicator. Replicator provides randomization tools that sample assets, materials, lighting and camera positions, annotators that produce labeled output such as 2D and 3D bounding boxes, normals and depth, and writers that emit training formats (NVIDIA Omniverse docs). It is general-purpose, so terrain, vegetation and vehicle-terrain interaction are yours to build. Wijayathunga et al. at Edith Cowan University used it to build a multimodal off-road dataset of four Western Australian scenarios, with monocular and stereo RGB, LiDAR and IMU and 16 semantic classes (Journal of Intelligent and Robotic Systems, 2025).
• Game engines directly. OffRoadSynth was rendered in a Unity-based simulator with varied weather conditions, and ForestSim in Unreal Engine with AirSim. This route gives the most control and the most engineering, and the datasets it has produced so far are a few thousand images each.

Managed platforms, including Stardust, sit above these. Our comparison of synthetic data platforms covers the vendor landscape, including Falcon, Applied Intuition and Isaac Sim, in more detail.

How Should You Use Synthetic Data Off-Road?

The procedure below follows from the datasets and results above. It assumes you have a target vehicle and sensor rig and at least a few hundred labeled real frames from it.

1. Choose a real benchmark that matches your platform. RUGD was shot from a small robot close to the ground. GOOSE and RELLIS-3D come from a full-size vehicle and a Warthog UGV, and GOOSE adds thermal and NIR. Camera height, field of view and LiDAR channel count change what the model sees, so pick the public set closest to your rig and hold part of it out as a real test split that never touches training.
2. Measure the gap before generating anything. Train on your synthetic set alone, test on the real split, and compare with a model trained on the real training split. Break the result down by class. Off-road sets are dominated by grass, tree and sky, so a gap concentrated in mud, puddle, rubble or log is the normal case and tells you what to render next.
3. Pretrain on synthetic, then fine-tune on real. This is the recipe OffRoadSynth validated and the one that the broader detection literature also favors. Mixed training is a reasonable second option, and it is the one Applied Intuition reported the 40 percent per-class gain with.
4. Spend the synthetic budget on conditions, not volume. Vary sun angle, time of day, dust, rain, fog and haze on the same terrain, since those are the conditions the real sets barely contain (ORFD is the only one in the table that covers sunny, rainy, foggy and snowy weather and daylight through darkness). Render the obstacles that real fleets rarely see, such as ditches, debris, wildlife and stalled vehicles, at the frequency you need the model to handle them.
5. Match the sensor, not just the scene. Off-road rigs are mixed. RELLIS-3D pairs a 64-channel Ouster with a 32-channel Velodyne; GOOSE runs a 128-channel Velodyne, two Ouster OS0s and a FLIR A615 thermal camera. Synthetic LiDAR, EO and IR should be rendered in registration from the same mounting positions, optics and noise as your vehicle, otherwise the model learns a sensor you do not have.
6. Label traversability for your vehicle, not a generic one. CaT's three-way labeling exists because the drivable region differs by vehicle class. If your deployment vehicle is a tracked platform or a 12-ton RACER Heavy Platform rather than a RZR, the traversability labels in a public set will not transfer, and a simulator that knows your vehicle's capability can produce the right ones.
7. Re-measure after every change. Re-run the synthetic-to-real and real-to-real numbers on the same held-out split after each generation batch. The 1.4-point gain in Dror et al. from a loss change on a large real set is a reminder that single interventions are small once the real data is good, so the loop matters more than any one batch.

If you have a large, well-matched real set already, as Dror et al. did, synthetic data is a smaller lever than it is for a team with a few hundred frames from a new vehicle. If you have no real data at all from the target vehicle, the evidence above says synthetic pretraining will help, and also that you still need the real fine-tuning set.

Where Stardust Fits

Stardust is Bifrost's synthetic data platform, and off-road is one of the verticals it is built for. It generates unstructured terrain (desert, steppe, scrub and forest) with rocks, ditches and vegetation placed fresh for every scene, and sweeps sun angle, dust, rain and time of day across the same terrain so you can find where a model fails and generate the patch. Output is EO, IR and depth in registration, with drivable surface, hazards and obstacles segmented per pixel and depth on every frame, and mounting, optics and noise matched to your vehicle. The rare obstacles in step 4 above, ditches, debris, wildlife and stalled vehicles, are the kind of content it is meant to produce on demand. Details are on the off-road page.

For the sensor-matching and measurement steps, the Real2Sim workflow takes captured field footage, rebuilds failures as 3D assets and measures the fix after retraining, and the neural rendering preview adds a generative pass for glare, grain and noise while the scene geometry and labels stay fixed. Policy evaluation for the driving stack itself, as opposed to the perception model, runs on Manifold, not Stardust. If you would rather run your own simulator, MAVS is free and built for this domain, and the procedure above applies either way.

Sources

• Wigness, Eum, Rogers, Han and Kwon, A RUGD Dataset for Autonomous Navigation and Visual Perception in Unstructured Outdoor Environments, IROS 2019.
• Jiang, Osteen, Wigness and Saripalli, RELLIS-3D Dataset: Data, Benchmarks and Analysis, 2020.
• Sharma et al., CaT: CAVS Traversability Dataset for Off-Road Autonomous Driving, IEEE Access 2022.
• Min et al., ORFD: A Dataset and Benchmark for Off-Road Freespace Detection, ICRA 2022.
• Triest et al., TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models, 2022.
• Mortimer et al., The GOOSE Dataset for Perception in Unstructured Environments, ICRA 2024.
• Autonomous Driving Lab, University of Tartu, The Rally Estonia Challenge.
• Małek, Dybała, Kordecki, Hondra and Kijania, OffRoadSynth Open Dataset for Semantic Segmentation using Synthetic-Data-Based Weight Initialization for Autonomous UGV in Off-Road Environments, Journal of Intelligent and Robotic Systems, 2024.
• Wagle, Chen and Liu, ForestSim: A Synthetic Benchmark for Intelligent Vehicle Perception in Unstructured Forest Environments, 2026.
• Müller et al., Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data, 2026.
• Dror, Aviv and Hadar, Off-Road Autonomous Vehicle Semantic Segmentation and Spatial Overlay Video Assembly, Sensors 26(6):1944, March 2026.
• Applied Intuition, Synthetic Data for Action Recognition and Terrain Segmentation, May 2024.
• Mississippi State University CAVS, Open Source Datasets for Off-road Autonomous Vehicles and MAVS.
• DARPA, RACER program page and RACER news, 13 January 2022.
• The Robot Report, Duality AI continues work with NASA JPL on DARPA RACER program, September 2024.
• Wijayathunga, Dabare, Rassau, Chai and Islam, A High-fidelity Multimodal Synthetic Dataset Generation Framework for Off-road Unstructured Terrain Navigation Training of Autonomous Robots, Journal of Intelligent and Robotic Systems, 2025.
• NVIDIA, Omniverse Replicator.