Synthetic Data For USV Collision Avoidance And Maritime Lookout
Synthetic data gives USV lookout models the rare, dangerous cases real footage lacks, like swimmers, debris, night and rough seas. Train on it, validate on real video.
The Short Answer
Synthetic data for USV collision avoidance is rendered imagery of the water around an autonomous boat, labeled automatically, that teams use to train the lookout model on the hazards real footage rarely contains, such as persons in the water, kayaks, debris, unlit buoys, night, fog, glare and rough seas. In MODS, the standard benchmark recorded from a real USV, persons make up just 0.7 percent of the dynamic obstacle labels, which is why a detector trained only on collected footage tends to miss the cases that matter most. In a Bifrost case study, a YOLOv11n detector trained on 2,500 synthetic images reached 82 percent F1 on a real MODS test sequence in two days of work. Across published work, the approach that holds up is to train with synthetic data and validate on real sequences from your own vessel.
As of October 2026, the pages that answer this question are academic papers, each describing one simulator or one dataset. This post covers what a lookout model has to detect, which public datasets help and what they lack, what a synthetic pipeline has to vary, how the published maritime simulators compare, and how to check the result against real data.
What A Lookout Model Has To Detect
Rule 5 of the COLREGs requires every vessel to keep a proper lookout "by sight and hearing as well as by all available means." For an uncrewed vessel, the camera-based lookout model carries much of that duty, and the planner behind it can only follow the rules if perception tells it what is out there.
Vessels, by type and aspect. The COLREGs treat a power-driven vessel, a sailing vessel, a vessel engaged in fishing and a vessel restricted in its ability to maneuver differently, and the correct maneuver depends on whether an encounter is head-on, crossing or overtaking. A lookout model therefore needs more than a "boat" box. It needs a class that maps to the rules and an estimate of heading, which comes from seeing each vessel type at every aspect angle, bow-on, beam-on and stern-on, by day and by its lights at night.
Buoys and navigation marks. Lateral and cardinal marks, mooring buoys and small floats come in many shapes. In our maritime case study, the first model consistently missed horizontal, oblong buoys because it had seen too few of them.
Small craft and people. Kayaks, paddle boards, rowing boats, swimmers and persons overboard are small, low in the water and often hidden behind waves, which makes them the hardest targets to see and the most important not to hit.
Debris and unexpected objects. Logs, lost containers, fishing gear and floating debris have no fixed shape, and a model trained on a closed list of classes can ignore them entirely. Segmentation that labels anything on the water that is not water as an obstacle is a common safeguard.
Static obstacles. Piers, breakwaters and shoreline matter for every harbor approach. MODS annotates them by the edge where the obstacle meets the water, because that edge is what a planner needs.
Why Real Data Misses These Cases
Collecting USV footage is slow and expensive, and the footage that is easy to collect is mostly open water and the vessels that happen to pass. The MODS authors recorded about 48 hours of footage over eight voyages and kept 94 sequences with 80,828 images, and even with an expert piloting toward obstacles on purpose, 42.6 percent of the dynamic labels are vessels, 56.7 percent are "other," and 0.7 percent are persons. Near-misses with swimmers, a capsized kayak in a swell, or a buoy dead ahead at dusk cannot be staged safely or often enough to train on. Night, fog and rough seas are also harder and riskier to record, so collection campaigns capture less of them.
Public Datasets For USV Obstacle Detection
These are the real datasets most used for this task. They are the right place to get a first test set and pretraining data, and each one leaves gaps a lookout model will hit in deployment. For a fuller list scored on ontology, diversity and label quality, see our post on the top datasets for maritime AI perception.
| Dataset | Sensor and platform | Contents | What it lacks for collision avoidance |
|---|---|---|---|
| MODS (Bovcon et al., IEEE T-ITS) | Stereo cameras and IMU on a small USV, Slovenian coast | 94 sequences, 80,828 images, over 60,000 annotated objects; labels vessel, person and other, plus water-edge polygons | Persons are 0.7 percent of labels; three coarse classes with no vessel type or heading; one region |
| LaRS (Žust et al., ICCV 2023) | Images from many sources across lakes, rivers and seas | Over 4,000 per-pixel labeled key frames (40,000+ frames with context), 8 obstacle classes including swimmer, paddle board and buoy, 19 scene attributes | Labels on key frames rather than full tracks; no range, heading or vessel-type labels beyond boat/ship and row boat |
| MaSTr1325 (Bovcon et al., IROS 2019) | Camera and IMU on a coastal USV, Gulf of Koper | 1,325 images with pixel labels for obstacle, water and sky | No object classes, so it supports obstacle segmentation but not COLREGs-relevant classification |
| Singapore Maritime Dataset (Prasad et al., 2017) | Visible and near-infrared video, on-shore and on-board | 81 videos (40 visible on-shore, 11 visible on-board, 30 NIR on-shore) from before sunrise to after sunset, plus haze and rain | Only 11 videos from a moving vessel; on-shore views sit higher and steadier than a USV camera |
| MassMIND (Nirgudkar et al., IJRR 2023) | LWIR thermal, Boston Harbor and Charles River | 2,916 images at 640×512, segmented into sky, water, obstacle, living obstacle, bridge, self and background | Segmentation classes only; one harbor; no paired RGB |
The common pattern is that each dataset covers one region or a handful, one or two sensors, and a limited set of encounters, and that people in the water are scarce wherever they are labeled at all. A model trained on them inherits those gaps.
What A Synthetic Pipeline Has To Vary
The point of generating data is to cover what the real set does not. These are the variables that change what a maritime camera sees, and how a pipeline should handle each.
| Variable | Why it matters | What to render |
|---|---|---|
| Sea state | Waves hide small objects and create foam and whitecaps that look like debris; SafeSea found detector mAP fell from sea state 1 to 4 | Calm to rough water with consistent wave physics, spray and partially submerged targets |
| Sun glare and glitter | Specular reflections saturate pixels and cause false positives; MODS deliberately includes prominent sun glitter | Sun elevation and azimuth sweeps, including low sun ahead of the vessel |
| Fog, haze and rain | Contrast falls with range, and the horizon disappears | Fog density and visibility distance, rain on the water and on the lens |
| Night and twilight | Vessels are recognized by navigation lights, and unlit objects nearly vanish in RGB | Night scenes with correct navigation lights by vessel type, plus thermal renders where a thermal camera is fitted |
| Horizon and shoreline | Shoreline clutter, piers and moored boats confuse detectors trained on open water | Open sea, coastal, harbor, river and canal backgrounds |
| Camera height and mount | A camera 1 m above the water sees small craft against the horizon; one on a mast sees them against water | The exact mount height, field of view and resolution of your camera |
| Platform motion | Pitch and roll move the horizon across the frame and blur small targets | Vessel motion matched to sea state, with motion blur and rolling-shutter effects |
| Target type, aspect and range | Classification and heading estimates fail for unseen aspects and at long range | Every class at full 360° aspect, at ranges from close quarters to the horizon |
| Sensor effects | Noise, compression, lens flare and salt on the lens change pixels in ways models learn | Your camera's noise profile, exposure behavior and lens contamination |
Two rules make the variation useful rather than random. First, log every parameter per frame (sea state, sun angle, visibility, range and bearing to each object) so failures can be traced to conditions. Second, weight the dataset toward the conditions your real test set shows the model failing on, then generate more of those. Our post on closing the sim-to-real gap for perception goes into how to measure which variables matter for a given sensor. If your vessel carries a thermal camera, Synthetic Thermal Infrared Training Data covers the IR side.
Academic Simulators And Synthetic Maritime Datasets
Most published work on synthetic data for autonomous boats comes from research groups, each building one tool for one question. This is how they compare.
| Work | Approach | What it provides | Limits for a lookout model |
|---|---|---|---|
| NAVHAZ-Synthetic (Ward, Harguess and Corelli, 2019) | Rendered shipboard views | One million labeled images of ten vessel classes, each with class and heading, across sea states, lighting, fog, sea spray and salt on the lens | Covers the ten vessel classes in the paper, not swimmers or debris |
| ASVSim (Lesy et al.) | Open-source simulator built on Cosys-AirSim and Unreal Engine, MIT license | Vessel dynamics, camera rendering and radar emulation, with synthetic dataset generation; demonstrated on waterway segmentation and navigation | Built for inland waterways and ports rather than open sea |
| MMUSV-Sim (Li et al., 2026) | Unreal Engine 5 with Project AirSim sensor models | Island, open-sea and port scenes, configurable weather, time of day and waves, RGB, depth, semantic, lidar and radar from several USVs at once | Aimed at multi-USV cooperative perception; its reported result is lidar detection (72.74 AP@0.5 with early fusion against 45.54 for one USV) |
| SafeSea (Tran et al., MaCVi workshop at WACV 2024) | Blended Latent Diffusion edits the sea background of real images while keeping the objects | 69,694 images made from 300 calm SeaDronesSee images, sorted by sea state 1 to 4 | Aerial viewpoint from drones; changes the sea, not the objects, so it cannot add new targets |
| ARAGON USV (Han et al., Journal of Field Robotics, 2020) | A real KRISO research USV with radar, lidar and cameras | Field-tested ship detection, sensor-fusion tracking and COLREGs-compliant avoidance maneuvers | A field system, not a data generator; shows the full perception-to-planning chain that synthetic data feeds |
| Maritime perception survey (Han et al., Intelligent Service Robotics, 2026) | Review of maritime datasets and perception algorithms | Datasets organized by sensor configuration and task, covering detection, segmentation, tracking, fusion and SLAM | A map of the field rather than a tool |
These are good starting points for a research team with engineering time, and ASVSim's open license makes it the easiest to adopt. None of them, as published, offers a broad library of small craft, people in the water and floating debris rendered across the conditions in the previous table.
How To Validate On Real Data
Synthetic data only earns its place if a model trained on it does better on real footage from your own vessel. A practical validation plan looks like this.
- Hold out real sequences before generating anything. Record footage with the production camera on the production mount, in the waters you will operate in, and keep it out of training. Split by whole sequences rather than random frames, because neighboring frames are nearly identical.
- Score inside a danger zone. MODS reports results both on the whole image and inside a danger zone around the USV, sized as the distance reachable in ten seconds at an average 1.5 m/s. Misses close to the boat matter more than misses at the horizon, and a single averaged score hides them.
- Report recall per class. Measure persons, small craft and buoys separately. A model can post a high overall F1 on vessel-heavy footage while missing most swimmers.
- Slice by condition. Tag the real test set by glare, sea state, visibility and time of day, and compare each slice. Our case study found its oblong-buoy gap by inspecting misses on the real test sequence, then fixed it with 500 targeted synthetic images.
- Check false positives on glitter and foam. Count false detections per hundred images on open-water clips with heavy glare and whitecaps, because a lookout that triggers constantly gets switched off.
- Test the whole loop in simulation before the sea trial. Run the perception model and the planner together through scripted head-on, crossing and overtaking encounters, then confirm the behavior at sea.
The broader evidence on training with synthetic data, including how much real data to mix in, is collected in Does Synthetic Data Work For Object Detection?.
Where Stardust Fits
Stardust is Bifrost's synthetic data platform, and collision avoidance and automated lookout are among the use cases on its maritime page. Its asset library includes commercial vessels (tankers, container ships, ferries, tugboats and more), defense vessels, USVs, buoys, inflatables, kayaks, debris and persons overboard. Scenarios set vessel positions, traffic patterns, sensor placement, weather, sea state and sun position, keyframed over time, and conditions include fog, mist, rain, snow, high winds and night. Sensors include RGB, stereo, SWIR, MWIR and thermal, with depth and segmentation in registration and marine radar in development. Every frame comes with pixel-level masks, 2D and 3D boxes, range, bearing and speed for each vessel, and per-frame scenario metadata.
In the case study, one engineer trained a sailboat and buoy detector on 2,500 synthetic images over two training runs on a single desktop GPU, and reached 82 percent F1 on a real MODS test sequence. The maritime page shows Saronic, Havoc, Seadronix, ClassNK and ST Engineering under "Trusted by the teams building autonomy on the water." If you want an open-source simulator to modify yourself, ASVSim is the better fit, and Synthetic Data Platforms Compared covers the commercial alternatives.
Sources
- Bovcon, Muhovič, Vranac, Mozetič, Perš and Kristan, MODS: A USV-oriented object detection and obstacle segmentation benchmark, IEEE T-ITS
- Žust, Perš and Kristan, LaRS: A Diverse Panoptic Maritime Obstacle Detection Dataset and Benchmark, ICCV 2023
- Bovcon, Muhovič, Perš and Kristan, The MaSTr1325 dataset for training deep USV obstacle detection models, IROS 2019
- Prasad et al., Singapore Maritime Dataset, from "Video Processing from Electro-optical Sensors for Object Detection and Tracking in Maritime Environment: A Survey," IEEE T-ITS 2017
- Nirgudkar, DeFilippo, Sacarny, Benjamin and Robinette, MassMIND: Massachusetts Maritime INfrared Dataset, IJRR 2023
- Ward, Harguess and Corelli, Leveraging synthetic imagery for collision-at-sea avoidance, 2019
- Lesy et al., ASVSim (AirSim for Surface Vehicles)
- Li et al., MMUSV-Sim: A Perception-Oriented Simulation and Data-Generation Platform for Multi-USV Cooperative Perception, 2026
- Tran, Shipard, Mulyono, Wiliem and Fookes, SafeSea: Synthetic Data Generation for Adverse and Low Probability Maritime Conditions, 2023
- Han, Cho, Kim, Kim, Son and Kim, Autonomous collision detection and avoidance for ARAGON USV: Development and field tests, Journal of Field Robotics, 2020
- Han, Lee, Kim and Kim, A survey on maritime perception datasets and technologies for autonomous surface vessels, Intelligent Service Robotics, 2026
- U.S. Coast Guard, Amalgamated Navigation Rules, International and Inland
Frequently Asked Questions
Can an autonomous boat be trained on synthetic data alone?
A detector can reach useful accuracy on synthetic data alone for simple classes. In one Bifrost case study, a YOLOv11n model trained only on 2,500 synthetic images reached 82 percent F1 detecting sailboats and buoys on a real MODS test sequence. Most teams still keep real footage from their own vessel for testing and fine-tuning, because the real camera, waters and traffic always differ from any simulation.
What is the best public dataset for maritime obstacle detection?
For a camera on a small USV, LaRS is the most diverse, with over 4,000 per-pixel labeled key frames from lakes, rivers and seas, and MODS is the standard benchmark for detection and segmentation from a moving USV. MaSTr1325 is a common training set, the Singapore Maritime Dataset covers larger vessels in visible and near-infrared, and MassMIND covers LWIR thermal. People in the water are scarce across them, at 0.7 percent of the dynamic obstacle labels in MODS, for example.
How do you detect persons overboard and swimmers from a USV?
Train on many examples of small heads and bodies in the water across sea states, ranges and lighting, and measure recall for that class separately. Persons are rare in public USV data, at 0.7 percent of the dynamic obstacle labels in MODS, so synthetic data is the practical way to get enough examples. Thermal cameras help at night because a person in cold water shows strong contrast.
Which simulators are used for USV perception research?
Published options include ASVSim, an open-source AirSim-based simulator for inland waterways and ports under the MIT license, and MMUSV-Sim, an Unreal Engine 5 platform for multi-USV cooperative perception. NAVHAZ-Synthetic used the Unity engine to render one million labeled vessel images, and SafeSea used diffusion models to change sea state in real images. Commercial platforms such as Bifrost Stardust are an alternative when a team does not want to build its own pipeline.
How does synthetic data help with COLREGs compliance?
COLREGs decisions depend on knowing what kind of vessel is nearby and which way it is heading, because rules differ for power-driven, sailing and fishing vessels and for head-on, crossing and overtaking situations. Synthetic data can render every vessel type at every aspect angle with labels for class, heading, range and bearing, which is hard to collect at sea. The perception output then feeds a planner that applies the rules.
