A robot model should have to pass before anyone trusts it.

The strongest evidence isn't forum complaints. It's a peer-reviewed-style survey of 82 robotics developers — Afzal, Katz, Le Goues & Timperley, "A Study on the Challenges of Using Robotics Simulators for Testing" (2020) — plus the ROS ecosystem's own quality-assurance reporting. Their findings read like a requirements document for this tool.

"Most simulations are NOT designed to run headless, nor are they easily scriptable for automatic invocation." — Survey respondent P34, in Afzal et al. 2020 Via index

• Headless is the exception, not the default. Participants preferred headless execution for automation and named its absence a major difficulty. One respondent (P37) had to connect a physical display to a Jenkins CI server to run simulation properly; another (P24): "no obvious test harnesses available for the simulation environments I use."
• CI deployment is a named failure point. Respondents reported simulators difficult to deploy in CI/cloud (P62), being unable to set up a CI pipeline on GPU machines (P66), and cloud GPU VMs being expensive (P77). Gazebo's own documentation now concedes that "setting up a Gazebo environment in a Continuous Integration (CI) pipeline can often be a complex and time-consuming task" — and ships a setup-gazebo GitHub Action to mitigate it. Gazebo CI docsVia index
• Resets aren't repeatable. P7: "Resetting gazebo simulations was not repeatable enough to get good data." The survey's conclusion about the de facto ROS simulator is blunt: Gazebo "does not adequately support headless execution, lacks reproducibility, and performs poorly" on complex environments.
• The GPU-sensor path was historically blocked. Headless runs once required an OpenGL X server; Gazebo's headless-rendering documentation calls the workaround steps "complex and error prone" and documents the EGL-based --headless-rendering path (OGRE2 only). The same doc notes running simulation on a remote computer is "often desirable." Headless rendering docsVia index
• ROS QA is inconsistent at the process level. The ROSIN / ROS-Industrial quality-assurance report found QA practices inconsistent across the ROS community, no QA ownership, and roughly 1 in 5 ROS bugs detected at runtime — often by end users — where a CI service was expected to catch many errors earlier. ROSIN D3.1 reportVia index
• Reproducibility is still an open research problem. A 2026 paper on deterministic ROS 2 execution (RSLCPP) notes ROS 2's asynchronous multiprocess design complicates reproducibility across hardware, "complicating benchmarking and CI." arXiv 2601.07052Via index

Design consequence, stated up front: because bit-exact determinism isn't on offer, the gate reports spread — run each maneuver k times and publish the distribution — instead of pretending one run is proof. See §7, item 8.

URDFs are written by hand, checked by tools that only verify the file parses, and converted to SDF by a pipeline that can drop parts of the model without a word. The failures surface later — as a robot that shakes, explodes, or quietly drives wrong.

"Making URDF files is a tremendous pain as the only good way to do it right now is by hand which is faulty and error prone." — Survey respondent P4, in Afzal et al. 2020 Via index

Why robots "explode" in Gazebo

Wrong inertia is the classic cause. A ROS Answers thread attributes Gazebo shaking to "bad inertia matrices, bad physics settings, joint problems," with debugging described as a process of elimination, parameter by parameter. A community wiki's list of "exploding robot" causes: PID gains, inertia values too small, collision overlaps, joints starting outside limits, solver instability. A beginner-facing differential-drive tutorial states it plainly: "Robot vibrates or explodes in Gazebo: inertia values are wrong… No value should be 0.0 or below 1e-6" — and warns wheels render sideways if the cylinder joint axis isn't rotated (rpy="${pi/2} 0 0").Via index

A simulation textbook chapter frames the deeper version: "a robot description good enough for visualization is not automatically good enough for contact-rich control" — a missing <inertial> block can yield high simulation success and near-zero hardware success while compiling without any error. Its audit checklist covers units, mesh scale, inertias, joint axes, damping, limits, and collision shapes.Via index

The inertia rules themselves are well specified — the matrix must be symmetric positive-definite and satisfy the triangle inequalities (ixx + iyy ≥ izz, etc.), and every link needs physically plausible mass, center of mass, and inertia. What's missing is any tool that enforces them automatically. Gazebo inertia tutorialVia index

The conceptual core of the gate — comparing what the robot reports against what actually happened — is a live, open source of confusion inside Gazebo itself. Two open gz-sim bugs are users hand-building exactly the comparison RobotForge automates.

DiffDrive odometry "is not ground truth… it simulates how odometry would be calculated from the model's kinematics and joint state." Ground truth comes from a different system: OdometryPublisher. — Gazebo maintainer, in gz-sim #872 Read live

• gz-sim #872 (open since 2021-06-21, labeled bug and help wanted): a robot spawned at (2, 2) reported odometry starting at (0, …). The reporter asked for clearer semantics; the maintainer's clarification above is the distinction the gate productizes — DiffDrive odometry is the device under test, OdometryPublisher is the reference. Read the issue
• gz-sim #2862 (open, labeled bug, filed 2025-04-15): odometry found "delayed or stale… usually behind robots actual position" — discovered by a user running an "exhaustive test of the diff drive system" to simulate an EKF filter node. That hand-built test is a ready-made regression scenario for the gate. Read the issue
• The maneuvers are classical, not invented. UMBmark (Borenstein & Feng, 1996) — the standard systematic-odometry-error method — uses straight runs and 90° spot turns precisely because they expose the two dominant systematic errors: unequal effective wheel diameters (E_d) and uncertain effective wheelbase (E_b). The gate's two tests are that method, miniaturized. UMBmark paper (PDF)Via index

Try the arithmetic the gate automates

From a wheeled-robot odometry guide's worked example: a nominal wheel radius of 33.0 mm against an actual 32.5 mm (tire compression) is a 1.5% distance error — 15 cm over 10 m — and wheel slip is undetectable from encoders alone. Drag the sliders and watch a sub-millimeter modeling error become a failed run.Via index

The same numbers live in two files that must agree

Wheel radius and separation appear in the URDF geometry, again in the DiffDrive plugin's SDF tags (<wheel_separation>, <wheel_radius>, <left_joint>, <right_joint>), and again in the ros2_control controller YAML. Project documentation warns that URDF-vs-YAML disagreement produces odometry error or controller startup failure — a cross-file consistency hazard no single-file linter catches. Vendors confirm real robots never match CAD: PAL Robotics' PMB-2 calibration docs cite wheel diameter tolerances of ±1–2 mm, tire wear, assembly tolerances, tire pressure, and floor interaction, corrected with empirically determined multipliers.Via index

Scope honesty: a gate comparing odometry to simulator ground truth catches model self-inconsistency — wrong radius, separation, or axes in the description versus what the simulated physics does. It does not close the sim-to-real gap, and the pitch shouldn't claim it does.

The primary audience source is Snider et al., "University Rover Challenge: Tutorials and Team Survey" — a book chapter surveying URC teams — plus current competition reporting. URC draws hundreds of students across 80+ teams (in 2017: 35 rovers, 500+ students) who spend a year building a rover from scratch under a $15,000 spending cap, which makes ROS's free ecosystem "a natural fit… to cut costs." This is precisely the segment that cannot buy test infrastructure.Via index

"Teams most often expressed difficulty using IMUs or not testing their rover enough." — Snider et al., URC team survey Via index

• The winning teams' advice is "test more, earlier, with everything." Of 2nd-place Team Continuum: asked what they wished they had done differently, "the answer was always the same: more field tests with all components of the rover" — advice the chapter notes is "often given by other teams." Field testing forces the team to finish sooner, but it demands the whole team organized and present with the finished rover. Today, that expensive in-person route is the only route to the feedback.
• Odometry failing at competition is documented. Team R3 (Ryerson): "As a result of wheel slippage on sand, Team R3 did not use wheel odometry… because our IMU was not working well enough to produce a good fused result, at the competition we did not actually use the ekf localization ROS nodes" — they fell back to visual odometry only. Another surveyed team "had plans to use MoveIt but due to lack of testing time used velocity control… at competition."
• Current practice is a human watching a GUI. The University of Michigan MRover team's autonomy starter project instructs new members to open Gazebo and a visualizer, run the navigation node, and "Watch it move to (5.5, 2)… see the tag, and finally reach the tag" — pass/fail is eyeballs, with no automated verdict and no recorded evidence. (Wiki-mirror source; indicative.) MRover starter projectVia index
• It is still happening in 2026. Michigan's MRover finished 3rd at CIRC and 5th at URC 2026 after "a summer filled with troubleshooting and continued testing" — and its write-up documents competition features, built for the new Heist mission, where "many of the tasks the rover had to perform had not yet been tested ahead of the competition" and the team "did not have enough time to fully test and get this feature working before the competition." (The same write-up's forfeits were hardware — a Jetson that failed to reboot, traced to a bad PCIe card, and damaged wiring. Relevant to under-testing generally; not model-validation failures.) MRover CIRC 2026 write-upVia index
• The robots are real field machines. CIRC's 2026 rules specify a single self-contained rover, a <50 kg target, a 0.8 m door constraint, ≥1 h per task, and operation in rain, mud, snow, and darkness — outdoor robots whose testing today is physical by default. CIRC 2026 rulesVia index

The headline question — does anything already produce a pass/fail verdict with evidence, for a small team, headlessly, for free? — answers no. Each tool below covers a slice. Static validators sit below the gate (no execution), Nav2's tests beside it (wrong scope), Isaac Sim above it (a platform, GPU-gated), and RoboMaker is gone.

Beyond syntax validation and the GUI inertia procedure, Gazebo itself ships headless server mode with the EGL --headless-rendering option for GPU sensors, and CI packaging via the setup-gazebo GitHub Action — the sanctioned building blocks for the gate's harness. No shipped "run maneuvers and grade the model" tool appears in the documentation reviewed.Via index

Every issue below was opened and read live during this research (October 25–26, 2026) and was open at read time, unless flagged. These are not decoration: two of them are users hand-building the gate's core comparison, and three define constraints its harness must be designed around.

Eight candidate tests for a 10-day MVP, ranked by strength of documented evidence × cheapness to implement in a headless Gazebo harness × directness of fit to the odometry-vs-ground-truth concept. Each item names the evidence it answers.

1. 1

   Structural URDF/SDF sanity

   File parses (check_urdf + gz sdf -k); joint/link references resolve; no kinematic loops or duplicate names; mesh files exist; and — the differentiator — every <gazebo> extension reference resolves to a real link, via a post-conversion diff of declared vs. actually-present sensors and plugins.

   Evidence: sdformat #1372 (extension "completely ignored, without any warning") · check_urdf's documented limits · Afzal P4 on hand-authored URDF pain.

2. 2

   Mass / inertia plausibility

   Every non-fixed link has mass > 0 and an <inertial>; inertia is symmetric positive-definite; triangle inequalities hold; center of mass sits within link geometry / support region; total mass compared against a user-declared expectation.

   Evidence: Gazebo inertia tutorial (rules; rotated inertia "happens often") · UR5e wrong vendor masses (#186) · Fetch/PR2 defects in urdf-validator's examples · shaking/exploding-robot threads.

3. 3

   Cross-file consistency — wheel parameters

   Wheel radius, wheel separation, wheel joint names, and joint axis direction agree across URDF geometry, DiffDrive plugin / SDF tags, and controller YAML.

   Evidence: URDF-vs-YAML mismatch → odometry error / startup failure · PMB-2 CAD-vs-real divergence and multiplier practice · DiffDrive plugin parameter surface.

4. 4

   Spawn-stability check

   Spawn headless, step N seconds uncommanded. FAIL if the robot falls through the ground plane, tips, translates or rotates beyond a small threshold, or produces NaN / exploding joint states.

   Evidence: "Robot vibrates or explodes in Gazebo: inertia values are wrong" · exploding-robots cause list (inertia, collision overlap, joint limits, solver) · ROS Answers shaking-by-elimination debugging.

5. 5

   Straight-line odometry test (~1 m)

   Command a straight drive; compare DiffDrive/controller odometry against OdometryPublisher ground truth: distance error %, lateral drift, heading drift. PASS within tolerance.

   Evidence: gz-sim #872 (odometry ≠ ground truth; maintainer clarification) · gz-sim #2862 (wrong/stale odometry, open bug) · wheel-radius error math (1.5% → 15 cm over 10 m) · UMBmark methodology.

6. 6

   Turn-in-place heading test (~90°)

   Rotate in place; compare odometry yaw change against ground-truth yaw change; report heading error in degrees and %.

   Evidence: UMBmark — 90° spot turns expose uncertain effective wheelbase (E_b), one of the two dominant systematic errors · PMB-2 / TurtleBot4 separation-multiplier practice.

7. 7

   Sensor / topic presence checks

   After spawn: odometry, joint_states, and each declared IMU/LiDAR/camera topic actually publishes at an expected rate; TF odom→base_link exists.

   Evidence: sdformat #1372 silent sensor drop · MRover-style manual topic/visual checking today · gz-sim #2668/#2803 ground-truth topic semantics.

8. 8

   Square-path closure + repeatability reportStretch

   Run 4× (straight + 90° turn); report closure error. Repeat each maneuver k times and report the spread, not just the mean — repeatability reporting is itself the feature, given the field's reproducibility findings.

   Evidence: full UMBmark procedure (4×4 m square, 5 CW + 5 CCW), scaled down · Afzal reproducibility findings ("not repeatable enough to get good data").

Design constraints the evidence imposes

• Ground truth from OdometryPublisher / PosePublisher; DiffDrive odometry is the device under test — per the maintainer semantics in #872. Document that OdometryPublisher pose is currently noise-free world pose (#2668), and resolve topics via the model namespace rather than hard-coding (#4104).
• The evidence report is half the product. Per test it should carry: commanded maneuver, measured odometry value, ground-truth value, error, tolerance, verdict, and the model/config hashes. No tool in §5 provides this.
• Ship the harness as a container / GitHub Action from day one. Gazebo's own CI doc calls setup "complex and time-consuming," and Afzal's respondents failed at exactly this step; the setup-gazebo action is the sanctioned building block.
• Phrase determinism honestly: fixed scenario + stated tolerances + repeatability statistics (§1/§3) — never bit-exactness.

Open questions, stated plainly. Anything on this list should not appear in a pitch as established fact.

1. ?

   "Teams can't test because the robot is in a lab while members are remote"

   No direct source found. Supported only indirectly (headless/CI demand in Afzal et al.; Gazebo's remote-execution rationale). Do not present as documented.

2. ?

   A competition failure traced to an unvalidated URDF / odometry model error

   Not found. Documented competition failures were hardware (MRover PCIe/wiring, CIRC 2026) and real-robot sensing (Team R3 sand slip / IMU, URC survey). Related, but distinct.

3. ?

   Nav2 system-tests documentation

   The docs page returned "Access Denied" when opened via the index; its characterization in §5 rests on the search-result snippet of that page.

4. ?

   ros2_controllers #2251

   Title and date come from a live GitHub search listing only; the issue page was not individually opened.

5. ?

   Isaac Sim requirements

   Read via the search-result text of NVIDIA's official requirements page (5.1.0), not a full page open; version-specific numbers will drift.

6. ?

   TurtleBot4 wheel_separation_multiplier

   The value 1.06793648481 and its comment come via a DeepWiki summary of the configuration (snippet-level), not the raw config file.

7. ?

   Formant and The Construct characterizations

   Light-touch, from vendor/roundup material at search level; neither was deep-read. Neither affects the §5 conclusion.

8. ?

   LinkForge / fieldpilot-urdf

   Physics-linting claims seen in search results were not opened and are excluded from this brief entirely.