Can AI Learn in a Virtual Space That Doesn't Match Reality? Simulation Environments for Training Physical AI

01What Is Physical AI? Why Train It in Simulation?

AI that perceives its surroundings through sensors, makes decisions, and moves real machines, such as robots, self-driving cars, and construction machinery, is called physical AI. The key difference from generative AI that handles text and images is that the AI's decisions come back as the movements of real machines.

AI that writes text can learn from vast amounts of text on the internet. But data on "the results of moving," such as how soil collapses depending on how an excavator's arm is moved, can only be collected by actually operating the machine. Collecting it on a real site compares with collecting it in a virtual space as follows.

In particular, reinforcement learning, in which the AI repeats trial and error and learns the actions that worked, involves an enormous number of attempts including failures, so doing it only with real machines is not realistic. Training data created in a virtual space is called synthetic data, and the correct labels for objects in images can also be attached automatically. NVIDIA, which provides development platforms for physical AI, also explains in its glossary that physically based simulation and synthetic data are used to train and validate models, including in rare and safety-critical scenarios, before they are deployed in robots and vehicles.

In construction, too, automating machinery has become national policy. "i-Construction 2.0," formulated by Japan's Ministry of Land, Infrastructure, Transport and Tourism (MLIT) in April 2024, aims to reduce the workforce needed on construction sites by at least 30% (a 1.5-fold increase in productivity) by FY2040, and makes "automation of construction work" one of its three pillars.

02How Simulators for People Differ from Simulators for Training AI

In the sense of reproducing a site in a virtual space, an AI training environment is the same as conventional training simulators. However, what matters when building it changes depending on whether the user is a person or an AI.

In human training, it does not matter if the leaves on distant trees are a little rough. People fill in rough visuals from experience and can focus on how the machine responds when they push a lever. AI, on the other hand, learns by taking the sensor data it is given at face value. In other words, a simulator for people focuses on matching "how people perceive it" to reality, while a simulator for AI focuses on matching "how things appear to the sensors and how objects move".

03What Is the Sim2Real Gap Between Virtual and Real?

When AI trained in simulation is run in the real world, its performance can drop compared with inside the virtual space. This difference between the virtual (Simulation) and the real (Real) is called the Sim2Real gap. It is similar to someone who drives well on a driving school course but is thrown off the first time they go out on public roads.

Gaps arise because a virtual space does not reproduce everything in reality. On top of that, AI may learn to rely even on quirks that exist only in the virtual space, such as always-identical lighting or noise-free sensors.

So should everything be made to look exactly like reality? Not necessarily. In a study published by Tobin et al. in 2017, an AI that finds the positions of objects was trained only on simulated images with random patterns that could never exist in reality. Even so, the researchers report that this AI was able to detect object positions in real camera images with an accuracy of 1.5 cm.

This method is called domain randomization: by learning in conditions where the appearance changes every time, the AI comes to make decisions without relying on appearance. However, this study was an experiment in finding object positions, and the same results will not necessarily hold for every task that involves moving machines. Even so, the idea that matching the parts the AI uses for its decisions to reality matters more than making everything look real is a starting point for building environments.

04Three Elements Where a Simulation Environment Needs Accuracy

While AI can still learn when appearance is varied, some parts cannot be glossed over: object geometry and position, physical motion, and sensor output.

Build geometry and space from real measurement data

If the terrain, buildings, and equipment in the virtual space differ from reality, the routes and work procedures the AI has learned will not fit the site. In an MLIT PLATEAU demonstration, "Autonomous Mobility Operation System Using 3D City Models and BIM," point cloud data created by running a virtual LiDAR through a virtual space containing a 3D city model was used by a real vehicle to estimate its own position. The report states that an issue remained: where there was equipment not reproduced in the 3D city model, such as plantings and fences, the virtual space and reality diverged, making autonomous driving difficult.

To match even small equipment to reality, the site is measured with laser scanners or photos, and the virtual space is built from that 3D data. Dimensions can be captured from the measured collection of points (a point cloud), and the look of the site can be captured from 3D Gaussian Splatting (3DGS) created from photos.

The differences between point clouds and 3DGS are explained in detail in What Is 3DGS (Gaussian Splatting)? Differences from Point Clouds and Uses in Construction.

Match physics and motion with real-machine data

If the weight of the arm, hydraulic response delays, how soil collapses, or the friction between tires and the ground differ from reality, AI that learns how to apply force will move incorrectly on site. Settings for physics simulation are not decided from catalog values alone; they are tuned by comparing them with data recorded by operating the real machine.

Reproduce sensors down to their real-world quirks

What AI receives on site is sensor data that contains shake and noise. Camera lens distortion and changes in brightness, LiDAR noise, and cases where lasers do not return from glass or puddles should also be built into the virtual sensors. It is also essential to make the sensor mounting position, orientation, and field of view the same as on the real machine.

05How to Reduce the Sim2Real Gap

Even if these three elements are made accurate, the gap will not reach zero, because events that were not anticipated in simulation always happen in reality. So instead of trying to finish everything in one go, you repeatedly feed the results confirmed in reality back into the virtual space. MLIT's "Physical AI Strategy for the Construction Sector (Interim Summary)," compiled in September 2026, also states that it is important to build a cycle of repeated improvement between the virtual space and actual sites, on the premise that differences between simulation and real-machine operation will arise, while narrowing those differences.

Stage 2 is especially important. If you compare the motion of the virtual and real machine under the same operation before training the AI, then when the AI performs poorly later, it is easier to tell whether the cause lies in the AI or in the environment.

Some parts, such as road surfaces on rainy days or fine soil particles, cannot be reproduced accurately no matter what. For those parts, the AI is trained with varied conditions so that it does not rely on those differences.

06WAN Builds AI Training Environments from Its Experience Creating Virtual Spaces and Simulators

Since its founding in 2010, WAN has created virtual spaces with 3DCG and XR and developed simulators for construction machinery and more. We have also built simulation environments for major companies in telecommunications, construction machinery, railways, and other industries.

Our in-house construction machinery simulator, METATORE, reproduces construction sites with 3DCG and physics simulation (Figure 1). We apply the site reproduction and physics tuning we have built up through developing simulators for people to building environments for training AI.

On the research side, we are working with universities on the research and development of METATORE+. The project was selected for the FY2026 Growth-Oriented SME R&D Support Program (Go-Tech Program), and works on using AI to quantify the skills of experienced operators and pass them on to the next generation and to machines.

• Contract development: We develop simulation environments usable for AI training, from measuring sites and equipment to reproducing the virtual space, machines, and sensors
• In-house development: Through developing METATORE, we have built up technology for reproducing construction machinery motion and for physics simulation
• Joint research: We work with universities and companies on research that connects decisions learned in virtual spaces to the movements of real machines

We welcome inquiries from the concept stage, such as "Can this site be reproduced in a form usable for AI training?" or "Can we start from the 3D data or drawings we already have?"

07Summary: Plan an AI Training Simulation Environment Around What to Teach, How Accurately to Reproduce, and How to Verify in Reality

As Waymo's more than 20 billion miles in simulation show, much of AI's experience is built up in virtual spaces. Here are three things those preparing the environment should do so that AI trained in a virtual space works on site.

What to teach: Decide first on the sensors and decisions the AI will use

First decide which machine will make what decisions based on which sensor data. Once this is decided, the parts to build in detail in the virtual space and the parts that can be varied become clear.

How accurately to reproduce: Match geometry, physics, and sensors with real data

Gather 3D measurement data of the site, records of the real machine's motion, and sensor specifications before building the virtual space. Before polishing the visuals, match the values of the parts the AI uses for its decisions to reality.

How to verify in reality: Feed real-machine test results back into the virtual space

Compare the virtual and real machine's motion before training, and after training, add the scenarios where performance dropped on the real machine to the virtual space. Building the verification stages and their costs into the plan from the start makes it less likely that the project will stall partway.

When using simulation for AI training, we recommend deciding three things before you start building the environment: the decisions the AI will learn, the sensors it will use, and where and when it will be tested on real machines.

Consultation

Development of simulation environments for AI training
We develop simulation environments usable for AI training, from 3D measurement of sites to reproducing the virtual space, machines, and sensors. We also welcome consultations at the concept stage and about joint research.

Talk to us about simulation environments