Hesai Launches Kosmo Spatial Camera to Build Robot Training Environments in 15 Minutes

The 3D world used by Amap and Pixar comes from this domestic AI spatial camera.

How much preparation is needed to hand over an ice cream shop to a robot?

Robotic arms, dexterous hands, cameras, and the models responsible for perception and decision-making constitute the visible system. However, according to Li Yifan, co-founder and CEO of Hesai Technology, making robots truly operational requires solving a series of "spatial" problems.

When discussing the collaboration between Sharpa and DQ at the launch event, he listed three goals the team aimed to achieve: minimizing modifications to the existing environment, completing the full task loop, and handling actual production work. He noted that making ice cream involves more than fifty steps, and the team hoped the robot could take over the entire process.

To enable the robot to master this workflow, the team first prepared a "digital store" for it: transferring the layout, equipment, and operational space of a real store into a virtual environment for training.

Hesai stated that while professional teams previously required about 300 hours to manually model and reconstruct a store, their device can complete scanning and generate a highly faithful 3D training environment in approximately 15 minutes. This reduced Sharpa's failure cases to one-tenth of the original number, achieving a task success rate of 99%.

This is also a key issue currently being discussed in the field of spatial intelligence. Fei-Fei Li summarizes spatial intelligence as the ability of AI to perceive, reason, and act within 3D space. Compared to visual models that only answer "what is seen," spatial intelligence further requires AI to understand where objects are located, the geometric relationships within space, and how the world changes after an action occurs.

So, how exactly does a real store become a place where robots can learn repeatedly?

On October 14, Hesai launched Kosmo, a 3D world data engine comprising the spatial camera Kosmo One and the cloud platform Kosmo World. The former is responsible for capturing real-world spaces, while the latter handles reconstruction, understanding, and editing, delivering data to workflows such as robot simulation, world models, and 3D creation. Currently, this data has already entered Pixar-related creative workflows and Amap's spatial display applications.

After scanning a store, what information is retained? How is this information transformed into training trajectories usable by robots?

Why build a dedicated spatial camera?

What kind of digital store do robots need?

The color and texture of counter surfaces affect visual recognition; the position of table edges and the spacing between equipment relate to approach paths and operational space; whether a cup is obstructed by equipment when viewed from a different angle changes the observations available to the robot.

Therefore, a digital environment suitable for training needs to simultaneously preserve appearance, geometry, and spatial relationships.

Professional teams can certainly accomplish this through traditional CG modeling, but the problem lies in the fact that stores, factories, and hotels in the real world are all different. If modeling must start from scratch every time a new scene is entered, costs increase rapidly with the complexity of the real world.

On-site scanning provides another path: instead of assuming a standard space beforehand, specific locations are directly recorded.

Kosmo One integrates lidar, imaging systems, and high-precision IMUs into a single device. Lidar provides spatial measurements, cameras record color and texture, and IMUs provide information for device motion estimation. Through synchronization, calibration, and fusion, data from different sources is placed into a unified spatial reference frame.

What does millimeter-level precision mean? Xiang Shaoqing demonstrated a tennis racket at the launch event.

He explained that the mesh holes on the racket have a spacing of about 5 millimeters, and Kosmo's custom lidar can clearly render these fine structures. This demonstration provided an intuitive reference for abstract precision metrics: when the scanning object shifts from room outlines to fine structures, measurement capability determines which details can be preserved.

The imaging system uses a large-format 1-inch CMOS sensor, supporting up to 8K capture, with final exported files reaching a maximum resolution of 6K. However, in 3D reconstruction, there is another issue beyond resolution: Can multiple cameras maintain consistency when capturing the same object?

Xiang Shaoqing used a floor tile to explain this problem. As the scanner moves around the room, the same floor tile repeatedly enters the frames of different cameras. As shooting angles and backgrounds change, automatic exposure, white balance, and other processing adjustments may vary, causing the same surface to appear with different brightness and colors.

These features improve the aesthetic quality of individual photos, but 3D reconstruction requires organizing multiple moments and viewpoints together. If the appearance of the same surface constantly changes, both texture restoration and continuity during viewpoint switching are affected.

Therefore, Kosmo customized the ISP (Image Signal Processor)—the system that processes raw signals from image sensors into images—for 3D reconstruction. It needs to balance consistency across cameras and time points, allowing different frames to jointly describe the same space.

Accurate correspondence between geometry and imagery is also necessary. When a scanner walks past a counter, the camera captures an image, the lidar obtains a set of measurements, and the device itself is moving. The system needs to know the exact time and location each piece of information originates from so that the edge of the counter in the image aligns with the edge in the spatial measurements.

Thus, the completeness of a spatial camera must be reflected throughout the entire acquisition chain: obtaining sufficiently detailed geometric and visual information while ensuring this information works together seamlessly.

The usage process must remain as simple as possible. When introducing Kosmo One, Xiang Shaoqing specifically emphasized a physical button and a screen button: users can perform operations such as powering on and scanning. His goal for this product was to make spatial modeling as easy as taking a photo.

Hesai stated that under typical indoor conditions, scanning a 100-square-meter space takes approximately 15 minutes, though the exact time varies depending on scene structure and acquisition requirements.

However, once scanning is complete, what is obtained is merely a spatial record. The next critical step is how to process it into reusable training assets.

How does a single scan transform into massive training trajectories?

During an interview, Li Yifan posed a question to the media: Do people tend to misunderstand 3D reconstruction as simply coloring point clouds or pasting photos onto walls?

He then gave an example: looking at a clock on the wall.

When people view the same clock from different positions, its shape, reflections, and details change. If only a front-facing photo is retained, it becomes difficult to fully represent how the clock appears when viewed from the side. As robots move through environments, such perspective changes occur continuously.

Kosmo Spatial Core, the underlying system of Kosmo World, can generate 3DGS (3D Gaussian Splatting), Meshes, and Depth maps in a single reconstruction process. These three representations serve distinct needs: appearance, geometric surfaces, and distance information, respectively.

3DGS, or 3D Gaussian Splatting, reconstructs scenes using numerous Gaussians parameterized by position, shape, opacity, and color, enabling rendering from novel viewpoints. When a virtual camera moves along a counter, the system can generate the objects, textures, and occlusion relationships visible from that specific position.

Meshes describe object surfaces using vertices, edges, and faces, providing the geometric foundation for collision detection and interaction modeling. Depth maps describe distance information from a specific viewpoint and can be combined with images for spatial perception.

The spatial alignment of these three representations means that the counter seen in the image corresponds to the counter in the geometric model and the distance data in the depth map. This allows training systems to obtain different forms of spatial information within the same environment.

Next, the virtual camera must be positioned where the robot actually observes.

A tabletop visible to a standing person might be obscured by the table edge from the perspective of a lower-mounted camera. Changes in camera mounting height, orientation, and field of view alter the size, visibility range, and occlusion relationships of objects in the frame. Hesai explained that data generated by Kosmo can be visually adapted to the specific characteristics of the target robot’s cameras, making observations in the digital scene more closely match actual usage conditions.

The scene itself can also vary.

Kosmo World supports AI-driven semantic segmentation, allowing it to identify and extract objects such as chairs, sofas, and cameras, which can then be moved and edited. This transforms static spatial records into reconfigurable environments.

For instance, if a chair in a store is moved from beside a table to the aisle, the walkable area shrinks, and some items may become occluded. Even if the camera follows the original path, the resulting imagery will differ. These adjustments can create new training or testing conditions to observe how well models adapt to layout and occlusion changes.

Editing also reveals areas not captured during initial acquisition. After removing a sofa, where does the carpet underneath come from?

In the launch demonstration, Kosmo used AI inference to complete the area previously hidden by the sofa. This processing makes objects easier to move and scenes more complete. However, since completed areas involve model inference, training teams must judge their suitability based on task requirements: there are different data quality standards for using them as background versus as precise contact and manipulation surfaces.

With environments that can be observed from various perspectives and edited, trajectory generation becomes feasible.

Capturing a video segment in a real store requires a person or device to move along a route; in a digital store, one can reset the virtual camera’s position, direction, height, and path to generate continuous frames along the route, along with corresponding depth and pose data.

More importantly, a single real space does not correspond to just one fixed dataset. By adjusting object positions and changing virtual camera observation conditions, new scenes and training trajectories can be continuously generated. Hesai stated that tens of thousands of training trajectories can be automatically generated based on a single 3D space.

A single spatial record can continuously generate data around different observation conditions, and teams can increase sampling in error-prone areas. For example, if a device frequently occludes the target, more observation paths can be set around the relevant area to check visibility from different angles.

However, different trajectories carry different types of information.

Observation sequences generated by virtual camera motion can support multi-view learning, spatial perception, and related world model training. Operational trajectories for robots performing tasks like grasping and moving involve joint actions, contact states, and task outcomes, requiring the joint participation of robot models, physics simulations, and policy systems.

So, if a scene can already be continuously edited to generate new training conditions, why is it still necessary to visit more real stores?

Because what robot training truly aims to solve is the ability to handle diverse scenarios. In an interview, Li Yifan noted that the essence of robot training data is generalization: once a room is digitized, it can serve as a training sample for similar scenarios, but moving from dining to retail, hotels, and homes, the spatial differences in the real world continue to increase.

One room can change endlessly, while more real rooms provide new structures, objects, and spatial relationships. The former amplifies the value of a single dataset, while the latter continuously provides new "worlds."

In Li Yifan’s judgment, the more scenarios AI can handle, the greater the demand for real-world data.

Thus, the value of a single scan extends beyond the scanning process itself.

From Spatial Cameras to Data Engines: How Far Does Hesai Aim to Go?

Over the past decade, Hesai has been best known for its LiDAR business, with applications extending from autonomous driving to robotics. Additionally, Hesai’s LiDAR was used for scanning ancient architecture in Black Myth: Wukong.

In the interview, Li Yifan first addressed a question: Why would a LiDAR company develop Kosmo?

The product originated from customer needs. Clients in surveying, construction, and other fields increasingly demanded higher-precision LiDAR, pushing accuracy from centimeter-level to millimeter-level. After obtaining the radar, these clients still had to configure cameras, inertial navigation systems, and computing platforms to build a complete spatial acquisition system.

Seeing different teams repeatedly perform similar integration work revealed a product opportunity to Hesai: designing sensors, acquisition systems, and subsequent data processing together. Capabilities accumulated in complex opto-mechanical-electrical systems, custom chips, calibration testing, and productization could extend into spatial acquisition.

This extension has clear engineering relevance. Kosmo requires simultaneous handling of geometric measurement, image quality, sensor synchronization, and on-site operation; the performance of individual components ultimately translates into the usability of the entire system. Hesai’s existing hardware and manufacturing experience provided a foundation for this work.

Once entering the domain of training assets, feedback from the application side became equally important.

Hesai and Sharpa operate independently, but their co-founding team facilitates close communication between both ends. Robot teams can specify concrete needs: which viewing angles tend to lose targets, whether segmented objects can easily enter simulation, and how much manual processing is required for reconstructed assets. The acquisition and reconstruction teams adjust products accordingly and validate effectiveness through actual use.

This established specific metrics for discussing data quality. Whether a 3D scene looks realistic only answers part of the question; whether it adapts to robot sensors and allows easy task configuration determines when development teams can begin training.

Hesai stated that Kosmo has connected with platforms such as Isaac Sim, Isaac Lab, and Omniverse. Tool compatibility, ease of object editing, and asset processing convenience all impact final delivery efficiency. As customers expand to different robot bodies and training frameworks, these aspects will remain areas for continuous product refinement.

Some needs emerged outside the team’s initial expectations.

Li Yifan described the use cases in the film and television industry as an "unexpected bonus and surprise" during an interview. After receiving Kosmo, Dylan Sisson, Creative Lead at Pixar’s RenderMan team, scanned the plaza of the Pixar campus. He imported the data into Houdini via USD format and then handed it over to RenderMan for rendering. The real-world campus was subsequently processed into visuals with painterly brushstrokes, while elements within the scene remained adjustable.

He also scanned a handmade cardboard TV frame. The texture of the cardboard, folded edges, and glue marks were preserved, becoming details that could be leveraged in subsequent creative work. For creators, the value of a single scan is reflected not only in capturing reality but also in how much editing and expressive freedom remains after those details are integrated into professional software.

Amap’s application extends 3D assets into spatial browsing. Reportedly, Kosmo has partnered with Amap to launch the first batch of reconstructed venues—including the Red Art Museum and PIFC Gallery—on Amap’s "Street View" leaderboard under the "Flying Street View" feature. Users can observe the interiors of these locations before visiting them.

The expansion of use cases has also led to changes in delivery methods.

Some clients require hardware devices for continuous data collection; others simply want to complete a specific project and obtain the reconstructed models; still others prefer to directly utilize pre-made spatial assets.

Accordingly, Hesai proposed two business models: providing integrated software-hardware products for clients with ongoing collection needs, and offering scalable spatial data production services for world model and physical AI enterprises.

The former requires making it convenient for clients to perform collection and processing, while the latter involves scene organization, quality control, and continuous delivery. After device sales, whether the data can be smoothly used and how many times it can be repeatedly called upon will further influence clients’ assessment of the product’s value.

This remains an area that Kosmo is currently exploring. When asked about its largest customer segment, Li Yifan did not provide a definitive answer in the interview, noting that demands from different industries are gradually emerging through actual usage.

For Hesai, extending from sensors to data engines signifies a shift in the target audience for deliverables. The ranging capabilities, accuracy, and reliability of LiDAR can be evaluated through relatively clear metrics. However, once 3D assets enter training pipelines, their value becomes dependent on tasks, models, and simulation configurations.

What Kosmo attempts to change is precisely the traditional paradigm of "one-time capture, one-time modeling, one-time use." Once a real-world space is digitized, objects can continue to be edited, viewing conditions altered, and new training scenarios generated. A single capture may thus serve as the starting point for multiple rounds of subsequent training.

Conclusion: Providing AI with a "Base Map of the Real World"

In recent years, AI capabilities have grown rapidly under everyone’s watchful eyes: from playing chess, generating text and images, to now exploring real physical spaces.

Undoubtedly, world models and spatial intelligence are among the hottest directions today. The future applications they correspond to are equally captivating—from autonomous driving and embodied intelligence to digital twins... Many scenarios that previously existed only in science fiction are gradually gaining technical feasibility.

However, as hype rises, discussions about an "embodied intelligence bubble" have also emerged. The way to address such concerns may not be to cool down the industry, but rather to fill in the foundational gaps piece by piece.

One unavoidable issue is data. At the launch event, Xiang Shaoqing summarized the data problem in embodied intelligence as an "impossible triangle": teleoperation data from real robots is sufficiently realistic but costly, inefficient, and difficult to scale; simulation data is easy to expand but struggles to fully replicate the complex physical laws of the real world; and 2D images and videos lack depth, pose information, and true spatial relationships.

If a significant portion of past AI training data came from text, images, and videos on the internet, then for physical AI to grow further, it requires more data from the real world, along with infrastructure capable of collecting, processing, and feeding this data into training pipelines.

In the interview, Li Yifan positioned Hesai as a "builder of infrastructure for the physical AI industry," emphasizing that this would not be a single product line, but rather efforts centered around multiple foundational components.

What Kosmo is attempting is to transform the real world into a resource that AI can continuously learn from and leverage.

© THE END

Please contact this official account for authorization before reprinting.

Submissions or media inquiries: [email protected]