LeRobot 集成 L2D:全球最大开源自动驾驶数据集发布
Hugging Face LeRobot 框架正式接入 L2D 数据集,这是目前全球最大的开源自动驾驶多模态数据集。该数据集包含来自德国 30 个城市的 90+ TB 数据(超 5000 小时驾驶记录),涵盖高清摄像头、车辆状态及自然语言指令,专为训练端到端自动驾驶模型设计。
报道正文
本文目录16 个章节
LeRobot 进军驾校
发布于 2025 年 3 月 11 日 在 GitHub 上更新 点赞 109
Harsimrat Sandhawalia sandhawalia yaak-ai Remi Cadene cadene
L2D 数据集摘要——全球最大的自动驾驶数据集!
- 超过 90 TB 的多模态数据(5000+ 小时的驾驶记录),源自德国 30 个城市
- 6 个环绕式高清摄像头及完整的车辆状态信息:速度/航向/GPS/IMU
- 连续控制信号:油门/刹车/转向;离散动作:档位/转向灯
- 环境状态:车道数量、道路类型(高速公路|居民区)、路面材质(沥青、鹅卵石、石块路)、最高限速。
- 环境条件:降水量、天气状况(雪天、晴天、雨天)、光照条件(黎明、白天、黄昏)
- 专为训练端到端模型而设计,支持以自然语言指令或未来路径点作为条件输入
- 自然语言指令。例如 "当绿灯亮起时,驶过有轨电车轨道,然后穿过环岛",每个片段均配有此类指令
- 未来路径点 对齐至 OpenStreetMap 图结构,并额外提供鸟瞰视图渲染
- 专家策略(驾驶教练)与学员策略(学习驾驶员)
前沿的视觉语言模型和大语言模型基于源自互联网的开源图文语料库进行训练,这推动了近期开源 AI 的加速发展。尽管取得了这些突破,端到端 AI 在机器人和汽车领域的采用率仍然较低,主要原因是缺乏高质量、大规模的多模态数据集 如 OXE。为了释放机器人 AI 的潜力,Yaak 与 🤗 的 LeRobot 团队合作,并很高兴向机器人 AI 社区宣布推出 Learning to Drive (L2D)。L2D 是全球最大的多模态数据集,旨在为汽车领域构建开源的空间智能,并对 🤗 的 LeRobot 训练流水线和模型提供一流支持。借鉴源代码版本控制的最佳实践,Yaak 还邀请 AI 社区在我们的整个数据集(超过 1 PB)中搜索和发现新的片段(episodes),并将其排队等待审核,以便合并到未来的发布版本中 (R5+)。
*表 1:开源自动驾驶数据集(不包括激光雷达和雷达)。来源
L2D 数据是通过安装在德国 30 个城市中由驾校运营的 60 辆电动汽车上的相同传感器套件收集的,时间跨度为 3 年。L2D 中的策略分为两组——由驾驶教练执行的专家策略和由学员司机执行的学员策略。这两组策略都包含针对驾驶任务的自然语言指令。例如,“当你拥有路权时,从环岛第三个出口驶出,小心地通过人行横道”。
图 1:可视化展示:Nutron(为清晰起见,仅显示 6 个摄像头中的 3 个)指令:“当你拥有路权时,穿过环岛并从第三个出口驶出”。
专家策略没有任何驾驶失误,被视为最优策略;而学员策略则存在已知的次优性(见图 2)。
图 2:学员策略为避免驶入对向卡车车道而出现方向盘操作抖动
这两组策略涵盖了在欧盟(德语版本)获取驾照所必须完成的所有驾驶场景,例如超车、环岛和铁路道口。在本次发布版本中(见下文 R3+),对于次优的学员策略,将附带关于次优性的自然语言推理说明。例如“在对向来车附近错误/抖动地操控方向盘”(见图 2)。
L2D:学习驾驶
L2D (R2+) 旨在成为最大的开源自动驾驶数据集,通过提供独特且多样化的“episode”来赋能 AI 社区训练端到端的空间智能。通过纳入全谱系的驾驶策略(包括学员和专家),L2D 捕捉了安全驾驶车辆的复杂性。为了全面代表一个实际运行的自动驾驶车队,我们收录了包含多样化环境条件、传感器故障、施工区域以及交通信号灯失效的 episode。
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
Both the expert and student policy groups are captured with the identical sensor setup detailed in the table below. Six RGB cameras capture the vehicle’s context in 360o, and on-board GPS captures the vehicle location and heading. An IMU collects the vehicle dynamics, and we read speed, gas/brake pedal, steering angle, turn signal and gear from the vehicle’s CAN interface. We synchronized all modality types with the front left camera (observation.images.front_left) using their respective unix epoch timestamps. We also interpolated data points where feasible to enhance precision (See Table 2.) and finally reduced the sampling rate to 10 hz.
Fig 3: Multimodal data visualization with Visualization: Nutron (only 3 of 6 cameras shown for clarity)
Table 2: Modality types, LeRobot v3.0 key, shape and interpolation strategy.
L2D follows the official German driving task catalog (detailed version) definition of driving tasks, driving sub-tasks and task definition. We assign a unique Task ID and natural language instructions to all episodes. The LeRobot for all episodes is set to “Follow the waypoints while adhering to traffic rules and regulations”. The table below shows a few sample episodes, their natural language instruction, driving tasks and subtasks. Both expert and student policies have an identical Task ID for similar scenarios, whereas the instructions vary with the episode.
| Episode | Instructions | Driving task | Driving sub-task | Task Definition | Task ID |
|---|---|---|---|---|---|
| Visualization LeRobot Visualization Nutron | Drive straight through going around the parked delivery truck and yield to the incoming traffic | 3 Passing, overtaking | 3.1 Passing obstacles and narrow spots | This sub-task involves passing obstacles or navigating narrow roads while following priority rules. | 3.1.1.3a Priority regulation without traffic signs (standard) |
| Visualization LeRobot Visualization Nutron | Drive through the unprotected left turn yielding to through traffic | 4 Intersections, junctions, entering moving traffic | 4.1 Crossing intersections & junctions | This sub-task involves crossing intersections and junctions while following priority rules and observing other traffic. | 4.1.1.3a Right before left |
| Visualization LeRobot Visualization Nutron | Drive straight up to the yield sign and take first exit from the roundabout | 5 Roundabouts | 5.1 Roundabouts | This sub-task involves safely navigating roundabouts, understanding right-of-way rules, and positioning correctly. | 5.1.1.3a With one lane |
Table 3: Sample episodes in L2D, their instructions and Task ID derived from EU driving task catalog
We automate the construction of the instructions and waypoints using the vehicle position (GPS), Open-Source Routing Machine, OpenStreetMap and a Large Language Model (LLM) (See below). The natural language queries are constructed to closely follow the turn-by-turn navigation available in most GPS navigation devices. The waypoints (Fig 4) are computed by map-matching the raw GPS trace to the OSM graph and sampling 10 equidistant points (orange) spanning 100 meters from the vehicles current location (green), and serve as drive-by-waypoints.

Fig 4: L2D 6x RGB cameras, waypoints (orange) and vehicle location (green) Instructions: drive straight up to the stop stop sign and then when you have right of way, merge with the moving traffic from the left
Search & Curation
![]() | ![]() | |
|---|---|---|
| Expert policies | Student policies | |
| GPS traces from the expert policies collected from the driving school fleet. Click here to see the full extent of expert policies in L2D. | Student policies cover the same geographical locations as expert policies. Click here to see the full extent of student policies in L2D. |
We collected the expert and student policies with a fleet of 60 KIA E-niro driving school vehicles operating in 30 German cities, with an identical sensor suite. The multimodal logs collected with the fleet are unstructured and void of any task or instructions information. To search and curate for episodes we enrich the raw multimodal logs with information extracted through map matching the GPS traces with OSRM and assigning node and way tags from OSM (See next section). Coupled with a LLM, this enrichment step enables searching for episodes through the natural language description of the task.
OpenStreetMap
For efficiently searching relevant episodes, we enrich the GPS traces with turn information obtained by map-matching the traces using OSRM. We additionally use the map-matched route and assign route features, route restrictions and route maneuvers, collectively referred to as route tasks, to the trajectory using OSM (See sample Map). Appendix A1-A2 provides for more details on the route tasks we assign to GPS traces.
Fig 5: Driving tasks assigned to raw GPS trace (View map)
The route tasks which get assigned to the map-matched route, are assigned the beginning and end timestamps (unix epoch), which equates to the time when the vehicle enters and exits the geospatial linestring or point defined by the task (Fig 6).
Fig 6: Pink: GNSS trace, Blue: Matched route, tasks: Yield, Train crossing and Roundabout (View Map)
Multimodal search
We structured the natural language queries (instructions) to closely resemble turn-by-turn navigation available in GPS navigation devices. To translate instructions to route tasks, we prompt the LLM with the instructions and steer its output to a list of route features, route restrictions, route maneuvers and retrieve episodes assigned to these route tasks. We perform a strict validation of the output from the LLM with a pydantic model to minimize hallucinations. Specifically we use llama-3.3-70b and steer the output to the schema defined by the pydantic model. To further improve the quality of the structured output, we used approx 30 pairs of known natural language queries and route tasks for in-context learning. Appendix A. 2 provides details on the in-context learning pairs we used.

Instructions: Drive up to the roundabout and when you have the right of way turn right
LeRobot
L2D on 🤗 is converted to LeRobotDataset v2.1 and LeRobotDataset v3.0 format to fully leverage the current and future models supported within LeRobot. The AI community can now build end-to-end self-driving models leveraging the state-of-the-art imitation learning and reinforcement learning models for real world robotics like ACT, Diffusion Policy, and Pi0.
Existing self-driving datasets (table below) focus on intermediate perception and planning tasks like 2D/3D object detection, tracking, segmentation and motion planning, which require high quality annotations making them difficult to scale. Instead L2D is focused on the development of end-to-end learning which learns to predict actions (policy) directly from sensor input (Table 1.). These models leverage internet pre-trained VLM and VLAM.
Releases
Robotics AI models’ performances are bounded by the quality of the episodes within the training set. To ensure the highest quality episodes, we plan a phased release for L2D. With each new release we add additional information about the episodes. Each release R1+ is a superset of the previous releases to ensure clean episode history.
- instructions: Natural language instruction of the driving task 2. task_id: Mapping of episodes to EU mandated driving tasks Task ID 3. observation.state.route : Information about lane count, turn lanes from OSM 4. suboptimal: Natural language description for the cause of sub-optimal policies
Table 5: L2D release dates
The entire multimodal dataset collected by Yaak with the driving school fleet is 5x larger than the planned release. To further the growth of L2D beyond R4, we invite the AI community to search and uncover scenarios within our entire data collection and build a community powered open-source L2D. The AI community can now search for episodes through our natural language search and queue their collection for review by the community for merging them into the upcoming releases. With L2D, we hope to unlock an ImageNet moment for spatial intelligence.

Fig 7: Searching episodes by natural language instructions
Using L2D with HF/LeRobot
For R0, R1 we recommend using LeRobotDataset, with revision=[R0|R1], which can be used directly from the pypi release of LeRobot. For R2+, please follow installation outlined here or install from main as below, as we recommend using StreamingLeRobotDataset as R3 is is Dataset v3.0 format.
# uv for python deps
curl -LsSf https://astral.sh/uv/install.sh | sh
# install python version and pin it
uv init && uv python install 3.12.4 && uv python pin 3.12.4
# add lerobot to deps for R0, R1
uv add lerobot
# for R2+
GIT_LFS_SKIP_SMUDGE=1 uv add "git+https://github.com/huggingface/lerobot.git@main"
uv run python
>>> from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset
# This will load 3 episodes=[0, 9999, 99999], to load all the episodes please remove it
>>> dataset = StreamingLeRobotDataset("yaak-ai/L2D", episodes=[0, 9999, 99999], streaming=True, buffer_size=1000)
>>> dataset.meta
LeRobotDatasetMetadata({
Repository ID: 'yaak-ai/L2D',
Total episodes: '100000',
Total frames: '19042712',
Features: '['observation.state.vehicle', 'observation.state.lanes', 'observation.state.road', 'observation.state.surface', 'observation.state.max_speed', 'observation.state.precipitation', 'observation.state.conditions', 'observation.state.lighting', 'observation.state.waypoints', 'observation.state.timestamp', 'task.policy', 'task.instructions', 'action.continuous', 'action.discrete', 'timestamp', 'frame_index', 'episode_index', 'index', 'task_index', 'observation.images.left_forward', 'observation.images.front_left', 'observation.images.right_forward', 'observation.images.left_backward', 'observation.images.rear', 'observation.images.right_backward', 'observation.images.map']',
})',Closed Loop Testing
LeRobot driver
For real world testing of the AI models trained with L2D and LeRobot, we invite the AI community to submit models for closed loop testing with a safety driver, starting summer of 2025. The AI community will be able to queue their models for closed loop testing, on our fleet and choose the tasks they’d like the model to be evaluated on and, for example, navigating roundabouts or parking. The model would run in inference mode (Jetson AGX or similar) on-board the vehicle. The models will drive the vehicle with LeRobot driver in two modes
- drive-by-waypoints: “Follow the waypoints adhering to driving rules and regulations” given observation.state.vehicle.waypoints
- drive-by-language: “Drive straight and turn right at the pedestrian crossing”
Additional Resources
- Driving task catalog (Fahraufgabenkatalog)
- Official German practical driving exam
- Groq
References
@article{yaak2023novel,
author = {Yaak team},
title ={A novel test for autonomy},
journal = {https://www.yaak.ai/blog/a-novel-test-for-autonomy},
year = {2023},
}
@article{yaak2023actiongpt,
author = {Yaak team},
title ={Next action prediction with GPTs},
journal = {https://www.yaak.ai/blog/next-action-prediction-with-gpts},
year = {2023},
}
@article{yaak2024si-01,
author = {Yaak team},
title ={Building spatial intelligence part - 1},
journal = {https://www.yaak.ai/blog/buildling-spatial-intelligence-part1},
year = {2024},
}
@article{yaak2024si-01,
author = {Yaak team},
title ={Building spatial intelligence part - 2},
journal = {https://www.yaak.ai/blog/building-spatial-intelligence-part-2},
year = {2024},
}Appendix
A.1 Route tasks
List of route restrictions. We consider route tags from OSM a restriction if it imposes restrictions on the policy, for example speed limit, yield or construction. Route features are physical structures along the route, for example inclines, tunnels and pedestrian crossing. Route maneuvers are different scenarios which a driver encounters during a normal operation of the vehicle in an urban environment, for example, multilane left turns and roundabouts.
| Type | Name | Assignment | Task ID | Release |
|---|---|---|---|---|
| Route restriction | CONSTRUCTION | VLM | R1 | |
| Route restriction | CROSS_TRAFFIC | VLM | 4.3.1.3a, 4.3.1.3b, 4.3.1.3d, 4.2.1.3a, 4.2.1.3b, 4.2.1.3d | R2 |
| Route restriction | INCOMING_TRAFFIC | VLM | R2 | |
| Route restriction | LIMITED_ACCESS_WAY | OSM | R0 | |
| Route restriction | LIVING_STREET | OSM | R0 | |
| Route restriction | LOW_SPEED_REGION (5, 10, 20 kph) | OSM | R0 | |
| Route restriction | ONE_WAY | OSM | 3.2.1.3b | R0 |
| Route restriction | PEDESTRIANS | VLM | 7.2.1.3b | R1 |
| Route restriction | PRIORITY_FORWARD_BACKWARD | OSM | 3.1.1.3b | R0 |
| Route restriction | ROAD_NARROWS | OSM | R0 | |
| Route restriction | STOP | OSM | 4.1.1.3b, 4.2.1.3b, 4.3.1.3b | R0 |
| Route restriction | YIELD | OSM | 4.1.1.3b, 4.2.1.3b, 4.3.1.3b | R0 |
| Route feature | BRIDGE | OSM | R0 | |
| Route feature | CURVED_ROAD | OSM (derived) | 2.1.1.3a, 2.1.1.3b | R0 |
| Route feature | BUS_STOP | OSM | 7.1.1.3a | R0 |
| Route feature | HILL_DRIVE | OSM | R0 | |
| Route feature | LOWERED_KERB | OSM | R0 | |
| Route feature | NARROW_ROAD | VLM | ||
| Route feature | PARKING | OSM | R0 | |
| Route feature | PEDESTRIAN_CROSSING | OSM | 7.2.1.3b | R0 |
| Route feature | TRAFFIC_CALMER | OSM | R0 | |
| Route feature | TRAIN_CROSSING | OSM | 6.1.1.3a, 6.1.1.3b | R0 |
| Route feature | TRAM_TRACKS | OSM | 6.2.1.3a | R0 |
| Route feature | TUNNEL | OSM | R0 | |
| Route feature | UNCONTROLLED_PEDESTRIAN_CROSSING | OSM | 7.2.1.3b | R0 |
| Route maneuver | ENTERING_MOVING_TRAFFIC | OSM (derived) | 4.4.1.3a | R0 |
| Route maneuver | CUTIN | VLM | R3 | |
| Route maneuver | LANE_CHANGE | VLM | 1.3.1.3a, 1.3.1.3b | R3 |
| Route maneuver | MERGE_IN_OUT_ON_HIGHWAY | OSM | 1.1.1.3a, 1.1.1.3b, 1.1.1.3c, 1.2.1.3a, 1.2.1.3b, 1.2.1.3c | R0 |
| Route maneuver | MULTILANE_LEFT | OSM (derived) | 4.3.1.3b, 4.3.1.3c, 4.3.1.3d | R0 |
| Route maneuver | MULTILANE_RIGHT | OSM (derived) | 4.2.1.3b, 4.2.1.3c, 4.2.1.3d | R0 |
| Route maneuver | PROTECTED_LEFT | OSM (derived) | 4.3.1.3c, 4.3.1.3d | R0 |
| Route maneuver | PROTECTED_RIGHT_WITH_BIKE | OSM (derived) | 4.2.1.3c, 4.2.1.3d | R0 |
| Route maneuver | RIGHT_BEFORE_LEFT | OSM (derived) | 4.1.1.3a, 4.2.1.3a, 4.3.1.3a | R0 |
| Route maneuver | RIGHT_TURN_ON_RED | OSM | 4.2.1.3c | R0 |
| Route maneuver | ROUNDABOUT | OSM | 5.1.1.3a, 5.1.1.3b | R0 |
| Route maneuver | STRAIGHT | OSM (derived) | 8.1.1.3a | R0 |
| Route maneuver | OVER_TAKE | VLM | 3.2.1.3a, 3.2.1.3b | R4 |
| Route maneuver | UNPROTECTED_LEFT | OSM (derived) | 4.3.1.3a, 4.3.1.3b | R0 |
| Route maneuver | UNPROTECTED_RIGHT_WITH_BIKE | OSM | 4.2.1.3a, 4.2.1.3b | R0 |
OSM = Openstreetmap, VLM= Vision Language Model, derived: Hand crafted rules with OSM data
A.2 LLM prompts
Prompt template and pseudo code for configuring the LLM using groq to parse natural language queries into structured prediction for route features, restrictions and maneuvers with a pydantic model. The natural language queries are constructed to closely follow the turn-by-turn navigation available in most GPS navigation devices.
prompt_template: "You are parsing natural language driving instructions into PyDantic Model's output=model_dump_json(exclude_none=True) as JSON. Here are a few example pairs of instructions and structured output: {examples}. Based on these examples parse the instructions. The JSON must use the schema: {schema}"
groq:
model: llama-3.3-70b-versatile
temperature: 0.0
seed: 1334
response_format: json_object
max_sequence_len: 60000Example pairs (showing 3 / 30) for in-context learning to steer the structured prediction of LLM, where ParsedInstructionModel is a pydantic model.
PROMPT_PAIRS = [
(
"Its snowing. Go straight through the intersection, following the right before left rule at unmarked intersection",
ParsedInstructionModel(
eventSequence=[
EventType(speed=FloatValue(value=10.0, operator="LT", unit="kph")),
EventType(osmRouteManeuver="RIGHT_BEFORE_LEFT"),
EventType(speed=FloatValue(value=25.0, operator="LT", unit="kph")),
],
turnSignal="OFF",
weatherCondition="Snow",
),
),
(
"stop at the stop sign, give way to the traffic and then turn right",
ParsedInstructionModel(
eventSequence=[
EventType(osmRouteRestriction="STOP"),
EventType(turnSignal="RIGHT"),
EventType(speed=FloatValue(value=5.0, operator="LT", unit="kph")),
EventType(osmRouteManeuver="RIGHT"),
],
),
),
(
"parking on a hill in the rain on a two lane road",
ParsedInstructionModel(
osmLaneCount=[IntValue(value=2, operator="EQ")],
osmRouteFeature=["PARKING", "HILL_DRIVE"],
weatherCondition="Rain",
),
),
]
EXAMPLES = ""
for idx, (instructions, parsed) in enumerate(PROMPT_PAIRS):
parsed_json = parsed.model_dump_json(exclude_none=True)
update = f"instructions: {instructions.lower()} output: {parsed_json}"
EXAMPLES += update
from groq import Groq
client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
chat_completion = client.chat.completions.create(
messages=[
{
"role": "system",
"content": prompt_template.format(examples=EXAMPLES, schema=json.dumps(ParsedInstructionModel.model_json_schema(), indent=2))
},
{
"role": "user",
"content": f"instructions : its daytime. drive to the traffic lights and when it turns green make a left turn",
},
],
model=config["groq"]["model"],
temperature=config["groq"]['temperature'],
stream=False,
seed=config["groq"]['seed'],
response_format={"type": config['groq']['response_format']},
)
parsed_obj = ParsedInstructionModel.model_validate_json(chat_completion.choices[0].message.content)
parsed_obj = parsed_obj.model_dump(exclude_none=True)A.2 Data collection hardware
Onboard compute: NVIDIA Jetson AGX Xavier
-
8 cores @ 2/2.2 GHz, 16/64 GB DDR5
-
100 TOPS , 8 lanes MIPI CSI-2 D-PHY 2.1 (up to 20Gbps)
-
8x 1080p30 video encoder (H.265)
-
Power: 10-15V DC input, ~90W power consumption
-
Storage: SSD M.2 (4gen PCIe 1x4)
-
Video input 8 cameras:
-
2x Fakra MATE-AX with 4x GMSL2 with Power-over-Coax support
Onboard compute: Connectivity
- Multi-band, Centimeter-level accuracy RTK module
- 5G connectivity: M.2 USB3 module with maximum downlink rates of 3.5Gbps and uplink rates of 900Mbps, dual SIM
| Component | # | Vendor | Specs |
|---|---|---|---|
| RGB: Camera | 1 | connect-tech | Techspecs |
| RGB: Rugged Camera | 5 | connect-tech | Techspecs |
| GNSS | 1 | Taoglas | Techspecs |
| 5G antenna | 2 | 2J Antenna | Datasheet |
| NVIDIA Jetson Orin NX - 64 GB | 1 | Nvidia | Techspecs |
Table 6: Information on hardware kit used for data collection











