openpilot is an operating system for robotics. Currently, it upgrades the driver assistance system in 275+ supported cars.
Go to file
Kacper Rączy f058b5d64e
webrtcd: webrtc streaming server (audio/video/cereal) (#30186)
* WebRTCClient and WebRTCServer abstractions

* webrtc client implementation

* Interactive test scripts

* Send localDescriptions as offer/asnwer, as they are different

* Tracks need to be added after setting remote description for multi-cam streaming to work

* Remove WebRTCStreamingMetadata

* Wait for tracks

* Move stuff to separate files, rename some things

* Refactor everything, create WebRTCStreamBuilder for both offer and answers

* ta flight done time to grind

* wait for incoming tracks and channels

* Dummy track and frame reader track. Fix timing.

* dt based on camera type

* first trial of the new api

* Fix audio track

* methods for checking for incoming tracks

* Web migration part 2

* Fixes for stream api

* use rtc description for web.py

* experimental cereal proxy

* remove old code from bodyav

* fix is_started

* serialize session description

* fix audio

* messaging channel wrapper

* fix audiotrack

* h264 codec preference

* Add codec preference to tracks

* override sdp codecs

* add logging

* Move cli stuff to separate file

* slight cleanup

* Fix audio track

* create codec_mime inside force_codec function

* fix incoming media estimation

* move builders to __init__

* stream updates following builders

* Update example script

* web.py support for new builder

* web speaker fixes

* StreamingMediaInfo API

* Move things around

* should_add_data_channel rename

* is_connected_and_ready

* fix linter errors

* make cli executable

* remove dumb comments

* logging support

* fix parse_info_from_offer

* improve type annotations

* satisfy linters

* Support for waiting for disconnection

* Split device tracks into video/audio files. Move audio speaker to audio.py

* default dt for dummy video track

* Fix cli

* new speaker fixes

* Remove almost all functionality from web.py

* webrtcd

* continue refactoring web.py

* after handling joystick reset in controlsd with #30409, controls are not necessary anymore

* ping endpoint

* Update js files to at least support what worked previously

* Fixes after some tests on the body

* Streaming fixes

* Remove the use of WebRTCStreamBuilder. Subclass use is now required

* Add todo

* delete all streams on shutdown

* Replace lastPing with lastChannelMessageTime

* Update ping text only if rtc is still on

* That should affect the chart too

* Fix paths in web

* use protocol in SSLContext

* remove warnings since aiortc is not used directly anymore

* check if task is done in stop

* remove channel handler wrapper, since theres only one channel

* Move things around

* Moved webrtc abstractions to separate repository

* Moved webrtcd to tools/webrtc

* Update imports

* Add bodyrtc as dependency

* Add webrtcd to process_config

* Remove usage of DummyVideoStreamTrack

* Add main to webrtcd

* Move webrtcd to system

* Fix imports

* Move cereal proxy logic outside of runner

* Incoming proxy abstractions

* Add some tests

* Make it executable

* Fix process config

* Fix imports

* Additional tests. Add tests to pyproject.toml

* Update poetry lock

* New line

* Bump aiortc to 1.6.0

* Added teleoprtc_repo as submodule, and linked its source dir

* Add init file to webrtc module

* Handle aiortc warnings

* Ignore deprecation warnings

* Ignore resource warning too

* Ignore the warnings

* find free port for test_webrtcd

* Start process inside the test case

* random sleep test

* test 2

* Test endpoint function instead

* Update comment

* Add system/webrtc to release

* default arguments for body fields

* Add teleoprtc to release

* Bump teleoprtc

* Exclude teleoprtc from static analysis

* Use separate event loop for stream session tests
2023-12-01 21:13:37 -08:00
.devcontainer Devcontainer: initial windows support (#30168) 2023-10-04 14:58:21 -07:00
.github CI: speedup docs build (#30574) 2023-12-01 19:55:27 -08:00
body@cad7ef43ad Bump submodules (#30528) 2023-11-26 09:31:08 -08:00
cereal@2cb2bfb015 Bump submodules (#30528) 2023-11-26 09:31:08 -08:00
common Pytest: more post-test path cleanup (#30556) 2023-11-30 09:55:48 -08:00
docs camerad cleanup (#30573) 2023-12-01 20:10:07 -08:00
opendbc@2b96bcc456 Bump submodules (#30528) 2023-11-26 09:31:08 -08:00
openpilot remove xx dependencies (#29263) 2023-08-11 20:05:44 -07:00
panda@b50455cc76 bump panda (#30563) 2023-11-30 15:28:46 -08:00
rednose_repo@44e8a891a2 locationd: remove models unused in openpilot (#30481) 2023-11-22 13:25:29 -08:00
release webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
scripts adding --local-partial-types mypy opt (#30491) 2023-11-18 22:47:11 -08:00
selfdrive webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
site_scons/site_tools
system webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
teleoprtc_repo@8ec4778685 webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
third_party Standardize line endings in repo (#30112) 2023-09-30 17:05:43 -07:00
tinygrad_repo@ae5d1407ee Tinygrad: use compile2 (#30400) 2023-11-07 09:54:48 -08:00
tools webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
.clang-tidy
.dockerignore convert vidindex to python (#30176) 2023-10-05 18:40:24 -07:00
.editorconfig
.gitattributes Standardize line endings in repo (#30112) 2023-09-30 17:05:43 -07:00
.gitignore modeld: parsing and publishing in python (#30273) 2023-10-19 14:23:51 -07:00
.gitmodules webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
.lfsconfig
.pre-commit-config.yaml webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
.python-version agnos 8 (#28843) 2023-07-20 21:56:57 -07:00
Dockerfile.openpilot locationd: remove models unused in openpilot (#30481) 2023-11-22 13:25:29 -08:00
Dockerfile.openpilot_base Docker: add batman to sudo group (#30188) 2023-10-05 15:53:07 -07:00
Dockerfile.openpilot_base_cl
Jenkinsfile jenkins auto cpu count (#30544) 2023-11-28 14:34:00 -08:00
LICENSE
README.md delete clocksd (#30252) 2023-10-22 14:47:38 -07:00
RELEASES.md bump to 0.9.6 2023-11-17 18:44:05 -08:00
SConstruct locationd: remove models unused in openpilot (#30481) 2023-11-22 13:25:29 -08:00
SECURITY.md
codecov.yml Pytest: consistent codecov (#30408) 2023-11-08 13:43:32 -08:00
conftest.py Pytest: load by group by default (#30466) 2023-11-14 17:33:47 -08:00
launch_chffrplus.sh remove extra sleep on startup 2023-11-27 14:57:40 -08:00
launch_env.sh agnos 8 (#28843) 2023-07-20 21:56:57 -07:00
launch_openpilot.sh
poetry.lock webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
pyproject.toml webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
rednose
teleoprtc webrtcd: webrtc streaming server (audio/video/cereal) (#30186) 2023-12-01 21:13:37 -08:00
tinygrad

README.md

openpilot on the comma 3X

Table of Contents


What is openpilot?

openpilot is an open source driver assistance system. Currently, openpilot performs the functions of Adaptive Cruise Control (ACC), Automated Lane Centering (ALC), Forward Collision Warning (FCW), and Lane Departure Warning (LDW) for a growing variety of supported car makes, models, and model years. In addition, while openpilot is engaged, a camera-based Driver Monitoring (DM) feature alerts distracted and asleep drivers. See more about the vehicle integration and limitations.

Running on a dedicated device in a car

To use openpilot in a car, you need four things

  1. Supported Device: A comma 3/3X. You can purchase these devices from (https://comma.ai/shop/comma-3x)

  2. Software: The setup procedure for the comma 3/3X allows users to enter a URL for custom software. To install the release version of openpilot, use the URL openpilot.comma.ai. To install openpilot master (for more advanced users), use the URL installer.comma.ai/commaai/master. You can replace "commaai" with another GitHub username to install a fork.

  3. Supported Car: Ensure that you have one of the 250+ supported cars. openpilot supports a wide range of car makes including Honda, Toyota, Hyundai, Nissan, Kia, Chrysler, Lexus, Acura, Audi, VW, Ford, and many more. If your car is not officially listed as supported but has adaptive cruise control and lane-keeping assist, it's likely capable of running openpilot.

  4. Car Harness: You will also need a car harness to connect your comma 3/3X to your car. We have detailed instructions for how to install the harness and device in a car.

Running on PC

All openpilot services can run as usual on a PC without requiring special hardware or a car. You can also run openpilot on recorded or simulated data to develop or experiment with openpilot.

With openpilot's tools, you can plot logs, replay drives, and watch the full-res camera streams. See the tools README for more information.

You can also run openpilot in simulation with the CARLA simulator. This allows openpilot to drive around a virtual car on your Ubuntu machine. The whole setup should only take a few minutes but does require a decent GPU.

A PC running openpilot can also control your vehicle if it is connected to a webcam, a black panda, and a harness.

Community and Contributing

openpilot is developed by comma and by users like you. We welcome both pull requests and issues on GitHub. Bug fixes and new car ports are encouraged. Check out the contributing docs.

Documentation related to openpilot development can be found on docs.comma.ai. Information about running openpilot (e.g. FAQ, fingerprinting, troubleshooting, custom forks, community hardware) should go on the wiki.

You can add support for your car by following guides we have written for Brand and Model ports. Generally, a car with adaptive cruise control and lane keep assist is a good candidate. Join our Discord to discuss car ports: most car makes have a dedicated channel.

Want to get paid to work on openpilot? comma is hiring.

And follow us on Twitter.

User Data and comma Account

By default, openpilot uploads the driving data to our servers. You can also access your data through comma connect. We use your data to train better models and improve openpilot for everyone.

openpilot is open source software: the user is free to disable data collection if they wish to do so.

openpilot logs the road-facing cameras, CAN, GPS, IMU, magnetometer, thermal sensors, crashes, and operating system logs. The driver-facing camera is only logged if you explicitly opt-in in settings. The microphone is not recorded.

By using openpilot, you agree to our Privacy Policy. You understand that use of this software or its related services will generate certain types of user data, which may be logged and stored at the sole discretion of comma. By accepting this agreement, you grant an irrevocable, perpetual, worldwide right to comma for the use of this data.

Safety and Testing

  • openpilot observes ISO26262 guidelines, see SAFETY.md for more details.
  • openpilot has software-in-the-loop tests that run on every commit.
  • The code enforcing the safety model lives in panda and is written in C, see code rigor for more details.
  • panda has software-in-the-loop safety tests.
  • Internally, we have a hardware-in-the-loop Jenkins test suite that builds and unit tests the various processes.
  • panda has additional hardware-in-the-loop tests.
  • We run the latest openpilot in a testing closet containing 10 comma devices continuously replaying routes.

Directory Structure

.
├── cereal              # The messaging spec and libs used for all logs
├── common              # Library like functionality we've developed here
├── docs                # Documentation
├── opendbc             # Files showing how to interpret data from cars
├── panda               # Code used to communicate on CAN
├── third_party         # External libraries
└── system              # Generic services
    ├── camerad         # Driver to capture images from the camera sensors
    ├── hardware        # Hardware abstraction classes
    ├── logcatd         # systemd journal as a service
    ├── loggerd         # Logger and uploader of car data
    ├── proclogd        # Logs information from /proc
    ├── sensord         # IMU interface code
    └── ubloxd          # u-blox GNSS module interface code
└── selfdrive           # Code needed to drive the car
    ├── assets          # Fonts, images, and sounds for UI
    ├── athena          # Allows communication with the app
    ├── boardd          # Daemon to talk to the board
    ├── car             # Car specific code to read states and control actuators
    ├── controls        # Planning and controls
    ├── debug           # Tools to help you debug and do car ports
    ├── locationd       # Precise localization and vehicle parameter estimation
    ├── manager         # Daemon that starts/stops all other daemons as needed
    ├── modeld          # Driving and monitoring model runners
    ├── monitoring      # Daemon to determine driver attention
    ├── navd            # Turn-by-turn navigation
    ├── test            # Unit tests, system tests, and a car simulator
    └── ui              # The UI

Licensing

openpilot is released under the MIT license. Some parts of the software are released under other licenses as specified.

Any user of this software shall indemnify and hold harmless Comma.ai, Inc. and its directors, officers, employees, agents, stockholders, affiliates, subcontractors and customers from and against all allegations, claims, actions, suits, demands, damages, liabilities, obligations, losses, settlements, judgments, costs and expenses (including without limitation attorneys fees and costs) which arise out of, relate to or result from any use of this software by user.

THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT. YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS. NO WARRANTY EXPRESSED OR IMPLIED.


openpilot tests codecov