Commit Graph

75 Commits

Author SHA1 Message Date
Jason Wen
03c8494dbc Merge branch 'upstream/openpilot/master' into sync-20251213
# Conflicts:
#	README.md
#	common/api.py
#	docs/CARS.md
#	opendbc_repo
#	panda
#	selfdrive/ui/mici/layouts/offroad_alerts.py
#	system/ui/README.md
#	system/version.py
2025-12-13 01:50:42 -05:00
Maxime Desroches
5fd0906164 allow restarting processes after crash (#36755)
more
2025-12-02 17:10:24 -08:00
discountchubbs
ceed203de7 Merge remote-tracking branch 'openpilot/master' into nov-19-sync 2025-11-21 09:42:54 -08:00
Adeeb Shihadeh
4ef0d3ee99 setup sound for DM test mode 2025-11-18 10:11:06 -08:00
Jason Wen
08e85808c5 Merge branch 'upstream/openpilot/master' into sync-20251114
# Conflicts:
#	.github/workflows/ci_weekly_run.yaml
#	.github/workflows/raylib_ui_preview.yaml
#	.github/workflows/tests.yaml
#	.gitmodules
#	README.md
#	SConstruct
#	common/api.py
#	common/params_keys.h
#	docs/CARS.md
#	msgq_repo
#	opendbc_repo
#	panda
#	selfdrive/car/tests/test_car_interfaces.py
#	selfdrive/controls/controlsd.py
#	selfdrive/controls/lib/latcontrol.py
#	selfdrive/controls/lib/latcontrol_angle.py
#	selfdrive/controls/lib/latcontrol_pid.py
#	selfdrive/controls/lib/latcontrol_torque.py
#	selfdrive/controls/tests/test_latcontrol.py
#	selfdrive/monitoring/helpers.py
#	selfdrive/ui/SConscript
#	selfdrive/ui/main.cc
#	selfdrive/ui/qt/body.h
#	selfdrive/ui/qt/home.cc
#	selfdrive/ui/qt/home.h
#	selfdrive/ui/qt/network/networking.cc
#	selfdrive/ui/qt/network/networking.h
#	selfdrive/ui/qt/network/wifi_manager.cc
#	selfdrive/ui/qt/offroad/developer_panel.cc
#	selfdrive/ui/qt/offroad/developer_panel.h
#	selfdrive/ui/qt/offroad/experimental_mode.cc
#	selfdrive/ui/qt/offroad/firehose.cc
#	selfdrive/ui/qt/offroad/firehose.h
#	selfdrive/ui/qt/offroad/onboarding.cc
#	selfdrive/ui/qt/offroad/onboarding.h
#	selfdrive/ui/qt/offroad/settings.cc
#	selfdrive/ui/qt/offroad/settings.h
#	selfdrive/ui/qt/offroad/software_settings.cc
#	selfdrive/ui/qt/onroad/alerts.cc
#	selfdrive/ui/qt/onroad/annotated_camera.h
#	selfdrive/ui/qt/onroad/buttons.cc
#	selfdrive/ui/qt/onroad/buttons.h
#	selfdrive/ui/qt/onroad/driver_monitoring.cc
#	selfdrive/ui/qt/onroad/hud.cc
#	selfdrive/ui/qt/onroad/hud.h
#	selfdrive/ui/qt/onroad/model.cc
#	selfdrive/ui/qt/onroad/model.h
#	selfdrive/ui/qt/onroad/onroad_home.cc
#	selfdrive/ui/qt/onroad/onroad_home.h
#	selfdrive/ui/qt/request_repeater.h
#	selfdrive/ui/qt/sidebar.cc
#	selfdrive/ui/qt/sidebar.h
#	selfdrive/ui/qt/util.cc
#	selfdrive/ui/qt/widgets/cameraview.h
#	selfdrive/ui/qt/widgets/controls.cc
#	selfdrive/ui/qt/widgets/controls.h
#	selfdrive/ui/qt/widgets/input.cc
#	selfdrive/ui/qt/widgets/input.h
#	selfdrive/ui/qt/widgets/prime.cc
#	selfdrive/ui/qt/widgets/prime.h
#	selfdrive/ui/qt/widgets/ssh_keys.h
#	selfdrive/ui/qt/widgets/toggle.h
#	selfdrive/ui/qt/widgets/wifi.cc
#	selfdrive/ui/qt/widgets/wifi.h
#	selfdrive/ui/qt/window.cc
#	selfdrive/ui/qt/window.h
#	selfdrive/ui/tests/cycle_offroad_alerts.py
#	selfdrive/ui/tests/test_ui/run.py
#	selfdrive/ui/translations/main_ar.ts
#	selfdrive/ui/translations/main_de.ts
#	selfdrive/ui/translations/main_es.ts
#	selfdrive/ui/translations/main_fr.ts
#	selfdrive/ui/translations/main_ja.ts
#	selfdrive/ui/translations/main_ko.ts
#	selfdrive/ui/translations/main_nl.ts
#	selfdrive/ui/translations/main_pl.ts
#	selfdrive/ui/translations/main_pt-BR.ts
#	selfdrive/ui/translations/main_th.ts
#	selfdrive/ui/translations/main_tr.ts
#	selfdrive/ui/translations/main_zh-CHS.ts
#	selfdrive/ui/translations/main_zh-CHT.ts
#	selfdrive/ui/ui.cc
#	selfdrive/ui/ui.h
#	system/manager/build.py
#	system/version.py
2025-11-16 02:50:28 -05:00
DevTekVE
4b5de0eddb stats: sunnylink integration (#1454)
* sunnylink: add statsd process and related telemetry logging infrastructure

- Introduced `statsd_sp` process for handling Sunnylink-specific stats.
- Enhanced metrics logging with improved directory structure and data handling.

* sunnylink: re-enable and refine stat_handler for telemetry processing

- Reactivated `stat_handler` thread with improved path handling.
- Made `stat_handler` more flexible by allowing directory injection.

* statsd: fix formatting issue in telemetry string generation

- Corrected missing comma between `sunnylink_dongle_id` and `comma_dongle_id`.

* update statsd_sp process configuration for enhanced readiness logic

- Modified `statsd_sp` initialization to include `always_run` alongside `sunnylink_ready_shim`.
- Ensures robust process activation conditions.

* refactor(statsd): enhance and unify StatLogSP implementation

- Replaced custom `StatLogSP` in sunnylink with centralized implementation from `system.statsd`.
- Ensures consistent logic for StatLogSP handling across modules.

* fix

* refactor(statsd): add intercept parameter to StatLogSP for configurable logging

- Introduced optional `intercept` parameter to `StatLogSP` to manage `comma_statlog` initialization.
- Updated usage in `sunnylink` to disable interception where unnecessary.

* Dont complain

* feat(statsd): add raw metric type and SunnyPilot-specific stats collection

- Introduced `METRIC_TYPE.RAW` for base64-encoded raw data metrics.
- Added `sp_stats` thread to export SunnyPilot params as raw metrics.
- Enhanced telemetry handling with decoding and serialization updates.

* refactor(statsd): improve `sp_stats` error handling and param processing

- Enhanced exception handling for `params.get` to prevent crashes.
- Added support for nested dict values to be included in stats.

* refactor(statsd): adjust imports and minor code formatting updates

- Updated `Ratekeeper` import path for consistency with the `openpilot` module structure.
- Fixed minor formatting for improved readability.

* refactor(statsd): update typings and remove unused NoReturn annotation

- Removed unnecessary `NoReturn` typing for `stats_main` to simplify function definition.
- Adjusted `get_influxdb_line_raw` to refine typing for `value` parameter.

* cleanup

* init

* init

* slightly more

* staticmethod

* handle them all

* get them models

* log with route

* more

* car

* Revert "car"

This reverts commit fe1c90cf4d.

* handle capnp

* Revert "handle capnp"

This reverts commit c5aea68803.

* 1 more time

* Revert "1 more time"

This reverts commit a364474fa5.

* Cleaning to expose wider

* feat(interfaces, statsd): log car params to stats system

- Added `STATSLOGSP` import and logging to capture `carFingerprint` in metrics.
- Improved error handling in `get_influxdb_line_raw` for robust metric generation.

* refactor(interfaces): streamline car params logging to stats

- Simplified logging by directly converting `CP` to a dictionary.
- Removed legacy stats aggregation for clarity.

* feat(sunnylink): enable compression for stats in SunnyLink

- Added optional compression for stats payload to support large data.
- Updated `stat_handler` to handle compression and base64 encoding.

* fix(statsd): filter complex types in `get_influxdb_line_raw`

- Skips unsupported types (dict, list, bytes) to prevent formatting errors.
- Simplifies type annotation for `value` parameter.

* fix(statsd): use `json.dumps` for string conversion in `get_influxdb_line_raw`

- Ensures proper handling of special characters in values.
- Prevents potential formatting issues with raw `str()` conversion.

* refactor(interfaces, statsd): update parameter keys for stats logging

- Renamed logged keys for better clarity (`sunnypilot_params` → `sunnypilot.car_params`, `device_params`).
- Ensures consistency across data logs.

* bet

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2025-11-04 16:53:31 -05:00
Shane Smiskol
1e73025f86 Remove Qt (#36427)
* rm qt from ui scons

* rm qt translation litter

* rm ccs

* more

* fix cabana

* more

* more

* more
2025-10-22 22:18:07 -07:00
Adeeb Shihadeh
5af1099fbf rm watchdog (#36425) 2025-10-22 15:36:09 -07:00
Maxime Desroches
92cd656c68 ui: remove watchdog (#36388)
out
2025-10-17 00:26:29 -07:00
Shane Smiskol
b3eba70b7a raylib: flip! (#36319)
* flip!

* add ui

* ?

* qt is extra

* low node

* add uiDebug

* fix

* fix dat

* bump double increase for tol

* it's ~11ms but double for tol

* fix report

* Update selfdrive/test/test_onroad.py
2025-10-11 07:41:29 -07:00
Jason Wen
a85f3ce11c Device: Onroad Uploads button (#1332) 2025-10-08 13:03:15 -04:00
Shane Smiskol
586e49cab3 Revert "Switch to raylib for UI (#36238)"
This reverts commit c88ab5cd12.
2025-10-04 01:04:20 -07:00
Shane Smiskol
c88ab5cd12 Switch to raylib for UI (#36238)
* flip

* change this
2025-10-03 23:38:10 -07:00
Shane Smiskol
ec7e3192bb revert that 2025-10-02 04:00:09 -07:00
Shane Smiskol
3fd352a7ef raylib: updater UI (#36235)
* auto attempt

* gpt5

* Revert "gpt5"

This reverts commit 556d6d9ee4d53aca0f4612023db6cfb2bed7ce29.

* clean up

* fixes

* use raylib

* fixes

* debug

* test update

* more

* rm

* add value to button like qt

* bump

* bump

* fixes

* bump

* fix

* bump

* clean up

* time ago like qt

rm

* bump

* clean up

* updated can fail to respond on boot leading to stuck state

* fix color

fix

* bump

* bump

* add back

* test update

* no unknown just ''

* ffix
2025-10-02 03:57:10 -07:00
Jason Wen
ecee67dd64 Locationd: reimplement liveLocationKalman and use position_geodetic in liveMapDataSP (#1275)
* init

* desc

* llk welcome back

* more

* new param to write

* update mapd

* no migration

* no refactor for now

* exec

* rename

* bearing

* fix test

* lint
2025-09-22 23:39:55 -04:00
Jason Wen
ec8f036850 Merge branch 'upstream/openpilot/master' into sync-20250908
# Conflicts:
#	.github/workflows/selfdrive_tests.yaml
#	README.md
#	docs/CARS.md
#	opendbc_repo
#	panda
#	selfdrive/car/tests/test_car_interfaces.py
#	selfdrive/modeld/modeld.py
#	selfdrive/selfdrived/selfdrived.py
#	selfdrive/ui/translations/main_ar.ts
#	selfdrive/ui/translations/main_de.ts
#	selfdrive/ui/translations/main_es.ts
#	selfdrive/ui/translations/main_fr.ts
#	selfdrive/ui/translations/main_ja.ts
#	selfdrive/ui/translations/main_ko.ts
#	selfdrive/ui/translations/main_pt-BR.ts
#	selfdrive/ui/translations/main_th.ts
#	selfdrive/ui/translations/main_tr.ts
#	selfdrive/ui/translations/main_zh-CHS.ts
#	selfdrive/ui/translations/main_zh-CHT.ts
#	system/hardware/hardwared.py
#	system/updated/updated.py
#	tinygrad_repo
#	uv.lock
2025-09-13 15:23:36 -04:00
Adeeb Shihadeh
8dca43881a Rewrite ubloxd in Python (#36112)
* Rewrite ubloxd in Python

* lil more

* rm from third_party/

* cleanup

* ubx replay

* try this

* back to kaitai

* Revert "ubx replay"

This reverts commit 570bd3d25fbabc590379ce0a9f98d30de55cf1b3.
2025-09-07 15:40:48 -07:00
Adeeb Shihadeh
608c16007e Rewrite logcatd in Python (#36111)
* Add Python logcatd implementation

* lil more
2025-09-07 11:32:44 -07:00
Adeeb Shihadeh
275abc1eb5 Rewrite proclogd in Python (#36110)
* Rewrite proclogd in Python

* lil more

* lil more

* Update system/proclogd.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update system/proclogd.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update system/proclogd.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-09-07 11:13:39 -07:00
DevTekVE
3e9545670b feature: Adding support for copyparty (#1116)
* feat: add support for copyparty-sfx

* feat: add toggle for Copyparty service in developer panel

* feat: enhance Copyparty configuration with additional volume mounts and options

* Update system/manager/process_config.py

* remove f string

* lint
2025-08-31 08:29:45 +02:00
James Vecellio-Grant
34f4aadca5 ci: enforce runner cutoff above 9.0V threshold (#1156)
* ci: github runner auto off when voltage is above 9.0v .

This ensures that a runner in vehicle doesn't accidentally break everything lol.

* suggestion for clarity.

* refactor: rename and update handling of `GithubRunnerVoltage` parameter

- Improve clarity by renaming to `GithubRunnerSufficientVoltage`.
- Changed attribute to `CLEAR_ON_MANAGER_START` for improved runtime state management.
- No need for this value to be backed up!

* refactor: streamline voltage check for GithubRunnerSufficientVoltage

---------

Co-authored-by: DevTekVE <devtekve@gmail.com>
2025-08-15 07:42:56 -07:00
DevTekVE
8910668e4e sunnylink: enable uploader option for admins (#1152)
* feat: add sunnylink uploader option for admins in sunnylink panel

* feat: enhance uploader to support zstd compression and improve route handling

* feat: update sunnylink uploader description and enablement criteria for admin tiers

* ui cleanup

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2025-08-13 12:12:01 -04:00
Jason Wen
97c2d7e655 Merge branch 'upstream/openpilot/master' into sync-20250809
# Conflicts:
#	common/params_keys.h
#	opendbc_repo
#	panda
#	selfdrive/car/cruise.py
#	selfdrive/controls/controlsd.py
#	selfdrive/selfdrived/selfdrived.py
#	selfdrive/test/process_replay/process_replay.py
#	selfdrive/ui/qt/setup/setup.cc
#	tinygrad_repo
2025-08-10 02:05:05 -04:00
Shane Smiskol
5117a8c3a6 ui: test raylib ui (#35949)
* add raylib ui

* test

* this is better for now

* rm

rm

* finalize it

* need this?

* ?

* shite

shite

* try

* ?

* huh

* simp

* ?

* wtf is going on

* ???????????????

* lock

* stash

* no 2 packages

* Revert "stash"

This reverts commit 9efb0d9bda6a6309e7a567634d1921bf1cd0fb59.

* debug

* noo

* debug

* ?

* and

* yeah yeah

* init one

* 2

* i wonder

* oooh

* make sure

* fix dat

* try this

* see if wifiman

* forgot

* ?

* ???

* fuck this we can rewrite it later
2025-08-08 00:34:53 -07:00
Jimmy
d7b0a5fa7e Record feedback with LKAS button (#35888)
* record feedback with LKAS button

* fix alert test

* slightly simplify feedbackd

* "Audio Feedback Saved" upon time expiration or early stop

* earlySend --> earlyStop

* userFlag --> userBookmark

* RecordAudioFeedback param/toggle

* add audioFeedback test

* simplify feedbackd

* send bookmark regardless of toggle, show feedback event with higher priority

* add userBookmark to selfdrived sm

* fix mispelled param name

* default off and move to main

* segmentNum --> blockNum, earlyStop --> lastBlock

* preserve audioFeedback

* get rid of lastBlock and just send bookmark saved at the end

* update raylib side

* update toggle description and add raylib toggle

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2025-08-05 13:41:41 -07:00
royjr
c5eb41fe37 mapd: suppress output
Update process_config.py
2025-08-01 09:45:41 -04:00
royjr
0a2e8dd81b mapd: bump version and fix missing folder (#994)
* fix mapd folder

* not needed

* unnecessary

* Remove unused import from mapd_installer.py

Deleted the unused import of Paths from openpilot.system.hardware.hw to clean up the code.

* Undo changes

* update: adjust `mapd` process to conditionally start based on readiness

- Added `mapd_ready` function to check `mapd` readiness.
- Modified `mapd` process to use this readiness condition.

* update: bump mapd version to v1.10.0 and add readiness utility

- Updated `VERSION` to v1.10.0 for `mapd_installer.py`.

* update: integrate `MapdInstallManager` versioning into main thread

- Added a call to `MapdInstallManager.update_installed_version` with `VERSION`.
- Ensures version management for `mapd` is synchronized during runtime.

* refactor: centralize `MAPD_PATH` and `MAPD_BIN_DIR` definitions in module init

- Moved constants to `sunnypilot/mapd/__init__.py` for cleaner imports and reuse.
- Updated references in `mapd_manager.py` and `mapd_installer.py`.

* revert back the file permission

* do this

* no longer static

---------

Co-authored-by: DevTekVE <devtekve@gmail.com>
Co-authored-by: rav4kumar <meetkumardesai@gmail.com>
Co-authored-by: James Vecellio-Grant <159560811+Discountchubbs@users.noreply.github.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2025-07-25 11:25:31 -04:00
Jason Wen
ab6d192714 mapd (#989)
* init

* some fixes

* move

* more

* old navd helpers

* bring back cereal

* fix linting

* more

* add to cereal first

* sp events

* lint

* implement in long plan

* fixme-sp

* refactor state machine

* wrong state

* start refactor controller

* some type hints

* init these

* enable debug print

* ui? ui!

* print them out

* fix spinner import

* fix path

* let's use gps chips directly for now

* service missing

* publish events

* no nav for now

* need to sub

* no car state speed yet

* missed event

* Car: `CarStateSP`

* fix tests

* bring back car state speed limit

* fix

* use old controller for now

* fix

* fix source

* type hints

* none for now

* formatting

* more

* create directory if does not exist

* mypy my bt

* policy param catch exceptions

* handle all params with exceptions

* more

* single method

* define types in init

* rename

* simpler op enabled check

* more mypy stuff

* rename

* no need for brake pressed

* don't reset if gas pressed

* type hint all

* type hint all

* back to upstream

* in another pr

* no longer need data type

* qlog

* slc in another pr

* use horizontal accuracy

* set core affinity for all realtime processes

* unused

* sort

* unused

* type hint and slight cleanup

* from old implementation

* use directly

* combine pm

* slight more cleanup

* type hints

* even more type hint

* lint

* more cleanup

* even less

* license
2025-06-07 03:47:09 -04:00
Jason Wen
4d12f880bd Merge branch 'upstream/openpilot/master' into sync-20250605
# Conflicts:
#	opendbc_repo
#	panda
2025-06-05 22:27:18 -04:00
Andrei Radulescu
3ad74b440c macos: no logcatd and proclogd (#35457)
* don't build proclogd on mac

* don't run logcatd and proclogd on mac
2025-06-05 10:23:27 -07:00
DevTekVE
82487ba136 Merge remote-tracking branch 'comma/master' into sync-20250531
# Conflicts:
#	.github/workflows/selfdrive_tests.yaml
#	common/params_keys.h
#	msgq_repo
#	opendbc_repo
#	panda
#	selfdrive/car/card.py
#	selfdrive/car/tests/test_car_interfaces.py
#	selfdrive/car/tests/test_models.py
#	selfdrive/test/process_replay/process_replay.py
#	selfdrive/ui/qt/offroad/settings.cc
#	selfdrive/ui/translations/main_ar.ts
#	selfdrive/ui/translations/main_es.ts
#	selfdrive/ui/translations/main_fr.ts
#	selfdrive/ui/translations/main_ja.ts
#	selfdrive/ui/translations/main_ko.ts
#	selfdrive/ui/translations/main_pt-BR.ts
#	selfdrive/ui/translations/main_th.ts
#	selfdrive/ui/translations/main_tr.ts
#	selfdrive/ui/translations/main_zh-CHS.ts
#	selfdrive/ui/translations/main_zh-CHT.ts
2025-05-31 14:12:30 -04:00
Adeeb Shihadeh
2f80854644 sensord: rewrite in Python (#35353)
* py sensord

* fix up mmc

* temp

* port over accel

* lil more

* kinda works

* rm that

* gpiochip

* mostly there

* lil more

* lil more

* irq timestamps

* fix ts

* fix double deg2rad

* test passes

* fix up mypy

* rm one more

* exception

* lint:

* read in all events

* bump that

* get under budget:

* accel self test

* gyro self-test

* keep these readable

* give it more cores

* debug

* valid

* rewrite that

---------

Co-authored-by: Comma Device <device@comma.ai>
2025-05-30 13:31:07 -07:00
Jason Wen
b9385a2368 Merge branch 'upstream/openpilot/master' into sync-20250422
# Conflicts:
#	.github/workflows/selfdrive_tests.yaml
#	.github/workflows/ui_preview.yaml
#	common/params_keys.h
#	opendbc_repo
#	panda
#	selfdrive/car/card.py
#	selfdrive/car/tests/test_car_interfaces.py
#	selfdrive/car/tests/test_models.py
#	selfdrive/controls/lib/longitudinal_planner.py
#	selfdrive/modeld/fill_model_msg.py
#	selfdrive/test/process_replay/process_replay.py
#	selfdrive/ui/qt/offroad/developer_panel.cc
#	selfdrive/ui/qt/offroad/settings.cc
#	selfdrive/ui/translations/main_de.ts
Sync: `commaai/opendbc:master` into `sunnypilot/opendbc:master-new`

Sync: `commaai/panda:master` into `sunnypilot/panda:master-new`
2025-04-23 00:44:53 -04:00
Kacper Rączy
f237649a7a Reapply "Online lateral lag learning" (#34975)
* Online lateral lag learning (#34974)

This reverts commit b4cc9e68d1.

* pad to the best size for fft

* Fix static analysis

* Add typing

* Fix typing

* MAX_LAG

* Calculate cross correlation regardless if the points are valid

* Back to lagd

* Add lagd to process_config

* Lagd in test onroad

* Move lag estimator for lagd

* Remove duplicate entry from test_onroad

* Update process replay

* pre-fill the data

* Update cpu usage

* 25sec window

* Change the meaning of lateralDelayEstimate

* No newline

* Fix typing

* Prefill

* Update ref commit

* Add a unit test

* Fix static issues

* Time limit

* Or timeout

* Use mocker

* Update estimate every time

* empty test

* DT const

* enable RIVIAN again

* Update ref commit

* Update that again

* Improve the tests

* Fix static

* Add masking test

* Increase timeout

* Add liveDelay to selfdrived

* Add liveDelay to selfdrived in process_replay

* Fix block_avg restore after num_blocks

* regen most

* Update bolt

* Update ref commit

* Change the key name

* Add assert

* True weighted average
2025-04-09 22:31:07 +00:00
DevTekVE
43e43849ad sunnylink: Settings backup & restore (#681)
* Add AES encryption and utility methods

This commit introduces a new AESCipher class that can be used for AES encryption with support for both AES-128 and AES-256. It also adds a set of utility functions, including methods for RSA to AES key derivation, file decryption and compression, and encryption and decompression. These changes provide fundamental cryptographic functionalities for data security within the system.

* Revised backup management system with new structures

This update introduces important revisions to the backup management system. A new struct named 'BackupManagerSP' has been integrated into the 'custom.capnp' file, replacing 'CustomReserved6'. This new struct includes several variables that help keep track of backup and restore status, progress and related messages.

Additional code modifications were made in 'base.py' and 'api.py' to include a 'json' parameter in the 'api_get' method. Lastly, 'manager.py' has been updated with functions to manage device configuration backups to and from sunnylink.

The changes promote better handling and management of data backup and restoration processes. The update is specifically designed to ensure that the backup information is well integrated into the system, with properly tracked status and progress.

* Refactor `allKeys` to filter by parameter key type

Updated the `allKeys` method to support filtering by `ParamKeyType`, allowing more specific key retrieval. Added a default value for backward compatibility and updated related bindings and keys to reflect this change.

* Improve Backup and Restore mechanisms

The commit refactors and improves several aspects of the backup and restore mechanisms in the `BackupManagerSP` class.

These improvements include removing redundant status tracking variables and replacing them with unified ones, updating the messaging system to handle all changes correctly, and including an enumeration `OperationType` to keep track of the type of operation currently being processed.

This commit also applies stricter conditions for restore operations, such that it only restores parameters that are currently marked as backupable, and skips those that are not. This is a preventive measure against potential issues when restoring parameters that are no longer relevant or could conflict with current versions.

Also, the encryption and decryption methods were updated to use AES-256 for more security. These changes have increased the robustness and reliability of the backup and restoration processes.

* copyright

* Add backup_manager process to offroad sunnypilot tasks

The backup_manager process is introduced to handle backups during offroad mode when SunnyLink is ready. This ensures proper backup management functionality integrated into the system.

* Simplify backup endpoint construction in restore method

Replaced conditional expression with a more concise and readable `or` operation for constructing the backup endpoint. This makes the code cleaner and easier to maintain while preserving functionality.

* Added support for backing up and restoring sunnypilot settings

An update to the sunnypilot functionality now provides two new features that allow users to backup and restore their sunnypilot settings. The changes include the addition of UI controls for initiating backup and restore operations, and the creation of a system-wide state management function for tracking these operations. This enhancement significantly improves the user experience by providing a safety net for user settings in case of software failures, bugs, or unintended changes.

* Refactor type hints to use PEP 604 syntax for clarity

Replaced `Optional` and `Dict` type hints with `|` and `dict` syntax for improved readability and compliance with Python 3.10+. Updated related imports and adjusted list comprehension for cleaner code.

* Update import path for hardware module in utils.py

Replaced the import path for the `Paths` module to align with the new directory structure under `openpilot`. This ensures compatibility with recent project reorganization and avoids import errors.

* Improve RSA key handling and fix backup status comparison

Added explicit RSA key type checks to handle invalid key formats. Enhanced type safety in `manager.py` by ensuring the backup status comparison returns a boolean. These changes improve robustness and error handling in backups.

* format

* more

* Improve backup and restore flow with progress tracking and fixes

Added proper progress tracking and cleanup logic during restore operations. Enhanced restore experience by resetting progress after completion and introducing confirmation dialogs for errors and completion. Updated API compatibility with a version query parameter for backups.

* Enable backup button only when restore process is complete

Previously, the backup button could be enabled during a restore operation, which might cause unintended behavior. This update introduces a check to ensure the backup button remains disabled while a restore process is active. This improves user experience and prevents potential conflicts.

* Fix restore button state handling during restore process

Ensure the restore button is disabled consistently when a restore is in progress. This prevents user interaction issues and aligns the button state with the restore operation status.

* "Refactor restore logic and improve button state handling"

Replaced `is_restoring` with `restore_request_pending` for clarity and better state management. Adjusted button behavior to immediately disable upon user action, ensuring improved UX and preventing repeat inputs. Refined restore completion flow for better consistency and reliability.

* Refine restore process logic for SunnyLink settings.

Introduced `restore_request_started` to improve handling of restore states and ensure accurate UI updates during the process. Adjusted case handling to enhance clarity and maintain proper behavior when restoring settings, especially during ongoing or completed requests.

* revert

* move around

* fix enabled states for different statuses

* add prompt to notify backup is complete

* same states as restore

* disable buttons if sunnylink is off

* can use the same texts

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2025-03-22 15:58:04 +00:00
DevTekVE
f0abacbc59 Merge branch 'refs/heads/master-at-17032025' into sync-17032025
# Conflicts:
#	opendbc_repo
#	selfdrive/car/card.py
#	selfdrive/car/tests/test_car_interfaces.py
#	selfdrive/car/tests/test_models.py
#	selfdrive/controls/controlsd.py
#	selfdrive/controls/lib/tests/test_latcontrol.py
#	selfdrive/pandad/panda.h
#	system/manager/process_config.py
Sync: `commaai/opendbc:master` into `sunnypilot/opendbc:master-new`
2025-03-18 01:05:34 -04:00
Adeeb Shihadeh
519ea9dcab Make model daemons Python processes (#34880) 2025-03-15 15:55:29 -07:00
DevTekVE
a4ef5ad982 Merge remote-tracking branch 'comma/master' into sync-20250309
# Conflicts:
#	.github/workflows/ui_preview.yaml
#	common/params_keys.h
#	msgq_repo
#	opendbc_repo
#	panda
#	selfdrive/ui/qt/offroad/firehose.h
#	selfdrive/ui/tests/test_ui/run.py
#	system/manager/process_config.py
2025-03-09 14:53:03 +01:00
Jason Wen
290383128e Sync: Fix conflicts 2025-03-09 00:53:06 -05:00
Adeeb Shihadeh
c32c405f65 remove model shims (#34750) 2025-03-02 13:39:36 -08:00
DevTekVE
1a8dd310ae Model: split modeld into it's own contained modeld implementation (#642)
* Add support for TinyGrad model runner processing

Introduced a new function `is_tinygrad_model` to detect TinyGrad as an active model runner. Updated the `is_stock_model` logic to account for TinyGrad models and added a new process entry for TinyGrad in the model manager. This enables handling TinyGrad models alongside existing configurations.

adding modeld back

Add support for `modeld_v2` and update paths for consistency

Updated `SConscript` files to integrate `modeld_v2` alongside `modeld` and adjusted script paths for correct metadata handling. Adjusted various configurations and scripts, such as `labeler.yaml` and `build_release.sh`, to include `modeld_v2` and ensure cohesive project structure.

Refactor imports to use updated `modeld_v2` paths.

Replaced outdated `modeld` references with their `modeld_v2` counterparts for consistency and clarity across the codebase. Also updated `.gitignore` to accommodate new directory structure. This change ensures better maintainability and alignment with the new directory schema.

Refactor and reorganize modeld to sunnypilot/modeld_v2 structure.

Moved and renamed `modeld` components to the new `sunnypilot/modeld_v2` directory for better organization and modularity. Updated imports and file references to align with the new structure, ensuring compatibility and functionality. Streamlined project structure to improve maintainability and future development.

* typo

* Use `stock` model runner and refactor model checks.

Replaces outdated model detection logic with unified `stock` runner integration, simplifying the decision flow for model selection. Includes `stock` as a new enum in the `Runner` type and updates affected references accordingly.

* Handle missing 'sim_pose' in model outputs gracefully.

Added conditional checks to ensure the code handles cases where 'sim_pose' is absent in the model outputs. Fallback behaviors use 'plan' data when 'sim_pose' is unavailable, preventing potential errors and enhancing robustness.
2025-03-02 20:49:30 +01:00
Adeeb Shihadeh
952def7abf start logging services first (#34646)
* start logging services first

* fix test

---------

Co-authored-by: Comma Device <device@comma.ai>
2025-02-20 16:53:07 -08:00
Jason Wen
7e5a84b897 Merge branch 'upstream/openpilot/master' into sync-20250125
# Conflicts:
#	opendbc_repo
#	panda
#	selfdrive/modeld/fill_model_msg.py
#	selfdrive/pandad/pandad.cc
#	selfdrive/test/process_replay/ref_commit
#	selfdrive/ui/qt/home.h
#	selfdrive/ui/qt/offroad/developer_panel.cc
#	selfdrive/ui/qt/offroad/developer_panel.h
#	selfdrive/ui/tests/test_ui/run.py
#	selfdrive/ui/translations/main_ar.ts
#	selfdrive/ui/translations/main_de.ts
#	selfdrive/ui/translations/main_es.ts
#	selfdrive/ui/translations/main_ko.ts
#	selfdrive/ui/translations/main_pt-BR.ts
#	selfdrive/ui/translations/main_th.ts
#	selfdrive/ui/translations/main_tr.ts
#	selfdrive/ui/translations/main_zh-CHS.ts
#	selfdrive/ui/translations/main_zh-CHT.ts
2025-01-25 22:50:03 -05:00
Shane Smiskol
4bb578ff1b Fix duplicate process names (#34446)
* test

* anything actually use this?

* keep it around
2025-01-21 17:29:56 -08:00
DevTekVE
9ccc5cc3ea models: retain pre-20hz drive model support (#531)
* modeld: Retain pre-20hz drive model support

* Method not available anymore on OP

* some fixes

* Revert "Long planner get accel: new function args (#34288)"

* Revert "Fix low-speed allow_throttle behavior in long planner (#33894)"

* Revert "long planner: allow throttle reflects usage (#33792)"

* Revert "Gate acceleration on model gas press predictions (#33643)"

* Reapply "Gate acceleration on model gas press predictions (#33643)"

This reverts commit 76b08e37cb.

* Reapply "long planner: allow throttle reflects usage (#33792)"

This reverts commit c75244ca4e.

* Reapply "Fix low-speed allow_throttle behavior in long planner (#33894)"

This reverts commit b2b7d21b7b.

* Reapply "Long planner get accel: new function args (#34288)"

This reverts commit 74dca2fccf.

* don't need

* retain snpe

* wrong

* they're symlinks

* remove

* put back into VCS

* add back

* don't include built

* Refactor model runner retrieval with caching support

Added caching for active model runner type via `ModelRunnerTypeCache` to enhance performance and avoid redundant checks. Introduced a `force_check` flag to bypass the cache when necessary. Updated related code to handle cache clearing during onroad transitions.

* Update model runner determination logic with caching fix

Enhances `get_active_model_runner` to utilize caching more effectively by ensuring type consistency and updating cache only when necessary. Also updates `is_snpe_model` to pass the `started` state to the runner determination function, improving behavior for dynamic checks.

* parse inputs via metadata

* load model and metadata dynamically

* cherry pick from devtekve as base

* lateral_control_params & prev_desired_curv: MLSIM V0 to Null Pointer

* old desired_curv data: MLSIM V1 to Postal Service

* Bringing what was on master back then

* Cleaning up

* Refactor model pipeline for modularity and dynamic input handling

Refactored the model pipeline by introducing helper functions to modularize model loading, metadata extraction, and input preparation. Improved flexibility in handling dynamic input keys and parsing outputs based on model configuration. Removed deprecated or unused code segments for cleaner and more maintainable structure.

* Push NDv2 because why not and fix modeld

* `Refactor model parsing and clean unused code dependencies`

Simplified `parse_outputs` by removing unnecessary `input_keys` parameter, ensuring cleaner logic. Updated `PROCESS_NAME` for standardization and eliminated deprecated `Pathlib` dependencies in model paths. Minor adjustments improve input handling for lateral control parameters.

* Refactor model and metadata loading functions.

Simplified and clarified `load_model` by renaming it to `get_model_path` and removing redundant variable assignments. Streamlined `load_metadata` by directly returning the loaded metadata without intermediate variables. These changes improve code readability and maintainability.

* Refactor modeld process selection based on SNPE support.

Introduce conditional logic to determine and start the appropriate modeld process (SNPE or default) based on hardware support. This improves flexibility and ensures correct process management.

* Walrus baby

* Update longitudinal_planner.py

* Improve model download progress handling and translations

Refactored model download status handling logic for better clarity and added mechanisms to track status changes efficiently. Updated UI text and translations across multiple languages to reflect consistent and accurate model download states.

* Revert "Update longitudinal_planner.py"

This reverts commit b44a687e4c.

* Fix variable naming for curvature size in modeld.py

Renamed the variable `len` to `length` to avoid conflict with the built-in `len()` function, improving code clarity and preventing potential errors. Also removed trailing spaces in commented-out sections for better formatting consistency.

* sync with upstream

* some are different

* should work

* sim_pose only exists in in ndv3 and prior

* dynamic meta constants

* fix

* uiview: disable power saving

* fix this

* ain't coasting for y'all

* Static analysis

* Refactor initialization of model inputs for clarity.

Removed redundant pre-initialization of `driving_style`, `nav_features`, and `nav_instructions` variables. Instead, directly initialize these within their respective conditional blocks for better readability and reduced memory usage.

* default to none

* enable in next PR

* more

* Revert "more"

This reverts commit f5a4220588.

* Revert "enable in next PR"

This reverts commit 621cc4f18e.

* no need to cast bool

* nuke

* fix desired curvature for pre LAv1 models

* mypy

* static

* fix

* new json

* Going to a test branch with a different model json list

* renamed model json

* Update model runner handling in custom.capnp and helpers

Refactored model runner logic by introducing a `runner` field in `custom.capnp` and simplifying `get_active_model_runner` logic. Removed deprecated function `get_model_runner_by_filename` and added a temporary filter in `fetcher.py` to enforce `snpe` until full tinygrad support is implemented.

* Revert "Update model runner handling in custom.capnp and helpers"

This reverts commit f34d872c13.

* Revert "renamed model json"

This reverts commit 15c6ed303b.

* Revert "Going to a test branch with a different model json list"

This reverts commit 4c1408fee5.

* Reapply "renamed model json"

This reverts commit c6fec6912a.

Reapply "Going to a test branch with a different model json list"

This reverts commit 83e253e9a3.

* Add 'runner' property to 'ModelBundle' and update relevant functions

The 'ModelBundle' class in 'custom.capnp' has been extended to include a 'runner' property. This required updating 'fetcher.py' to handle the 'runner' property when parsing model bundles. Additionally, the helper function 'get_model_runner_by_filename' has been removed from 'helpers.py' as it is no longer needed because the 'runner' property provides this information. The 'get_active_model_runner' function has also been updated in light of these changes.

* Refine bundle selection logic in SoftwarePanelSP

Improve logic for determining which model bundle to display by considering download status and failure state. Remove unnecessary function call to enhance clarity and maintainability.

* tmp

* Add retrieval of active model bundle in manager loop

Introduce a call to `get_active_bundle` to fetch the active model bundle and store it in `self.active_bundle`. This ensures the active bundle is always up-to-date during the model management process.

* Add "Use Default" option for model selection

Introduced a "Use Default" option in the model selection dropdown, allowing users to reset to the default model. Adjusted logic to handle default selection and ensure proper parameter updates. Fixed bundle status handling during the download process in the model manager.

* Refactor type hint in active_bundle assignment.

Removed an unnecessary type hint in the active_bundle assignment for cleaner and more maintainable code. This change does not affect functionality but improves code readability.

* no nested

* update json url

* split out

* format

* condense

* more

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
Co-authored-by: Kumar <36933347+rav4kumar@users.noreply.github.com>
2025-01-11 22:32:01 -05:00
Jason Wen
acd46aa94b modeld: retain SNPE and thneed drive model support (#555)
* modeld: Retain pre-20hz drive model support

* Method not available anymore on OP

* some fixes

* Revert "Long planner get accel: new function args (#34288)"

* Revert "Fix low-speed allow_throttle behavior in long planner (#33894)"

* Revert "long planner: allow throttle reflects usage (#33792)"

* Revert "Gate acceleration on model gas press predictions (#33643)"

* Reapply "Gate acceleration on model gas press predictions (#33643)"

This reverts commit 76b08e37cb.

* Reapply "long planner: allow throttle reflects usage (#33792)"

This reverts commit c75244ca4e.

* Reapply "Fix low-speed allow_throttle behavior in long planner (#33894)"

This reverts commit b2b7d21b7b.

* Reapply "Long planner get accel: new function args (#34288)"

This reverts commit 74dca2fccf.

* don't need

* retain snpe

* wrong

* they're symlinks

* remove

* put back into VCS

* add back

* don't include built

* Refactor model runner retrieval with caching support

Added caching for active model runner type via `ModelRunnerTypeCache` to enhance performance and avoid redundant checks. Introduced a `force_check` flag to bypass the cache when necessary. Updated related code to handle cache clearing during onroad transitions.

* Update model runner determination logic with caching fix

Enhances `get_active_model_runner` to utilize caching more effectively by ensuring type consistency and updating cache only when necessary. Also updates `is_snpe_model` to pass the `started` state to the runner determination function, improving behavior for dynamic checks.

* default to none

* enable in next PR

* more

---------

Co-authored-by: DevTekVE <devtekve@gmail.com>
2025-01-10 18:34:06 -05:00
Jason Wen
402ee444b7 Revert "modeld: retain SNPE and thneed drive model support" (#551)
Revert "modeld: retain SNPE and thneed drive model support (#530)"

This reverts commit 56aa07844a.
2025-01-10 16:25:16 -05:00
Jason Wen
56aa07844a modeld: retain SNPE and thneed drive model support (#530)
* modeld: Retain pre-20hz drive model support

* Method not available anymore on OP

* some fixes

* Revert "Long planner get accel: new function args (#34288)"

* Revert "Fix low-speed allow_throttle behavior in long planner (#33894)"

* Revert "long planner: allow throttle reflects usage (#33792)"

* Revert "Gate acceleration on model gas press predictions (#33643)"

* Reapply "Gate acceleration on model gas press predictions (#33643)"

This reverts commit 76b08e37cb.

* Reapply "long planner: allow throttle reflects usage (#33792)"

This reverts commit c75244ca4e.

* Reapply "Fix low-speed allow_throttle behavior in long planner (#33894)"

This reverts commit b2b7d21b7b.

* Reapply "Long planner get accel: new function args (#34288)"

This reverts commit 74dca2fccf.

* don't need

* retain snpe

* wrong

* they're symlinks

* remove

* put back into VCS

* add back

* don't include built

* Refactor model runner retrieval with caching support

Added caching for active model runner type via `ModelRunnerTypeCache` to enhance performance and avoid redundant checks. Introduced a `force_check` flag to bypass the cache when necessary. Updated related code to handle cache clearing during onroad transitions.

* Update model runner determination logic with caching fix

Enhances `get_active_model_runner` to utilize caching more effectively by ensuring type consistency and updating cache only when necessary. Also updates `is_snpe_model` to pass the `started` state to the runner determination function, improving behavior for dynamic checks.

* default to none

* enable in next PR

* more

---------

Co-authored-by: DevTekVE <devtekve@gmail.com>
2025-01-10 08:56:10 -05:00
Jason Wen
8edeb458d4 Merge branch 'openpilot/master' into sync-20250108
# Conflicts:
#	common/base.py
#	common/comma_connect.py
#	opendbc_repo
#	panda
2025-01-08 23:42:52 -05:00