Commit Graph

177 Commits

Author SHA1 Message Date
Kacper Rączy
e62756596b controlsd: require joystick mode param for the body (#30680)
* Update joystick mode in step

* Stop putting the body in joystick mode by default

* Toggle JoystickDebugMode by default for notCars in process_replay

* Remove subsection about web joystick

* Update param only in non-cars

* Make joystickDebug non-static event, and add it in a loop

* Add comment
2023-12-12 14:14:51 -08:00
Adeeb Shihadeh
a2b48efa20 process replay: check valid flag (#30588) 2023-12-02 10:47:08 -08:00
Adeeb Shihadeh
b97e5b0e03 rename carEvents -> onroadEvents (#30577)
* rename carEvents -> onroadEvents

* bump cereal

* bump cereal
2023-12-01 22:48:29 -08:00
Vivek Aithal
2eb487c9a5 params: Remove separate CarParams from each daemon (#30405)
* remove separate previous route carparams from each daemon and add centrally to controlsd

* extract out sigint handler

* make process replay work for torqued

* don't write param if None
2023-11-08 15:25:41 -08:00
Kacper Rączy
4094d584f3 process_replay: use frame dimensions from FrameReader (#30332)
Use dimensions from FrameReader to setup vipc server
2023-10-25 15:46:16 -07:00
Harald Schäfer
56dea7b0b4 Rm laikad (#30299)
* rm laika

* Rm laika

* Needed this

* More rm

* More rm
2023-10-23 00:15:39 -07:00
Justin Newberry
1820b5f259 Proc Replay: cleanup iterable log typing (#30261)
* cleanup log typing

* use iterable

* fix those

* Cleanup

* cleanup

* missed that one
2023-10-17 12:58:28 -07:00
Kacper Rączy
898acb13f2 process_replay: migrate panda/peripheral/manager states (#30141)
* Migrate pandaStates

* Fix import

* Make pandaStates migration optional, so process_replay still works on logs without carParams

* peripheralState migration if its not available

* Do nothing if log file has peripheralStates

* Add migrate_managerState

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2023-10-04 13:41:50 -07:00
Kacper Rączy
9c5bbe1a91 process_replay: fix support for single cam segments (#30143)
* Fix assertion for vision_pubs

* Get camera states from meta

* Convert deviceType to string

* Use neo instead of eon for frame sizes dict

* update vision_pubs of process config when creating vipc
2023-10-02 19:36:47 -07:00
Adeeb Shihadeh
a4aa801fbe simplify sensor alert (#29824)
* simplify

* update proc replay
2023-09-26 16:47:21 -07:00
Shane Smiskol
7f073483bb services: capitalize constant service list (#30005)
* rename

* bump

* Apply suggestions from code review
2023-09-21 18:25:15 -07:00
Adeeb Shihadeh
7b6afbc162 pytest: use a clean environment for all tests (#29788)
* pytest: use a clean environment for all tests

* rm that

* fix pj

* put build back

* fix params

* fix that

* handle no key

* that was removed

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
2023-09-05 18:52:40 -07:00
Kacper Rączy
bb14ba4e5e process_replay: clean env vars after run (#29694)
Clean environ vars at the end of replay
2023-08-28 19:44:40 -07:00
Harald Schäfer
55b391509d Revert: Locationd: enable laikad (#29687)
* Revert: Locationd: enable laikad

* update ref
2023-08-28 16:27:11 -07:00
Harald Schäfer
8205590624 Locationd: enable laikad (#29570)
* experiment

* llk back

* update refs

* update ref

* no internet

* Update ref
2023-08-24 16:51:45 -07:00
Harald Schäfer
fda3106c28 Revert laikad llk (#29535)
* Revert "bump laika (#29522)"

This reverts commit 3523bef732.

* Revert "Locationd: switch to laikad (#29278)"

This reverts commit fd3b8a3d75.
2023-08-22 14:12:51 -07:00
Harald Schäfer
fd3b8a3d75 Locationd: switch to laikad (#29278)
* use_laika

* Update test

* fix spacing

* fix proc replay

* small fix

* Try correct std

* cast correctly

* experimental

* simpler

* update reset thresholds

* improve

* Ready for live running

* outlier reject

* update laika

* add fine speeds

* more experiments

* fixes

* all in laika

* more small tweaks

* outlier rejection not needed

* fix conflict

* no more gpslocation

* update test

* refactor to laika

* runs again

* Update ref
2023-08-21 22:51:02 -07:00
Adeeb Shihadeh
a9626f95b6 add openpilot prefix to imports (#29498)
* add openpilot prefix to imports

* more

* more

* fix docs

* fix linter

* bump submodules

* fix patched tests

* update dynamic imports

* debug

* Revert "debug"

This reverts commit db5e13b9911cc74438bee123bc3430da6c31b24b.

* fix pm test
2023-08-20 20:49:55 -07:00
Harald Schäfer
1dea20e574 Proc replay: field tolerance unused (#29479)
field tolerance unused
2023-08-20 13:05:18 -07:00
Shane Smiskol
1a08c2986e Revert "radard: tie radard frequency to modelV2 vol. 2 (#29240)" (#29474)
* Revert "static analysis fixes"

This reverts commit a1e85f3661.

* Revert "radard: tie radard frequency to modelV2 vol. 2 (#29240)"

This reverts commit 460f5c8e0b.

* update refs
2023-08-18 22:56:43 -07:00
Kacper Rączy
f07e4fb49a process_replay: prevent creating orphaned processes if replay fails during setup (#29426)
Fix the ability to cleanup, if process container fails during start
2023-08-16 10:37:19 -07:00
Kacper Rączy
460f5c8e0b radard: tie radard frequency to modelV2 vol. 2 (#29240)
* radard: tie radard frequency to modelV2

Accumulate parsed messages until state is updated (toyota)

Same for honda

Rename rr to something more descriptive

* Change _update method name, since signature has changed

* Update ref commit

* Some renames

* Check for number of cans
2023-08-11 18:07:06 -07:00
Justin Newberry
8793cbff40 Ruff: enable flake8-builtins (#29315)
* enable flake8-builtins

* replace any with contains

* fix typo in pack

* fix type

* format is from the parent module, has to be enabled

* item_id

* fix item_id

* disable for id since that's what the remote server returns
2023-08-10 18:58:22 -07:00
Justin Newberry
4a9afd7554 Ruff: comprehensions conventions (#29317)
ignore c408
2023-08-10 16:40:30 -07:00
Justin Newberry
7fdd9fc37c CI: Pylint to ruff (#29294)
* pylint to riff

* pylint to riff

* pylint to riff

* revert more

* undo exclude removal

* exclude tinygrad

* set line length

* pylint exclusions to ruff

* same excludes as old linter

* fix tools QA

* remove unrequired check

* revert linting third_party

* ignore e402
2023-08-09 13:26:54 -07:00
Justin Newberry
cf64b27168 Linter: Fix >160 line length test (#29287)
* test

* fix line length

* fix line length

* fix file size

* revert poetry

* cleanup for PR
2023-08-08 17:13:35 -07:00
Justin Newberry
a91764954b Revert "CI: pylint to ruff (#29276)" (#29284)
This reverts commit cb31a53d8a.
2023-08-08 14:24:04 -07:00
Justin Newberry
cb31a53d8a CI: pylint to ruff (#29276)
* pylint to riff

* pylint to riff

* pylint to riff

* revert more

* undo exclude removal

* exclude tinygrad
2023-08-08 13:41:30 -07:00
Kacper Rączy
1774b4fc61 Revert "radard: tie radard frequency to modelV2 (#29063)" (#29243)
This reverts commit 255f5c13f2.
2023-08-04 20:25:40 -07:00
Kacper Rączy
255f5c13f2 radard: tie radard frequency to modelV2 (#29063) 2023-08-03 22:10:21 -07:00
Kacper Rączy
547a033a3c process_replay: capture process output (#29027)
* Add ProcessOutputProxy

* Move launcher to its own field

* Move ProcessOutputCapture to its own file

* Return itself from __enter__ of OpenpilotPrefix

* Integrate ProcessOutputCapture into process_replay

* Add note about capture_output_store to README

* ipykernel import is optional

* Disable type checking for link_with_current_proc

* Remove assertion

* Decode outputs to utf-8

* read(self): return empty buf if its none

* Fix type annotations

* Replace fifo with regular file, to avoid hitting fifo size limit
2023-07-31 16:30:58 -07:00
Kacper Rączy
81e8cf414e process_replay: ability to permanently unlock some sockets (#29070)
Add unlocked_pubs for keeping some sockets permanently unlocked
2023-07-20 16:03:58 -07:00
Kacper Rączy
b2d0f74ec3 process_replay: support for replaying multiple processes simultaneously (#28829)
* first working ProcessContainer and _replay_multi_process implementation

* Setup controlsd config_callback

* Add vision ipc support

* Remove old code

* add todo

* Update estimated processing times

* Use deepcopy when getting configs

* Rework the API to support passing multiple names or configs

* Add get_custom_params_from_lr

* Add more typing

* Enable simulation for controlsd

* Fix typing issues

* Regen refactor

* Fix linter issues

* Make wideRoadCameraState optional

* Move DummySocket to helpers

* Improve typing in regen

* Minor improvements

* Maintain sort order of internal_pub_queue using heapq

* Move ProcessContainer elsewhere

* Fix internal heap

* Change comment

* Remove subtest_name from ProcessConfig

* Update cereal

* Add newline
2023-07-18 14:56:24 -07:00
Kacper Rączy
72579a0e10 controlsd: support for SIMULATION mode for process replay (#28895)
* Support for simulation flag while replay

* Update cereal

* Enable simulation for controlsd

* Update ref commit

* Update cereal
2023-07-12 10:32:02 -07:00
Kacper Rączy
2c617e0c7a process_replay: modeld and dmonitoringmodeld support (#28463)
* vipc support. modeld and dmonitoringmodeld configs

* Update cereal

* Add vision meta

* Use vision meta in process replay

* Exclude modeld and dmonitoringmodeld from test_processes

* Refactor other pieces of code to use vision meta

* remove unnecessary imports

* Add MAX_FRAMES_PER_SEGMENT

* cameraState migration

* Move camera state migration to migration module

* model_replay refactor

* Fix lint issues

* Reduce number of vipc buffers to 2

* Update spinner message

* Update model replay ref

* Support the new log order

* Replace CameraFrameSizes with dict. Add encode_index field

* Migrate camera states based on encode indices

* Refactor process replay

* Refactor model replay

* Remove unnecessary print

* Update model replay ref

* Make camera state migration optional

* Remove unused import

* Add ignore_alive_pubs field

* Add assertion for carParams inside fingerprinting callback

* Remove WideCameraOnly param setup

* Refactor ModeldCameraSyncRcvCallback

* Update model replay ref commit

* Fix camera state migration

* Only migrate camera states when process has vision pubs

* Refactor model replay again

* Fix linter issues

* One more linter fix

* Update model replay ref commit
2023-06-22 17:36:25 -07:00
Kacper Rączy
3cae32a85f process_replay: custom params (#28624)
* Add custom params argument

* Fix default arguments

* Fix typing issues
2023-06-21 20:16:14 -07:00
Dean Lee
4ae0378f63 automatically detect available camera streams (#27640)
* remove WideCameraOnly

* check in set_initial_state

* no block

* remove try block

* apply reviews
2023-06-12 13:40:43 -07:00
Kacper Rączy
838346d8a3 process_replay: deprecated message migration (#28456)
* Move migration code to new file

* Migrate messages in process replay. Remove old code

* Fix old_logtime

* Keep old logtimes on migrated messages

* Update ref commit
2023-06-09 19:17:50 -07:00
Kacper Rączy
a2e75fdea6 process_replay: progress bar support (#28447)
* Add support for tqdm progress bar

* Fix argument name
2023-06-07 16:23:40 -07:00
Kacper Rączy
d932aae327 process_replay: improved support for drained pubs (#28452)
Stop triggering empty update if there're no drained messages in the queue
2023-06-07 16:23:02 -07:00
Kacper Rączy
02841cede3 process_replay: remove get_car_params_callback when it's not needed (#28451)
* Add assertion for can msgs in get_car_params_callback

* Remove get_car_params_callback from processes that don't need it
2023-06-07 16:22:48 -07:00
Kacper Rączy
64377a88f9 process_replay: helpers (#28367)
* process_replay helper

* Refactor

* refactor check_enabled uses

* fix __init__

* noqa in __init__ imports

* Move compare_logs outside of process_replay

* Move save_logs to tools.lib.helpers

* Remove save_log from compare_logs

* fix lint issues
2023-06-06 10:31:13 -07:00
Kacper Rączy
1a0f482d4d regen: fix params (#28375)
Fix environment setup for regen
2023-06-02 15:39:23 -07:00
Kacper Rączy
a075905c4a process_replay: fix crash on segments without carParams (#28286)
* Fix crash on segments without carParams

* Add assertion for carParams
2023-05-27 02:10:10 +02:00
Kacper Rączy
ac0272d0cc process_replay: use PARAMS_ROOT (#28282)
Use PARAMS_ROOT in process_replay
2023-05-24 13:16:22 -07:00
Kacper Rączy
e6ac6320ac process_replay: universal replay mechanism using cereal fake sockets (#28115)
* replay_process_with_fake_sockets implementation

* add missing polled_pubs to configs

* drained_pubs field

* updated cereal

* Remove python/native variations of process replay. Replace with universal one using cereal fake sockets

* Replace old py FakeSocket with DummySocket

* Invalidate and deregister fake sockets after replay is done

* Remove unused import

* Set up new prefix for each replay

* Fixes for radard

* Refactor ReplayContext and ProcessConfig

* Minor fixes

* Reimplement controlsd fingerprinting callback

* time.sleep for sockets to safely reconnect

* Fix fingerprinting for controlsd

* Fixes for regen to work

* Fix replay loop to respect submaster frames

* Fix profiler to use new ProcessConfig fields

* Remove tqdm

* Refactor tests to use new ProcessConfig

* Add FrequencyBasedRcvCallback

* Make tolerance None by default

* Update cereal

* Add get_process_config utility func

* Update cereal. Simplify sync procedure

* Chain context managers

* New sub-socket reconnection procedure

* Fix linter issues

* Revert chaining of context managers

* Init controlsState only when replaying controlsd. Update cereal

* Update cereal

* Update process_replay to use new cereal API

* Update cereal

* Update cereal

* Update cereal

* Simplify radard recv callback

* Update release/files_common
2023-05-23 17:24:16 -07:00
Kacper Rączy
12b3ffcc49 process_replay: drain sockets instead recv_one (#28161)
* Drain the sockets until the next cycle in python process replay

* Add waiting mechanism to native process replay, which mimics the syncing

* Sort logs when comparing them. Drain all the sockets instead of only recv_socks.

* Reverse the retrieval order of FakeSocket

* Change pub order for torqued

* Update ref commit
2023-05-23 12:59:09 -07:00
Harald Schäfer
bb3388f9d2 Laikad: blocking raw_gnss receive (#28083)
* Blocking receive

* redundant

* rm clocks

* timeout irrelevant now
2023-05-03 13:03:25 -07:00
Kacper Rączy
ba4a1c1b87 regen: whitelist-proc (#27980)
* Add support for whitelist-procs to toggle on/off specific processes during regen

* Use process CONFIGS from process_replay to generate fake processes instead of hard-coded values

* Add rawgpsd to fake_daemons. Remove duplicates from replayable_messages.

* Remove duplicate block of code from setup_env

* Remove rawgpsd comment
2023-04-28 19:36:31 -07:00
Kacper Rączy
eee7dd082f regen: ecamera support (#27999)
* Add wideRoadCameraState support in regen

* Fix for ecam FrameReader when use_route_meta

* Pylint fixes
2023-04-22 02:43:04 +00:00