Commit Graph

179 Commits

Author SHA1 Message Date
Vivek Aithal
f8dce94688 params: Remove separate CarParams from each daemon (#30405)
* remove separate previous route carparams from each daemon and add centrally to controlsd

* extract out sigint handler

* make process replay work for torqued

* don't write param if None
old-commit-hash: 2eb487c9a5
2023-11-08 15:25:41 -08:00
Kacper Rączy
1027a60851 process_replay: use frame dimensions from FrameReader (#30332)
Use dimensions from FrameReader to setup vipc server
old-commit-hash: 4094d584f3
2023-10-25 15:46:16 -07:00
Harald Schäfer
7c4d29c3d1 Rm laikad (#30299)
* rm laika

* Rm laika

* Needed this

* More rm

* More rm
old-commit-hash: 56dea7b0b4
2023-10-23 00:15:39 -07:00
Justin Newberry
ef1acd4e6c Proc Replay: cleanup iterable log typing (#30261)
* cleanup log typing

* use iterable

* fix those

* Cleanup

* cleanup

* missed that one
old-commit-hash: 1820b5f259
2023-10-17 12:58:28 -07:00
Kacper Rączy
c33f68047b process_replay: migrate panda/peripheral/manager states (#30141)
* Migrate pandaStates

* Fix import

* Make pandaStates migration optional, so process_replay still works on logs without carParams

* peripheralState migration if its not available

* Do nothing if log file has peripheralStates

* Add migrate_managerState

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 898acb13f2
2023-10-04 13:41:50 -07:00
Kacper Rączy
71ad6ce228 process_replay: fix support for single cam segments (#30143)
* Fix assertion for vision_pubs

* Get camera states from meta

* Convert deviceType to string

* Use neo instead of eon for frame sizes dict

* update vision_pubs of process config when creating vipc
old-commit-hash: 9c5bbe1a91
2023-10-02 19:36:47 -07:00
Adeeb Shihadeh
9c8de3e08b simplify sensor alert (#29824)
* simplify

* update proc replay
old-commit-hash: a4aa801fbe
2023-09-26 16:47:21 -07:00
Shane Smiskol
13381ed446 services: capitalize constant service list (#30005)
* rename

* bump

* Apply suggestions from code review
old-commit-hash: 7f073483bb
2023-09-21 18:25:15 -07:00
Adeeb Shihadeh
53e0f5eb49 pytest: use a clean environment for all tests (#29788)
* pytest: use a clean environment for all tests

* rm that

* fix pj

* put build back

* fix params

* fix that

* handle no key

* that was removed

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
old-commit-hash: 7b6afbc162
2023-09-05 18:52:40 -07:00
Kacper Rączy
a40c0c9b6e process_replay: clean env vars after run (#29694)
Clean environ vars at the end of replay
old-commit-hash: bb14ba4e5e
2023-08-28 19:44:40 -07:00
Harald Schäfer
3a28e97a95 Revert: Locationd: enable laikad (#29687)
* Revert: Locationd: enable laikad

* update ref
old-commit-hash: 55b391509d
2023-08-28 16:27:11 -07:00
Harald Schäfer
bc017ab936 Locationd: enable laikad (#29570)
* experiment

* llk back

* update refs

* update ref

* no internet

* Update ref
old-commit-hash: 8205590624
2023-08-24 16:51:45 -07:00
Harald Schäfer
a967592f8f Revert laikad llk (#29535)
* Revert "bump laika (#29522)"

This reverts commit 19ab859e302d52b4f1667f05543ac622ad259e57.

* Revert "Locationd: switch to laikad (#29278)"

This reverts commit 5dda7be8ac08ef6b51283eda74569517bba905f6.
old-commit-hash: fda3106c28
2023-08-22 14:12:51 -07:00
Harald Schäfer
0174c5f21c Locationd: switch to laikad (#29278)
* use_laika

* Update test

* fix spacing

* fix proc replay

* small fix

* Try correct std

* cast correctly

* experimental

* simpler

* update reset thresholds

* improve

* Ready for live running

* outlier reject

* update laika

* add fine speeds

* more experiments

* fixes

* all in laika

* more small tweaks

* outlier rejection not needed

* fix conflict

* no more gpslocation

* update test

* refactor to laika

* runs again

* Update ref
old-commit-hash: fd3b8a3d75
2023-08-21 22:51:02 -07:00
Adeeb Shihadeh
16ec56398c add openpilot prefix to imports (#29498)
* add openpilot prefix to imports

* more

* more

* fix docs

* fix linter

* bump submodules

* fix patched tests

* update dynamic imports

* debug

* Revert "debug"

This reverts commit db5e13b9911cc74438bee123bc3430da6c31b24b.

* fix pm test
old-commit-hash: a9626f95b6
2023-08-20 20:49:55 -07:00
Harald Schäfer
92d1e3fabb Proc replay: field tolerance unused (#29479)
field tolerance unused
old-commit-hash: 1dea20e574
2023-08-20 13:05:18 -07:00
Shane Smiskol
1a2818d775 Revert "radard: tie radard frequency to modelV2 vol. 2 (#29240)" (#29474)
* Revert "static analysis fixes"

This reverts commit 2271d0adf9a02aee1359717c47cb6280893a6b9b.

* Revert "radard: tie radard frequency to modelV2 vol. 2 (#29240)"

This reverts commit 6757e90415766e84ad1b6c921a2edb29ceb3bbbe.

* update refs
old-commit-hash: 1a08c2986e
2023-08-18 22:56:43 -07:00
Kacper Rączy
deb8257322 process_replay: prevent creating orphaned processes if replay fails during setup (#29426)
Fix the ability to cleanup, if process container fails during start
old-commit-hash: f07e4fb49a
2023-08-16 10:37:19 -07:00
Kacper Rączy
4017b1707c radard: tie radard frequency to modelV2 vol. 2 (#29240)
* radard: tie radard frequency to modelV2

Accumulate parsed messages until state is updated (toyota)

Same for honda

Rename rr to something more descriptive

* Change _update method name, since signature has changed

* Update ref commit

* Some renames

* Check for number of cans
old-commit-hash: 460f5c8e0b
2023-08-11 18:07:06 -07:00
Justin Newberry
715f028a0e Ruff: enable flake8-builtins (#29315)
* enable flake8-builtins

* replace any with contains

* fix typo in pack

* fix type

* format is from the parent module, has to be enabled

* item_id

* fix item_id

* disable for id since that's what the remote server returns
old-commit-hash: 8793cbff40
2023-08-10 18:58:22 -07:00
Justin Newberry
a14f162bc7 Ruff: comprehensions conventions (#29317)
ignore c408
old-commit-hash: 4a9afd7554
2023-08-10 16:40:30 -07:00
Justin Newberry
3ee0786de3 CI: Pylint to ruff (#29294)
* pylint to riff

* pylint to riff

* pylint to riff

* revert more

* undo exclude removal

* exclude tinygrad

* set line length

* pylint exclusions to ruff

* same excludes as old linter

* fix tools QA

* remove unrequired check

* revert linting third_party

* ignore e402
old-commit-hash: 7fdd9fc37c
2023-08-09 13:26:54 -07:00
Justin Newberry
7b677910df Linter: Fix >160 line length test (#29287)
* test

* fix line length

* fix line length

* fix file size

* revert poetry

* cleanup for PR
old-commit-hash: cf64b27168
2023-08-08 17:13:35 -07:00
Justin Newberry
ff96892511 Revert "CI: pylint to ruff (#29276)" (#29284)
This reverts commit ffad7279cbad3c4f2ea786287b4624d3a3aa18d7.
old-commit-hash: a91764954b
2023-08-08 14:24:04 -07:00
Justin Newberry
99006365a3 CI: pylint to ruff (#29276)
* pylint to riff

* pylint to riff

* pylint to riff

* revert more

* undo exclude removal

* exclude tinygrad
old-commit-hash: cb31a53d8a
2023-08-08 13:41:30 -07:00
Kacper Rączy
c92e7265a6 Revert "radard: tie radard frequency to modelV2 (#29063)" (#29243)
This reverts commit 4b221cb4fd271664ab7d39c5cfefb4225ca07c2a.
old-commit-hash: 1774b4fc61
2023-08-04 20:25:40 -07:00
Kacper Rączy
79fa322d4a radard: tie radard frequency to modelV2 (#29063)
old-commit-hash: 255f5c13f2
2023-08-03 22:10:21 -07:00
Kacper Rączy
67ac96c8b4 process_replay: capture process output (#29027)
* Add ProcessOutputProxy

* Move launcher to its own field

* Move ProcessOutputCapture to its own file

* Return itself from __enter__ of OpenpilotPrefix

* Integrate ProcessOutputCapture into process_replay

* Add note about capture_output_store to README

* ipykernel import is optional

* Disable type checking for link_with_current_proc

* Remove assertion

* Decode outputs to utf-8

* read(self): return empty buf if its none

* Fix type annotations

* Replace fifo with regular file, to avoid hitting fifo size limit
old-commit-hash: 547a033a3c
2023-07-31 16:30:58 -07:00
Kacper Rączy
d50f057f02 process_replay: ability to permanently unlock some sockets (#29070)
Add unlocked_pubs for keeping some sockets permanently unlocked
old-commit-hash: 81e8cf414e
2023-07-20 16:03:58 -07:00
Kacper Rączy
4c6efa638c process_replay: support for replaying multiple processes simultaneously (#28829)
* first working ProcessContainer and _replay_multi_process implementation

* Setup controlsd config_callback

* Add vision ipc support

* Remove old code

* add todo

* Update estimated processing times

* Use deepcopy when getting configs

* Rework the API to support passing multiple names or configs

* Add get_custom_params_from_lr

* Add more typing

* Enable simulation for controlsd

* Fix typing issues

* Regen refactor

* Fix linter issues

* Make wideRoadCameraState optional

* Move DummySocket to helpers

* Improve typing in regen

* Minor improvements

* Maintain sort order of internal_pub_queue using heapq

* Move ProcessContainer elsewhere

* Fix internal heap

* Change comment

* Remove subtest_name from ProcessConfig

* Update cereal

* Add newline
old-commit-hash: b2d0f74ec3
2023-07-18 14:56:24 -07:00
Kacper Rączy
fe84dadaa5 controlsd: support for SIMULATION mode for process replay (#28895)
* Support for simulation flag while replay

* Update cereal

* Enable simulation for controlsd

* Update ref commit

* Update cereal
old-commit-hash: 72579a0e10
2023-07-12 10:32:02 -07:00
Kacper Rączy
1fa42d0b97 process_replay: modeld and dmonitoringmodeld support (#28463)
* vipc support. modeld and dmonitoringmodeld configs

* Update cereal

* Add vision meta

* Use vision meta in process replay

* Exclude modeld and dmonitoringmodeld from test_processes

* Refactor other pieces of code to use vision meta

* remove unnecessary imports

* Add MAX_FRAMES_PER_SEGMENT

* cameraState migration

* Move camera state migration to migration module

* model_replay refactor

* Fix lint issues

* Reduce number of vipc buffers to 2

* Update spinner message

* Update model replay ref

* Support the new log order

* Replace CameraFrameSizes with dict. Add encode_index field

* Migrate camera states based on encode indices

* Refactor process replay

* Refactor model replay

* Remove unnecessary print

* Update model replay ref

* Make camera state migration optional

* Remove unused import

* Add ignore_alive_pubs field

* Add assertion for carParams inside fingerprinting callback

* Remove WideCameraOnly param setup

* Refactor ModeldCameraSyncRcvCallback

* Update model replay ref commit

* Fix camera state migration

* Only migrate camera states when process has vision pubs

* Refactor model replay again

* Fix linter issues

* One more linter fix

* Update model replay ref commit
old-commit-hash: 2c617e0c7a
2023-06-22 17:36:25 -07:00
Kacper Rączy
0b2580b20a process_replay: custom params (#28624)
* Add custom params argument

* Fix default arguments

* Fix typing issues
old-commit-hash: 3cae32a85f
2023-06-21 20:16:14 -07:00
Dean Lee
eddc84d104 automatically detect available camera streams (#27640)
* remove WideCameraOnly

* check in set_initial_state

* no block

* remove try block

* apply reviews
old-commit-hash: 4ae0378f63
2023-06-12 13:40:43 -07:00
Kacper Rączy
0541586636 process_replay: deprecated message migration (#28456)
* Move migration code to new file

* Migrate messages in process replay. Remove old code

* Fix old_logtime

* Keep old logtimes on migrated messages

* Update ref commit
old-commit-hash: 838346d8a3
2023-06-09 19:17:50 -07:00
Kacper Rączy
d3a3c73b0c process_replay: progress bar support (#28447)
* Add support for tqdm progress bar

* Fix argument name
old-commit-hash: a2e75fdea6
2023-06-07 16:23:40 -07:00
Kacper Rączy
0a3d0b48f7 process_replay: improved support for drained pubs (#28452)
Stop triggering empty update if there're no drained messages in the queue
old-commit-hash: d932aae327
2023-06-07 16:23:02 -07:00
Kacper Rączy
8b4e704df3 process_replay: remove get_car_params_callback when it's not needed (#28451)
* Add assertion for can msgs in get_car_params_callback

* Remove get_car_params_callback from processes that don't need it
old-commit-hash: 02841cede3
2023-06-07 16:22:48 -07:00
Kacper Rączy
2edd65ad4e process_replay: helpers (#28367)
* process_replay helper

* Refactor

* refactor check_enabled uses

* fix __init__

* noqa in __init__ imports

* Move compare_logs outside of process_replay

* Move save_logs to tools.lib.helpers

* Remove save_log from compare_logs

* fix lint issues
old-commit-hash: 64377a88f9
2023-06-06 10:31:13 -07:00
Kacper Rączy
64e8f332dc regen: fix params (#28375)
Fix environment setup for regen
old-commit-hash: 1a0f482d4d
2023-06-02 15:39:23 -07:00
Kacper Rączy
e6855f049f process_replay: fix crash on segments without carParams (#28286)
* Fix crash on segments without carParams

* Add assertion for carParams
old-commit-hash: a075905c4a
2023-05-27 02:10:10 +02:00
Kacper Rączy
7ab3f88015 process_replay: use PARAMS_ROOT (#28282)
Use PARAMS_ROOT in process_replay
old-commit-hash: ac0272d0cc
2023-05-24 13:16:22 -07:00
Kacper Rączy
a48b67f720 process_replay: universal replay mechanism using cereal fake sockets (#28115)
* replay_process_with_fake_sockets implementation

* add missing polled_pubs to configs

* drained_pubs field

* updated cereal

* Remove python/native variations of process replay. Replace with universal one using cereal fake sockets

* Replace old py FakeSocket with DummySocket

* Invalidate and deregister fake sockets after replay is done

* Remove unused import

* Set up new prefix for each replay

* Fixes for radard

* Refactor ReplayContext and ProcessConfig

* Minor fixes

* Reimplement controlsd fingerprinting callback

* time.sleep for sockets to safely reconnect

* Fix fingerprinting for controlsd

* Fixes for regen to work

* Fix replay loop to respect submaster frames

* Fix profiler to use new ProcessConfig fields

* Remove tqdm

* Refactor tests to use new ProcessConfig

* Add FrequencyBasedRcvCallback

* Make tolerance None by default

* Update cereal

* Add get_process_config utility func

* Update cereal. Simplify sync procedure

* Chain context managers

* New sub-socket reconnection procedure

* Fix linter issues

* Revert chaining of context managers

* Init controlsState only when replaying controlsd. Update cereal

* Update cereal

* Update process_replay to use new cereal API

* Update cereal

* Update cereal

* Update cereal

* Simplify radard recv callback

* Update release/files_common
old-commit-hash: e6ac6320ac
2023-05-23 17:24:16 -07:00
Kacper Rączy
dd70d2bdf3 process_replay: drain sockets instead recv_one (#28161)
* Drain the sockets until the next cycle in python process replay

* Add waiting mechanism to native process replay, which mimics the syncing

* Sort logs when comparing them. Drain all the sockets instead of only recv_socks.

* Reverse the retrieval order of FakeSocket

* Change pub order for torqued

* Update ref commit
old-commit-hash: 12b3ffcc49
2023-05-23 12:59:09 -07:00
Harald Schäfer
c8de368187 Laikad: blocking raw_gnss receive (#28083)
* Blocking receive

* redundant

* rm clocks

* timeout irrelevant now
old-commit-hash: bb3388f9d2
2023-05-03 13:03:25 -07:00
Kacper Rączy
46e075aa36 regen: whitelist-proc (#27980)
* Add support for whitelist-procs to toggle on/off specific processes during regen

* Use process CONFIGS from process_replay to generate fake processes instead of hard-coded values

* Add rawgpsd to fake_daemons. Remove duplicates from replayable_messages.

* Remove duplicate block of code from setup_env

* Remove rawgpsd comment
old-commit-hash: ba4a1c1b87
2023-04-28 19:36:31 -07:00
Kacper Rączy
650ed84920 regen: ecamera support (#27999)
* Add wideRoadCameraState support in regen

* Fix for ecam FrameReader when use_route_meta

* Pylint fixes
old-commit-hash: eee7dd082f
2023-04-22 02:43:04 +00:00
Adeeb Shihadeh
6f52cc09f4 process replay: fix socket-ful replay flakiness (#27721)
* process replay: fix socket-ful replay flakiness

* bump up laikad timeout

* bump

* bump again

* revert that
old-commit-hash: 85908f7e2b
2023-03-28 17:48:16 -07:00
Shane Smiskol
cdcf06e9e2 boardd: ability to switch between ELM safety params (#27656)
* indecisive

* rename to generic FW query

* remove code and update comment

* we need this to start off, unless we set multiplexing immediately

* draft

* draft 2

* try that

* can't do this either, boardd might read Enabled after removing, but before setting new Request param

* this should work

* use one less param

* fix params

* match behavior (set all pandas to safety param of 1, disabling multiplexing for fingerprinting

* clean up (some tests may temp break)

* fix param name and sort

* time it

* yes it does matter

* add to hyundai's bus 5 query

* remove hyundai for now

* this should work

* clean up

* clean up

* flip argument around, clean up

* fix test_startup

* some clean up

* rm line

* comment makes more sense

* required typing

* clean up common type

* comments

* Update selfdrive/car/car_helpers.py

* line

* whoops, need to set before vin!

* fix debug

* annoying

* more debugging

* bug fix (needs both keys always)

* debuGG

debuGG

* Revert "debuGG"

This reverts commit 55b2f429324c0b92d5cfb2cabf8b20db1e166248.

* Revert "more debugging"

This reverts commit 02934c3403ad5270f03093508b704c151d1ccb2a.

* Revert "annoying"

This reverts commit 8b4e5e09989f9a0217e3ec1c0ba68735929b7366.

* clean that up

* bumpback

* bumpback

* every second write param

* flip

* stuff

* move up?

* fix timing out in CI

* rm
old-commit-hash: 42449b482d
2023-03-23 00:14:31 -07:00
Adeeb Shihadeh
c2fc5175fc process replay: prep for laikad subsock + QCOM GPS (#27632)
* process replay: prep for laikad subsock

* cleanup

* fix for qcomGnss

* detect ublox

* more debug info in err

* wip

* cleanup
old-commit-hash: 7319afbd51
2023-03-21 16:43:40 -07:00