Commit Graph

107 Commits

Author SHA1 Message Date
Kacper Rączy
b2d0f74ec3 process_replay: support for replaying multiple processes simultaneously (#28829)
* first working ProcessContainer and _replay_multi_process implementation

* Setup controlsd config_callback

* Add vision ipc support

* Remove old code

* add todo

* Update estimated processing times

* Use deepcopy when getting configs

* Rework the API to support passing multiple names or configs

* Add get_custom_params_from_lr

* Add more typing

* Enable simulation for controlsd

* Fix typing issues

* Regen refactor

* Fix linter issues

* Make wideRoadCameraState optional

* Move DummySocket to helpers

* Improve typing in regen

* Minor improvements

* Maintain sort order of internal_pub_queue using heapq

* Move ProcessContainer elsewhere

* Fix internal heap

* Change comment

* Remove subtest_name from ProcessConfig

* Update cereal

* Add newline
2023-07-18 14:56:24 -07:00
Kacper Rączy
2c617e0c7a process_replay: modeld and dmonitoringmodeld support (#28463)
* vipc support. modeld and dmonitoringmodeld configs

* Update cereal

* Add vision meta

* Use vision meta in process replay

* Exclude modeld and dmonitoringmodeld from test_processes

* Refactor other pieces of code to use vision meta

* remove unnecessary imports

* Add MAX_FRAMES_PER_SEGMENT

* cameraState migration

* Move camera state migration to migration module

* model_replay refactor

* Fix lint issues

* Reduce number of vipc buffers to 2

* Update spinner message

* Update model replay ref

* Support the new log order

* Replace CameraFrameSizes with dict. Add encode_index field

* Migrate camera states based on encode indices

* Refactor process replay

* Refactor model replay

* Remove unnecessary print

* Update model replay ref

* Make camera state migration optional

* Remove unused import

* Add ignore_alive_pubs field

* Add assertion for carParams inside fingerprinting callback

* Remove WideCameraOnly param setup

* Refactor ModeldCameraSyncRcvCallback

* Update model replay ref commit

* Fix camera state migration

* Only migrate camera states when process has vision pubs

* Refactor model replay again

* Fix linter issues

* One more linter fix

* Update model replay ref commit
2023-06-22 17:36:25 -07:00
Kacper Rączy
a2e75fdea6 process_replay: progress bar support (#28447)
* Add support for tqdm progress bar

* Fix argument name
2023-06-07 16:23:40 -07:00
Kacper Rączy
64377a88f9 process_replay: helpers (#28367)
* process_replay helper

* Refactor

* refactor check_enabled uses

* fix __init__

* noqa in __init__ imports

* Move compare_logs outside of process_replay

* Move save_logs to tools.lib.helpers

* Remove save_log from compare_logs

* fix lint issues
2023-06-06 10:31:13 -07:00
Harald Schäfer
598343aad1 Calibrationd: make recalibrating alert (#28149)
* Initial

* fixes

* not an int anymore

* elif

* revert ref

* update ref

* fix alert text

* regen refs

* update ref

* add recalibration unit test

* set into recalibration state

* fix words

* recalib

* text

* Update selfdrive/controls/lib/events.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2023-05-15 09:29:57 -07:00
Cameron Clough
51d41a3336 test Ford in process replay (#28112) 2023-05-04 14:47:39 -07:00
Shane Smiskol
d05d9dac04 Ford: remove from dashcam (#28103)
* no dashcam 🎉

* update docs

* not these

* update refs

* bump panda
2023-05-04 06:55:09 +00:00
Harald Schäfer
06756f4533 New process_replay segment with latest ublox logs (#28038)
* Add new segment

* Update ref_commit
2023-04-30 18:24:20 -07:00
Bruce Wayne
613bbcd626 Revert "LaikaD: new proc_replay segment and no internet laikad (#28029)"
This reverts commit fbb7dcb19c.
2023-04-26 19:20:05 -07:00
Harald Schäfer
fbb7dcb19c LaikaD: new proc_replay segment and no internet laikad (#28029)
* no internet in CI

* add new segment

* new ref
2023-04-26 16:01:39 -07:00
Adeeb Shihadeh
7319afbd51 process replay: prep for laikad subsock + QCOM GPS (#27632)
* process replay: prep for laikad subsock

* cleanup

* fix for qcomGnss

* detect ublox

* more debug info in err

* wip

* cleanup
2023-03-21 16:43:40 -07:00
Shane Smiskol
e729f5b763 Remove dashcam override param (#27509)
* remove dashcam override

* sort

* remove ford from segments
2023-03-06 15:36:39 -08:00
Adeeb Shihadeh
e3de6b8834 Chrysler: revert torque tuning (#27327)
* Chrysler: revert torque tuning

* update refs

* update refs + route
2023-02-13 16:15:59 -08:00
Adeeb Shihadeh
ed52916d55 update chrysler test segment with new flag 2023-02-10 23:47:47 -08:00
Shane Smiskol
cb62a5397e process replay: add Ford Bronco Sport segment (#27112)
* Add Ford segment to process replay

* fix dashcam cars

* clean up

* fix NotImplementedError

* Update refs
2023-01-27 14:14:04 -08:00
Shane Smiskol
e971bb11c2 Hyundai: bump safety params (#26407)
* bump panda

* bump panda

* bump panda

* update refs

* bump panda

* bump panda
2022-11-07 22:14:47 -08:00
Shane Smiskol
995c74a994 process replay: test Bolt EUV (camera-ACC platform) (#26270)
ADD BOLT TO PROCESS REPLAY
2022-10-27 14:36:47 -07:00
Shane Smiskol
f6416e7769 Hyundai HDA2 detection: use ADAS ECU (#26130)
* use adas ecu

* new route

* update refs

* new route

* update refs

* new test routes

* bump
2022-10-19 14:21:03 -07:00
Adeeb Shihadeh
c782380fc1 Hyundai: share panda flags with CAN-FD platform (#26058)
* Hyundai: share panda flags with CAN-FD platform

* move that

* only set bit

* bump panda

* panda master

* regen + update refs for new param
2022-10-12 21:54:08 -07:00
Shane Smiskol
4bd146ee7a Longitudinal planner: clip a_desired to cruise limits (#25928)
* Clip a_desired to cruise limits

* Update selfdrive/controls/lib/longitudinal_planner.py

* fix

* update refs

* explicit
2022-09-30 15:43:04 -07:00
Shane Smiskol
75735675bd process replay: rename second HYUNDAI entry 2022-09-30 14:49:27 -07:00
Kurt Nistelberger
96ed5aa581 Sensor events regen (#25903) 2022-09-27 20:33:45 -07:00
Vivek Aithal
4fa62f1464 Live torque (#25456)
* wip torqued

* add basic logic

* setup in manager

* check sanity and publish msg

* add first order filter to outputs

* wire up controlsd, and update gains

* rename intercept to offset

* add cloudlog, live values are not updated

* fix bugs, do not reset points for now

* fix crashes

* rename to main

* fix bugs, works offline

* fix float in cereal bug

* add latacc filter

* randomly choose points, approx for iid

* add variable decay

* local param to capnp instead of dict

* verify works in replay

* use torqued output in controlsd

* use in controlsd; use points from past routes

* controlsd bugfix

* filter before updating gains, needs to be replaced

* save all points to ensure smooth transition across routes, revert friction factor to 1.5

* add filters to prevent noisy low-speed data points; improve fit sanity

* add engaged buffer

* revert lat_acc thresh

* use paramsd realtime process config

* make latacc-to-torque generic, and overrideable

* move freq to 4Hz, avoid storing in np.array, don't publish points in the message

* float instead of np

* remove constant while storing pts

* rename slope, offset to lat_accet_factor, offset

* resolve issues

* use camelcase in all capnp params

* use camelcase everywhere

* reduce latacc threshold or sanity, add car_sane todo, save points properly

* add and check tag

* write param to disk at end of route

* remove args

* rebase op, cereal

* save on exit

* restore default handler

* cpu usage check

* add to process replay

* handle reset better, reduce unnecessary computation

* always publish raw values - useful for debug

* regen routes

* update refs

* checks on cache restore

* check tuning vals too

* clean that up

* reduce cpu usage

* reduce cpu usage by 75%

* cleanup

* optimize further

* handle reset condition better, don't put points in init, use only in corolla

* bump cereal after rebasing

* update refs

* Update common/params.cc

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* remove unnecessary checks

* Update RELEASES.md

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-09-19 15:19:26 -07:00
Vivek Aithal
b133a4c9a8 regenerate replay segments for torqued (#25805)
* update segments in test_processes

* bump cereal

* update refs
2022-09-15 20:15:57 -07:00
Jason Wen
bd5e28909f Hyundai: Car Port for Tucson Hybrid 2022 (#25276)
* Hyundai: Car Port for Tucson Hybrid 2022

* Update RELEASES.md

* Init gear_msg at the top

* FW versions from script

* Button send attempt

* start with some cleanup

* Send button fixed bits

* Define all bytes and only send PAUSE/RESUME

* Use CRUISE_INFO to cancel cruise and resume

* 8-bit counter

* Cleanup ish

* 8 bit counter

* Send at 20ms

* Disengage bits

* Revert bump submodules

* Allow tx on 0x1a0

* Fix byte counts

* Send LFA and HDA icons based on engageability

* Send cruise buttons only on HDA2 cars for now

* Add comments

* Add FLAG_HYUNDAI_CANFD_HDA2 flag

* Update interface.py

* Update carstate.py

* Update carstate.py

* Update carstate.py

* Bump submodules

* Bump panda

* Bump opendbc

* Allow tx with CRUISE_INACTIVE

* GEAR has 24 bytes only

* Generate car docs

* Fix CRUISE_INFO copy

* Remove unused class

* Add CAN-FD busses to unit test

* Bump opendbc

* Revert "Add CAN-FD busses to unit test"

This reverts commit 2f751640408a7f73a9100947cbd95ea13fbb8a48.

* Remove duplicate

* New tune based on data

* Panda safety cleanup

* Include bus 0 in rx checks

* Missed one

* bus 6 check

* Remove redundant check

* Add comments

* Bump opendbc

* Sync with DBC

* Hide LFA icon when disengaged

* Little endian

* fix comment

* more conditions in carcontroller

* update pedal signal

* update tuning

* cleanup carcontroller

* bump panda

* fix mismatch

* alt buttons

* little more cleanup

* update refs for EV6 new safety param

* bump panda

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-08-19 20:14:30 -07:00
Shane Smiskol
87ca42e993 process replay: initialize controlsState with original route (#25461)
* push

* do process replay

* commit

* update refs

* clean up

* clean up controlsd

* clean up controlsd

* Add assert

* debubuggier param name

* can be peristent

* Revert "can be peristent"

This reverts commit 6e6d3f6423c26a202623cef728e038259e9e46cd.

* Update selfdrive/test/process_replay/process_replay.py
2022-08-16 22:02:03 -07:00
Shane Smiskol
d7095d0d43 process replay: show errors when updating refs (#25460)
* Update test_processes.py

* Apply suggestions from code review
2022-08-16 19:39:11 -07:00
Jason Young
c27d7913f2 VW PQ: Volkswagen Passat NMS (#24768)
* VW PQ: Volkswagen Passat NMS

* regen CARS.md

* vEgo from Bremse_1 vehicle speed

* sync opendbc to master

* handle checksums and counters in opendbc

* LDW HUD message handling

* GRA_Neu_Zaehler -> COUNTER

* bump opendbc

* stub in till we find platform ACC standstill

* bump opendbc

* bump opendbc

* placeholder lateral accel data

* regen CARS.md

* counters now directly supported in opendbc

* additional door-open signals

* add trunk lid state

* add doors and trunk lid to signals list

* LDW_Status updates and passthrough

* bump opendbc for typo fix

* update AWV comment

* another comment update

* regen CARS.md with PQ in dashcam only

* don't show NMS footnotes while still in dashcam

* polish

* add stubbed-out dashcamOnly prep

* VW MQB: Cleanup stock ACC button handling

* bump opendbc and panda

* use controls resume output as trigger

* these can wait until taco bell

* bump opendbc

* pass through of previously fixed value

* retry CI

* checks already done in carcontroller

* don't need these anymore

* reduce diff for now

* slightly better abstraction

* more engine and trans FW

* turn signal is instantaneous stalk position

* weak sauce :(

* better clarity

* try torque tune

* add test route references

* bump opendbc and panda for OP long

* don't show steering faults for 3 seconds after start

* longitudinal control senders

* a little more torque

* test hax to torque control

* test a little more delay

* allow use of manufacturer ramp-up rate

* soften wheel-touch threshold

* Revert "test hax to torque control"

This reverts commit d1af459c29e36264aae406f72b8fcbc9ef22b9e0.

* punch it Chewie

* better ACC state and mainswitch handling

* a little more

* tweak max accel gradient

* oops

* also oops

* stuff

* srsly

* that's not how this works

* regen CARS.md

* footnotes now properly excluded for dashcam cars

* this wasn't a problem

* update network location detection

* bump submodules for ACC main switch

* clean up DBC references and long flag

* bump one more time

* one more time

* follow CANPacker counter refactor

* bump opendbc

* sync opendbc to master

* bump panda to fix Subaru tests

* DBC handling cleanup

* fix

* model-year stretch

* cleanup and rate bugfixes

* better abstractions

* simplify create_lka_hud_control

* volkswagencan -> mqbcan

* bump panda

* fix doc data bug, regen CARS.md

* style updates; diff reduction

* use common button enable logic

* not needed anymore

* refactor TSK and HUD enum values

* make common button events function

* consistency

* bump panda

* bump panda

* dashcam only

* don't need process_replay yet

* regen CARS.md with Passat NMS in dashcam

* can't handle dashcam-orphaned footnotes yet

* remove outdated standstill handling

* editor tried to be too helpful at some point

* don't need to import this anymore

* Apply suggestions from code review

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* follow parkingBrake refactor

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-08-16 14:32:59 -07:00
Shane Smiskol
4602e5a8f9 process replay: print correct ref path (#25448)
* fix log paths ref printing in process replay

* rm that

* this is easier to read

* fix model_replay
2022-08-16 00:32:42 -07:00
Adeeb Shihadeh
d462a08056 remove CS.steeringRateLimited (#25251)
* remove CS.steeringRateLimited

* bump cereal

* update refs
2022-07-24 14:56:55 -07:00
Adeeb Shihadeh
505bbce29d controlsd: handle radar missing + cleanup system error handling (#25179)
* controlsd: handle radar missing + cleanup system erorr handling

* fix carla

* update refs
2022-07-14 19:55:18 -07:00
Adeeb Shihadeh
780c60324b process replay: fix string failures after #25127 2022-07-12 11:12:35 -07:00
Gijs Koning
f0b5ff5c1a Replay: Fix --upload-only (#25127)
Add checking for list back
2022-07-12 05:03:35 -07:00
Gijs Koning
4d7b7483d7 Replay: tolerances per field (#25116)
* tolerances per field in replay

* refactor

* Remove laikad parameters

* Small comment change
2022-07-12 03:36:57 -07:00
Adeeb Shihadeh
2ae52e9b22 process replay: ensure enabled for significant amount of time (#25121)
* process replay: ensure enabled for significant amount of time

* update refs

* 10s is reasonable
2022-07-11 17:20:56 -07:00
Adeeb Shihadeh
d08a231774 Ship EV6 (#25095)
* Ship EV6

* bump opendbc
2022-07-08 22:46:20 -07:00
Adeeb Shihadeh
8d98d8c657 process replay: add Ram route (#25063) 2022-07-07 10:28:55 -07:00
Gijs Koning
edf170103e Process replay: Fix subtest diff (#25054)
Fix subtest diff
2022-07-07 02:14:31 -07:00
Shane Smiskol
ea241bf3dc FW fingerprinting: log all FW versions (#25042)
* get_fw_versions returns all fw versions with request's brand

* keep track of everything received

* debug

* need to regen or write a hack in build_fw_dict

* to be safe, still replace old responses within same brands (hyundai responds to two queries, can fix later)

to be safe, still replace old responses within same brands (hyundai responds to two queries, can fix later)

* update test_fw_query_on_routes

* clean up

* better name

* slightly cleaner

* fix test_startup unit test

del

* fix imports

* fix test_fw_fingerprint

fix test_fw_fingerprint

fix

* fingerprint on all FW_VERSIONS, not just brands with requests

* support old routes in test_fw_query_on_routes

* regen and update refs

* similar function style to before

* better comment

* space

switch name

* try to exact match first

* useless else

* fix debug script

* simpler dictionary

* bump cereal to master
2022-07-06 19:42:58 -07:00
Gijs Koning
c49f997be5 Add laikadOffline subtest to process replay. (#24995)
* Add subtests to process replay.
Adds laikadOffline subtest

* Update cpp.

* Update ref

* Update ref again

* Update ref again

* update ref

* Fix disabling fetching orbits

* Add proc name to event exception

* update ref

* Update setup_env

* Fix offline test and update refs
2022-06-30 08:39:12 -07:00
Shane Smiskol
170ed3d761 process replay: clean up common code (#24855)
* regen and process replay clean up

* test_fuzzy actually uses fingerprint hardcoding

fix

* revert

* revert

* this can be a url or path so just print full variable
2022-06-14 12:03:30 -07:00
Adeeb Shihadeh
0fce5d9045 Move a bunch of stuff to system/ part 3 (#24829)
* move swaglog.py

* timezoned

* logmessaged

* version.py

* fix linter
2022-06-11 23:19:27 -07:00
Lukas Petersson
50a8b30580 process replay: upload-only bug (#24722)
dont look for lr with upload only
2022-06-02 17:25:44 -07:00
Shane Smiskol
6a21cc83ff process replay: fix tested_cars 2022-06-02 14:20:19 -07:00
Shane Smiskol
90af436a12 process replay: fix updating refs (#24716)
fix updating refs
2022-06-02 13:49:21 -07:00
Lukas Petersson
86f73a507e process replay: logreader as bytes (#24610)
* willem's changes

* classmethod for bytes

* submodules

* submodules

* Update tools/lib/logreader.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* add back files

* little cleanup

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-06-01 22:02:42 -07:00
ClockeNessMnstr
3fb22c0289 process replay: only download used logs (#24661)
* only download wanted logs

* cleanup

* cache as an option

* cleanup

* Readme

* Revert "cache as an option"

This reverts commit 060ed4ade548696505e3185fc7d20f1897072ded.

* cleanup

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-06-01 21:46:45 -07:00
Shane Smiskol
dff70876c3 process replay: fix updating refs after regen (#24656)
* fix updating refs after regen

* match spacing

* same as before

* fix

* minimize diff

* update

* not used
2022-05-26 00:39:33 -07:00
Lukas Petersson
3e975eb3aa process replay: update regen body route (#24648)
update to a preserved route
2022-05-24 17:07:10 -07:00
Lukas Petersson
444d265821 process replay: support running in parallel (#24534)
* prefix params

* set env

* prefix in manager

* filesystem except

* dont delete manager folder

* Update selfdrive/common/params.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* debug same path

* remove cleanup + same default

* dont use filesystem lib

* param symlink path

* prefix helpers

* path

* dont delete d

* parallel loop

* refactor loop

* msgq

* clean msgs

* spelling

* nestable pool

* spelling

* logreaders in parallel

* bugfix

* assert msgq

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* assert in cereal

* folder exists

* create dirs in test

* bump cereal

* Update selfdrive/test/process_replay/process_replay.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* PR feedback

* params path in basedir

* ref commit

* param path

* bugfix

* upload_only

* param path

* Update selfdrive/test/process_replay/process_replay.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* msgq path name

* python concurrency.features

* progress bar

* remove progress bar from compare logs

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* defaultdict

* context manager

* update refs

* dont get logs if upload only

* upload refs in parallel

* cleanup

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* cleanup

* text

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: Shane Smiskol <shane@smiskol.com>
2022-05-18 19:59:30 -07:00