Commit Graph

149 Commits

Author SHA1 Message Date
Kacper Rączy
1774b4fc61 Revert "radard: tie radard frequency to modelV2 (#29063)" (#29243)
This reverts commit 255f5c13f2.
2023-08-04 20:25:40 -07:00
Kacper Rączy
255f5c13f2 radard: tie radard frequency to modelV2 (#29063) 2023-08-03 22:10:21 -07:00
Kacper Rączy
547a033a3c process_replay: capture process output (#29027)
* Add ProcessOutputProxy

* Move launcher to its own field

* Move ProcessOutputCapture to its own file

* Return itself from __enter__ of OpenpilotPrefix

* Integrate ProcessOutputCapture into process_replay

* Add note about capture_output_store to README

* ipykernel import is optional

* Disable type checking for link_with_current_proc

* Remove assertion

* Decode outputs to utf-8

* read(self): return empty buf if its none

* Fix type annotations

* Replace fifo with regular file, to avoid hitting fifo size limit
2023-07-31 16:30:58 -07:00
Kacper Rączy
81e8cf414e process_replay: ability to permanently unlock some sockets (#29070)
Add unlocked_pubs for keeping some sockets permanently unlocked
2023-07-20 16:03:58 -07:00
Kacper Rączy
b2d0f74ec3 process_replay: support for replaying multiple processes simultaneously (#28829)
* first working ProcessContainer and _replay_multi_process implementation

* Setup controlsd config_callback

* Add vision ipc support

* Remove old code

* add todo

* Update estimated processing times

* Use deepcopy when getting configs

* Rework the API to support passing multiple names or configs

* Add get_custom_params_from_lr

* Add more typing

* Enable simulation for controlsd

* Fix typing issues

* Regen refactor

* Fix linter issues

* Make wideRoadCameraState optional

* Move DummySocket to helpers

* Improve typing in regen

* Minor improvements

* Maintain sort order of internal_pub_queue using heapq

* Move ProcessContainer elsewhere

* Fix internal heap

* Change comment

* Remove subtest_name from ProcessConfig

* Update cereal

* Add newline
2023-07-18 14:56:24 -07:00
Kacper Rączy
72579a0e10 controlsd: support for SIMULATION mode for process replay (#28895)
* Support for simulation flag while replay

* Update cereal

* Enable simulation for controlsd

* Update ref commit

* Update cereal
2023-07-12 10:32:02 -07:00
Kacper Rączy
2c617e0c7a process_replay: modeld and dmonitoringmodeld support (#28463)
* vipc support. modeld and dmonitoringmodeld configs

* Update cereal

* Add vision meta

* Use vision meta in process replay

* Exclude modeld and dmonitoringmodeld from test_processes

* Refactor other pieces of code to use vision meta

* remove unnecessary imports

* Add MAX_FRAMES_PER_SEGMENT

* cameraState migration

* Move camera state migration to migration module

* model_replay refactor

* Fix lint issues

* Reduce number of vipc buffers to 2

* Update spinner message

* Update model replay ref

* Support the new log order

* Replace CameraFrameSizes with dict. Add encode_index field

* Migrate camera states based on encode indices

* Refactor process replay

* Refactor model replay

* Remove unnecessary print

* Update model replay ref

* Make camera state migration optional

* Remove unused import

* Add ignore_alive_pubs field

* Add assertion for carParams inside fingerprinting callback

* Remove WideCameraOnly param setup

* Refactor ModeldCameraSyncRcvCallback

* Update model replay ref commit

* Fix camera state migration

* Only migrate camera states when process has vision pubs

* Refactor model replay again

* Fix linter issues

* One more linter fix

* Update model replay ref commit
2023-06-22 17:36:25 -07:00
Kacper Rączy
3cae32a85f process_replay: custom params (#28624)
* Add custom params argument

* Fix default arguments

* Fix typing issues
2023-06-21 20:16:14 -07:00
Dean Lee
4ae0378f63 automatically detect available camera streams (#27640)
* remove WideCameraOnly

* check in set_initial_state

* no block

* remove try block

* apply reviews
2023-06-12 13:40:43 -07:00
Kacper Rączy
838346d8a3 process_replay: deprecated message migration (#28456)
* Move migration code to new file

* Migrate messages in process replay. Remove old code

* Fix old_logtime

* Keep old logtimes on migrated messages

* Update ref commit
2023-06-09 19:17:50 -07:00
Kacper Rączy
a2e75fdea6 process_replay: progress bar support (#28447)
* Add support for tqdm progress bar

* Fix argument name
2023-06-07 16:23:40 -07:00
Kacper Rączy
d932aae327 process_replay: improved support for drained pubs (#28452)
Stop triggering empty update if there're no drained messages in the queue
2023-06-07 16:23:02 -07:00
Kacper Rączy
02841cede3 process_replay: remove get_car_params_callback when it's not needed (#28451)
* Add assertion for can msgs in get_car_params_callback

* Remove get_car_params_callback from processes that don't need it
2023-06-07 16:22:48 -07:00
Kacper Rączy
64377a88f9 process_replay: helpers (#28367)
* process_replay helper

* Refactor

* refactor check_enabled uses

* fix __init__

* noqa in __init__ imports

* Move compare_logs outside of process_replay

* Move save_logs to tools.lib.helpers

* Remove save_log from compare_logs

* fix lint issues
2023-06-06 10:31:13 -07:00
Kacper Rączy
1a0f482d4d regen: fix params (#28375)
Fix environment setup for regen
2023-06-02 15:39:23 -07:00
Kacper Rączy
a075905c4a process_replay: fix crash on segments without carParams (#28286)
* Fix crash on segments without carParams

* Add assertion for carParams
2023-05-27 02:10:10 +02:00
Kacper Rączy
ac0272d0cc process_replay: use PARAMS_ROOT (#28282)
Use PARAMS_ROOT in process_replay
2023-05-24 13:16:22 -07:00
Kacper Rączy
e6ac6320ac process_replay: universal replay mechanism using cereal fake sockets (#28115)
* replay_process_with_fake_sockets implementation

* add missing polled_pubs to configs

* drained_pubs field

* updated cereal

* Remove python/native variations of process replay. Replace with universal one using cereal fake sockets

* Replace old py FakeSocket with DummySocket

* Invalidate and deregister fake sockets after replay is done

* Remove unused import

* Set up new prefix for each replay

* Fixes for radard

* Refactor ReplayContext and ProcessConfig

* Minor fixes

* Reimplement controlsd fingerprinting callback

* time.sleep for sockets to safely reconnect

* Fix fingerprinting for controlsd

* Fixes for regen to work

* Fix replay loop to respect submaster frames

* Fix profiler to use new ProcessConfig fields

* Remove tqdm

* Refactor tests to use new ProcessConfig

* Add FrequencyBasedRcvCallback

* Make tolerance None by default

* Update cereal

* Add get_process_config utility func

* Update cereal. Simplify sync procedure

* Chain context managers

* New sub-socket reconnection procedure

* Fix linter issues

* Revert chaining of context managers

* Init controlsState only when replaying controlsd. Update cereal

* Update cereal

* Update process_replay to use new cereal API

* Update cereal

* Update cereal

* Update cereal

* Simplify radard recv callback

* Update release/files_common
2023-05-23 17:24:16 -07:00
Kacper Rączy
12b3ffcc49 process_replay: drain sockets instead recv_one (#28161)
* Drain the sockets until the next cycle in python process replay

* Add waiting mechanism to native process replay, which mimics the syncing

* Sort logs when comparing them. Drain all the sockets instead of only recv_socks.

* Reverse the retrieval order of FakeSocket

* Change pub order for torqued

* Update ref commit
2023-05-23 12:59:09 -07:00
Harald Schäfer
bb3388f9d2 Laikad: blocking raw_gnss receive (#28083)
* Blocking receive

* redundant

* rm clocks

* timeout irrelevant now
2023-05-03 13:03:25 -07:00
Kacper Rączy
ba4a1c1b87 regen: whitelist-proc (#27980)
* Add support for whitelist-procs to toggle on/off specific processes during regen

* Use process CONFIGS from process_replay to generate fake processes instead of hard-coded values

* Add rawgpsd to fake_daemons. Remove duplicates from replayable_messages.

* Remove duplicate block of code from setup_env

* Remove rawgpsd comment
2023-04-28 19:36:31 -07:00
Kacper Rączy
eee7dd082f regen: ecamera support (#27999)
* Add wideRoadCameraState support in regen

* Fix for ecam FrameReader when use_route_meta

* Pylint fixes
2023-04-22 02:43:04 +00:00
Adeeb Shihadeh
85908f7e2b process replay: fix socket-ful replay flakiness (#27721)
* process replay: fix socket-ful replay flakiness

* bump up laikad timeout

* bump

* bump again

* revert that
2023-03-28 17:48:16 -07:00
Shane Smiskol
42449b482d boardd: ability to switch between ELM safety params (#27656)
* indecisive

* rename to generic FW query

* remove code and update comment

* we need this to start off, unless we set multiplexing immediately

* draft

* draft 2

* try that

* can't do this either, boardd might read Enabled after removing, but before setting new Request param

* this should work

* use one less param

* fix params

* match behavior (set all pandas to safety param of 1, disabling multiplexing for fingerprinting

* clean up (some tests may temp break)

* fix param name and sort

* time it

* yes it does matter

* add to hyundai's bus 5 query

* remove hyundai for now

* this should work

* clean up

* clean up

* flip argument around, clean up

* fix test_startup

* some clean up

* rm line

* comment makes more sense

* required typing

* clean up common type

* comments

* Update selfdrive/car/car_helpers.py

* line

* whoops, need to set before vin!

* fix debug

* annoying

* more debugging

* bug fix (needs both keys always)

* debuGG

debuGG

* Revert "debuGG"

This reverts commit 55b2f429324c0b92d5cfb2cabf8b20db1e166248.

* Revert "more debugging"

This reverts commit 02934c3403ad5270f03093508b704c151d1ccb2a.

* Revert "annoying"

This reverts commit 8b4e5e09989f9a0217e3ec1c0ba68735929b7366.

* clean that up

* bumpback

* bumpback

* every second write param

* flip

* stuff

* move up?

* fix timing out in CI

* rm
2023-03-23 00:14:31 -07:00
Adeeb Shihadeh
7319afbd51 process replay: prep for laikad subsock + QCOM GPS (#27632)
* process replay: prep for laikad subsock

* cleanup

* fix for qcomGnss

* detect ublox

* more debug info in err

* wip

* cleanup
2023-03-21 16:43:40 -07:00
Adeeb Shihadeh
ee36c106af process replay: update after laika publish policy change (#27647)
* laika pubs all

* lr for locationd

* fix locationd qcom

* update refs
2023-03-21 16:14:22 -07:00
Shane Smiskol
e729f5b763 Remove dashcam override param (#27509)
* remove dashcam override

* sort

* remove ford from segments
2023-03-06 15:36:39 -08:00
Harald Schäfer
eb8bdc0026 MPC path in UI (#27380)
* 10s lat

* Full length MPC

* redfine N

* Leave controls the same for now

* Updates

* use long plan in lat plan

* interp plan

* simplergit add selfdrive/controls/plannerd.py selfdrive/controls/

* expand to 10s

* revert this

* fix linter

* vizualize

* fix long test

* typo

* cleanup

* compiles

* unused

* unused

* bump cereal

* bump cereal

* use model if no uiplanm

* update replay

* update ref commit

* bump cereal to master
2023-02-17 19:58:30 -08:00
Shane Smiskol
2bc2dbfb70 process replay: fix hang 2023-02-15 15:55:08 -08:00
Adeeb Shihadeh
6fa9b555a4 rename carState.radarOffCan to radarUnavailable (#27187)
* rename carState.radarOffCan to radarUnavailable

* bump cereal
2023-02-02 11:38:09 -08:00
Shane Smiskol
4f25bd05f2 process replay: test routes in dashcam (#27118)
* fix routes with dashcam override param

* move to bottom
2023-01-27 13:06:33 -08:00
Adeeb Shihadeh
0eed31f1a6 process replay: show process name in timeout error 2023-01-27 11:27:07 -08:00
Kurt Nistelberger
0756c7f374 make process replay more reliable 2023-01-25 10:46:06 -08:00
Adeeb Shihadeh
c408b7e306 UI: show experimental long available to release users (#26910)
* UI: show experimental long available to release users

* more description

* cleanup

* hide toggle

* fix process replay

* clear param

* actually fix
2023-01-19 23:06:18 -08:00
Kurt Nistelberger
9131da9103 Revert "Add low Gnss laikad test (#26987)"
This reverts commit b7ce77b3aa.
2023-01-19 11:46:22 -08:00
Kurt Nistelberger
b7ce77b3aa Add low Gnss laikad test (#26987)
* add low gnss test

* enable laikad

* temp fix for cache

* update replay

* save LaikadEphemeris for testing

* rem

* update refs

* add comment

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>
2023-01-18 22:29:12 -08:00
Harald Schäfer
4c33d94d3f Revert locationd to gpsLocation (#26963)
* revert locationd laika change

* switch msg

* change proc replay

* back to gpslocation

* update ref

* no np floats in msg

* Ignore empty laikad messages in mapos

* more tolerance

* c++ doesnt work like that
2023-01-14 16:56:14 -08:00
Kurt Nistelberger
88423e25df Laikad: set active (#26850)
* laikad update, renaming

* update locationd

* fix naming

* address PR comments

* upsi

* .

* draft to fix replay

* fix process relay to allow no response for messages

* final fix for process replay

* .

* bump cereal

* update process replay ref commit

* reduce wait time

* .

* last ref change

* move laikad helpers to laika

* .

* fix ublox test

* update refs

* add proper qcom replay support

* fix gnss support if both is available

* update refs

* remove left over

* revert laikad msg

* move laika back to master

* init

* fix gps valid flag

* change time

* add gnss to ignore

* remove gps_valid flag

* .

* adopt orientation reset threshold

* .

* update laikad

* .

* fix stanstill KF resets

* test orienation reset count

* update laika

* bump cereal

* fix process replay

* update laika repo

* remove handle gps

* add extra logging for cache

* .

* add more log

* .

* .

* update laika

* dont remove gps code

* inc min satellite count

* update magic vals and add acc drop

* update laika

* upsi

* rem

* bump laika

* use nav and correct

* more fixes

* use sftp

* No more glonass

* Revert "No more glonass"

This reverts commit a76124da50a1e25f423ad1137c7a046e1d57811d.

* nump laika

* back support old ephemeris cache

* add health to ephemeris message

* bump laika

* remove print

* fix laikad tests

* clean

* remove extra log

* bump laika

* inc timeout for plotjuggler build

* rem cache clear

* .

* enable gps after checks

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>
Co-authored-by: Bruce Wayne <harald.the.engineer@gmail.com>
Co-authored-by: Shane Smiskol <shane@smiskol.com>
2023-01-13 11:37:38 -08:00
Shane Smiskol
b00bc4f57e Car interface: require fingerprint and FW versions to get params (#26932)
* require fingerprint and FW versions

* add get_non_essential_params()

* comment

* all required

* classmethod, need to allow subclasses to override _get_params

* fix that

* fix

* fix

* wrong fix 🤦
2023-01-12 12:25:24 -08:00
Adeeb Shihadeh
f14deae475 Revert "Car interface: require fingerprint and FW versions to get params (#26766)"
This reverts commit b68dabb689.
2023-01-12 11:28:04 -08:00
Shane Smiskol
b68dabb689 Car interface: require fingerprint and FW versions to get params (#26766)
* require fingerprint and FW versions

* add get_non_essential_params()

* comment

* all required

* classmethod, need to allow subclasses to override _get_params

* fix that

* fix
2023-01-11 23:44:50 -08:00
Kurt Nistelberger
981532f0c3 Laikad preperation (#26800)
* laikad update, renaming

* update locationd

* address PR comments

* draft to fix replay

* fix process relay to allow no response for messages

* bump cereal

* update process replay ref commit

* move laikad helpers to laika

* fix ublox test

* update refs

* add proper qcom replay support

* fix gnss support if both is available

* update refs

* move laika back to master

* move cereal back to master

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>
2022-12-29 16:56:35 -08:00
Shane Smiskol
c2326a4219 process replay: support experimental long (#26233)
Support experimental long
2022-10-27 16:44:22 -07:00
Adeeb Shihadeh
01d05f66fe auto-detect pigeon or quectel (#25991)
* auto-detect pigeon or quectel

* persistent

* fix sim

* fix process replay

* fix locationd unit tests

* fix that

Co-authored-by: Comma Device <device@comma.ai>
2022-10-06 16:46:15 -07:00
Adeeb Shihadeh
80c40048cb sensord: test fixups (#25935)
* sensord: test fixups

* fix linter

* check freq

* fp seems reliable

* clean that up

* update refs

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Bruce Wayne <batman@z840-provisioning.internal>
2022-09-30 12:25:11 -07:00
Kurt Nistelberger
29d3ed2ce6 Sensor events splitup (#25714)
* PoC of reading sensors via interrupts instead of polling

* add Gyro and draft for magn

* add more functionality to gpio.cc

* change LSM gyro to interrupt

* resolve rebase conflict

* update BMX accel interrupt impl

* add interrupt collector thread to fetch in parallel

* change get_event interface to return true on successful read

* update BMX gyro interrupt impl

* update gpio.h/.cc according to comments

* address comments, rename Edgetype enum

* Edgetype to EdgeType

* update sensor interrupt interface

* add error handling, and read fd on trigger

* avoid sending empty messages

* fix build

* use gpiochip

* less diff

* gpiochip on both edges, but skip falling edge if rising edge is detected

* init last_ts with 0

* update sensord testcases

* update sensord testsweet

* test for pipeline

* readd with_process

* add null check

* move tests update to seperate PR

* sensord: improve test coverage (#25683)

* update sensord-interrupt testsweet

* address review comments

* inc stddev threshold

* fix format string

* add version 0 check again

* relax strictness after c3 with bmx tests

* relax strictness after tests

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>

* address PR comments

* fix typo

* remove 4ms limit, and skip first 0.5sec of data

* revert disable_interuppt change to destructor

* fix and remove timing skip

* make gpiochip generic

* sensord port

* change from sensorEvents to separated events

* fix gyro usage

* add splitted sensor tests

* modify debug script sensor_data_to_hist.py

* refactor get_event interface to remove sensorEvent message type

* update locationd to non sensorEvent usage

* tmp commit

* fix replay

* fix accelerometer type

* fix sensor to hist debug script

* update sensord tests to split events

* remove rebase artifacts

* port test_sensord.py

* small clean up

* change cereal to sensorEvents-splitup branch

* upate sensorEvents in regen

* fix route generation for splitted sensor events

* regen cleanUp from sensorEvents change

* .

* remove light and temp from locationd

* add generic init delay per sensor

* .

* update routes

* move bmx gyro/accel to its own channel

* adopt sensor tests to bmx channel

* remove rebase artifacts

* fix sensord test

* handle bmx not present

* add bmx sockets to regen

* .

* .

* code cleanUp

* .

* address PR comments

* address PR comments

* address PR comments

* lsm clean up

* readd sensorEvents

* rever regen.py

* .

* update replay refs

* move channels

* fix artifact

* bump cereal

* update refs

* fix timing issue

Co-authored-by: Bruce Wayne <batman@workstation-eu-intern2.eu.local>
Co-authored-by: gast04 <kurt.nistelberger@gmail.com>
Co-authored-by: Willem Melching <willem.melching@gmail.com>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-09-29 14:31:54 -07:00
Vivek Aithal
4fa62f1464 Live torque (#25456)
* wip torqued

* add basic logic

* setup in manager

* check sanity and publish msg

* add first order filter to outputs

* wire up controlsd, and update gains

* rename intercept to offset

* add cloudlog, live values are not updated

* fix bugs, do not reset points for now

* fix crashes

* rename to main

* fix bugs, works offline

* fix float in cereal bug

* add latacc filter

* randomly choose points, approx for iid

* add variable decay

* local param to capnp instead of dict

* verify works in replay

* use torqued output in controlsd

* use in controlsd; use points from past routes

* controlsd bugfix

* filter before updating gains, needs to be replaced

* save all points to ensure smooth transition across routes, revert friction factor to 1.5

* add filters to prevent noisy low-speed data points; improve fit sanity

* add engaged buffer

* revert lat_acc thresh

* use paramsd realtime process config

* make latacc-to-torque generic, and overrideable

* move freq to 4Hz, avoid storing in np.array, don't publish points in the message

* float instead of np

* remove constant while storing pts

* rename slope, offset to lat_accet_factor, offset

* resolve issues

* use camelcase in all capnp params

* use camelcase everywhere

* reduce latacc threshold or sanity, add car_sane todo, save points properly

* add and check tag

* write param to disk at end of route

* remove args

* rebase op, cereal

* save on exit

* restore default handler

* cpu usage check

* add to process replay

* handle reset better, reduce unnecessary computation

* always publish raw values - useful for debug

* regen routes

* update refs

* checks on cache restore

* check tuning vals too

* clean that up

* reduce cpu usage

* reduce cpu usage by 75%

* cleanup

* optimize further

* handle reset condition better, don't put points in init, use only in corolla

* bump cereal after rebasing

* update refs

* Update common/params.cc

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* remove unnecessary checks

* Update RELEASES.md

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-09-19 15:19:26 -07:00
Adeeb Shihadeh
90a4565eb2 params: make python and c++ API match (#25573)
* params: make python and c++ API match

* few more
2022-08-26 20:46:19 -07:00
Shane Smiskol
87ca42e993 process replay: initialize controlsState with original route (#25461)
* push

* do process replay

* commit

* update refs

* clean up

* clean up controlsd

* clean up controlsd

* Add assert

* debubuggier param name

* can be peristent

* Revert "can be peristent"

This reverts commit 6e6d3f6423c26a202623cef728e038259e9e46cd.

* Update selfdrive/test/process_replay/process_replay.py
2022-08-16 22:02:03 -07:00
Chris Souers
5b1ba4f135 Lateral planner: Don't prompt/initiate lane changes if lateral control is not active (#25436)
* Don't set lane change events if we're not able to steer.

* more specific variable name

* update process replay socket config

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-08-15 11:37:24 -07:00