Commit Graph

58 Commits

Author SHA1 Message Date
Lukas Petersson
444d265821 process replay: support running in parallel (#24534)
* prefix params

* set env

* prefix in manager

* filesystem except

* dont delete manager folder

* Update selfdrive/common/params.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* debug same path

* remove cleanup + same default

* dont use filesystem lib

* param symlink path

* prefix helpers

* path

* dont delete d

* parallel loop

* refactor loop

* msgq

* clean msgs

* spelling

* nestable pool

* spelling

* logreaders in parallel

* bugfix

* assert msgq

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* assert in cereal

* folder exists

* create dirs in test

* bump cereal

* Update selfdrive/test/process_replay/process_replay.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* PR feedback

* params path in basedir

* ref commit

* param path

* bugfix

* upload_only

* param path

* Update selfdrive/test/process_replay/process_replay.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* msgq path name

* python concurrency.features

* progress bar

* remove progress bar from compare logs

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* defaultdict

* context manager

* update refs

* dont get logs if upload only

* upload refs in parallel

* cleanup

* Update selfdrive/test/process_replay/test_processes.py

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* cleanup

* text

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: Shane Smiskol <shane@smiskol.com>
2022-05-18 19:59:30 -07:00
Shane Smiskol
d686f9e527 process replay: save logs to folder (#24509)
* save logs to folder for proc replay

* use fakedata dir

* fix

* only once and where needed
2022-05-12 19:19:47 -07:00
Shane Smiskol
934bc1e6fb process replay: don't print diff when only uploading (#24514)
when only uploading no need to print success status or update diff

don't print succeed/fail when only uploading
2022-05-12 17:05:22 -07:00
grekiki
17e33978cd Mypy fixes for --check-untyped-defs (#24372)
more type fixes
2022-05-12 14:08:11 +02:00
Shane Smiskol
d2d3b7b823 process replay: automatically push refs on fail (#24414)
* test failure()

* let's just change a tune here

* debug

revert

* debug

* use current commit, not ref_commit

fix

* need to figure out better place for this

* quick test

* test without upload

* temp

* use azure token

* fixes

* shouldn't need this

* debug

* debug

* not getting anything?

* does this mean nothing gets envvars?

* add azure token to docker environment variables

* quote

* move back

* clean up a bit

* more clean up

* like this sorting better

* replace flags with CI and clean up

* test FULL_TEST and minimalize diff a bit

* now test all

* revert tests comments

* remove flags

* revert this

revert this

* now make it fail

* now update ref_commit to last commit (make sure we can re-start this test if we commit before last one finishes uploading)

* fix

fix

fix

fix

* bad commit

* why is it not throwing an exception?

* debug

* URLFile returns empty bytes if using cache and remote file doesn't exist

* we always need to download anyway

* debug...

* duh, wrong file. but neither should have it

* add that back and just check explicitly

* check both

* clean up and make a diff

* stylize

* see if this is a better diff on files changed

* update refs

* revert changes

* only for owners or members

* if we have token access

* if we have token access

* if we have token access

* move up

* clean up

* revert

* move update refs to test_processes

* clean up

* update messages

* update msg

* update README and delete update_refs

* this isn't possible to reach anymore

* fix readme

* better help message

better help message

better help message

* only show basename when uploading, only if failed to find

* test diff

* fix printing old ref commit

* change to using

* update refs

* Revert "update refs"

This reverts commit 2e352a736a6de68e2c7064daa4e2e9409ce77686.

* revert

* ref refers to reference commit/logs, cur refers to current logs/commit (future ref)

* like for better

* Apply suggestions from code review

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/test/process_replay/test_processes.py

* every time lol

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-05-11 15:03:05 -07:00
Shane Smiskol
4190d2608a Change safetyParam to uint16_t (#24376)
* bump panda & cereal

* bump panda

* rest of references

* regen TOYOTA with safety param 578, TOYOTA2 with 329

* regen rest of routes

* update ref

* bump cereal
2022-05-02 13:34:31 -07:00
Shane Smiskol
c1caca104f Support RAV4 Hybrid 2022 with stock longitudinal (#23969)
* add panda flag for toyota stock long with camera

* clean up

* Add 2022 RAV4 Hybrid from Philly

* fix wrong fw in interface, did this ever work?

* Must be a hybrid

* no radar parsing

* fix can error

* move to own platform

* generate docs

* fix

* Add 2022 Rav4 XSE Australia fingerprint parameters (#24303)

* Update values.py

Add 2022 Rav4 XSE Australia

* add commas

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* bump panda

* wait, the camera doesn't even send 0x343, right?

* use a set instead, more obvious

* don't test without a parser

* bump panda

* flip panda flag

* bump panda

* add commas

* regen and update refs

* set to none by default

* revert parenthesis

* update comment

* bump panda

* regen and update refs

* add test models and update readme

* bump to master

Co-authored-by: BrettLynch123 <34538435+BrettLynch123@users.noreply.github.com>
2022-04-28 22:37:46 -07:00
Shane Smiskol
66dd8f934c safety param: make uint16_t (#24362)
* only use half of the available bytes

* move to set_safety_model

* regen and update refs

* last two

* bump to master
2022-04-28 17:56:34 -07:00
Adeeb Shihadeh
3e098c9ba5 Toyota: add safety param flag for alt brake message (#24341)
* Toyota: add safety param flag for alt brake message

* update refs

* update refs

* bump panda
2022-04-27 12:08:11 -07:00
Shane Smiskol
6e62df65d7 Make safetyParam unsigned (#24160)
* bump panda & cereal

* make unsigned in boardd

* update health packet version

bump panda

* fix definition

* bump

* regen TOYOTA2 route

* upload route

* oh, ndog param wasn't being set

* regenerate routes with non-zero safety params

* update refs

* regen routes and update refs

* bump panda

* update refs

* update refs

* bump submodules to master
2022-04-21 00:01:05 -07:00
Shane Smiskol
1a5ddfaedd regen: check if regenerated route is enabled (#24207)
* check if regenerated route is enabled

* elif

* initialize CUDA runner before all other processes

* Revert "initialize CUDA runner before all other processes"

This reverts commit 124e3dd4d3408b6649f40b3e6e7f217ca347f59f.

* add back fixes

* fix typo
2022-04-13 18:01:16 -07:00
Adeeb Shihadeh
6c5b0685da body: set fault state (#24125)
* body: set fault state

* fix that

* update refs
2022-04-04 23:05:40 -07:00
Adeeb Shihadeh
00ebd10b87 body: only control while engaged (#24114)
* body: only control while engaged

* bump panda

* remove that

* bump panda

* update refs

* update refs
2022-04-04 19:37:21 -07:00
Adeeb Shihadeh
e40c36f22b better support non-car robots (#24106)
* better support non-car robots

* bump cereal

* update refs
2022-04-01 20:03:20 -07:00
Igor Biletskyy
031f79ba88 car port: comma body (#24019)
* body FPv2

* ..

* ..

* ..

* Temp, REVERT!

* more cleanup

* typo

* ..

* del eyes

* should work?

* fix

* new dbc

* ..

* fixes

* static analysis

* cln balancing code

* no test route

* excluded_interfaces

* THE DOCS!

* comments on steer/speed mixin

* switch to bus 0

* less UDS

* FAKE bus 0 vin and fingerprint

* FAKE locationd

* Keep steady and remove handcoded offset

* Improve startup sequence, get closer to stock openpilot

* Forgot to define angle

* lowercase

* revert that

* little cleanup

* change safety model

* update refs

* body is gold

* handle no llk

* oops

* litte more

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-03-31 17:16:15 -07:00
Willem Melching
d3d10e014a Ensure controlsd can engage again in process replay (#23556)
* hacks to make process replay engage again

* dont change that

* enable engagement check again

* chmod +x

* first working regen

* proper logMonoTime

* fix video framerate consistency

* mpc is valid again

* proper alive checks

* revert loggerd change

* ensure engaged in regen, fix managerState, peripheralState

* ubloxRaw is unused

* add submaster config for gm

* regen all services we can run without HW

* fix loggerd

* loggerd: matroska without the extension

* update ref
2022-01-21 13:13:34 +01:00
Ewout ter Hoeven
332f568a82 Pyupgrade 3.6: Update syntax with Python 3.6+ features (#23305)
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default

Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>

Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
2021-12-24 11:18:39 -08:00
Ewout ter Hoeven
55390d273f Convert format strings strings to f-strings (#23241)
* Convert all text strings to f-strings

Reformats all the text from the old "%-formatted" and .format(...) format to the newer f-string format, as defined in PEP 498. This requires Python 3.6+.

Flynt 0.69 was used to reformat the strings. 120 f-strings were created in 51 files.

F-strings are in general more readable, concise and performant. See also: https://www.python.org/dev/peps/pep-0498/#rationale

* revert pyextra changes

* revert ublox.py

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-12-16 14:58:17 +01:00
Adeeb Shihadeh
2252c6bba7 Mazda: move CX-9 out of dashcam mode (#22717)
* move cx9 out of dashcam

* no alc below 28

* process replay route
2021-10-31 22:02:36 -07:00
Robbe Derks
91987f38d4 Make pandaState and safetyMode a list (#22454)
* wip: move to pandaStates

* bump cereal

* wip: SafetyMode struct

* move to safetyMode

* fix typo

* this can be None

* fix potential empty pandaStates list

* fix thermald

* fix controlsd

* rename safetyModes to safetyConfigs

* update process_replay

* fix test_models

* bump cereal
2021-10-08 17:54:34 +02:00
Adeeb Shihadeh
86f68ff22e add TSS2 replay route (#21953)
* add TSS2 replay route

* skip on sim

* do not set on import time

* update ref

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-08-31 10:48:21 -07:00
Adeeb Shihadeh
f7dc898853 update prius test route (#21791) 2021-07-29 20:04:07 -07:00
Harald Schafer
1979005e82 update 2021-07-09 16:31:34 -07:00
HaraldSchafer
6838e1c82c Refactor lateral lag compensation (#21334)
* add T_IDXS

* refactor

* fix test

* unused

* typo

* needs casting

* Update selfdrive/controls/lib/drive_helpers.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* deprecate field

* regen all

* new segs

* add todo

* split back

* clean

* bad names

* do in controls

* add arg

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-06-30 14:19:39 -07:00
Adeeb Shihadeh
4a1aec27ef segment regen (#21148)
* start seg regen

* send vipc

* fix up some paths

* set fp

* fix no entries

* engages

* seperate camera procs

* send real frames

* regen test routes

* nice argparser

* fix valgrind test

* move that

* fix that
2021-06-28 21:44:40 -07:00
robbederks
cdfba105ed Tesla AP2 port (#20074)
* squashed the PR

* remove AP passthrough

* disengage on steer override

* bump panda

* add test routes

* bump panda

* tesla is still dashcam only

* dashcam only

* rerun CI

* add to release files
2021-06-23 14:56:59 +02:00
Adeeb Shihadeh
7d8480cffe Update VW process replay route (#21142) 2021-06-04 15:29:13 -07:00
Adeeb Shihadeh
c45dbbc737 fix process replay on local files (#20758) 2021-04-26 20:50:47 -07:00
Jason Young
48fa8ffb33 VW MQB: Update to Audi A3 Mk3 (#20489)
* Add A3 and A3 e-tron FPv2

* Update README

* Catch up with model renaming

* Trim duplicate FW values

* Add 2014 A3 (bankster)

* Update model-year range

* Revert more model name changes

* Revert more model name changes

* More FW for 2017 A3 e-tron (fastzeek)

* Update A3 models and packages in README

* Update A3 packages again

* AUDI_A3 -> AUDI_A3_MK3

* Update test route reference

* CAN valid should pass for VW now

* Update process_replay route

* Expand tested A3 range (we have known 2014-15,17)

* Revert "CAN valid should pass for VW now"

This reverts commit 73a4b4886f572045ada05cd13e60790ca3e17a94.

* revert route changes and update refs

* update refs again

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-03-30 14:49:46 -07:00
Adeeb Shihadeh
f3f44e43e6 VW: update process replay route (#20239)
* VW: update process replay route

* update refs
2021-03-04 14:58:28 -08:00
HaraldSchafer
c3707328f5 Different lag comp (#20024)
* more sensible

* different names for different segs

* new ref

* this is checked somewhere
2021-02-05 14:01:22 -08:00
HaraldSchafer
124100d0fa remove old model packet (#19769)
* remove publisher

* move to V2

* radard new model

* fix plant

* change packety

* need hack here too

* change to new

* this has been wrong all along

* no more model msg

* subscribe to new model

* not needed anymore

* make work

* need to ignore that too

* should pass tests, needs car test

* fix process replay

* no more poly

* update refs

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-01-19 17:15:16 -08:00
Adeeb Shihadeh
13ea3ccf6d update VW and nissan test routes 2021-01-19 14:22:50 -08:00
HaraldSchafer
a4098d7bb0 Update Subaru and GM test routes (#19828)
* 2 more

* update refs

* note?

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-01-18 15:54:02 -08:00
HaraldSchafer
158210cde8 Mpc rework2 (#19660)
* start again

* need that too

* this actually works

* not needed

* do properly

* still works

* still works

* still good

* all G without ll

* still works

* all still good

* cleanup building

* cleanup sconscript

* new lane planner

* how on earth is this silent too....

* update

* add rotation radius

* update

* pathplanner first pass

* misc fixes

* fix

* need deep_interp

* local again

* fix

* fix test

* very old

* new replay

* interp properly

* correct length

* another horrible silent bug

* like master

* fix that

* do doubles

* different delay compensation

* make robust to empty msg

* make pass with hack for now

* add some extra

* update ref for increased leg

* test cpu usage on this pr

* tiny bit faster

* purge numpy

* update ref

* not needed

* ready for merge

* try again after recompile

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-01-14 18:43:50 -08:00
HaraldSchafer
95eb5b6fcb Extra test segs (#19749)
* new routes

* update refs

* new ref
2021-01-13 21:56:29 -08:00
HaraldSchafer
307ba858db Test segs 08 (#19701)
* start updating routes

* update that too

* new ref commit

* try again
2021-01-08 16:39:47 -08:00
Adeeb Shihadeh
59d7c5e64a move to github container registry + CI speedup (#19621) 2020-12-31 14:57:01 -08:00
Adeeb Shihadeh
89d02dcf97 check for safety mode mismatch (#2443) 2020-11-04 11:12:15 -08:00
Adeeb Shihadeh
3795920750 update process replay test routes (#2482)
* update bosch route

* update rfs

* bump refs
2020-11-03 18:44:35 -08:00
Adeeb Shihadeh
516da927f9 add cars with LQR and INDI lat control to process replay (#2363)
* add cars that use LQR and INDI lat control to process replay

* bump ref commit
2020-10-19 21:18:50 -07:00
Adeeb Shihadeh
1e5a935ff0 paramsd process replay test (#2118)
* paramsd process replay test

* cleanup
2020-09-01 18:31:26 -07:00
HaraldSchafer
8e2d344135 Misc locationd improvements (#1714)
* I like this more

* rewind less

* bump rednose

* falling off windshield detectopr

* adjust thresholds

* this is a soft disable now

* move that

* process replay fixes

* update refs

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2020-07-30 15:33:22 -07:00
Willem Melching
5139b827af add tolerance to process replay compare (#1904) 2020-07-22 11:50:58 +02:00
Willem Melching
843a64c72f Make pylint more strict (#1626)
* make pylint more strict

* cleanup in progress

* done cleaning up

* no opendbc
2020-06-02 16:29:32 -07:00
Adeeb Shihadeh
27754a277c enable E261 in flake8: two spaces before inline comment 2020-05-31 14:07:29 -07:00
Adeeb
d9bf9f0a40 Enable more flake8 checks (#1602)
* enable some more flake8 checks

* some more quick ones

* bump opendbc

* e401

* e711 e712

* e115 e116

* e222

* e301

* remove that

* e129

* e701 e702

* e125 e131

* e227

* e306

* e262

* W503

* e713

* e704

* e731

* bump opendbc

* fix some e722
2020-05-30 20:14:58 -07:00
Jafar Al-Gharaibeh
34b9b5e849 Bounty: Mazda (#988)
* Mazda Port

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Update checksum to account for steer angle signal used in some cars

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Add test drive/segment ID to pass CI tests

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Process gear and brake signals, add new fingerprints

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Refactor some car interface code

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Drop redundant call, use Ecu, move warning to carstate, fix cruise speed

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Drop unused variables, cleanup handsoff tracking

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Update to steerError and steeringRate

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Refactor parse gear, update lkas signals from the new dbc

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Better tracking of engage and warning events

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* remove commented lines, update speed_kph to match panda

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Steer Error

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Set lkas enable speed to 52 kph

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Drop block signal use, fix LGTM alerts

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* When gas is pressed OP will disengage, sync local state with it

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Use car's speed signal instead of wheel speed

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Tidy up disengage events

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Rebase/Refactor with upstream

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Sync stock ACC state with OP

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* mazda dbc file renamed

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Improve acc stock and go by removing the 3 seconds limit

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Dashcam

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* PR Feedback

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* Send ACC cancel only if we are sure cruise is enabled

Otherwise we run the risk of disabling main cruise

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* updated route

Signed-off-by: Jafar Al-Gharaibeh <to.jafar@gmail.com>

* no process replay for now

* add to release files

* Revert "updated route"

This reverts commit 233db4f1bc2d8eefaa86488be4f6f88360d20793.

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2020-05-28 22:15:19 -07:00
Willem Melching
3d08dcc3b2 Run mypy commit hook (#1591)
* run mypy commit hook

* fix mypy errors
2020-05-28 15:05:04 -07:00
Willem Melching
71589fc92e Logreader can take car of downloading the files 2020-04-30 15:09:19 -07:00