Commit Graph

98 Commits

Author SHA1 Message Date
cl0cks4fe
dd9d5d4528 Unittest to pytest (#32366)
* add pytest-asyncio

* switch common

* switch selfdrive

* switch system

* switch tools

* small fixes

* fix setUp and valgrind pytest

* switch to setup

* fix random

* switch mock

* switch test_lateral_limits

* revert test_ui

* fix poetry.lock

* add unittest to banned-api

* add inline ignores to remaining unittest imports

* revert test_models

* revert check_can_parser_performance

* one more skip

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2024-05-17 11:01:44 -07:00
Shane Smiskol
d0d44a51a0 card: create pedal pressed event (#32417)
* card: create pedal pressed event (#32393)

* move pedalPressed to card

* rm

* needs to be a builder

* move these

* clean up

* reader later

* Update ref_commit

* moved to card
2024-05-13 16:18:55 -07:00
Shane Smiskol
e5107b1eab debug test_models script fixes (#32362)
* some things were renamed without checking

* fix that
2024-05-06 15:01:06 -07:00
Cameron Clough
90ff0dd047 test_models: migrate fingerprint (#31936) 2024-03-20 10:57:28 -07:00
Justin Newberry
b0eb3ba4f4 cars: platform enums -> prepend brand name (#31927)
* with brand name

* migrate

* Fix

* fixes

* more

* passes

* fix

* fix the doc

* collects

* these too

* more stuff

* body exception :/

* more

* hardcode i guess

* update ref

* toyota

* more toyota

* and here

* final!

* fix notebooks and ccs

* move this here
2024-03-19 17:29:50 -07:00
Justin Newberry
ca5a2ed942 move getting platform to get_params (#31871)
* better

* string

* not here
2024-03-14 21:58:25 -07:00
Justin Newberry
a919d27afc fix car test routes typing (#31773)
* Fix typing

* and fix test_car_model

* fix
2024-03-07 11:33:40 -08:00
Justin Newberry
5734d7c2bf Honda: move to platformconfig (#31637)
* first

* Fixes

* not

* not

* Fixes

* fix

* cleanup + use sets for now

* update ref

* update ref
2024-02-29 11:29:40 -08:00
Cameron Clough
995250ae49 use pyupgrade to update to new typing syntax (#31580)
* add pyupgrade hook

* run pyupgrade (pre-commit run -a)

* ruff --fix

* Revert "add pyupgrade hook"

This reverts commit 56ec18bb6b8602a0b612f3803d96cdad14b52066.

* revert changes to third_party/

* manual type fixes

* explicit Optional wrapping capnp objects
2024-02-24 16:41:23 -08:00
Justin Newberry
1161d33c18 Subaru: move carspecs to PlatformConfig (#31550)
* subaru platform config

* forester wrong dbc

* spacing

* subaru car specs

* someday!

* more red diff

* all brands can be done like this

* but this should be done first and thats subaru specific

* that seems very low but we shouldn't change it here

* as long as it subclasses str its fine

* fix
2024-02-23 13:23:40 -08:00
Adeeb Shihadeh
6f905ed979 radard: remove sleep for cars without radar (#31467)
* radard: remove sleep for cars without radar

* fix

* update refs
2024-02-14 13:12:54 -08:00
Shane Smiskol
9479686fa3 test models: more examples in CI (#31444)
* more examples

* how is it so fast?!

* 500

* is there a hypothesis cache?

all

* Revert "is there a hypothesis cache?"

This reverts commit e628ba73d5b639e36f95df0ab9049697b34a58b8.

* same as jenkins
2024-02-13 23:00:44 -08:00
Justin Newberry
66dc7030de test_models: cleanup by using new logreader (#31267)
* cleanup

* only check error

* this segment only

* fix

* fix

* keep this

* fix internal
2024-02-06 16:50:30 -08:00
Justin Newberry
caa9153974 CommaCarSegments: public database of segments for each platform (#31114)
* comma car segments

* comma car segments for test_models

* oneliner

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* better name

* not used here

* sort

* remove print

* better comment

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2024-01-22 17:33:03 -08:00
Justin Newberry
3846130d8e pytest: add marker for sharing the download cache (#31082)
* fix cache

* with a marker
2024-01-22 16:30:59 -08:00
Justin Newberry
2c86c023fe sanitizer: sanitize vin number (#31090) 2024-01-21 12:22:32 -05:00
Justin Newberry
5c24527683 test_models: sanitize segment before running test (#31026)
to ensure sanetized routes will work
2024-01-16 15:08:44 -08:00
Justin Newberry
7d400112df LogReader: move opci to tools/lib (#31021)
move opci
2024-01-16 10:58:51 -08:00
Justin Newberry
1148c65d23 add read_seglist helper (#30908)
add helper
2024-01-04 17:52:07 -08:00
Shane Smiskol
7f398e1cf9 test_models: fallback to public route when route isn't uploaded to CI bucket (#30893)
* not sure why this is a test

* Revert "not sure why this is a test"

This reverts commit c13e16311eb882df65d216bbd59acb5702e985e6.

* no ruff for this pattern :(

* this is always caught

* should fail

* simpler

* Revert "should fail"

This reverts commit 21bd15275c46cd42fd19b058245334bae83cce77.

* fix

* cleanup this

* better comment

* better

* this can be more localized

* better names

* better names

* should fail

* Revert "should fail"

This reverts commit c58495e8b3417ce351df8048752ec60177827cb3.

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
2024-01-03 11:54:22 -08:00
Shane Smiskol
8c4930d2d3 Revert "test_models: fallback to public route when route isn't uploaded to CI bucket (#30794)" (#30896)
* fails if no skip function

* Revert "fails if no skip function"

This reverts commit ff3d697d38dce56057c83f06033a689f2301a29d.

* Revert "test_models: fallback to public route when route isn't uploaded to CI bucket (#30794)"

This reverts commit 05dc0b51e6.
2024-01-02 17:22:06 -08:00
Justin Newberry
05dc0b51e6 test_models: fallback to public route when route isn't uploaded to CI bucket (#30794)
* wip

* add public comment

* fix that

* wip

* retrigger
2023-12-20 11:29:19 -08:00
Shane Smiskol
5052b55c44 test_models: fuzz test panda and CarState (#30443)
* pre-hypothesis

* some hypothesis junk

* this kinda works but is really slow due to counter check

* choose addrs from fingerprint

* stash

* honda nidec brake pressed mismatches fixed

* bump panda

* stash

* tesla: use DI_torque2 (panda msg)

* run

* run

* ah this honda mismatch too

* no more multi can msgs

* clean up, remove old file

* add todo

* prob can remove urandom

* stash, huge examples

* fix pq standstill mismatch

* yuge

* yup there's a leak somewhere

* try to find leak

* skip dashcam (pq and tesla)

* PR comments

* bump

* draft stash

* fix alt brake hondas

* bump

* bump

* bump

* some clean up

* minor clean up

* more clean up

* stash

* fix honda bug

* more

* 100 examples

* revert tesla

* no memory leak any more?

* bring back tests with skips

* parameterize max_examples

* skip interceptor

* is jenkins on my branch?

* ooh that's fast

* 50 is not bad for GH CI

* 300 might be better with rest of test_models

* no more detection

* bump

* need CS_prev to catch bugs where openpilot changes and panda doesn't (eg. not setting interceptor safety mode)

* need to simplify all this

* need a warm up first, since some signals are 1 by default (toyota's gas_released!=1)

* changes

* set honda safety param

* set toyota safety param

* bump panda

* clean up honda

* rm interceptor

* thought interleaving addrs might help, but we can fine tune later

* Revert "thought interleaving addrs might help, but we can fine tune later"

This reverts commit 153301384b48c9f33f9e2af3c224241eaeec41c1.

* get size from dict

* what

* add nocapture marker

* clean up

* try to raise logging level

* need to run last as pytest_runtest_call, since it starts capturing

* get capman conditionally

* mark

* type fingerprint

* should use gen_empty_fingerprint

* no longer needed

* draft

* no longer need gc

* clean that up

* test everything!

* more clean up

* more

* no point

* fix that

* fix errors

* bump

* nice even 300 examples for 300 segs

* final bump :fingers_crossed:

* better import order

* remove debugging prints

* warm up kinda works

* Revert "warm up kinda works"

This reverts commit 7fc77b07d592edb13eadca77deb49540954a7d69.

* random seed

* revert

* strat

strat

* add expl comment

* cmt

* check controls allowed

* Revert "check controls allowed"

This reverts commit e82a0e5396810dd4670e6847aa555194a709e10f.

* not unittests

* run tests!

* run tests 2!

* run tests 3!

* seed unused

* revert

* add shrink phase, and remove health check suppression

* hello

* oncemore

* Update selfdrive/car/tests/test_models.py
2023-12-19 03:18:54 -06:00
Shane Smiskol
d9a8846d58 test_models: use gen_empty_fingerprint (#30752)
use gen_empty_fingerprint
2023-12-15 17:16:19 -08:00
Shane Smiskol
123ef46af3 Revert "test_models: check consitency between panda and openpilot for velocity (#30750)" (#30753)
This reverts commit 1ca1cf3922.
2023-12-15 17:16:01 -08:00
Justin Newberry
1ca1cf3922 test_models: check consitency between panda and openpilot for velocity (#30750)
* sanity check on velocity

* import from there

* use raw

* just check last

* ignore body

* reduce tolerance and bump panda

* its not average anymore

* slightly higher tolerance
2023-12-15 14:50:37 -08:00
Shane Smiskol
ac83318ac4 test_models: randomize internal segment list (#30653)
* randomize internal seg list segments

* random

* pytest-randomly sets random.seed to a consistent value for all the workers/processes

* noeol

* update

* Revert "update"

This reverts commit aff9a69c4e5e3934deebaa33986b42f44b55b002.

* lock

* don't randomize by default

* remove random-order

* strict

* random

* one fix

* test

* does nothing

* rm tests

* Revert "rm tests"

This reverts commit b548e3fcd48e60538695506888d863c01b459d27.

* (can't repro locally) just athena should be fine

* bs1

* bs2

* bs3

* bs4

* bs5

* wrong way

* no controls

* no car

* no board

* controls?

* crazy -common

* Revert "crazy -common"

This reverts commit 02365d712b3d09cab1893cce2261a4b418bb3851.

* test athena

* test athena 2

* test athena 3

* test athena 4

* test athena 5

* test athena 6

* test athena 7

* test athena 8

* test athena 9

* ??

* in one commit

* common?

* car and board

* -controls -board

* random-order

* no board

* revert

* car/tests

* least likely

* try

* try 2

* draft

* draft

* so much better

* cmt

* use randomly

* not needed here

* directly modify option works

* bb

* test time

* Revert "test time"

This reverts commit 2c5caabe2b470b47b7322e37800680b92773fccc.

* tmut

* i concur

* revert
2023-12-09 14:28:32 -08:00
Justin Newberry
f16df8e4b7 pytest: group test_models by route instead of car name (#30625)
* group by route

* dont need those
2023-12-06 16:00:59 -08:00
Adeeb Shihadeh
613ccf4b74 falsify rx checks (#30564) 2023-11-30 14:53:33 -08:00
Adeeb Shihadeh
afd61352d0 bump panda (#30501)
* bump panda

* bump

* bump

* fix renamed variables

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2023-11-20 14:45:51 -08:00
Shane Smiskol
fc8cfe1280 test_models: check if in relay-breaking safety mode (#30476)
* remove initData, remove tolerance

* can check faster now

* don't
2023-11-15 18:57:37 -08:00
Shane Smiskol
e44d84a7f9 test_models: remove ignored address check list (#30475)
* remove ignore_addr_checks_valid

* lessgoooo
2023-11-15 18:50:39 -08:00
Justin Newberry
31c682d407 Pytest: load by group by default (#30466)
* wip

* use a class marker

* this one should be grouped too
2023-11-14 17:33:47 -08:00
Justin Newberry
0251742079 CI: use "slow" marker for pytest (#30063)
* skip to mark

* fix quotes
2023-09-26 13:44:18 -07:00
Shane Smiskol
1410a1150c test_models: check controlsd events (#29907)
* test_moedls: check controlsd events

* clean up

* sort

* controlsd modifies CP

* fix

* can use any segment now

* final fix

* use constant
2023-09-14 20:20:49 -07:00
Lee Jong Mun
238f9d65db cleanup remnants of LQR and INDI tuning (#29860)
* not use 'indi' cleanup

* bump cereal

* rm more

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2023-09-10 16:30:13 -07:00
Shane Smiskol
5d1550de87 test_models: detect when OBD multiplexing/fingerprinting (#29759)
* wait to leave elm mode before checking relay malfunction

* clean up

* clean up

* pycharm is best

* clean up

* more

* use pandaStatesDEPRECATED

* fix

* use constant

* only radar

* cmt

* set inline

* comment and re-arrange

* enable for relay malfunc check

* down here

* Update selfdrive/car/tests/test_models.py

* rm

* don't affect other tests

* up here

* one line

* update cmt

* no tolerance

* Revert "no tolerance"

This reverts commit 41b1c7e8beda87a878c3bb1f37ec96b256965966.

* comment is real solution

---------

Co-authored-by: Kacper Rączy <gfw.kra@gmail.com>
2023-09-05 21:22:36 -07:00
Justin Newberry
4c039ab022 Tests: disable laika tests in CI (#29742)
skip laika
2023-08-31 17:17:07 -07:00
Justin Newberry
1ee6ed4b16 Linter: remove pylint (#29611)
remove pylint
2023-08-24 16:30:54 -07:00
Adeeb Shihadeh
a9626f95b6 add openpilot prefix to imports (#29498)
* add openpilot prefix to imports

* more

* more

* fix docs

* fix linter

* bump submodules

* fix patched tests

* update dynamic imports

* debug

* Revert "debug"

This reverts commit db5e13b9911cc74438bee123bc3430da6c31b24b.

* fix pm test
2023-08-20 20:49:55 -07:00
Shane Smiskol
3563292e65 test_models: remove body standstill exception (#29453)
* remove body exception

* bump to master
2023-08-17 16:26:30 -07:00
Shane Smiskol
9d0d59ac9b Hyundai: match panda standstill check (#29442)
* bump panda

* clean up

* common?

* Revert "common?" (not now)

This reverts commit 197af7cdbb6a8c8e58fcbccca94f7db004ce91e4.

* fix

* no failures!

* remove time

* bump

* bump

* undo
2023-08-17 00:06:38 -07:00
Shane Smiskol
2d636deb4e test_models: check panda cruise state (#29423)
* cruise state test

* add todo

* rename

* clean up

* remove
2023-08-15 23:55:53 -07:00
Shane Smiskol
e2910d0720 CANParser: perf benchmark on route script (#29324)
* timing test

* test

* fix

* print

* loop

* clean up

* wider range

* Update selfdrive/car/tests/test_models.py

* Apply suggestions from code review

* run many times

* Update selfdrive/car/tests/test_models.py

* run for many its

* run with unittest and print as list

* Update .github/workflows/selfdrive_tests.yaml

* Update .github/workflows/selfdrive_tests.yaml

* total time is super inconsistent (body)

* Update selfdrive/car/tests/test_models.py

* clean up

* clean up

* clean up

* this works!

* draft

* test suite not as modular

* try something like this

* can do kb, but not too representative

* clean up

* remove kb? it depends on signals

* clean up

* more clean up

* rename

* just measure all CANParsers

* can do all this manually

* but this is way simpler

* comment

* fix mypy

* this is correct too

* 3 seconds instead of 1.5
2023-08-15 20:06:15 -07:00
Shane Smiskol
6ec00f5ee2 test_models: more typing (#29420)
* not type checked, but we set this in test_car_model

* type this
2023-08-15 20:01:33 -07:00
Shane Smiskol
936f8b9817 test_models: some typing (#29413)
* type

* also type check this func

* unused
2023-08-15 15:25:36 -07:00
Shane Smiskol
fb235238ad jenkins: re-enable test_models (#29394)
* re-enable test_models

* now?

* Remove bad Volt segment

* Update Jenkinsfile

* use first n segments
2023-08-14 16:08:34 -07:00
Shane Smiskol
001d4e8729 test_models: match online fingerprinting behavior (#29306)
* draft

* 1x

* clean up

* final clean up

* fix

* self explan

* space

* space
2023-08-11 18:33:11 -07:00
Shane Smiskol
1099b3486a test_models: check relay malfunction (#29270)
* check relay malfunction in test_models

* draft

* actual relay malfunction (or ASCM was turned on)

* rav4: not dashcam and op enabled, but we're not sending anything

* clean up

* spaces

* more space

* flip

* better?

* combine

* for

* move up
2023-08-11 18:10:27 -07:00
Shane Smiskol
70b1e9dd75 test_models: run on route list (#26226)
* move to openpilot

* draft

* support internal urls

* not used

* update seg list

* regen segment list

* regen seg list

* regen

* regen

* regen

* no dirty segments

* regen seg list with no fuzzy or fixed fp

* regen segments with hda2 exception

* regen with added filters

* regen without bad dongle

* regen

* regenerate!

* regenerate, only OP enabled

* regen

* regenerate!

* regenerate!

* stash

* use SegmentName

* new route list

* add temp comment

* remove comment

* let's see if this works

* comment out other tests to speed up

* fix agent

* ?

* hmm

* oh whoops...

* add matrix

* fix

* how many levels do we need?

* fix values

* matrix can't be in parallel

* how about this

* try this

* Revert matrix

Revert "try this"

This reverts commit 8d0d91fd70c467c1fbb4e4e9aed511d15b2a30ef.

Revert "how about this"

This reverts commit a8e4fc53234824e77cbfb1e471277bc033e9dea1.

Revert "matrix can't be in parallel"

This reverts commit daaa6fcc3c75c74cbb90e97c565099a94123994c.

Revert "fix values"

This reverts commit df554b6a3371d124a574eb8d26bc51ef5b5b8fde.

Revert "how many levels do we need?"

This reverts commit 1a17320fa1c5e7220ef60e29981bbb3bb7da16c6.

Revert "fix"

This reverts commit e7eb6e404358fbd2eac3fea1901a8d30ea92d729.

Revert "add matrix"

This reverts commit a1b57e5725417d3c2f639f8edfc0c889b84b6753.

* use pytest instead! (5 jobs is ~150 mins)

* split lines, uncomment

* This Sienna seg has a fault SDSU that stopped forwarding/sending msgs

* picked a route with no PSCMStatus and no panda errors, recent routes are working

* this cadillac was dashcammed (no radar)

* opened an issue for this, it's 'expected' right now

* small clean up

* small clean up

* i don't think that worked

* is this needed?

* add to new PC tests

* cache

* draft

* Revert "draft"

This reverts commit 3b7f740dd4883118747300bc3687074c2d3c2116.

* probably should be function

* draft

* clean up

* add todo

* 600 random segments

* debug

* does this fix pythonpath issues?

fix

* try this

* mount?

* pytest again!

* no need for PYTHONPATH now

* Update Jenkinsfile

* ?

* convention

* clean up

* would be even more complex (have to unset ci which is class level)

* track

* is lfs pulled at all?

* ah no it's not
2023-08-10 00:45:23 -07:00