Commit Graph

177 Commits

Author SHA1 Message Date
Shane Smiskol
5052b55c44 test_models: fuzz test panda and CarState (#30443)
* pre-hypothesis

* some hypothesis junk

* this kinda works but is really slow due to counter check

* choose addrs from fingerprint

* stash

* honda nidec brake pressed mismatches fixed

* bump panda

* stash

* tesla: use DI_torque2 (panda msg)

* run

* run

* ah this honda mismatch too

* no more multi can msgs

* clean up, remove old file

* add todo

* prob can remove urandom

* stash, huge examples

* fix pq standstill mismatch

* yuge

* yup there's a leak somewhere

* try to find leak

* skip dashcam (pq and tesla)

* PR comments

* bump

* draft stash

* fix alt brake hondas

* bump

* bump

* bump

* some clean up

* minor clean up

* more clean up

* stash

* fix honda bug

* more

* 100 examples

* revert tesla

* no memory leak any more?

* bring back tests with skips

* parameterize max_examples

* skip interceptor

* is jenkins on my branch?

* ooh that's fast

* 50 is not bad for GH CI

* 300 might be better with rest of test_models

* no more detection

* bump

* need CS_prev to catch bugs where openpilot changes and panda doesn't (eg. not setting interceptor safety mode)

* need to simplify all this

* need a warm up first, since some signals are 1 by default (toyota's gas_released!=1)

* changes

* set honda safety param

* set toyota safety param

* bump panda

* clean up honda

* rm interceptor

* thought interleaving addrs might help, but we can fine tune later

* Revert "thought interleaving addrs might help, but we can fine tune later"

This reverts commit 153301384b48c9f33f9e2af3c224241eaeec41c1.

* get size from dict

* what

* add nocapture marker

* clean up

* try to raise logging level

* need to run last as pytest_runtest_call, since it starts capturing

* get capman conditionally

* mark

* type fingerprint

* should use gen_empty_fingerprint

* no longer needed

* draft

* no longer need gc

* clean that up

* test everything!

* more clean up

* more

* no point

* fix that

* fix errors

* bump

* nice even 300 examples for 300 segs

* final bump :fingers_crossed:

* better import order

* remove debugging prints

* warm up kinda works

* Revert "warm up kinda works"

This reverts commit 7fc77b07d592edb13eadca77deb49540954a7d69.

* random seed

* revert

* strat

strat

* add expl comment

* cmt

* check controls allowed

* Revert "check controls allowed"

This reverts commit e82a0e5396810dd4670e6847aa555194a709e10f.

* not unittests

* run tests!

* run tests 2!

* run tests 3!

* seed unused

* revert

* add shrink phase, and remove health check suppression

* hello

* oncemore

* Update selfdrive/car/tests/test_models.py
2023-12-19 03:18:54 -06:00
Adeeb Shihadeh
7aeefaad65 jenkins: only do replay when relevant files change 2023-12-16 23:35:43 -08:00
Adeeb Shihadeh
6f143f6c45 full clean for onroad device 2023-12-14 23:00:07 -08:00
Adeeb Shihadeh
0830f62b36 also run qcomgpsd on path update 2023-12-14 20:44:14 -08:00
Adeeb Shihadeh
2cad6dfa58 jenkins: only run pandad tests when paths change (#30741)
* jenkins: only run pandad when paths changed

* fix

* ugh shadow
2023-12-14 20:41:11 -08:00
Adeeb Shihadeh
303f1cf9cf jenkins: run test_onroad in the normal env (#30740) 2023-12-14 19:59:16 -08:00
Justin Newberry
5d668701cb CI: move test_maprenderer to GH actions (#30610)
maprenderer to ci
2023-12-14 16:59:28 -08:00
Justin Newberry
02373fcb05 jenkins: retry docker_build (#30629)
* retry docker build

* retry docker build

* retry docker build

* does this work nicely?

* docker fail!

* Revert "docker fail!"

This reverts commit ab227a1ea37fad3d6413081e4516211b474b0712.
2023-12-12 18:12:19 -08:00
Justin Newberry
93cd19e3a5 jenkins: retry pc setup (#30659)
* retry pc setup

* sleep and retry

* exception

* throw e

* remove force exit

* make a function for it

* remove force exit

* move to top

* separate exception
2023-12-12 17:54:17 -08:00
Justin Newberry
c066fdd7a3 jenkins: remove old root user permission fixes (#30684)
* Batman

* add debug stuff

* Revert that for now

* remove debug stuff
2023-12-12 17:48:03 -08:00
Shane Smiskol
e78b80c8fe jenkins: run all car tests with one command (#30661)
* run all jenkins car tests in one script, less down time?

* fix

* speed up

* u
2023-12-08 22:15:27 -08:00
Justin Newberry
36bebb1aa0 jenkins: retry build_devel (#30628)
* will it work

* retry build devel

* correct p[ath

* try a failure

* Revert "try a failure"

This reverts commit ba4e6a0a2096b94887fcfbf7d011e34aa2a16c41.
2023-12-06 17:22:32 -08:00
Justin Newberry
f16df8e4b7 pytest: group test_models by route instead of car name (#30625)
* group by route

* dont need those
2023-12-06 16:00:59 -08:00
Justin Newberry
3777bf7fb9 jenkins: disable xdist on tici (#30624)
to preserve it
2023-12-06 15:51:29 -08:00
Justin Newberry
31ab43ce41 jenkins: remove pytest tici conf (#30621)
* remove pytest conf

* remove that too
2023-12-06 11:13:22 -08:00
Justin Newberry
5030824572 jenkins auto cpu count (#30544)
* auto cpu

* build too

* need bash
2023-11-28 14:34:00 -08:00
Adeeb Shihadeh
4cae08e636 rename rawgpsd to qcomgpsd (#30493)
* rename rawgpsd to qcomgpsd

* fix more paths

* that moved

* forgot the d
2023-11-18 23:23:16 -08:00
Justin Newberry
2ad82cbfb0 Pytest: default settings on PC improvements (#30441)
* always multithreaded

* multithreaded

* pkill correct process

* escape more

* load for interfaces
2023-11-13 10:45:24 -08:00
Adeeb Shihadeh
5f7143df02 jenkins: use build.py to manage cache size (#30440)
* jenkins: use build.py to manage cache size

* label

* set pp

* double

* cleanup

* non mimimal build on PC

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
2023-11-10 16:55:25 -08:00
Adeeb Shihadeh
57c8304c25 jenkins: retry on failed git checkouts (#30345)
* jenkins: retry on failed git checkouts

* test

* Revert "test"

This reverts commit ea57ba025630eb3105a9ccdce3bdc17f72678c72.
2023-10-30 09:58:57 -07:00
Adeeb Shihadeh
13d780a5f2 test_onroad: log sizes check (#30265)
* test_onroad: check log sizes

* fix

* check logs

* rb

* len

* pytest

* mb

* corolla log is small

* adjust qlog
2023-10-18 19:44:46 -07:00
Adeeb Shihadeh
71a9b2c24c jenkins: clear commacache dir (#30290) 2023-10-18 19:37:29 -07:00
Justin Newberry
8b2b72499f CI: run jenkins as batman (#30184)
* I'm batman

* chown batman

* sudo

* add batman to sudo group

* try without cleanup

* Revert "try without cleanup"

This reverts commit f59554f2f373417f2c59fd250463eca2fda4d49e.

* comment
2023-10-06 11:12:41 -07:00
Adeeb Shihadeh
d036148550 jenkins speedup (#29869)
* jenkins speedup

* cleanup
2023-09-10 16:47:07 -07:00
Adeeb Shihadeh
a411af4f02 jenkins scripted pipeline (#29865)
* scripted

* exclude branches

* try that

* fix credentials

* fix

* tag with commit
2023-09-09 21:30:21 -07:00
Adeeb Shihadeh
947abbcba0 fixup power draw test output (#29844)
* fixup power draw test

* no pytest
2023-09-08 13:01:04 -07:00
Shane Smiskol
821cfff56d car interfaces: long running fuzzy test (#29592)
* draft

* draft 2

* increase deadline and run

* imports

* run -> use
2023-08-24 10:45:50 -07:00
Shane Smiskol
ac3f87bad5 internal test_models: run on more segments (#29443)
100 more
2023-08-17 00:25:14 -07:00
Shane Smiskol
fb235238ad jenkins: re-enable test_models (#29394)
* re-enable test_models

* now?

* Remove bad Volt segment

* Update Jenkinsfile

* use first n segments
2023-08-14 16:08:34 -07:00
Adeeb Shihadeh
de66ba0187 jenkins: remove test that's now run in Actions 2023-08-13 21:21:54 -07:00
Adeeb Shihadeh
09ffce8073 add small ui for jenkins runners (#29382) 2023-08-13 18:01:51 -07:00
Adeeb Shihadeh
b11b14aff8 CI: enable PYTHONWARNINGS=error (#29364)
* CI: enable PYTHONWARNINGS=error

* fixes

* more ignore

* more modeld ignore
2023-08-12 17:15:16 -07:00
Shane Smiskol
70b1e9dd75 test_models: run on route list (#26226)
* move to openpilot

* draft

* support internal urls

* not used

* update seg list

* regen segment list

* regen seg list

* regen

* regen

* regen

* no dirty segments

* regen seg list with no fuzzy or fixed fp

* regen segments with hda2 exception

* regen with added filters

* regen without bad dongle

* regen

* regenerate!

* regenerate, only OP enabled

* regen

* regenerate!

* regenerate!

* stash

* use SegmentName

* new route list

* add temp comment

* remove comment

* let's see if this works

* comment out other tests to speed up

* fix agent

* ?

* hmm

* oh whoops...

* add matrix

* fix

* how many levels do we need?

* fix values

* matrix can't be in parallel

* how about this

* try this

* Revert matrix

Revert "try this"

This reverts commit 8d0d91fd70c467c1fbb4e4e9aed511d15b2a30ef.

Revert "how about this"

This reverts commit a8e4fc53234824e77cbfb1e471277bc033e9dea1.

Revert "matrix can't be in parallel"

This reverts commit daaa6fcc3c75c74cbb90e97c565099a94123994c.

Revert "fix values"

This reverts commit df554b6a3371d124a574eb8d26bc51ef5b5b8fde.

Revert "how many levels do we need?"

This reverts commit 1a17320fa1c5e7220ef60e29981bbb3bb7da16c6.

Revert "fix"

This reverts commit e7eb6e404358fbd2eac3fea1901a8d30ea92d729.

Revert "add matrix"

This reverts commit a1b57e5725417d3c2f639f8edfc0c889b84b6753.

* use pytest instead! (5 jobs is ~150 mins)

* split lines, uncomment

* This Sienna seg has a fault SDSU that stopped forwarding/sending msgs

* picked a route with no PSCMStatus and no panda errors, recent routes are working

* this cadillac was dashcammed (no radar)

* opened an issue for this, it's 'expected' right now

* small clean up

* small clean up

* i don't think that worked

* is this needed?

* add to new PC tests

* cache

* draft

* Revert "draft"

This reverts commit 3b7f740dd4883118747300bc3687074c2d3c2116.

* probably should be function

* draft

* clean up

* add todo

* 600 random segments

* debug

* does this fix pythonpath issues?

fix

* try this

* mount?

* pytest again!

* no need for PYTHONPATH now

* Update Jenkinsfile

* ?

* convention

* clean up

* would be even more complex (have to unset ci which is class level)

* track

* is lfs pulled at all?

* ah no it's not
2023-08-10 00:45:23 -07:00
Adeeb Shihadeh
7f02cb9faf jenkins: rm tools copy in release tests (#29273)
* jenkins: rm tools copy in release tests

* rm that too
2023-08-07 19:14:45 -07:00
Adeeb Shihadeh
036888ec15 jenkins: reset USB (#29155)
* jenkins: reset USB

* fix non aux
2023-07-26 15:11:20 -07:00
Adeeb Shihadeh
6b83724c50 jenkins: run tests with pytest (#29143)
* jenkins: run tests with pytest

* typo

* fix
2023-07-25 17:06:43 -07:00
Adeeb Shihadeh
5fb2f7072b rawgpsd: speedup tests (#29110)
* rawgpsd: speedup tests

* fix that one

* little more

* prevent subtle bugs

---------

Co-authored-by: Comma Device <device@comma.ai>
2023-07-23 15:38:47 -07:00
Adeeb Shihadeh
d92c9aef23 tici: speedup power save setting (#29100)
* speedup

* update type

---------

Co-authored-by: Comma Device <device@comma.ai>
2023-07-22 17:15:58 -07:00
Maxime Desroches
97eec45d42 Test scons multithreaded random build (#29082)
* random build

* name

* 2 pass

* clean after

* fail on this

* fix deps

* format

* var

* try without lfs
2023-07-21 21:57:27 -07:00
Adeeb Shihadeh
e8b20b7f81 CI: fix no DNS after tests killed (#29090)
* CI: fix no DNS after tests killed

* fix
2023-07-21 18:35:29 -07:00
Adeeb Shihadeh
e2b5d1e29a mapsd: rendering tests (#28981)
* mapsd: rendering tests

* run

* speedup
2023-07-16 14:39:34 -07:00
Adeeb Shihadeh
fdf1c328b9 jenkins: speedup build stage (#28687)
* jenkins: speedup build stage

* run in gh actions

* skip

* set that

* build

* python path

* cleanup
2023-06-24 22:33:06 -07:00
Adeeb Shihadeh
5992bf4acc rawgpsd tests (#28683)
* rawgpsd tests

* test xtra

* check freq

* fix

* restart lte

---------

Co-authored-by: Comma Device <device@comma.ai>
2023-06-24 20:04:58 -07:00
Adeeb Shihadeh
16e3d4f69f jenkins: new replay devices (#28643)
* replay

* restart nm
2023-06-22 14:21:11 -07:00
Adeeb Shihadeh
6d91b7c341 jenkins: ensure panda is in good state (#28414) 2023-06-05 15:28:38 -07:00
Mitchell Goff
2d34083cd8 Test navmodel (#28272)
* Test navmodel

* removed comment

* Updated model_replay ref

* read tiles from azure bucket

* updated ref commit
2023-05-30 20:33:01 -07:00
Adeeb Shihadeh
2664dba52e jenkins: switch order of tests 2023-05-22 00:48:51 -07:00
Adeeb Shihadeh
39c5d492bf time to onroad test (#28242)
* time to onroad test

* run few times

* test on stripped builds

* manager

* cleanup

* split those out

* run first

---------

Co-authored-by: Comma Device <device@comma.ai>
2023-05-20 17:59:41 -07:00
Adeeb Shihadeh
c4501e4bc1 controlsd: remove procs from ignore list (#26848)
* trim list

* check for file
2023-05-07 15:43:45 -07:00
Adeeb Shihadeh
d5a0a4b3e8 Jenkins tuneup (#28095)
* tuneup

* use service messages

* logmono

* not passive
2023-05-03 16:52:20 -07:00