Commit Graph

177 Commits

Author SHA1 Message Date
Shane Smiskol
f65ccca8e0 test_models: fuzz test panda and CarState (#30443)
* pre-hypothesis

* some hypothesis junk

* this kinda works but is really slow due to counter check

* choose addrs from fingerprint

* stash

* honda nidec brake pressed mismatches fixed

* bump panda

* stash

* tesla: use DI_torque2 (panda msg)

* run

* run

* ah this honda mismatch too

* no more multi can msgs

* clean up, remove old file

* add todo

* prob can remove urandom

* stash, huge examples

* fix pq standstill mismatch

* yuge

* yup there's a leak somewhere

* try to find leak

* skip dashcam (pq and tesla)

* PR comments

* bump

* draft stash

* fix alt brake hondas

* bump

* bump

* bump

* some clean up

* minor clean up

* more clean up

* stash

* fix honda bug

* more

* 100 examples

* revert tesla

* no memory leak any more?

* bring back tests with skips

* parameterize max_examples

* skip interceptor

* is jenkins on my branch?

* ooh that's fast

* 50 is not bad for GH CI

* 300 might be better with rest of test_models

* no more detection

* bump

* need CS_prev to catch bugs where openpilot changes and panda doesn't (eg. not setting interceptor safety mode)

* need to simplify all this

* need a warm up first, since some signals are 1 by default (toyota's gas_released!=1)

* changes

* set honda safety param

* set toyota safety param

* bump panda

* clean up honda

* rm interceptor

* thought interleaving addrs might help, but we can fine tune later

* Revert "thought interleaving addrs might help, but we can fine tune later"

This reverts commit 153301384b48c9f33f9e2af3c224241eaeec41c1.

* get size from dict

* what

* add nocapture marker

* clean up

* try to raise logging level

* need to run last as pytest_runtest_call, since it starts capturing

* get capman conditionally

* mark

* type fingerprint

* should use gen_empty_fingerprint

* no longer needed

* draft

* no longer need gc

* clean that up

* test everything!

* more clean up

* more

* no point

* fix that

* fix errors

* bump

* nice even 300 examples for 300 segs

* final bump :fingers_crossed:

* better import order

* remove debugging prints

* warm up kinda works

* Revert "warm up kinda works"

This reverts commit 7fc77b07d592edb13eadca77deb49540954a7d69.

* random seed

* revert

* strat

strat

* add expl comment

* cmt

* check controls allowed

* Revert "check controls allowed"

This reverts commit e82a0e5396810dd4670e6847aa555194a709e10f.

* not unittests

* run tests!

* run tests 2!

* run tests 3!

* seed unused

* revert

* add shrink phase, and remove health check suppression

* hello

* oncemore

* Update selfdrive/car/tests/test_models.py
old-commit-hash: 5052b55c440f3a778f9db9c854fa385077dd7ab9
2023-12-19 03:18:54 -06:00
Adeeb Shihadeh
6f13688747 jenkins: only do replay when relevant files change
old-commit-hash: 7aeefaad65b1d31789031a904a0a5e66180537af
2023-12-16 23:35:43 -08:00
Adeeb Shihadeh
030614e180 full clean for onroad device
old-commit-hash: 6f143f6c45b04702de24371db2d836210be9b860
2023-12-14 23:00:07 -08:00
Adeeb Shihadeh
129c2b9f16 also run qcomgpsd on path update
old-commit-hash: 0830f62b36fd4e802ea6f1a74062ae9e8bd7ecd1
2023-12-14 20:44:14 -08:00
Adeeb Shihadeh
71d4a05d86 jenkins: only run pandad tests when paths change (#30741)
* jenkins: only run pandad when paths changed

* fix

* ugh shadow
old-commit-hash: 2cad6dfa58f58af09ff70c70ad557fdf761b9293
2023-12-14 20:41:11 -08:00
Adeeb Shihadeh
2a08d484a1 jenkins: run test_onroad in the normal env (#30740)
old-commit-hash: 303f1cf9cf2f0c933fca103f104b382523db6cf2
2023-12-14 19:59:16 -08:00
Justin Newberry
0939b25bc4 CI: move test_maprenderer to GH actions (#30610)
maprenderer to ci
old-commit-hash: 5d668701cb69c1e3d170109a8e788a2acbacebd3
2023-12-14 16:59:28 -08:00
Justin Newberry
a20c934d2f jenkins: retry docker_build (#30629)
* retry docker build

* retry docker build

* retry docker build

* does this work nicely?

* docker fail!

* Revert "docker fail!"

This reverts commit ab227a1ea37fad3d6413081e4516211b474b0712.
old-commit-hash: 02373fcb05e3c5fcb3fe2596024bca02fa7f8ab8
2023-12-12 18:12:19 -08:00
Justin Newberry
758426375d jenkins: retry pc setup (#30659)
* retry pc setup

* sleep and retry

* exception

* throw e

* remove force exit

* make a function for it

* remove force exit

* move to top

* separate exception
old-commit-hash: 93cd19e3a553d34d2f8bcd1aea121d09a7b2196b
2023-12-12 17:54:17 -08:00
Justin Newberry
60d5e98920 jenkins: remove old root user permission fixes (#30684)
* Batman

* add debug stuff

* Revert that for now

* remove debug stuff
old-commit-hash: c066fdd7a32b2d5b419419ac3f19ab4e8ba34280
2023-12-12 17:48:03 -08:00
Shane Smiskol
930ff9a0c1 jenkins: run all car tests with one command (#30661)
* run all jenkins car tests in one script, less down time?

* fix

* speed up

* u
old-commit-hash: e78b80c8fe4f1cf6e6350c9f613ba8c9a43e9d57
2023-12-08 22:15:27 -08:00
Justin Newberry
12da944f1c jenkins: retry build_devel (#30628)
* will it work

* retry build devel

* correct p[ath

* try a failure

* Revert "try a failure"

This reverts commit ba4e6a0a2096b94887fcfbf7d011e34aa2a16c41.
old-commit-hash: 36bebb1aa064b4b40580aa48d35b21bd9ebe14c2
2023-12-06 17:22:32 -08:00
Justin Newberry
62d0647ab3 pytest: group test_models by route instead of car name (#30625)
* group by route

* dont need those
old-commit-hash: f16df8e4b7901f5e38b6f90a8900eef02faab504
2023-12-06 16:00:59 -08:00
Justin Newberry
a8c97eec3f jenkins: disable xdist on tici (#30624)
to preserve it
old-commit-hash: 3777bf7fb9b6d63f49118d010ac13cee89d13d06
2023-12-06 15:51:29 -08:00
Justin Newberry
f945b8ebaa jenkins: remove pytest tici conf (#30621)
* remove pytest conf

* remove that too
old-commit-hash: 31ab43ce41d6d0f88cb8a9b6df8e79f23b6309fe
2023-12-06 11:13:22 -08:00
Justin Newberry
31eede7722 jenkins auto cpu count (#30544)
* auto cpu

* build too

* need bash
old-commit-hash: 5030824572c90862dbe0acca2b095b3899311e39
2023-11-28 14:34:00 -08:00
Adeeb Shihadeh
ca85e5b82a rename rawgpsd to qcomgpsd (#30493)
* rename rawgpsd to qcomgpsd

* fix more paths

* that moved

* forgot the d
old-commit-hash: 4cae08e6369c0f63229222c0b3815da983928787
2023-11-18 23:23:16 -08:00
Justin Newberry
efaa3ea500 Pytest: default settings on PC improvements (#30441)
* always multithreaded

* multithreaded

* pkill correct process

* escape more

* load for interfaces
old-commit-hash: 2ad82cbfb064676401876a20e49e3965c40344a0
2023-11-13 10:45:24 -08:00
Adeeb Shihadeh
8d3a5970d4 jenkins: use build.py to manage cache size (#30440)
* jenkins: use build.py to manage cache size

* label

* set pp

* double

* cleanup

* non mimimal build on PC

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
old-commit-hash: 5f7143df02b857201e2615240ccd1be06daa32fe
2023-11-10 16:55:25 -08:00
Adeeb Shihadeh
554528351b jenkins: retry on failed git checkouts (#30345)
* jenkins: retry on failed git checkouts

* test

* Revert "test"

This reverts commit ea57ba025630eb3105a9ccdce3bdc17f72678c72.
old-commit-hash: 57c8304c2562e807d5b95d0edd78636de215e526
2023-10-30 09:58:57 -07:00
Adeeb Shihadeh
87f77825db test_onroad: log sizes check (#30265)
* test_onroad: check log sizes

* fix

* check logs

* rb

* len

* pytest

* mb

* corolla log is small

* adjust qlog
old-commit-hash: 13d780a5f2b049731df2cbf617cc6dfdd648e622
2023-10-18 19:44:46 -07:00
Adeeb Shihadeh
bf4b441687 jenkins: clear commacache dir (#30290)
old-commit-hash: 71a9b2c24c4a23a7b6d0ed3911076be373436bda
2023-10-18 19:37:29 -07:00
Justin Newberry
845cb6b670 CI: run jenkins as batman (#30184)
* I'm batman

* chown batman

* sudo

* add batman to sudo group

* try without cleanup

* Revert "try without cleanup"

This reverts commit f59554f2f373417f2c59fd250463eca2fda4d49e.

* comment
old-commit-hash: 8b2b72499fb49fa2c01c806b54e2bfe6a2c28844
2023-10-06 11:12:41 -07:00
Adeeb Shihadeh
52752a3131 jenkins speedup (#29869)
* jenkins speedup

* cleanup
old-commit-hash: d0361485504fe5af47a9e8ffaf0b64cfcdf03ac9
2023-09-10 16:47:07 -07:00
Adeeb Shihadeh
58bb1afb9a jenkins scripted pipeline (#29865)
* scripted

* exclude branches

* try that

* fix credentials

* fix

* tag with commit
old-commit-hash: a411af4f02ab778b78a98bfc424769e0c6a5974a
2023-09-09 21:30:21 -07:00
Adeeb Shihadeh
15e071f88c fixup power draw test output (#29844)
* fixup power draw test

* no pytest
old-commit-hash: 947abbcba0f685ae89d6d6a1b05f3f55f9739848
2023-09-08 13:01:04 -07:00
Shane Smiskol
ce1c61d59a car interfaces: long running fuzzy test (#29592)
* draft

* draft 2

* increase deadline and run

* imports

* run -> use
old-commit-hash: 821cfff56d00b61fc3b4fba12bf0830cdb21ed27
2023-08-24 10:45:50 -07:00
Shane Smiskol
11466f786b internal test_models: run on more segments (#29443)
100 more
old-commit-hash: ac3f87bad519df408a295d2e007e0de58b11690c
2023-08-17 00:25:14 -07:00
Shane Smiskol
6be9e66073 jenkins: re-enable test_models (#29394)
* re-enable test_models

* now?

* Remove bad Volt segment

* Update Jenkinsfile

* use first n segments
old-commit-hash: fb235238addb73e04f59fca941040e3785be103e
2023-08-14 16:08:34 -07:00
Adeeb Shihadeh
1e72e9b094 jenkins: remove test that's now run in Actions
old-commit-hash: de66ba01874c630bebd2468237f4eda35171f4d2
2023-08-13 21:21:54 -07:00
Adeeb Shihadeh
1ca3343d0c add small ui for jenkins runners (#29382)
old-commit-hash: 09ffce80737ec54ada27be5643e223ba1db12bd1
2023-08-13 18:01:51 -07:00
Adeeb Shihadeh
e76af64065 CI: enable PYTHONWARNINGS=error (#29364)
* CI: enable PYTHONWARNINGS=error

* fixes

* more ignore

* more modeld ignore
old-commit-hash: b11b14aff86a4c993bf5342ee37af169788c9026
2023-08-12 17:15:16 -07:00
Shane Smiskol
11492496fc test_models: run on route list (#26226)
* move to openpilot

* draft

* support internal urls

* not used

* update seg list

* regen segment list

* regen seg list

* regen

* regen

* regen

* no dirty segments

* regen seg list with no fuzzy or fixed fp

* regen segments with hda2 exception

* regen with added filters

* regen without bad dongle

* regen

* regenerate!

* regenerate, only OP enabled

* regen

* regenerate!

* regenerate!

* stash

* use SegmentName

* new route list

* add temp comment

* remove comment

* let's see if this works

* comment out other tests to speed up

* fix agent

* ?

* hmm

* oh whoops...

* add matrix

* fix

* how many levels do we need?

* fix values

* matrix can't be in parallel

* how about this

* try this

* Revert matrix

Revert "try this"

This reverts commit 8d0d91fd70c467c1fbb4e4e9aed511d15b2a30ef.

Revert "how about this"

This reverts commit a8e4fc53234824e77cbfb1e471277bc033e9dea1.

Revert "matrix can't be in parallel"

This reverts commit daaa6fcc3c75c74cbb90e97c565099a94123994c.

Revert "fix values"

This reverts commit df554b6a3371d124a574eb8d26bc51ef5b5b8fde.

Revert "how many levels do we need?"

This reverts commit 1a17320fa1c5e7220ef60e29981bbb3bb7da16c6.

Revert "fix"

This reverts commit e7eb6e404358fbd2eac3fea1901a8d30ea92d729.

Revert "add matrix"

This reverts commit a1b57e5725417d3c2f639f8edfc0c889b84b6753.

* use pytest instead! (5 jobs is ~150 mins)

* split lines, uncomment

* This Sienna seg has a fault SDSU that stopped forwarding/sending msgs

* picked a route with no PSCMStatus and no panda errors, recent routes are working

* this cadillac was dashcammed (no radar)

* opened an issue for this, it's 'expected' right now

* small clean up

* small clean up

* i don't think that worked

* is this needed?

* add to new PC tests

* cache

* draft

* Revert "draft"

This reverts commit 3b7f740dd4883118747300bc3687074c2d3c2116.

* probably should be function

* draft

* clean up

* add todo

* 600 random segments

* debug

* does this fix pythonpath issues?

fix

* try this

* mount?

* pytest again!

* no need for PYTHONPATH now

* Update Jenkinsfile

* ?

* convention

* clean up

* would be even more complex (have to unset ci which is class level)

* track

* is lfs pulled at all?

* ah no it's not
old-commit-hash: 70b1e9dd756cd2f6929cdf33dd1061a33f3b52b1
2023-08-10 00:45:23 -07:00
Adeeb Shihadeh
5fb047c963 jenkins: rm tools copy in release tests (#29273)
* jenkins: rm tools copy in release tests

* rm that too
old-commit-hash: 7f02cb9fafb05e77dcbd2b11f41ecb8244f641fe
2023-08-07 19:14:45 -07:00
Adeeb Shihadeh
36846e5374 jenkins: reset USB (#29155)
* jenkins: reset USB

* fix non aux
old-commit-hash: 036888ec15388cacca49302c6459270fa81930aa
2023-07-26 15:11:20 -07:00
Adeeb Shihadeh
a5f5b72c99 jenkins: run tests with pytest (#29143)
* jenkins: run tests with pytest

* typo

* fix
old-commit-hash: 6b83724c50d54b99448e9c7aec0aa5dc8b690952
2023-07-25 17:06:43 -07:00
Adeeb Shihadeh
a51824c339 rawgpsd: speedup tests (#29110)
* rawgpsd: speedup tests

* fix that one

* little more

* prevent subtle bugs

---------

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 5fb2f7072bec37ab4da3ca6558834a9e2fffcb9b
2023-07-23 15:38:47 -07:00
Adeeb Shihadeh
3dfd770fb0 tici: speedup power save setting (#29100)
* speedup

* update type

---------

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: d92c9aef23fc789d50bfd03ae09a3abbf23a8429
2023-07-22 17:15:58 -07:00
Maxime Desroches
06125f1309 Test scons multithreaded random build (#29082)
* random build

* name

* 2 pass

* clean after

* fail on this

* fix deps

* format

* var

* try without lfs
old-commit-hash: 97eec45d42a67c3e840a6a08fab2e421b2825de7
2023-07-21 21:57:27 -07:00
Adeeb Shihadeh
13436b4893 CI: fix no DNS after tests killed (#29090)
* CI: fix no DNS after tests killed

* fix
old-commit-hash: e8b20b7f81ff612450b03fbc924506d3d37589f1
2023-07-21 18:35:29 -07:00
Adeeb Shihadeh
30a6fa35aa mapsd: rendering tests (#28981)
* mapsd: rendering tests

* run

* speedup
old-commit-hash: e2b5d1e29a3efeefc0679e116e6f6e16a815425c
2023-07-16 14:39:34 -07:00
Adeeb Shihadeh
52d1de0e79 jenkins: speedup build stage (#28687)
* jenkins: speedup build stage

* run in gh actions

* skip

* set that

* build

* python path

* cleanup
old-commit-hash: fdf1c328b944421036c7e6128b1c9e23064a4ef3
2023-06-24 22:33:06 -07:00
Adeeb Shihadeh
8d80b8445c rawgpsd tests (#28683)
* rawgpsd tests

* test xtra

* check freq

* fix

* restart lte

---------

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 5992bf4acc11bd49206b2b0c16926a129cd11f20
2023-06-24 20:04:58 -07:00
Adeeb Shihadeh
83fd21a79e jenkins: new replay devices (#28643)
* replay

* restart nm
old-commit-hash: 16e3d4f69f5ee4eea23f0440da6a73cf842f80d2
2023-06-22 14:21:11 -07:00
Adeeb Shihadeh
e65200b8ed jenkins: ensure panda is in good state (#28414)
old-commit-hash: 6d91b7c341afd1bf38e1bb044338a2026d0a6c94
2023-06-05 15:28:38 -07:00
Mitchell Goff
3dc3978a10 Test navmodel (#28272)
* Test navmodel

* removed comment

* Updated model_replay ref

* read tiles from azure bucket

* updated ref commit
old-commit-hash: 2d34083cd8a507ed06f0139f496930cfe4bde820
2023-05-30 20:33:01 -07:00
Adeeb Shihadeh
dbe615db9e jenkins: switch order of tests
old-commit-hash: 2664dba52e03b45d7a55f17211eb17abb35598c9
2023-05-22 00:48:51 -07:00
Adeeb Shihadeh
a3dcc5cd97 time to onroad test (#28242)
* time to onroad test

* run few times

* test on stripped builds

* manager

* cleanup

* split those out

* run first

---------

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 39c5d492bfb4ca1630fed1a4ab6990c171c5e797
2023-05-20 17:59:41 -07:00
Adeeb Shihadeh
4e87efb78c controlsd: remove procs from ignore list (#26848)
* trim list

* check for file
old-commit-hash: c4501e4bc1cb7a242207e910c0b61af20c3c677d
2023-05-07 15:43:45 -07:00
Adeeb Shihadeh
e631d24d77 Jenkins tuneup (#28095)
* tuneup

* use service messages

* logmono

* not passive
old-commit-hash: d5a0a4b3e8a52b5a0e1da325c15cdb99d2a71350
2023-05-03 16:52:20 -07:00