Commit Graph

259 Commits

Author SHA1 Message Date
Maxime Desroches
6a37d8a89e fix framereader indent 2025-06-19 10:38:26 -07:00
Harald Schäfer
87a6e369aa Framereader: minor cleanup (#35577)
* No wrapping

* unused test

* another list

* mypy

* cleaner

* Revert "cleaner"

This reverts commit ccc1446b9d649d64b20175e22a66e135c44b21e5.

* mypy
2025-06-19 09:49:51 -07:00
Harald Schäfer
5f3d876aaa model replay: framereader cache (#35576)
* Simpler cache version

* cachetools

* different LRU

* lint

* smaller

* just write LRU

* mypy

* same length
2025-06-18 16:29:22 -07:00
Harald Schäfer
d9b6c16037 Cleanup framereader (#35513)
* squash

* misc cleanup

* no LLM garbage

* misc fixes

* typo

* fix CI

* fix hints

* LLM soo wordy

* improve
2025-06-09 22:39:35 -07:00
Harald Schäfer
a8639d3feb rgb is default (#35467)
* rgb is default

* change dummy default
2025-06-04 23:40:41 -07:00
Harald Schäfer
5505338ffb model replay: less frames and less complexity (#35427)
* Revert "ci: faster model_replay (#34036)"

This reverts commit 847a5ce1f3.

* fix conflict

* trigger on test change

* zst

* give start and end frame

* unused flags

* no print

* whitespace

* fix plotting

* slice correct

* no print

* Just start from beginning
2025-06-02 21:54:37 -07:00
Trey Moen
19ad50a185 feat(route.py): add metadata, events retrieval (#35261)
* feat(route.py): add metadata, events retrieval

* whitespace

* raise for status
2025-05-21 11:33:47 -07:00
Trey Moen
60113632b4 feat(clip): delete get_metadata call (#35229)
* feat: delete get_metadata call

* no one else is using this now
2025-05-15 10:02:46 -07:00
Maxime Desroches
997a99926c ci: overwrite previous comment for model_replay (#35209)
* less

* test

* fix

* cleanup
2025-05-13 21:06:30 -07:00
Trey Moen
f704d18a8b feat(clip): title and metadata overlay (#35099)
* wip

* moar

* ensure inter is installed

* line len

* refactor

* dont need this

* no longer than

* show meta for 4s
2025-05-04 10:16:35 -07:00
Shane Smiskol
27d882cc0a Revert "logreader.py: concurrent file existence checks" (#34942)
Revert "logreader.py: concurrent file existence checks (#34875)"

This reverts commit e11b239b2e.
2025-03-26 14:45:35 -07:00
Shane Smiskol
e652f435f9 max_lat_accel.py: use parallel processing (#34936)
* use run_across_segments

* disable tqdm
2025-03-25 13:26:16 -07:00
Dean Lee
e11b239b2e logreader.py: concurrent file existence checks (#34875)
* concurrent file existence checks

* upper bound on parallel queries

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2025-03-24 15:46:42 -07:00
Shane Smiskol
65d667e60e Revert "logreader: avoid repeated prompts in auto_strategy when rlogs are missing" (#34919)
Revert "logreader: avoid repeated prompts in auto_strategy when rlogs are mis…"

This reverts commit b8229ebca6.
2025-03-21 17:28:19 -07:00
Dean Lee
b8229ebca6 logreader: avoid repeated prompts in auto_strategy when rlogs are missing (#34890)
* avoid repeated prompts in auto_strategy when rlogs are missing

* shorter

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2025-03-21 14:43:48 -07:00
Dean Lee
5cffaf5bb5 url_file.py: remove the unnecessary str() wrapping around the hexdigest() (#34891)
remove the unnecessary str() wrapping around the hexdigest()
2025-03-17 12:23:57 +00:00
Adeeb Shihadeh
071f8ba658 LogReader: add time_series helper 2025-02-19 16:43:54 -08:00
Adeeb Shihadeh
a30d3f7d72 test_onroad: cleanup with time series LogReader (#34613) 2025-02-17 21:26:21 -08:00
Adeeb Shihadeh
bab1254a02 LogReader: add time series helper (#34608)
* simple time series

* example
2025-02-17 16:26:41 -08:00
Dean Lee
9ec54f59c6 loggerd: switch logging from raw file format to zstd compressed files (#34549)
* switch logging from raw file format to zstd compressed files

* more zst suffix

* compress bootlog

* remove class RawFile

* Optimize ZstdFileWriter by adding input caching

* use ZSTD_compressStream2

* cleanup

* LOG_COMPRESSION_LEVEL=10

* space

* add zst suffix to LOGS_SIZE_RATE
2025-02-08 12:55:31 -08:00
Kacper Rączy
5dd1972f96 logreader: support filesystem DATA_ENDPOINT (#34340)
* Check if DATA_ENDPOINT is a dir

* Change url resolution to support joining dirs
2025-01-09 01:20:59 +00:00
Shane Smiskol
da2c70e097 Revert "LogReader: fix issue when your dns resolves all requests" (#34229)
Revert "LogReader: fix issue when your dns resolves all requests (#34089)"

This reverts commit 7fc5040ed9.
2024-12-13 15:11:30 -08:00
Maxime Desroches
e9246df02e ci: faster unit tests (#34161)
* fast

* lint

* try

* clean

* this

* try even more

* more

* maybe

* what

* maybe?

* maybe

* try this

* fix

* fast?

* now cache?

* debug

* again...

* maybe

* NOW!

* revert, this is already enough (for now?)
2024-12-05 16:41:38 -08:00
Dean Lee
e19ecbf75c CommaApi: use context manager for response handling (#34118)
use context manager for response handling
2024-12-05 15:28:11 -08:00
Justin Newberry
7fc5040ed9 LogReader: fix issue when your dns resolves all requests (#34089)
* terrible :(

* keep this spacing
2024-11-22 16:01:53 -08:00
Maxime Desroches
847a5ce1f3 ci: faster model_replay (#34036)
* cache draft

* fix

* fix

* fix

* better

* zst

* more

* try

* pool

* fix

* fix

* revert :C

* better

* cleanup

* no cache

* this too
2024-11-16 15:34:21 -08:00
Maxime Desroches
c995d5b9ae ci: faster test_models (#33941)
* test

* try 4 runners

* try

* 100 for fun

* the fun is over

* try even less
2024-11-06 19:17:15 -08:00
Shane Smiskol
2af9f68147 LogReader: more specific exceptions (#33914)
* more specific logreader exceptions

* huh

* fix
2024-11-01 15:59:46 -07:00
Maxime Desroches
f11e5492f8 ci: keep old comment on model replay report (#33759)
keep old
2024-10-09 15:29:41 -07:00
Maxime Desroches
596d8b13bb ci: model_replay improvements (#33753)
* diff

* tr

* cleanup

* at the end

* fix
2024-10-08 17:39:13 -07:00
Maxime Desroches
2e83e37984 jenkins: model replay report in PR (#33723)
* first

* first

* first

* token

* edit previous comment

* clean

* plots

* linter

* cleaner

* comment

* save ref

* fix

* remove refs

* add tokens

* fix branch

* table

* fix

* real data

* title

* github

* fix

* github api

* better

* clean

* errors

* create bucket

* true

* fix
2024-10-07 21:40:00 -07:00
Shane Smiskol
606943010e Reapply "joystickd is a real process (#33490)" (#33514)
* Reapply "joystickd is a real process (#33490)"

This reverts commit c8465e3a21.

* catch this

* reset to 0 when unplugged

* catch this too

* pytest capturing breaks stdin (pytest -s) fixes
2024-09-07 17:03:22 -07:00
commaci-public
1e92af9b3c [bot] Update Python packages (#33436)
* Update Python packages

* bump

* specify any list of sources

ooh this is pretty nice

* update

* y

* fix

* fix that

* bump

---------

Co-authored-by: Vehicle Researcher <user@comma.ai>
Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 239cc69a20b7d3c2f285bd1907cecad9c9604d50
2024-09-03 18:57:28 -07:00
Shane Smiskol
9da055853f LogReader clean ups (#33446)
* buggin me

* no caps

* remove at least one level of indirection

* no recursion, remove a bunch of junk

* default is now allfault ??

* back

* fix tests
old-commit-hash: 5796bf12b6dac4f3fc8e55ecd9785c73044d8a08
2024-09-03 18:34:18 -07:00
Shane Smiskol
730924e7f2 LogReader: formatting and remove unnecessary type alias
old-commit-hash: 34305bec091c1088249fe6fe8df8de64a7ce1f86
2024-09-03 17:58:50 -07:00
Kacper Rączy
786aff27da azure_container: pass the filename explicitely when using get_url (#33407)
* Pass filename explicitely when using azure container

* Do not update segments in test_processes

* openpilotci_source_zst

* Fix static analysisc
old-commit-hash: b5958ebb60
2024-08-29 21:51:55 -07:00
Shane Smiskol
4f019b5f60 move selfdrive/car to opendbc (#32630)
* move most of /car

* move some car tests

move some car tests

* fix selfdrive/car/tests

* fix selfdrive/controls tests

* fix the rest of the selfdrive tests

* bump opendbc

* fix all tests

* few more non-test references

* remove opcar and move docs to car

fix these debugging scripts

fix docs

* bump opendbc and panda

forgot panda
old-commit-hash: e735a7f379
2024-08-17 00:54:51 -07:00
Maxime Desroches
c551f1d239 fix bash + python3 shebangs (#33287)
fix shebangs
old-commit-hash: 1dcd660815
2024-08-13 21:22:13 -07:00
Jimmy
937ed5f700 Add testing closet to LogReader source (#33242)
* add testing closet to logreader source

* check source availability

* use internal_source_available
old-commit-hash: 118932acd3
2024-08-11 15:10:14 -07:00
Hoang Bui
6a2b4dedfc tools/logreader: add desc arg to run_across_segments() (#33226)
old-commit-hash: 9e648e428d
2024-08-08 07:51:14 -07:00
Shane Smiskol
7ce33fbe2b process replay: use zst (#33138)
* test

* update refs to zst

* update
old-commit-hash: dfd387520e
2024-07-30 01:46:03 -07:00
Shane Smiskol
7b67994a05 use zstandard library (#33110)
* use zstandard

* run 10

* bye bye

* this was a little overzealous

* clean up
old-commit-hash: 1b3b5ab758
2024-07-28 02:03:47 -07:00
Shane Smiskol
539c828861 tools: keep re pattern file small (#33084)
remove bz2 from re file
old-commit-hash: e68bb26e14
2024-07-25 22:05:10 -07:00
Shane Smiskol
309ccf6482 LogReader: support multiple log extensions in route (#33066)
* support rlog.zst

* TODO
old-commit-hash: e38a1428aa
2024-07-24 14:47:33 -07:00
Kacper Rączy
4efe8d8118 Fix msgq context overrides (#33014)
* Fix msgq context overrides

* Remove unneccesary changes

* add reset_context helper

* Remove redundant import
old-commit-hash: c96dbd5a0b
2024-07-18 20:19:39 -07:00
Shane Smiskol
5333e8ba20 LogReader: show total segment count in missing rlog prompt
old-commit-hash: 3e41b1b287
2024-07-01 22:43:48 -07:00
Maxime Desroches
8adfc992e3 fix pytest tests (#32867)
* fix tests

* no dot
old-commit-hash: 6f10b8a209
2024-06-28 23:58:19 +00:00
StefanLekanic-SYRMIA
5887093eed LogReader: print number of missing rlogs (#32830)
* count rlogs

* minimization changes

* cache to avoid duplicate checks

* no real point in checking qlogs to just show the warning

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 504dd4543d
2024-06-26 22:15:13 -07:00
Shane Smiskol
9f50db15c0 LogReader: try zst on internal source (#32751)
* internal source list files like azure api

* messy but works

* no limit

* simpler

* clean up

* clean up

* clean up

* that's obvious

* better

* we need to unfortunately return a url, so best to take a naive approach for now

* todo

* fix

* clean up
old-commit-hash: b45caf4033
2024-06-14 01:12:19 -07:00
Shane Smiskol
644156420b LogReader: revert broken check
it returns files OR None

old-commit-hash: 68e22faa29
2024-06-14 00:30:14 -07:00