Adeeb Shihadeh
ea01a53711
switch from mypy to ty ( #36961 )
2025-12-28 10:42:49 -08:00
Bruce Wayne
5393308d03
Logreader: print errors
2025-12-02 15:54:21 -08:00
Shane Smiskol
870d19f33d
Reapply "File sourcing: Not all files are logs ( #36025 )"
...
This reverts commit 3570022b9a
2025-08-19 19:59:50 -07:00
Maxime Desroches
3570022b9a
Revert "File sourcing: Not all files are logs ( #36025 )"
...
This reverts commit 18b7ddef8f
2025-08-19 17:11:53 -07:00
Harald Schäfer
18b7ddef8f
File sourcing: Not all files are logs ( #36025 )
...
* Not all files are logs
* more refactor
* linting ok
* fix tests
* import exception
* whoops forgot to git add
* fix
---------
Co-authored-by: Shane Smiskol <shane@smiskol.com >
2025-08-19 16:25:13 -07:00
Shane Smiskol
5ec9aee216
File sourcing: simplify return type ( #36028 )
...
* rm str | none pattern
* clean up
* more clean up
* stash
* Revert "stash"
This reverts commit 3e2472160cc97e9d11922137757d9ef942a0312d.
* fix da prints
* fix cmt
2025-08-19 15:39:44 -07:00
Shane Smiskol
6005b12f94
format logreader
2025-08-19 15:04:17 -07:00
Shane Smiskol
8ec61991ee
LogReader sourcing: remove redundant file existence checks ( #35991 )
...
* speed up sourcing but avoiding checking for existence of collected files already from previous sources
* clean up
* been meaning to make them return dicts
* no longer true
* no longer true
* clean up
* more
* more
* revert
2025-08-14 19:28:37 -07:00
Shane Smiskol
1eef956cad
LogReader sourcing: return dict ( #35994 )
...
* new return type
* fix test
* why not
2025-08-14 19:19:37 -07:00
Shane Smiskol
aa91a02db8
LogReader sourcing: check comma API source before CI source ( #35992 )
...
sort
2025-08-14 18:26:19 -07:00
Shane Smiskol
e0f51bdbb6
Reapply "LogReader: wrap events to cache which() ( #35882 )" ( #35909 )
...
* Reapply "LogReader: wrap events to cache which() (#35882 )"
This reverts commit ba2dced54c
2025-08-08 23:42:54 -07:00
Shane Smiskol
52a4b52628
FileName clean up ( #35938 )
...
two spaces!
2025-08-06 14:07:02 -07:00
Harald Schäfer
d1e0a60408
Filename refactor: no enum ( #35930 )
...
* conflict
* typing
* typing
* no value
* fix typing
* whitespace
* whitespace
* unused
* Reapply "Filename: minor refactor (#35927 )"
This reverts commit 8c7d53004f
2025-08-05 20:37:09 -07:00
Adeeb Shihadeh
8c7d53004f
Revert "Filename: minor refactor ( #35927 )"
...
This reverts commit 96313fa4c0
2025-08-05 19:07:17 -07:00
Harald Schäfer
96313fa4c0
Filename: minor refactor ( #35927 )
...
* Filename
* rest of refactor
2025-08-05 16:29:25 -07:00
Adeeb Shihadeh
ba2dced54c
Revert "LogReader: wrap events to cache which() ( #35882 )"
...
This reverts commit 0ebee55050
2025-08-02 15:53:20 -07:00
Shane Smiskol
0ebee55050
LogReader: wrap events to cache which() ( #35882 )
...
* speed up lr
* lazy caching
* clean up
* it fast
* stash
* stash
* chatgpt code is bad as usual
* clean up
* clean up
* clean up
* clean up
* clean up
* clean up
* match behavior
* cmt
2025-08-01 19:07:16 -07:00
Harald Schäfer
103b4df18c
Log filenames: make tuples ( #35771 )
...
* make tuples
* spelling: this linter is hardcore
2025-07-19 19:19:13 -07:00
Shane Smiskol
ffc8ea5a86
LogReader: use file name constants ( #35765 )
...
use common
2025-07-18 20:29:22 -07:00
Shane Smiskol
48892e339d
LogReader: remove testing closet source ( #35763 )
...
rm
2025-07-18 19:21:04 -07:00
Shane Smiskol
278781e5af
LogReader: extend FileName ( #35760 )
...
extend filename
2025-07-18 17:00:39 -07:00
Shane Smiskol
4a5385a5b0
LogReader: support connect URLs ( #35757 )
...
* support connect
* clean up
2025-07-18 16:03:30 -07:00
Shane Smiskol
618a25a612
LogReader: simplify sourcing logic and improve AUTO performance ( #35753 )
...
* sourcing supports any file type!
* stuff
* no camera for now
* i guess these are fine without
* rename
* get rid of these too!
* fix
* fix
* this is better
* start to clean up!
* better
* holy
holy
* clean up sources
* more robust
* working but needs some clean up
* clean up
* remove some trash
* nl
* auto_source can only return if it finds acceptable logs from sources
* double negative is confusing
* default
* list isn't hashable
* fix typing
* clean up
* speed up -- test_models got zst before bz2 in openpilotci, so do that (some segments have both bz2 and zst!)
* don't be a hero
* same behavior for now
2025-07-18 05:15:55 -07:00
Shane Smiskol
b50b351b15
LogReader: remove commaCarSegments selector ( #35750 )
...
* a source should not be a readmode... and readmode should be logtype
* fix that
* fixup
2025-07-18 02:17:11 -07:00
Shane Smiskol
862a816215
LogReader: add data endpoint option ( #35751 )
...
* add data endpoint option
* clean up
2025-07-17 23:12:08 -07:00
Shane Smiskol
c553c1f872
LogReader: support passing list of sources ( #35749 )
...
* far too long
* this is a better experience
* no rename for now
2025-07-17 19:54:38 -07:00
Shane Smiskol
27d882cc0a
Revert "logreader.py: concurrent file existence checks" ( #34942 )
...
Revert "logreader.py: concurrent file existence checks (#34875 )"
This reverts commit e11b239b2e
2025-03-26 14:45:35 -07:00
Shane Smiskol
e652f435f9
max_lat_accel.py: use parallel processing ( #34936 )
...
* use run_across_segments
* disable tqdm
2025-03-25 13:26:16 -07:00
Dean Lee
e11b239b2e
logreader.py: concurrent file existence checks ( #34875 )
...
* concurrent file existence checks
* upper bound on parallel queries
---------
Co-authored-by: Shane Smiskol <shane@smiskol.com >
2025-03-24 15:46:42 -07:00
Shane Smiskol
65d667e60e
Revert "logreader: avoid repeated prompts in auto_strategy when rlogs are missing" ( #34919 )
...
Revert "logreader: avoid repeated prompts in auto_strategy when rlogs are mis…"
This reverts commit b8229ebca6
2025-03-21 17:28:19 -07:00
Dean Lee
b8229ebca6
logreader: avoid repeated prompts in auto_strategy when rlogs are missing ( #34890 )
...
* avoid repeated prompts in auto_strategy when rlogs are missing
* shorter
---------
Co-authored-by: Shane Smiskol <shane@smiskol.com >
2025-03-21 14:43:48 -07:00
Adeeb Shihadeh
071f8ba658
LogReader: add time_series helper
2025-02-19 16:43:54 -08:00
Dean Lee
9ec54f59c6
loggerd: switch logging from raw file format to zstd compressed files ( #34549 )
...
* switch logging from raw file format to zstd compressed files
* more zst suffix
* compress bootlog
* remove class RawFile
* Optimize ZstdFileWriter by adding input caching
* use ZSTD_compressStream2
* cleanup
* LOG_COMPRESSION_LEVEL=10
* space
* add zst suffix to LOGS_SIZE_RATE
2025-02-08 12:55:31 -08:00
Shane Smiskol
2af9f68147
LogReader: more specific exceptions ( #33914 )
...
* more specific logreader exceptions
* huh
* fix
2024-11-01 15:59:46 -07:00
commaci-public
1e92af9b3c
[bot] Update Python packages ( #33436 )
...
* Update Python packages
* bump
* specify any list of sources
ooh this is pretty nice
* update
* y
* fix
* fix that
* bump
---------
Co-authored-by: Vehicle Researcher <user@comma.ai >
Co-authored-by: Shane Smiskol <shane@smiskol.com >
old-commit-hash: 239cc69a20b7d3c2f285bd1907cecad9c9604d50
2024-09-03 18:57:28 -07:00
Shane Smiskol
9da055853f
LogReader clean ups ( #33446 )
...
* buggin me
* no caps
* remove at least one level of indirection
* no recursion, remove a bunch of junk
* default is now allfault ??
* back
* fix tests
old-commit-hash: 5796bf12b6dac4f3fc8e55ecd9785c73044d8a08
2024-09-03 18:34:18 -07:00
Shane Smiskol
730924e7f2
LogReader: formatting and remove unnecessary type alias
...
old-commit-hash: 34305bec091c1088249fe6fe8df8de64a7ce1f86
2024-09-03 17:58:50 -07:00
Kacper Rączy
786aff27da
azure_container: pass the filename explicitely when using get_url ( #33407 )
...
* Pass filename explicitely when using azure container
* Do not update segments in test_processes
* openpilotci_source_zst
* Fix static analysisc
old-commit-hash: b5958ebb60
2024-08-29 21:51:55 -07:00
Jimmy
937ed5f700
Add testing closet to LogReader source ( #33242 )
...
* add testing closet to logreader source
* check source availability
* use internal_source_available
old-commit-hash: 118932acd3
2024-08-11 15:10:14 -07:00
Hoang Bui
6a2b4dedfc
tools/logreader: add desc arg to run_across_segments() ( #33226 )
...
old-commit-hash: 9e648e428d
2024-08-08 07:51:14 -07:00
Shane Smiskol
7ce33fbe2b
process replay: use zst ( #33138 )
...
* test
* update refs to zst
* update
old-commit-hash: dfd387520e
2024-07-30 01:46:03 -07:00
Shane Smiskol
7b67994a05
use zstandard library ( #33110 )
...
* use zstandard
* run 10
* bye bye
* this was a little overzealous
* clean up
old-commit-hash: 1b3b5ab758
2024-07-28 02:03:47 -07:00
Shane Smiskol
539c828861
tools: keep re pattern file small ( #33084 )
...
remove bz2 from re file
old-commit-hash: e68bb26e14
2024-07-25 22:05:10 -07:00
Shane Smiskol
309ccf6482
LogReader: support multiple log extensions in route ( #33066 )
...
* support rlog.zst
* TODO
old-commit-hash: e38a1428aa
2024-07-24 14:47:33 -07:00
Shane Smiskol
5333e8ba20
LogReader: show total segment count in missing rlog prompt
...
old-commit-hash: 3e41b1b287
2024-07-01 22:43:48 -07:00
StefanLekanic-SYRMIA
5887093eed
LogReader: print number of missing rlogs ( #32830 )
...
* count rlogs
* minimization changes
* cache to avoid duplicate checks
* no real point in checking qlogs to just show the warning
---------
Co-authored-by: Shane Smiskol <shane@smiskol.com >
old-commit-hash: 504dd4543d
2024-06-26 22:15:13 -07:00
Shane Smiskol
9f50db15c0
LogReader: try zst on internal source ( #32751 )
...
* internal source list files like azure api
* messy but works
* no limit
* simpler
* clean up
* clean up
* clean up
* that's obvious
* better
* we need to unfortunately return a url, so best to take a naive approach for now
* todo
* fix
* clean up
old-commit-hash: b45caf4033
2024-06-14 01:12:19 -07:00
Shane Smiskol
644156420b
LogReader: revert broken check
...
it returns files OR None
old-commit-hash: 68e22faa29
2024-06-14 00:30:14 -07:00
Shane Smiskol
8f402f29a3
SegmentRange: remove timestamp ( #32750 )
...
* add deprecation warning
* remove timestamp
* Update logreader.py
old-commit-hash: d5ce4b3085
2024-06-13 23:19:32 -07:00
Shane Smiskol
ed80e9357a
LogReader: improve error messages ( #32747 )
...
* better error messages
* clean up
old-commit-hash: 3ede1e2a7c
2024-06-13 22:37:00 -07:00
