mawei/sunnypilot
1
0
Fork 0
mirror of https://github.com/sunnypilot/sunnypilot.git synced 2026-02-19 06:33:57 +08:00
Code Issues Packages Projects Releases Wiki Activity
16,167 Commits 185 Branches 163 Tags
9ee66008dbcbc67e744db99d6c19f38282d885d5
Commit Graph

110 Commits

Author SHA1 Message Date
Shane Smiskol
870d19f33d Reapply "File sourcing: Not all files are logs (#36025)" 
This reverts commit 3570022b9a.

Fix test
 2025-08-19 19:59:50 -07:00
Maxime Desroches
3570022b9a Revert "File sourcing: Not all files are logs (#36025)" 
This reverts commit 18b7ddef8f.
 2025-08-19 17:11:53 -07:00
Harald Schäfer
18b7ddef8f File sourcing: Not all files are logs (#36025) 
* Not all files are logs

* more refactor

* linting ok

* fix tests

* import exception

* whoops forgot to git add

* fix

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
 2025-08-19 16:25:13 -07:00
Shane Smiskol
5ec9aee216 File sourcing: simplify return type (#36028) 
* rm str | none pattern

* clean up

* more clean up

* stash

* Revert "stash"

This reverts commit 3e2472160cc97e9d11922137757d9ef942a0312d.

* fix da prints

* fix cmt
 2025-08-19 15:39:44 -07:00
Shane Smiskol
6005b12f94 format logreader 2025-08-19 15:04:17 -07:00
Shane Smiskol
8ec61991ee LogReader sourcing: remove redundant file existence checks (#35991) 
* speed up sourcing but avoiding checking for existence of collected files already from previous sources

* clean up

* been meaning to make them return dicts

* no longer true

* no longer true

* clean up

* more

* more

* revert
 2025-08-14 19:28:37 -07:00
Shane Smiskol
1eef956cad LogReader sourcing: return dict (#35994) 
* new return type

* fix test

* why not
 2025-08-14 19:19:37 -07:00
Shane Smiskol
aa91a02db8 LogReader sourcing: check comma API source before CI source (#35992) 
sort
 2025-08-14 18:26:19 -07:00
Shane Smiskol
e0f51bdbb6 Reapply "LogReader: wrap events to cache which() (#35882)" (#35909) 
* Reapply "LogReader: wrap events to cache which() (#35882)"

This reverts commit ba2dced54c.

* fix lr

* speed up

* clean up

* more

* should be fast

* clean up

* only supports Event

* rmrmr

* bye

* simple

* gix
 2025-08-08 23:42:54 -07:00
Shane Smiskol
52a4b52628 FileName clean up (#35938) 
two spaces!
 2025-08-06 14:07:02 -07:00
Harald Schäfer
d1e0a60408 Filename refactor: no enum (#35930) 
* conflict

* typing

* typing

* no value

* fix typing

* whitespace

* whitespace

* unused

* Reapply "Filename: minor refactor (#35927)"

This reverts commit 8c7d53004f.

* unused import

* done
 2025-08-05 20:37:09 -07:00
Adeeb Shihadeh
8c7d53004f Revert "Filename: minor refactor (#35927)" 
This reverts commit 96313fa4c0.
 2025-08-05 19:07:17 -07:00
Harald Schäfer
96313fa4c0 Filename: minor refactor (#35927) 
* Filename

* rest of refactor
 2025-08-05 16:29:25 -07:00
Adeeb Shihadeh
ba2dced54c Revert "LogReader: wrap events to cache which() (#35882)" 
This reverts commit 0ebee55050.
 2025-08-02 15:53:20 -07:00
Shane Smiskol
0ebee55050 LogReader: wrap events to cache which() (#35882) 
* speed up lr

* lazy caching

* clean up

* it fast

* stash

* stash

* chatgpt code is bad as usual

* clean up

* clean up

* clean up

* clean up

* clean up

* clean up

* match behavior

* cmt
 2025-08-01 19:07:16 -07:00
Harald Schäfer
103b4df18c Log filenames: make tuples (#35771) 
* make tuples

* spelling: this linter is hardcore
 2025-07-19 19:19:13 -07:00
Shane Smiskol
ffc8ea5a86 LogReader: use file name constants (#35765) 
use common
 2025-07-18 20:29:22 -07:00
Shane Smiskol
48892e339d LogReader: remove testing closet source (#35763) 
rm
 2025-07-18 19:21:04 -07:00
Shane Smiskol
278781e5af LogReader: extend FileName (#35760) 
extend filename
 2025-07-18 17:00:39 -07:00
Shane Smiskol
4a5385a5b0 LogReader: support connect URLs (#35757) 
* support connect

* clean up
 2025-07-18 16:03:30 -07:00
Shane Smiskol
618a25a612 LogReader: simplify sourcing logic and improve AUTO performance (#35753) 
* sourcing supports any file type!

* stuff

* no camera for now

* i guess these are fine without

* rename

* get rid of these too!

* fix

* fix

* this is better

* start to clean up!

* better

* holy

holy

* clean up sources

* more robust

* working but needs some clean up

* clean up

* remove some trash

* nl

* auto_source can only return if it finds acceptable logs from sources

* double negative is confusing

* default

* list isn't hashable

* fix typing

* clean up

* speed up -- test_models got zst before bz2 in openpilotci, so do that (some segments have both bz2 and zst!)

* don't be a hero

* same behavior for now
 2025-07-18 05:15:55 -07:00
Shane Smiskol
b50b351b15 LogReader: remove commaCarSegments selector (#35750) 
* a source should not be a readmode... and readmode should be logtype

* fix that

* fixup
 2025-07-18 02:17:11 -07:00
Shane Smiskol
862a816215 LogReader: add data endpoint option (#35751) 
* add data endpoint option

* clean up
 2025-07-17 23:12:08 -07:00
Shane Smiskol
c553c1f872 LogReader: support passing list of sources (#35749) 
* far too long

* this is a better experience

* no rename for now
 2025-07-17 19:54:38 -07:00
Shane Smiskol
27d882cc0a Revert "logreader.py: concurrent file existence checks" (#34942) 
Revert "logreader.py: concurrent file existence checks (#34875)"

This reverts commit e11b239b2e.
 2025-03-26 14:45:35 -07:00
Shane Smiskol
e652f435f9 max_lat_accel.py: use parallel processing (#34936) 
* use run_across_segments

* disable tqdm
 2025-03-25 13:26:16 -07:00
Dean Lee
e11b239b2e logreader.py: concurrent file existence checks (#34875) 
* concurrent file existence checks

* upper bound on parallel queries

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
 2025-03-24 15:46:42 -07:00
Shane Smiskol
65d667e60e Revert "logreader: avoid repeated prompts in auto_strategy when rlogs are missing" (#34919) 
Revert "logreader: avoid repeated prompts in auto_strategy when rlogs are mis…"

This reverts commit b8229ebca6.
 2025-03-21 17:28:19 -07:00
Dean Lee
b8229ebca6 logreader: avoid repeated prompts in auto_strategy when rlogs are missing (#34890) 
* avoid repeated prompts in auto_strategy when rlogs are missing

* shorter

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
 2025-03-21 14:43:48 -07:00
Adeeb Shihadeh
071f8ba658 LogReader: add time_series helper 2025-02-19 16:43:54 -08:00
Dean Lee
9ec54f59c6 loggerd: switch logging from raw file format to zstd compressed files (#34549) 
* switch logging from raw file format to zstd compressed files

* more zst suffix

* compress bootlog

* remove class RawFile

* Optimize ZstdFileWriter by adding input caching

* use ZSTD_compressStream2

* cleanup

* LOG_COMPRESSION_LEVEL=10

* space

* add zst suffix to LOGS_SIZE_RATE
 2025-02-08 12:55:31 -08:00
Shane Smiskol
2af9f68147 LogReader: more specific exceptions (#33914) 
* more specific logreader exceptions

* huh

* fix
 2024-11-01 15:59:46 -07:00
commaci-public
1e92af9b3c [bot] Update Python packages (#33436) 
* Update Python packages

* bump

* specify any list of sources

ooh this is pretty nice

* update

* y

* fix

* fix that

* bump

---------

Co-authored-by: Vehicle Researcher <user@comma.ai>
Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 239cc69a20b7d3c2f285bd1907cecad9c9604d50
 2024-09-03 18:57:28 -07:00
Shane Smiskol
9da055853f LogReader clean ups (#33446) 
* buggin me

* no caps

* remove at least one level of indirection

* no recursion, remove a bunch of junk

* default is now allfault ??

* back

* fix tests
old-commit-hash: 5796bf12b6dac4f3fc8e55ecd9785c73044d8a08
 2024-09-03 18:34:18 -07:00
Shane Smiskol
730924e7f2 LogReader: formatting and remove unnecessary type alias 
old-commit-hash: 34305bec091c1088249fe6fe8df8de64a7ce1f86
 2024-09-03 17:58:50 -07:00
Kacper Rączy
786aff27da azure_container: pass the filename explicitely when using get_url (#33407) 
* Pass filename explicitely when using azure container

* Do not update segments in test_processes

* openpilotci_source_zst

* Fix static analysisc
old-commit-hash: b5958ebb60
 2024-08-29 21:51:55 -07:00
Jimmy
937ed5f700 Add testing closet to LogReader source (#33242) 
* add testing closet to logreader source

* check source availability

* use internal_source_available
old-commit-hash: 118932acd3
 2024-08-11 15:10:14 -07:00
Hoang Bui
6a2b4dedfc tools/logreader: add desc arg to run_across_segments() (#33226) 
old-commit-hash: 9e648e428d
 2024-08-08 07:51:14 -07:00
Shane Smiskol
7ce33fbe2b process replay: use zst (#33138) 
* test

* update refs to zst

* update
old-commit-hash: dfd387520e
 2024-07-30 01:46:03 -07:00
Shane Smiskol
7b67994a05 use zstandard library (#33110) 
* use zstandard

* run 10

* bye bye

* this was a little overzealous

* clean up
old-commit-hash: 1b3b5ab758
 2024-07-28 02:03:47 -07:00
Shane Smiskol
539c828861 tools: keep re pattern file small (#33084) 
remove bz2 from re file
old-commit-hash: e68bb26e14
 2024-07-25 22:05:10 -07:00
Shane Smiskol
309ccf6482 LogReader: support multiple log extensions in route (#33066) 
* support rlog.zst

* TODO
old-commit-hash: e38a1428aa
 2024-07-24 14:47:33 -07:00
Shane Smiskol
5333e8ba20 LogReader: show total segment count in missing rlog prompt 
old-commit-hash: 3e41b1b287
 2024-07-01 22:43:48 -07:00
StefanLekanic-SYRMIA
5887093eed LogReader: print number of missing rlogs (#32830) 
* count rlogs

* minimization changes

* cache to avoid duplicate checks

* no real point in checking qlogs to just show the warning

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 504dd4543d
 2024-06-26 22:15:13 -07:00
Shane Smiskol
9f50db15c0 LogReader: try zst on internal source (#32751) 
* internal source list files like azure api

* messy but works

* no limit

* simpler

* clean up

* clean up

* clean up

* that's obvious

* better

* we need to unfortunately return a url, so best to take a naive approach for now

* todo

* fix

* clean up
old-commit-hash: b45caf4033
 2024-06-14 01:12:19 -07:00
Shane Smiskol
644156420b LogReader: revert broken check 
it returns files OR None

old-commit-hash: 68e22faa29
 2024-06-14 00:30:14 -07:00
Shane Smiskol
8f402f29a3 SegmentRange: remove timestamp (#32750) 
* add deprecation warning

* remove timestamp

* Update logreader.py
old-commit-hash: d5ce4b3085
 2024-06-13 23:19:32 -07:00
Shane Smiskol
ed80e9357a LogReader: improve error messages (#32747) 
* better error messages

* clean up
old-commit-hash: 3ede1e2a7c
 2024-06-13 22:37:00 -07:00
Shane Smiskol
e88cbb0162 add zstd Python library (#32731) 
* add zstd

* fix

* uploader

* logreader: zst support

* caps

* only logreader
old-commit-hash: 04a93dd849
 2024-06-12 21:14:23 -07:00
Dean Lee
450d012166 logreader.py: optimize event loading and sorting in _LogFileReader (#32729) 
* improve logfilereader

* less changes
old-commit-hash: c64bca97ba
 2024-06-12 14:20:06 -07:00