Commit Graph

162 Commits

Author SHA1 Message Date
Justin Newberry
824ddffec8 CommaCarSegments: BIG Database! (#31169)
* sanetized selector

* god damn

* add to re

* slog branch

* slog

* source

* big

* not slog

* cleanup

* no space

* works!

* trigger

* main!
2024-01-27 01:07:29 -08:00
Adeeb Shihadeh
2504341070 bootlog: remove timestamp from filename (#31187)
* bootlog: remove timestamp from filename

* revert route

* test

* revert that

* fix loggerd test
2024-01-26 19:17:38 -08:00
Adeeb Shihadeh
52c7514452 tools: add LOG_ID regex pattern (#31186) 2024-01-26 16:44:12 -08:00
Justin Newberry
9518b66c9e commaCarSegments cleanup (#31182)
* cleanup

* no space
2024-01-26 13:20:25 -08:00
Greg Hogan
6901aada14 filereader should use connection pool for file_exists (#31158)
* filereader should use connection pool for file_exists

* remove debug var
2024-01-25 13:26:12 -08:00
Justin Newberry
edd2428b60 LogReader: fail-fast on invalid segments (#31152)
* don't check all

* is
2024-01-25 11:09:09 -08:00
Justin Newberry
dd0c474e6c LogReader: replace get_first_message with LogReader.first (#31146)
* first

* str
2024-01-24 10:21:57 -08:00
Justin Newberry
88dcaa51c4 LogReader: no redownloading on multiple iterations (#31141)
* no redownload

* sort
2024-01-23 23:54:11 -08:00
Justin Newberry
ac71c30ad5 auto_source: add commaCarSegments source (#31137)
add source
2024-01-23 21:40:04 -08:00
Justin Newberry
3d9b483e10 comma_api_source: fix qlogs (#31136)
fix qlogs
2024-01-23 20:51:15 -08:00
Justin Newberry
d574013d7b LogReader: don't use route for max_segment (#31129)
* dont use route

* fix and cleanup
2024-01-23 14:51:45 -08:00
Justin Newberry
0234cadec3 LogReader: remove redirect message for internal data (#31088)
fix logging
2024-01-23 12:55:50 -08:00
Justin Newberry
65490bb3c5 plotjuggler: auto fallback to qlogs with prompt (#31110)
* qlog fallback

* show a warning for that

* qlog fallback

* fix

* wording
2024-01-23 12:53:19 -08:00
Justin Newberry
35e1e8ecd2 SegmentRange: __str__ get preferred format (#31126)
get canonical
2024-01-23 11:40:22 -08:00
Justin Newberry
389b8ca30d add example of using the comma car segments database (#31117)
* add vin fingerprinting example

* perhaps!

* remove segment

* easier
2024-01-22 18:23:07 -08:00
Justin Newberry
caa9153974 CommaCarSegments: public database of segments for each platform (#31114)
* comma car segments

* comma car segments for test_models

* oneliner

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* better name

* not used here

* sort

* remove print

* better comment

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2024-01-22 17:33:03 -08:00
Justin Newberry
2c86c023fe sanitizer: sanitize vin number (#31090) 2024-01-21 12:22:32 -05:00
Justin Newberry
840db1f1b7 prevent typing issue in azure_container (#31044)
prvent that issue
2024-01-17 15:34:24 -08:00
Justin Newberry
3f0b56b364 cleanup azure handling (#31034)
* wip cleanup

* Wip

* fixes

* fix

* cleanup

* keep this for now

* dest
2024-01-17 14:21:47 -08:00
Justin Newberry
2967cada71 auto_source: optimize api calls and use head to determine if file exists (#31025)
* fast

* catch all

* source

* fix file_exists

* remove duplicate reset

* test multiple loops

* iterations

* cleanup imports
2024-01-16 17:24:37 -08:00
Justin Newberry
61fe705160 LogReader: fix plotjuggler not reading properly (#31023)
* fix plotjuggler

* simpler

* fix juggler

* fix that
2024-01-16 12:40:49 -08:00
Justin Newberry
7d400112df LogReader: move opci to tools/lib (#31021)
move opci
2024-01-16 10:58:51 -08:00
Justin Newberry
0739d5d7b2 LogReader: support lists of identifiers (#31020)
* support lists of logs

* mark as slow instead
2024-01-16 10:40:00 -08:00
Justin Newberry
4da8d16514 LogReader: support only_union_types (#31019)
support only-union-types
2024-01-16 10:03:55 -08:00
Adeeb Shihadeh
e17e66180c CI: speedup unit test job (#30995)
* start

* mv to buildjet

* fix
2024-01-13 21:16:35 -08:00
Justin Newberry
3d73512c07 rename segmentrangereader to logreader (#30981)
* Replace multilogiterator

* replace logreader

* update readme

* fix from_Bytes

* new section

* reset before iter

* selector

* fix internal

* whitespace
2024-01-12 12:53:50 -08:00
Justin Newberry
9f1b72ac79 Replace multilogiterator (#30980) 2024-01-12 11:43:32 -08:00
Justin Newberry
eb09294fc2 segmentrangereader: support direct parsing (#30973)
* use correct source

* revert

* cleanup imports

* clean

* direct parsing

* rename

* move up

* fixes

* fix that

* better error message
2024-01-11 16:56:48 -08:00
Justin Newberry
fb994ae782 segmentrangereader: support more sources (#30970)
* support more sources

* remove this
2024-01-11 12:55:19 -08:00
Justin Newberry
be814ed447 SegmentRangeReader: log type selector (#30960)
* log type selector

* test this too

* and this

* invalid selector

* use strenum

* don't hardcode size

* oneline
2024-01-11 10:25:24 -08:00
Justin Newberry
cfb23eb2d1 segmentrangereader: support sort_by_time (#30954)
support sort-by-time
2024-01-10 13:36:55 -08:00
Justin Newberry
0d126e1e9e SegmentRangeReader: new format for reading multiple segments (#30940)
* segment range reader

* rename that

* revert that

* cleanup

* revert this for now

* revert this for now

* Fix + test

* rm that

* rm that

* use for auto_fingerprint

* simpler

* for notebook too

* match numpy indexing

* just use numpy directly

* remove that

* spacing

* spacing

* use qlog for auto fingerprint

* add 'read mode'

* pass in read mode

* add test for modes

* numpy indexing

* fix that case

* more examples

* fix the notebook

* cleanup the notebook

* cleaner

* fix those
2024-01-10 11:55:19 -08:00
Justin Newberry
fba521ecc6 url_file: fix non-200 files being cached (#30886)
* fix + test

* fix unclosed

* easier to read

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* fix that

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2024-01-02 12:42:08 -08:00
Adeeb Shihadeh
51a93cf9aa fix unused 2023-12-17 16:32:56 -08:00
royjr
70624ffc81 dependency: remove pycurl package (#30771)
* Update qcomgpsd.py

* Update url_file.py

* remove pycurl

* requests -> urllib3

* unused

* redundant

* fix import

* Revert "requests -> urllib3"

This reverts commit 7ca39e618917986f50356519d2795a432e3a5280.

* headless

* fix trail

* use requests.exceptions.RequestException

* use fp.tell

* fix indents

* reorder imports

* change timeout

* fix debug timing

* remove exception

* add timeout

* missing headers

* move to constructor

* move import

* unused import

* fix debug

* try

* no retries
2023-12-17 16:25:25 -08:00
Greg Hogan
db35dcd0b5 replace common.file_helpers.mkdirs_exists_ok with python os.makedirs funtion (#30618)
replace common.file_helpers.mkdirs_exists_ok with python os.makedirs function
2023-12-06 09:55:29 -08:00
Justin Newberry
4086795c29 Pytest: more post-test path cleanup (#30556)
* test cleanup

* missed this one

* remove that

* static class
2023-11-30 09:55:48 -08:00
Kacper Rączy
c27e977475 URLFile: exception type for failed requests (#30330)
URLFileException for URLFile request errors
2023-10-25 16:20:12 -07:00
Greg Hogan
68acb26aa2 framereader: eliminate tempfile usage (#30289) 2023-10-18 19:48:26 -07:00
Greg Hogan
59e947dc6c fix ffprobe on device (#30286)
* fix ffprobe on device

* use FileReader

* no more requests
2023-10-18 18:31:14 -07:00
Justin Newberry
1820b5f259 Proc Replay: cleanup iterable log typing (#30261)
* cleanup log typing

* use iterable

* fix those

* Cleanup

* cleanup

* missed that one
2023-10-17 12:58:28 -07:00
Greg Hogan
6f98a987af vidindex improvements (#30196)
* vidindex improvements

* fix spelling
2023-10-05 22:48:50 -07:00
Greg Hogan
86b90a8ba2 filereader cleanup (#30191)
* filereader cleanup

* make name return url

* remove unused imports
2023-10-05 19:45:46 -07:00
Greg Hogan
f8e488f881 convert vidindex to python (#30176)
* convert vidindex to python

* fix whitespace

* corrupt file option

* fix up typings

* fix return type

* update framereader

* change length delimiter to uint32 value

* change length to uint32 value

* move url_file changes to separate PR

* cleanup caching

* revert whitespace change

* fix frame type param type
2023-10-05 18:40:24 -07:00
Justin Newberry
fa51bbc236 URLFile: don't cache non-existent file's lengths (#30071)
* urlfile empty

* simplify with mock

* better test name

* PR cleanup

* cleanup the length file
2023-10-04 12:57:56 -07:00
YassineYousfi
880157f5a6 framereader: add cache_dir argument (#29904)
* framereader: add cache_dir argument

* make it an env var
2023-09-14 14:36:32 -07:00
Greg Hogan
140a4ba558 importing auth_config.py should not have side effects (#29893)
import should not create directory
2023-09-12 18:00:18 +00:00
Justin Newberry
73eda51a11 System: use paths for download folders too (#29818)
* use paths for download folders too

* trailing slash

* reset that
2023-09-07 12:49:03 -07:00
Justin Newberry
4e69937d0d Precommit: ensure executable bit is set (#29784)
* precommit ensure executable

* exclude tinygrad

* bump submodules

* exclude tinygrad globally
2023-09-05 16:33:26 -07:00
Justin Newberry
405d09714c Tests: expand temporary directory helpers for more directories (#29735)
* create helpers for testing

* document it

* fix other tests
2023-08-31 14:00:31 -07:00