Commit Graph

243 Commits

Author SHA1 Message Date
Adeeb Shihadeh
071f8ba658 LogReader: add time_series helper 2025-02-19 16:43:54 -08:00
Adeeb Shihadeh
a30d3f7d72 test_onroad: cleanup with time series LogReader (#34613) 2025-02-17 21:26:21 -08:00
Adeeb Shihadeh
bab1254a02 LogReader: add time series helper (#34608)
* simple time series

* example
2025-02-17 16:26:41 -08:00
Dean Lee
9ec54f59c6 loggerd: switch logging from raw file format to zstd compressed files (#34549)
* switch logging from raw file format to zstd compressed files

* more zst suffix

* compress bootlog

* remove class RawFile

* Optimize ZstdFileWriter by adding input caching

* use ZSTD_compressStream2

* cleanup

* LOG_COMPRESSION_LEVEL=10

* space

* add zst suffix to LOGS_SIZE_RATE
2025-02-08 12:55:31 -08:00
Kacper Rączy
5dd1972f96 logreader: support filesystem DATA_ENDPOINT (#34340)
* Check if DATA_ENDPOINT is a dir

* Change url resolution to support joining dirs
2025-01-09 01:20:59 +00:00
Shane Smiskol
da2c70e097 Revert "LogReader: fix issue when your dns resolves all requests" (#34229)
Revert "LogReader: fix issue when your dns resolves all requests (#34089)"

This reverts commit 7fc5040ed9.
2024-12-13 15:11:30 -08:00
Maxime Desroches
e9246df02e ci: faster unit tests (#34161)
* fast

* lint

* try

* clean

* this

* try even more

* more

* maybe

* what

* maybe?

* maybe

* try this

* fix

* fast?

* now cache?

* debug

* again...

* maybe

* NOW!

* revert, this is already enough (for now?)
2024-12-05 16:41:38 -08:00
Dean Lee
e19ecbf75c CommaApi: use context manager for response handling (#34118)
use context manager for response handling
2024-12-05 15:28:11 -08:00
Justin Newberry
7fc5040ed9 LogReader: fix issue when your dns resolves all requests (#34089)
* terrible :(

* keep this spacing
2024-11-22 16:01:53 -08:00
Maxime Desroches
847a5ce1f3 ci: faster model_replay (#34036)
* cache draft

* fix

* fix

* fix

* better

* zst

* more

* try

* pool

* fix

* fix

* revert :C

* better

* cleanup

* no cache

* this too
2024-11-16 15:34:21 -08:00
Maxime Desroches
c995d5b9ae ci: faster test_models (#33941)
* test

* try 4 runners

* try

* 100 for fun

* the fun is over

* try even less
2024-11-06 19:17:15 -08:00
Shane Smiskol
2af9f68147 LogReader: more specific exceptions (#33914)
* more specific logreader exceptions

* huh

* fix
2024-11-01 15:59:46 -07:00
Maxime Desroches
f11e5492f8 ci: keep old comment on model replay report (#33759)
keep old
2024-10-09 15:29:41 -07:00
Maxime Desroches
596d8b13bb ci: model_replay improvements (#33753)
* diff

* tr

* cleanup

* at the end

* fix
2024-10-08 17:39:13 -07:00
Maxime Desroches
2e83e37984 jenkins: model replay report in PR (#33723)
* first

* first

* first

* token

* edit previous comment

* clean

* plots

* linter

* cleaner

* comment

* save ref

* fix

* remove refs

* add tokens

* fix branch

* table

* fix

* real data

* title

* github

* fix

* github api

* better

* clean

* errors

* create bucket

* true

* fix
2024-10-07 21:40:00 -07:00
Shane Smiskol
606943010e Reapply "joystickd is a real process (#33490)" (#33514)
* Reapply "joystickd is a real process (#33490)"

This reverts commit c8465e3a21.

* catch this

* reset to 0 when unplugged

* catch this too

* pytest capturing breaks stdin (pytest -s) fixes
2024-09-07 17:03:22 -07:00
commaci-public
1e92af9b3c [bot] Update Python packages (#33436)
* Update Python packages

* bump

* specify any list of sources

ooh this is pretty nice

* update

* y

* fix

* fix that

* bump

---------

Co-authored-by: Vehicle Researcher <user@comma.ai>
Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 239cc69a20b7d3c2f285bd1907cecad9c9604d50
2024-09-03 18:57:28 -07:00
Shane Smiskol
9da055853f LogReader clean ups (#33446)
* buggin me

* no caps

* remove at least one level of indirection

* no recursion, remove a bunch of junk

* default is now allfault ??

* back

* fix tests
old-commit-hash: 5796bf12b6dac4f3fc8e55ecd9785c73044d8a08
2024-09-03 18:34:18 -07:00
Shane Smiskol
730924e7f2 LogReader: formatting and remove unnecessary type alias
old-commit-hash: 34305bec091c1088249fe6fe8df8de64a7ce1f86
2024-09-03 17:58:50 -07:00
Kacper Rączy
786aff27da azure_container: pass the filename explicitely when using get_url (#33407)
* Pass filename explicitely when using azure container

* Do not update segments in test_processes

* openpilotci_source_zst

* Fix static analysisc
old-commit-hash: b5958ebb60
2024-08-29 21:51:55 -07:00
Shane Smiskol
4f019b5f60 move selfdrive/car to opendbc (#32630)
* move most of /car

* move some car tests

move some car tests

* fix selfdrive/car/tests

* fix selfdrive/controls tests

* fix the rest of the selfdrive tests

* bump opendbc

* fix all tests

* few more non-test references

* remove opcar and move docs to car

fix these debugging scripts

fix docs

* bump opendbc and panda

forgot panda
old-commit-hash: e735a7f379
2024-08-17 00:54:51 -07:00
Maxime Desroches
c551f1d239 fix bash + python3 shebangs (#33287)
fix shebangs
old-commit-hash: 1dcd660815
2024-08-13 21:22:13 -07:00
Jimmy
937ed5f700 Add testing closet to LogReader source (#33242)
* add testing closet to logreader source

* check source availability

* use internal_source_available
old-commit-hash: 118932acd3
2024-08-11 15:10:14 -07:00
Hoang Bui
6a2b4dedfc tools/logreader: add desc arg to run_across_segments() (#33226)
old-commit-hash: 9e648e428d
2024-08-08 07:51:14 -07:00
Shane Smiskol
7ce33fbe2b process replay: use zst (#33138)
* test

* update refs to zst

* update
old-commit-hash: dfd387520e
2024-07-30 01:46:03 -07:00
Shane Smiskol
7b67994a05 use zstandard library (#33110)
* use zstandard

* run 10

* bye bye

* this was a little overzealous

* clean up
old-commit-hash: 1b3b5ab758
2024-07-28 02:03:47 -07:00
Shane Smiskol
539c828861 tools: keep re pattern file small (#33084)
remove bz2 from re file
old-commit-hash: e68bb26e14
2024-07-25 22:05:10 -07:00
Shane Smiskol
309ccf6482 LogReader: support multiple log extensions in route (#33066)
* support rlog.zst

* TODO
old-commit-hash: e38a1428aa
2024-07-24 14:47:33 -07:00
Kacper Rączy
4efe8d8118 Fix msgq context overrides (#33014)
* Fix msgq context overrides

* Remove unneccesary changes

* add reset_context helper

* Remove redundant import
old-commit-hash: c96dbd5a0b
2024-07-18 20:19:39 -07:00
Shane Smiskol
5333e8ba20 LogReader: show total segment count in missing rlog prompt
old-commit-hash: 3e41b1b287
2024-07-01 22:43:48 -07:00
Maxime Desroches
8adfc992e3 fix pytest tests (#32867)
* fix tests

* no dot
old-commit-hash: 6f10b8a209
2024-06-28 23:58:19 +00:00
StefanLekanic-SYRMIA
5887093eed LogReader: print number of missing rlogs (#32830)
* count rlogs

* minimization changes

* cache to avoid duplicate checks

* no real point in checking qlogs to just show the warning

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 504dd4543d
2024-06-26 22:15:13 -07:00
Shane Smiskol
9f50db15c0 LogReader: try zst on internal source (#32751)
* internal source list files like azure api

* messy but works

* no limit

* simpler

* clean up

* clean up

* clean up

* that's obvious

* better

* we need to unfortunately return a url, so best to take a naive approach for now

* todo

* fix

* clean up
old-commit-hash: b45caf4033
2024-06-14 01:12:19 -07:00
Shane Smiskol
644156420b LogReader: revert broken check
it returns files OR None

old-commit-hash: 68e22faa29
2024-06-14 00:30:14 -07:00
Shane Smiskol
8f402f29a3 SegmentRange: remove timestamp (#32750)
* add deprecation warning

* remove timestamp

* Update logreader.py
old-commit-hash: d5ce4b3085
2024-06-13 23:19:32 -07:00
Shane Smiskol
cd8d440c86 LogReader: support reading zst logs from Azure (#32745)
* staging api

* other stuff

* ugh this should really not be duplicated, we already check the max segnum in Route class

* better error message!!

* more better

* clean up

* clean up

* breaks again
old-commit-hash: a0bbc005b6
2024-06-13 22:43:24 -07:00
Shane Smiskol
ed80e9357a LogReader: improve error messages (#32747)
* better error messages

* clean up
old-commit-hash: 3ede1e2a7c
2024-06-13 22:37:00 -07:00
Shane Smiskol
e88cbb0162 add zstd Python library (#32731)
* add zstd

* fix

* uploader

* logreader: zst support

* caps

* only logreader
old-commit-hash: 04a93dd849
2024-06-12 21:14:23 -07:00
Dean Lee
450d012166 logreader.py: optimize event loading and sorting in _LogFileReader (#32729)
* improve logfilereader

* less changes
old-commit-hash: c64bca97ba
2024-06-12 14:20:06 -07:00
Mauricio Alvarez Leon
008008b03a Update to Python 3.12 (#32548)
* 3.12

* pprofile is broken

* use modified metadrivepy3-12

* 0.3.0 metadrive

* add metadrive/commaai git dependency

* metadrive git set

* pin sounddevice 0.4.6

* datetime.utcnow() deprecation

* poetry lock

* make datetime not aware

* poetry lock

* pin pytools

* google_crc32c wheel

* unpin sounddevice

* clean metadrive

* use python crc

* mypy

* 3.12.4

* allow python3.11

* test

* no pip

* poetry

* better

* better

* merge

* remove

* try

* test

* try  this

* snok

* python

* simpler

* setuptools

* lower

* try

* try

* work?

* ubuntu deps

* ubuntu

* try

* remove

* move

* remove this

* names

* Update .github/workflows/tools_tests.yaml

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* python<4

* <3.13

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>
old-commit-hash: 148eaf8fa6
2024-06-11 20:45:15 +00:00
Adeeb Shihadeh
86b6103657 test_*.py files are no longer executable (#32610)
* test_*.py files are no longer executable

* cleanup empty lines

* update precommit

* fixme

* enable

* build
old-commit-hash: c59887c810
2024-06-03 15:48:56 -07:00
Adeeb Shihadeh
3358745103 ruff: enable UP
old-commit-hash: b2e4c64cf8
2024-05-20 17:43:54 -07:00
cl0cks4fe
639a634f7c Unittest to pytest (#32366)
* add pytest-asyncio

* switch common

* switch selfdrive

* switch system

* switch tools

* small fixes

* fix setUp and valgrind pytest

* switch to setup

* fix random

* switch mock

* switch test_lateral_limits

* revert test_ui

* fix poetry.lock

* add unittest to banned-api

* add inline ignores to remaining unittest imports

* revert test_models

* revert check_can_parser_performance

* one more skip

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: dd9d5d4528
2024-05-17 11:01:44 -07:00
Greg Hogan
a17639bddb logreader: skip internal source if connection refused (#32418)
* logreader: skip internal source if connection refused

* fix indentation

* fix spacing

* explicit ipv4 and tcp
old-commit-hash: 2a46d71fc8
2024-05-13 16:02:28 -07:00
Justin Newberry
6cc3c1ee0f jenkins: publish casync releases from device (#32142)
* publish in ci

* overwrite

* publish in ci

* fix

* test it

* Revert "test it"

This reverts commit b3de51dc693df79b980d7dffc18bcc3c88d83375.

* use right token

* cleanup after uploading

---------

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 8364cd2f2c
2024-04-09 11:38:18 -07:00
Justin Newberry
881ccd321e fix logreader after segment_numbers api removal (#32073)
fix
old-commit-hash: fc9f50c90d
2024-04-01 10:01:50 -07:00
Cameron Clough
c401892058 tools: remove unused timestamp_to_datetime (#31950)
old-commit-hash: 27d2a6066d
2024-03-21 10:58:35 -07:00
Justin Newberry
b899592b18 cars: platform enums -> prepend brand name (#31927)
* with brand name

* migrate

* Fix

* fixes

* more

* passes

* fix

* fix the doc

* collects

* these too

* more stuff

* body exception :/

* more

* hardcode i guess

* update ref

* toyota

* more toyota

* and here

* final!

* fix notebooks and ccs

* move this here
old-commit-hash: b0eb3ba4f4
2024-03-19 17:29:50 -07:00
thenhnn
77d6d1a8f1 PlatformConfig: automatically get platform_str from the enum name (#31868)
* get platform_str from the enum name

* fix tests

* add migration table

* remove impossible todo

* Add link to PR in MIGRATION table

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Remove useless brand name comments and rename RAM_1500 to RAM_1500_5TH_GEN

* rename RAM_HD to RAM_HD_5TH_GEN

* rename references to RAM_HD and RAM_1500

* change "mock" to "MOCK" and rename torque data of Nissan Leaf 2018 IC

* remove MOCK from fingerprints.py

* change hard-coded car model in test_can_fingerprint.py/test_timing

* migration

* update ref

* space

* prius

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: justin newberry <justin@comma.ai>
Co-authored-by: Justin Newberry <jnewberry0502@gmail.com>
old-commit-hash: 489528dcae
2024-03-19 11:16:02 -07:00
Justin Newberry
529699f3fe logreader: fix auto source + interactive modes (#31847)
* interactive modes

* these exceptions don't matter
old-commit-hash: a819292073
2024-03-12 20:04:17 -07:00