Commit Graph

217 Commits

Author SHA1 Message Date
Shane Smiskol
539c828861 tools: keep re pattern file small (#33084)
remove bz2 from re file
old-commit-hash: e68bb26e14d4e85f61ab3f58ac9667ca9d6ea565
2024-07-25 22:05:10 -07:00
Shane Smiskol
309ccf6482 LogReader: support multiple log extensions in route (#33066)
* support rlog.zst

* TODO
old-commit-hash: e38a1428aa20aad134b4ad9fd1abab33a1de70b9
2024-07-24 14:47:33 -07:00
Kacper Rączy
4efe8d8118 Fix msgq context overrides (#33014)
* Fix msgq context overrides

* Remove unneccesary changes

* add reset_context helper

* Remove redundant import
old-commit-hash: c96dbd5a0b681c76b94191d1b928e29eb2b72119
2024-07-18 20:19:39 -07:00
Shane Smiskol
5333e8ba20 LogReader: show total segment count in missing rlog prompt
old-commit-hash: 3e41b1b287c956ed17ab065cf4391aae05333e52
2024-07-01 22:43:48 -07:00
Maxime Desroches
8adfc992e3 fix pytest tests (#32867)
* fix tests

* no dot
old-commit-hash: 6f10b8a20989031d4cc04e8b8ce38eaea4a4824e
2024-06-28 23:58:19 +00:00
StefanLekanic-SYRMIA
5887093eed LogReader: print number of missing rlogs (#32830)
* count rlogs

* minimization changes

* cache to avoid duplicate checks

* no real point in checking qlogs to just show the warning

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 504dd4543dd7fd6c466817faa82cda42b6b55542
2024-06-26 22:15:13 -07:00
Shane Smiskol
9f50db15c0 LogReader: try zst on internal source (#32751)
* internal source list files like azure api

* messy but works

* no limit

* simpler

* clean up

* clean up

* clean up

* that's obvious

* better

* we need to unfortunately return a url, so best to take a naive approach for now

* todo

* fix

* clean up
old-commit-hash: b45caf403366ed4161516e49b9e433805d9de6c4
2024-06-14 01:12:19 -07:00
Shane Smiskol
644156420b LogReader: revert broken check
it returns files OR None

old-commit-hash: 68e22faa296d335f2d497c52339f536f761602c3
2024-06-14 00:30:14 -07:00
Shane Smiskol
8f402f29a3 SegmentRange: remove timestamp (#32750)
* add deprecation warning

* remove timestamp

* Update logreader.py
old-commit-hash: d5ce4b308588f2bd4eee300110ddcd66e787a110
2024-06-13 23:19:32 -07:00
Shane Smiskol
cd8d440c86 LogReader: support reading zst logs from Azure (#32745)
* staging api

* other stuff

* ugh this should really not be duplicated, we already check the max segnum in Route class

* better error message!!

* more better

* clean up

* clean up

* breaks again
old-commit-hash: a0bbc005b62f1e6183ce4fd167e78b4fedb06df0
2024-06-13 22:43:24 -07:00
Shane Smiskol
ed80e9357a LogReader: improve error messages (#32747)
* better error messages

* clean up
old-commit-hash: 3ede1e2a7c850622beef2549315f320aa018105a
2024-06-13 22:37:00 -07:00
Shane Smiskol
e88cbb0162 add zstd Python library (#32731)
* add zstd

* fix

* uploader

* logreader: zst support

* caps

* only logreader
old-commit-hash: 04a93dd849461df7a3496706c6172d95e83149c9
2024-06-12 21:14:23 -07:00
Dean Lee
450d012166 logreader.py: optimize event loading and sorting in _LogFileReader (#32729)
* improve logfilereader

* less changes
old-commit-hash: c64bca97bac68462a63b47d33aedf869d21ac97e
2024-06-12 14:20:06 -07:00
Mauricio Alvarez Leon
008008b03a Update to Python 3.12 (#32548)
* 3.12

* pprofile is broken

* use modified metadrivepy3-12

* 0.3.0 metadrive

* add metadrive/commaai git dependency

* metadrive git set

* pin sounddevice 0.4.6

* datetime.utcnow() deprecation

* poetry lock

* make datetime not aware

* poetry lock

* pin pytools

* google_crc32c wheel

* unpin sounddevice

* clean metadrive

* use python crc

* mypy

* 3.12.4

* allow python3.11

* test

* no pip

* poetry

* better

* better

* merge

* remove

* try

* test

* try  this

* snok

* python

* simpler

* setuptools

* lower

* try

* try

* work?

* ubuntu deps

* ubuntu

* try

* remove

* move

* remove this

* names

* Update .github/workflows/tools_tests.yaml

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* python<4

* <3.13

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>
old-commit-hash: 148eaf8fa6c9b23cdd346e926f58fad61f79ff1d
2024-06-11 20:45:15 +00:00
Adeeb Shihadeh
86b6103657 test_*.py files are no longer executable (#32610)
* test_*.py files are no longer executable

* cleanup empty lines

* update precommit

* fixme

* enable

* build
old-commit-hash: c59887c8109dd8d33ea2e7169995184fbe38cc3a
2024-06-03 15:48:56 -07:00
Adeeb Shihadeh
3358745103 ruff: enable UP
old-commit-hash: b2e4c64cf8fe3256d1e411e8715c29e2eb6f4f00
2024-05-20 17:43:54 -07:00
cl0cks4fe
639a634f7c Unittest to pytest (#32366)
* add pytest-asyncio

* switch common

* switch selfdrive

* switch system

* switch tools

* small fixes

* fix setUp and valgrind pytest

* switch to setup

* fix random

* switch mock

* switch test_lateral_limits

* revert test_ui

* fix poetry.lock

* add unittest to banned-api

* add inline ignores to remaining unittest imports

* revert test_models

* revert check_can_parser_performance

* one more skip

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: dd9d5d4528ff0a61c7699abc1b63ba8fd282632f
2024-05-17 11:01:44 -07:00
Greg Hogan
a17639bddb logreader: skip internal source if connection refused (#32418)
* logreader: skip internal source if connection refused

* fix indentation

* fix spacing

* explicit ipv4 and tcp
old-commit-hash: 2a46d71fc885201d88ca63bc52d26ef715545a23
2024-05-13 16:02:28 -07:00
Justin Newberry
6cc3c1ee0f jenkins: publish casync releases from device (#32142)
* publish in ci

* overwrite

* publish in ci

* fix

* test it

* Revert "test it"

This reverts commit b3de51dc693df79b980d7dffc18bcc3c88d83375.

* use right token

* cleanup after uploading

---------

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 8364cd2f2cc4f4b753fc417333df4c707c11eb86
2024-04-09 11:38:18 -07:00
Justin Newberry
881ccd321e fix logreader after segment_numbers api removal (#32073)
fix
old-commit-hash: fc9f50c90d166ce8bef7a711b2e391071fb0fe02
2024-04-01 10:01:50 -07:00
Cameron Clough
c401892058 tools: remove unused timestamp_to_datetime (#31950)
old-commit-hash: 27d2a6066d944255608d953c9a8fc7c8cb79f828
2024-03-21 10:58:35 -07:00
Justin Newberry
b899592b18 cars: platform enums -> prepend brand name (#31927)
* with brand name

* migrate

* Fix

* fixes

* more

* passes

* fix

* fix the doc

* collects

* these too

* more stuff

* body exception :/

* more

* hardcode i guess

* update ref

* toyota

* more toyota

* and here

* final!

* fix notebooks and ccs

* move this here
old-commit-hash: b0eb3ba4f477142797624580eb3fdb6448f068d6
2024-03-19 17:29:50 -07:00
thenhnn
77d6d1a8f1 PlatformConfig: automatically get platform_str from the enum name (#31868)
* get platform_str from the enum name

* fix tests

* add migration table

* remove impossible todo

* Add link to PR in MIGRATION table

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Remove useless brand name comments and rename RAM_1500 to RAM_1500_5TH_GEN

* rename RAM_HD to RAM_HD_5TH_GEN

* rename references to RAM_HD and RAM_1500

* change "mock" to "MOCK" and rename torque data of Nissan Leaf 2018 IC

* remove MOCK from fingerprints.py

* change hard-coded car model in test_can_fingerprint.py/test_timing

* migration

* update ref

* space

* prius

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: justin newberry <justin@comma.ai>
Co-authored-by: Justin Newberry <jnewberry0502@gmail.com>
old-commit-hash: 489528dcae3888311f92075ba9e0f68626725121
2024-03-19 11:16:02 -07:00
Justin Newberry
529699f3fe logreader: fix auto source + interactive modes (#31847)
* interactive modes

* these exceptions don't matter
old-commit-hash: a8192920737216fe07bd2f273efdee67669b8483
2024-03-12 20:04:17 -07:00
Shane Smiskol
bc7664aa2b LogReader: fix sort by time and union types (#31565)
* fix :(

* test_sort_by_time

* this isn't required

* not slow, and just compare sorted

* messy

* works

* clean up

* clean up

* not here

* clean up

* clean up

* clean up

* makes network call

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
old-commit-hash: e964c5944d4342b6e08ca872eacf9d524a49b479
2024-03-08 02:49:24 -08:00
Adeeb Shihadeh
9735cf2eb6 timeless routes (#31119)
* timeless route

* update sort

* update test

* fix param name
old-commit-hash: 7cfc571f5698c8f5d5bce2e39ccbd8a56c729395
2024-03-05 20:43:33 -08:00
Justin Newberry
6f9f45182b disable commaCarSegments tests (#31632)
disable this test too
old-commit-hash: f4a7e8eae6e4102a35c0ba787d61861bd40ed8f4
2024-02-28 16:47:08 -08:00
Justin Newberry
c799790b9b move with_http_server to selfdrive/test/helpers (#31597)
* move

* fix
old-commit-hash: 9d1cafd0fccc0406a91309a2f9aa1053b498601d
2024-02-26 12:53:17 -08:00
Cameron Clough
9bd90112d0 use pyupgrade to update to new typing syntax (#31580)
* add pyupgrade hook

* run pyupgrade (pre-commit run -a)

* ruff --fix

* Revert "add pyupgrade hook"

This reverts commit 56ec18bb6b8602a0b612f3803d96cdad14b52066.

* revert changes to third_party/

* manual type fixes

* explicit Optional wrapping capnp objects
old-commit-hash: 995250ae4967943ee33a3699a1d89a7a770d95e9
2024-02-24 16:41:23 -08:00
Justin Newberry
cc77d0ca34 segment range docs update (#31560)
* the format

* cleaner
old-commit-hash: 556f9738960526d6eeaab9fe639d0340a6d5d6c9
2024-02-23 11:50:33 -08:00
Greg Hogan
b4479b4155 URLFile: default value test (#31544)
* URLFile: default value test

* cleanup

* fix env

* improvements

* fix GET response

* only delete cache dir if it exists

* env pop

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 43f64ec5005b38b9d2f3ccab9b8bd99a2c1cd6fd
2024-02-22 13:17:02 -08:00
Shane Smiskol
aeb904d5da test_fw_query_on_routes: get first qlog (#31496)
* fast

* this isn't internal

* see
old-commit-hash: c5f1f4c67663161d335048238e6dd5506bde3555
2024-02-16 17:55:15 -08:00
Justin Newberry
8dbdd4736c test_logreader: test internal scenarios (#31484)
* test source scenario

* test source scenario

* fix
old-commit-hash: 16d13395536e84ddf492b2352d827d33befd601e
2024-02-15 12:10:56 -08:00
Justin Newberry
42f292b3ce test_logreader: test interactive mode + fix typo in AUTO_INTERACTIVE (#31481)
* fix spelling

* test interactive

* remove that

* test taht

* move that
old-commit-hash: 8aee0d2af2c9e9c1755afa2b46f68b199958ccdb
2024-02-15 11:24:27 -08:00
Greg Hogan
6c51d1563d URLFile: add typing and internalize pool manager (#31466)
* URLFile: add typing and internalize pool manager

* cleanup
old-commit-hash: e59fe0014af6e97485c89502c747d3117f95e91a
2024-02-14 14:09:01 -08:00
Justin Newberry
edd4649d24 LogReader: add typing hints (#31464)
logreader typing
old-commit-hash: 33cf6bda9ef6b1ed19f3a0fed4a5914a414ae653
2024-02-14 10:34:17 -08:00
Justin Newberry
38fc6f9dd6 LogReader: retain old behavior for direct parsing of files (#31419)
* maintain exception

* test that head is not called

* annoying mock

* test with cache
old-commit-hash: 1436f576df5f43f78b9253b5b2072fbf1414d9be
2024-02-14 10:05:25 -08:00
Shane Smiskol
9d941a9682 SegmentRange: more explicit slice matching (#31451)
* more explicit

* fix it

* use non capturing group

* only needed for regex101

* make test_logreader.py executable

* fix

* stash

* passes test

* none

* type anotate

* test

* fix

* nice for syntax highlighting
old-commit-hash: 796671fe1a9131275c4ff0d67293bb669d0dcd98
2024-02-14 04:36:58 -08:00
Shane Smiskol
5c16ae62d8 SegmentRange: simplify slice (#31455)
* simplify slicing

* rm
old-commit-hash: 8fe9bc7a6977f9f901b1ddf8eb0916ac80224644
2024-02-14 04:17:03 -08:00
Shane Smiskol
ae65a01afd SegmentRange: test API call (#31456)
* test

* better

* better
old-commit-hash: c4f7991bb63ba7953f37eb08f14d227fec8a5524
2024-02-14 03:29:08 -08:00
Shane Smiskol
7afdb2393d SegmentRange: type annotations (#31453)
* type annotate SegmentRange

* proper formatting

* oops

* numpy?

format test too

* draft

* fixed

* clean up

* rm

* more

* clean up

* clean up

* rm

* not here

* revert
old-commit-hash: 82763710097cbfe629502502163a7d2d6319f3fd
2024-02-14 02:05:55 -08:00
Shane Smiskol
4c7ad7f3b7 tools/lib: format code (#31454)
easier to read. pylint used to catch all this stuff, but it's mostly missing in ruff :'(
old-commit-hash: 0846175f44ac9364527087db4ec54767e6884fca
2024-02-14 01:56:17 -08:00
Shane Smiskol
a4e61c1022 SegmentRange: define __repr__
old-commit-hash: 884bd5c7f68018028620821c6b25b5f37e2f7841
2024-02-13 22:58:44 -08:00
Justin Newberry
8b5f1235f4 live_logreader to its own file (#31436)
own file
old-commit-hash: d6762c3035fac31cc9582c685c07441d61da0f83
2024-02-13 13:29:52 -08:00
Adeeb Shihadeh
1efa8db1b4 dongle id is only hex (#31426)
* dongle id is only hex

* so is the count
old-commit-hash: e9071f11988f780063e74c01c11bbb2dd7448bee
2024-02-13 13:01:25 -08:00
Justin Newberry
fbf5ada69f LogReader: live_logreader helpers (#31416)
live helper
old-commit-hash: 0a92c5bf966530335e804a8f240b10ae982bfcb2
2024-02-12 12:58:25 -08:00
Greg Hogan
1384c10fb0 logreader: skip internal source if not available (#31400)
* logreader: skip internal source if not available

* raise exception

* but only when appropriate
old-commit-hash: 667693b8c2ca7ef0b0457823cd76b93b22764038
2024-02-10 00:20:43 -08:00
Justin Newberry
02aea8b757 test_caching: use with_http_server (#31393)
use http server
old-commit-hash: 664a3c86babf84b64ae7856bc25789b165cf1e83
2024-02-09 20:06:11 -08:00
Justin Newberry
bcd22c1683 LogReader: test auto mode fallback (#31390)
* test auto mode

* better

* slow

* better
old-commit-hash: f87322423534a28ef1680bc86ec072b2914fd217
2024-02-09 15:22:43 -08:00
Greg Hogan
cb6a47c6bf simplify URLFile (#31365)
* simplify URLFile

* more space
old-commit-hash: ec9f3dcef367e6c29756ededa7e83783ef280cfa
2024-02-08 10:24:45 -08:00