Commit Graph

60 Commits

Author SHA1 Message Date
Justin Newberry
529699f3fe logreader: fix auto source + interactive modes (#31847)
* interactive modes

* these exceptions don't matter
old-commit-hash: a819292073
2024-03-12 20:04:17 -07:00
Shane Smiskol
bc7664aa2b LogReader: fix sort by time and union types (#31565)
* fix :(

* test_sort_by_time

* this isn't required

* not slow, and just compare sorted

* messy

* works

* clean up

* clean up

* not here

* clean up

* clean up

* clean up

* makes network call

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
old-commit-hash: e964c5944d
2024-03-08 02:49:24 -08:00
Cameron Clough
9bd90112d0 use pyupgrade to update to new typing syntax (#31580)
* add pyupgrade hook

* run pyupgrade (pre-commit run -a)

* ruff --fix

* Revert "add pyupgrade hook"

This reverts commit 56ec18bb6b8602a0b612f3803d96cdad14b52066.

* revert changes to third_party/

* manual type fixes

* explicit Optional wrapping capnp objects
old-commit-hash: 995250ae49
2024-02-24 16:41:23 -08:00
Justin Newberry
8dbdd4736c test_logreader: test internal scenarios (#31484)
* test source scenario

* test source scenario

* fix
old-commit-hash: 16d1339553
2024-02-15 12:10:56 -08:00
Justin Newberry
42f292b3ce test_logreader: test interactive mode + fix typo in AUTO_INTERACTIVE (#31481)
* fix spelling

* test interactive

* remove that

* test taht

* move that
old-commit-hash: 8aee0d2af2
2024-02-15 11:24:27 -08:00
Justin Newberry
edd4649d24 LogReader: add typing hints (#31464)
logreader typing
old-commit-hash: 33cf6bda9e
2024-02-14 10:34:17 -08:00
Justin Newberry
38fc6f9dd6 LogReader: retain old behavior for direct parsing of files (#31419)
* maintain exception

* test that head is not called

* annoying mock

* test with cache
old-commit-hash: 1436f576df
2024-02-14 10:05:25 -08:00
Shane Smiskol
5c16ae62d8 SegmentRange: simplify slice (#31455)
* simplify slicing

* rm
old-commit-hash: 8fe9bc7a69
2024-02-14 04:17:03 -08:00
Shane Smiskol
4c7ad7f3b7 tools/lib: format code (#31454)
easier to read. pylint used to catch all this stuff, but it's mostly missing in ruff :'(
old-commit-hash: 0846175f44
2024-02-14 01:56:17 -08:00
Justin Newberry
8b5f1235f4 live_logreader to its own file (#31436)
own file
old-commit-hash: d6762c3035
2024-02-13 13:29:52 -08:00
Justin Newberry
fbf5ada69f LogReader: live_logreader helpers (#31416)
live helper
old-commit-hash: 0a92c5bf96
2024-02-12 12:58:25 -08:00
Greg Hogan
1384c10fb0 logreader: skip internal source if not available (#31400)
* logreader: skip internal source if not available

* raise exception

* but only when appropriate
old-commit-hash: 667693b8c2
2024-02-10 00:20:43 -08:00
Justin Newberry
9b4b5d3cf5 LogReader: skip file_exists check for comma_api (#31251)
* valid file

* missed this one
old-commit-hash: 6010a39bf8
2024-01-31 16:43:58 -08:00
Justin Newberry
16e0907129 LogReader: add ``.filter`` (#31222)
add filter
old-commit-hash: c782fbc34e
2024-01-29 17:33:47 -08:00
Justin Newberry
223b1ed065 LogReader: add tqdm back to run_across_segments (#31216)
tqdm
old-commit-hash: 7835f9cce0
2024-01-29 13:18:49 -08:00
Justin Newberry
abfa8fcda9 CommaCarSegments: BIG Database! (#31169)
* sanetized selector

* god damn

* add to re

* slog branch

* slog

* source

* big

* not slog

* cleanup

* no space

* works!

* trigger

* main!
old-commit-hash: 824ddffec8
2024-01-27 01:07:29 -08:00
Justin Newberry
9edc902106 LogReader: fail-fast on invalid segments (#31152)
* don't check all

* is
old-commit-hash: edd2428b60
2024-01-25 11:09:09 -08:00
Justin Newberry
7d76ac8eba LogReader: replace get_first_message with LogReader.first (#31146)
* first

* str
old-commit-hash: dd0c474e6c
2024-01-24 10:21:57 -08:00
Justin Newberry
67bb21219c LogReader: no redownloading on multiple iterations (#31141)
* no redownload

* sort
old-commit-hash: 88dcaa51c4
2024-01-23 23:54:11 -08:00
Justin Newberry
0d558d02ee auto_source: add commaCarSegments source (#31137)
add source
old-commit-hash: ac71c30ad5
2024-01-23 21:40:04 -08:00
Justin Newberry
6a1d7fd732 comma_api_source: fix qlogs (#31136)
fix qlogs
old-commit-hash: 3d9b483e10
2024-01-23 20:51:15 -08:00
Justin Newberry
8768be968b LogReader: don't use route for max_segment (#31129)
* dont use route

* fix and cleanup
old-commit-hash: d574013d7b
2024-01-23 14:51:45 -08:00
Justin Newberry
c5c522c984 plotjuggler: auto fallback to qlogs with prompt (#31110)
* qlog fallback

* show a warning for that

* qlog fallback

* fix

* wording
old-commit-hash: 65490bb3c5
2024-01-23 12:53:19 -08:00
Justin Newberry
c03fe6bb13 add example of using the comma car segments database (#31117)
* add vin fingerprinting example

* perhaps!

* remove segment

* easier
old-commit-hash: 389b8ca30d
2024-01-22 18:23:07 -08:00
Justin Newberry
f447062b66 CommaCarSegments: public database of segments for each platform (#31114)
* comma car segments

* comma car segments for test_models

* oneliner

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* better name

* not used here

* sort

* remove print

* better comment

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: caa9153974
2024-01-22 17:33:03 -08:00
Justin Newberry
b29eea47ce sanitizer: sanitize vin number (#31090)
old-commit-hash: 2c86c023fe
2024-01-21 12:22:32 -05:00
Justin Newberry
a9676e7086 auto_source: optimize api calls and use head to determine if file exists (#31025)
* fast

* catch all

* source

* fix file_exists

* remove duplicate reset

* test multiple loops

* iterations

* cleanup imports
old-commit-hash: 2967cada71
2024-01-16 17:24:37 -08:00
Justin Newberry
5df01b3d22 LogReader: fix plotjuggler not reading properly (#31023)
* fix plotjuggler

* simpler

* fix juggler

* fix that
old-commit-hash: 61fe705160
2024-01-16 12:40:49 -08:00
Justin Newberry
4b7edec2b0 LogReader: move opci to tools/lib (#31021)
move opci
old-commit-hash: 7d400112df
2024-01-16 10:58:51 -08:00
Justin Newberry
35dcd7ccd8 LogReader: support lists of identifiers (#31020)
* support lists of logs

* mark as slow instead
old-commit-hash: 0739d5d7b2
2024-01-16 10:40:00 -08:00
Justin Newberry
13e7366b6d LogReader: support only_union_types (#31019)
support only-union-types
old-commit-hash: 4da8d16514
2024-01-16 10:03:55 -08:00
Justin Newberry
c8cc30112d rename segmentrangereader to logreader (#30981)
* Replace multilogiterator

* replace logreader

* update readme

* fix from_Bytes

* new section

* reset before iter

* selector

* fix internal

* whitespace
old-commit-hash: 3d73512c07
2024-01-12 12:53:50 -08:00
Justin Newberry
a729e102c7 Replace multilogiterator (#30980)
old-commit-hash: 9f1b72ac79
2024-01-12 11:43:32 -08:00
Justin Newberry
ef1acd4e6c Proc Replay: cleanup iterable log typing (#30261)
* cleanup log typing

* use iterable

* fix those

* Cleanup

* cleanup

* missed that one
old-commit-hash: 1820b5f259
2023-10-17 12:58:28 -07:00
Adeeb Shihadeh
16ec56398c add openpilot prefix to imports (#29498)
* add openpilot prefix to imports

* more

* more

* fix docs

* fix linter

* bump submodules

* fix patched tests

* update dynamic imports

* debug

* Revert "debug"

This reverts commit db5e13b9911cc74438bee123bc3430da6c31b24b.

* fix pm test
old-commit-hash: a9626f95b6
2023-08-20 20:49:55 -07:00
Justin Newberry
d487c0501f Ruff: enable most of bugbear (#29320)
* added mutable default args

* most of the Bs

* add comment about lrucache
old-commit-hash: 62c1e65924
2023-08-11 15:33:49 -07:00
Lukas Petersson
a16a20005b process replay: logreader as bytes (#24610)
* willem's changes

* classmethod for bytes

* submodules

* submodules

* Update tools/lib/logreader.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* add back files

* little cleanup

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 86f73a507e
2022-06-01 22:02:42 -07:00
Willem Melching
f957c16a7b make normal logreader more robust (#24577)
old-commit-hash: 194e5fdf1c
2022-05-18 12:21:01 +02:00
Lukas Petersson
2d6ae65ea7 Latency logger bad data robustness (#24343)
* robustness against bad data

* sort by time logreader

* loop until last frameid
old-commit-hash: 1439867d16
2022-04-29 13:32:21 -07:00
Lukas Petersson
df8476118b Latency logging 2 (#24058)
* msg_order and gantt

* frameId in long/lat planner

* track frame id

* controls frame id

* graph tracked events

* graph json

* cloudlog timestamp

* c++ cloudlog

* add frame id

* bug fixes

* bug fixes

* frame id visionicp

* bug fixes and debug level

* timestamp log placement

* print timestamps in table

* translate events

* more logging

* bug fixes

* daemon boardd

* print logs with boardd

* more timestamp logs

* cleanup

* remove publish logs

* bug fix

* timestamp received

* timestamp received

* bug fixes

* use json lib

* ignore driver camera

* prep for new timestamp pipeline

* bug fix

* read new pipeline unfinnished

* read new pipeline

* bug fix

* add frame to controlsstate

* remove controlsstate

* print

* cleanup

* more cleanup + bug fix

* clock build issue

* remove unused imports

* format durations

* increase speed

* pr comments fixes

* conflicts

* set MANAGER_DAEMON for boardd

* clean script code

* bug fix + argparse

* remove rcv time

* bug fixes

* print without tabulate

* fix pre-commits

* plot gnatt

* color bug fix

* read without timestampextra

* bump panda

* mono time instead of frame id

* finnish script

* clean unused

* clean unused logging

* monotonic + json fixes

* del test

* remove whilelines

* bump laika

* cleanup

* remove deps

* logs nicer strings

* remove plotting from scirpt

* reset pipfile

* reset pipfile

* nicer strings

* bug fix

* bug fix

* pr comments cleaning

* remove plotting

* bug fix

* new demo route

* bump opendbc and panda

* cereal master

* cereal master

* script less komplex

* assertions

* matplotlib

* readme

* Update README.md

* graph html

* design fixes

* more code design

* Update common/logging_extra.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* whitespace

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update tools/latency_logger/latency_logger.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* pr comments

* bug fix

* readme + env once

* clean swaglog

* bug fix

* Update tools/latencylogger/README.md

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* revert

* revert

* clean swaglog with error

* remove typo file

* revert graph

* cereal

* submodules

* whitespaces

* update refs

Co-authored-by: Bruce Wayne <batman@workstation-openpilot2.internal>
Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 65fca83abe
2022-04-05 21:05:45 -07:00
Adeeb Shihadeh
6f88ca1c6f MultiLogIterator: add function to reset
old-commit-hash: c95ec211e1
2022-03-25 19:56:22 -07:00
Adeeb Shihadeh
8a682597ca can_print_changes improvements
old-commit-hash: 1680f1e251
2022-03-10 14:32:19 -08:00
Greg Hogan
0b659ce9b6 eliminate xx filereader (#23514)
* eliminate xx filereader

* framereader cleanup
old-commit-hash: d5f4fdb1db
2022-01-13 15:56:54 -08:00
Adeeb Shihadeh
ba30137b2a MultiLogIterator: fix after adding sort_by_time
old-commit-hash: bbb47dc860
2022-01-02 20:08:57 -08:00
Adeeb Shihadeh
c036e64c86 MultiLogIterator: remove wraparound mode (#23360)
old-commit-hash: 92f194decc
2022-01-02 15:16:44 -08:00
Greg Hogan
2f843c3023 LogReader: add arg to sort by time (#23346)
* logreader optional sort by time

* robust logreader sort by time option
old-commit-hash: 9992ea1ae8
2022-01-02 11:56:53 -08:00
Ewout ter Hoeven
f206ebd054 Pyupgrade 3.6: Update syntax with Python 3.6+ features (#23305)
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default

Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>

Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
old-commit-hash: 332f568a82
2021-12-24 11:18:39 -08:00
Adeeb Shihadeh
0827af77e5 MultiLogIterator: default wraparound to false
old-commit-hash: c8356d3b8f
2021-12-02 08:14:48 -08:00
Adeeb Shihadeh
6c90a5d445 logreader: remove log path print
old-commit-hash: 01b99eae35
2021-11-29 13:44:20 -08:00
Greg Hogan
44f03a9401 fix log values with non-utf8 characters and remove index_log (#20173)
* remove index_log

* handle non-unicode characters in values

* dump.py better handling of printing non-unicode characters

* only replace errors in main
old-commit-hash: 78808961c5
2021-03-01 13:03:44 -08:00