Commit Graph

50 Commits

Author SHA1 Message Date
Justin Newberry
fbf5ada69f LogReader: live_logreader helpers (#31416)
live helper
old-commit-hash: 0a92c5bf96
2024-02-12 12:58:25 -08:00
Greg Hogan
1384c10fb0 logreader: skip internal source if not available (#31400)
* logreader: skip internal source if not available

* raise exception

* but only when appropriate
old-commit-hash: 667693b8c2
2024-02-10 00:20:43 -08:00
Justin Newberry
9b4b5d3cf5 LogReader: skip file_exists check for comma_api (#31251)
* valid file

* missed this one
old-commit-hash: 6010a39bf8
2024-01-31 16:43:58 -08:00
Justin Newberry
16e0907129 LogReader: add ``.filter`` (#31222)
add filter
old-commit-hash: c782fbc34e
2024-01-29 17:33:47 -08:00
Justin Newberry
223b1ed065 LogReader: add tqdm back to run_across_segments (#31216)
tqdm
old-commit-hash: 7835f9cce0
2024-01-29 13:18:49 -08:00
Justin Newberry
abfa8fcda9 CommaCarSegments: BIG Database! (#31169)
* sanetized selector

* god damn

* add to re

* slog branch

* slog

* source

* big

* not slog

* cleanup

* no space

* works!

* trigger

* main!
old-commit-hash: 824ddffec8
2024-01-27 01:07:29 -08:00
Justin Newberry
9edc902106 LogReader: fail-fast on invalid segments (#31152)
* don't check all

* is
old-commit-hash: edd2428b60
2024-01-25 11:09:09 -08:00
Justin Newberry
7d76ac8eba LogReader: replace get_first_message with LogReader.first (#31146)
* first

* str
old-commit-hash: dd0c474e6c
2024-01-24 10:21:57 -08:00
Justin Newberry
67bb21219c LogReader: no redownloading on multiple iterations (#31141)
* no redownload

* sort
old-commit-hash: 88dcaa51c4
2024-01-23 23:54:11 -08:00
Justin Newberry
0d558d02ee auto_source: add commaCarSegments source (#31137)
add source
old-commit-hash: ac71c30ad5
2024-01-23 21:40:04 -08:00
Justin Newberry
6a1d7fd732 comma_api_source: fix qlogs (#31136)
fix qlogs
old-commit-hash: 3d9b483e10
2024-01-23 20:51:15 -08:00
Justin Newberry
8768be968b LogReader: don't use route for max_segment (#31129)
* dont use route

* fix and cleanup
old-commit-hash: d574013d7b
2024-01-23 14:51:45 -08:00
Justin Newberry
c5c522c984 plotjuggler: auto fallback to qlogs with prompt (#31110)
* qlog fallback

* show a warning for that

* qlog fallback

* fix

* wording
old-commit-hash: 65490bb3c5
2024-01-23 12:53:19 -08:00
Justin Newberry
c03fe6bb13 add example of using the comma car segments database (#31117)
* add vin fingerprinting example

* perhaps!

* remove segment

* easier
old-commit-hash: 389b8ca30d
2024-01-22 18:23:07 -08:00
Justin Newberry
f447062b66 CommaCarSegments: public database of segments for each platform (#31114)
* comma car segments

* comma car segments for test_models

* oneliner

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* better name

* not used here

* sort

* remove print

* better comment

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: caa9153974
2024-01-22 17:33:03 -08:00
Justin Newberry
b29eea47ce sanitizer: sanitize vin number (#31090)
old-commit-hash: 2c86c023fe
2024-01-21 12:22:32 -05:00
Justin Newberry
a9676e7086 auto_source: optimize api calls and use head to determine if file exists (#31025)
* fast

* catch all

* source

* fix file_exists

* remove duplicate reset

* test multiple loops

* iterations

* cleanup imports
old-commit-hash: 2967cada71
2024-01-16 17:24:37 -08:00
Justin Newberry
5df01b3d22 LogReader: fix plotjuggler not reading properly (#31023)
* fix plotjuggler

* simpler

* fix juggler

* fix that
old-commit-hash: 61fe705160
2024-01-16 12:40:49 -08:00
Justin Newberry
4b7edec2b0 LogReader: move opci to tools/lib (#31021)
move opci
old-commit-hash: 7d400112df
2024-01-16 10:58:51 -08:00
Justin Newberry
35dcd7ccd8 LogReader: support lists of identifiers (#31020)
* support lists of logs

* mark as slow instead
old-commit-hash: 0739d5d7b2
2024-01-16 10:40:00 -08:00
Justin Newberry
13e7366b6d LogReader: support only_union_types (#31019)
support only-union-types
old-commit-hash: 4da8d16514
2024-01-16 10:03:55 -08:00
Justin Newberry
c8cc30112d rename segmentrangereader to logreader (#30981)
* Replace multilogiterator

* replace logreader

* update readme

* fix from_Bytes

* new section

* reset before iter

* selector

* fix internal

* whitespace
old-commit-hash: 3d73512c07
2024-01-12 12:53:50 -08:00
Justin Newberry
a729e102c7 Replace multilogiterator (#30980)
old-commit-hash: 9f1b72ac79
2024-01-12 11:43:32 -08:00
Justin Newberry
ef1acd4e6c Proc Replay: cleanup iterable log typing (#30261)
* cleanup log typing

* use iterable

* fix those

* Cleanup

* cleanup

* missed that one
old-commit-hash: 1820b5f259
2023-10-17 12:58:28 -07:00
Adeeb Shihadeh
16ec56398c add openpilot prefix to imports (#29498)
* add openpilot prefix to imports

* more

* more

* fix docs

* fix linter

* bump submodules

* fix patched tests

* update dynamic imports

* debug

* Revert "debug"

This reverts commit db5e13b9911cc74438bee123bc3430da6c31b24b.

* fix pm test
old-commit-hash: a9626f95b6
2023-08-20 20:49:55 -07:00
Justin Newberry
d487c0501f Ruff: enable most of bugbear (#29320)
* added mutable default args

* most of the Bs

* add comment about lrucache
old-commit-hash: 62c1e65924
2023-08-11 15:33:49 -07:00
Lukas Petersson
a16a20005b process replay: logreader as bytes (#24610)
* willem's changes

* classmethod for bytes

* submodules

* submodules

* Update tools/lib/logreader.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* add back files

* little cleanup

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 86f73a507e
2022-06-01 22:02:42 -07:00
Willem Melching
f957c16a7b make normal logreader more robust (#24577)
old-commit-hash: 194e5fdf1c
2022-05-18 12:21:01 +02:00
Lukas Petersson
2d6ae65ea7 Latency logger bad data robustness (#24343)
* robustness against bad data

* sort by time logreader

* loop until last frameid
old-commit-hash: 1439867d16
2022-04-29 13:32:21 -07:00
Lukas Petersson
df8476118b Latency logging 2 (#24058)
* msg_order and gantt

* frameId in long/lat planner

* track frame id

* controls frame id

* graph tracked events

* graph json

* cloudlog timestamp

* c++ cloudlog

* add frame id

* bug fixes

* bug fixes

* frame id visionicp

* bug fixes and debug level

* timestamp log placement

* print timestamps in table

* translate events

* more logging

* bug fixes

* daemon boardd

* print logs with boardd

* more timestamp logs

* cleanup

* remove publish logs

* bug fix

* timestamp received

* timestamp received

* bug fixes

* use json lib

* ignore driver camera

* prep for new timestamp pipeline

* bug fix

* read new pipeline unfinnished

* read new pipeline

* bug fix

* add frame to controlsstate

* remove controlsstate

* print

* cleanup

* more cleanup + bug fix

* clock build issue

* remove unused imports

* format durations

* increase speed

* pr comments fixes

* conflicts

* set MANAGER_DAEMON for boardd

* clean script code

* bug fix + argparse

* remove rcv time

* bug fixes

* print without tabulate

* fix pre-commits

* plot gnatt

* color bug fix

* read without timestampextra

* bump panda

* mono time instead of frame id

* finnish script

* clean unused

* clean unused logging

* monotonic + json fixes

* del test

* remove whilelines

* bump laika

* cleanup

* remove deps

* logs nicer strings

* remove plotting from scirpt

* reset pipfile

* reset pipfile

* nicer strings

* bug fix

* bug fix

* pr comments cleaning

* remove plotting

* bug fix

* new demo route

* bump opendbc and panda

* cereal master

* cereal master

* script less komplex

* assertions

* matplotlib

* readme

* Update README.md

* graph html

* design fixes

* more code design

* Update common/logging_extra.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* whitespace

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update tools/latency_logger/latency_logger.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* pr comments

* bug fix

* readme + env once

* clean swaglog

* bug fix

* Update tools/latencylogger/README.md

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* revert

* revert

* clean swaglog with error

* remove typo file

* revert graph

* cereal

* submodules

* whitespaces

* update refs

Co-authored-by: Bruce Wayne <batman@workstation-openpilot2.internal>
Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 65fca83abe
2022-04-05 21:05:45 -07:00
Adeeb Shihadeh
6f88ca1c6f MultiLogIterator: add function to reset
old-commit-hash: c95ec211e1
2022-03-25 19:56:22 -07:00
Adeeb Shihadeh
8a682597ca can_print_changes improvements
old-commit-hash: 1680f1e251
2022-03-10 14:32:19 -08:00
Greg Hogan
0b659ce9b6 eliminate xx filereader (#23514)
* eliminate xx filereader

* framereader cleanup
old-commit-hash: d5f4fdb1db
2022-01-13 15:56:54 -08:00
Adeeb Shihadeh
ba30137b2a MultiLogIterator: fix after adding sort_by_time
old-commit-hash: bbb47dc860
2022-01-02 20:08:57 -08:00
Adeeb Shihadeh
c036e64c86 MultiLogIterator: remove wraparound mode (#23360)
old-commit-hash: 92f194decc
2022-01-02 15:16:44 -08:00
Greg Hogan
2f843c3023 LogReader: add arg to sort by time (#23346)
* logreader optional sort by time

* robust logreader sort by time option
old-commit-hash: 9992ea1ae8
2022-01-02 11:56:53 -08:00
Ewout ter Hoeven
f206ebd054 Pyupgrade 3.6: Update syntax with Python 3.6+ features (#23305)
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default

Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>

Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
old-commit-hash: 332f568a82
2021-12-24 11:18:39 -08:00
Adeeb Shihadeh
0827af77e5 MultiLogIterator: default wraparound to false
old-commit-hash: c8356d3b8f
2021-12-02 08:14:48 -08:00
Adeeb Shihadeh
6c90a5d445 logreader: remove log path print
old-commit-hash: 01b99eae35
2021-11-29 13:44:20 -08:00
Greg Hogan
44f03a9401 fix log values with non-utf8 characters and remove index_log (#20173)
* remove index_log

* handle non-unicode characters in values

* dump.py better handling of printing non-unicode characters

* only replace errors in main
old-commit-hash: 78808961c5
2021-03-01 13:03:44 -08:00
Greg Hogan
d200f048b7 fix pylint errors
old-commit-hash: 1e6c7c94e4
2020-09-03 18:46:13 -07:00
Adeeb
9027018368 Update Flake8 config (#1624)
* update flake8 checks

* add E502

* no whitespace warnings

* fix violations

* no W391

Co-authored-by: Jason Young <jason@comma.ai>
old-commit-hash: 367155168a
2020-06-03 16:13:34 -07:00
George Hotz
d47b2ef576 More LGTM fixups (#1457)
* exclude lib dirs, fix 3 cpp errors

* fix more alerts, exclude mkvparse

Co-authored-by: George Hotz <geohot@gmail.com>
old-commit-hash: e568d3cadc
2020-05-02 20:00:41 -07:00
George Hotz
6fbcde48e5 Using lgtm.com and fixing found alerts (#1452)
* lgtm in readme, and mac nui fixes

* call super inits in radars

* unused imports, dup lines

* more radars, more unused imports

* pass CP into RadarInterfaceBase

* more fixups

* unused imports

* delete unused lines

* ugh, new unused import

Co-authored-by: George Hotz <geohot@gmail.com>
old-commit-hash: 1295cfe06c
2020-05-02 12:07:34 -07:00
Willem Melching
6993b1ecca Build log_index using scons
old-commit-hash: 845abed5db
2020-04-30 16:26:11 -07:00
George Hotz
640c41a3ff Add sentinels to logs and qlogs (#1205)
* sentinel WIP

* logger is now a cc, not a c

* simpler code doesn't touch loggerd.cc

* oops, double lock. and logreader is nicer

old-commit-hash: 2f2b85581c
2020-03-05 14:29:34 -08:00
Andy Haden
03acf56736 Revert "no more xx imports in tools"
This reverts commit 2de79769f0e7d29cc1e4f5544ae1681127ed3c5c.

old-commit-hash: 4a94e507cf
2020-02-25 11:36:02 -08:00
Willem Melching
6929d5b6e0 no more xx imports in tools
old-commit-hash: 1f9485171e
2020-02-25 11:06:27 -08:00
Andy
738c3cca79 Tools: Storage API (#1161)
* filereader

* support URLs in filereader, logreader

* unused

* use route files api; add auth file

* Implement browser auth

* Update readme, fix up cache paths

* Add tests, clear token on 401

* Factor out URLFile

* space

old-commit-hash: c4af05868b
2020-02-24 21:24:54 -05:00
George Hotz
99c5800ea5 merge in tools
old-commit-hash: 29ac3da7b8
2020-01-17 10:07:22 -08:00