Commit Graph

133 Commits

Author SHA1 Message Date
Justin Newberry
be814ed447 SegmentRangeReader: log type selector (#30960)
* log type selector

* test this too

* and this

* invalid selector

* use strenum

* don't hardcode size

* oneline
2024-01-11 10:25:24 -08:00
Justin Newberry
cfb23eb2d1 segmentrangereader: support sort_by_time (#30954)
support sort-by-time
2024-01-10 13:36:55 -08:00
Justin Newberry
0d126e1e9e SegmentRangeReader: new format for reading multiple segments (#30940)
* segment range reader

* rename that

* revert that

* cleanup

* revert this for now

* revert this for now

* Fix + test

* rm that

* rm that

* use for auto_fingerprint

* simpler

* for notebook too

* match numpy indexing

* just use numpy directly

* remove that

* spacing

* spacing

* use qlog for auto fingerprint

* add 'read mode'

* pass in read mode

* add test for modes

* numpy indexing

* fix that case

* more examples

* fix the notebook

* cleanup the notebook

* cleaner

* fix those
2024-01-10 11:55:19 -08:00
Justin Newberry
fba521ecc6 url_file: fix non-200 files being cached (#30886)
* fix + test

* fix unclosed

* easier to read

Co-authored-by: Shane Smiskol <shane@smiskol.com>

* fix that

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2024-01-02 12:42:08 -08:00
Adeeb Shihadeh
51a93cf9aa fix unused 2023-12-17 16:32:56 -08:00
royjr
70624ffc81 dependency: remove pycurl package (#30771)
* Update qcomgpsd.py

* Update url_file.py

* remove pycurl

* requests -> urllib3

* unused

* redundant

* fix import

* Revert "requests -> urllib3"

This reverts commit 7ca39e618917986f50356519d2795a432e3a5280.

* headless

* fix trail

* use requests.exceptions.RequestException

* use fp.tell

* fix indents

* reorder imports

* change timeout

* fix debug timing

* remove exception

* add timeout

* missing headers

* move to constructor

* move import

* unused import

* fix debug

* try

* no retries
2023-12-17 16:25:25 -08:00
Greg Hogan
db35dcd0b5 replace common.file_helpers.mkdirs_exists_ok with python os.makedirs funtion (#30618)
replace common.file_helpers.mkdirs_exists_ok with python os.makedirs function
2023-12-06 09:55:29 -08:00
Justin Newberry
4086795c29 Pytest: more post-test path cleanup (#30556)
* test cleanup

* missed this one

* remove that

* static class
2023-11-30 09:55:48 -08:00
Kacper Rączy
c27e977475 URLFile: exception type for failed requests (#30330)
URLFileException for URLFile request errors
2023-10-25 16:20:12 -07:00
Greg Hogan
68acb26aa2 framereader: eliminate tempfile usage (#30289) 2023-10-18 19:48:26 -07:00
Greg Hogan
59e947dc6c fix ffprobe on device (#30286)
* fix ffprobe on device

* use FileReader

* no more requests
2023-10-18 18:31:14 -07:00
Justin Newberry
1820b5f259 Proc Replay: cleanup iterable log typing (#30261)
* cleanup log typing

* use iterable

* fix those

* Cleanup

* cleanup

* missed that one
2023-10-17 12:58:28 -07:00
Greg Hogan
6f98a987af vidindex improvements (#30196)
* vidindex improvements

* fix spelling
2023-10-05 22:48:50 -07:00
Greg Hogan
86b90a8ba2 filereader cleanup (#30191)
* filereader cleanup

* make name return url

* remove unused imports
2023-10-05 19:45:46 -07:00
Greg Hogan
f8e488f881 convert vidindex to python (#30176)
* convert vidindex to python

* fix whitespace

* corrupt file option

* fix up typings

* fix return type

* update framereader

* change length delimiter to uint32 value

* change length to uint32 value

* move url_file changes to separate PR

* cleanup caching

* revert whitespace change

* fix frame type param type
2023-10-05 18:40:24 -07:00
Justin Newberry
fa51bbc236 URLFile: don't cache non-existent file's lengths (#30071)
* urlfile empty

* simplify with mock

* better test name

* PR cleanup

* cleanup the length file
2023-10-04 12:57:56 -07:00
YassineYousfi
880157f5a6 framereader: add cache_dir argument (#29904)
* framereader: add cache_dir argument

* make it an env var
2023-09-14 14:36:32 -07:00
Greg Hogan
140a4ba558 importing auth_config.py should not have side effects (#29893)
import should not create directory
2023-09-12 18:00:18 +00:00
Justin Newberry
73eda51a11 System: use paths for download folders too (#29818)
* use paths for download folders too

* trailing slash

* reset that
2023-09-07 12:49:03 -07:00
Justin Newberry
4e69937d0d Precommit: ensure executable bit is set (#29784)
* precommit ensure executable

* exclude tinygrad

* bump submodules

* exclude tinygrad globally
2023-09-05 16:33:26 -07:00
Justin Newberry
405d09714c Tests: expand temporary directory helpers for more directories (#29735)
* create helpers for testing

* document it

* fix other tests
2023-08-31 14:00:31 -07:00
Justin Newberry
5dc7028f91 Tests: logmessaged reduce global state (#29680)
* reduce global state

* use a decorator here too

* use that one

* use base imports

* ipchandler in setup

* move to common dir

* move to helpers

* wip

* fix the decorator
2023-08-29 13:36:26 -07:00
Justin Newberry
dd9b0231c6 Tests: test_caching reduce global state (#29681) 2023-08-29 10:06:32 -07:00
Justin Newberry
1ee6ed4b16 Linter: remove pylint (#29611)
remove pylint
2023-08-24 16:30:54 -07:00
Dean Lee
21da5a216f cpplint: add filter build/include_subdir (#29585) 2023-08-24 15:42:17 +01:00
Adeeb Shihadeh
a9626f95b6 add openpilot prefix to imports (#29498)
* add openpilot prefix to imports

* more

* more

* fix docs

* fix linter

* bump submodules

* fix patched tests

* update dynamic imports

* debug

* Revert "debug"

This reverts commit db5e13b9911cc74438bee123bc3430da6c31b24b.

* fix pm test
2023-08-20 20:49:55 -07:00
Adeeb Shihadeh
b11b14aff8 CI: enable PYTHONWARNINGS=error (#29364)
* CI: enable PYTHONWARNINGS=error

* fixes

* more ignore

* more modeld ignore
2023-08-12 17:15:16 -07:00
Justin Newberry
62c1e65924 Ruff: enable most of bugbear (#29320)
* added mutable default args

* most of the Bs

* add comment about lrucache
2023-08-11 15:33:49 -07:00
Justin Newberry
8793cbff40 Ruff: enable flake8-builtins (#29315)
* enable flake8-builtins

* replace any with contains

* fix typo in pack

* fix type

* format is from the parent module, has to be enabled

* item_id

* fix item_id

* disable for id since that's what the remote server returns
2023-08-10 18:58:22 -07:00
Justin Newberry
af3ed37c5c Ruff: flake8 pie (misc lints) (#29318)
flake8 pie
2023-08-10 15:28:51 -07:00
Shane Smiskol
acc732efbb replay: new demo route (#29215)
* new demo route

* replace other references (except gps test)

* Apply suggestions from code review

* it's 13 segments
2023-08-03 13:18:44 -07:00
Kacper Rączy
64377a88f9 process_replay: helpers (#28367)
* process_replay helper

* Refactor

* refactor check_enabled uses

* fix __init__

* noqa in __init__ imports

* Move compare_logs outside of process_replay

* Move save_logs to tools.lib.helpers

* Remove save_log from compare_logs

* fix lint issues
2023-06-06 10:31:13 -07:00
Adeeb Shihadeh
94eb215980 loggerd: move to system/ (#27534) 2023-03-08 11:20:49 -08:00
Adeeb Shihadeh
285fd56a1d bootlog: add helper to get a bootlog by segment id 2022-10-04 17:47:37 -07:00
Adeeb Shihadeh
c6b749fb96 add pyside2 package (#25602) 2022-08-30 15:10:52 -07:00
Adeeb Shihadeh
6590fb2b93 pre-commit: add codespell (#25571) 2022-08-30 11:20:55 -07:00
Mitchell Goff
acbb230e0a Update DATA_ENDPOINT default to data-raw.comma.internal (#25597) 2022-08-29 13:31:43 -07:00
Adeeb Shihadeh
e6e8607306 implement string representation for Bootlog 2022-07-21 14:14:34 -07:00
Adeeb Shihadeh
1139fe507b Move selfdrive/hardware/ to system/ (#24725)
* move hardware to system/

* fix mypy
2022-06-11 16:38:24 -07:00
Lukas Petersson
86f73a507e process replay: logreader as bytes (#24610)
* willem's changes

* classmethod for bytes

* submodules

* submodules

* Update tools/lib/logreader.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* add back files

* little cleanup

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-06-01 22:02:42 -07:00
George Hotz
ea5b8cdfb1 nv12: encoderd avoids a full frame copy (#24519)
* rgb to nv12

* nv12 works (w memcpy)

* correct now

* no copy

* fix nv12 with fast debayer

* reverts of unused stuff

* ui use nv12

* comment out thumbnails for now

* rebase fix

* dm read nv12

* model read nv12

* fix ffmpeg encoder

* thumbnails from nv12

* replay to nv12

* python framereader support nv12

* remove hardcoded frame/buffer sizes

* fix build

* ffmpeg encoder fix buffers

* small cleanup

* reduce power usage test

* fix cpu usage test

* fix snapshot

* fix loggerd test

* bump cereal

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Joost Wooning <jwooning@gmail.com>
2022-06-01 17:18:28 +02:00
Willem Melching
194e5fdf1c make normal logreader more robust (#24577) 2022-05-18 12:21:01 +02:00
ntegan1
07f7570840 Tools: allow using plotjuggler on local data (#24473)
local juggler
2022-05-17 19:50:23 +02:00
Dylan Herman
d2eef1955c add mypy check for return-any (#24379)
* add mypy check for return-any

* remove unused import

* typing

* remove unnecessary variable typing

* cleanup

* cleanup

* bump submodules

* small fixes

* only a problem on mac

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2022-05-12 14:59:00 +02:00
grekiki
17e33978cd Mypy fixes for --check-untyped-defs (#24372)
more type fixes
2022-05-12 14:08:11 +02:00
Shane Smiskol
6163dd5ca0 URLFile: raise exception if remote URL doesn't exist when using cache (#24432)
* URLFile returns empty bytes if using cache and remote file doesn't exist

* better exception

* assert on cached files
2022-05-09 13:31:55 -07:00
ntegan1
bf269bd883 Tools: allow uncompressed logs (#24471) 2022-05-09 12:07:19 -07:00
Lukas Petersson
1439867d16 Latency logger bad data robustness (#24343)
* robustness against bad data

* sort by time logreader

* loop until last frameid
2022-04-29 13:32:21 -07:00
grekiki
049b163ea0 mypy: enable --warn-unused-ignores (#24287)
* this is the easy part to check

* Not sure what this fixes, 8aaf00c6a4

* bump laika

* works here

* cleanup

* add types-certifi

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-04-27 14:26:15 -07:00
Lukas Petersson
65fca83abe Latency logging 2 (#24058)
* msg_order and gantt

* frameId in long/lat planner

* track frame id

* controls frame id

* graph tracked events

* graph json

* cloudlog timestamp

* c++ cloudlog

* add frame id

* bug fixes

* bug fixes

* frame id visionicp

* bug fixes and debug level

* timestamp log placement

* print timestamps in table

* translate events

* more logging

* bug fixes

* daemon boardd

* print logs with boardd

* more timestamp logs

* cleanup

* remove publish logs

* bug fix

* timestamp received

* timestamp received

* bug fixes

* use json lib

* ignore driver camera

* prep for new timestamp pipeline

* bug fix

* read new pipeline unfinnished

* read new pipeline

* bug fix

* add frame to controlsstate

* remove controlsstate

* print

* cleanup

* more cleanup + bug fix

* clock build issue

* remove unused imports

* format durations

* increase speed

* pr comments fixes

* conflicts

* set MANAGER_DAEMON for boardd

* clean script code

* bug fix + argparse

* remove rcv time

* bug fixes

* print without tabulate

* fix pre-commits

* plot gnatt

* color bug fix

* read without timestampextra

* bump panda

* mono time instead of frame id

* finnish script

* clean unused

* clean unused logging

* monotonic + json fixes

* del test

* remove whilelines

* bump laika

* cleanup

* remove deps

* logs nicer strings

* remove plotting from scirpt

* reset pipfile

* reset pipfile

* nicer strings

* bug fix

* bug fix

* pr comments cleaning

* remove plotting

* bug fix

* new demo route

* bump opendbc and panda

* cereal master

* cereal master

* script less komplex

* assertions

* matplotlib

* readme

* Update README.md

* graph html

* design fixes

* more code design

* Update common/logging_extra.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* whitespace

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update tools/latency_logger/latency_logger.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* pr comments

* bug fix

* readme + env once

* clean swaglog

* bug fix

* Update tools/latencylogger/README.md

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* revert

* revert

* clean swaglog with error

* remove typo file

* revert graph

* cereal

* submodules

* whitespaces

* update refs

Co-authored-by: Bruce Wayne <batman@workstation-openpilot2.internal>
Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-04-05 21:05:45 -07:00