Commit Graph

96 Commits

Author SHA1 Message Date
Adeeb Shihadeh
e6e8607306 implement string representation for Bootlog 2022-07-21 14:14:34 -07:00
Adeeb Shihadeh
1139fe507b Move selfdrive/hardware/ to system/ (#24725)
* move hardware to system/

* fix mypy
2022-06-11 16:38:24 -07:00
Lukas Petersson
86f73a507e process replay: logreader as bytes (#24610)
* willem's changes

* classmethod for bytes

* submodules

* submodules

* Update tools/lib/logreader.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* add back files

* little cleanup

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-06-01 22:02:42 -07:00
George Hotz
ea5b8cdfb1 nv12: encoderd avoids a full frame copy (#24519)
* rgb to nv12

* nv12 works (w memcpy)

* correct now

* no copy

* fix nv12 with fast debayer

* reverts of unused stuff

* ui use nv12

* comment out thumbnails for now

* rebase fix

* dm read nv12

* model read nv12

* fix ffmpeg encoder

* thumbnails from nv12

* replay to nv12

* python framereader support nv12

* remove hardcoded frame/buffer sizes

* fix build

* ffmpeg encoder fix buffers

* small cleanup

* reduce power usage test

* fix cpu usage test

* fix snapshot

* fix loggerd test

* bump cereal

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Joost Wooning <jwooning@gmail.com>
2022-06-01 17:18:28 +02:00
Willem Melching
194e5fdf1c make normal logreader more robust (#24577) 2022-05-18 12:21:01 +02:00
ntegan1
07f7570840 Tools: allow using plotjuggler on local data (#24473)
local juggler
2022-05-17 19:50:23 +02:00
Dylan Herman
d2eef1955c add mypy check for return-any (#24379)
* add mypy check for return-any

* remove unused import

* typing

* remove unnecessary variable typing

* cleanup

* cleanup

* bump submodules

* small fixes

* only a problem on mac

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2022-05-12 14:59:00 +02:00
grekiki
17e33978cd Mypy fixes for --check-untyped-defs (#24372)
more type fixes
2022-05-12 14:08:11 +02:00
Shane Smiskol
6163dd5ca0 URLFile: raise exception if remote URL doesn't exist when using cache (#24432)
* URLFile returns empty bytes if using cache and remote file doesn't exist

* better exception

* assert on cached files
2022-05-09 13:31:55 -07:00
ntegan1
bf269bd883 Tools: allow uncompressed logs (#24471) 2022-05-09 12:07:19 -07:00
Lukas Petersson
1439867d16 Latency logger bad data robustness (#24343)
* robustness against bad data

* sort by time logreader

* loop until last frameid
2022-04-29 13:32:21 -07:00
grekiki
049b163ea0 mypy: enable --warn-unused-ignores (#24287)
* this is the easy part to check

* Not sure what this fixes, 8aaf00c6a4

* bump laika

* works here

* cleanup

* add types-certifi

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-04-27 14:26:15 -07:00
Lukas Petersson
65fca83abe Latency logging 2 (#24058)
* msg_order and gantt

* frameId in long/lat planner

* track frame id

* controls frame id

* graph tracked events

* graph json

* cloudlog timestamp

* c++ cloudlog

* add frame id

* bug fixes

* bug fixes

* frame id visionicp

* bug fixes and debug level

* timestamp log placement

* print timestamps in table

* translate events

* more logging

* bug fixes

* daemon boardd

* print logs with boardd

* more timestamp logs

* cleanup

* remove publish logs

* bug fix

* timestamp received

* timestamp received

* bug fixes

* use json lib

* ignore driver camera

* prep for new timestamp pipeline

* bug fix

* read new pipeline unfinnished

* read new pipeline

* bug fix

* add frame to controlsstate

* remove controlsstate

* print

* cleanup

* more cleanup + bug fix

* clock build issue

* remove unused imports

* format durations

* increase speed

* pr comments fixes

* conflicts

* set MANAGER_DAEMON for boardd

* clean script code

* bug fix + argparse

* remove rcv time

* bug fixes

* print without tabulate

* fix pre-commits

* plot gnatt

* color bug fix

* read without timestampextra

* bump panda

* mono time instead of frame id

* finnish script

* clean unused

* clean unused logging

* monotonic + json fixes

* del test

* remove whilelines

* bump laika

* cleanup

* remove deps

* logs nicer strings

* remove plotting from scirpt

* reset pipfile

* reset pipfile

* nicer strings

* bug fix

* bug fix

* pr comments cleaning

* remove plotting

* bug fix

* new demo route

* bump opendbc and panda

* cereal master

* cereal master

* script less komplex

* assertions

* matplotlib

* readme

* Update README.md

* graph html

* design fixes

* more code design

* Update common/logging_extra.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* whitespace

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update tools/latency_logger/latency_logger.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* pr comments

* bug fix

* readme + env once

* clean swaglog

* bug fix

* Update tools/latencylogger/README.md

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* revert

* revert

* clean swaglog with error

* remove typo file

* revert graph

* cereal

* submodules

* whitespaces

* update refs

Co-authored-by: Bruce Wayne <batman@workstation-openpilot2.internal>
Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-04-05 21:05:45 -07:00
Adeeb Shihadeh
c95ec211e1 MultiLogIterator: add function to reset 2022-03-25 19:56:22 -07:00
Adeeb Shihadeh
8af20af66d pylint: enforce indentation (W0311) (#24039)
* pylint: enforce indentation (W0311)

* few more
2022-03-24 23:23:29 -07:00
mitchellgoffpc
5c8c6faa9a Ignore BrokenPipeError 2022-03-22 12:52:25 -07:00
Adeeb Shihadeh
1680f1e251 can_print_changes improvements 2022-03-10 14:32:19 -08:00
xx979xx
da50930450 tools: add MultiLogIterator example to readme (#23917)
* update LogReader doc

use MultiLogIterator to read the entire route, get timestamps and deal with exceptions

* bring back the old example

* clean f-strings

* simplify

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-03-07 10:04:41 -08:00
Greg Hogan
a6b46c311e filereader better name for cd:/ replacement 2022-01-13 18:21:19 -08:00
Greg Hogan
df35ebdceb fix filereader cd:/ replacement 2022-01-13 16:09:56 -08:00
Greg Hogan
d5f4fdb1db eliminate xx filereader (#23514)
* eliminate xx filereader

* framereader cleanup
2022-01-13 15:56:54 -08:00
Adeeb Shihadeh
7d7c0ff35f tools: add bootlog helpers (#23504)
* tools: add bootlog helpers

* this is nice

* types
2022-01-12 21:37:43 -08:00
Greg Hogan
71132edf17 plotjuggler: support segment names (#23263)
* plotjuggler: support segment names

* update docs

* generic parser

* convert segment number to int when parsing

* add SegmentName

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2022-01-04 15:55:46 +01:00
Adeeb Shihadeh
bbb47dc860 MultiLogIterator: fix after adding sort_by_time 2022-01-02 20:08:57 -08:00
Adeeb Shihadeh
92f194decc MultiLogIterator: remove wraparound mode (#23360) 2022-01-02 15:16:44 -08:00
Greg Hogan
9992ea1ae8 LogReader: add arg to sort by time (#23346)
* logreader optional sort by time

* robust logreader sort by time option
2022-01-02 11:56:53 -08:00
Ewout ter Hoeven
332f568a82 Pyupgrade 3.6: Update syntax with Python 3.6+ features (#23305)
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default

Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>

Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
2021-12-24 11:18:39 -08:00
Ewout ter Hoeven
55390d273f Convert format strings strings to f-strings (#23241)
* Convert all text strings to f-strings

Reformats all the text from the old "%-formatted" and .format(...) format to the newer f-string format, as defined in PEP 498. This requires Python 3.6+.

Flynt 0.69 was used to reformat the strings. 120 f-strings were created in 51 files.

F-strings are in general more readable, concise and performant. See also: https://www.python.org/dev/peps/pep-0498/#rationale

* revert pyextra changes

* revert ublox.py

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-12-16 14:58:17 +01:00
Adeeb Shihadeh
c8356d3b8f MultiLogIterator: default wraparound to false 2021-12-02 08:14:48 -08:00
Adeeb Shihadeh
01b99eae35 logreader: remove log path print 2021-11-29 13:44:20 -08:00
Adeeb Shihadeh
08f9316b56 remove dead + duplicate python code 2021-11-28 14:02:06 -08:00
Willem Melching
f1c77f97b9 cleanup unused pip packages (#22947)
* cleanup unused pip packages

* fix linter
2021-11-17 16:23:03 +01:00
Willem Melching
6d6f989b7f RobustLogReader that can recover corrupted bz2 files (#22835)
* LogReader with bzip2 recovery

* only rlogs

* add comment

* plotjuggler should also use robust logreader
2021-11-10 16:41:00 +01:00
Adeeb Shihadeh
ee33f8a8cd tools/lib/route: expose raw file paths 2021-11-04 20:32:02 -07:00
Willem Melching
252f3c1c87 tools/lib/auth.py: support github/apple login (#22766)
* tools/lib/auth.py: support github/apple login

* print some info about logged in user

* add docstring
2021-11-01 15:00:00 +01:00
Willem Melching
b816b5b644 Sphinx docs generation (#22697)
* add sphinx

* switch theme

* Experiment: sphinx docs generation updated (#22708)

* moved build to root gitignore, added .gitkeep

* Improved makefile doc build process

- Removed auto-generated docs from source control
- Moved apidoc.sh into Makefile
- Removed make.bat (can add back if Windows support desired)
- Added sphinx viewcode and markdown extensions
- Added feature to source /docs in build, so any .rst file in /docs will
  override the respective file during the build process
- Added feature to copy all markdown/rst files from source into /build/
  during build process so they can be easily referenced while writing
  docs (see examples in index.md)
- Wrote basic starter index.md file

TODO: Add new dependencies to Pipfile [dev-packages]

* Revert accidental modification to Pipfile

* fix command substitution

* exclude xx

* improve docs

* dont include all docs in release build

* Add dockerfile

* update title

* include normal readme

* build container in CI

* use buildkit

* add login

Co-authored-by: Chad Bailey <chadbailey904@gmail.com>
2021-10-28 15:14:37 +02:00
Adeeb Shihadeh
431f6eac95 docs for tools/lib 2021-10-12 20:58:04 -07:00
Adeeb Shihadeh
b6514a5cb3 lib/route: support for e/d cams (#22529)
* lib/route: support for e/d cams

* comment
2021-10-12 00:34:53 -07:00
Dean Lee
c43d35a04d remove tools/lib/route_framereader.py (#22486) 2021-10-08 11:10:44 +02:00
Willem Melching
a31c6ce3f9 tools: make auth optional for public routes (#22386)
* tools: make auth optional for public routes

* handle 403

* show warning from c++ replay
2021-09-30 20:13:46 +02:00
George Hotz
3a6c8d4678 Fix test relying on SAFETY.md being in the root 2021-09-14 16:02:36 -07:00
Willem Melching
dddab597bc Less open file permissions (#21922)
* Less open file permissions

* add test back

* remove params test for permissions

* remove umask

* bump cereal

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-08-28 18:25:05 -07:00
arfy slowy
d74def61f8 fix: spelling typos (#21861)
* fix: typo spelling grammar

* revert

* Update selfdrive/locationd/calibrationd.py

* more revert

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-08-05 12:05:49 -07:00
Greg Hogan
8f1806602c FrameIterator that uses 1+ GB less RAM (#21687) 2021-07-22 12:08:56 -07:00
Adeeb Shihadeh
594877e4a8 fix route_framereader when segments are missing 2021-06-10 23:12:56 -07:00
Adeeb Shihadeh
4b6ec3bca6 unlogger: put YUV vipc send behind flag 2021-06-03 21:53:16 -07:00
Josh Smith
77321dbac4 Add type hints, small cleanups (#21080)
* improve tools.lib.kbhit and tools.sim.lib.keyboard_ctrl

* unpack more efficiently

* minor improvements

* agnos.py match spec better

* manual_ctrl test missing queue arg

* fix incorrect type annotation

* queues are generic

* varname reuse resulting in incorrect type inference

* bytes().hex() rather than bytes.hex(bytes())

* a bit of type hinting stuff
2021-06-03 12:21:04 +02:00
Willem Melching
476ee91845 fix route framereader when using signed urls (#21029) 2021-05-26 11:36:53 +02:00
Mitchell Goff
0dc27e3d6d Updated RouteFrameReader to work on routes with missing segments (#20971) 2021-05-23 14:15:48 -07:00
Willem Melching
0252072729 tools/lib/auth.py update oauth redirect url 2021-05-17 14:51:29 +02:00