Commit Graph

271 Commits

Author SHA1 Message Date
grekiki
17e33978cd Mypy fixes for --check-untyped-defs (#24372)
more type fixes
2022-05-12 14:08:11 +02:00
Willem Melching
75f2149267 encoderd: update deprecated ffmpeg functions (#24501) 2022-05-12 12:43:07 +02:00
George Hotz
bd0cc655fb encoderd: support remote encoder behind env var (#24490)
* remote encoder

* remote encoder on PC, fix type

Co-authored-by: Comma Device <device@comma.ai>
2022-05-10 16:53:47 -07:00
George Hotz
2008db47c2 encoderd: make work on PC (#24483)
* don't use the codec in video_writer

* this produces broken videos for some reason

* bugfix

* refactor on the class

* works on device

* fix codec

* no codec enum

* fix pc

* move into dirs

* these includes also

* rename it ffmpegencoder

* add avcodec_close

Co-authored-by: Comma Device <device@comma.ai>
2022-05-10 15:20:51 -07:00
Willem Melching
ba89faa5f7 uploader: ensure requests.put gets file like object (#24462) 2022-05-07 15:42:47 +02:00
George Hotz
ac343433a2 sync qcam p frames (#24459)
Co-authored-by: Comma Device <device@comma.ai>
2022-05-06 21:47:58 -07:00
Joost Wooning
df8f024e19 uploader: size limit for automatic uploading of qlogs/qcams (#24403)
* uploader: size limit for automatic uploading of qlogs/qcams

* move check to add logging

* use constant

* mark as uploaded

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2022-05-06 11:47:58 +02:00
Adeeb Shihadeh
66fbce638f uploader: compress rlogs (#24447) 2022-05-05 19:13:06 -07:00
Willem Melching
77a6f3d034 delay bz2 compression from logging to uploading (#24392)
* remove log_name

* log without compression

* fix tests

* remove extension for bootlog

* another test fix

* uploader compresses

* also compress in athena

* only compress qlog

* more generic check in do_upload

* fix bootlog compression

* lower loggerd cpu usage

* dont link against bz2

* set core affinity to little cluster

* handle old files
2022-05-03 14:09:17 +02:00
George Hotz
0baa4c3e2a loggerd: switch to v4l encoder try 2 (#24380)
* start v4l encoder

* v4l encoder starts

* start and stop

* fill in proper controls

* it dequeued a buffer

* getting bytes

* it made a video

* it does make files

* getting close

* ahh, so that's how dequeue works

* qcam works (no remuxing)

* remuxing works

* we just need to make shutdown and rollover graceful

* graceful destruction

* switch to polling

* should work now

* fix pc build

* refactors, stop properly

* touchups, remove a copy

* add v4l encoder to release

* inlcude file

* move writing to it's own thread

* fix minor memory leak

* block instead of dropping frames

* add counter, fix tests maybe

* better debugging and test print

* print file path in assert

* format string in test

* no more oversized qlogs

* match qcam

* touchups, remove omx encoder

* remove omx include files

* checked ioctl, better debugging, open by name

* unused import

* move linux includes to third_party/linux/include

* simple encoderd

* full packet

* encoderd should be complete

* lagging print

* updates

* name dq thread

* subset idx

* video file writing works

* debug

* potential bugfix

* rotation works

* iframe

* keep writing support

* ci should pass

* loggerd, not encoderd

* remote encoder code

* support remote encoder

* cereal to master, add encoderd

* header no longer required

* put that back there

* realtime

* lower decoder latency

* don't use queue for VisionIpcBufExtra, disable realtime again

* assert all written

* hmm simpler

* only push to to_write if we are writing

* assert timestamp is right

* use at and remove assert

* revert to queue

Co-authored-by: Comma Device <device@comma.ai>
2022-04-30 09:22:52 -07:00
George Hotz
8d24655787 loggerd: Separate writer from encoder (#24257)
* seperate writer from encoder

* video writer in release

* confirm loggerd on PC works, add YUV option to compressed_vipc

* make raw_logger use video_writer

* put this back to master

* close codec

* put this back

* use unique_ptr

Co-authored-by: Comma Device <device@comma.ai>
2022-04-20 14:51:24 -07:00
George Hotz
1795a2ac03 C2: remove some dead code (#24256)
* remove some dead code

* still needed

Co-authored-by: Comma Device <device@comma.ai>
2022-04-18 20:38:03 -07:00
George Hotz
0ac35a656a loggerd: prereqs for deanlee loggerd omx encoder (try 2) (#24252)
* refactor encoders

* fix pc build

* buf_info

Co-authored-by: Comma Device <device@comma.ai>
2022-04-18 18:21:46 -07:00
Adeeb Shihadeh
5c48e7bc86 remove comma two support (#24248)
* remove comma two support

* cleanup release files

* little more

* more libs

* no more gralloc

* add snpe back
2022-04-18 17:55:23 -07:00
Comma Device
df7275f1f7 encodeData: add flags support 2022-04-16 15:54:22 -07:00
George Hotz
35e776d2fd loggerd: add qcam broadcast support, don't recreate pubmaster (#24226)
* add qcam broadcast support, don't recreate pubmaster

* fixed

Co-authored-by: Comma Device <device@comma.ai>
2022-04-16 14:42:36 -07:00
George Hotz
dab978083b loggerd: broadcast encoder data (#24177)
* encode data try 1

* fix pc build

* low quality compressed streamer with pyav

* nvidia streamer

* timestamp support

* fix latency measurement

* refactor

* camerastream updates

* fix linter

* compressed to vipc, clean

* remove print

* touchups

Co-authored-by: Comma Device <device@comma.ai>
2022-04-12 12:08:50 -07:00
George Hotz
cc8430b668 broadcast encode idx packets (#24176)
Co-authored-by: Comma Device <device@comma.ai>
2022-04-10 15:12:58 -07:00
Willem Melching
e7f23ceca2 uploader: azure returns 401 for files already uploaded (#24141) 2022-04-05 20:09:05 -07:00
Adeeb Shihadeh
8af20af66d pylint: enforce indentation (W0311) (#24039)
* pylint: enforce indentation (W0311)

* few more
2022-03-24 23:23:29 -07:00
Adeeb Shihadeh
53909c1757 deleter: handle files (#24023) 2022-03-23 11:31:29 -07:00
Willem Melching
da5a0c41a0 C3: detect metered Android hotspot (#23734)
* C3: detect metered networks

* show in ui

* fix text layout

* bump cereal

* revert ui changes

* set networkMetered

* add athena method

* add metered logging to uploader

* use in athena uploader

* remove param

* use networkmanager properties to set cell to unmetered

* fix indentation

* no need to check

* bump cereal

* review

* bump cereal
2022-03-09 11:36:52 +01:00
Adeeb Shihadeh
3243d1a81e move lock clearing to uploader (#23826)
* move lock clearing to uploader

* test case
2022-02-22 22:30:11 -08:00
Willem Melching
2f00271ce6 omx encoder: move writing into thread (#23771)
* omx encoder: move writing to separate thread

* fix include

* pop

* log buffers sizes

* split copy and write
2022-02-22 13:02:50 +01:00
Willem Melching
724bab8ef3 Update raw_logger/framereader to new new ffmpeg api (#23799)
* Update raw_logger/framereader to new new ffmpeg api

* Can be removed now

* Handled by avcodec_free_context

* handle EAGAIN and clean up
2022-02-21 16:39:32 +01:00
Willem Melching
7765bc2166 improve logging around uploading (#23750)
* improve logging around uploading

* use int

* use raw

* duplicate
2022-02-11 12:58:41 +01:00
Shane Smiskol
e5005b8f44 scons: fix duplicate environments with test flag (#23702)
* fix duplicate environments when built with --test

* lib

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-02-06 14:08:02 -08:00
Adeeb Shihadeh
652ef27a72 loggerd: less error logging 2022-01-24 21:15:14 -08:00
Dean Lee
a4fefdcb94 RawLogger: downscale qcamera before encoding (#23607) 2022-01-24 13:41:44 -08:00
Ryan
aa9e635311 Add more type hinting (#23595)
* Add more type hinting.

* Revert joystick_alert changes.

* Add typing to statsd.

* Update selfdrive/statsd.py

* Update selfdrive/test/test_fingerprints.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-01-21 15:11:16 -08:00
Willem Melching
d3d10e014a Ensure controlsd can engage again in process replay (#23556)
* hacks to make process replay engage again

* dont change that

* enable engagement check again

* chmod +x

* first working regen

* proper logMonoTime

* fix video framerate consistency

* mpc is valid again

* proper alive checks

* revert loggerd change

* ensure engaged in regen, fix managerState, peripheralState

* ubloxRaw is unused

* add submaster config for gm

* regen all services we can run without HW

* fix loggerd

* loggerd: matroska without the extension

* update ref
2022-01-21 13:13:34 +01:00
Dean Lee
22d43211d9 loggerd: make test_loggerd.py run on PC (#23574)
* run on PC

* set frameId

* remove todo

* rawlogger: remove suffix mkv

* remove unused import PC

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-01-19 13:55:02 -08:00
Robbe Derks
1b49ce6ec4 Statsd (#23413)
* device side of statsd

* need to start it

* enable in manager

* add sleep

* cleanup

* remove aggregates for now and standardize on industry terms

* manager needs main

* need to have a try/except

* atomic_write_on_fs_tmp does not work

* cleaner

* use dump

Co-authored-by: Willem Melching <willem.melching@gmail.com>

* one file at a time

* limit amount of files

* move to influx line protocol and cleanup

* needs to be a list

* fix timezone bug

* actually rate limit

* add to release

* normalized origin

* also log deviceType

* more stats

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2022-01-10 15:21:48 +01:00
Ewout ter Hoeven
332f568a82 Pyupgrade 3.6: Update syntax with Python 3.6+ features (#23305)
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default

Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>

Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
2021-12-24 11:18:39 -08:00
Ewout ter Hoeven
55390d273f Convert format strings strings to f-strings (#23241)
* Convert all text strings to f-strings

Reformats all the text from the old "%-formatted" and .format(...) format to the newer f-string format, as defined in PEP 498. This requires Python 3.6+.

Flynt 0.69 was used to reformat the strings. 120 f-strings were created in 51 files.

F-strings are in general more readable, concise and performant. See also: https://www.python.org/dev/peps/pep-0498/#rationale

* revert pyextra changes

* revert ublox.py

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-12-16 14:58:17 +01:00
Dean Lee
f042962933 put logcat in bootlog (#23217)
* put logcat in boot log

* continue

* use std::unique_ptr

* Revert "use std::unique_ptr"

This reverts commit c671f8ca7160691791bade508c4a8eb5cdb73fb8.
2021-12-14 11:54:54 -08:00
Dean Lee
4aa291c406 Use std::unique_ptr to manage memory (#23168)
* use unique_ptr

* trigger ci

* move down loggerdstate
2021-12-13 14:16:53 +01:00
Dean Lee
cfad1f35e8 RawLogger: fix wrong pts (#23205) 2021-12-13 11:46:52 +01:00
Dean Lee
9decd3d8a2 util: move all functions into util namespace (#23203) 2021-12-12 14:42:23 -08:00
Adeeb Shihadeh
5f36b48199 loggerd: fix delayed exit from main thread 2021-12-07 18:22:00 -08:00
Willem Melching
2e0c73fc0c Remove Raw Upload toggle (#23143)
* Remove Raw Upload toggle

* fix tests
2021-12-07 11:25:40 +01:00
Adeeb Shihadeh
f71a03f967 increase loggerd per-cycle service limit to 200 2021-12-06 22:08:17 -08:00
Adeeb Shihadeh
baf1a77ca2 loggerd: don't let a single service clog up the main thread 2021-12-06 19:31:11 -08:00
Dean Lee
ddc612e9f6 LogCameraInfo: remove unused variables (#23142) 2021-12-06 14:25:28 +01:00
Dean Lee
480fb54926 move struct LogCameraInfo into loggerd.h (#23129) 2021-12-04 21:08:55 -08:00
grekiki
6951b3271d Python: Replace more lists with generators (#23116)
* Replace lists with generators v2

* Replace set with {}

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Replace more set() with {}

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-12-03 22:57:19 -08:00
Dean Lee
348d2d2b0d rename yuv streams (#23071)
* rename yuv streams

* bump cereal

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-11-29 18:14:59 -08:00
Willem Melching
f7c46c6949 Do not run code on version.py import (#23063)
* Do not run code on version.py import

* fix athena
2021-11-29 19:38:55 +01:00
Willem Melching
a2f32fd3e1 Clear loggerd locks once on boot (#23060)
* add test for clear_locks

* move to bootlog

* simplify test
2021-11-29 12:30:28 +01:00
Dean Lee
1d323e0fd6 loggerd: add test case for trigger_rotate (#23038)
* test rotate

* remove global LoggerdState
2021-11-26 14:53:09 +01:00