Commit Graph

88 Commits

Author SHA1 Message Date
Adeeb Shihadeh
3c48bc47b7 Move a bunch of stuff to system/ part 3 (#24829)
* move swaglog.py

* timezoned

* logmessaged

* version.py

* fix linter
old-commit-hash: 0fce5d9045
2022-06-11 23:19:27 -07:00
Adeeb Shihadeh
9a45a3b99b Move selfdrive/hardware/ to system/ (#24725)
* move hardware to system/

* fix mypy
old-commit-hash: 1139fe507b
2022-06-11 16:38:24 -07:00
Willem Melching
753045c73d less TICI when not needed (#24698)
* less TICI when not needed

* fix process replay

* move reading voltages into hw abstraction layer

* Update selfdrive/hardware/tici/hardware.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/hardware/hw.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/hardware/base.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* rename init function

* Update selfdrive/athena/athenad.py

Co-authored-by: Robbe Derks <robbe.derks@gmail.com>

* Update selfdrive/boardd/boardd.cc

* Apply suggestions from code review

* Update selfdrive/thermald/thermald.py

* update ref

* fix alert width if all cameras are bad

* add ecam to test_loggerd

* bump cereal

* bump cereal

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: Robbe Derks <robbe.derks@gmail.com>
old-commit-hash: f49a9c9fd2
2022-06-02 15:20:51 +02:00
George Hotz
2a6ffaf71d nv12: encoderd avoids a full frame copy (#24519)
* rgb to nv12

* nv12 works (w memcpy)

* correct now

* no copy

* fix nv12 with fast debayer

* reverts of unused stuff

* ui use nv12

* comment out thumbnails for now

* rebase fix

* dm read nv12

* model read nv12

* fix ffmpeg encoder

* thumbnails from nv12

* replay to nv12

* python framereader support nv12

* remove hardcoded frame/buffer sizes

* fix build

* ffmpeg encoder fix buffers

* small cleanup

* reduce power usage test

* fix cpu usage test

* fix snapshot

* fix loggerd test

* bump cereal

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Joost Wooning <jwooning@gmail.com>
old-commit-hash: ea5b8cdfb1
2022-06-01 17:18:28 +02:00
Adeeb Shihadeh
3e8625e50b visionipc python shim (#24660)
old-commit-hash: 32a9ccfc37
2022-05-25 20:49:56 -07:00
Adeeb Shihadeh
3abe307374 encoder test speedup (#24601)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: d87f30ea06
2022-05-19 22:33:17 -07:00
Adeeb Shihadeh
f4675fc260 Merge common/ and selfdrive/common (#24556)
* Merge common/ and selfdrive/common

* fix that

* fix version

* fix unit tests
old-commit-hash: cb8885cffb
2022-05-18 14:11:57 -07:00
Adeeb Shihadeh
fb1c3b0d5d common: replace custom xattr implementation with os module's (#24448)
* use os module's xattr function

* fix that

* handle in helper
old-commit-hash: 81dacbedca
2022-05-15 17:13:50 -07:00
Adeeb Shihadeh
6a8fefd5de loggerd: adjust qcamera threshold, seemed to be tuned for comma two
old-commit-hash: 5a58ebf183
2022-05-14 01:04:46 -07:00
George Hotz
a053d070be encoderd: enable (#24492)
* enable encoderd

* correct enable line

* fix loggerd tests

* fix power draw and cpu tests

* correct cpu for encoderd

* fix a bug, video_writer is shared

* fix issue with not recording dcam

* add recording state

* wooo tests pass. encode id keeps counting

* core 3

* loggerd then encoderd

* stop loggerd first

* core 3 always online

* see the camera when we see encoder packet

* encoderd on small core uses 37%

* remove encoder logic from loggerd

* delete unit test that doesn't really make sense anymore

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 94b9972eb7
2022-05-13 17:56:02 -07:00
Willem Melching
be08693d5c delay bz2 compression from logging to uploading (#24392)
* remove log_name

* log without compression

* fix tests

* remove extension for bootlog

* another test fix

* uploader compresses

* also compress in athena

* only compress qlog

* more generic check in do_upload

* fix bootlog compression

* lower loggerd cpu usage

* dont link against bz2

* set core affinity to little cluster

* handle old files
old-commit-hash: 77a6f3d034
2022-05-03 14:09:17 +02:00
George Hotz
7119a98414 loggerd: switch to v4l encoder try 2 (#24380)
* start v4l encoder

* v4l encoder starts

* start and stop

* fill in proper controls

* it dequeued a buffer

* getting bytes

* it made a video

* it does make files

* getting close

* ahh, so that's how dequeue works

* qcam works (no remuxing)

* remuxing works

* we just need to make shutdown and rollover graceful

* graceful destruction

* switch to polling

* should work now

* fix pc build

* refactors, stop properly

* touchups, remove a copy

* add v4l encoder to release

* inlcude file

* move writing to it's own thread

* fix minor memory leak

* block instead of dropping frames

* add counter, fix tests maybe

* better debugging and test print

* print file path in assert

* format string in test

* no more oversized qlogs

* match qcam

* touchups, remove omx encoder

* remove omx include files

* checked ioctl, better debugging, open by name

* unused import

* move linux includes to third_party/linux/include

* simple encoderd

* full packet

* encoderd should be complete

* lagging print

* updates

* name dq thread

* subset idx

* video file writing works

* debug

* potential bugfix

* rotation works

* iframe

* keep writing support

* ci should pass

* loggerd, not encoderd

* remote encoder code

* support remote encoder

* cereal to master, add encoderd

* header no longer required

* put that back there

* realtime

* lower decoder latency

* don't use queue for VisionIpcBufExtra, disable realtime again

* assert all written

* hmm simpler

* only push to to_write if we are writing

* assert timestamp is right

* use at and remove assert

* revert to queue

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 0baa4c3e2a
2022-04-30 09:22:52 -07:00
Adeeb Shihadeh
5a9644261f remove comma two support (#24248)
* remove comma two support

* cleanup release files

* little more

* more libs

* no more gralloc

* add snpe back
old-commit-hash: 5c48e7bc86
2022-04-18 17:55:23 -07:00
Adeeb Shihadeh
bd71c594a2 pylint: enforce indentation (W0311) (#24039)
* pylint: enforce indentation (W0311)

* few more
old-commit-hash: 8af20af66d
2022-03-24 23:23:29 -07:00
Adeeb Shihadeh
ef2106ebd2 move lock clearing to uploader (#23826)
* move lock clearing to uploader

* test case
old-commit-hash: 3243d1a81e
2022-02-22 22:30:11 -08:00
Dean Lee
75a22388c7 loggerd: make test_loggerd.py run on PC (#23574)
* run on PC

* set frameId

* remove todo

* rawlogger: remove suffix mkv

* remove unused import PC

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 22d43211d9
2022-01-19 13:55:02 -08:00
Ewout ter Hoeven
f206ebd054 Pyupgrade 3.6: Update syntax with Python 3.6+ features (#23305)
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default

Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>

Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
old-commit-hash: 332f568a82
2021-12-24 11:18:39 -08:00
Willem Melching
e7bf88f29f Remove Raw Upload toggle (#23143)
* Remove Raw Upload toggle

* fix tests
old-commit-hash: 2e0c73fc0c
2021-12-07 11:25:40 +01:00
grekiki
9e4cc1f7ff Python: Replace more lists with generators (#23116)
* Replace lists with generators v2

* Replace set with {}

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Replace more set() with {}

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 6951b3271d
2021-12-03 22:57:19 -08:00
Willem Melching
28dd415759 Do not run code on version.py import (#23063)
* Do not run code on version.py import

* fix athena
old-commit-hash: f7c46c6949
2021-11-29 19:38:55 +01:00
Willem Melching
3b8daea061 Clear loggerd locks once on boot (#23060)
* add test for clear_locks

* move to bootlog

* simplify test
old-commit-hash: a2f32fd3e1
2021-11-29 12:30:28 +01:00
Dean Lee
7f0beef092 loggerd: add test case for trigger_rotate (#23038)
* test rotate

* remove global LoggerdState
old-commit-hash: 1d323e0fd6
2021-11-26 14:53:09 +01:00
Dean Lee
c1ef2b7493 loggerd: add test case for sync_encoders (#23020)
old-commit-hash: b5eb02181c
2021-11-24 13:34:16 -08:00
Adeeb Shihadeh
44568de155 faster jenkins (#22973)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: e13630dfee
2021-11-18 23:43:09 -08:00
Willem Melching
425349a55c swaglog.py: respect LOGPRINT env variable (#22950)
* swaglog.py: respect LOGPRINT env variable

* only change handler that prints

* keep usefull manager prints

* only print running list

* revert process changes

* fix test
old-commit-hash: 77b334a698
2021-11-18 16:05:06 +01:00
Adeeb Shihadeh
35bfdce477 loggerd: trigger rotate on frame id instead of frame count (#22848)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: ea761cbbd5
2021-11-13 15:38:10 -08:00
Mayfield
43e1903073 enable wshadow (#22756)
* enable wshadow (#22714)

* fix replay

* more build fixes

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 5246f0231e
2021-11-02 17:08:53 +01:00
Dean Lee
a031b938b0 replay: improve segment download and merge (#22654)
* no-cache mode

* fix test cases build error

* space

* don't create cache dir in no-cache mode

* fix errors in test cases

* no_local_cache_

* set the number of connections by chunk_size

* use size_t instead of int64_t

* add test case for no-cache mode

* rename variables

* fix SIGSEGV

* cleanup

* faster decompressBZ2

* always decompress bz2

* add test cases

* prepare for python interface

* fix test cases build error

* continue

* camera_replay: cache remote file

* protected inheritance

* single option name

* TODO

* test_case for LogReader&FrameReader

* fix wrong require

* test case for FileReader

* cleanup test

* test:fix wrong filename

* check cached file's checksum

* fix mkdir permissions err

cleanup filereader

* remove initialize libav network libraries.

dd

* abort all loading if one failed

* cleanup tests

* use threadpool to limit concurrent downloads

* cache more segments

* merge 3 segments for replay

* one segment uses about 100M of memory

* use segments_need_merge.size()

* shutdown

* fix stuck if exit replay  before keyboard thread started

* load one segment at a time

* small cleanup

* cleanup filereader

* space

* tiny cleanup

* merge master

* cleanup test cases

* use util:create_directories

* cleanup framereader
old-commit-hash: 2b4a477fbc
2021-11-01 11:55:56 +01:00
Willem Melching
0f77dd4506 loggerd: set encodeIdx.valid if frame id is correct (#22634)
* loggerd: set encodeIdx.valid if frame id is correct

* check valid flag in test

* test cleanup

* bump cereal
old-commit-hash: 89f311714c
2021-10-26 16:00:43 +02:00
Dean Lee
66ca3985c9 replay : move utility functions into separate file (#22414)
* move functions into util

* read bz2 into stream

* pre-decompress log in the download thread

* cleanup logreader

* cache sha256 path

* use readBZ2file in test_logger

* Revert "cache sha256 path"

This reverts commit 60459d3ea09a2c80f4560cf95b1ce7d6af59f06d.

* use macro

* use ostringstream

* cleanup readBZ2File

* move precise_nano_sleep into util
old-commit-hash: d28b98c602
2021-10-04 15:39:59 +02:00
Dean Lee
2dac83c02b loggerd: remove obsolete testraw.cc (#22138)
old-commit-hash: 5a8ca8b449
2021-09-06 12:10:12 -07:00
Willem Melching
60e7e09127 Ensure that first segment starts with all cameras on the same frameId (#22096)
* test that segment starts with the same frame id

* improve test

* add sync logic

* atomic max

* review comments
old-commit-hash: a93271a518
2021-09-01 14:28:50 -07:00
Dean Lee
0fac38e34f LoggerState: test cases (#21855)
* test multiple threads writing to log

* fix typo

* use util::getenv

* check INIT_DATA and SENTINEL

* test END_OF_SEGMENT

* add comment

* test multiple threads logging and rotation

* cleanup

* update

* LoggerHandle:quick fix

* cleanup

* revert test_loggerd.py

* refactor test case

* check lock file

* check refcnt after close

* test_runner.cc

d

* int eixt_signal
old-commit-hash: 935cbd3139
2021-08-30 17:12:09 -07:00
Willem Melching
ae4f3c98d5 refactor loggerd: trigger rotate in encoder thread (#21860)
* trigger rotate in eoncode thread

* rotate in time

* lgtm

* dcam trigger rotate on C3

* check trigger rotate field

* Use >=

* add rotator thread

* set cnt to 0

* test encodeIdx is increasing across segments

* test both segmentId and encodeId

* fix encodeIdx

* no thread needed

* add log for failed to encode

* stricter C3 test, dont check start encodeId on C2 dcam

* only update last_camera_seen_tms when considered for rotate

Co-authored-by: deanlee <deanlee3@gmail.com>
old-commit-hash: a39873872d
2021-08-06 11:10:33 +02:00
Adeeb Shihadeh
db62739c00 fix loggerd test failure when camerad is slow to start
old-commit-hash: d748b44f48
2021-08-02 09:10:29 -07:00
Adeeb Shihadeh
dc0c4bd99b debug msg for loggerd tests
old-commit-hash: 48652db706
2021-07-28 14:01:27 -07:00
Adeeb Shihadeh
255dfffd7f fix qlog decimation calculation in loggerd tests
old-commit-hash: d8411e7a62
2021-07-10 23:20:15 -07:00
Adeeb Shihadeh
19bc1b7397 fast jenkins (#21472)
* speed up setup

* speedup onroad test

* .git doesn't like inplace

* tici2

* no inplace

* revert

* fix

* that one too
old-commit-hash: 0bfd994f73
2021-07-02 16:29:57 -07:00
Adeeb Shihadeh
61d3b1f177 disable uploading raw by default (#21128)
old-commit-hash: c76c30fa2d
2021-06-03 15:24:30 -07:00
Willem Melching
8b785f4744 store all files in /sys/fs/pstore in bootlog (#20875)
* store all files in /sys/fs/pstore in bootlog

* whitespace

* fix bootlog test
old-commit-hash: 4742f55749
2021-05-12 11:15:54 +02:00
Dean Lee
e333e4f189 Cleanup selfdrive/ includes (#20822)
* cleanup include path

* continue

* format includes

* fix testraw.cc

* remove include path from SConstruct

* regroup

* rebase master

* almost done

* apply review

* rename FileReader.xx to filereader.xx

* rename Unlogger.x->unlogger.x

* rename FrameReader.xx -> framereader.xx

* apply reviews

* ui.h

* continue

* fix framebuffer.cc build error:mv util.h up

* full path to msm_media_info

* fix qcom2 camerad

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 7222d0f20d
2021-05-08 22:15:17 -07:00
Adeeb Shihadeh
3e4eb2e343 handle unregistered devices (#20728)
* handle unregistered devices

* cleanup

* no uploader or athena

* clean up
old-commit-hash: 1786d04239
2021-04-22 21:03:57 -07:00
Willem Melching
badae8669a More encoder test cleanup (#20623)
* more encode test cleanup

* fix check at end of function
old-commit-hash: c4e2d374c9
2021-04-08 17:48:57 +02:00
Willem Melching
9f84ff75cc check encodeIdx in loggerd test (#20621)
old-commit-hash: ec5465ffd5
2021-04-08 16:50:48 +02:00
Willem Melching
8f4a445171 add putBool/getBool wrappers to cython params class (#20611)
* add putBool/getBool wrappers to cython class

* use new API

* some more puts

* fix mockparams arguments

* add get_bool to MockParams

* typo
old-commit-hash: ae094042ad
2021-04-07 15:36:37 +02:00
Willem Melching
c0098cec09 Manager cleanup (#20231)
* move manager in folder

* inital refactor

* call start

* small cleanup

* add comments

* use self.signal()

* order shouldnt matter

* newlines

* add helpers

* newlines

* add process config

* split out build part of manager

* this should fix most tests

* no sensord on pc

* dont start athena

* remove comment

* fix old athena test

* fix inject model

* fix test car models

* should be not none

* fix helpers exitcode

* ignore manage_athenad

* Use time.monotonic()

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* combine init, remove spinner

* move manager test

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 5a3b511306
2021-03-05 11:03:23 +01:00
Willem Melching
c1606b60fd test loggerd: limit samples to number of services
old-commit-hash: 6e26403cee
2021-02-23 16:45:08 +01:00
ZwX1616
46840807fa up tici hevc bitrate (#20093)
* 2x tici bitrate

* update test

* clean up
old-commit-hash: e2044ca5a2
2021-02-22 09:15:56 -08:00
Greg Hogan
d59a13e22a adjust qcamera quality (#20066)
* better ts

* fix test
old-commit-hash: 1cd9105e96
2021-02-11 17:55:00 -08:00
Adeeb Shihadeh
bfa5674d4f fix up encoder test for tici rotation (#19941)
* fix encoder test

* revert for c2

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 0411015103
2021-01-27 14:20:48 -08:00