Commit Graph

20 Commits

Author SHA1 Message Date
Dean Lee
dbec761941 logreader: support reading from corrupt log (#23050)
* catch exception outside loop

* print decompress error

* add test case for corrupt log

* fix decompressbz2 stuck if log is corrupt

* recovered from corrupt data

* add output

* ass space

* std::endl

* override load(), load from buffer

* override FrameReader::load to load from the buffer

* replace NULL with nullptr

* fix test case for corrupt log

* Trigger Build

* check bzerror too

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-11-29 14:10:24 +01:00
Dean Lee
f5e4867864 replay: refactor http download (#23052)
* refactor http download

* use chunk_size instead of parts

testcase:set chunksize to 5mb

* use template

space

* cleanup

* remove unused include

* check buffer overfllow

* simplify print download speed
2021-11-29 11:13:30 +01:00
Dean Lee
1d2dfab81c replay: add test to check dcam and ecam (#22993) 2021-11-20 18:56:27 -08:00
Dean Lee
fb8ba34f31 replay: add qcam and YUV flags (#22788)
* common flags

* cleanup

* remove double semicolon

* camera

* apply reviews
2021-11-09 14:08:24 -08:00
Dean Lee
79651b6c49 replay: fix qcam download error (#22777)
* fix qcam download error

* test loading qlog/qcamera
2021-11-02 11:24:59 +01:00
Dean Lee
2b4a477fbc replay: improve segment download and merge (#22654)
* no-cache mode

* fix test cases build error

* space

* don't create cache dir in no-cache mode

* fix errors in test cases

* no_local_cache_

* set the number of connections by chunk_size

* use size_t instead of int64_t

* add test case for no-cache mode

* rename variables

* fix SIGSEGV

* cleanup

* faster decompressBZ2

* always decompress bz2

* add test cases

* prepare for python interface

* fix test cases build error

* continue

* camera_replay: cache remote file

* protected inheritance

* single option name

* TODO

* test_case for LogReader&FrameReader

* fix wrong require

* test case for FileReader

* cleanup test

* test:fix wrong filename

* check cached file's checksum

* fix mkdir permissions err

cleanup filereader

* remove initialize libav network libraries.

dd

* abort all loading if one failed

* cleanup tests

* use threadpool to limit concurrent downloads

* cache more segments

* merge 3 segments for replay

* one segment uses about 100M of memory

* use segments_need_merge.size()

* shutdown

* fix stuck if exit replay  before keyboard thread started

* load one segment at a time

* small cleanup

* cleanup filereader

* space

* tiny cleanup

* merge master

* cleanup test cases

* use util:create_directories

* cleanup framereader
2021-11-01 11:55:56 +01:00
Dean Lee
317deeae84 replay: improve seeking, queuing segments (#22625)
* improve seeking, queuing segments

* cleanup,init current_segment_ to 0

* use isSegmentLoaded

* remove to isSegmentMerged
2021-10-19 22:23:53 -07:00
Dean Lee
8d1d0c2cd7 replay: improve segment downloading (#22504)
* improve segment downloading

dd

* continue

* log retrying

* check aborting_ in loop

* std::endl

* log download information

* cleanup

* continue

* dd

* move download to seperate funciton

* simplify logging

* continue

* guard ts

* cleanup

* typo
2021-10-19 12:33:12 +02:00
Dean Lee
0189a19b8e replay: refactor FrameReader (#22438)
* decode from the previous keyframe after seek

* less memory

* some stream seems to contian no keyframes

* test random seek

* merge master

* continue

update test_cases

use fr

* merge master
2021-10-18 14:19:23 -07:00
Dean Lee
f6de10b55a replay: start streaming after segment loaded (#22575)
* start streaming  after segment loaded

dd

* loop from beginning if reaches the end

* isSegmentLoaded

* one loop

* move to ctor

* delete stream_thread_ on exit

* pause streaming while testing seek

* Revert "one loop"

This reverts commit f029cd118f7ac876dee1dbf2b91478403211ad47.

* test:dummy stream thread

* cleanup

* start thread after vipcserver
2021-10-18 11:05:03 +02:00
Dean Lee
acc52ece20 replay: refactor Route and Segment (#22531)
* new functions

* fix wrong call to qUrl::isLocalFile

* cleanup

* keep extension in cached files

* cleanup

* simplify segment

* delete thread

* add output

* pre-decompress

* remove suffix

* revert remove suffix

* 1 connection for log file

* cleanup

* segment may not be continuous,use map

* don't emit finish if aborting_

* use QFuture and thread pool

* cleanup

* fix segfault in LoadFromLocal

* cleanup

* handle segment failed to load

* output info

* continue error handling

* Remove redundant testSeekTo

* cleanup

* always return true

* keep time is ok now

change to 1s

write 1 byte at the end of the sparse file

* log loading segment

* merge #22476

* Update selfdrive/ui/replay/logreader.cc

* correct connect

* pub message in function

* typo

* Update selfdrive/ui/replay/replay.cc

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-10-15 14:35:17 -07:00
Dean Lee
5527736df6 replay: simplify seek&merge (#22463)
* simplify seek&merge

* update test cases

update test cases

* cleanup test cases

* new function currentSeconds

* add TODO

* thread safe publishFrame

* cleanup

* fix 'at x S' not printed if seek back to old time

* exit replay if failed to load route

* move out setCurrentSegment from if statement

* cleanup

* use std::find

* const variables
2021-10-07 17:32:16 +02:00
Dean Lee
261d51de41 test_replay: temporarily comment out some unit tests (#22447) 2021-10-06 12:27:52 +02:00
Dean Lee
fd801c454a c++ replay: get socket name from capnp::Schema (#22367)
* get socket name from canp::Schema

* use vector
2021-10-04 15:44:06 +02:00
Dean Lee
d28b98c602 replay : move utility functions into separate file (#22414)
* move functions into util

* read bz2 into stream

* pre-decompress log in the download thread

* cleanup logreader

* cache sha256 path

* use readBZ2file in test_logger

* Revert "cache sha256 path"

This reverts commit 60459d3ea09a2c80f4560cf95b1ce7d6af59f06d.

* use macro

* use ostringstream

* cleanup readBZ2File

* move precise_nano_sleep into util
2021-10-04 15:39:59 +02:00
Dean Lee
fa8ddd992d c++ replay: more robust seek (#22375)
* skip invalid segment

* test seek

* fix wrong cur_mono_time when route_start_ts is 0

* don't notify stream thread if current segment not merged

* continue merge next segment if current is invalid

* cleanup seekTo

* continue

* fix seek cross-boundary

* new demo route has 11 segments

* continue

* fix cross-boundary

* cleanup & better test cases

cleanup

remoe &

typo

update comment

* update test

* reserve space for new_events

* parallel sorting

* usleep(0)

* Revert "parallel sorting"

This reverts commit d7d1b42f49944aef0b6ca2962e45a7f5318a1152.
2021-10-04 15:00:57 +02:00
Dean Lee
4e6ff308a8 c++ replay: chunking and concurrent downloads (#22308)
* download segment files by chunks in multiple threads

* remove easy_handl on aborting

* add test cases

* better error handling

* update test

* cleanup

* add CURLGlobalInitializer

* check http code

* finish
2021-09-28 12:24:48 +02:00
Dean Lee
50ae7dd6a9 replay improvements (#22203)
* refactor replay

* cleanup

small cleanup

* merge 22239

* cleanup

* add optional argument for start time

* small cleaup
2021-09-19 12:24:28 -07:00
Dean Lee
f3683dd4cb test_replay.cc: temporarily comment out the rand seek test (#22192)
* Temporarily comment out the rand seek test

* enable ci

* trigger ci

* trigger ci

* trigger ci

* trigger ci

* trigger ci
2021-09-10 21:53:46 -07:00
Dean Lee
81a128cd76 FrameReader: do not depend on Qtunit tests (#21379)
* add timeout and test case

* remove useless comment

* reduce to 1s

* fix gotframe fail when Duplicate POC in a sequence

* cleanup

* remove space

* fix
2021-09-09 13:40:12 -07:00