Commit Graph

144 Commits

Author SHA1 Message Date
Dean Lee
33849245d8 uploader.py: fix empty string handing in AthenadRecentlyViewedRoutes parameter (#35139)
Fix empty strings in AthenadRecentlyViewedRoutes parameter
2025-05-08 16:31:06 +01:00
Maxime Desroches
47ed90c6cf Reapply "Mypy: Got passing on macos (#34591)" (#35126) (#35153)
* Mypy: Got passing on macos (#34591)

* Mypy: Got mypy passing on macos

* common/realtime.py refactor

* Mypy: mypy passing on darwin

* Refactor: Removed else: pass statement

* Refactor: Removed unnecessary check

* added xattr to pyproject

* loggerd: switched to xatter module

* loggerd: removed unused module in xattr_cache.py

* UV: update uv.lock

* Update system/athena/athenad.py



* athenad: fixed blank lines

* loggerd: refactor of xattr_cache

* cleanup

---------



* fix getxattr no attribute on macOS

* try fixing missing ENOATTR on Linux

---------

Co-authored-by: Andrei Radulescu <andi.radulescu@gmail.com>
Co-authored-by: BrainLess <116778989+BrainLessPea@users.noreply.github.com>
2025-05-07 19:11:37 -07:00
Dean Lee
7c16e65347 loggerd: remove redundant Params Construction (#35138)
remove redundant Params Construction
2025-05-07 16:39:05 +01:00
Maxime Desroches
aa1b790708 revert agnos 12 2025-05-06 21:38:50 -07:00
Andrei Radulescu
bbf37ae5c7 Reapply "Mypy: Got passing on macos (#34591)" (#35126)
* Mypy: Got passing on macos (#34591)

* Mypy: Got mypy passing on macos

* common/realtime.py refactor

* Mypy: mypy passing on darwin

* Refactor: Removed else: pass statement

* Refactor: Removed unnecessary check

* added xattr to pyproject

* loggerd: switched to xatter module

* loggerd: removed unused module in xattr_cache.py

* UV: update uv.lock

* Update system/athena/athenad.py

Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>

* athenad: fixed blank lines

* loggerd: refactor of xattr_cache

* cleanup

---------

Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>

* fix getxattr no attribute on macOS

* try fixing missing ENOATTR on Linux

---------

Co-authored-by: BrainLess <116778989+BrainLessPea@users.noreply.github.com>
Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>
2025-05-06 21:09:59 -07:00
Adeeb Shihadeh
b33342c18f loggerd: limit encoded queue size (#34837)
* loggerd: limit encoded queue size

* less spammy
2025-03-11 10:22:49 -07:00
Dean Lee
a0a9df2d55 loggerd: enhance ZstdFileWriter test coverage with variable data sizes (#34812)
enhance ZstdFileWriter test coverage
2025-03-07 10:23:46 -08:00
Dean Lee
2c2c6e6437 deleter: fix duplicate segments in get_preserved_segments return value (#34795)
fix duplicate segments in get_preserved_segments return value
2025-03-06 10:16:56 -08:00
Dean Lee
c52112ad4e loggerd: eliminate recursion from handle_encoder_msg (#33453)
* refactor handle_encoder_msg

* removing the recursion only

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2025-03-02 20:01:21 -08:00
Adeeb Shihadeh
54f1b00447 Log git source commit (#34749)
* Log git source commit

* strip

* fix trim

* no trim
2025-03-02 13:11:38 -08:00
Dean Lee
0619e9a581 v4l_encoder: used encoder_info.fps instead of hardcoded 20 (#34701)
Used encoder_info.fps instead of hardcoded 20
2025-02-24 07:06:38 -08:00
Adeeb Shihadeh
e00b5337d2 misc encoder cleanups 2025-02-21 10:41:30 -08:00
Adeeb Shihadeh
6f5f8e7746 loggerd: reduce debug log frequency 2025-02-17 15:06:16 -08:00
Adeeb Shihadeh
27b5a727e3 Revert "Mypy: Got passing on macos (#34591)"
This reverts commit b09b48130e.
2025-02-16 08:50:46 -08:00
BrainLess
b09b48130e Mypy: Got passing on macos (#34591)
* Mypy: Got mypy passing on macos

* common/realtime.py refactor

* Mypy: mypy passing on darwin

* Refactor: Removed else: pass statement

* Refactor: Removed unnecessary check

* added xattr to pyproject

* loggerd: switched to xatter module

* loggerd: removed unused module in xattr_cache.py

* UV: update uv.lock

* Update system/athena/athenad.py

Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>

* athenad: fixed blank lines

* loggerd: refactor of xattr_cache

* cleanup

---------

Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>
2025-02-16 00:05:52 -08:00
Dean Lee
644130ca3d encoderd: use AV_CODEC_ID_H264 codec for qcamera encoding (#34498) 2025-02-11 12:46:43 +00:00
Dean Lee
3c0990e716 move thumbnail generation from camerad to encoderd (#34554)
Co-authored-by: Cameron Clough <cameronjclough@gmail.com>
2025-02-10 23:43:46 +00:00
Dean Lee
27c11eb597 athena upload: streaming file upload (#34559)
streaming file upload
2025-02-10 13:57:31 -08:00
Dean Lee
4b7a25efc3 loggerd: remove unused zlib and libswscale linkage (#34552)
remove unused link
2025-02-09 09:13:58 -08:00
Dean Lee
9ec54f59c6 loggerd: switch logging from raw file format to zstd compressed files (#34549)
* switch logging from raw file format to zstd compressed files

* more zst suffix

* compress bootlog

* remove class RawFile

* Optimize ZstdFileWriter by adding input caching

* use ZSTD_compressStream2

* cleanup

* LOG_COMPRESSION_LEVEL=10

* space

* add zst suffix to LOGS_SIZE_RATE
2025-02-08 12:55:31 -08:00
Adeeb Shihadeh
9bc35c0919 Revert "athena upload: reduce memory usage and improve efficiency with streaming (#34528)"
This reverts commit 4c65f51a55.
2025-02-06 11:50:04 -08:00
Dean Lee
4c65f51a55 athena upload: reduce memory usage and improve efficiency with streaming (#34528)
* reduce memory usage and improve upload efficiency with streaming

* space
2025-02-04 13:10:13 -08:00
Dean Lee
e7b63dce65 encoderd: enhance robustness of open() with HANDLE_EINTR Macro (#34478)
HANDLE_EINTR
2025-01-27 19:46:43 -08:00
Dean Lee
3219fe1472 encoder: optimize MessageBuilder data handling by adopting encoded data (#34391)
optimize data handing by adopting external data
2025-01-19 16:22:40 -08:00
Dean Lee
47b13f54f6 encoderd: refactor VideoEncoder::publisher_publish to standardize member variable access (#34342)
remove redundant pointer parameter
2025-01-12 11:38:20 -08:00
Cameron Clough
08779941e5 loggerd: typing and remove unused default arg (#34349) 2025-01-09 22:12:10 +00:00
Cameron Clough
f761f53205 deleter cleanups (#34345) 2025-01-09 20:00:45 +00:00
Adeeb Shihadeh
29577a3346 bootlog: monotonic ts for journalctl 2024-11-25 15:41:21 -08:00
Adeeb Shihadeh
c287232374 uploader: increase max qlog size (#34106) 2024-11-25 09:40:23 -08:00
Maxime Desroches
7d5031a6d6 ci: reduce sleep in test_uploader (#34009)
less
2024-11-12 19:41:00 -08:00
Maxime Desroches
ce7317407f ci: faster test_encoder (#33968)
* try

* timeout

* fix
2024-11-08 11:49:04 -08:00
Maxime Desroches
e0acd86ca1 ci: faster unit_tests (#33962)
faster
2024-11-07 18:44:12 -08:00
Adeeb Shihadeh
853febe21d visionipc: remove RGB support (#33739)
* visionipc: remove RGB support

* bump

* msgq master

* fix sim
2024-10-07 13:24:57 -07:00
Dean Lee
0bdad1fee0 loggerd: direct Initialize params instead of copy (#33622)
Direct Initialize Instead of Copy
2024-09-22 14:12:07 -07:00
Dean Lee
2b37624dee encoderd: use const reference for VisionBuf access (#33621)
use const reference for VisionBuf access
2024-09-22 14:11:59 -07:00
Dean Lee
467bd74944 camerad: remove CameraType enum from camera_common.h (#33588)
remove CameraType from camera_common.h
2024-09-19 10:33:24 -07:00
Adeeb Shihadeh
102f1b0d85 loggerd: fix mac build 2024-09-18 22:16:25 -07:00
Adeeb Shihadeh
718835a825 camerad: prep part 2 (#33584)
* no camerastate in sensors

* bring this over

* and encoder
2024-09-18 21:10:45 -07:00
Adeeb Shihadeh
139f4034a8 Revert "uploader.py: simplify clear_locks by using rglob (#32974)"
This reverts commit 870f5cbeb39fa63829e6f3a64fa8df57e82d52c3.

old-commit-hash: 9474523add
2024-08-03 12:12:42 -07:00
Dean Lee
7fcba71725 uploader.py: simplify clear_locks by using rglob (#32974)
simplify clear_locks
old-commit-hash: cafca4f891
2024-08-02 13:40:57 -07:00
Dean Lee
d802a423a6 loggerd: Initialize atomic variables to zero (#33130)
Initialize atomic variables in LoggerdState to zero
old-commit-hash: 8f3fc699d4
2024-08-02 12:01:56 -07:00
Hoang Bui
cb306fd7a8 CI: fix recent test_rotation skip (#33167)
old-commit-hash: 5b4b7a08fc
2024-08-01 16:28:26 -07:00
Adeeb Shihadeh
d59f9262c8 disable test_rotation until it's fixed
old-commit-hash: bfa8e0b2ec
2024-08-01 15:36:44 -07:00
Dean Lee
3c1af12490 uploader.py: ensure proper resource management with io.BytesIO (#33108)
* ensure proper resource management with io.BytesIO

* improve
old-commit-hash: 2728c95b0d
2024-07-29 13:43:31 -07:00
Adeeb Shihadeh
8cc6b66cde docs: add concepts section
old-commit-hash: 5ebc65f254
2024-07-28 16:38:41 -07:00
Shane Smiskol
7b67994a05 use zstandard library (#33110)
* use zstandard

* run 10

* bye bye

* this was a little overzealous

* clean up
old-commit-hash: 1b3b5ab758
2024-07-28 02:03:47 -07:00
Shane Smiskol
b0a8b3bc9d uploader: compress with zstd (#32736)
* zstd uploader

* fix that

* fix name of function

* comment

* log failed

* fix comma_api_source for routes with both bz2 and zst rlogs

* TODO

* 10-14 achieves almost no benefit on qlogs in a few cases, but takes 2x the time

* these aren't written out

* regen: specify any list of sources

ooh this is pretty nice

* regen and process replay

* damn, actually we don't need all this (cool tho)

Revert "regen: specify any list of sources"

This reverts commit ceb0b4abed9ad463a9fe98d9b98a05875a52806f.

* just let it auto resolve

* fix athenad/uploader tests

* zst here too

* TODOs

* yes

* Revert "TODOs"

This reverts commit 8c7da1dbd0340c72290b5eb5563b642080ddc131.

* Revert "zst here too"

This reverts commit 23b0023ddfd22c8090be7a7caa09e7026a12aa5c.

* Revert "just let it auto resolve"

This reverts commit f296d62424227ad05facc62abc18a6f81b474e84.

* Revert "regen and process replay"

This reverts commit 0768330e96974a42616d229d159780619d049cd0.

* revert readme

* not in save_log either

* lfg

* Revert "lfg"

This reverts commit 3718559c6c4de7d1f0c80dc9f1a1d335fe679a89.
old-commit-hash: 7dec7c39be
2024-07-26 19:33:35 -07:00
Shane Smiskol
82ebea3d12 deleter: preserve two previous segments on user flag (#32822)
preserve 2 segs
old-commit-hash: dc41848597
2024-06-24 15:11:20 -07:00
Harald Schäfer
05ed3d8c15 Restructure msgq (#32652)
* Update ref

* Compiles

* compiles

* Refactor rest of libs

* import all

* small fiex

* cleanup import

* Need msgq simlink too

* Add to openpilot docker too

* try repo

* Updates

* Fix lint

* fix docs

* Try blank slate

* Revert "Try blank slate"

This reverts commit f078ce04acacfe115c19e23e86038b01e2b84a6d.

* Maybe scons needs this to clear cache

* fix tests

* Disable test for now

* Update SConstruct

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Fix whitespace

* Write skip normal

* small fixes

* add test path

* Revert repo

* linting

* whitespace

* Bump msgq

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: e70dc90a45
2024-06-09 17:44:34 -07:00
Adeeb Shihadeh
ad42b61427 test_encoder: adjust file size tolerance
old-commit-hash: b51472048c
2024-06-07 23:09:41 -07:00