* Refactor squash and merge script for improved simplicity
Simplified the squash_and_merge.py script by replacing redundant utility functions and consolidating logic. Enhanced usability by aligning command-line arguments and leveraging streamlined git operations to improve maintainability and reliability.
* Fix argument names in squash PR script
Renamed CLI arguments from '--base' and '--source' to '--target' and '--base' to align with expected input format. This ensures the script runs correctly with proper argument mapping.
* Fix incorrect base branch argument in squash script
Updated the `--base` argument to use `source_branch` instead of `branch` to ensure the squash script processes the correct base branch. Also adjusted the command to include `branch` as a separate argument for clarity and correctness.
* Reset to a clean state after squash error.
Add a `git reset --hard` command to ensure the repository returns to a clean state after encountering errors during the squash and merge process. This prevents lingering changes from affecting subsequent operations.
* Improve error handling in squash_and_merge_prs.py
Capture and display both stdout and stderr in error cases to provide more informative feedback. Adjust the PR comment to include available output for better debugging.
* Refactor PR squash process to enhance error handling.
Modify subprocess handling to use `result.returncode` for error checks instead of relying on exceptions. Consolidate error output retrieval and logging for better clarity, while maintaining the workflow for resetting changes on failure.
* Fix incorrect return in PR processing loop
Replaced `return` with `continue` to ensure all PRs in the loop are processed before exiting. This prevents premature termination of the function and ensures accurate success count reporting.
* Simplify subprocess output handling in squash_and_merge.py
Replaced labeled print statements with direct output of stdout and stderr. This change ensures cleaner logs and remains consistent with the function's purpose of output handling during subprocess execution.
* Update subprocess.run calls to use capture_output parameter
Replaced `stdout` and `stderr` with the `capture_output` parameter for cleaner and more concise subprocess handling. Also removed extraneous whitespace for improved code readability.
* testing moving the squash script given that it's called iteratively and switching branch might miss it
* format
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Refactor OnroadEventSP structure to contain list of events
A restructuring of the OnroadEventSP structure has been undertaken to accommodate a list of 'Event' substructures. The change is reflected in different files where OnroadEventSP is used. This update allows for more efficient management of multiple events by grouping them together under the revised OnroadEventSP structure.
* Rename `OnroadEventSP` to `OnroadEventsSP` across codebase.
Updated all references to `OnroadEventSP` to ensure consistency with the renamed struct `OnroadEventsSP`. This change improves code clarity and aligns naming conventions across modules.
* Add optional debug logging to schema validation script
Introduced a `DEBUG` flag and a `print_debug` function to streamline debug output management. This replaces direct `print` calls with conditional logging to control verbosity during execution.
Refactor structural validation logic in cereal test
Simplify the iteration over read_instances to streamline structural validation. Removed redundant comparisons and improved error handling to detect unreadable fields more effectively. Updated error messages for better clarity during debugging.
Update build command to include 'cereal' target in CI
Modified the scons build command in selfdrive_tests workflow to explicitly build the 'cereal' target. This ensures necessary components are included during the CI process, improving reliability and consistency.
Added workflow for cereal validation artifacts generation and validation against upstream
This commit encompasses significant changes to .github/workflows/selfdrive_tests.yaml, including the addition of two new jobs. One is responsible for 'Generating cereal validation artifacts' and the other for 'Validating cereal with Upstream'. This includes generating cereal schemas, building openpilot, and running validation schema instances against master. Furthermore, a new Python script (validate_sp_cereal_upstream.py) was also added to perform cereal schema instance generation and validation. These changes aim to enhance the testing process, ensuring schema compatibility and integration quality.
* Relocate cereal validation to a dedicated GitHub workflow
This commit introduces a distinct GitHub workflow for cereal validation named 'cereal_validation.yaml'. This workflow includes two jobs: one for generating cereal validation artifacts and another for validating cereal with the upstream project. Previously, these operations were included as separate jobs in 'selfdrive_tests.yaml'. However, the decoupling in this commit allows for a better organization of GitHub workflows within the project. Additionally, this separation allows these workflows to be individually configured and run, providing a greater degree of flexibility in managing our continuous integration activities.
* Rename workflow to "cereal validation" for clarity.
Updated the workflow name in the GitHub Actions configuration to better reflect its purpose. This change improves maintainability and ensures clearer identification of the workflow's function.
* Add LFS configuration and GitLab SSH setup to workflow
Integrate GitLab LFS handling by configuring LFS URLs and enabling SSH setup. This includes adding public GitLab keys and updating the workflow to support secure connections for LFS operations. Ensures proper handling of large files and seamless integration with GitLab.
* rename
* format
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
Integrate GitLab LFS handling by configuring LFS URLs and enabling SSH setup. This includes adding public GitLab keys and updating the workflow to support secure connections for LFS operations. Ensures proper handling of large files and seamless integration with GitLab.
Convert safe_desired_curvature to float before returning.
This ensures the returned value is explicitly a float, avoiding an issue when serializing it on capnp as it does not recognize numpy.float
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
Move the exact_match calculation after updating model_path and max_similarity to ensure consistency. This prevents potential discrepancies when rechecking NN paths and enhances code maintainability.
This change includes the current commit SHA in the workflow outputs of the sunnypilot-build-prebuilt.yaml file. It provides better traceability for builds, ensuring each workflow run is linked to the exact commit it was triggered from.
Changed the PR reference format from (`#123`) to (PR-123) in squash commit messages
to prevent GitHub from automatically adding reference comments to PRs when
`master-dev-c3-new` is force pushed.
* init alc controller
* only for sunny
* rebase fix
* ui
* add ui preview
* Update common/params_keys.h
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Update selfdrive/ui/sunnypilot/SConscript
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Update selfdrive/ui/sunnypilot/qt/offroad/settings/lateral/lane_change_settings.h
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Update selfdrive/ui/sunnypilot/qt/offroad/settings/lateral/lane_change_settings.cc
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* review sugg
* code review
* renaming
* move around
* style
* fix types and params exceptions handling
* take out magic numbers
* more
* rename
* shorter
* make sure reset happens at the end of every DH loop
* split into multiple updates
* just 3 seconds
* use default states
* oops
* more readable
* oops
* some space and lines
* run in DH loop directly
* adjust ui preview
* nudgeless should process immediately
* check option instead
* more explicit
* even more explicit
* tests
* brake pedal release should not allow auto lane change (caught by test)
* unnecessary
* no continuous auto lane change
* Revert "unnecessary"
This reverts commit 93d135b54a.
* more tests
* less
* less less
* update again
* more cleanup
* better
* AutoLaneChangeState -> AutoLaneChangeMode
* update
* lint
* unused
* test all states
* license
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* ui: exclude certain params from deletion during settings reset
* clear all and exclude directly and only reboot after confirmation
* cleaner
* handle this instead
* send it
* 1-liner
* nuke nuke
* space
* less
* NNLC: use torque substitutes as fuzzy fingerprints
* substitutes are not exact matches
* update nnlc helpers
for extra nnlc model path checks, take the first one that passes.
* Update sunnypilot/selfdrive/controls/lib/nnlc/helpers.py
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Add AES encryption and utility methods
This commit introduces a new AESCipher class that can be used for AES encryption with support for both AES-128 and AES-256. It also adds a set of utility functions, including methods for RSA to AES key derivation, file decryption and compression, and encryption and decompression. These changes provide fundamental cryptographic functionalities for data security within the system.
* Revised backup management system with new structures
This update introduces important revisions to the backup management system. A new struct named 'BackupManagerSP' has been integrated into the 'custom.capnp' file, replacing 'CustomReserved6'. This new struct includes several variables that help keep track of backup and restore status, progress and related messages.
Additional code modifications were made in 'base.py' and 'api.py' to include a 'json' parameter in the 'api_get' method. Lastly, 'manager.py' has been updated with functions to manage device configuration backups to and from sunnylink.
The changes promote better handling and management of data backup and restoration processes. The update is specifically designed to ensure that the backup information is well integrated into the system, with properly tracked status and progress.
* Refactor `allKeys` to filter by parameter key type
Updated the `allKeys` method to support filtering by `ParamKeyType`, allowing more specific key retrieval. Added a default value for backward compatibility and updated related bindings and keys to reflect this change.
* Improve Backup and Restore mechanisms
The commit refactors and improves several aspects of the backup and restore mechanisms in the `BackupManagerSP` class.
These improvements include removing redundant status tracking variables and replacing them with unified ones, updating the messaging system to handle all changes correctly, and including an enumeration `OperationType` to keep track of the type of operation currently being processed.
This commit also applies stricter conditions for restore operations, such that it only restores parameters that are currently marked as backupable, and skips those that are not. This is a preventive measure against potential issues when restoring parameters that are no longer relevant or could conflict with current versions.
Also, the encryption and decryption methods were updated to use AES-256 for more security. These changes have increased the robustness and reliability of the backup and restoration processes.
* copyright
* Add backup_manager process to offroad sunnypilot tasks
The backup_manager process is introduced to handle backups during offroad mode when SunnyLink is ready. This ensures proper backup management functionality integrated into the system.
* Simplify backup endpoint construction in restore method
Replaced conditional expression with a more concise and readable `or` operation for constructing the backup endpoint. This makes the code cleaner and easier to maintain while preserving functionality.
* Added support for backing up and restoring sunnypilot settings
An update to the sunnypilot functionality now provides two new features that allow users to backup and restore their sunnypilot settings. The changes include the addition of UI controls for initiating backup and restore operations, and the creation of a system-wide state management function for tracking these operations. This enhancement significantly improves the user experience by providing a safety net for user settings in case of software failures, bugs, or unintended changes.
* Refactor type hints to use PEP 604 syntax for clarity
Replaced `Optional` and `Dict` type hints with `|` and `dict` syntax for improved readability and compliance with Python 3.10+. Updated related imports and adjusted list comprehension for cleaner code.
* Update import path for hardware module in utils.py
Replaced the import path for the `Paths` module to align with the new directory structure under `openpilot`. This ensures compatibility with recent project reorganization and avoids import errors.
* Improve RSA key handling and fix backup status comparison
Added explicit RSA key type checks to handle invalid key formats. Enhanced type safety in `manager.py` by ensuring the backup status comparison returns a boolean. These changes improve robustness and error handling in backups.
* format
* more
* Improve backup and restore flow with progress tracking and fixes
Added proper progress tracking and cleanup logic during restore operations. Enhanced restore experience by resetting progress after completion and introducing confirmation dialogs for errors and completion. Updated API compatibility with a version query parameter for backups.
* Enable backup button only when restore process is complete
Previously, the backup button could be enabled during a restore operation, which might cause unintended behavior. This update introduces a check to ensure the backup button remains disabled while a restore process is active. This improves user experience and prevents potential conflicts.
* Fix restore button state handling during restore process
Ensure the restore button is disabled consistently when a restore is in progress. This prevents user interaction issues and aligns the button state with the restore operation status.
* "Refactor restore logic and improve button state handling"
Replaced `is_restoring` with `restore_request_pending` for clarity and better state management. Adjusted button behavior to immediately disable upon user action, ensuring improved UX and preventing repeat inputs. Refined restore completion flow for better consistency and reliability.
* Refine restore process logic for SunnyLink settings.
Introduced `restore_request_started` to improve handling of restore states and ensure accurate UI updates during the process. Adjusted case handling to enhance clarity and maintain proper behavior when restoring settings, especially during ongoing or completed requests.
* revert
* move around
* fix enabled states for different statuses
* add prompt to notify backup is complete
* same states as restore
* disable buttons if sunnylink is off
* can use the same texts
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
The `showEvent` method in `NeuralNetworkLateralControl` was removed as it duplicated functionality now handled in `LateralPanel`. The unused `refresh` method was also removed to streamline the code and improve maintainability.
* init
* more init
* keep it alive
* fixes
* more fixes
* more fix
* new submodule for nn data
* bump submodule
* update path to submodule
* spacing???
* update submodule path
* update submodule path
* bump
* dump
* bump
* introduce params
* Add Neural Network Lateral Control toggle to developer panel
This introduces a new toggle for enabling Neural Network Lateral Control (NNLC), providing detailed descriptions of its functionality and compatibility. It includes UI integration, car compatibility checks, and feedback links for unsupported vehicles.
* decouple even more
* static
* codespell
* remove debug
* in structs
* fix import
* convert to capnp
* fixes
* debug
* only initialize if NNLC is enabled or allow to enable
* oops
* fix initialization
* only allow engage if nnlc is off
* fix toggle param
* fix tests
* lint
* fix more test
* capnp test
* try this out
* validate if it's not None
* make it 33 to match
* align
* share the same friction input calculation
* return stock values if not enabled
* unused
* split base and child
* space
* rename
* NeuralNetworkFeedForwardModel
* less
* just use file name
* try this
* more explicit
* rename
* move it
* child class for additional controllers
* rename
* time to split out custom lateral acceleration
* move around
* space
* fix
* TODO-SP
* TODO-SP
* update regardless, it's an extension now
* update name and expose toggle
* ui: sunnypilot Panel -> Steering Panel
* Update selfdrive/ui/sunnypilot/qt/offroad/settings/lateral_panel.h
* merge
* move to steering panel
* no need for this
* live params in a thread
* no live for now
* new structs
* more ui
* more flexible
* more ui
* no longer needed
* another ui
* cereal changes
* bump opendbc
* simplify checks
* all in one place
* split Enhanced Lat Accel
* handle unrecognized platform
* test for fingerprinting
* fix fingerprint
* NNLC: Mock data for unrecognized cars
* fix fingerprints
* test to verify model loading
* bump neural_network_data
* use pytest
* use different one for now
* fix ui preview alignments
* typing
* more type
* show a platform
* show match
* init params for tests
* ok ruff
* make sure to loop through tests
* ok again ruff
* ok we need this lol
---------
Co-authored-by: DevTekVE <devtekve@gmail.com>
* init
* more init
* keep it alive
* fixes
* more fixes
* more fix
* new submodule for nn data
* bump submodule
* update path to submodule
* spacing???
* update submodule path
* update submodule path
* bump
* dump
* bump
* introduce params
* Add Neural Network Lateral Control toggle to developer panel
This introduces a new toggle for enabling Neural Network Lateral Control (NNLC), providing detailed descriptions of its functionality and compatibility. It includes UI integration, car compatibility checks, and feedback links for unsupported vehicles.
* decouple even more
* static
* codespell
* remove debug
* in structs
* fix import
* convert to capnp
* fixes
* debug
* only initialize if NNLC is enabled or allow to enable
* oops
* fix initialization
* only allow engage if nnlc is off
* fix toggle param
* fix tests
* lint
* fix more test
* capnp test
* try this out
* validate if it's not None
* make it 33 to match
* align
* share the same friction input calculation
* return stock values if not enabled
* unused
* split base and child
* space
* rename
* NeuralNetworkFeedForwardModel
* less
* just use file name
* try this
* more explicit
* rename
* move it
* child class for additional controllers
* rename
* time to split out custom lateral acceleration
* move around
* space
* fix
* TODO-SP
* TODO-SP
* update regardless, it's an extension now
* update name and expose toggle
* ui: sunnypilot Panel -> Steering Panel
* Update selfdrive/ui/sunnypilot/qt/offroad/settings/lateral_panel.h
* merge
* move to steering panel
* no need for this
* live params in a thread
* no live for now
* new structs
* more ui
* more flexible
* more ui
* no longer needed
* another ui
* cereal changes
* bump opendbc
* simplify checks
* all in one place
* split Enhanced Lat Accel
---------
Co-authored-by: DevTekVE <devtekve@gmail.com>
* init
* more init
* keep it alive
* fixes
* more fixes
* more fix
* new submodule for nn data
* bump submodule
* update path to submodule
* spacing???
* update submodule path
* update submodule path
* bump
* dump
* bump
* introduce params
* Add Neural Network Lateral Control toggle to developer panel
This introduces a new toggle for enabling Neural Network Lateral Control (NNLC), providing detailed descriptions of its functionality and compatibility. It includes UI integration, car compatibility checks, and feedback links for unsupported vehicles.
* decouple even more
* static
* codespell
* remove debug
* in structs
* fix import
* convert to capnp
* fixes
* debug
* only initialize if NNLC is enabled or allow to enable
* oops
* fix initialization
* only allow engage if nnlc is off
* fix toggle param
* fix tests
* lint
* fix more test
* capnp test
* try this out
* validate if it's not None
* make it 33 to match
* align
* share the same friction input calculation
* return stock values if not enabled
* unused
* split base and child
* space
* rename
* NeuralNetworkFeedForwardModel
* less
* just use file name
* try this
* more explicit
* rename
* move it
* child class for additional controllers
* rename
* time to split out custom lateral acceleration
* move around
* space
* fix
* TODO-SP
* TODO-SP
* split nnlc and custom lat accel
* more
* not yet
* comment
* fix
---------
Co-authored-by: DevTekVE <devtekve@gmail.com>
* Add error log to developer panel
* Add lang files
* Refactor GitHub runner control to local variable
Replaced the `enableGithubRunner` member variable with a local variable, as it is not used elsewhere in the class. Simplified visibility management for developer panel toggles in release branches.
* Modifying visibility rules for developer panel controls
This commit modifies the visibility settings for controls within the developer panel such as `enableGithubRunner`, `errorLogBtn`, `joystickToggle` and `longManeuverToggle`. The visibility of these controls is now dependent on whether the software is running in release mode. This update ensures that certain settings are hidden from the end user in the release version, enhancing the user interface by simplifying it and preventing potential user confusion.
* Translation files
* we already handle this in stock
---------
Co-authored-by: DevTekVE <devtekve@gmail.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
Refactor sunnylinkd for better readability and SSL support
Reorganized variable handling for Sunnylink API initialization. Updated WebSocket connection to use Authorization header and added SSL configuration based on the connection type. This enhances code clarity and ensures secure connections when not on localhost.
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Add error log to developer panel
* Testing debug
* Fix condition for retry handling in action workflow
Updated the logic to properly check for 'NONE' author association instead of 'FIRST_TIME_CONTRIBUTOR' when determining retry limits. This ensures the workflow behaves as intended for contributors without prior activity.
* Clean
* Refine workflow condition logic for PR checks
Updated the `if` condition in the lfs-maintenance workflow to improve clarity and ensure proper evaluation. This change aligns the logical grouping for better readability and correctness in forked PR handling.
* Add lang files
* Update selfdrive/ui/qt/offroad/developer_panel.cc
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Refactor GitHub runner control to local variable
Replaced the `enableGithubRunner` member variable with a local variable, as it is not used elsewhere in the class. Simplified visibility management for developer panel toggles in release branches.
* Updating translations
---------
Co-authored-by: DevTekVE <devtekve@gmail.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* format
* ci: update existing comment during Squash and Merge
* dynamic target branch
* test label
* try again
* no f
* test
* should be edit-last
* quote
* fix arg
* final
* bold it
* revert
* Refactor sunnylink panel code for clarity and initialization fixes.
Replaced explicit pointer types with `auto` for cleaner code and added proper initialization for the `offroad` boolean member. Simplified toggle logic by consolidating description updates for enabling/disabling sunnylink. These changes improve code readability and maintainability.
* Add Sunnylink sponsor and GitHub pairing functionality
This update introduces a feature to manage sponsorship-based roles and GitHub account pairing for Sunnylink. It includes new sponsor popups, sponsor-specific widgets, QR code logic, and backend API integrations. Additionally, new models and services support sponsor tier management and user-role synchronization.
* Translation files
* Param keys
* Add setup functions for SunnyLink sponsor and pair buttons
Introduce `setup_settings_sunnylink_sponsor_button` and `setup_settings_sunnylink_pair_button` to handle specific SunnyLink UI interactions. These functions streamline button clicks for sponsor and pairing actions within SunnyLink settings.
* Add new SunnyLink test cases for sponsor and pair buttons
Added `settings_sunnylink_sponsor_button` and `settings_sunnylink_pair_button` to the UI test case dictionary. This extends the SunnyLink test coverage to include sponsor and pairing functionalities.
* No need to import sunnylink from here, and it causes just circular dependency
* Enhance SunnylinkPanel functionality in off-road settings
This commit enhances the functionality of the SunnylinkPanel in the off-road settings of the SunnyPilot user interface. A paramWatcher is added to the SunnylinkPanel to observe "SunnylinkEnabled" parameter changes. Update functionalities are enhanced to handle showing and hiding of components based on various circumstances, such as whether the system is 'on-road' or 'off-road', and whether Sunnylink is enabled or not. The stopSunnylink and startSunnylink functions were also added to start or stop processes accordingly when Sunnylink is enabled or disabled. Additionally, the ui.h file is updated to efficiently handle Sunnylink roles and device users.
* Refactor SunnylinkPanel initialization and handling.
Reorganized SunnylinkPanel to improve structure and clarity by separating sunnylink client initialization and list widget setup. Enabled automatic sunnylink startup when the feature is enabled. Added minor formatting fixes for label display consistency.
* Add missing include for <optional> in ui.h
Including <optional> ensures compatibility with standard C++ features and prevents potential compilation errors. This addition aligns with best practices for maintaining robust and clean code.
* Updated setup_settings_sunnylink_sponsor_button and setup_settings_sunnylink_pair_button function signatures
Added an optional 'scroll' parameter to the setup_settings_sunnylink_sponsor_button and setup_settings_sunnylink_pair_button functions in the test_ui module. The modifications were made to allow for more flexible function usage by potentially enabling scroll operations during the execution of these UI setup steps.
* Enable Sunnylink initialization on panel show event
Begin Sunnylink connection automatically when the panel is displayed, ensuring the feature is active if enabled. Additionally, update the sponsor button text formatting for more concise styling.
* Translations
* Added checks for new UI files in PRs
The git workflow script `ui_preview.yaml` has been modified. The script now checks if the master branch contains a file corresponding to a UI file present in the PR. If a UI file in the PR does not have a match on the master branch, it is marked as new. These enhancements improve the comparison of UI changes between the master and PR branches, particularly with the identification of new UI files.
* cleanup
* duh
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
Added checks for new UI files in PRs
The git workflow script `ui_preview.yaml` has been modified. The script now checks if the master branch contains a file corresponding to a UI file present in the PR. If a UI file in the PR does not have a match on the master branch, it is marked as new. These enhancements improve the comparison of UI changes between the master and PR branches, particularly with the identification of new UI files.
