* Add job to fetch and upload model artifacts
Introduced a new `get_model` job to retrieve and upload model artifacts as part of the workflow. Updated the `build_model` job to download and clean up model files before building, ensuring an isolated and streamlined process. These changes improve modularity and artifact handling in the CI pipeline.
* Refine model file handling in build workflow
Restrict deletion and download commands to ".onnx" files to prevent unintended removal or overwriting of non-ONNX files. This ensures a more targeted and safer workflow for handling model artifacts.
* Update model artifact handling in build workflow
Changed artifact upload path to include only .onnx files and commented out the deletion of existing models. Also added a dependency for the build stage to ensure proper execution sequence.
* Update build workflow to refine model artifact handling
Removed unused code for deleting models and fixed the artifact path. Added a reference file logging custom name and upstream branch for better traceability. These changes streamline the workflow and improve clarity.
* Set dynamic run-name for sunnypilot build workflow
Added a customizable `run-name` field in the sunnypilot build model workflow. This allows more informative and dynamic naming based on input parameters such as `custom_name` or `upstream_branch`. Improves clarity and traceability in workflow runs.
* Add customization options and metadata generation to build
Introduce new inputs for file name prefix and 20Hz model specification in the workflow. Enable renaming of model files, generation of file hashes, and creation of a `metadata.json` file containing build details. This improves flexibility and enhances artifact information management.
* remove EOF
* No need for this hack anymore
* Add support for compiling ONNX models to Tinygrad format
* Add model directory environment variable and metadata generation script
* Fix quoting in model generator script command
* Refactor conditional flag for 20Hz input in build model script
* meh
* Add powersave management to build model workflow
* Add environment variable setup to build model workflow
* make it path call it a day
* kill me
* kill me x2
* mmm
* mmm
* Generate metadata.pkl
* Generate metadata.pkl
* ignore failure on getting model metadata
* Cleanup the existing onnxs
* Fix rm command syntax in build model script
* Cleaning a little
* Add optional file name parameter to generate_metadata function
* update
* stg
* linter
* more formatting
* Update sunnypilot model build workflow: improve custom naming, handle 20Hz default, and add commit date as output.
* Update commit date format in workflow script
Changed the commit date format in sunnypilot-build-model.yaml to a more human-readable format ('Month DD, YYYY'). This improves readability and consistency in the workflow outputs.
* Fix output naming in sunnypilot-build-model workflow
Corrected the output name from 'model_date' to 'commit_date' to match its usage. This ensures consistency and prevents potential errors in the workflow.
Add debug echo for model date in GitHub workflow
This change adds a debug echo statement to display the model date in the sunnypilot-build-model workflow. It helps verify the correct commit date is being set during the workflow process.
* Update BUILD_DIR path in sunnypilot workflow
Replaced hardcoded BUILD_DIR with a dynamic path using `github.workspace`. This enhances flexibility and ensures compatibility with different environments or runners.
* Revert "Update BUILD_DIR path in sunnypilot workflow"
This reverts commit 376971b616.
* Refactor model metadata generation to use short names
Replaced `file_name` with generated `short_name` for metadata and file naming. Introduced `create_short_name` function to simplify short name creation from custom names. Updated scripts and workflow to reflect the new parameter and improve metadata handling.
* `Use timezone-aware UTC timestamp for build_time`
Updated the code to use `datetime.now(timezone.utc)` instead of `datetime.utcnow()`. This ensures that the generated timestamps are timezone-aware, improving consistency and correctness in the output.
* Refactor short name generation logic in model metadata
Enhanced the `create_short_name` function to better handle name conversion by incorporating new rules for handling parentheses, alphanumeric words, and version pairs. Updated related variable names in metadata functions for improved clarity and consistency.
* Fix syntax error in assigning 'models' key in metadata
Corrected a typo where 'models' was missing quotes, causing it to be interpreted as a variable instead of a string key. This ensures proper metadata dictionary structure.
Update model metadata structure and JSON generation
Revised metadata to include detailed download URIs and a type field for artifacts. Enhanced the JSON generator with new fields like environment, runner, and overrides for better configurability. These changes aim to improve metadata clarity and compatibility with downstream processes.
* Refactor variable name from "escaped_name" to "short_name"
Renamed "escaped_name" to "short_name" across function parameters and internal logic for clarity and consistency. This improves readability and better aligns with its use case in the metadata generation process. No functional changes were introduced.
* Refactor variable names and improve type annotations.
Updated parameter names to align with Python naming conventions, enhancing readability and consistency. Added type annotations for function parameters to improve code clarity and maintainability.
* Refactor type annotations and clarify comments.
Updated type annotations to use "Str" for consistency with custom typing conventions. Improved clarity of a comment describing the normal case logic in the shortening function.
* Fix dict formatting in model_generator.py
Moved the "overrides" key for better alignment and consistency in the dictionary format. This improves readability without changing functionality.
Refactor model short name generation logic
Simplify and clarify the `create_short_name` function by improving readability and condensing comments. Adjust casing and truncation logic for single-word cases and handle versioned names more intuitively.
* Ensure custom file names are converted to lowercase.
When renaming files using a custom short name, enforce lowercase conversion to maintain consistency and prevent potential issues with case-sensitive file systems. This change applies to both the tinygrad and metadata file names.
* Use UTC alias for timezone in datetime imports
Replaced `timezone.utc` with `UTC` from the `datetime` module for clarity and consistency. This simplifies the representation of time zones and aligns with Python's updated best practices.
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* enable cache for Mac brew and scons
* bump
* save cache by default, explicitly opt-out
* Delete bump-ci
---------
Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>
Enhance action to fail on non-successful workflow runs
Add logic to check the conclusion of the watched workflow run. If the run ends with a non-successful status, the action now exits with an error to improve error handling and ensure reliability.
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Refactor workflow to centralize branch source handling
Moved branch resolution logic to a single `SOURCE_BRANCH` variable for consistency and readability. Updated related conditionals and parameters to use this new variable, reducing redundancy and improving maintainability of branch-specific configurations.
* Refactor workflow to use repository-defined variables.
Replaced hardcoded branch names with repository-defined variables for improved configurability and maintainability. This ensures flexibility by allowing changes via repository settings without modifying the workflow file.
pull_request_target is needed for security, but at the same time it makes life a little harder because the ref is different so we need to do some work to make sure we get what we are supposed to get
Otherwise don't let it cancel as it seems that when running with pull_request_target we are with ref pointing to target and thus the cancel condition was being met
* ci: enhance PR checks to validate individual check runs before merging and ignore reset-and-squash as candidate for fail
* reset and squash script shouldn't will be cancelled only if a push to master has been made which always should take priority. The rest will be put on hold to avoid parallel squash scripts running
* Set concurrencty at workflow level?
* Set concurrencty at workflow level?
* change trigger to pull_request to validate
* Reducing the types of "pull request" events
* Keep it as target
* Playing eith run name a little
* Change to pull request not target
* simplifying
* set to just pull request without target
* maintain as pull request target
* ci: enhance pull request handling for prebuilt workflows
* ci: add wait-for-action step to monitor selfdrive_tests workflow
* ci: add validation job to monitor selfdrive_tests for prebuilt workflows
* test
* force negative condition to validate flow
* force negative condition to validate flow
* ci: add wait_for_tests input to control selfdrive_tests workflow execution
* ci: update description for wait_for_tests input in workflow
* Cleaning
* Remove PR label when triggered by the label
* Rename
* Changing to target and only caring about labeled for the time being
* gh action needs cancelled not canceled
* Line ending
* force commit to force PR
* ci: add job to manage PR labels and remove trust label on new commits
* Add pull request trigger
* empty commit to test
* ci: update pull request label management logic and remove FORCE_RUN flag
* ci: update trust label management to use TRUST_FORK_PR_LABEL
* ci: update workflow to manage PR labels and adjust trigger conditions
* ci: rename jobs in workflow for consistency and clarity
* ci: add unleash-nightly-squash branch to workflow
* ci: add checkout step with GITHUB_TOKEN to workflow
* force wait
* what happens if negated the wait
* ci: add 'Wait for Tests' action to monitor workflow execution
* ci: update wait-for-tests action conditions and cleanup
* ci: refine conditions for managing PR labels and waiting for tests
* ci: enhance PR label conditions for workflow execution
* Cant use env on if
* maybe
* ci: update pull request workflow conditions and cleanup
* Missing end of line
* ci: rename workflow and update job name for clarity
* Cleaning
* ci: refine conditions for pull request handling in workflow
* ci: update handling of TRUST_FORK_LABEL in PR processing
* ci: remove fork trust warning from PR processing
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Get head ref info
* Simpify adding the comment with the newest flagg by gh cli
* Dynamic origin
* Enhance PR processing: add label handling for forked PRs and improve variable naming
* Refactor PR label handling: use constant for trust-fork label and improve comments
* Fix remote addition in PR processing: change subprocess call to not check for errors
* Refactor PR comment handling: support multiple comments and improve clarity
* Refactor PR processing: streamline comment handling and improve error management
* Refactor add_pr_comments function: specify comments type as list of strings
* Maybe we don't prevent "behind" branches to be merged as lon as the PR itself is mergeable
* lint, leave me alone
---------
Co-authored-by: Discountchubbs <159560811+Discountchubbs@users.noreply.github.com>
The Panda build step was unnecessary as it is not utilized in this workflow. This change simplifies the workflow and reduces redundant actions, improving efficiency.
* ci: update package management for `ui_preview`
Add `apt-get update` before installing ImageMagick to ensure the package list is up-to-date. This prevents potential installation issues due to outdated package information.
* test commit
* try this
* revert
* Refactor squash and merge script for improved simplicity
Simplified the squash_and_merge.py script by replacing redundant utility functions and consolidating logic. Enhanced usability by aligning command-line arguments and leveraging streamlined git operations to improve maintainability and reliability.
* Fix argument names in squash PR script
Renamed CLI arguments from '--base' and '--source' to '--target' and '--base' to align with expected input format. This ensures the script runs correctly with proper argument mapping.
* Fix incorrect base branch argument in squash script
Updated the `--base` argument to use `source_branch` instead of `branch` to ensure the squash script processes the correct base branch. Also adjusted the command to include `branch` as a separate argument for clarity and correctness.
* Reset to a clean state after squash error.
Add a `git reset --hard` command to ensure the repository returns to a clean state after encountering errors during the squash and merge process. This prevents lingering changes from affecting subsequent operations.
* Improve error handling in squash_and_merge_prs.py
Capture and display both stdout and stderr in error cases to provide more informative feedback. Adjust the PR comment to include available output for better debugging.
* Refactor PR squash process to enhance error handling.
Modify subprocess handling to use `result.returncode` for error checks instead of relying on exceptions. Consolidate error output retrieval and logging for better clarity, while maintaining the workflow for resetting changes on failure.
* Fix incorrect return in PR processing loop
Replaced `return` with `continue` to ensure all PRs in the loop are processed before exiting. This prevents premature termination of the function and ensures accurate success count reporting.
* Simplify subprocess output handling in squash_and_merge.py
Replaced labeled print statements with direct output of stdout and stderr. This change ensures cleaner logs and remains consistent with the function's purpose of output handling during subprocess execution.
* Update subprocess.run calls to use capture_output parameter
Replaced `stdout` and `stderr` with the `capture_output` parameter for cleaner and more concise subprocess handling. Also removed extraneous whitespace for improved code readability.
* testing moving the squash script given that it's called iteratively and switching branch might miss it
* format
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Refactor OnroadEventSP structure to contain list of events
A restructuring of the OnroadEventSP structure has been undertaken to accommodate a list of 'Event' substructures. The change is reflected in different files where OnroadEventSP is used. This update allows for more efficient management of multiple events by grouping them together under the revised OnroadEventSP structure.
* Rename `OnroadEventSP` to `OnroadEventsSP` across codebase.
Updated all references to `OnroadEventSP` to ensure consistency with the renamed struct `OnroadEventsSP`. This change improves code clarity and aligns naming conventions across modules.
* Add optional debug logging to schema validation script
Introduced a `DEBUG` flag and a `print_debug` function to streamline debug output management. This replaces direct `print` calls with conditional logging to control verbosity during execution.
Refactor structural validation logic in cereal test
Simplify the iteration over read_instances to streamline structural validation. Removed redundant comparisons and improved error handling to detect unreadable fields more effectively. Updated error messages for better clarity during debugging.
Update build command to include 'cereal' target in CI
Modified the scons build command in selfdrive_tests workflow to explicitly build the 'cereal' target. This ensures necessary components are included during the CI process, improving reliability and consistency.
Added workflow for cereal validation artifacts generation and validation against upstream
This commit encompasses significant changes to .github/workflows/selfdrive_tests.yaml, including the addition of two new jobs. One is responsible for 'Generating cereal validation artifacts' and the other for 'Validating cereal with Upstream'. This includes generating cereal schemas, building openpilot, and running validation schema instances against master. Furthermore, a new Python script (validate_sp_cereal_upstream.py) was also added to perform cereal schema instance generation and validation. These changes aim to enhance the testing process, ensuring schema compatibility and integration quality.
* Relocate cereal validation to a dedicated GitHub workflow
This commit introduces a distinct GitHub workflow for cereal validation named 'cereal_validation.yaml'. This workflow includes two jobs: one for generating cereal validation artifacts and another for validating cereal with the upstream project. Previously, these operations were included as separate jobs in 'selfdrive_tests.yaml'. However, the decoupling in this commit allows for a better organization of GitHub workflows within the project. Additionally, this separation allows these workflows to be individually configured and run, providing a greater degree of flexibility in managing our continuous integration activities.
* Rename workflow to "cereal validation" for clarity.
Updated the workflow name in the GitHub Actions configuration to better reflect its purpose. This change improves maintainability and ensures clearer identification of the workflow's function.
* Add LFS configuration and GitLab SSH setup to workflow
Integrate GitLab LFS handling by configuring LFS URLs and enabling SSH setup. This includes adding public GitLab keys and updating the workflow to support secure connections for LFS operations. Ensures proper handling of large files and seamless integration with GitLab.
* rename
* format
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
Integrate GitLab LFS handling by configuring LFS URLs and enabling SSH setup. This includes adding public GitLab keys and updating the workflow to support secure connections for LFS operations. Ensures proper handling of large files and seamless integration with GitLab.
This change includes the current commit SHA in the workflow outputs of the sunnypilot-build-prebuilt.yaml file. It provides better traceability for builds, ensuring each workflow run is linked to the exact commit it was triggered from.
Added checks for new UI files in PRs
The git workflow script `ui_preview.yaml` has been modified. The script now checks if the master branch contains a file corresponding to a UI file present in the PR. If a UI file in the PR does not have a match on the master branch, it is marked as new. These enhancements improve the comparison of UI changes between the master and PR branches, particularly with the identification of new UI files.
* stash
* fastest
* faster but not fastest
* clean up
* here too
* fix that
* revert
* already sorted
* rev
* clean up
* allow empty
* lower tm time for cache miss
* test
* Fix runs-on so that things acutally run otherwise they get stuck due to gh change
* Add repository condition to process_replay in CI workflow
Restrict the process_replay job to run only for the commaai/openpilot repository. This change temporarily blocks execution for forks or other repositories.
* in another PR
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Update workflow condition to skip forked PRs during sync
This change ensures the LFS maintenance workflow does not run for pull requests originating from forked repositories. It enhances efficiency by avoiding unnecessary sync actions in these cases.
* Modified retry conditions in GitHub workflow file
Adjusted the retry conditions in the GitHub workflow setup file. The updated condition allows for a second attempt if the pull request originator is a 'FIRST_TIME_CONTRIBUTOR' from a forked repository. This modification lends more flexibility to first-time contributors by providing them with an additional chance to pass the tests.
* test
* test2
* test3
* Fix conditional logic for retry check in workflow script
The condition now correctly checks if the pull request is not from a fork using 'false' instead of 'true'. This ensures proper validation of retry attempts based on the contributor and repository source.
* Fix placement of event JSON logging in workflow script
Moved the `toJSON(github.event)` logging statement to ensure it runs after retry logic validation. This improves the logical flow of the script and maintains appropriate debugging output.
* Add debug logs for PR metadata and run attempt values
This change introduces debug logs to output key PR metadata, such as fork status, author association, and the GitHub run attempt value. These logs will aid in debugging and provide better visibility into workflow execution.
* Add debug log for PR fork and author association check
This change adds a debug log to display the evaluation of the fork and author association condition in the workflow. It helps in troubleshooting PR handling logic and ensures better visibility into the condition being assessed.
* Fix logical negation for fork checks in workflow script
Correct the evaluation of fork status using proper negation syntax. This ensures accurate logic in pull request conditions, improving the reliability of the workflow execution.
* Add log for first-time contributor check in workflow
This adds a debug statement to log whether the PR author is a first-time contributor. It helps improve observability and makes it easier to debug workflows involving author associations.
* Simplify log output in GitHub action script
Removed unnecessary debug echo statements to streamline the script and improve readability. Retained essential logic to handle retries and enforce restrictions on test flakiness.
* Simplify retry logic for pull request contributors.
Updated the condition to correctly identify and handle retries for first-time contributors and forked repositories. This ensures a clearer and more accurate retry policy in the workflow.
* Refine workflow conditions for PR checks and retries
Adjusted conditional logic in `lfs-maintenance.yaml` to improve readability and ensure proper handling of draft PRs. Updated retry logic in `setup/action.yaml` to reflect correct author association for stricter validations.
* Update workflow condition to skip forked PRs during sync
This change ensures the LFS maintenance workflow does not run for pull requests originating from forked repositories. It enhances efficiency by avoiding unnecessary sync actions in these cases.
* Modified retry conditions in GitHub workflow file
Adjusted the retry conditions in the GitHub workflow setup file. The updated condition allows for a second attempt if the pull request originator is a 'FIRST_TIME_CONTRIBUTOR' from a forked repository. This modification lends more flexibility to first-time contributors by providing them with an additional chance to pass the tests.
