Files
sunnypilot/selfdrive/controls/lib/longitudinal_planner.py
DevTekVE 9b7502bd85 model: Refactor modeld with modular runners and split model support (#877)
* Refactor model runner methods for improved abstraction.

Moved slicing logic to a private `_slice_outputs` method and decoupled `_run_model` for clearer subclass implementation. Removed redundant `output` attribute in `ModelState` to streamline data handling.

* Add output parsing to model_runner and remove duplicate logic

Integrates an output parser directly into `model_runner` for streamlined inference and parsing. Removes redundant parser initialization from `modeld` to avoid duplication and enhance maintainability.

* Reordering

* linter

* linter

* Refactor model handling with `ModelData` abstraction

Introduce a `ModelData` class to encapsulate model and metadata logic, improving code clarity and modularity. Refactor `ModelRunner` to manage multiple models and add conditional handling for fallback scenarios. Adjust `TinygradRunner` to validate and use the new `ModelData` structure.

* Refactor model handling to use dictionary-based structure

Replaces the model list with a dictionary keyed by model type to improve clarity and maintainability. Updates related logic and ensures consistent handling of model metadata and inputs. Adds `slice_outputs` implementation to the `TinygradRunner` for proper output parsing.

* Refactor model runners to support policy and vision separation

Introduced `TinygradPolicyRunner`, `TinygradVisionRunner`, and `TinygradSplitRunner` to enable separate handling of policy and vision models. Updated `TinygradRunner` initialization and input preparation to accommodate modular processing. Adjusted `modeld` to utilize the new runners, ensuring compatibility with separated model workflows.

* Refactor model runner initialization and simplify logic

Introduce `get_model_runner` to centralize model runner selection logic, replacing multiple conditional instantiations. Simplify the handling of model metadata by removing fallback logic and restructuring output slicing to enforce proper loading of model data. These changes improve code maintainability and clarity.

* Refactor model data access for TinygradRunner initialization

Update references to access nested artifact properties and align with structural changes to the model data schema. This simplifies input shapes handling and ensures compatibility with updated model attributes.

* Refactor imports and clean up redundant code.

Removed unused imports and improved formatting for clarity and maintainability. These changes simplify the codebase by eliminating unnecessary dependencies and ensuring consistency.

* Refactor model output parsing with specialized parsers.

Introduce abstract `_parse_outputs` method to standardize parsing logic. Add `SplitParser` for specialized parsing in `TinygradVisionRunner` and `TinygradPolicyRunner`. This improves modularity and paves the way for easier parser customization.

* Add parser for model output processing in modeld_v2

Introduce a new `Parser` class to handle parsing and processing of model outputs, including MDN, binary cross-entropy, and categorical cross-entropy outputs. This modularizes the logic, improves clarity, and prepares for handling various types of model data.

* Add `input_shapes` property to model runners

Introduce a new `input_shapes` property in the abstract base class and its implementation in derived classes. This provides a standardized way to access the input shapes of models, improving clarity and consistency in the model runners.

* Refactor model runner to use private `_model_data` attribute

Replaced public `model_data` with private `_model_data` for improved encapsulation. Updated all references and property accessors accordingly. Simplified model type handling by using raw types where applicable.

* Remove debug print statement from model_runner.py

The unnecessary `print(model_type)` statement was removed as it served no functional purpose in the code. This improves code cleanliness and avoids unintended console output during execution.

* Refactor `_parse_outputs` call in `run_model`.

Replaced the use of `self.parser.parse_outputs` with `self._parse_outputs` for clarity and consistency. Updated method signature to align with the revised usage.

* Refactor model output parsing for clarity and scope separation

Moved specific parsing logic (e.g., lane_lines, lead) from `parse_model_outputs` to `parse_policy_outputs` to better align with functional responsibilities. This improves modularity and readability while maintaining existing functionality.

* Refactor model_runner to simplify result handling

Renamed variable `result` to `parsed_result` for clarity and removed unnecessary slicing during model output parsing. These changes improve code readability and maintain consistency within the `run_model` method.

* Adjust _parse_outputs method signature in model_runner

    Updated the method signature of _parse_outputs to accept a single np.ndarray instead of a dictionary. This aligns with the intended data structure and ensures consistency across subclasses implementing this abstract method.

* Refactor ModelRunner to enforce abstract base class compliance

Updated `ModelRunner` and its subclasses to properly inherit from `ABC` while refactoring methods to ensure compliance with Python's abstract base class standards. Streamlined the handling of `_parse_outputs` and added a new `input_shapes` property for improved functionality.

* Fix buffer length issue in 20Hz model initialization

Adjusted `FULL_HISTORY_BUFFER_LEN` by adding +1 for `full_features_20Hz` to address compatibility issues with the current FoF model. Added a comment noting potential failure for other models with this adjustment.

* Refactor TinygradRunner to remove abstract methods.

Simplified the TinygradRunner class by removing unnecessary @abstractmethod decorators and redundant method definitions. This streamlines the code and aligns it more effectively with its current usage and implementation.

* Refactor model runner classes and enhance type annotations

Simplified model runner implementations, added type annotations, and improved code readability and maintainability. Introduced new type definitions, updated metadata handling, and standardized input/output parsing across all runner classes. Minor comment update in `modeld.py` for clarity.

* Refactor model runner classes with detailed docstrings.

Enhanced class and function docstrings across model_runner.py for better clarity and maintainability. Descriptions now include detailed explanations of attributes, purposes, and workflows to aid understanding and future development.

* Refactor model runner classes and add TICI hardware optimization

Simplified and clarified class definitions, comments, and functionality for ModelRunner subclasses. Introduced the use of QCOM environment variable on TICI for potential hardware acceleration. Enhanced input/output handling and error reporting across Tinygrad and ONNX implementations.

* Update parser import and usage to use CombinedParser

Replaced the Parser class with CombinedParser in model_runner.py. This change ensures consistency with the updated parsing logic, aligning with the latest requirements for combined model output handling.

* Refactor TinygradRunner hierarchy for modular parsers

Reorganized the TinygradRunner and its specialized runners (Vision, Policy, and Supercombo) into a cleaner, modular structure using composable classes. This consolidates parser logic, removes redundancy, and simplifies initialization by leveraging a shared base class with a dictionary-based parser method.

* Refactor model runners to use ModularRunner as abstract base.

Introduce a new `ModularRunner` class to enforce a consistent interface across model runners. Updated existing runners, including `ModelRunner`, `SupercomboTinygrad`, `PolicyTinygrad`, and `VisionTinygrad`, to extend `ModularRunner`. Added abstract methods and properties to enhance modularity and code maintainability.

* Refactor model runners into modular components.

This commit separates the logic for Tinygrad, ONNX, and split runners into clearly defined modules and components. It introduces `PolicyTinygrad`, `VisionTinygrad`, `SupercomboTinygrad`, and centralized helpers for cleaner architecture. The changes improve modularity and maintainability of the model running and parsing workflows.

* Simplify imports and clean up unused code in ONNXRunner.

Removed unused imports and redundant environmental variables to streamline the codebase. Consolidated necessary imports and organized type definitions for improved readability and maintenance.

* Standardize imports and add model data validation.

Updated import paths to ensure consistency across modules by using `openpilot` as the base. Introduced validation in `_parse_outputs` methods to handle cases where `_model_data` is not initialized, preventing potential runtime errors.

* Remove unused import and fix whitespace in runners

The unused import `ModelData` was removed from `tinygrad_runner.py` to clean up the code. Additionally, extraneous whitespace was corrected in `onnx_runner.py` for improved readability and consistency.

* Remove unnecessary blank line in import statements

Cleaned up import section by removing an extra blank line. This helps maintain consistency and adheres to code style conventions.

* BROKEN!! Staging code but its not gonna work. Also I realized we need to run the split models in 2 stages because the output of one is immediately needed for the input of the other. We might handle it inside of the model_runner instead

* update smooth

* Revert "update smooth"

This reverts commit c335712e6e1ee189459ce34dfc9d4028feb9470f.

* match case made this very hard to read.

* shouldnt be there

* Refactor to allow TR (soon TM)

* TR 7 is .1

* metadata

* .2=3

* Remove redundant comments and clean up conditional blocks in modeld.py.

* Undoing wrong buffer

* Refactor model initialization and adjust ONNX runner import

Reorganized numpy input buffer initialization and updated `temporal_idxs` logic for better clarity and efficiency. Conditional import for ONNXRunner added for non-TICI platforms to optimize imports. These changes improve maintainability and compatibility across platforms.

* Update CURRENT_SELECTOR_VERSION to 4

Bump the CURRENT_SELECTOR_VERSION constant from 3 to 4 to reflect changes in the selector logic or requirements. This ensures compatibility with the updated selector version while maintaining the minimum required version as 2.

* Add output_slices property to model runners

Introduce output_slices property to provide access to the output slices for individual and combined models. This ensures consistent handling of output slices across vision and policy models, improving modularity and usability.

* Refactor imports to use SplitModelConstants consistently

Updated import references to use the renamed `SplitModelConstants` class for consistency across files. This change ensures clarity and better alignment with the updated class naming convention.

* Refactor buffer initialization and desired curvature handling

Refactored model input buffer initialization for improved clarity and consistency, leveraging dynamic shape calculations. Extracted `process_desired_curvature` method to encapsulate logic for handling 3D and non-3D cases. Simplified temporal index generation and related calculations for better maintainability.

* Refactor desire reshape dims logic in modeld.py

Adjust logic for setting desire reshape dimensions to handle `is_20hz_3d` separately. This improves clarity and ensures proper handling of different model runner configurations.

* Fix off-by-one error in full_desire buffer initialization

The full_desire buffer length was mistakenly set to full_history_buffer_len + 1. This change corrects it to match the intended full_history_buffer_len, ensuring proper alignment with other buffers.

* Simplify desire reshape logic by removing unused condition.

Removed the `is_20hz_3d` condition and associated reshape logic since it is no longer needed. This streamlines the code and avoids unnecessary checks for unused configurations.

* Refactor buffer initialization for 20Hz model variants

Reorganized buffer initialization logic to prioritize the 20Hz_3D condition. This improves clarity and ensures specific handling of different 20Hz configurations. Adjusted the order of conditions to streamline execution flow.

* Refactor: Move `get_action_from_model` and constants to `Model` class to improve encapsulation and readability.

* 12 line reshaping red diff

* .2 to match old models delay

* mypy fixes

* Revert "12 line reshaping red diff"

This reverts commit 8c7280f629.

* mypy

* remove this

* Fix desired curvature for models which do not output desired_curvature

* fix FoF

* flip policy and vision outs to allow FoF and tomb raider to live in harmony using conditional `if 'this' in outs:'`

* noqa

* single

* sunnypilot modeld.py

* action

* overrides methodology

* combine split outputs to its own method

* comments

* Fix static checker line length

* static will fail on line length
lines:

286,
206,
70 - 77,
159,
168

* Address E501 line length violations

* This will make TR better while not effecting FoF/VFF at all

* Reduce this to one conditional and just call normally in vision/policy

* Align with upstream in our own way.

* check for desired curvature in outputs first

* outputs

* Use a cleaner import method

* Fix output

* Clean up some values

* Only call on init

* slight cleanup

* names!!!!!!!!!

* Refactor overrides structure to support key-value pairs.

The overrides structure now uses a list of key-value pairs instead of fixed lat/long fields. This change improves flexibility, allowing dynamic addition of override parameters. Code adjustments ensure backward compatibility and consistent behavior throughout the application.

* Refactor: Use local variable for SplitModelConstants

Introduce a local `constants` variable to replace repeated access to `SplitModelConstants`. This simplifies code readability and adheres to linter recommendations for line length.

* Refactor model constant handling to improve modularity

Replaced direct usage of model constants with dynamic access through model runners for better scalability and maintainability. This change centralizes constant definitions, reduces redundancy, and ensures clearer integration with different model types.

---------

Co-authored-by: discountchubbs <alexgrant990@gmail.com>
Co-authored-by: Discountchubbs <159560811+Discountchubbs@users.noreply.github.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2025-05-25 17:10:16 +02:00

212 lines
9.5 KiB
Python
Executable File

#!/usr/bin/env python3
import math
import numpy as np
import cereal.messaging as messaging
from opendbc.car.interfaces import ACCEL_MIN, ACCEL_MAX
from openpilot.common.conversions import Conversions as CV
from openpilot.common.filter_simple import FirstOrderFilter
from openpilot.common.realtime import DT_MDL
from openpilot.selfdrive.modeld.constants import ModelConstants
from openpilot.selfdrive.controls.lib.longcontrol import LongCtrlState
from openpilot.selfdrive.controls.lib.longitudinal_mpc_lib.long_mpc import LongitudinalMpc
from openpilot.selfdrive.controls.lib.longitudinal_mpc_lib.long_mpc import T_IDXS as T_IDXS_MPC
from openpilot.selfdrive.controls.lib.drive_helpers import CONTROL_N, get_speed_error, get_accel_from_plan
from openpilot.selfdrive.car.cruise import V_CRUISE_MAX, V_CRUISE_UNSET
from openpilot.common.swaglog import cloudlog
from openpilot.sunnypilot.selfdrive.controls.lib.longitudinal_planner import LongitudinalPlannerSP
LON_MPC_STEP = 0.2 # first step is 0.2s
A_CRUISE_MAX_VALS = [1.6, 1.2, 0.8, 0.6]
A_CRUISE_MAX_BP = [0., 10.0, 25., 40.]
CONTROL_N_T_IDX = ModelConstants.T_IDXS[:CONTROL_N]
ALLOW_THROTTLE_THRESHOLD = 0.5
MIN_ALLOW_THROTTLE_SPEED = 2.5
# Lookup table for turns
_A_TOTAL_MAX_V = [1.7, 3.2]
_A_TOTAL_MAX_BP = [20., 40.]
def get_max_accel(v_ego):
return np.interp(v_ego, A_CRUISE_MAX_BP, A_CRUISE_MAX_VALS)
def get_coast_accel(pitch):
return np.sin(pitch) * -5.65 - 0.3 # fitted from data using xx/projects/allow_throttle/compute_coast_accel.py
def limit_accel_in_turns(v_ego, angle_steers, a_target, CP):
"""
This function returns a limited long acceleration allowed, depending on the existing lateral acceleration
this should avoid accelerating when losing the target in turns
"""
# FIXME: This function to calculate lateral accel is incorrect and should use the VehicleModel
# The lookup table for turns should also be updated if we do this
a_total_max = np.interp(v_ego, _A_TOTAL_MAX_BP, _A_TOTAL_MAX_V)
a_y = v_ego ** 2 * angle_steers * CV.DEG_TO_RAD / (CP.steerRatio * CP.wheelbase)
a_x_allowed = math.sqrt(max(a_total_max ** 2 - a_y ** 2, 0.))
return [a_target[0], min(a_target[1], a_x_allowed)]
class LongitudinalPlanner(LongitudinalPlannerSP):
def __init__(self, CP, init_v=0.0, init_a=0.0, dt=DT_MDL):
self.CP = CP
self.mpc = LongitudinalMpc(dt=dt)
self.mpc.mode = 'acc'
LongitudinalPlannerSP.__init__(self, self.CP, self.mpc)
self.fcw = False
self.dt = dt
self.allow_throttle = True
self.a_desired = init_a
self.v_desired_filter = FirstOrderFilter(init_v, 2.0, self.dt)
self.prev_accel_clip = [ACCEL_MIN, ACCEL_MAX]
self.v_model_error = 0.0
self.output_a_target = 0.0
self.output_should_stop = False
self.v_desired_trajectory = np.zeros(CONTROL_N)
self.a_desired_trajectory = np.zeros(CONTROL_N)
self.j_desired_trajectory = np.zeros(CONTROL_N)
self.solverExecutionTime = 0.0
@staticmethod
def parse_model(model_msg, model_error):
if (len(model_msg.position.x) == ModelConstants.IDX_N and
len(model_msg.velocity.x) == ModelConstants.IDX_N and
len(model_msg.acceleration.x) == ModelConstants.IDX_N):
x = np.interp(T_IDXS_MPC, ModelConstants.T_IDXS, model_msg.position.x) - model_error * T_IDXS_MPC
v = np.interp(T_IDXS_MPC, ModelConstants.T_IDXS, model_msg.velocity.x) - model_error
a = np.interp(T_IDXS_MPC, ModelConstants.T_IDXS, model_msg.acceleration.x)
j = np.zeros(len(T_IDXS_MPC))
else:
x = np.zeros(len(T_IDXS_MPC))
v = np.zeros(len(T_IDXS_MPC))
a = np.zeros(len(T_IDXS_MPC))
j = np.zeros(len(T_IDXS_MPC))
if len(model_msg.meta.disengagePredictions.gasPressProbs) > 1:
throttle_prob = model_msg.meta.disengagePredictions.gasPressProbs[1]
else:
throttle_prob = 1.0
return x, v, a, j, throttle_prob
def update(self, sm):
self.mode = 'blended' if sm['selfdriveState'].experimentalMode else 'acc'
LongitudinalPlannerSP.update(self, sm)
if dec_mpc_mode := self.get_mpc_mode():
self.mode = dec_mpc_mode
if len(sm['carControl'].orientationNED) == 3:
accel_coast = get_coast_accel(sm['carControl'].orientationNED[1])
else:
accel_coast = ACCEL_MAX
v_ego = sm['carState'].vEgo
v_cruise_kph = min(sm['carState'].vCruise, V_CRUISE_MAX)
v_cruise = v_cruise_kph * CV.KPH_TO_MS
v_cruise_initialized = sm['carState'].vCruise != V_CRUISE_UNSET
long_control_off = sm['controlsState'].longControlState == LongCtrlState.off
force_slow_decel = sm['controlsState'].forceDecel
# Reset current state when not engaged, or user is controlling the speed
reset_state = long_control_off if self.CP.openpilotLongitudinalControl else not sm['selfdriveState'].enabled
# PCM cruise speed may be updated a few cycles later, check if initialized
reset_state = reset_state or not v_cruise_initialized
# No change cost when user is controlling the speed, or when standstill
prev_accel_constraint = not (reset_state or sm['carState'].standstill)
if self.mode == 'acc':
accel_clip = [ACCEL_MIN, get_max_accel(v_ego)]
steer_angle_without_offset = sm['carState'].steeringAngleDeg - sm['liveParameters'].angleOffsetDeg
accel_clip = limit_accel_in_turns(v_ego, steer_angle_without_offset, accel_clip, self.CP)
else:
accel_clip = [ACCEL_MIN, ACCEL_MAX]
if reset_state:
self.v_desired_filter.x = v_ego
# Clip aEgo to cruise limits to prevent large accelerations when becoming active
self.a_desired = np.clip(sm['carState'].aEgo, accel_clip[0], accel_clip[1])
# Prevent divergence, smooth in current v_ego
self.v_desired_filter.x = max(0.0, self.v_desired_filter.update(v_ego))
# Compute model v_ego error
self.v_model_error = get_speed_error(sm['modelV2'], v_ego)
x, v, a, j, throttle_prob = self.parse_model(sm['modelV2'], self.v_model_error)
# Don't clip at low speeds since throttle_prob doesn't account for creep
self.allow_throttle = throttle_prob > ALLOW_THROTTLE_THRESHOLD or v_ego <= MIN_ALLOW_THROTTLE_SPEED
if not self.allow_throttle:
clipped_accel_coast = max(accel_coast, accel_clip[0])
clipped_accel_coast_interp = np.interp(v_ego, [MIN_ALLOW_THROTTLE_SPEED, MIN_ALLOW_THROTTLE_SPEED*2], [accel_clip[1], clipped_accel_coast])
accel_clip[1] = min(accel_clip[1], clipped_accel_coast_interp)
if force_slow_decel:
v_cruise = 0.0
self.mpc.set_weights(prev_accel_constraint, personality=sm['selfdriveState'].personality)
self.mpc.set_cur_state(self.v_desired_filter.x, self.a_desired)
self.mpc.update(sm['radarState'], v_cruise, x, v, a, j, personality=sm['selfdriveState'].personality)
self.v_desired_trajectory = np.interp(CONTROL_N_T_IDX, T_IDXS_MPC, self.mpc.v_solution)
self.a_desired_trajectory = np.interp(CONTROL_N_T_IDX, T_IDXS_MPC, self.mpc.a_solution)
self.j_desired_trajectory = np.interp(CONTROL_N_T_IDX, T_IDXS_MPC[:-1], self.mpc.j_solution)
# TODO counter is only needed because radar is glitchy, remove once radar is gone
self.fcw = self.mpc.crash_cnt > 2 and not sm['carState'].standstill
if self.fcw:
cloudlog.info("FCW triggered")
# Interpolate 0.05 seconds and save as starting point for next iteration
a_prev = self.a_desired
self.a_desired = float(np.interp(self.dt, CONTROL_N_T_IDX, self.a_desired_trajectory))
self.v_desired_filter.x = self.v_desired_filter.x + self.dt * (self.a_desired + a_prev) / 2.0
action_t = self.CP.longitudinalActuatorDelay + DT_MDL
output_a_target_mpc, output_should_stop_mpc = get_accel_from_plan(self.v_desired_trajectory, self.a_desired_trajectory, CONTROL_N_T_IDX,
action_t=action_t, vEgoStopping=self.CP.vEgoStopping)
output_a_target_e2e = sm['modelV2'].action.desiredAcceleration
output_should_stop_e2e = sm['modelV2'].action.shouldStop
if self.mode == 'acc':
output_a_target = output_a_target_mpc
self.output_should_stop = output_should_stop_mpc
else:
output_a_target = min(output_a_target_mpc, output_a_target_e2e)
self.output_should_stop = output_should_stop_e2e or output_should_stop_mpc
for idx in range(2):
accel_clip[idx] = np.clip(accel_clip[idx], self.prev_accel_clip[idx] - 0.05, self.prev_accel_clip[idx] + 0.05)
self.output_a_target = np.clip(output_a_target, accel_clip[0], accel_clip[1])
self.prev_accel_clip = accel_clip
def publish(self, sm, pm):
plan_send = messaging.new_message('longitudinalPlan')
plan_send.valid = sm.all_checks(service_list=['carState', 'controlsState', 'selfdriveState'])
longitudinalPlan = plan_send.longitudinalPlan
longitudinalPlan.modelMonoTime = sm.logMonoTime['modelV2']
longitudinalPlan.processingDelay = (plan_send.logMonoTime / 1e9) - sm.logMonoTime['modelV2']
longitudinalPlan.solverExecutionTime = self.mpc.solve_time
longitudinalPlan.speeds = self.v_desired_trajectory.tolist()
longitudinalPlan.accels = self.a_desired_trajectory.tolist()
longitudinalPlan.jerks = self.j_desired_trajectory.tolist()
longitudinalPlan.hasLead = sm['radarState'].leadOne.status
longitudinalPlan.longitudinalPlanSource = self.mpc.source
longitudinalPlan.fcw = self.fcw
longitudinalPlan.aTarget = float(self.output_a_target)
longitudinalPlan.shouldStop = bool(self.output_should_stop)
longitudinalPlan.allowBrake = True
longitudinalPlan.allowThrottle = bool(self.allow_throttle)
pm.send('longitudinalPlan', plan_send)
self.publish_longitudinal_plan_sp(sm, pm)