基于参数描述的换道场景自动驾驶精确决策学习

Precise Decision-Making Learning for Automated Vehicles in Lane-Change Scenario Based on Parameter Description

摘要:

为提高车辆驾驶安全性并充分考虑人类驾驶员对于自动驾驶控制系统的接受度，研究并实现了自动驾驶车辆在换道场景下的精确决策学习。汽车自动驾驶不仅需要决策是否换道，还需要决定汽车的具体微观行为，如换道时间和期望加速度的确定等，因此，车道变换的精确决策需使用3个参数来描述，并需要通过强化学习求解。这种基于参数精确决策的合理性体现在两个方面：首先是不同的决策参数值会影响规划的轨迹，如果决策不精确，将产生运动的不确定性；其次是基于真实交通数据（NGSIM）的分析，因为人类换道行为在换道时间和期望加速度上存在显著的差异性，在当前的决策研究中很少被明确考虑。此外，发现NGSIM数据中存在一些潜在的紧急情况，可以通过优化部分决策参数来提升其安全性；在强化学习算法的设计中，动作过程中加入换道时间和期望加速度；奖励函数考虑了安全性、当前驾驶员的意愿和平均人类驾驶风格；问题求解中，自定义了基函数，并通过基于核函数的最小二乘策略迭代强化学习方法学习精确的安全决策行为。仿真结果表明，使用强化学习参数决策可以实现更精确的决策，从而提高安全性能，并可在变道场景中模仿人类驾驶员的行为。

Abstract:

To promote safety and fully consider human drivers' acceptance， precise decision-making is realized for automated vehicles under the lane-change scenario in this paper. More specifically， automated vehicles not only decide to change lanes or not but also decide specific microcosmic behaviors， such as lane-change time and expected acceleration. Thus， precise decisions for lane-change are described with three parameters and learned by reinforcement learning. The rationality of such parameter-based precise decisions is shown in two aspects. First， different values of decision parameters will notably influence the planned trajectory， which means other microcosmic behaviors will be a significant uncertainty when they are not precisely decided in the decision-making layer. Secondly， based on the analysis of real traffic data， NGSIM， changeable lane-change time， and expected acceleration are revealed in lane-change behaviors， which is seldom explicitly considered in the decision-making layer of current researches. The decision parameters that include lane-change time and expected acceleration are learned with kernel-based least-squares policy iteration reinforcement learning （KLSPI）. Safety， current driver's willingness， and average human driving style are considered in the reward function. Simulation results demonstrate that using reinforcement learning （RL） to learn decision parameters can realize more precise decisions， promote safety performance， and imitate human drivers' behaviors in the lane-change scenario.