Supplementary material

Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning

This page hosts the supplementary material referenced in our submission. It contains additional model details, experiment details, and results. The key formulations, architectures and calibrated parameters of the human behaviour models are summarised below; the complete derivations and all remaining details are in the supplementary PDF.

Highway driving models

Summary of Sec. II of supp.pdf. All models are trained or calibrated on the highD dataset and control the surrounding vehicles in highway-fast-v0.

Rule-based models

Car-following parameters are inferred with Automatic Differentiation Variational Inference (ADVI) under a hierarchical Bayesian model, so each vehicle $i$ gets its own parameter drawn from a population distribution. Positive parameters use a softplus link, $\theta_i = \log(1+e^{\theta_i^*})$ with $\theta_i^* \sim \mathcal{N}(\mu_\theta, \sigma_\theta^2)$, and observed speeds have Gaussian noise $v \sim \mathcal{N}(\mu_v, \sigma_{\text{obs}}^2)$. The tables report the posterior of the population-level hyperparameters $(\mu_\theta, \sigma_\theta)$ on the latent (pre-softplus) scale, so negative means are possible. 2,000 posterior samples are drawn for all downstream experiments.

MOBIL (lane changing)

A lane change is executed only if it is safe for the new follower and gives enough incentive:

$$\tilde{a}_n \ge -b_{\text{safe}}, \qquad \tilde{a}_c - a_c + p\left(\tilde{a}_n - a_n + \tilde{a}_o - a_o\right) \ge \Delta a_{\text{th}}$$

Here $c$, $o$ and $n$ are the ego, old follower and new follower, and the tilde marks accelerations after the change. Lane changes are rare in highD (about 5,600 complete events), so MOBIL is not calibrated. Its parameters are sampled per vehicle around standard literature values:

  • $p \sim |\mathcal{N}(0,\,0.25)|$
  • $\Delta a_{\text{th}} \sim \mathcal{N}(0.1,\,0.05)$ m/s²
  • $b_{\text{safe}} \sim \mathcal{N}(4,\,2)$ m/s²

Details: supp.pdf, Sec. II-A

Gipps' model

$$v_n(t+\tau) = \min\left\{v_n^{\text{free}},\; v_n^{\text{safe}}\right\}, \quad v_n^{\text{free}} = v_n + 2.5\,a_n\tau\Big(1-\tfrac{v_n}{v_n^{\max}}\Big)\sqrt{0.025+\tfrac{v_n}{v_n^{\max}}}$$
$$v_n^{\text{safe}} = b_n\tau + \sqrt{(b_n\tau)^2 - b_n\Big[2s_n - v_n\tau - \tfrac{v_{n-1}^2}{b_{n-1}}\Big]}, \qquad s_n = x_{n-1}-x_n-l_{n-1}-L$$

The $\min$ is not differentiable, so the data are split by distance headway (DHW). Samples with DHW < 25 m fit the safe margin $L$ through $v^{\text{safe}}$, and samples with DHW ≥ 25 m fit the desired speed $v^{\max}$ through $v^{\text{free}}$.

Fixed

  • $a_n = 1.7$ m/s²
  • $b_n = -3.4$ m/s²
  • $b_{n-1} = -3$ m/s²
  • $\tau = 0.6$ s

Priors

  • $\mu_L \sim \mathcal{N}(2,1)$, $\sigma_L \sim \text{HN}(0.5)$
  • $\mu_{v}^{\max} \sim \mathcal{N}(20,1)$, $\sigma_{v}^{\max} \sim \text{HN}(0.5)$
  • $\sigma_{\text{obs}} \sim \text{HN}(1)$
  • ADVI: 30k iters ($L$), 100k iters ($v^{\max}$)
ADVI posterior: population-level hyperparameters
ParameterMeanSDHDI 3%HDI 97%
$\mu_L$2.4860.0682.3582.610
$\sigma_L$1.3320.0691.2111.466
$\sigma_{\text{obs}}$ ($L$ fit)1.4660.0211.4301.508
$\mu_v^{\max}$32.3360.21631.92632.734
$\sigma_v^{\max}$5.5410.1365.2925.805
$\sigma_{\text{obs}}$ ($v^{\max}$ fit)0.2430.0020.2400.246

Details: supp.pdf, Sec. II-B (Tables I–II)

Krauß model

$$v_{\text{safe}} = v_{n-1} + \frac{d_n - v_{n-1}\tau}{\frac{v_n+v_{n-1}}{2b_n}+\tau}, \qquad v_n(t+\Delta t) = \min\left(v_{\text{safe}},\; v_n + a_n\Delta t,\; v_n^{\max}\right)$$

Here $d_n = x_{n-1}-x_n-l_{n-1}$, with no safe margin. Only the reaction time $\tau$ is inferred, from samples with DHW < 25 m. All other parameters are the same as for Gipps, except $b_n = 3.4$ m/s² (positive).

  • $\mu_\tau \sim \mathcal{N}(0.8,\,0.3^2)$, $\sigma_\tau \sim \text{HN}(0.5)$
  • ADVI: 20k iters
ADVI posterior: population-level hyperparameters
ParameterMeanSDHDI 3%HDI 97%
$\mu_\tau$0.0870.051−0.0060.184
$\sigma_\tau$1.3410.0451.2561.423
$\sigma_{\text{obs}}$0.9950.0200.9591.034

Details: supp.pdf, Sec. II-C (Table III)

Intelligent Driver Model (IDM)

$$\dot{v}_n = a_n\left[1-\Big(\frac{v_n}{v^{\max}}\Big)^{\delta} - \Big(\frac{d_n^*}{d_n}\Big)^2\right], \qquad d_n^* = d_0 + T v_n + \frac{v_n(v_n - v_{n-1})}{2\sqrt{a_n b_n}}$$

IDM is a second-order model and fully differentiable, so every observation is used. Three parameters are inferred: desired speed $v^{\max}$, jam distance $d_0$ and safe time headway $T$.

Fixed

  • $\delta = 4$
  • $a_n = 0.73$ m/s²
  • $b_n = 1.67$ m/s²

Priors

  • $\mu_v^{\max} \sim \mathcal{N}(30,1)$, $\sigma_v^{\max} \sim \text{HN}(0.5)$
  • $\mu_{d_0} \sim \mathcal{N}(2,1)$, $\sigma_{d_0} \sim \text{HN}(1)$
  • $\mu_T \sim \mathcal{N}(1.5,\,0.5^2)$, $\sigma_T \sim \text{HN}(0.5)$
  • ADVI: 15k iters
ADVI posterior: population-level hyperparameters
ParameterMeanSDHDI 3%HDI 97%
$\mu_v^{\max}$32.6710.09632.49132.848
$\sigma_v^{\max}$1.2490.0851.1021.408
$\mu_{d_0}$0.3710.0880.2100.540
$\sigma_{d_0}$1.3380.0851.1831.504
$\mu_T$−0.8120.086−0.979−0.664
$\sigma_T$1.2580.0841.1101.427
$\sigma_{\text{obs}}$2.0940.1371.8632.371

Details: supp.pdf, Sec. II-D (Table IV)

Unif and Unif-W (random baselines)

  • Unif: $a^{\text{lon}} \sim U(-4,3)$, $a^{\text{lat}} \sim U(-2,2)$
  • Unif-W: $a^{\text{lon}} \sim U(-4,3)$, $a^{\text{lat}} \sim U(-6,6)$

Details: supp.pdf, Sec. II-E

Data-driven models

All three are trained by behaviour cloning on highD state–action pairs. The data are downsampled from 25 to 5 fps and split 80/10/10, giving about 1.7M training samples. The 48-dimensional ego-centric state is $\mathbf{s} = [v_e^{\text{lon}}, v_e^{\text{lat}}, l_e, \delta y_e, \mathbb{1}_{\text{left}}, \mathbb{1}_{\text{right}}, s_p, s_{la}, s_{lp}, s_{lf}, s_{ra}, s_{rp}, s_{rf}]$. Each of the seven neighbours is encoded as $s_i = [\mathbb{1}_i, l_i, \Delta x_i, \Delta y_i, \Delta v_i^{\text{lon}}, \Delta v_i^{\text{lat}}]$ and set to zero if absent. The action is $\mathbf{a} = [a^{\text{lon}}, a^{\text{lat}}]$.

MLP

$$\mathcal{L}_{\text{MLP}} = \frac{1}{B}\sum_{i=1}^{B}\big\|\hat{\mathbf{a}}_i - \mathbf{a}_i^{\text{gt}}\big\|_2^2$$
  • 2 hidden layers × 64, ReLU
  • batch 1024
  • lr $10^{-3}$
  • ≤100 epochs, patience 10
  • test loss 0.62

Details: supp.pdf, Sec. II-F

Mixture density network (MDN)

$$p(\mathbf{a}\mid\mathbf{s}) = \sum_{k=1}^{K}\alpha_k(\mathbf{s})\,\mathcal{N}\big(\mathbf{a}\mid\boldsymbol{\mu}_k(\mathbf{s}), \operatorname{diag}(\sigma_{k,\text{lon}}^2, \sigma_{k,\text{lat}}^2)\big)$$
$$\mathcal{L}_{\text{MDN}} = \text{NLL} + \lambda_{\text{usage}}\, D_{\mathrm{KL}}(\bar{\mathbf{r}}\,\|\,U) + \lambda_{\text{div}}\,\frac{1}{K(K-1)B}\sum_{i}\sum_{k\neq j}\exp\!\big(-D_{\mathrm{KL}}(\mathcal{N}_k^{(i)}\|\mathcal{N}_j^{(i)})\big)$$

$\bar{\mathbf{r}}$ is the batch-averaged component responsibility. The usage term discourages mode collapse, and the diversity term penalises overlapping components.

  • $K = 15$ components
  • 2 hidden layers × 128, ReLU
  • $\lambda_{\text{usage}} = 0.05$
  • $\lambda_{\text{div},e} = 0.01\,e^{-0.05e}$
  • batch 512, lr $10^{-3}$
  • test loss 1.88

Details: supp.pdf, Sec. II-G

Attention-based model (CVAE)

Attention-based highway model: the current ego state queries its history via multi-head attention to give z_ego, which queries the surrounding-vehicle states to give z_inter; their concatenation z conditions a CVAE that outputs the ego action.
The current ego state $\mathbf{s}^t_{\text{ego}}$ attends to its past $h$ states to produce $\mathbf{z}_{\text{ego}}$, which then attends to the $N=7$ surrounding vehicles over $h+1$ steps to produce $\mathbf{z}_{\text{inter}}$. Their concatenation $\mathbf{z}$ conditions a CVAE. During training the encoder also sees the ground-truth action $\mathbf{a}^t_{\text{ego}}$; at inference $\mathbf{z}_{\text{lat}}$ is sampled from the prior.
$$\mathcal{L}_{\text{CVAE}} = \mathbb{E}_{q_\phi(\mathbf{z}_{\text{lat}}\mid\mathbf{a},\mathbf{z})}\big\|\mathbf{a} - f_\theta([\mathbf{z},\mathbf{z}_{\text{lat}}])\big\|_2^2 + \beta\, D_{\mathrm{KL}}\big(q_\phi(\mathbf{z}_{\text{lat}}\mid\mathbf{a},\mathbf{z})\,\|\,\mathcal{N}(0,I)\big)$$

At inference $\mathbf{z}_{\text{lat}} \sim \mathcal{N}(0,I)$, which gives stochastic actions. The design is a simplified attention-based variant of Trajectron++, with attention in place of LSTMs and direct action prediction.

  • history $h = 4$
  • 4 heads, $d_z = 32$, dropout 0.1, LayerNorm + residual
  • encoder 2-layer MLP (64), decoder 3-layer MLP (64)
  • $d_{\text{lat}} = 16$, $\beta = 1$
  • AdamW, lr $10^{-3}$, wd $10^{-4}$, batch 64, grad clip 1.0
  • test loss 0.52

Details: supp.pdf, Sec. II-H (Fig. 1)

Validation against highD

$D_{\mathrm{KL}}(P_{\text{highD}}\,\|\,Q_{\text{model}})$ of driving-feature distributions (lower is better; best in bold)
ModelVelocityRel. preceding vel.Distance headwayAcceleration
Gipps0.0240.5100.2140.148
Krauß0.0670.5150.3160.188
IDM0.1420.6870.2190.087
MLP0.0100.0710.0130.029
MDN0.0020.3380.0310.024
Attention0.0010.3400.0530.267
Unif0.0300.0480.0240.099
Unif-W0.0320.0550.0310.099

Feature distributions and summary statistics: supp.pdf, Sec. II-I

Highway scenario

Simulated trajectories from our calibrated human model in the highway setting.

Handover giver models

Summary of Sec. III of supp.pdf. Models generate the human giver's right-hand trajectory and are fitted to 1,281 preprocessed giver trajectories from handover@RPL.

Rule-based models

Minimum jerk

$$\min_{\mathbf{p}(t)}\int_0^T\Big\|\tfrac{d^3\mathbf{p}}{dt^3}\Big\|_2^2 dt \;\;\Rightarrow\;\; \mathbf{p}(t) = \mathbf{p}_0 + \big(10\tau^3 - 15\tau^4 + 6\tau^5\big)(\mathbf{p}_T - \mathbf{p}_0), \quad \tau = t/T$$

The model has no learnable parameters. The start position $\mathbf{p}_0$, end position $\mathbf{p}_T$ and duration $T$ are each sampled from Gaussians fitted to the human training data.

Details: supp.pdf, Sec. III-A

Elliptical model

Each trajectory is approximately planar: PCA variance along PC3 is significantly smaller than along PC1 and PC2 (RM-ANOVA, $F(2,1136)=6628.24$, $p_{\mathrm{GG}}\lt 0.001$). Each trajectory is therefore projected onto its PC1–PC2 plane, and an ellipse is fitted there by direct least squares (Fitzgibbon et al.):

$$Ax^2 + Bxy + Cy^2 + Dx + Ey + F = 0$$
  • Duration is modelled with a univariate Gaussian; start/end points plus ellipse parameters with a multivariate Gaussian; principal axes plus 3-D plane offset with a separate multivariate Gaussian.
  • Generation: sample the parameters, take the shortest elliptical arc between the projected endpoints, map it back to 3-D, retime it with the minimum-jerk profile $s(\tau)=10\tau^3-15\tau^4+6\tau^5$, then apply a linear endpoint correction.

Details: supp.pdf, Sec. III-B

Dynamic Movement Primitive (DMP)

$$\tau\dot{v} = K(g-y) - Dv + (g-y_0)f(s), \qquad \tau\dot{y} = v, \qquad s(t) = \exp(-\alpha_s t/\tau)$$
$$f(s) = \frac{\sum_{i=1}^{N} w_i\psi_i(s)}{\sum_{i=1}^{N}\psi_i(s)}\,s, \quad \psi_i(s) = e^{-h_i(s-c_i)^2}, \quad c_i = e^{-\alpha_s\frac{i-1}{N-1}}, \quad h_i = N/c_i^2$$

A DMP is fitted to every demonstration (per dimension, by locally weighted regression). A Gaussian is then fitted over the weight vectors, $\mathbf{w}\sim\mathcal{N}(\boldsymbol{\mu}_w, \boldsymbol{\Sigma}_w)$. At generation time, $y_0$, $g$, $\tau$ and $\mathbf{w}$ are sampled and the system is Euler-integrated.

  • $N = 20$ basis functions
  • $K = 100$
  • $D = 20$
  • $\alpha_s = 4.6$

Details: supp.pdf, Sec. III-C

Data-driven models

Trajectories are downsampled to 20 Hz. The input is a sliding window of 5 frames ($T-4$ to $T$) of the standardised 3-D positions of the giver's and taker's right hands, $\mathbf{s}_T \in \mathbb{R}^{5\times 6}$. The target is the standardised 3-D displacement of the giver's hand from $T$ to $T+1$.

MLP

Same architecture and training as the highway MLP (2 × 64, ReLU, MSE). The input is flattened to $\mathbb{R}^{30}$ and the output is in $\mathbb{R}^{3}$.

Details: supp.pdf, Sec. III-D

MDN

Same architecture, loss and training as the highway MDN, but with $K = 5$ mixture components (down from 15) to match the lower behavioural complexity of the task.

Details: supp.pdf, Sec. III-E

Attention-based model

$\mathbf{X}\in\mathbb{R}^{5\times 6}$→ linear → $\mathbf{Z}\in\mathbb{R}^{5\times128}$→ 4-head self-attn + residual + LayerNorm→ mean pool → $\mathbb{R}^{128}$→ Linear(128) · ReLU · Linear(3)
$$\mathbf{Z}' = \operatorname{LayerNorm}\big(\mathbf{Z} + \operatorname{MHA}(\mathbf{Z},\mathbf{Z},\mathbf{Z})\big), \qquad \mathbf{z}_{\text{pool}} = \tfrac{1}{T}\textstyle\sum_{t}\mathbf{Z}'_t$$

Details: supp.pdf, Sec. III-F

Validation against human data

$D_{\mathrm{KL}}(P_{\text{human}}\,\|\,Q_{\text{model}})$ of giver trajectory features (lower is better; best in bold)
ModelEnd-to-end len.Path len.Avg. speedAvg. accel.SmoothnessCurvature
Min. jerk0.0540.0640.0840.69611.7431.602
Elliptical0.0600.0600.0690.3992.2981.109
DMP0.0670.0570.0680.2502.2990.288
MLP0.1180.3361.0590.9512.3974.083
MDN0.0370.0934.0880.6447.4490.270
Attention0.1400.2111.8241.6191.8713.770

Feature distributions and summary statistics: supp.pdf, Sec. III-G

Handover scenario

Simulated trajectories from our calibrated human model in the handover setting.

Full supplementary PDF

Also covers dataset and environment details, trajectory generation, trajectory representations, evaluation metrics and RL training details.

Your browser can't display the PDF inline.
Use the link below to open it directly.
Open supplementary PDF supp.pdf