Algorithm

ARM allows to fuse a set of models M according to an estimation of their accuracy. The fused model z obtained with ARM is a linear combination of the models $m \in M$ . Given a test sample $X_j$ , the prediction $\hat{z}_j$ issued by the fused model is the weighted average of model predictions

$\hat{z}_j=\sum_{m=1}^{o}W_m\hat{Y}_{mj}$

Thus, the fusion process consists of learning the weight $W_m$ for each model. Let $r=|D|$ be the size of the fusion training set, and $o=|$M$|$ be the number of models in the ensemble. Here, we assume that the errors for each model are normally distributed. We then use the variance in these errors to identify the weights by executing the following steps:

Split $D$ randomly into two equally sized subsets $D^{(1)} \text{ and } D^{(2)}$
For each model m, evaluate $\sigma_m^2$ which is the maximum likelihood estimate of the variance of the errors $\overline{e}_{m} \text{ on } D^{(1)}$
$\overline{e}_{m}=\{\hat{Y}_{mj}-Y_j|\overline{X_j},Y_j \in D_{f}^{(1)}\}}$ Compute the sum of squared errors on $D^{(2)}$
$\beta_m= \sum_{j=\frac{r}{2}+1}^r(\hat{Y_{mj}}-Y_j)^2$
Estimate the weights using:
$W_m=\frac{(\sigma_m)^{-r/2} exp(-\sigma_m^{-2}{\beta_m/2})}{\sum_{j=1}^{o}(\sigma_j)^{-r/2}exp(-\sigma_j^{-2}{\beta_j/2})}$
Repeat steps 1-3 for a fixed number of times. Average the weights from each iteration to get the final weights for the models.

Transformation for large r: for large values of r, the calculation of the weights encounters an underflow error. To avoid this problem we equivalently compute the weights using:

$A_m=-\frac{r}{2}log(\sigma_m) + \frac{{-\sigma_m}^{-2}\beta_m}{2}$
and
$W_m=exp(A_m - log(\sum_{q=1}^{o}A_q))$

Authors and Contributors

This project is developed is by Ignacio Arnaldo (@ignacioarnaldo) of the Any-Scale Learning For All (ALFA) group at MIT. Contact us by email at iarnaldo@mit.edu