Chapter 12 Ensemble models

Let us be honest. When facing a prediction task, it is not obvious to determine the best choice between ML tools: penalized regressions, tree methods, neural networks, SVMs, etc. A natural and tempting alternative is to combine several algorithms (or the predictions that result from them) to try to extract value out of each engine (or learner). This intention is not new and contributions towards this goal go back at least to Bates and Granger (1969) (for the prupose of passenger flow forecasting).

Below, we outline a few books on the topic of ensembles. The latter have many names and synonyms, such as forecast aggregation, model averaging, mixture of experts or prediction combination. The first four references below are monographs while the last two are compilations of contributions:

  • Zhou (2012): a very didactic book that covers the main ideas of ensembles;
  • Schapire and Freund (2012): the main reference for boosting (and hence, ensembling) with many theoretical results and thus strong mathematical groundings;
  • Seni and Elder (2010): an introduction dedicated to tree methods mainly;
  • Claeskens and Hjort (2008): an overview of model selection techniques with a few chapter focused on model averaging;
  • Zhang and Ma (2012): a collection of thematic chapters on ensemble learning;
  • Okun, Valentini, and Re (2011): examples of applications of ensembles.

In this chapter, we cover the basic ideas and concepts behind the notion of ensembles. We refer to the above books for deeper treatments on the topic. We underline that several ensemble methods have already been mentioned and covered earlier, notably in Chapter 7. Indeed, random forests and boosted trees are examples of ensembles. Hence, other early articles on the combination of learners are Schapire (1990), Jacobs et al. (1991) (for neural networks particularly), and Freund and Schapire (1997).

12.1 Linear ensembles

12.1.1 Principles

In this chapter we adopt the following notations. We work with \(M\) models where \(\tilde{y}_{i,m}\) is the prediction of model \(m\) for instance \(i\) and errors \(\epsilon_{i,m}=y_i-\tilde{y}_{i,m}\) are stacked into a \((I\times M)\) matrix \(\textbf{E}\). A linear combination of models has sample errors equal to \(\textbf{Ew}\), where \(\textbf{w}=w_m\) are the weights assigned to each model and we assume \(\textbf{w}'\textbf{1}_M=1\). Minimizing the total (squared) error is thus a simple quadratic program with unique constraint. The Lagrange function is \(L(\textbf{w})=\textbf{w}'\textbf{E}'\textbf{E}\textbf{w}-\lambda (\textbf{w}'\textbf{1}_M-1)\) and hence \[\frac{\partial}{\partial \textbf{w}}L(\textbf{w})=\textbf{E}'\textbf{E}\textbf{w}-\lambda \textbf{1}_M=0 \quad \Leftrightarrow \quad \textbf{w}=\lambda(\textbf{E}'\textbf{E})^{-1}\textbf{1}_M,\]

and the constraint imposes \(\textbf{w}^*=\frac{(\textbf{E}'\textbf{E})^{-1}\textbf{1}_M}{(\textbf{1}_M'\textbf{E}'\textbf{E})^{-1}\textbf{1}_M}\). This form is similar to that of minimum variance portfolios. If errors are unbiased (\(\textbf{1}_I'\textbf{E}=\textbf{0}_M'\)), then \(\textbf{E}'\textbf{E}\) is the covariance matrix of errors.

This expression shows an important feature of optimized linear ensembles: they can only add value if the models tell different stories. If two models are redundant, \(\textbf{E}'\textbf{E}\) will be close to singular and \(\textbf{w}^*\) will arbitrage one against the other in a spurious fashion. This is the exact same problem as when mean-variance portfolios are constituted with highly correlated assets: in this case, diversification fails because when things go wrong, all assets go down. Another problem arises when the number of observations is too small compared to the number of assets so that the covariance matrix of returns is singular. This is not an issue for ensembles because the number of observations will usually be much larger than the number of models (\(I>>M\)).

In the limit when correlations increase to one, the above formulation becomes highly unstable and ensembles cannot be trusted. One heuristic way to see this is when \(M=2\) and \[\textbf{E}'\textbf{E}=\left[ \begin{array}{cc} \sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2 \\ \end{array} \right] \quad \Leftrightarrow \quad (\textbf{E}'\textbf{E})^{-1}=\frac{1}{1-\rho^2}\left[ \begin{array}{cc} \sigma_1^{-2} & -\rho(\sigma_1\sigma_2)^{-1} \\ -\rho(\sigma_1\sigma_2)^{-1} & \sigma_2^{-2} \\ \end{array} \right]\]

so that when \(\rho \rightarrow 1\), the model with the smallest errors (minimum \(\sigma_i^2\)) will see its weight increasing towards infinity while the other model will have a similarly large negative weight: the model arbitrages between two highly correlated variable. This seems like a very bad idea.

There is another illustration of the issues caused by correlations. Let’s assume we face \(M\) correlated errors \(\epsilon_m\) with pairwise correlation \(\rho\), zero mean and variance \(\sigma^2\). The variance of errors is \[\begin{align*} \mathbb{E}\left[\frac{1}{M}\sum_{m=1}^M \epsilon_m^2 \right]&=\frac{1}{M^2}\left[\sum_{m=1}^M\epsilon_m^2+\sum_{m\neq n}\epsilon_n\epsilon_m\right] \\ &=\frac{\sigma^2}{M}+\frac{1}{M^2}\sum_{n\neq m} \rho \sigma^2 \\ & =\rho \sigma^2 +\frac{\sigma^2(1-\rho)}{M} \end{align*}\] where while the second term converges to zero as \(M\) increases, the second term remains and is linearly increasing with \(\rho\). In passing, because variances are always positive, this result implies that the common pairwise correlation between \(M\) variables is bounded by below by \(-(M-1)^{-1}\). This result is interesting but rarely found in textbooks.

One improvement proposed to circumvent the trouble caused by correlations, advocated in a seminal publication (Breiman (1996)), is to enforce positivity constraints on the weights and solve

\[\underset{\textbf{w}}{\text{argmin}} \ \textbf{w}'\textbf{E}'\textbf{E}\textbf{w} , \quad \text{s.t.} \quad \left\{ \begin{array}{l} \textbf{w}'\textbf{1}_M=1 \\ w_m \ge 0 \quad \forall m \end{array}\right. .\]

Mechanically, if several models are highly correlated, the constraint will impose that only one of them will have a nonzero weight. If there are many models, then just a few of them will be selected by the minimization program. In the context of portfolio optimization, Jagannathan and Ma (2003) have shown the benefits of constraint in the construction mean-variance allocations In our setting, the constraint will similarly help discriminate wisely among the ‘best’ models.

In the literature, forecast combination and model averaging (which are synonyms of ensembles) have been tested on stock markets as early as in Von Holstein (1972). Surprisingly, the articles were not published in Finance journals but rather in fields such as Management (Virtanen and Yli-Olli (1987), Wang et al. (2012)), Economics and Econometrics (Donaldson and Kamstra (1996), Clark and McCracken (2009)), Operations Reasearch (Huang, Nakamori, and Wang (2005), Leung, Daouk, and Chen (2001), and Bonaccolto and Paterlini (2019)), and Computer Science (Harrald and Kamstra (1997), Hassan, Nath, and Kirley (2007)).

In the general forecasting literature, many alternative (refined) methods for combining forecasts have been studied. Trimmed opinion pools (Grushka-Cockayne, Jose, and Lichtendahl Jr (2016)) compute averages over the predictions that are not too extreme. We refer to Gaba, Tsetlin, and Winkler (2017) for a more exhaustive list of combinations as well as for an empirical study of their respective efficiency. Overall, findings are mixed and the heuristic simple average is, as usual, hard to beat (see, e.g., Genre et al. (2013)).

12.1.2 Example

In order to build an ensemble, we must gather the predictions and the corresponding errors into the \(\textbf{E}\) matrix. We will work with 5 models that were trained in the previous chapters: penalized regression, simple tree, random forest, xgboost and feedforward neural network. The training errors have zero means, hence \(\textbf{E}'\textbf{E}\) is the covariance matrix of errors between models.

##           Pen_reg      Tree        RF       XGB        NN
## Pen_reg 1.0000000 0.9984394 0.9968224 0.9475378 0.9963000
## Tree    0.9984394 1.0000000 0.9974647 0.9461082 0.9970501
## RF      0.9968224 0.9974647 1.0000000 0.9446178 0.9973220
## XGB     0.9475378 0.9461082 0.9446178 1.0000000 0.9442154
## NN      0.9963000 0.9970501 0.9973220 0.9442154 1.0000000

As is shown by the correlation matrix, the models fail to generate heterogeneity in their predictions. The minimum correlation (though above 95%!) is obtained by the boosted tree models. Below, we compare the training accuracy of models by computing the average absolute value of errors.

##    Pen_reg       Tree         RF        XGB         NN 
## 0.08345916 0.08362133 0.08327121 0.08916813 0.08369367

The best performing ML engine is the random forest. The boosted tree model is the worst, by far. Below, we compute the optimal (non constrained) weights for the combination of models.

##                 [,1]
## Pen_reg -0.620973725
## Tree    -0.061358846
## RF       1.269625185
## XGB     -0.003952533
## NN       0.416659919

Because of the high correlations, the optimal weights are not balanced and diversified: they load heavily on the random forest learner (best in sample model) and ‘short’ a few models in order to compensate. As one could expect, the model with the largest negative weights (Pen_reg) has a very high correlation with the random forest algorithm (0.997).

Note that the weights are of course computed with training errors. The optimal combination is then tested on the testing sample. Below, we compute out-of-sample (testing) errors and their average absolute value.

##    Pen_reg       Tree         RF        XGB         NN 
## 0.06618181 0.06653527 0.06710349 0.07149006 0.06785248

The boosted tree model is still the worst performing algorithm while the simple models (regression and simple tree) are the ones that fare the best. The most naive combination is the simple average of model and predictions.

## [1] 0.06701863

Because the errors are very correlated, the equally-weighted combination of forecasts yields an average error which lies ‘in the middle’ of individual errors. The diversification benefits are too small. Let us now test the ‘optimal’ combination \(\textbf{w}^*=\frac{(\textbf{E}'\textbf{E})^{-1}\textbf{1}_M}{(\textbf{1}_M'\textbf{E}'\textbf{E})^{-1}\textbf{1}_M}\).

## [1] 0.06861566

Again, the result is disappointing because of the lack of diversification across models. The correlations between errors are high not only on the training sample, but also on the testing sample, as shown below.

##           Pen_reg      Tree        RF       XGB        NN
## Pen_reg 1.0000000 0.9987069 0.9968882 0.9706767 0.9957290
## Tree    0.9987069 1.0000000 0.9978366 0.9753921 0.9970938
## RF      0.9968882 0.9978366 1.0000000 0.9776554 0.9976921
## XGB     0.9706767 0.9753921 0.9776554 1.0000000 0.9789965
## NN      0.9957290 0.9970938 0.9976921 0.9789965 1.0000000

The leverage from the optimal solution only exacerbates the problem and underperforms the heuristic uniform combination. We end this section with the constrained formulation of Breiman (1996) using the quadprog package. If we write \(\mathbf{\Sigma}\) for the covariance matrix of errors, we seek \[\mathbf{w}^*=\underset{\mathbf{w}}{\text{argmin}} \ \mathbf{w}'\mathbf{\Sigma}\mathbf{w}, \quad \mathbf{1}'\mathbf{w}=1, \quad w_i\ge 0,\] The constraints will be handled as:

\[\mathbf{A} \mathbf{w}= \begin{bmatrix} 1 & 1 & 1 \\ 1 & 0 & 0\\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \mathbf{w} \hspace{9mm} \text{ compared to} \hspace{9mm} \mathbf{b}=\begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \end{bmatrix}, \]

where the first line will be an equality (weights sum to one) and the last three will be inequalities (weights are all positive).

## [1] 0.000 0.000 0.813 0.000 0.187

Compared to the unconstrained solution, the weights are sparse and concentrated in one or two models, usually those with small training sample errors.

12.2 Stacked ensembles

12.2.1 Two stage training

Stacked ensembles are a natural generalization of linear ensembles. The idea of generalizing linear ensembles goes back at least to Wolpert (1992b). In the general case, the training is performed in two stages. The first stage is the simple one, whereby the \(M\) models are trained independently, yielding the predictions \(\tilde{y}_{i,m}\) for instance \(i\) and model \(m\). The second step is to consider the output of the trained models as input for a new level of machine learning optimization. The second level predictions are \(\breve{y}_i=h(\tilde{y}_{i,1},\dots,\tilde{y}_{i,M})\), where \(h\) is a new learner (see Figure 12.1). Linear ensembles are of course stacked ensembles in which the second layer is a linear regression.

The same techniques are then applied to minimize the error between the true values \(y_i\) and the predicted ones \(\breve{y}_i\).

Scheme of Stacked Ensembles.

FIGURE 12.1: Scheme of Stacked Ensembles.

12.2.2 Code and results

Below, we create a low-dimensional neural network which takes in the individual predictions of each model and compiles them into a synthetic forecast.

The configuration is very simple. We do not include any optional arguments and hence the model is likely to overfit. As we seek to predict returns, the loss function is the standard \(L^2\) norm.

## Model: "sequential_12"
## __________________________________________________________________________________________
## Layer (type)                            Output Shape                        Param #       
## ==========================================================================================
## dense_35 (Dense)                        (None, 8)                           48            
## __________________________________________________________________________________________
## dense_36 (Dense)                        (None, 4)                           36            
## __________________________________________________________________________________________
## dense_37 (Dense)                        (None, 1)                           5             
## ==========================================================================================
## Total params: 89
## Trainable params: 89
## Non-trainable params: 0
## __________________________________________________________________________________________

Training metrics for the ensemble model.

FIGURE 12.2: Training metrics for the ensemble model.

The performance of the ensemble is again disappointing: the learning curve is flat, hence the rounds of backpropagation are useless. The training adds little value which means that the new overarching layer of ML does not enhance the original predictions. Again, this is because all ML engines seem to be capturing the same patterns and both their linear and non-linear combinations fail to improve their performance.

12.3 Extensions

12.3.1 Exogenous variables

In a financial context, macro-economic indicators could add value to the process. It is possible that some models perform better under certain conditions and exogenous predictors can help introduce a flavor of economic-driven conditionality in the predictions.

Adding macro variables to the set of predictors (here, predictions) \(\tilde{y}_{i,m}\) could seem like one way to achieve this. However, this would amount to mix predicted values with (possibly scaled) economic indicators and that would not make much sense.

One alternative outside the perimeter of ensembles is to train simple trees on a set of macroeconomic indicators. If the labels are the (possibly absolute) errors stemming from the original predictions, then the trees will create clusters of homogeneous error values. This will hint towards which conditions lead to the best and worst forecasts. We test this idea below, using aggregate data from the Federal Reserve of Saint Louis. A simple downloading function is available in the quantmod package. We download and format the data in the next chunk. CPIAUCSL is a code for consumer price index and T10Y2YM is a code for the term spread (10Y minus 2Y).

## [1] "CPIAUCSL"
## [1] "T10Y2YM"
##         date  err_NN_test      Index CPIAUCSL   inflation termspread
## 1 2014-01-31 -0.146864512 2014-01-01  235.288 0.002424175       2.47
## 2 2014-02-28  0.079719784 2014-02-01  235.547 0.001100779       2.38
## 3 2014-03-31 -0.002269834 2014-03-01  236.028 0.002042055       2.32
## 4 2014-04-30 -0.066518659 2014-04-01  236.468 0.001864186       2.29
## 5 2014-05-31 -0.080330425 2014-05-01  236.918 0.001903006       2.17
## 6 2014-06-30  0.048578314 2014-06-01  237.231 0.001321132       2.15

We can now build a tree that tries to explain the accuracy of models as a function of macro variables.

Conditional performance of a ML engine.

FIGURE 12.3: Conditional performance of a ML engine.

The tree creates clusters which have homogeneous values of absolute errors. One big cluster gathers 92% of predictions (the left one) and is the one with the smallest average. It corresponds to the periods when the term spread is above 0.29 (in percentage points). The other two groups (when the term spread is below 0.29%) are determined according to the level of inflation. If the latter is positive, then the average absolute error is 7%, if not, it is 12%. This last number, the highest of the three clusters, indicates that when the term spread is low and the inflation negative, the model’s predictions are not trustworthy because their errors have a magnitude twice as large as in other periods. Under these circumstances (which seem to be linked to a dire economic environment), it may be wiser not to use ML-based forecasts.

12.3.2 Shrinking inter-model correlations

As shown earlier in this chapter, one major problem with ensembles arises when the first layer of predictions is highly correlated. In this case, ensemble are pretty much useless. Their are several tricks that can help reduce this correlation but the simplest and best is probably to alter training samples. If algorithms do not see the same data, they will probably infer different patterns.

There are several ways to split the training data so as to build different subsets of training samples. The first dichotomy is between random versus deterministic splits. Random splits are easy and require only the target sample size to be fixed. Note that the training samples can be overlapping as long as the overlap is not too large. Hence if the original training sample has \(I\) instance and the ensemble requires \(M\) models, then a subsample size of \(\lfloor I/M \rfloor\) may be too conservative especially if the training sample is not very large. In this case \(\lfloor I/\sqrt{M} \rfloor\) may be a better alternative. Random forests are one example of ensembles built in random training samples.

One advantage of deterministic splits is that they are easy to reproduce and their outcome does not depend on the random seed. By the nature of factor-based training samples, the second splitting dichotomy is between time and assets. A split within assets is straightforward: each model is trained on a different set of stocks. Note that the choices of sets can be random, or dictacted by some factor-based criterion: size, momentum, book-to-market ratio, etc.

A split in dates requires other decisions: is the data split in large blocks (like years) and each model gets a block, which may stand for one particular kind of market condition? Or are the training dates divided more regularly? For instance, if there are 12 models in the ensemble, each model can be trained on data from a given month (e.g., January for the first models, February for the second, etc.).

Below, we train four models on four different years to see if this help reduce the inter-model correlations. This process is a bit lengthy because the samples and models need to be all redefined. We start by creating the four training samples. The third model works on the small subset of features, hence the sample is smaller.

Then, we proceed to the training of the models. The syntaxes are those used in the previous chapters, nothing new here. We start with a penalized regression. In all predictions below, the original testing sample is used for all models.

We continue with a random forest.

The third model is a boosted tree.

Finally, the last model is a simple neural network.

Endowed with the errors of the four models, we can compute their correlation matrix.

##              err_ens_2007 err_ens_2009 err_ens_2011 err_ens_2013
## err_ens_2007    1.0000000    0.9610165    0.6460091    0.9987795
## err_ens_2009    0.9610165    1.0000000    0.6340258    0.9657441
## err_ens_2011    0.6460091    0.6340258    1.0000000    0.6460924
## err_ens_2013    0.9987795    0.9657441    0.6460924    1.0000000

The results are overall disappointing. Only one model manages to extract patterns that are somewhat different from the other ones, resulting in a 65% correlation across the board. Neural networks (on 2013 data) and penalized regressions (2007) remain highly correlated. One possible explanation could be that the models capture mainly noise and little signal. Working with long term labels like annual returns could help improve diversification across models.

12.4 Exercise

Build an integrated ensemble of top of 3 neural networks trained entirely with Keras. Each network obtains one third of predictors as input. The three networks yield a classification (yes/no or buy/sell). The overarching network aggregates the three outputs into a final decision. Evaluate its performance on the testing sample. Use the functional API.

References

Bates, John M, and Clive WJ Granger. 1969. “The Combination of Forecasts.” Journal of the Operational Research Society 20 (4): 451–68.

Bonaccolto, Giovanni, and Sandra Paterlini. 2019. “Developing New Portfolio Strategies by Aggregation.” Annals of Operations Research, 1–39.

Breiman, Leo. 1996. “Stacked Regressions.” Machine Learning 24 (1): 49–64.

Claeskens, Gerda, and Nils Lid Hjort. 2008. Model Selection and Model Averaging. Cambridge University Press.

Clark, Todd E, and Michael W McCracken. 2009. “Improving Forecast Accuracy by Combining Recursive and Rolling Forecasts.” International Economic Review 50 (2): 363–95.

Donaldson, R Glen, and Mark Kamstra. 1996. “Forecast Combining with Neural Networks.” Journal of Forecasting 15 (1): 49–61.

Freund, Yoav, and Robert E Schapire. 1997. “A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting.” Journal of Computer and System Sciences 55 (1): 119–39.

Gaba, Anil, Ilia Tsetlin, and Robert L Winkler. 2017. “Combining Interval Forecasts.” Decision Analysis 14 (1): 1–20.

Genre, Véronique, Geoff Kenny, Aidan Meyler, and Allan Timmermann. 2013. “Combining Expert Forecasts: Can Anything Beat the Simple Average?” International Journal of Forecasting 29 (1): 108–21.

Grushka-Cockayne, Yael, Victor Richmond R Jose, and Kenneth C Lichtendahl Jr. 2016. “Ensembles of Overfit and Overconfident Forecasts.” Management Science 63 (4): 1110–30.

Harrald, Paul G, and Mark Kamstra. 1997. “Evolving Artificial Neural Networks to Combine Financial Forecasts.” IEEE Transactions on Evolutionary Computation 1 (1): 40–52.

Hassan, Md Rafiul, Baikunth Nath, and Michael Kirley. 2007. “A Fusion Model of Hmm, Ann and Ga for Stock Market Forecasting.” Expert Systems with Applications 33 (1): 171–80.

Huang, Wei, Yoshiteru Nakamori, and Shou-Yang Wang. 2005. “Forecasting Stock Market Movement Direction with Support Vector Machine.” Computers & Operations Research 32 (10): 2513–22.

Jacobs, Robert A, Michael I Jordan, Steven J Nowlan, Geoffrey E Hinton, and others. 1991. “Adaptive Mixtures of Local Experts.” Neural Computation 3 (1): 79–87.

Jagannathan, Ravi, and Tongshu Ma. 2003. “Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps.” Journal of Finance 58 (4): 1651–83.

Leung, Mark T, Hazem Daouk, and An-Sing Chen. 2001. “Using Investment Portfolio Return to Combine Forecasts: A Multiobjective Approach.” European Journal of Operational Research 134 (1): 84–102.

Okun, Oleg, Giorgio Valentini, and Matteo Re. 2011. Ensembles in Machine Learning Applications. Vol. 373. Springer Science & Business Media.

Schapire, Robert E. 1990. “The Strength of Weak Learnability.” Machine Learning 5 (2): 197–227.

Schapire, Robert E, and Yoav Freund. 2012. Boosting: Foundations and Algorithms. MIT press.

Seni, Giovanni, and John F Elder. 2010. “Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions.” Synthesis Lectures on Data Mining and Knowledge Discovery 2 (1): 1–126.

Virtanen, Ilkka, and Paavo Yli-Olli. 1987. “Forecasting Stock Market Prices in a Thin Security Market.” Omega 15 (2): 145–55.

Von Holstein, Carl-Axel S Staël. 1972. “Probabilistic Forecasting: An Experiment Related to the Stock Market.” Organizational Behavior and Human Performance 8 (1): 139–58.

Wang, Ju-Jie, Jian-Zhou Wang, Zhe-George Zhang, and Shu-Po Guo. 2012. “Stock Index Forecasting Based on a Hybrid Model.” Omega 40 (6): 758–66.

Wolpert, David H. 1992b. “Stacked Generalization.” Neural Networks 5 (2): 241–59.

Zhang, Cha, and Yunqian Ma. 2012. Ensemble Machine Learning: Methods and Applications. Springer.

Zhou, Zhi-Hua. 2012. Ensemble Methods: Foundations and Algorithms. Chapman; Hall/CRC.