# Chapter 12 Ensemble models

Let us be honest. When facing a prediction task, it is not obvious to determine the best choice between ML tools: penalized regressions, tree methods, neural networks, SVMs, etc. A natural and tempting alternative is to **combine** several algorithms (or the predictions that result from them) to try to extract value out of each engine (or learner). This intention is not new and contributions towards this goal go back at least to Bates and Granger (1969) (for the purpose of passenger flow forecasting).

Below, we outline a few books on the topic of ensembles. The latter have many names and synonyms, such as **forecast aggregation**, **model averaging**, **mixture of experts** or **prediction combination**. The first four references below are monographs, while the last two are compilations of contributions:

- Zhou (2012): a very didactic book that covers the main ideas of ensembles;

- Schapire and Freund (2012): the main reference for boosting (and hence, ensembling) with many theoretical results and thus strong mathematical groundings;

- Seni and Elder (2010): an introduction dedicated to tree methods mainly;

- Claeskens and Hjort (2008): an overview of model selection techniques with a few chapters focused on model averaging;

- Zhang and Ma (2012): a collection of thematic chapters on ensemble learning;

- Okun, Valentini, and Re (2011): examples of applications of ensembles.

In this chapter, we cover the basic ideas and concepts behind the notion of ensembles. We refer to the above books for deeper treatments on the topic. We underline that several ensemble methods have already been mentioned and covered earlier, notably in Chapter 7. Indeed, random forests and boosted trees are examples of ensembles. Hence, other early articles on the combination of learners are Schapire (1990), Jacobs et al. (1991) (for neural networks particularly), and Freund and Schapire (1997). Finally, for a theoretical view on ensembles with a Bayesian perspective, we refer to Razin and Levy (2020).

## 12.1 Linear ensembles

### 12.1.1 Principles

In this chapter we adopt the following notations. We work with \(M\) models where \(\tilde{y}_{i,m}\) is the prediction of model \(m\) for instance \(i\) and errors \(\epsilon_{i,m}=y_i-\tilde{y}_{i,m}\) are stacked into a \((I\times M)\) matrix \(\textbf{E}\). A linear combination of models has sample errors equal to \(\textbf{Ew}\), where \(\textbf{w}=w_m\) are the weights assigned to each model and we assume \(\textbf{w}'\textbf{1}_M=1\). Minimizing the total (squared) error is thus a simple quadratic program with unique constraint. The Lagrange function is \(L(\textbf{w})=\textbf{w}'\textbf{E}'\textbf{E}\textbf{w}-\lambda (\textbf{w}'\textbf{1}_M-1)\) and hence \[\frac{\partial}{\partial \textbf{w}}L(\textbf{w})=\textbf{E}'\textbf{E}\textbf{w}-\lambda \textbf{1}_M=0 \quad \Leftrightarrow \quad \textbf{w}=\lambda(\textbf{E}'\textbf{E})^{-1}\textbf{1}_M,\]

and the constraint imposes \(\textbf{w}^*=\frac{(\textbf{E}'\textbf{E})^{-1}\textbf{1}_M}{(\textbf{1}_M'\textbf{E}'\textbf{E})^{-1}\textbf{1}_M}\). This form is similar to that of minimum variance portfolios. If errors are unbiased (\(\textbf{1}_I'\textbf{E}=\textbf{0}_M'\)), then \(\textbf{E}'\textbf{E}\) is the covariance matrix of errors.

This expression shows an important feature of optimized linear ensembles: they can only add value if the models tell different stories. If two models are redundant, \(\textbf{E}'\textbf{E}\) will be close to singular and \(\textbf{w}^*\) will arbitrage one against the other in a spurious fashion. This is the exact same problem as when mean-variance portfolios are constituted with highly correlated assets: in this case, diversification fails because when things go wrong, all assets go down. Another problem arises when the number of observations is too small compared to the number of assets so that the covariance matrix of returns is singular. This is not an issue for ensembles because the number of observations will usually be much larger than the number of models (\(I>>M\)).

In the limit when correlations increase to one, the above formulation becomes highly unstable and ensembles cannot be trusted. One heuristic way to see this is when \(M=2\) and \[\textbf{E}'\textbf{E}=\left[ \begin{array}{cc} \sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2 \\ \end{array} \right] \quad \Leftrightarrow \quad (\textbf{E}'\textbf{E})^{-1}=\frac{1}{1-\rho^2}\left[ \begin{array}{cc} \sigma_1^{-2} & -\rho(\sigma_1\sigma_2)^{-1} \\ -\rho(\sigma_1\sigma_2)^{-1} & \sigma_2^{-2} \\ \end{array} \right]\]

so that when \(\rho \rightarrow 1\), the model with the smallest errors (minimum \(\sigma_i^2\)) will see its weight increasing towards infinity while the other model will have a similarly large **negative weight**: the model arbitrages between two highly correlated variables. This seems like a very bad idea.

There is another illustration of the issues caused by correlations. Let’s assume we face \(M\) correlated errors \(\epsilon_m\) with pairwise correlation \(\rho\), zero mean and variance \(\sigma^2\). The variance of errors is
\[\begin{align*}
\mathbb{E}\left[\frac{1}{M}\sum_{m=1}^M \epsilon_m^2 \right]&=\frac{1}{M^2}\left[\sum_{m=1}^M\epsilon_m^2+\sum_{m\neq n}\epsilon_n\epsilon_m\right] \\
&=\frac{\sigma^2}{M}+\frac{1}{M^2}\sum_{n\neq m} \rho \sigma^2 \\
& =\rho \sigma^2 +\frac{\sigma^2(1-\rho)}{M}
\end{align*}\]
where while the second term converges to zero as \(M\) increases, the second term remains and is **linearly increasing** with \(\rho\). In passing, because variances are always positive, this result implies that the common pairwise correlation between \(M\) variables is bounded below by \(-(M-1)^{-1}\). This result is interesting but rarely found in textbooks.

One improvement proposed to circumvent the trouble caused by correlations, advocated in a seminal publication (Breiman (1996)), is to enforce positivity constraints on the weights and solve

\[\underset{\textbf{w}}{\text{argmin}} \ \textbf{w}'\textbf{E}'\textbf{E}\textbf{w} , \quad \text{s.t.} \quad \left\{ \begin{array}{l} \textbf{w}'\textbf{1}_M=1 \\ w_m \ge 0 \quad \forall m \end{array}\right. .\]

Mechanically, if several models are highly correlated, the constraint will impose that only one of them will have a nonzero weight. If there are many models, then just a few of them will be selected by the minimization program. In the context of portfolio optimization, Jagannathan and Ma (2003) have shown the benefits of constraint in the construction mean-variance allocations. In our setting, the constraint will similarly help discriminate wisely among the ‘best’ models.

In the literature, forecast combination and model averaging (which are synonyms of ensembles) have been tested on stock markets as early as in Von Holstein (1972). Surprisingly, the articles were not published in Finance journals but rather in fields such as Management (Virtanen and Yli-Olli (1987), Wang et al. (2012)), Economics and Econometrics (Donaldson and Kamstra (1996), Clark and McCracken (2009), Mascio, Fabozzi, and Zumwalt (2020)), Operations Reasearch (Huang, Nakamori, and Wang (2005), Leung, Daouk, and Chen (2001), and Bonaccolto and Paterlini (2019)), and Computer Science (Harrald and Kamstra (1997), Hassan, Nath, and Kirley (2007)).

In the general forecasting literature, many alternative (refined) methods for combining forecasts have been studied. Trimmed opinion pools (Grushka-Cockayne, Jose, and Lichtendahl Jr (2016)) compute averages over the predictions that are not too extreme. We refer to Gaba, Tsetlin, and Winkler (2017) for a more exhaustive list of combinations as well as for an empirical study of their respective efficiency. Overall, findings are mixed and the heuristic simple average is, as usual, hard to beat (see, e.g., Genre et al. (2013)).

### 12.1.2 Example

In order to build an ensemble, we must gather the predictions and the corresponding errors into the \(\textbf{E}\) matrix. We will work with 5 models that were trained in the previous chapters: penalized regression, simple tree, random forest, xgboost and feed-forward neural network. The training errors have zero means, hence \(\textbf{E}'\textbf{E}\) is the covariance matrix of errors between models.

```
err_pen_train <- predict(fit_pen_pred, x_penalized_train) - training_sample$R1M_Usd # Reg.
err_tree_train <- predict(fit_tree, training_sample) - training_sample$R1M_Usd # Tree
err_RF_train <- predict(fit_RF, training_sample) - training_sample$R1M_Usd # RF
err_XGB_train <- predict(fit_xgb, train_matrix_xgb) - training_sample$R1M_Usd # XGBoost
err_NN_train <- predict(model, NN_train_features) - training_sample$R1M_Usd # NN
E <- cbind(err_pen_train, err_tree_train, err_RF_train, err_XGB_train, err_NN_train) # E matrix
colnames(E) <- c("Pen_reg", "Tree", "RF", "XGB", "NN") # Col. names
cor(E) # Cor. mat.
```

```
## Pen_reg Tree RF XGB NN
## Pen_reg 1.0000000 0.9984394 0.9968224 0.9310186 0.9969135
## Tree 0.9984394 1.0000000 0.9974647 0.9296081 0.9974810
## RF 0.9968224 0.9974647 1.0000000 0.9281725 0.9976131
## XGB 0.9310186 0.9296081 0.9281725 1.0000000 0.9282508
## NN 0.9969135 0.9974810 0.9976131 0.9282508 1.0000000
```

As is shown by the correlation matrix, the models fail to generate heterogeneity in their predictions. The minimum correlation (though above 95%!) is obtained by the boosted tree models. Below, we compare the training accuracy of models by computing the average absolute value of errors.

```
## Pen_reg Tree RF XGB NN
## 0.08345916 0.08362133 0.08327121 0.08986993 0.08369588
```

The best performing ML engine is the random forest. The boosted tree model is the worst, by far. Below, we compute the optimal (non-constrained) weights for the combination of models.

```
w_ensemble <- solve(t(E) %*% E) %*% rep(1,5) # Optimal weights
w_ensemble <- w_ensemble / sum(w_ensemble)
w_ensemble
```

```
## [,1]
## Pen_reg -0.5817327676
## Tree -0.0543806114
## RF 1.3769780491
## XGB -0.0006271105
## NN 0.2597624405
```

Because of the high correlations, the optimal weights are not balanced and diversified: they load heavily on the random forest learner (best in sample model) and ‘short’ a few models in order to compensate. As one could expect, the model with the largest negative weights (Pen_reg) has a very high correlation with the random forest algorithm (0.997).

Note that the weights are of course computed with **training errors**. The optimal combination is then tested on the testing sample. Below, we compute out-of-sample (testing) errors and their average absolute value.

```
err_pen_test <- predict(fit_pen_pred, x_penalized_test) - testing_sample$R1M_Usd # Reg.
err_tree_test <- predict(fit_tree, testing_sample) - testing_sample$R1M_Usd # Tree
err_RF_test <- predict(fit_RF, testing_sample) - testing_sample$R1M_Usd # RF
err_XGB_test <- predict(fit_xgb, xgb_test) - testing_sample$R1M_Usd # XGBoost
err_NN_test <- predict(model, NN_test_features) - testing_sample$R1M_Usd # NN
E_test <- cbind(err_pen_test, err_tree_test, err_RF_test, err_XGB_test, err_NN_test) # E matrix
colnames(E_test) <- c("Pen_reg", "Tree", "RF", "XGB", "NN")
apply(abs(E_test), 2, mean) # Mean absolute error or columns of E
```

```
## Pen_reg Tree RF XGB NN
## 0.06618181 0.06653527 0.06710349 0.07170802 0.06716501
```

The boosted tree model is still the worst performing algorithm while the simple models (regression and simple tree) are the ones that fare the best. The most naive combination is the simple average of model and predictions.

`## [1] 0.06693156`

Because the errors are very correlated, the equally weighted combination of forecasts yields an average error which lies ‘in the middle’ of individual errors. The diversification benefits are too small. Let us now test the ‘optimal’ combination \(\textbf{w}^*=\frac{(\textbf{E}'\textbf{E})^{-1}\textbf{1}_M}{(\textbf{1}_M'\textbf{E}'\textbf{E})^{-1}\textbf{1}_M}\).

`## [1] 0.06835538`

Again, the result is disappointing because of the lack of diversification across models. The correlations between errors are high not only on the training sample, but also on the testing sample, as shown below.

```
## Pen_reg Tree RF XGB NN
## Pen_reg 1.0000000 0.9987069 0.9968882 0.9537914 0.9966959
## Tree 0.9987069 1.0000000 0.9978366 0.9583641 0.9976702
## RF 0.9968882 0.9978366 1.0000000 0.9606570 0.9979694
## XGB 0.9537914 0.9583641 0.9606570 1.0000000 0.9609153
## NN 0.9966959 0.9976702 0.9979694 0.9609153 1.0000000
```

The leverage from the optimal solution only exacerbates the problem and underperforms the heuristic uniform combination. We end this section with the constrained formulation of Breiman (1996) using the *quadprog* package. If we write \(\mathbf{\Sigma}\) for the covariance matrix of errors, we seek
\[\mathbf{w}^*=\underset{\mathbf{w}}{\text{argmin}} \ \mathbf{w}'\mathbf{\Sigma}\mathbf{w}, \quad \mathbf{1}'\mathbf{w}=1, \quad w_i\ge 0,\]
The constraints will be handled as:

\[\mathbf{A} \mathbf{w}= \begin{bmatrix} 1 & 1 & 1 \\ 1 & 0 & 0\\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \mathbf{w} \hspace{9mm} \text{ compared to} \hspace{9mm} \mathbf{b}=\begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \end{bmatrix}, \]

where the first line will be an equality (weights sum to one) and the last three will be inequalities (weights are all positive).

```
library(quadprog) # Package for quadratic programming
Sigma <- t(E) %*% E # Unscaled covariance matrix
nb_mods <- nrow(Sigma) # Number of models
w_const <- solve.QP(Dmat = Sigma, # D matrix = Sigma
dvec = rep(0, nb_mods), # Zero vector
Amat = rbind(rep(1, nb_mods), diag(nb_mods)) %>% t(), # A matrix for constraints
bvec = c(1,rep(0, nb_mods)), # b vector for constraints
meq = 1 # 1 line of equality constraints, others = inequalities
)
w_const$solution %>% round(3) # Solution
```

`## [1] 0 0 1 0 0`

Compared to the unconstrained solution, the weights are sparse and concentrated in one or two models, usually those with small training sample errors.

## 12.2 Stacked ensembles

### 12.2.1 Two-stage training

**Stacked ensembles** are a natural generalization of linear ensembles. The idea of generalizing linear ensembles goes back at least to Wolpert (1992b). In the general case, the training is performed in two stages. The first stage is the simple one, whereby the \(M\) models are trained independently, yielding the predictions \(\tilde{y}_{i,m}\) for instance \(i\) and model \(m\). The second step is to consider the output of the trained models as input for a new level of machine learning optimization. The second level predictions are \(\breve{y}_i=h(\tilde{y}_{i,1},\dots,\tilde{y}_{i,M})\), where \(h\) is a new learner (see Figure 12.1). Linear ensembles are of course stacked ensembles in which the second layer is a linear regression.

The same techniques are then applied to minimize the error between the true values \(y_i\) and the predicted ones \(\breve{y}_i\).

### 12.2.2 Code and results

Below, we create a low-dimensional neural network which takes in the individual predictions of each model and compiles them into a synthetic forecast.

```
model_stack <- keras_model_sequential()
model_stack %>% # This defines the structure of the network, i.e. how layers are organized
layer_dense(units = 8, activation = 'relu', input_shape = nb_mods) %>%
layer_dense(units = 4, activation = 'tanh') %>%
layer_dense(units = 1)
```

The configuration is very simple. We do not include any optional arguments and hence the model is likely to overfit. As we seek to predict returns, the loss function is the standard \(L^2\) norm.

```
model_stack %>% compile( # Model specification
loss = 'mean_squared_error', # Loss function
optimizer = optimizer_rmsprop(), # Optimisation method (weight updating)
metrics = c('mean_absolute_error') # Output metric
)
summary(model_stack) # Model architecture
```

```
## __________________________________________________________________________________________
## Layer (type) Output Shape Param #
## ==========================================================================================
## dense_11 (Dense) (None, 8) 48
## __________________________________________________________________________________________
## dense_12 (Dense) (None, 4) 36
## __________________________________________________________________________________________
## dense_13 (Dense) (None, 1) 5
## ==========================================================================================
## Total params: 89
## Trainable params: 89
## Non-trainable params: 0
## __________________________________________________________________________________________
```

```
y_tilde <- E + matrix(rep(training_sample$R1M_Usd, nb_mods), ncol = nb_mods) # Train predictions
y_test <- E_test + matrix(rep(testing_sample$R1M_Usd, nb_mods), ncol = nb_mods) # Testing
fit_NN_stack <- model_stack %>% fit(y_tilde, # Train features
training_sample$R1M_Usd, # Train labels
epochs = 12, batch_size = 512, # Train parameters
validation_data = list(y_test, # Test features
testing_sample$R1M_Usd) # Test labels
)
plot(fit_NN_stack) # Plot, evidently!
```

The performance of the ensemble is again disappointing: the learning curve is flat in Figure 12.2, hence the rounds of back-propagation are useless. The training adds little value which means that the new overarching layer of ML does not enhance the original predictions. Again, this is because all ML engines seem to be capturing the same patterns and both their linear and non-linear combinations fail to improve their performance.

## 12.3 Extensions

### 12.3.1 Exogenous variables

In a financial context, macro-economic indicators could add value to the process. It is possible that some models perform better under certain conditions and exogenous predictors can help introduce a flavor of **economic-driven conditionality** in the predictions.

Adding macro-variables to the set of predictors (here, predictions) \(\tilde{y}_{i,m}\) could seem like one way to achieve this. However, this would amount to mix predicted values with (possibly scaled) economic indicators and that would not make much sense.

One alternative outside the perimeter of ensembles is to train simple trees on a set of macro-economic indicators. If the labels are the (possibly absolute) errors stemming from the original predictions, then the trees will create clusters of homogeneous error values. This will hint towards which conditions lead to the best and worst forecasts.
We test this idea below, using aggregate data from the Federal Reserve of Saint Louis. A simple downloading function is available in the *quantmod* package. We download and format the data in the next chunk. CPIAUCSL is a code for consumer price index and T10Y2YM is a code for the term spread (10Y minus 2Y).

```
library(quantmod) # Package that extracts the data
library(lubridate) # Package for date management
getSymbols("CPIAUCSL", src = "FRED") # FRED is the Fed of St Louis
```

`## [1] "CPIAUCSL"`

`## [1] "T10Y2YM"`

```
cpi <- fortify(CPIAUCSL) %>%
mutate (inflation = CPIAUCSL / lag(CPIAUCSL) - 1) # Inflation via Consumer Price Index
ts <- fortify(T10Y2YM) # Term spread (10Y minus 2Y rates)
colnames(ts)[2] <- "termspread" # To make things clear
ens_data <- testing_sample %>% # Creating aggregate dataset
dplyr::select(date) %>%
cbind(err_NN_test) %>%
mutate(Index = make_date(year = lubridate::year(date), # Change date to first day of month
month = lubridate::month(date),
day = 1)) %>%
left_join(cpi) %>% # Add CPI to the dataset
left_join(ts) # Add termspread
head(ens_data) # Show first lines
```

```
## date err_NN_test Index CPIAUCSL inflation termspread
## 1 2014-01-31 -0.15700543 2014-01-01 235.288 0.002424175 2.47
## 2 2014-02-28 0.07107934 2014-02-01 235.547 0.001100779 2.38
## 3 2014-03-31 -0.01740334 2014-03-01 236.028 0.002042055 2.32
## 4 2014-04-30 -0.08128143 2014-04-01 236.468 0.001864186 2.29
## 5 2014-05-31 -0.09547639 2014-05-01 236.918 0.001903006 2.17
## 6 2014-06-30 0.03042763 2014-06-01 237.231 0.001321132 2.15
```

We can now build a tree that tries to explain the accuracy of models as a function of macro-variables.

```
library(rpart.plot) # Load package for tree plotting
fit_ens <- rpart(abs(err_NN_test) ~ inflation + termspread, # Tree model
data = ens_data,
cp = 0.001) # Complexity parameter (size of tree)
rpart.plot(fit_ens) # Plot tree
```

The tree creates clusters which have homogeneous values of absolute errors. One big cluster gathers 92% of predictions (the left one) and is the one with the smallest average. It corresponds to the periods when the term spread is above 0.29 (in percentage points). The other two groups (when the term spread is below 0.29%) are determined according to the level of inflation. If the latter is positive, then the average absolute error is 7%, if not, it is 12%. This last number, the highest of the three clusters, indicates that when the term spread is low and the inflation negative, the model’s predictions are not trustworthy because their errors have a magnitude twice as large as in other periods. Under these circumstances (which seem to be linked to a dire economic environment), it may be wiser not to use ML-based forecasts.

### 12.3.2 Shrinking inter-model correlations

As shown earlier in this chapter, one major problem with ensembles arises when the first layer of predictions is highly correlated. In this case, ensembles are pretty much useless. There are several tricks that can help reduce this correlation, but the simplest and best is probably to alter training samples. If algorithms do not see the same data, they will probably infer different patterns.

There are several ways to split the training data so as to build different subsets of training samples. The first dichotomy is between random versus deterministic splits. Random splits are easy and require only the target sample size to be fixed. Note that the training samples can be overlapping as long as the overlap is not too large. Hence if the original training sample has \(I\) instance and the ensemble requires \(M\) models, then a subsample size of \(\lfloor I/M \rfloor\) may be too conservative especially if the training sample is not very large. In this case \(\lfloor I/\sqrt{M} \rfloor\) may be a better alternative. Random forests are one example of ensembles built in random training samples.

One advantage of deterministic splits is that they are easy to reproduce and their outcome does not depend on the random seed. By the nature of factor-based training samples, the second splitting dichotomy is between time and assets. A split within assets is straightforward: each model is trained on a different set of stocks. Note that the choices of sets can be random, or dictacted by some factor-based criterion: size, momentum, book-to-market ratio, etc.

A split in dates requires other decisions: is the data split in large blocks (like years) and each model gets a block, which may stand for one particular kind of market condition? Or are the training dates divided more regularly? For instance, if there are 12 models in the ensemble, each model can be trained on data from a given month (e.g., January for the first models, February for the second, etc.).

Below, we train four models on four different years to see if this helps reduce the inter-model correlations. This process is a bit lengthy because the samples and models need to be all redefined. We start by creating the four training samples. The third model works on the small subset of features, hence the sample is smaller.

```
training_sample_2007 <- training_sample %>%
filter(date > "2006-12-31", date < "2008-01-01")
training_sample_2009 <- training_sample %>%
filter(date > "2008-12-31", date < "2010-01-01")
training_sample_2011 <- training_sample %>%
dplyr::select(c("date",features_short, "R1M_Usd")) %>%
filter(date > "2010-12-31", date < "2012-01-01")
training_sample_2013 <- training_sample %>%
filter(date > "2012-12-31", date < "2014-01-01")
```

Then, we proceed to the training of the models. The syntaxes are those used in the previous chapters, nothing new here. We start with a penalized regression. In all predictions below, the original testing sample is used *for all models*.

```
y_ens_2007 <- training_sample_2007$R1M_Usd # Dep. variable
x_ens_2007 <- training_sample_2007 %>% # Predictors
dplyr::select(features) %>% as.matrix()
fit_ens_2007 <- glmnet(x_ens_2007, y_ens_2007, alpha = 0.1, lambda = 0.1) # Model
err_ens_2007 <- predict(fit_ens_2007, x_penalized_test) - testing_sample$R1M_Usd # Prediction errors
```

We continue with a random forest.

```
fit_ens_2009 <- randomForest(formula, # Same formula as for simple trees!
data = training_sample_2009, # Data source: 2011 training sample
sampsize = 4000, # Size of (random) sample for each tree
replace = FALSE, # Is the sampling done with replacement?
nodesize = 100, # Minimum size of terminal cluster
ntree = 40, # Nb of random trees
mtry = 30 # Nb of predictive variables for each tree
)
err_ens_2009 <- predict(fit_ens_2009, testing_sample) - testing_sample$R1M_Usd # Prediction errors
```

The third model is a boosted tree.

```
train_features_2011 <- training_sample_2011 %>%
dplyr::select(features_short) %>% as.matrix() # Independent variable
train_label_2011 <- training_sample_2011 %>%
dplyr::select(R1M_Usd) %>% as.matrix() # Dependent variable
train_matrix_2011 <- xgb.DMatrix(data = train_features_2011,
label = train_label_2011) # XGB format!
fit_ens_2011 <- xgb.train(data = train_matrix_2011, # Data source
eta = 0.4, # Learning rate
objective = "reg:linear", # Objective function
max_depth = 4, # Maximum depth of trees
nrounds = 18 # Number of trees used
)
err_ens_2011 <- predict(fit_ens_2011, xgb_test) - testing_sample$R1M_Usd # Prediction errors
```

Finally, the last model is a simple neural network.

```
NN_features_2013 <- dplyr::select(training_sample_2013, features) %>%
as.matrix() # Matrix format is important
NN_labels_2013 <- training_sample_2013$R1M_Usd
model_ens_2013 <- keras_model_sequential()
model_ens_2013 %>% # This defines the structure of the network, i.e. how layers are organized
layer_dense(units = 16, activation = 'relu', input_shape = ncol(NN_features_2013)) %>%
layer_dense(units = 8, activation = 'tanh') %>%
layer_dense(units = 1)
model_ens_2013 %>% compile( # Model specification
loss = 'mean_squared_error', # Loss function
optimizer = optimizer_rmsprop(), # Optimisation method (weight updating)
metrics = c('mean_absolute_error') # Output metric
)
model_ens_2013 %>% fit(NN_features_2013, # Training features
NN_labels_2013, # Training labels
epochs = 9, batch_size = 128 # Training parameters
)
err_ens_2013 <- predict(model_ens_2013, NN_test_features) - testing_sample$R1M_Usd
```

Endowed with the errors of the four models, we can compute their correlation matrix.

```
## err_ens_2007 err_ens_2009 err_ens_2011 err_ens_2013
## err_ens_2007 1.0000000 0.9610165 0.6460091 0.9985485
## err_ens_2009 0.9610165 1.0000000 0.6340258 0.9638768
## err_ens_2011 0.6460091 0.6340258 1.0000000 0.6456426
## err_ens_2013 0.9985485 0.9638768 0.6456426 1.0000000
```

The results are overall disappointing. Only one model manages to extract patterns that are somewhat different from the other ones, resulting in a 65% correlation across the board. Neural networks (on 2013 data) and penalized regressions (2007) remain highly correlated. One possible explanation could be that the models capture mainly noise and little signal. Working with long-term labels like annual returns could help improve diversification across models.

## 12.4 Exercise

Build an integrated ensemble on top of 3 neural networks trained entirely with Keras. Each network obtains one third of predictors as input. The three networks yield a classification (yes/no or buy/sell). The overarching network aggregates the three outputs into a final decision. Evaluate its performance on the testing sample. Use the functional API.

### References

Bates, John M, and Clive WJ Granger. 1969. “The Combination of Forecasts.” *Journal of the Operational Research Society* 20 (4): 451–68.

Bonaccolto, Giovanni, and Sandra Paterlini. 2019. “Developing New Portfolio Strategies by Aggregation.” *Annals of Operations Research*, 1–39.

Breiman, Leo. 1996. “Stacked Regressions.” *Machine Learning* 24 (1): 49–64.

Claeskens, Gerda, and Nils Lid Hjort. 2008. *Model Selection and Model Averaging*. Cambridge University Press.

Clark, Todd E, and Michael W McCracken. 2009. “Improving Forecast Accuracy by Combining Recursive and Rolling Forecasts.” *International Economic Review* 50 (2): 363–95.

Donaldson, R Glen, and Mark Kamstra. 1996. “Forecast Combining with Neural Networks.” *Journal of Forecasting* 15 (1): 49–61.

Freund, Yoav, and Robert E Schapire. 1997. “A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting.” *Journal of Computer and System Sciences* 55 (1): 119–39.

Gaba, Anil, Ilia Tsetlin, and Robert L Winkler. 2017. “Combining Interval Forecasts.” *Decision Analysis* 14 (1): 1–20.

Genre, Véronique, Geoff Kenny, Aidan Meyler, and Allan Timmermann. 2013. “Combining Expert Forecasts: Can Anything Beat the Simple Average?” *International Journal of Forecasting* 29 (1): 108–21.

Grushka-Cockayne, Yael, Victor Richmond R Jose, and Kenneth C Lichtendahl Jr. 2016. “Ensembles of Overfit and Overconfident Forecasts.” *Management Science* 63 (4): 1110–30.

Harrald, Paul G, and Mark Kamstra. 1997. “Evolving Artificial Neural Networks to Combine Financial Forecasts.” *IEEE Transactions on Evolutionary Computation* 1 (1): 40–52.

Hassan, Md Rafiul, Baikunth Nath, and Michael Kirley. 2007. “A Fusion Model of Hmm, Ann and Ga for Stock Market Forecasting.” *Expert Systems with Applications* 33 (1): 171–80.

Huang, Wei, Yoshiteru Nakamori, and Shou-Yang Wang. 2005. “Forecasting Stock Market Movement Direction with Support Vector Machine.” *Computers & Operations Research* 32 (10): 2513–22.

Jacobs, Robert A, Michael I Jordan, Steven J Nowlan, Geoffrey E Hinton, and others. 1991. “Adaptive Mixtures of Local Experts.” *Neural Computation* 3 (1): 79–87.

Jagannathan, Ravi, and Tongshu Ma. 2003. “Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps.” *Journal of Finance* 58 (4): 1651–83.

Leung, Mark T, Hazem Daouk, and An-Sing Chen. 2001. “Using Investment Portfolio Return to Combine Forecasts: A Multiobjective Approach.” *European Journal of Operational Research* 134 (1): 84–102.

Mascio, David A, Frank J Fabozzi, and J Kenton Zumwalt. 2020. “Market Timing Using Combined Forecasts and Machine Learning.” *Journal of Forecasting* Forthcoming.

Okun, Oleg, Giorgio Valentini, and Matteo Re. 2011. *Ensembles in Machine Learning Applications*. Vol. 373. Springer Science & Business Media.

Razin, Ronny, and Gilat Levy. 2020. “A Maximum Likelihood Approach to Combining Forecasts.” *Theoretical Economics* Forthcoming.

Schapire, Robert E. 1990. “The Strength of Weak Learnability.” *Machine Learning* 5 (2): 197–227.

Schapire, Robert E, and Yoav Freund. 2012. *Boosting: Foundations and Algorithms*. MIT Press.

Seni, Giovanni, and John F Elder. 2010. “Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions.” *Synthesis Lectures on Data Mining and Knowledge Discovery* 2 (1): 1–126.

Virtanen, Ilkka, and Paavo Yli-Olli. 1987. “Forecasting Stock Market Prices in a Thin Security Market.” *Omega* 15 (2): 145–55.

Von Holstein, Carl-Axel S Staël. 1972. “Probabilistic Forecasting: An Experiment Related to the Stock Market.” *Organizational Behavior and Human Performance* 8 (1): 139–58.

Wang, Ju-Jie, Jian-Zhou Wang, Zhe-George Zhang, and Shu-Po Guo. 2012. “Stock Index Forecasting Based on a Hybrid Model.” *Omega* 40 (6): 758–66.

Wolpert, David H. 1992b. “Stacked Generalization.” *Neural Networks* 5 (2): 241–59.

Zhang, Cha, and Yunqian Ma. 2012. *Ensemble Machine Learning: Methods and Applications*. Springer.

Zhou, Zhi-Hua. 2012. *Ensemble Methods: Foundations and Algorithms*. Chapman Hall/CRC.