Cracking The Dark Pool: Forecasting S&P 500 Using Machine Learning

Creating a Random Forest Algorithm to Predict S&P 500 Returns

Dark pools, also known as alternative trading systems, are private, off-exchange platforms used by institutional investors and high-frequency traders to execute large block orders of stocks and other securities. Unlike traditional stock exchanges, dark pools do not display order book information or the identities of trading participants to the public.

This article will download historical values relating to dark pool activities and will develop a random forest model that aims to predict the returns of the S&P 500 using the dark pool values.

The Dark Index

The dark index (DIX) provides a way of peeking into the secret (dark) exchanges. it is calculated as an aggregate value of many dark pool indicators and measures the hidden market sentiment. When the values of this indicator are higher than usual, it means that more buying occurred in dark pools than usual. We can profit from this informational gap. It is therefore a trail of liquidity providers (i.e. negatively correlated with the market it follows due to hedging activities).

The following graph shows the historical values of the DIX.