Do Not Fall into These Financial Back-Testing Traps.

Do not be Misled by these Errors When Back-testing your Trading Strategy.

Back-testing is your proof of work for any strategy you might think of. If the trader’s friend is the trend, then back-testing is his best friend. Many traders out there just “accept” the mainstream strategies that are already out there and use them in hopes of becoming rich.

Overfitting and Underfitting

After reading the brilliant work of Dr. Marco Lopez De Prado on biases a few months ago, I have come to understand that there is always something that escapes us no matter how perfect we think the analysis was.

  • Variance is also known as overfitting and this is simply when the model encounters noise and thinks it is a signal.
  • Decreasing the training period so that model does not exactly fit the training data.
  • If you are using neural networks, consider a dropout function and an early stopping technique.
  • Increasing the training period so that model finds more data to work with and be able to make predictions based on sufficient information.
Image for post
Image for post
Summary Table. (Image by Author)

Forgetting Transaction Costs

Something you should always be aware of is that back-testing results are mostly false. You are likely to never get a good estimate of future results except perhaps by luck. You cannot accurately estimate the actual fees, spreads, slippage, and any other unexpected events that will occur during live execution and therefore when including a proxy of these costs in your back-tests, it is always helpful to bias them upwards.

Image for post
Image for post

Look-Ahead Bias

This bias is known in the field of back-testing and research and although it is considered more of a rookie mistake, it must never be forgotten. Look-ahead bias is when you are using the future to predict the past. Consider a strategy that relies on the daily closing values of the S&P500 to make predictions. When designing the model, you erroneously, use today’s close to predict today’s close, hence, you are using a future information to try to predict the future and this is of course unfeasible.

Image for post
Image for post

Not Accounting for Market Regime Changes

Most of the time, markets are either trending or ranging and while we can develop strategies for both market regimes, it is difficult to find one strategy that is able to capture the change in the regime and adapt itself all while continuing to be profitable. This Strategy-of-Everything is unlikely to exist at the moment as financial time series are highly complex and dynamic.

Image for post
Image for post
The S&P500 Index. A Mostly Trending Market.

Non-Stationary Data

Stationarity is synonymous to a constant mean over time. A changing mean will cause the model to produce erroneous forecasts. A time series data is stationary if it has a constant mean and variance, that is, its mean does not change much by time. The same goes for its variance (volatility).

Image for post
Image for post
A stationary data series with a mean ≈ 0.09. (Image by Author)

As time series (Prices) are significantly autocorrelated, the model will always follow the latest value, and therefore, it will likely reproduce the previous value and call it a forecast. When time series are transformed into stationary data either by differencing or by taking returns, this problem more often than not goes away.

Let us see what happens when we use a normal auto-regression technique to forecast Bitcoin. The first test will use non-stationary (i.e. BTCUSD prices) data and the second test will use stationary data (i.e. BTCUSD returns).

  • Model used: Linear regression.
  • Training days: 2221.
  • Testing days: 100.
Image for post
Image for post
R-square between the predicted and actual values shows a superior modelling power of our algorithm. (Image by author)
Image for post
Image for post
Actual values vs predicted values. The model seems to be simply replicating the value of yesterday. (Image by author)
  • Model used: Linear regression.
  • Training days: 2221.
  • Testing days: 100.
  • Accuracy: 49%
Image for post
Image for post
The plot of predicted vs actual values shows no correlation between the two whatsoever, giving a stronger conviction of the underperformance of our model. (Image by author)
Image for post
Image for post
The returns of BTCUSD were much more volatile than predicted with the linear regression model. The model has been wrong about 51% of the time. (Image by author)
Image for post
Image for post

Focusing too Much on the Hit Ratio

Sure, an 80% hit ratio on your trades is great. But what if you are risking $1 each time to earn $0.20 (20 cents)? Well, then you will lose money and get wiped out because your risk reward will be 0.2. If you make 100 trades where you always use the same position sizing and you get your 80% hit ratio which translates to 80 profitable trades with each gaining $0.20.

Image for post
Image for post
Risk-reward Trade-off and the Hit Ratio. At a risk-reward Ratio of 1.00, we need 50% to breakeven. (Image by author)
  • Total loss = 0.6 x 1.00 = -$0.600
  • Net profit = 0.728–0.600 = +$0.128
Image for post
Image for post

Not Taking Into Account Yearly Performances

Look at the below equity curve and tell me what do you see? Clearly, it is upwards sloping and looks attractive. After all, you have started with $1,000 and now have around $4,500. Now, let us zoom in.

Image for post
Image for post
An example of a Strategy that has produced positive cumulative returns. (Image by author)

Not Taking Into Account Risk Management Before Going Live

When you back-test a strategy, you must account for stops and profit orders. In other words, when you do apply the strategy in real life, you will place stops and profit orders. You should know that these orders change drastically the performance. Here is the RSI strategy with and without placing fixed stops.

Image for post
Image for post
EURUSD Hourly with 14-Hour RSI. (Image by author)
Image for post
Image for post
Comparison between the two RSI Strategies. (Image by author)

Conclusion

There will always be some form of bias in the back-test. Our job as researchers and traders is to minimize them so as to maximize the probability of realization. We are all of course familiar with the saying that the history does not repeat itself or that the past is not a reflection for future profits but the past is the best we have in our fight against the future. If you manage to at least incorporate some of the above points, then you are likely on the right track. Remember, finding your strategy is not an overnight process, be patient.

Image for post
Image for post
Image by Nattanan Kanchanaprat from Pixabay

Written by

Institutional FOREX Strategist | Trader | Data Science Enthusiast. Author of the Book of Back-tests: https://www.amazon.com/dp/B089CWQWF8

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store