Backtesting: Dangers of Over-fit

I’d rather be lucky than good.

Lefty Gomez

NinjaTrader Cumulative Profit Report, 01_01_2012 - 01_01_2013

This chart should grab your attention. The strategy makes $68k using a $5k account using an aggressive 10% trade size with reinvestment in a single year. The drawdown is a little painful at $7k but has a very respectable Sharpe Ratio of 1.62. The 680 trades are 56% winners with average win/loss return of 1.17 and average trade length of 422 minutes.

Without knowledge of how the strategy works, it does actually appear quite good. But the following chart shows it was just a single good year.

NinjaTrader Cumulative Profit Report, 01_01_2011 - 01_01_2014

This shows a clear example of overfit, also known as curve fitting.

The strategy was actually completely random, using a 5% chance to go short and 5% to go long on each bar of a 30 minute chart. A key fact for random number generation is that the system will generate the same random numbers in the same order if given the same “seed” which is simply a given number. I backtested the strategy using seeds of zero to 50,000 and 27720 was the winner.

The key parts of the strategy looked like this,

private Random generator;

if( generator == null )

{

generator = new Random( seed );

}

int signal = generator.Next( 100 );

if( signal < 5 )

{

EnterShort();

}

if( signal > 95 )

{

EnterLong();

}

A common method to avoid such pitfalls is called out-of-sample testing. In my example, I used the year 2012 as the sample period, and thus trying a backtest on prior years showed that I had just over-optimized for this period.

Some people might consider building an obnoxious website claiming a 1360% yearly return and definitely too much flashing coloured text. And a picture of what large piles of money can buy, like a car or yacht, just in case you were unsure.

Merely as a thought experiment, consider generating several hundred such strategies. Clearly most of them will haemorrhage money and should be deleted. However, a lucky handful will make decent money over the initial few months.These can then be made public, and heavily promoted based on their proven and verifiable track record.

Since the strategies look amazing, they would gain the public interest and thus earn commission for the owner. Unfortunately, the luck will run out one day, and investment in the strategies will fall. 

When looking at such strategies, remember the Latin phrase, caveat emptor.