Back Testing, Forward Testing and Mathematical Nonsense

I typically get sent a lot of systems by people and they ask me for my thoughts.

This is great, but I do suggest that this is only a good idea if you’ve read our site and have a good understanding of what is involved with building trading systems – as you are likely to get some seriously honest feedback.

Read Sam’s series on designing and building Quant Models

Swerve the curve

The problem in this market place at the moment is there are a lot of systems out there built on very little, that don’t actually work. I am well aware that the previous top five systems on Myfxbook all blew up spectacularly when they got to market conditions they couldn’t cope with.

This has bred similar systems in the chase for unrealistic returns and short-run curve-fitted systems, but they typically always fall at similar hurdles, so I thought I would offer some insight.

Firstly, 3 months, 1 year even 3 years is not sufficient enough a test to see if a system works. My head of tech did an article ages ago about selecting a purely random system that looked amazing but actually was nonsense, precisely because it was random.

Read techy Ctrl-Alt-Dave’s articles on backtesting here

Why does your system work?

Typically when I challenge people on this I ask them why their system works; what is the fundamental point that means their system will work. When people ask me the same question I typically refer to the fact that we spent a large amount of time working on underlying order books int he OTC market and we use these to derive some underlying information about where price is likely to trend to next. What I typically receive are comments along the lines of ‘price action’, ‘patterns’ and market fundamentals (by this they typically mean more candlestick patterns).

The challenge with this is you can show quite easily that most candlestick patterns are no better than flipping a coin. What’s important is all the confluent signals that combine with them; you can also show that unless you know exactly the kind of market you are in right now, the probabilities of candlestick patterns massively changes.

Price action seems to be a favourite catch-all term now. The term is in itself rather useless, as typically most technical indicators and all charts are based on the action of price. Thus it is all price action.

If you are building a system based on a pattern though, I would at least expect you to know the probabilities of that pattern playing out in different market conditions and why that pattern works; i.e. what is it telling you about the market that allows you to gain some statistically significant advantage.

Sample testing

This brings me to the biggest challenge. Normally by this point I have either upset a few people or they have upset me with their continued delusion. But if not then we get to the challenge of sample testing.

This is one of the biggest hurdles I faced early on. It is the difference between back and forward testing or in sample and out of sample testing.

The concept is beautifully simple but difficult to get right.

Say we have 3 years of data, what we would do is build the system using 2 years of that data then test it on the other year of data.

I’d say 90% of the systems I see fail straight away as they are obviously curve-fitted nonsense that someone too arrogant to understand their own weaknesses is determined that they have made the holy grail of all systems. Of the remaining 10%; the majority work beautifully well in their back test period they were designed on, but fail on the ability to forward test; i.e. they work well in the period they were designed for but don’t perform the same in live or forward tests.

The humble few

The remaining few (i.e. let’s say less than 1% of what I am sent), work as described and they have been developed by some of the most interesting and smartest people I have met.

They aren’t arrogant, deluded or starry-eyed. They are as varied in character and background as one could imagine, but they do all have one thing in common: they tend to come with a passion for this field and humble understanding of the huge challenges involved.

Got a question for Sam? Tweet him directly at @LFXSam or contact us here

Want more? Read CEO Sam Barry’s series on building Quant Models

Retail versions of similar components to what we use in our systems feature as part of our Forex Trading course – find out more about it here.