This article forms part of a series by CEO Sam Barry on Quant Models. To read the first three parts, follow the links below:
- An Intro to Quant Models: Part 1
- An Intro to Quant Models: Part 2
- An Intro to Quant Models: Part 3
- An Intro to Quant Models: Part 4
So in the final part of this series introducing quant models, I want to talk to you about measuring performance.
This doesn’t just mean when your system is running, but also when you are determining which system to run. Part of this process is looking at your principles and metrics aligned to them.
For total return systems, primary focus is typically return per unit of risk or risk vs reward. For consistent systems, typically the measure is more aligned to a standard deviation or a metric incorporating these; typically this also then results in any number of ratios that can be used, Sharpe, Calmer, Sortino… the list is endless – the key here really is what you are measuring.
Firstly, it is usually some measure of reward, whether return, average return etc.. Then we work out how much of this key measure we get, to some unit of a measure we don’t like, like variance or drawdown or negative returns etc.
This then gives us a basic score to judge and compare systems by, but often this is barely scratches the surface of what we want to see.
For each of these individual metrics you can find an exception to the rule that scores highly, but you would never run it.
Often the key to the total measure is viewing several of these factors together.
This then becomes a very large mathematical problem in how you determine the best system and frankly there is no right answer and the debate has raged for years, in somewhat pointless fashion in my view.
I think the challenge with this is what we said in the previous article which circulates around your principles and who this is for.
Another one of our key assets is our scoring system.
Whenever we are building new systems to run we look at over 5 million individual strategies we could run (I expect by the time you read this that number is a lot higher). You therefore need a way of filtering them. We do that with our own scoring system which builds systems we like to run and operate.
But to give some idea, it doesn’t rely on one or two metrics. It has a portfolio of its own scoring systems within it that it uses; the number of metrics is now in the high double digits. So if you are just looking at Sharpe or Sortino, you need to be careful.
However, for those looking for a starting point (or ever want to present systems to us / want a job with us) I would consider the following key metrics:
- Sharpe Ratio
- Sortino Ratio
- Sterling Ratio
- Calmer Ratio
- Return Distribution curve (Skewness, Kurtosis)
- Drawdown (peak to trough for both equity and balance)
- Standard Deviation
- Downside Deviation
- Averages (lots of them)
- Correlations (a few)
- Rate of Return
These are probably the key ones. One note; if you are looking at win ratios as well, that’s great but focus on time periods such as days, weeks, months and years. Be careful with high win ratios on per trade basis as it either means arbitrage or potential lack of risk management, meaning lots of small gains for large blow outs.
Above and beyond all of this though is the need for accuracy.
The challenge I have with a lot of systems that are presented to me is that they take such a rosy view of the world that you could never achieve the results they claim.
This can range from the fact that they assume they will get filled at best bid and best ask (they won’t), and that they get zero slippage and – my favourite – that there is no spread.
Most of these aren’t all the traders fault, although he should have questioned the performance in detail, often they are a challenge of building cost effective trading platforms for people.
A typical hedge fund will be paying over £300,000 per year for a reasonable trading platform (that’s not even a good / bespoke one for their needs) and the cost frequently isn’t the technology within it, but its ability to get accurate data and manage trades to its best ability.
Herein lies the key to measuring how good your system really is: how accurate can you make it.
We spend probably 70% of our development time ensuring everything we produce is as accurate as can possibly get it, our tolerance levels are now extremely low but that has taken a significant amount of bespoke development work to achieve this, and it doesn’t come lightly at all.
I think the other key, especially when back testing, is the amount of data you use.
A lot of people slip up when they use a small subset of data for the trading style they have, and, don’t get me wrong, getting good data is extremely hard as anything with serious integrity is costly; but it is critical.
For any system trading on a daily we stipulate a minimum of 10 years worth of accurate (proven accurate) backtesting. In fact, for many of our 1 minute or lower systems we will use 10 to 15 years worth of data to ensure we cover as many possible market conditions as we can.
This helps us understand and evaluate the key characteristics of the system in lots of potential scenarios. The truth is we have no idea what the future holds, but if we have designed the systems and measured them in their most stressed conditions and they hold up, then hopefully they will continue to do so in the future.
Got a question for Sam? Tweet him directly at @LFXSam or contact us here…
Get your hands on all the trading plans, indicators and strategies required to gain a comprehensive understanding of the Order Flow trading techniques successfully used by the traders here at Littlefish FX. Find out more here.