Backtesting an investment strategy
Sep. 14th, 2018 06:19 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
mtbcYears ago I was ongoingly involved in research relating to irrationality in human decision-making. Some of that work led to an investment strategy idea. From EODData I downloaded a decade's worth of data for multiple stock exchanges and did some extensive analysis and backtesting. I was very careful: for example, rather than just have the software tell me how much it made, I wrote its data input component as a stream to make lookahead impossible and had it output a series of buys and sells that were independently tallied and could be spot-checked by hand. The results looked very promising. We tried to think of how we might be mistaken and wrote statistical tests and suchlike accordingly and everything still looked good. Then, real money was put into the scheme and it failed to perform. Fortunately it wasn't my money.
I recently came across my old code for this project and was pleasantly surprised at just how much I did in verification and documentation. Frustratingly, what I don't know is: why the strategy worked so well on paper but not in real life. It's not like enough was being invested to affect the markets, I was careful to filter out low-volume stocks. It's known that strategies that once worked can unexpectedly stop working but my background is more in modeling and simulation and more physics-y math, my statistics is too weak for me to even have been able to say confidently that the strategy did badly enough for long enough with real money to clearly refute the backtesting. I wonder if there is some odd issue with how the historical data sets may have been adjusted for stock splits and whatnot that I failed to uncover. A lesson I take from the experience is to try strategies for some time on paper with ongoing data before committing any funds of my own.
I recently came across my old code for this project and was pleasantly surprised at just how much I did in verification and documentation. Frustratingly, what I don't know is: why the strategy worked so well on paper but not in real life. It's not like enough was being invested to affect the markets, I was careful to filter out low-volume stocks. It's known that strategies that once worked can unexpectedly stop working but my background is more in modeling and simulation and more physics-y math, my statistics is too weak for me to even have been able to say confidently that the strategy did badly enough for long enough with real money to clearly refute the backtesting. I wonder if there is some odd issue with how the historical data sets may have been adjusted for stock splits and whatnot that I failed to uncover. A lesson I take from the experience is to try strategies for some time on paper with ongoing data before committing any funds of my own.
