Black Box Financial Model

Black Box Machine Learning

Einstein observed that “if you can’t explain it simply, you don’t understand it”, and so it is with complex machine learning (ML), where the most powerful models are black-boxes. Unabated, ML now measures ESG risk, execute trades, drives stock selection, portfolio construction, and more. ML’s now accelerating expansion across the investment industry, creates completely novel concerns over “reduced transparency”, and how to “explain and justify investment decisions” (CFA Standards of Practice Guidance, 2019). Frankly, “unexplainable ML algorithms […] expose the firm to unacceptable levels of legal and regulatory risk” (IOSCO, 2020). In plain English, that means if you can’t explain your decision making, you, your firm and your stakeholders are in deep trouble. Explanations, or better still direct interpretation, is therefore essential. Great minds have wrestled with this question in other major industries that have already widely deployed machine learning, as we shall see. It changes everything for those in our industry who would favour computer scientists over investment professionals, or try to throw naïve, and out of the box ML into investment decision marking. 

In this article, I reflect on 20 years of quantitative investing and ML research to contrast the two solutions on offer:

1. Interpretable-AI : Use less complex ML that you can directly read and interpret

2. Explainable-AI (or XAI) : Use complex ML, and attempt to explain it

While XAI could be the solution of the future, that’s the future. For the present and foreseeable, for my money, interpretability is where you should look to gain the power of machine learning and AI. Let me explain why.

​

Finance’ second tech revolution.

It is now generally accepted that ML will form a material part of the future of modern investment management, promising to reduce expensive front-office headcount, replace legacy factor models, lever vast and growing data-pools, and ultimately achieve asset-owner objectives in a more targeted, bespoke way. The slow take up of technology in investment management is an old story, and so it has been with ML up until recently, but the rise of ESG over the past 18months, and the scouring of the vast data-pools needed to assess it, have been one key force that has turbo charged the move to ML. Speaking anecdotally, the demand for these new expertise, and solutions has outstripped anything I have witnessed over the last decade, and since the last major tech revolution hit finance in the mid-1990s. The pace of the ML arms race is concerning though, the apparent uptake of newly self-minted experts alarming, and that this revolution is being hijacked by the geeks rather than the business, is most concerning of all. Explanations for investment decisions will always lie in the hard rationales of the business.

​

Interpretable simplicity? Or Explainable complexity?

Interpretable-AI, (also called Symbolic AI (SAI), or ‘good old fashioned AI’), has its roots in the 1960s but is again at the forefront of AI research. These systems are generally rules based, like decision trees[1], but now have far more powerful processes for rule-learning. However the rules are learned – and this learning process can be very sophisticated indeed– it is these rules that should be applied to the data. The rules can therefore be directly examined, scrutinised and interpreted, just as you might Graham and Dodd’s investment rules. Simple perhaps, but powerful, and if the rule learning has been done well, safe.

Explainable-AI (XAI) is the alternative and is completely different. XAI attempts to find an explanation for the inner-workings of black-box models that are impossible to directly interpret (see Figure 1). For black-boxes, inputs and out-comes can be observed, but one can only guess at the processes in between. This is what XAI generally attempts, to guess and test its way to explaining the black-box processes, using visualisations to show how different inputs might influence outcomes. It is early days for XAI, and it has proved challenging. Two very good reasons to defer judgement and go Interpretable.

​

Black Box Investment

One of the most common XAI approaches used in finance is SHAP, which has its origins in Shapely Values from game theory, and was fairly recently developed by researchers at IBM (Lundberg and Lee, 2017)[2]. Figure 2 shows the SHAP explanation of a stock selection model, resulting from only a few lines of Python code, but it is an explanation that needs its own explanation. It is as super idea, and very useful for developing ML systems, but it would take a brave PM to rely on such tools to explain a trading error to a Compliance Executive.

​

Drones, nukes, cancer diagnoses… and stock selection

Medical researchers and the defence community got to the question of explain or interpret, before financiers, and while powerful application specific solutions have been achieved, no general conclusion has yet been reached. The US Defence Advanced Research Projects Agency (DARPA) has conducted thought leading research in this area, and has presented interpretability as being a cost, hobbling the power of a machine learning system. Figure 3, left side, illustrates this with various different ML approaches. In this line of thinking the more interpretable an approach, the less complex it will be and therefore the less powerful. This would certainly be true if complexity was associated with accuracy, but the principle of parsimony, and some heavy weight researchers in the field beg to differ, perhaps making the reality more like the right side of Figure 3.

​

Complexity bias in the C-suite

The assumption baked into the explanability camp, that complexity is warranted, which may be true in some applications such as predicting protein folding[3], where deep learning is certainly needed, but it has been challenged in many other applications, including stock selection. This was exemplified by an upset in the landmark “Explainable Machine Learning Challenge” proposed at the Neural Information Processing Systems (NeurIPS) conference (the leading AI research event of the year) by Google, the Fair Isaac Corporation (FICO), and academics at Berkeley, Oxford, Imperial, UC Irvine, and MIT. It was supposed to be a black-box challenge for neural-nets, but a team involving Prof Cynthia Rudin of Duke University had different ideas. They proposed an interpretable, ie more simple, machine learning model that was not neural net based, and therefore did not require any explanation. It was already interpretable. As Rudin has since put it, “the false dichotomy between the accurate black box and the not-so accurate transparent model has gone too far. When hundreds of leading scientists and financial company executives are misled by this dichotomy, imagine how the rest of the world might be fooled as well”. Perhaps her most striking comments are that “trusting a black box model means that you trust not only the model’s equations, but also the entire database that it was built from”. What Rudin is addressing here is what we in finance would recognise as yet another behavioural bias in ourselves: "Complexity bias". Our tendency to find the complex more appealing than the simple. The C-suites that are driving the AI arms race might pause for to reflect on this, rather than continue the all-out quest for perhaps excessive complexity.

Black Box Models in AI

Interpretable, Auditable machine learning for stock selection

While some objectives demand complexity, there are others that will suffer from it. Stock selection is one example. The recent technical publication in the JFDS from myself, David Tilles and Timothy Law strongly supports interpretable-AI, as a scalable alternative to factor-investing for stock selection in equities investment management. Our approach learns simple, interpretable investment rules, learning these rules using the non-linear power of a simple machine learning approach we explain in detail. The novelty is that it is simple, interpretable, scalable, and – we believe - could replace, and far exceed factor investing. We also anecdotally note that our interpretable-AI approach does almost as well as far more complex, black-boxes approaches we have experimented with over the years. This approach is simple, fully interpretable and transparent, and therefore auditable, able to be communicated to stakeholders without an advanced degree in Computer Science. It does not need XAI to explain it, as it can be directly interpreted. Our motivation for revealing this IP was our long held belief that excessive complexity is not needed for stock selection, and in fact almost certainly harms it. In short, interpretability is paramount. Your alternative is the red herring of making things so complexity that you have use another complex approach to explain it. Where does it end?

​

One to the humans

The debate over explain or interpret is raging. $100s of millions are being spent on its research to support the machine learning surge we are seeing in the most forward thinking financial companies. As with any cutting edge technologies, we can expect false starts, blow ups, and wasted capital. But for now and the foreseeable future, the solution is interpretable-AI. Consider two truisms: explanations are generally needed for matters that are too complex; there is no need to explain something if it can readily be interpreted. In the future XAI will be better established and understood, and much more powerful. For now it is in its infancy, and it would take a brave investment manager to expose their firm and stakeholders to the chance of unacceptable levels of legal and regulatory risk. General purpose XAI does not currently provide a simple explanation, and as Einstein once said, “if you can’t explain it simply, you don’t understand it”.

 

[1] Although, please note that decision trees are terrible forecasting tools and typically over-fit to the data. Useful as a tool to understand what has happened in the past, but I would strongly advise against their use as forecasting tools.

[2] https://arxiv.org/abs/1705.07874

[3] DeepMind’s AlphaFold probably represents on of the challenges of our generation, and an awesome achievement of engineering https://www.nature.com/articles/s41586-021-03819-2