Balancing control and chaos, with Dan Davies

Dan Davies, one of my favorite authors (Lying for Money is perhaps my favorite non-fiction book of this decade), came on the podcast to discuss his new book The Unaccountability Machine on organizational design (and why it occasionally creates perverse outcomes in recent times).

[Patrick notes: I add after-the-fact notes to the transcript, set aside in this fashion.

As befits a conversation between a serious writer and a serious reader, we mention a lot of books. Not all mentions are necessarily endorsements; some of these are historical texts at this point. I took the liberty of linking to Amazon for convenience. No, they're not affiliate links.]

Sponsor:

This podcast is sponsored by Check, the leading payroll infrastructure provider and pioneer of embedded payroll. Check makes it easy for any SaaS platform to build a payroll business, and already powers 60+ popular platforms. Head to checkhq.com/complex and tell them patio11 sent you.

Timestamps

(00:00) Intro
(00:26) The Unaccountability Machine
(01:38) History and fundamentals of cybernetics
(08:10) Operations research and its evolution
(12:08) Theory of the Firm, revisited
(15:21) Monopolizing math for fun and profit
(18:38) Sponsor: Check
(19:50) Role of black boxes in systems
(25:11) AI and the future of system management
(30:02) Accountability sinks and organizational issues
(38:44) Optimism about future of organizational design
(43:45) Empowering employees: the CEO’s open door policy
(46:31) Lying for Money
(51:57) Psychology of fraudsters
(01:02:52) Fraudogenic environments
(01:09:49) Journey of becoming a published author
(01:18:13) Effective ways to sell books
(01:22:33) Wrap

(You can Ctrl-F for most of these titles to skip to that point in the transcript. Sorry, due to technical reasons, I cannot simply link them everywhere this transcript appears.)

Transcript

Patrick: Hi everybody. I'm Patrick McKenzie, better known as patio11 on the internets, and I'm here with one of my favorite nonfiction authors, Dan Davies.

Dan: Hi! I'm really pleased to be here.

The Unaccountability Machine

Patrick: Honored to have you here, Dan. So I'd love to talk about your new book, The Unaccountability Machine. I've been reading it with interest the last couple of weeks.

Dan: Oh, thanks. It's been a bit of a labor of love for me. It's taken about five years, start to finish, although I took a break in the middle of that to write a different book. 

It's really talking about a kind of thing that's adjacent to the themes of this podcast: how people build systems to manage things because they are too complicated for any one human being to hold them in their head.

But then the systems take on a life of their own, and in particular, I'm most interested in the situations where a decision-making process has been industrialized – and where it now really seems like the only sensible way that you can look at the situation is one where the system is in control, and where the system is making the decisions, and they can't be attributed easily or fairly to any one natural person, any one single human being.

History and fundamentals of cybernetics

Patrick: Yep. So the book starts with a deep dive into the history of cybernetics, which is honestly something I was under-informed about, but I suppose that's why we read. The brief version is that cybernetics is sort of the history of looking at organizations – not so much as social systems, which we have a bit of in ethnography and other places, but as more technical systems for getting work done.

Dan: Yeah, absolutely. The way I would describe it is that cybernetics is applied mathematics in the same way that economics is applied mathematics – but like economics, it's mathematics being applied in an area where most of the important things aren't necessarily mathematically quantifiable.

So, I mean, maybe I should unpack that a bit. 

Originally the word was coined in the 1940s by Norbert Wiener, who was at work trying to design an automated gun site. He was working on how to get a gun site to automatically track aircraft, and he started realizing that there was some single quantity that was being preserved between the radar, the operator, the server motors, and the site. This quantity might be what you would call information.

The idea that if you could treat this abstract quantity of information as a single entity, and ignore all of the physical processes going on, just lent itself to a whole load of analysis, in the early days of computer science, in the early days of control engineering, optimization – all of these techniques were being invented at the end of the second world war. 

There's a whole parallel strand of that analysis, which ended up becoming information theory – and another way to look at cybernetics is that it's information theory kind of turned through 90 degrees, to be thinking about things in terms of control. (I'm kind of going a bit fast here, but my personal favorite nonfiction book in the last 20 years is James Gleick's The Information, which is all about the discovery of information theory at Bell Labs by Claude Shannon.)

Patrick: I was going to name drop Claude Shannon and prove that I actually have a CS degree, darn it.

Dan: Shannon's fundamental theorem of information theory is showing that information is entropy – and in Shannon's formulation, it's entropy because information is the opposite of redundancy. It's the extent to which a signal can't be compressed.

Now, Wiener actually came up with the exact same theorem close enough in time to Shannon's theorem that they both decided that, you know, they were friends and it would be in nobody's interest to have a big battle about who invented what first. But in Wiener's formulation of information theory, it's entropy with a negative sign in front of it, because the way that Wiener and cybernetics interpreted information is that it's a measure of disorder in the system. It's a measure of the chaos that the system can throw up. 

From there you get other extensions of that in applied mathematics, and the absolute fundamental theorem being discovered by a guy in Britain called Ross Ashby. He was actually a brain doctor, but he was trying to build an artificial intelligence – even back in 1950, before the first IBM machine had been imported into the UK.

Ashby's Theorem of requisite variety is just the very simple principle that the bandwidth and capacity of a controller has to be at least as great as the amount of entropy and variety generated by the system that it's trying to control. If we think of economics as being basically the science of trying to make sure that supply and demand balance with one another, then cybernetics, at that same level of generality, is the same principle of trying to make sure that the information balance sheet balances, and that the capacity to manage the system is equal or greater to the amount of chaos and uncertainty that the system itself can throw up. 

The big difference is that in economics, you talk about equilibrium because if supply and demand are out of balance, then they will tend towards equilibrium – prices will move, supply and demand will adjust, and they will tend towards equilibrium. 

In cybernetics there's no such easy equilibrium process for the information matching; it has to be produced by constant effort. That's why for someone like Stafford Beer and the early cyberneticians, they're always talking about the word homeostasis. Homeostasis is basically the same thing as equilibrium, it's just the Greek word – homeostasis, same state – but it has that implication that this is not a self-sustaining stable process. This is a state which requires constant input of energy in order to maintain it. 

The way that a guy explained it to me was that, if your body's temperature is in homeostasis, that's good, because it's being maintained at a constant 98 degrees Fahrenheit, 37 and a half degrees Celsius. If your body temperature is in equilibrium, that's probably bad, because that means you're in equilibrium with the surrounding temperature and that usually means that you're dead.

Operations research and its evolution

Patrick: So it seems to me from the outside that cybernetics is one of a number of more rigorous intellectual approaches to analyzing organizations and production systems as systems themselves that we got right around the Second World War – largely done by people who were given incredible incentives to understand how factories and logistics and similar worked, many times for the first time in history because we were in total war mode. I think operations research was essentially invented in parallel at roughly the same time.

One of the enduring mysteries of the 20th century for me is how we invented operations research in the United States and the UK during the war, and then promptly forgot about it for, it seems, about 40, 50 years afterwards – meanwhile W. Edwards Deming took it over to Japan and successfully evangelized it in large Japanese manufacturing companies during the interim, and then they sort of ran the table on the world for much of the next 50 years by basically having a worldwide monopoly on applied mathematics.

[Patrick notes: There are many books written about this, some implicating a company which I never officially worked at but which many people in the vicinity of Nagoya do not officially work for. You might find this one or this one useful on the general topic. There is a bit of a cargo cult around the whole field, much like there is a cargo cult around Agile, for the same reason: this model won, decisively. Practice in the field these days is not merely using or not using the model, it exists in conversation with a world transformed by that model, in a way which influences customers, suppliers, the ecosystem, etc, whether on uses the specific words/tactics/etc of the model or not.] 

Dan: Well, there's a whole story there – it's, again, a story partly about information, but it's also a story about ideology. The thing about operations research, management cybernetics, all of these things which just basically mean applied mathematics in the context of controlling systems, big systems, is that, as you say, they did develop from the military.

They developed from a very command and control-oriented kind of philosophy, in which you were always going to assume that you could hand out orders and make things happen and that you would have reasonable control over the information environment – which is to say not perfect control, but you'd be aware of the major sources of uncertainty in the process.

And of course, this was the Cold War. This was not a time at which it was particularly ideologically simpatico in the USA or in Western Europe to be talking about things in terms of command and control structures and information gathering, information management and processing and making of five-year plans.

It was a period in which we were very ideologically committed to the idea that the free market was effectively our way of processing information – that, you know, the financial markets coupled with competitive goods markets was our equivalent of a distributed supercomputer – and to be honest, who's to say that was wrong?

I'm not going to disagree with you about the influence of Deming in Japan and Lean Production and all those things, but, you know, the USA didn't do too badly. I've been there. It's a great place.

[Patrick notes: Lean Production is a reference to the constellation of practices associated with a certain automobile manufacturer. That particular one is about radically minimizing the amount of “work in progress” and inventory levels, in the pursuit of greater efficiency, quality, and cost control.]

Dan: The problem was that as things got more and more complicated, to my mind, the financial market in particular just stopped working as a regulator. Maybe we can kind of come onto this in some other context, but particular pathologies of the financial markets – in particular the overuse of debt – just made it work much less well as a massive distributed supercomputer to organize our whole economy, and maybe we're reaching some kind of crisis point now.

Theory of the Firm, revisited

Patrick: I can't remember who first articulated this [Patrick notes: Ronald Coase, in Theory of the Firm], but one of the idiosyncrasies of capitalism is that we rely on price as a signal in competitive, vibrant markets for trade between firms – but then in trade within firms, virtually no one has a true internal markets operating (with the possible exception of Amazon with regards to some capital allocation decisions, but that's definitely not the main way that firms operate).

They operate more on the command and control model, where you have an org chart, and the org chart passes orders and memos around, and hopefully that converges on actually producing something of value in the economy.

One thing that we see repeatedly when we try to analyze systems is that we have a theory for how the system operates – maybe we have mathematics that can describe that in some fashion – and then reality confounds our expectations because all models are wrong and some models are useful but some models are, let's say, wronger than others.

Is there, like, a Seeing Like A State vibe with regards to cybernetics, in that we created this wonderful theory of the firm, but perhaps it does not describe exactly the firms that we have in front of us?

Dan: Well, it's always the problem. There's a guy – don't know if you've come across him, incredibly smart guy – up at the University of Berkeley called Ben Recht, and he's very much into control engineering, borderlines of machine learning, recommendation engines.

I was talking to him at a conference a few months ago and he said, you know, “in every one of these theories of organization, there comes a point where complexity increases and you switch from math to metaphor.” For economics, you can have things like derivatives pricing and finance theory, and that's pure math – bond pricing is simply math, and there is a right answer to however many decimal places you want to give it.

It's precise because you have access to the entire relevant information environments – everything that is worth knowing about a US government bond can be held on one single Bloomberg screen. (If it's a junk bond or orderline investment grade bond there might be a bunch of other things that you have to know that aren't on that Bloomberg screen however, but most of the time you can still say “Pricing this thing is basically math.”)

[Patrick notes: This ties into the banking business, among many other things. U.S. government bonds and bank deposits are “information insensitive”, meaning that no one transacting in them reasonably has an informational edge on anyone else transacting in them. That is an important reason they are usable as money. Startup stock, while it is valuable, is “information sensitive” and so you have far wider error bars on what a pile of it worth a notional $10 million is “actually” worth.]

Dan: Then you start moving into, “well, I'm going to try and say what inflation in the USA is going to be like in three years time.”

Now you do not have the entire relevant information environment. You have to simplify, you have to abstract, you have to collect statistics – and every time you're making a decision about how you're going to collect and tabulate the statistics, you're throwing information away. Because that's the purpose of doing so. You're trying to get it back to something that can be held on maybe not one Bloomberg screen anymore, but one of those kinds of skyscraper screen arrangements that traders like to show off with.

Monopolizing math for fun and profit

Patrick: One of the things that I find interesting in reading histories of Wall Street from effectively the end of the war through the present, is that a lot of things we take for granted now – like “there is math that describes the proper pricing for bonds” – had to be invented, and there was a period of more than a decade where… I can't remember if it was Salomon Brothers or Goldman with regards to bonds specifically, but where I think the formula wasn't exactly a state secret, but the notion of actually telling traders to apply it effectively was.

And so for more than a decade, they got excess returns simply by willingness to actually do the math and not make up numbers off the top of their heads for prices.

Dan: Yeah. I think it was Salomon. 

I think it was Sidney somebody [Patrick notes: Sidney Homer, when he was not writing about the history of interest rates] who wrote the book and the formulas and gave it to the traders, but yeah – in those kinds of things, you can see that some people think about this mathematically and some people are moving towards rules of thumb.

In economics, you might then say, well, “Is China or America going to be the world's dominant economy in 10 years time?” Right now you've got no hope of summarizing that on however many Bloomberg screens you want to put on your wall. You're purely saying, “I have some training in the mathematics; I know what kinds of things make an economy work; I know what kinds of things go into production – but I'm using this almost entirely to tell stories. I'm not making exact calculations anymore. I've got the mathematical framework in my mind, but I'm using it entirely metaphorically.”

There's something exactly similar going on with cybernetics. When you are designing a telephone switching system, it’s math. It's purely math. When you say, “how much bandwidth have I got in my house?” you can get an answer down to as many decimal points as you want to measure in terms of kilobytes per second. That's going to tell you how many movies you can stream at one time, how many video calls you can make, at what kind of quality – the whole system works exactly, because the people designing it have the exact same information environment. They've got everything there. 

When you start talking about organizations or countries or entire world systems, you'll still believe that the bandwidth of the management system has to exceed the entropy of the system they're trying to manage, but there's no way of quantifying that in terms of bits per second – all you're saying is that one unimaginably huge number has to be bigger than another unimaginably huge number, neither of which I have any way of measuring. 

You might ask, “Well, doesn't that mean everything you say is going to be unfalsifiable then?” Not necessarily, because there's one great way of finding out whether one number is bigger than the other number: that is, has the whole thing blown up? If the whole thing blows up and becomes uncontrollable, then you can be pretty sure that there wasn't enough bandwidth presence to be able to manage it.

Patrick: I think one of the challenges with all analyses of systems is that every subsystem is essentially an abstraction where we believe we can model the effectiveness of the subsystem reasonably well – but it turns out when you actually peek under the covers we have a model for the effectiveness of the subsystem that does not necessarily track reality 100 percent, using the example of networks that you just gave.

I agree that in principle it should be possible to measure bandwidth with a very high degree of precision. As someone who has known talented network engineers before and understands himself to not be one I have an appropriate amount of healthy respect for the Lovecraftian horrors that sit under every network stack.

You can reason about networking, but you can't predict networking in the same fashion that, you know, some clock-maker type of omniscient deity could predict the rotation of the planets.

That becomes more relevant when we build systems that assume, like, "Okay, networking is a basically solved problem, so when I send a request out to the Internet, that will probably work.” A lot of systems engineering at scale is building in redundancy and other countermeasures for reducing the uncertainty with regards to the word “probably” – maybe quantifying it, and then maybe having some sort of error term at the end where it's like, “Okay, we can model this pretty well… except on days like last Friday where there are huge outages at many counterparties because a particular software company that very few people have heard of shipped a poorly formatted file, and now many Windows machines across institutions will fail to boot.”

[Patrick notes: This podcast was recorded soon after the CloudStrike outage of mid-2024. I wrote about it for Bits about Money since it hit U.S. banks disproportionately hard.]

Role of black boxes in systems 

Dan: Yeah, that was absolutely correct. The way that my guy, Stafford Beer, David Bowie's favorite management scientist – which I can prove, by the way; his book Brain of the Firm was on David Bowie's top 10 books as late as 1995 – the way he dealt with this is to say, “When you're analyzing the system, you're drawing black boxes.

The first step is always to decide what you're going to try to understand right the way down to the physical causation processes, and what you're going to just draw a black box around and say, “I'm going to declare that this telecom switching network is a black box that has these 10 inputs and these 12 outputs, and it generates output from inputs in a predictable way.”

Obviously that initial step of deciding what you're going to treat as a black box, thereby throwing away all information about what happens inside the black box – now that's going to determine the quality of your model. If you’ve fundamentally got the black boxes analyzed wrongly, then however clever you are at analyzing the connections between them, you've fundamentally got the whole thing wrong. 

To an extent that's just natural as part of modeling, and it's good to have that step put out in the open because a lot of the time, particularly in the economic sphere that I'm more familiar with, people just take, for example, a set of U.S. GAAP accounts as if they were objective facts about the world – whereas, in fact, that's simply the equivalence of someone taking a bunch of black boxes and saying, “This is going to count as earnings; this is going to count as revenue; this is going to count as an asset;” assigning things to times and then handing them out. 

If you don't ever realize that there's been a process of censoring that information, tabulating it and combining it before it ever reaches you, you can go down some merry old rabbit holes – same with government economic statistics. 

And so Stafford Beer was very keen on saying you don't gain anything by opening up a black box. If you can model the black box reliably in terms of its inputs and outputs, then that's probably where to stop. 

If you start diving into the innards of it, then firstly, you're probably stepping on someone else's job and you're trying to interfere with things that you're never going to understand well enough to take part in, and second, it's going to interfere with your understanding of the system, at the level of being a system.

Patrick: I think I would have a polite disagreement with that gentleman, since part of my job has been opening up black boxes and seeing what horrors they contain for many years, partly it's explaining them to other people. 

AI and the future of system management

Patrick: One of the memes that has been floating around the last while: with the new ways we've finally not quite achieved the dreams of the 1950s, but made substantial progress on AI tools that actually work in the same world we live in, people are wondering, “Is this the end of programming as a field? Is this the end of accounting as a field?" 

Even if there were a magic wand that we could wave that would successfully transfer all the thoughts that we had down to executable computer code with no errors in it, there would still, I think, be a huge necessity for people to go out and model the world and make contextually appropriate choices for what information we decide is the core focus of our models and what can be appropriately discarded in a given context.

We have existence proofs for ChatGPT and Claude being able to do relatively simple programs very, very quickly relative to even skilled engineers. I don't think we have existence proofs for them being good at modeling truly complex systems yet, and it might be years until we get there.

[Patrick notes: I am aware that you can easily get an LLM to riff with you on how to model airplane tickets. I am saying that you cannot get an LLM to usefully model airplane tickets the way that Delta has to model airplane tickets. The job of engineers is not to merely sound intelligent when discussing why the international transit network is a hard problem, it is to model the world in sufficient fidelity to be able to accomplish physical results within it.] 

Dan: And of course, in those many years, everything changes. 

It's easy for someone like me glibly to say that “you've got to manage the bandwidth. You've got to match the bandwidth of the management capability to the entropy of the system” – but systems drift, and systems produce things suddenly which are way outside what you thought their general range of entropy was. That's the distinction, I think, between just a sort of stable system and what they call a viable system. 

A viable system is one that can re-engineer itself to respond to shocks that were not anticipated at the time of its design. If human beings have that property, I would say it’s related to (but not exactly the same as) Nassim Taleb's concept of anti-fragility – it’s this idea that something which is set up simply as a model in the sense of a rule will always end up collapsing because something will always come that wasn't anticipated by the original designers. 

There's another kind of fundamental theorem in this branch of applied mathematics called the good regulator theorem, which says that every regulatory system, every rulebook is implicitly a model of the system. 

The example I have in the book: If you have a sign on the wall saying that staff do not have access to the combination for the safe, that's implicitly a model of the kind of people who might walk in off the street asking for the combination to the safe. If you have a sign up on the wall saying “no refunds, no exceptions,” that's a model of the legal situation with respect to having to give refunds. 

Over time, there are going to be cases that come in which are outside that model, and when the model fails, then either the system will fail because it's completely bound to a rule, which is bound to a model which did not work, or there has to be some way of changing the system to incorporate something that was outside the model. 

And that's, to my mind, something that ChatGPT, large language models, kind of recurrent transformer neural networks seem to me to be quite badly suited to doing because they've got this kind of averaging property over them – you know, here we are really talking about metaphor rather than math, but they're digging through and averaging over past outcomes and past responses. When something actually brand new happens, maybe it's hard to predict how they're going to respond to that, but it doesn't feel like they're going to be able to respond to it in the way that a human would, or that a long-lived organization or other kind of regulatory system would, by reorganizing themselves to cope with something that’s completely outside their previous information environment.

Patrick: I think one of the points you made in Lying for Money (which we'll address in a bit more detail later) – it’s that any system of controls you have for an organization, a corporation, and similar has to be robust against determined adversaries who are capable of examining the source code, as it were, of that organization, and then attempting to get it to perform in a manner opposite to its stated values.

Basically any system for moving money around is a massive target for determined adversaries attempting to make it move all the money to them, and there are ways to make systems better at moving money around, which perform pathologically if and only if a determined adversary then jackpots them for all the money.

But we'll get back to financial fraud in a moment. One concept that came up in The Unaccountability Machine is this notion of accountability sinks. Can you just explore that for a moment for the audience?

Dan: An accountability sink – and this was the original kind of idea I had, which grew into the whole book – was that in the modern world, industrialized decision making happens, and because decisions are taken through industrial processes, people don't feel like they are responsible for them.

An individual manager will not feel like they can reverse a decision that's taken in accordance with the procedures, or in accordance with the employee handbook, or, you know, is the output of a literal algorithm that they're running on their computer, on their desk – and if you can't change something, it's really difficult to be accountable for it.

Being held accountable for things that you can't actually do anything about is one of the most psychologically horrible things that there is. I argue later on in the book that it's the main source of the kind of job stress that really does, with empirical evidence, lead to serious health problems over time. It’s not unnatural that when people find themselves in that situation, they try to create management structures which divert that accountability.

What the accountability sink is: firstly, it's an industrialized decision-making process that takes a decision so that an individual human being doesn't have to – there are many reasons why you might do that, lots of them actually, very good reasons, like you want consistency and efficiency over your decision making – but then you add to that another system which severs the feedback link and makes it impossible to appeal against the results of that decision.

That way you've created something which stops negative feedback coming back to any individuals in the system – which is a great stress reduction tool for them, but you've also broken one of the vital communication channels of the whole machine. In particular, going back to what we were talking about five or 10 minutes ago, you've cut out the communication channel that will tell higher levels of the system that something unusual has happened and that they've got to change things. 

As a result, as the accountability sinks build up, as more things become deterministic results of the algorithm or the procedure, the whole system gets stiffer and more fragile, in an engineering sense, and you're just gradually building up the kind of rigidity that's going to mean that, someday, the world's going to drift outside that system’s model of the universe and it's going to start reaching outcomes that it can't handle and falling apart.

Patrick: I think the notion of severing the feedback loop is one, you know, “are we talking about math or are we talking about metaphor?” I think most people might assume, “Oh, this is metaphorically that the feedback loop gets severed.” But the thing is that an actual failure case that we see over and over again in complex systems is like, no, literally, there is no one monitoring that inbox anymore.

To pick a particular example that people would be able to Google, there was a Free Application for Federal Student Aid – FAFSA. It's a United States institution that is involved in helping people pay for college. There was a particular procedure that needed to get done for the FAFSA where there was an edge case and they said, “ah, for this edge case, you need to write into a help center for FAFSA and then the help center will make a determination and tell you how to proceed” – and 70,000 emails ended up in an inbox that went to nowhere and stuck there for a year and no one noticed that they weren't getting the emails because the inbox was to nobody. 

I don't want to throw the federal government under the bus for that one… well, actually I do, because that was a thing that happened and negatively impacted tens of thousands of people. But we see that pattern so frequently partly because I think organizational cultures often devalue even the notion of having a help desk as something that says like, “okay, these are the hourly employees at the lowest rung of the totem pole who will be dealing with the unwashed masses that are doing business with us, while our decision makers up here do not have to deal with that nonsense.”

So there's already a tremendous gap. And then like the specific sort of, I don't know if you would call it “post-sale support,” is often not impacting the core metrics that the business or government organization thinks that are top of mind and brought up in the weekly meeting with the people that matter.

So those tend to naturally atrophy – and then once they have completely atrophied, it becomes impossible to discover that fact unless you have some sort of non-blessed pathway in the system to let it know that it's there. The system is now totally out of control.

Dan: Absolutely. This is an example of how these things need to evolve. 

Back in the day, if you read a great book like Structure and Strategy by Alfred Chandler, the split between line management and staff management, or line officers and staff officers in the military, or operation staff and policy staff in government – like every terrible idea, it probably seemed like a good idea to someone at the time. It was a way of optimizing management capability. It was a way of allowing scope for dividing and conquering, gaining the benefits of division of labor. 

But once you've made that split between line and staff, you've created two classes of people who aren't in the habit of talking to each other. You’ve usually created a few rivalries, internal turf wars. Anyone who's had a job or worked in a big organization knows that there's certain people that only some people in your division are allowed to talk to. Anytime you've got someone saying, “I want all of this to go through me,” that's where you're seeing a broken communication link within the organization.

That's always going to lead to problems. But while it always leads to problems, it doesn't necessarily mean problems for the people who created that system. A lot of the time, as with post-sales support, this is a solution for the people making the decisions, because this system was specifically set up to protect them from being bothered by customers and by, you know, other horrible people that were always angry and complaining about things. 

They’re never going to change that because it’s not a problem for them. It's a solution to that. 

Optimism about future of organizational design

Patrick: What is the optimistic upshot to this book? We've gone through cybernetics, we understand that systems are more complicated than we give them credit for, and that they tend to pathological behavior over time. For those of us who are actually in charge of building systems in the world, what should we do differently?

Dan: I mean, I am fundamentally optimistic and I'm surprised that so many people who've reviewed the book think that it's really pessimistic. I do kind of think that we're genuinely going through a crisis at present, and that that crisis is that the world has grown too complicated for our existing institutions to handle.

[Patrick notes: I find this a deeply unsatisfying core explanation for the failure of institutions in the modern age, because they seem to fail at simple things within their core competencies. The U.S. successfully prosecuted World War II. This required moving boxes of ammunition to small islands in the Pacific. It had a mostly adequate system of records attached to very long, complicated supply chains that ensured that people did not run out of ammunition. The same U.S. entrusted vials of the covid vaccine to the most efficient logistics system the world had ever built then lost them immediately, had no plan for fixing this state of affairs, and appeared to be mostly unbothered by this fact.

If we were to summon the shades of our ancestors and ask them for advice, do you think they would say “Ugh, hundreds of thousands of individual items over tens of thousands of individual locations? That’s just too complex for me to understand.” No, I think, with very high probability, that some junior officer in logistics would have had a working operational plan written within two hours.]

Dan: But we've been through numerous such crises since the Second World War and the dawn of really big companies and really big governments – that is the nature of these things. If you look in any management textbook that's worth the paper it's printed on, they all have the same basic message because they're all describing the same basic reality: organizations grow and they get more complex as they grow. 

Without wanting to kind of cheat on the math-metaphor distinction, I would argue that they grow in a sense exponentially – as Fred Brooks pointed out, the number of connections grows as some power of the number of nodes. Every time you're adding more things, you are adding a multiplicative number of connections between things. 

Organizations deal with this first by throwing manpower at the problem, sometimes by taking advantage of technological improvements to get a step jump in the kind of information productivity of individual management. But after a while, the distinction between linear and exponential growth will not be easily defied, and it becomes a crisis – at which point they reorganize.

Structure and Strategy by Chandler was written in the 1960s about DuPont and General Electric since the 1940s, and they’d been through two or three major reorganizations since then. It was the development of the multidivisional corporation, which was already on the way out by the time Chandler published.

We've had conglomerates, we've had deconglomerates, we've had outsourcing, we've had leveraged buyouts. All of these things were solutions to a previous crisis of complexity.

Maybe this is the first time in human history that humanity has reached a level of complexity that it can't do anything with, but I wouldn't think that's the way to bet, particularly since this crisis of complexity is coming along at the same time as what looks like a very strong general-purpose technology for dealing with complexity. 

The one thing that we do know that language models and transformer neural networks can do is that they can reduce information to a summary and they can blow it up from a summary, in ways that really –  for the first six months of anyone playing with this thing – really do look like magic. I think that what we have to do is take advantage of that technology and respect it, take it seriously.

There was a wonderful joke that Stafford Beer made about the first mainframe computers, which were being installed in industry across the sixties and seventies. He said that he worked with many of these projects, and companies were just taking mainframe computers and using them to speed up their existing financial reporting and cost accounting.

He said it was as if these companies had been given the opportunity to hire Einstein, Newton, and Galileo, and they put them to work memorizing the phone books so they could make phone calls quicker – it was that level of wastage of a new resource. 

I think we have to take the potential of “artificial intelligence,” so called, seriously. Most of these algorithms are things that have been around for a very long time. The fundamental math of the transformer neural network is 1950s. It's just 1950s math that no one in the 50s, 60s, 70s, 80s, 90s, 90s, 00s or 10s ever thought that you'd ever be actual to able to implement on silicon and now we can – and yeah, I think that's a great opportunity. That’s my reason for being optimistic at the moment.

Patrick: My compound reasons to be optimistic: One, we have existence-proofs of dealing with some of the issues that we're dealing with. Those of us who are students of history and/or read the newspaper can realize that we have historical examples and contemporary examples of organizations that don't seem to be as adversely affected by complexity as some organizations do currently.

In the 2020-21 rollout of the vaccine in the United States, for example, I often thought the people who did the flu vaccine back in the 1910s with vastly worse telecommunications technology, no computing systems worthy of the name, et cetera, et cetera, would look at our vaccine rollout and think, “How did you regress so much with regards to your ability to do relatively simple coordination of people?” I think that is something we can rediscover.

[Patrick notes: We primarily used wide scale non-pharmaceutical interventions to respond to the flu pandemic, not vaccines.] 

Empowering employees: the CEO’s open door policy

Patrick: Then, you know, tactical things: well before Slack existed and anyone could just DM the CEO and get his opinion on things, I worked in the order entry operation for a large American retailer of office supplies. They sell pens, paper, and other staples.

[Patrick notes: Some people consider this affectation of mine a bit much, and to you I plead cultural difference.

A Japanese salaryman is by custom allowed to share a reasonable amount of detail about his prior employment, including doing so in a fashion which makes it utterly unambiguous who that employer was, but the fig leaf is important. It is a mutual acknowledgement of sacred values (loyalty), a playful wink at one’s conversational partner that implies you have sufficient regard for them that the reference is as transparent as naming directly would have been, and an intentional performance of class and culture. Both salarymen, in the dance, appreciate that they are dancing with another salaryman, and therefore make confident predictions about what the future holds, far afield from the current conversation.

Which is not at all transparent if one comes from a culture with different shibboleths, and thus this parenthetical note.]

Patrick: On their first day, they gave everyone the CEO's phone number and explained: “No one has ever been fired at this company for calling the CEO. You'll probably never need to do it. But if you do, place the call versus not placing the call.”

I think that was powerful for a number of reasons – one, for demonstrating to employees that the company actually respected them and wanted them to use their best judgment, and two, for actually surfacing problems in a way that didn't do, as you were discussing earlier, that sort of psychologically damaging notion, saying “You have absolutely no agency within the system. You are implementing a business process that has been designed by your betters, and then when a negative outcome is associated with that business process, you wash your hands of it and move on immediately to the next ticket.”

Everyone at the company understood: no matter what is happening to me right now, there's always one option that is not on the sheet of paper in front of me, and that option is pick up a phone to the CEO and say, “boss, we've got a problem.”

Maintaining the space for that is I think much more important than people believe it is. I think that's probably one of the secrets for startup success, and also why a few very large enterprises that have extremely powerful CEOs, who have a great deal of cultural power and willingness to dig into the weeds, continue outperforming. 

Apple is a classic example; the Elon Musk industrial complex is probably the other big one, but there's a number of companies like that in the tech sector.

Dan: Couldn't agree more.

Lying for Money

Patrick: So, switching gears for a moment. The book that put you on my radar – my favorite nonfiction book of recent decades – was Lying for Money. It was extremely relevant to my interest when I was working in the financial industry, because all sorts of people try to tell lies for money far, far more than civilians would appreciate, and at a much higher rate and a much sort of wider segment of the population, but – be that as it may, why don't you kick it off with where did that book come from?

Dan: That book came from having recently finished my career as an investment banker. I was an equity analyst covering the banking sector. 

It just kind of struck me, as I was sort of looking around for something to do, how little the real world of financial services has to do with the kind of friction-free world of financial services as it's modeled You look in economics textbooks, or talk to people at central banks, or talk to economists about the financial industry, and half the time they're assuming perfect information.

You know better than I do, Patrick: the reality of financial services is a constant struggle to maintain adequate information, let alone perfect. In particular, the case that always sticks in my mind (because I was in the office at the Bank of England, next door to the guy who ended up being blamed for it as the bank supervisor) was the Collapse of Barings Bank in, I think 1995, although it might have been ’96. This was a classic rogue trader: this guy called Nick Leeson was in charge of the Singapore office, and he built up huge futures and options positions on Japanese equity indices which went entirely the wrong way. Then the Kobe earthquake happened and the bank was driven bankrupt overnight. 

There was a lot of work done on this, including by the economics department, including by me. My one publication in an economics journal – and I think I got two citations for it, so, you know, no one can question that I'm a serious economist (chuckles) – but I explained this with absolutely orthodox economic theory, saying that what you had here was someone who was unable to be monitored. It was a classic principal-agent problem: he got all of the upside from the bonus from profits, but didn't bear any of the downside from losses.

Consequently, it was inevitable that someone in this situation would take huge speculative positions and eventually will blow up. I was pretty pleased with that, and my boss was pretty pleased with that as a base of analysis. 

Then looking back on it with this perspective of 10 years, I reread that and I thought, “My God, I did not get a single thing right here.”

At the end of the day, if you're relying on incentive compatibility for a guy who's got limitless authority to pledge your firm's credits and you're not able to monitor their positions, that's your problem right there – not being able to monitor those positions. That's a big problem.

The big problem was that he was sending completely false P&L reports by fax to London and people were funding them with borrowed US dollars without questioning them. 

I started thinking, well, how would you put something like that into an economic model? How would you put into it, you know, “this is what the top management thinks is happening, but in fact, it's completely untrue because someone, for reasons that are still not very well understood, decided that he was going to fight the market and that he needed to tell lies to have to happen?”

I ended up thinking, well, the answer is you can’t. And you’ve got huge parts of the economy that are based on people trying to decide what they're going to check up on, and people are trying to decide what lies they're going to tell, you know: white lies, black lies, constructive summaries, and all of the other little things that make the world go round. It's a huge part of our economic life, but no one ever really talks about it, because economics just studies the economy as it's working well. It's like trying to discover the structure of the brain by only studying healthy patients.

If you want to understand the structure of the brain, you want to study people who've had odd injuries and find out what goes wrong. So I thought, well, let's look around and let's kind of see what other frauds there have been throughout history – and maybe this can teach us a little bit about the general kind of concept of what we choose to check up on and what we choose not to check up on.

Psychology of fraudsters

Patrick: Can I give you an example of two things that I really got from the book? (And disclaimer, I worked at Stripe for a number of years. Stripe does not necessarily endorse what I say in my personal spaces.) 

I will say you made Stripe many millions of dollars from one throughline of the book, which shows up in the Barings case as well, where people who have a great deal of facility with understanding the physical reality of transactions and how those transactions are controlled. In the case of the Barings trader, it was that this person had previously worked in the back office, and so was extremely comfortable with understanding controls around the back office, and was also the sole person responsible for doing that down in Singapore at the time for inexplicable reasons – that puts someone in a position where they can do much more damage, especially when controls around the back office are implemented in a high-trust environment. 

I can't exactly tell you an example of that affecting a particular customer, but I've got one seared in my memory – and oh goodness, if you take nothing from that book other than, “when someone has a huge amount of interest in the physical reality of transaction processing and controls around transaction processing, file that away as a thing that you need to think about for five seconds more than you would otherwise,” you will more than pay for the price of that book.

[Patrick notes: The discourse on control frauds is very good, too. That is a pattern which very much exists in the world but which is not in the formal education of many people.] 

Patrick: The other one is – and I have a special interest in reading ethnographies about fraud, and we see this over and over and over again – that to make it sound a little bit trite, oftentimes the first person the fraudster successfully deceives is themselves. I think that we want to, like, homo economicus model fraudsters as well: incentive compatibility, they are doing it for some combination of self aggrandizement and taking unprincipled risks. Then they're going to, in a case where those unprincipled risks work out, end up being billionaires is our model. 

In reality, we often actually see people have this almost insane belief that this thing, which in physical reality cannot possibly work, is actually working. Sometimes they maintain this belief in a superposition of lying about the thing, but they often seem like they really believe it on some level.

Dan: The trouble is that, as you say, you have to slip into this sort of thing – because the thing about fraud is that it's a job for life. You know, if you're a drug dealer, then you can just stop dealing drugs; if you're a bank robber, you can just stop robbing banks. But if you're an embezzler, Then you have to keep on concealing the existence of the actual crime.

There’s a really good couple of lines in the autobiography of the guy who was the accountant for the Kray twins, who were British gangsters of the 1960s. (Apparently no one's heard of them in America, so the US publisher told me to put something else in the book.) 

[Patrick notes: I don’t know if I count as American for the purposes of being a barometer reading of what an educated American would know, but I had no idea who these people were prior to reading the book.]

This guy was an accountant for a crime syndicate. He wrote an autobiography, and it's just an incredibly interesting book – it even contains two template letters for if you want to start up a long firm fraud, which I'm surprised that they allowed him to print.

[Patrick notes: A “long firm” is a fraud against a supplier, in which a fictitious enterprise or real enterprise which has been puppeted gets trade credit from a supplier and uses it to induce that supplier to send it goods. The fraudsters then sell the goods while ducking the invoices from the supplier. Eventually, the supplier discovers that they have no goods, no money, and that their purported customer—perhaps one whose offices they visited and whose references they checked—does not exist.]

Dan: He pointed out that the weird thing about fraud as a crime is that, if you see a house that's been broken into and you're investigating it, you know what the crime is; the mystery is just who did it. But usually with respect to a fraud, as soon as you've recognized that there's crime being committed, it's pretty obvious who it was that committed it. So rather than kind of concealing your connection, you have to conceal the existence of the crime itself, and that's because it includes this time dimension. 

If I steal five BMWs by hotwiring the ignition or, you know, whatever you do with electronic starters these days, the owners are going to notice them gone the next morning. The police are going to be investigating the crime. 

If I steal five BMWs by buying them on 90 day terms from an auto distributor and selling, putting them in the yard of my dealership, then for 90 days, the person who I've stolen those cars from is going to look in my yard, see his stolen property and think things are going pretty well, you know, and then I sell them for cash and disappear.

[Patrick notes: This is a worked example of a long firm fraud. And, yes, people very literally do that.]

Dan: As you say, as the fraud is hidden, it tends to grow because you're keeping two sets of books, effectively – the real set of books and the fraudulent set of books that you show to the world. There's a gap between those, and that gap is going to grow at a constant rate because, normally, it's the nature of businesses to either continue growing or to kind of shrink – but you can't allow it to shrink because you're taking cash out, and if it shrinks, then you're going to have to find some of that cash in order to pay the bills. 

What tends to happen is that the fake books grow at a much more rapid rate than the real books; the fraudster extracts cash from the business, which widens the wedge ever more, and so you end up in a situation whereby the actual amount of money that you've managed to steal as a fraudster is quite small compared to the amount of fraudulent money that you've allowed to go out into the whole economy. 

The classic example of this, of course, is Bernie Madoff, who, you know, was a big thief – but nobody can really put the amount of money that Bernie Madoff diverted to his own uses at more than a few hundreds of millions of US dollars, whereas the amount of money that his victims believed themselves to have lost was in the tens of billions.

Patrick: Yep. Much of the ill-gotten gains never really existed. They simply existed on paper being the consequences to the business model, if it were hypothetically run as described.

I think we hear echoes of this in cryptocurrency all the time – for example, in the, I don't know what exactly to call it, the FTX umbrella organization and the multifaceted frauds they were deploying.

Dan: The FTX bankruptcy claims are just hilarious because, of course, you know, I kind of have a twisted kind of sympathy for people because basically, they handed over their bitcoins, most of their bitcoins were stolen, some of them weren’t – or, well “stolen,” you know, not to be taken as me accusing anyone of any specific crime, but they should have been there and they weren't – and the bankruptcy judge ended up saying, “I'm going to be able to give you the entire US dollar value of your claims at 100 cents in the dollar.” 

The reason for that is that it happened that it all fell apart at a local low in the bitcoin price, and the remaining bitcoins more than tripled in value in order to make up the loss.

[Patrick notes: A minor clarification here, since the mechanics are very important: the FTX collapse did not merely happen at a local low. Rumors of FTX’s weakness caused the local low. Bitcoin went down by ~20% in two days following CZ pulling out of a rescue deal for FTX, which itself followed CZ’s leak of their balance sheet to CoinDesk in early November 2022. Then the bankruptcy was filed on November 11th. That filing dollarized all claims.

Note that, very relevantly to creditors here, so-called Samcoins were hit much harder than Bitcoin was. Solana, very relevantly, was down about 50%.

] 

Dan: So on the one hand, the bankruptcy administrator is saying, “pay back all the creditors at 100 cents in the dollar.”  Sam Bankman-Fried was, at the time of his pleading against sentence, saying “no harm, no foul, everyone got their money back, 100 cents in the dollar” – and of course, all of the creditors were going, you know, “what about 100 satoshis in the Bitcoin?”

Because the reason they got 100 cents in the dollar is that the value of their claims were frozen, and they were unable to get their Bitcoins out of this exchange – so they missed out on a huge amount of fictitious wealth. Now, in a world in which I was more important than I am in the justice system, I might certainly have told those creditors, “You've got a Bitcoin, it's actually worth zero – be thankful that you're getting even one cent in the dollar. Go away, and stop bothering me about this fictitious value.” But I can kind of see that point, you know – there's a genuine sort of psychic wealth there. Not everyone could have spent that all at the same time, but some of them could have bought a yacht with it if they'd made different decisions.

[Patrick notes: Claims against FTX traded as low as three cents in the immediate aftermath of the bankruptcy filing, and months later were going for twenty cents. John Ray III explains persuasively that diligent execution by his (expensive) team of professionals on behalf of the bankruptcy estate accounts for much of the difference. I’d also mention good fortune in investment selection (Anthropic) and good crypto weather.]

Dan: J. K. Galbraith calls it the bezel. The bezel is the increment to national wealth during that period when the fraudster knows that he has stolen the money, but the victim is not yet aware that they've lost it. And that bezel can be a significant amount. It can be a large proportion of even a quite developed economy.

Patrick: I think another thing that we see frequently in narratives of fraud in your book as well is that fraudsters, often for operational reasons or to avoid people defecting against the fraud, will involve their victims in the instrumentalities of the fraud – sort of dirty their hands semi-intentionally. So one thing which advocates for the FTX creditors don't exactly make a huge deal of is that, when Sam Bankman-Fried was doing bank fraud, often the victims were doing the bank fraud too – they just thought that they weren't the victims of the fraud, it was the banks and or government.

[Patrick notes: It is considered impolite to accuse someone of a crime. And so I offer crypto hedge funds and other sophisticated individuals, who wired money to either Alameda Research or to North Dimension, a choice: you can tell your LPs that you’re stunningly incompetent at routine banking operations, accidentally wiring money to a company which is either plainly a direct competitor (Alameda) or a money laundering front whose website said it was an electronics retailer (North Dimension), or, you were aware of bank fraud, actively participated in it, and didn’t think it targeted you.

There is no option C.]  

Dan: Oh, absolutely.

Patrick: A thing FTX did as a standard business practice was to tell people, “so because the banks are unfair and won't give us accounts in the name of FTX, you're going to have to wire money to this shell corporation and lie about it to your banks and our banks. Are you cool with that?” And everyone was like, if that's what I need to do to make money, it sounds great. North Dimension? Selling fictitious electronics? Sure, I will send them a wire for a hundred million dollars.”

Dan: Yeah – same thing with Bernie Madoff. People pointed out, “This looks like a fraud, because you can't possibly declare that level of smooth, positive above-market returns.” And lots of the kind of savvy people in the fund of hedge funds market who were encouraging people to invest with Madoff were saying, “Don't worry, he makes this money by frontrunning the retail clients of his brokerage firm.” So, you know, that's all great. And everyone again went, “Oh, brilliant – a frontrunning scam and I can take part in it, have my money!” 

When someone does that, it just makes you feel a little less sorry for them, does it not?

[Patrick notes: While I have difficulty pointing to a specific fund-of-funds making that case for Madoff, there was an industry whistleblower with a contemporaneously recorded view. The World’s Largest Hedge Fund Is A Fraud (a memo to the SEC which the SEC spectacularly failed to follow up on, and also one of the best titled memos ever), said “I have outlined a detailed set of Red Flags that make me very suspicious that Bernie Madoff’s returns aren’t real and, if they are real, then they would almost certainly have to be generated by front-running customer order flow from the broker-dealer arm of Madoff Investment Securities, LLC.”] 

Patrick: Yeah. I think it exists in some tension with the sort of received dicta in our social class that you don't blame the victim for being victimized in a crime – and yet in fraud, it is a repeated tactic of fraudsters to involve the the victim in a way where they can no longer be morally blameless, but be that as it may.

Fraudogenic environments

Patrick: Another thing that I liked about the book was your description of fraudogenic environments, where essentially you can have arbitrarily-scaled frauds without that fraud having a central actor pulling the strings. Can you chat a little bit more about that?

Dan: Yeah. This is one of the other strands that ended up leading into the second book. It’s just a strange thing, where systems make decisions or systems not structurally lead to outcomes, which you can't pin on any individual person kind of at the center of the thing.

In the English edition of the book, I talked about what we call the pension mis-selling and mortgage mis-selling scandals, but for US listeners of the podcast, think about Wells Fargo bank and the phantom account scandal, because it's got exactly the same mechanics. 

There were people at both banks who committed illegal acts, produced false documents, and victimized customers. Why aren't they in jail? Because it seems absolutely pointless to put some poor harassed counter staff in jail for having just kind of pushed something a little bit wrong for the customer that they were dealing with.

[Patrick notes: In one of these two the more frequent mechanism was not “pushing something” but rather creating accounts without the knowledge of the customer. Many less sophisticated observers of the financial industry, such as journalists and regulators, believed there was a great theft happening in the creation of those accounts. In most cases, it was simply ticking a box that activated online banking for someone or ordered them a debit card.] 

Dan: If you look at the people who you really want to put in jail, the people right at the top of this organization, then you might think, “I really want to see that guy in prison, but what did he do that was wrong?” You know, all he did was set ambitious targets for his staff and try to deliver the revenue numbers that Wall Street wanted to see. And that's what we want CEOs to do.

No one really noticed that we'd developed an environment in which you set unrealistic targets, you inform your low level staff that there are really bad consequences for not making those unrealistic targets – and, well, if you constantly demand the impossible, then what you will get is the unethical.

It interests me – the former banking regulator in me wants to say, “they may not have realized it, but they thundering well ought to have done, and they ought to have created a compliance environment in which that did not happen.” But that very much was not the spirit of the age – spending loads of money on processes which don't directly generate revenue, which in many ways prevents or makes cross-selling more difficult. That was not what people thought that good bankers did in those days. And it's really easy for an organization to fool itself, you know, like HBOS in the UK or like Wells Fargo in America, that what it's doing is really clever and it's increasing revenue per customer, when what it's actually doing is building up a huge mis-selling and operational risk liability for itself five or 10 years down the line.

[Patrick notes: It feels very important to me, and underremarked upon, that all sales-focused organizations track performance closely, push management of individual sales reps to line managers who are far-removed from top management, and manage their results statistically and stochastically. And yet we don’t see this result every day. Yet nobody tells the story of “What Bank of Bigness gets right about debit card activation that results in its personal bankers, who make shoe store money, not improperly activating unasked-for debit cards to make their quotas.” That is probably more important to society than the fraud was.]

Dan: The other example I use in the second book – but it could have gone into Lying for Money if it had happened then – is with Boeing.

If you've cut off all channels of information from the shop floor to the boardroom other than financial reporting, then it looks like a great idea to outsource manufacturing. It looks like a great idea to outsource the software developments for your stall sensors because you're saving money, and you can report your incremental saving of however many single digit millions of dollars – without realizing that what you're doing is something that will (A) kill people, and (B) cost vastly more money than it's ever possibly saved when things fall apart.

A guy was very influential on me on this, and I think you'd love his book if you haven't already read it – a guy called Malcolm Sparrow. His book (License to Steal) about Medicare is just amazing. It’s about the Medicare system in the 80s and 90s, the time when 30 percent of the program was fraud.

The thing I really took away from that book was him saying that “the way you cut costs successfully in a business is by doing things right, not by doing them cheap – because making a mistake and going back and correcting it always costs you 10 times as much as the couple of cents that you thought you were saving in the first place.”

Patrick: There have been a number of culture and counterculture currents in the United States or Japan – the two cultures I'm most familiar with – where we had a few decades of industry largely deciding to outsource software development, and then the tech industry went in a radically different direction and said, you know, “Engineers at Google are among the best paid W-2 employees in the world” (traditionally full time employees of a company) “and yet this is the cheapest way we know to develop software that works because we have high levels of confidence that we can actually deliver on our quality and similar targets.” That culture is only after a span of decades getting sort of re-exported to the “traditional economy” in the United States, to the financial sector, etc.

So it will be interesting to see where things shake out on how much software gets developed internally to banks in the coming couple of decades. One of the reasons I think that the physical reality of the artifacts that people interact with on their telephone has gotten good over the last couple of years is that the banks got extremely burned, as you might say, on that side of the pond – being caught on left foot on software development for a period of about a decade after the release of the iPhone, and then went to Silicon Valley, hired some teams, put them at the bank and said, “We want to win in mobile apps, make us win.” And then the teams set out (largely successfully) to do that.

Journey of becoming a published author

Patrick: Switching gears a little bit, another complex system that people probably under-appreciate is the mechanism of being a published author. How does one end up in your quirky situation in life?

Dan: Well, I almost kind of stumbled into it. I mean, I had been a banker for 15, 16 years, and in that time I'd done little bits of articles for trade press and for mainstream journalists, newspapers, when my bosses would allow me; I had a few friends in the media industry. I cowrote a book with my wife for a publisher that either went bust or got taken over very shortly afterwards, but it fell into the hands of someone who worked at a company called Profile Books in the UK, which is a great little independent publishing house. Then just one day he sent me an email and said, “I think you could write a better book than this. Come down to London and have lunch.” 

We talked a bit, threw around ideas, and that was when I said, “What I'd really like to do is write a book about fraud, about the history of financial fraud.” So that was kind of how I fell into it. 

I was very lucky because Profile is a great little old fashioned publishing house; they have editors and the editors are really good at – well, they're really good at quality control basically.

All three of the books that I've written with them, I've sent in a first draft, and always you sort of think, “Ooh, I hope it doesn't come back with lots of comments” – but in fact, when you're writing a book, you want it to come back with lots of comments, because if it's lots of comments, they're all individually going to be pretty minor, and you can just go through going, “Oh, yes, no, that sentence doesn't make sense. I can rewire that. I can change that adjective.” 

When you get one saying “this chapter doesn't work,” and you get a few comments like, “there's no structure here,” or “I don't understand why this goes this way”... for one thing, it's incredibly flattering to have someone read something that you wrote that closely – but also, these guys are such professionals, they probably read more books by Wednesday morning than I will in an entire year, and they know what makes things go forward. 

The biggest thing I learned is that you can write something down that you think is great, and it's just a collection of anecdotes. Turning those anecdotes into something with an actual narrative structure is a huge skill, and it always happens, for me at least, in the second draft.

Patrick: Yeah. I think one of the biggest leaps – and I've done millions upon millions of words of writing in my career – the biggest leap between the sort of long essay, long magazine piece that I typically write and writing a book is that need for structure and narrative that will sustain someone for several hundred pages.

Then hopefully years after reading the book, they will be able to pick out not just the anecdotes they've probably forgotten, but the underlying themes and mental models that you're attempting to leave them with – like the mental model of a fraudogenic environment, for example. 

So how long did it take you to write the book from that lunch in London?

[Patrick notes: This is not me doing career planning in real time but is not entirely not me doing career planning in real time.] 

Dan: For Lying for Money, I think that was 18 months, start to finish – that was basically a year to write it and then six months to rewrite it. 

Well, actually, I think that the version that you might have got in the USA benefited from another three months. When the US rights were bought by Scribner, I had a really good editor there who said, you know, “Look, we could just take the ‘u’ out of the word colour and change all the pounds to dollars, but actually this book kind of deserves rewrites of some chapters to use things that are more relevant to the USA.” I had things about, like, London gangsters, and I had a whole chapter about value-added tax, which doesn't even exist in the USA.

[Patrick notes: VAT carousel fraud, which is similar in character to Japanese gold smuggling, in that it results in the government coming to the mistaken impression that it owes the fraudster money. This results in the odd truth that the government commits money laundering (very effectively!) for the fraudster.

Gold, in Japan, has a tax asset embedded in it, equivalent to the consumption tax (which is also the duty on gold). When one sells gold, one pays the buyer for that tax asset in addition to the bullion. The tax asset is currently 10% of the price of gold. The crime goes: buy gold anywhere in the world ex-Japan, smuggle it into Japan, sell it in Japan, receive the 10% tax asset from the seller offsetting the 10% you didn’t pay when you bought/imported your gold. The seller will, in the ordinary course, net the cost of that tax asset against their own consumption taxes for the year. i.e. The Japanese state has now paid you money, through a middleman.

This was exploited at industrial scale but was also for years a socially viral phenomenon, since you can buy gold at a well-lit shop in a commercial district in Hawaii, sell it at home for more, and never stop to consider that one is now an amateur smuggler (and not simply a prudent gold investor taking advantage of an arbitrage opportunity caused by local conditions in Hawaiian gold markets). There are now very prominent signs in the airports, and an ongoing public education campaign, attempting to reduce the level of ambiguity here.] 

Dan: It was really nice to get a third go-around at some of the things which, reading it back when it was a printed thing in the UK, I was going, “oh, that doesn't work, I wish I'd done that differently.” So I got another three months worth to rewrite two or three chapters, and I took the advantage to pretty much completely rewrite the conclusion. He also allowed me to expand it by 10,000 words, so the US version is actually slightly longer than the UK version.

Patrick: One thing that I've been asked before: why did you decide to write a book? You know, you've done a number of interesting things in your career, and if you had known going into this that this was going to occupy your professional attention for upwards of a year and a half, why make that decision?

Dan: I mean, there's always that joke of like, “we do these things not because they are easy, but because we thought they would be easy.” The first one of these things I ever wrote, I thought it would be a lot easier than it was. 

The other thing actually that people underestimate with writing books is how difficult it is to hold that whole structure in your head at one time. It's an almost physical discomfort at trying to stretch your brain around an object that large, and I think it improves with practice. People who have done doctoral theses say to me that it's a fairly similar kind of discomfort. 

But yeah, I thought it would be easier than it would – then the reason I kept coming back after doing the first one is that you just meet really interesting people. Very few people in this life get rich from writing books, and certainly compared to consulting on financial regulation, which is my day job, it is not as lucrative – but you meet a lot more interesting people in the publishing world, and people just get in touch with you, they get in touch with you on Twitter or drop you an email and say, “Oh, I really like this book,” and you go, “I'll meet up for a coffee next time we're in London.” I've made some good friends, made some good business contacts that way.

So it's certainly not been a waste of time. It's been a great use of time in terms of continuing to meet people, continuing to keep mentally active (which is important at my age). Even in purely commercial terms, you meet interesting people in the same field as you – who wouldn't want to do that?

Patrick: I think it does work as an extremely high effort calling card. The barrier to entry of producing a high-quality calling card is high. You want it to be actually good when reading it – and again, Lying for Money, my favorite nonfiction book of probably several decades. So you can definitely pry interesting conversations out of the economy with that as your introduction – hopefully with people much more interesting than me.

Dan: Oh, no. I mean, actually thinking about it, most of the advantages I've talked about you can get by doing a really good substack – so you've probably already got them Patrick.

Patrick: Eh – we shall see. I think it's underexplored. 

I have a love-hate relationship with the way that Substack has, like, branded the thing that is “having a beat and publishing on it for money without being a full time journalist.” But be that as it may, there must exist something between a Substack and a traditionally published book which is sort of an encapsulation of someone's beat in a single artifact that is produced at a particular point in time, which is not a multi-year commitment.

I think the world and the economy and the internet urgently want that form factor to exist, and no one has successfully created it yet – but hopefully someone will, so I can make several of them.

Effective ways to sell books

Patrick: One more question on the business of being an author: What have you found is effective as an author for moving book sales?

[Patrick notes: This is not simply career planning. I’ll ask any business owner in any industry what works in marketing/sales if I don’t think they’ll be offended by the question. It often provides a fascinating lens on how their business operates. I’ll also ask about… well, everything, really, but early priorities would include the production function, hiring practices, and the waste stream.] 

Dan: Well, I started my own newsletter as a way of promoting the book. I don't know whether it works, but I've met as many interesting people through the newsletter as I have through the book, so that was a great thing to do. 

I wish I knew better what shifted books. I think to a large extent, it feels like the sort of thing that is a matter of self-reinforcing luck. If it gets into the hands of the right people, then some people really make connections, some people really recommend it if you've done something that they like.

If you've done something that you like and it falls into the hands of, you know, someone with a really big profile and they say so, then you really do notice a pickup in sales. I think that social media networks are useful, but actually mainstream media publicity seems to do more in terms of driving sales.

But then on the other hand, as I say, if you're in our kind of space in the nonfiction world and you're writing a book, then it is very likely that the way that you end up kind of being financially rewarded for writing that book is not going to be through your 12 percent royalty on the hardback price of every book sold – it's through meeting someone and doing something interesting with them who you wouldn't have otherwise met. 

I used to play in bands and someone always used to say to me, “If you've played all night, then if even one person in the audience really appreciates what you're doing then that's great – as long as he's the one that owns the venue.”

Patrick: Just for helping people who are outside of this ecosystem, I think that when people (publishers, etc.) try to set expectations with trade authors like yourself, potentially me if I was hypothetically having that conversation, they try to say like 2,000 copies is a good reach for a book for a first time author, for example.

(I imagine Lying for Money certainly has sold more than 2,000 at this point, because I think I've given out a hundred copies.) (Patrick laughs)

[Patrick notes: I certainly expect that a NYT mention probably sells more books than me, but when I tweet about dragon/banking crossovers, a neglected fantasy subgenre with two titans frolicking in a blue ocean, that apparently causes enough sales to warrant an email from the author.

You’re either the kind of person who, told to subvert AML controls on behalf of a dragon, starts thinking that is a fun intellectual challenge, or you are not. If you are, you will enjoy those books. (The Dagger and the Coin series is probably the most underrated fantasy series I am aware of.)] 

Patrick: Every book creates a small community around it, and some create a larger community, but then the interesting connections within that system in the shadow of a book are maybe more economically significant than the work itself.

Dan: I definitely think so. I came into this from the world of sell-side research, where I was used to the idea that I was writing things every week that would go to no more than 200 people – I've never had that idea that you need to be reaching tens of thousands of people to be doing anything valid.

Wrap

Patrick: Dan, where can people find you on the internet, aside from the books The Unaccountability Machine and Lying for Money?

Dan: Well, if people really want to find me… I mean, to be honest, I'll find them, I'm such a loud mouth these days. I am @dsquareddigest on the network known as Twitter to me and X to its owner, and the corresponding address on BlueSky, and probably on all of the other competitors that I don't really post very much on. My newsletter is called backofmind.substack.com, I think, whatever the kind of equivalence is.

Patrick:. I'll put links in the show notes for everybody. As always, I'm @patio11 on Twitter or patrick@ any of my domain names. Thanks very much for listening, everybody. Thanks very much for coming on the podcast, Dan.

Dan: Oh, thanks very much. It's been a real pleasure. It's great to meet you after all this time.

Patrick: Honored to meet you as well. All right. Have a good time, everyone. We'll see you around the Internets. Bye bye.

Dan: Bye.