Retrospective on 2024, with Sammy Cottrell
This week I sat down with Sammy Cottrell, my EA/producer and the person who makes Complex Systems actually happen every week, to examine the production function behind the podcast. We discuss theory and practice of good shows (including e.g. the importants of transcripts and how to curate guests), reflect on our subjectively favorite episodes, and explore how LLMs might transform both the systems we cover and how we write about them.
Sponsors
Vanta automates security compliance and builds trust, helping companies streamline ISO, SOC 2, and AI framework certifications. Learn more at https://vanta.com/complex
Support proven charities that deliver measurable results and learn how to maximize your charitable impact with GiveWell. Go to givewell.org (and type in "Complex Systems" at checkout).
Check is the leading payroll infrastructure provider and pioneer of embedded payroll. Check makes it easy for any SaaS platform to build a payroll business, and already powers 60+ popular platforms. Head to checkhq.com/complex and tell them patio11 sent you.
Timestamps
(01:04) Introducing Sammy
(01:37) The offscreen symphony behind Complex Systems
(07:19) Production supply chain and transcript philosophy
(08:40) LLMs of choice
(15:11) Sponsors: Vanta | Check
(21:21) Favorite episodes
(25:49) Sponsor: GiveWell
(27:15) Beyoncé
(28:47) More subjectively favorite episodes
(32:34) Thesis of Complex Systems
(34:07) Canon in operations research
(40:51) Sabbatical mode
(48:52) Bits about Money and other writing in 2024
(51:22) On paywalls and not implementing them
(54:29) LLM plans for 2025
[Patrick notes: As always, I have included commentary below, set out in this fashion. This transcript has been massaged by an LLM and is designed for readability rather than for being a court stenographer.
Complex Systems is a personal project of mine, and not everything said by myself or guests is endorsed by e.g. advertisers or other organizations I may be affiliated with.]
Transcript
Patrick McKenzie: Hi everybody, my name is Patrick McKenzie, better known as patio11 on the Internet, and I'm here with Sammy Cottrell.
Sammy Cottrell: Hey Patrick.
Patrick: One thing I really like about Tyler Cowen's podcast is that annually he does a retrospective on the year's events with his producer, Jeff Holmes. I often feel that the "off-camera" talent in media production doesn't get enough on-air recognition for their efforts. Are we a media production business? I guess we kind of are.
Sammy: We're a little bit in media production.
Patrick: We sort of cosplay as this on the Internet. [Patrick notes: This is partially tongue in cheek. While I don’t particularly feel like a professional commentator/researcher/writer in the field of financial services, there is no reasonable bright line for that that I don’t easily pass.] I wanted to recognize you for your work as the de facto producer of this podcast. Why don't you tell the world a little bit about yourself before we get into things?
Introducing Sammy
Sammy: I'm Sammy. I'm Patrick's EA. More generally, I do part-time chief of staff, general fixer, and executive assistant work for various people. I enjoy working on interesting projects with interesting people, and I particularly enjoy working with Patrick. I help produce Complex Systems, which these days mostly involves lining up guests and managing scheduling.
The offscreen symphony behind Complex Systems
Patrick: Like every infrastructure project, there's an offscreen symphony making things happen. For example, we're recording this from a pair of recording studios, and ensuring studio availability, microphone setup, and other logistics takes non-trivial work. We'll discuss the fruits of your labors in a moment.
I had this on my list for years—restarting a podcast after having one before. From 2012 through 2016, I had a podcast with my friend Keith Perhac where we produced 16 episodes over four years, which reflects how consistently I could release them. How many episodes has Complex Systems done?
Sammy: This will be episode 25.
Production supply chain and transcript philosophy
Patrick: Yes, 25 since July. To use another Tyler Cowen-ism, the production function seems to be working. Thank you for that. So we've been doing about one per week. How do we feel about it? Give me your impressions, and then I'll give you mine.
Sammy: I'm a little bit biased. I really enjoy this podcast. I like reading the episode transcripts, I like the guests that we have on—many of whom I helped choose, but I would have liked them anyway. We've managed to make an episode every week except Thanksgiving and Christmas, which was intentional.
Patrick: I feel somewhat passionate about transcripts. Many well-produced professional podcasts don't put much effort into transcripts for various reasons. They're seen as epiphenomena of audio work. They require significant effort to do right and don't fit neatly into the weekly workflow of podcast production. As a result, they're often neglected.
About 50% of my work on each podcast is reviewing the transcript, which is produced either by a human transcriptionist or by AI tools, then re-edited by me with inline notes added. [Patrick notes: Actually, if you’re counting strictly by minutes expended, this is substantially more than 50%, but if you’re counting by mental energy, 50% seems about right.]
Why is this important? I'm far more committed to the written word than to speaking as a format. I want people to be able to read things if they get more value from reading than listening. It's much more time-efficient for many busy people. Additionally, I often find myself doing better reflection on topics offline when I have time to think, versus simultaneously managing the mechanics of speaking while thinking about complex topics and reacting to guest comments. [Patrick notes: Although I don’t like to be defined by it, I have at times struggled with a minor speech impediment, and the physical mechanics of speech take me a bit more of my attention budget to juggle than the typical person. You’re most likely to hear it slip through years of effort when either I’m quite excited and speaking quickly or when I’m ill or tired and not able to summon the effort to police my speech like I do in every other conversation.]
The written format allows thoughts to breathe and produces a more cohesive finished product.
Sammy: I'd add that your particular manner of speaking, and sometimes writing, is often implicit. There's a lot of subtext. The nice thing about inline notes is that they explicitly spell out the subtext for those who might not parse it the same way as your intended sophisticated audience. Having the industry references spelled out with links to examples, rather than brief references, is really valuable.
Patrick: The podcast has also developed a diverse audience—though that doesn't quite capture it. We have listeners from various positions: tech, finance, government, policy roles, and beyond. Things that are common knowledge in some communities might not be worth explaining there. For instance, in banking, while many actually do need Regulation E explained to them, those with my particular confluence of interests don't need it explained every time. However, many Complex Systems listeners don't have Regulation E cached in their short-term memory, so I'll frequently include an explainer, particularly when using it as an example of larger regulatory frameworks.
[Patrick notes: Regulation E governs, most prominently, liability for fraudulent use of (many) consumer-oriented payments systems within the United States. It is, broadly, extremely customer-friendly, and shifts liability from consumers to financial institutions in (effectively) almost all cases. The financial and payments industries have complex business processes downstream of Reg E to re-shift that liability to e.g. businesses which take credit card payments. Reg E is very much not a dead letter. I used to have a hobby of writing banks on behalf of (mostly) unsophisticated consumers asserting their rights under it, and banks were indeed quite solicitous about following their obligations under the law… contingent on someone presenting as being “the sort of person who knows what Regulation E is” taking a keen interest in the affairs of a Kansan grandmother who had a routine dispute with their bank.]
Sammy: That's how I first learned terms like OpEx and CapEx—from Bits About Money, because you explained them outright with examples. Now they're fundamental terms in my vocabulary.
Patrick: You're welcome. By the way, Sammy is quite young—not saying that to embarrass you, but it reminds me of a great XKCD comic about how for any well-understood topic, 30,000 people are learning about it today. [Patrick notes: The original comic cites 10,000.]
Current technology doesn't make it possible to explain every possible allusion in a piece, though LLMs make that tantalizingly close. But with minimal effort, we can increase accessibility for those 30,000 daily learners of concepts like operating expenses versus capital expenses.
Sammy: It really helps.
Patrick: Continuing education is something we're all subscribed to for the rest of our lives, hopefully.
Regarding our supply chain: Complex Systems is produced by Turpentine, a media network focused on the tech industry and its adjacencies. They handle our production and advertising sales. If your company could use more exposure to an intelligent audience, email me for introductions.
The show is hosted on Ghost and Transistor—Ghost for the website, which I also use for Bits About Money, and Transistor.fm for podcast hosting, which is by far the best way to host a podcast. We record through Riverside, which makes the physical dynamics of recording much better than having guests set up their own recording tools.
LLMs of choice
We use a human transcriptionist and extensively use LLMs in post-production, mainly for rough-to-fine transcription editing. We've been experimenting with different LLMs. I know you prefer Claude. I just signed up for Claude this week, having previously used ChatGPT.
Sammy: Have you tested Claude yet?
Patrick: I just started using it and haven't used it "in anger" yet.
Sammy: Ah, gotcha.
Patrick: The technology works far better than we would have expected two years ago. Point it at an MP3 file, and you get a mostly accurate transcription. The failure modes are somewhat insane if you don't understand what LLMs are doing under the hood—sometimes they'll invent entire paragraphs that never happened, even when explicitly prompted to reduce hallucinations. It's a hilarious software paradigm where you have to tell your software in plain English, "Please invent less while cleaning up this transcription."
Sammy: You mentioned an incident where you didn't actually paste in the transcript, and it just invented a plausible podcast episode.
Patrick: Yes, and it fooled me for the first few paragraphs, which was mind-blowing. Then I realized, "No, this wasn't the first topic we discussed. I have distinct memory of the sequence, and it was not this sequence."
I don't want court transcript-style verbatim records. For those unfamiliar, court transcriptionists produce exact, word-by-word records for the judicial system. Instead, I want to present guests—and myself—in their best light, which sometimes means transforming complex, exploratory conversations into more structured thoughts, as if they had been written with an editing process.
When I ask AIs to help with this editing, they sometimes do well conveying the core message but remove people's unique phrases, speaking styles, and specific points. We're working to find the right combination of LLM, version, and prompting to preserve the best version of what was said without losing the speaker's personality and detail.
Sammy: Empirically, Claude does this well. I know you've had issues with ChatGPT's "sandblasting." A recent example: in an automated transcript, something was transcribed as "factory" when you actually said "Factorio," and Claude corrected it to Factorio based on context and knowing it was you.
Patrick: There are pros and cons to being recognizable to LLMs. For better or worse, large amounts of my written work are in various AI labs' datasets. I might still be used as a test case in some places—I don't know. Funny things happen when you're willing to say yes to things. (For context: "eval" is a term used by AI labs for testing their LLMs' functionality, through automated, semi-automated, and manual approaches, both during and after training.)
We should probably do an episode on LLMs at some point. So, what episodes were our most popular of the first 26?
Favorite episodes
Sammy: The Ricki Heicklen episode was most popular by a decent margin. It was also our first episode, so when people check out the podcast and scroll down to the beginning, that's the one they click on. It probably got the most promotion as our launch episode. Also, it was fantastic—Ricki is an incredible guest, and the conversation was excellent.
Patrick: And pip pip for you, you're totally responsible for making that happen. I wasn't aware of Ricki’s work before you introduced us, and she was indeed a fantastic guest. Going just by the numbers, what are our other top episodes?
Sammy: Based on downloads and transcript views, Casey Handmer, Dan Davies, and Zvi are all roughly tied for second place, in about that order.
Patrick: I have some thoughts about judging the nascent success of internet publishing operations after writing for the internet for many years. Sorting purely by engagement is suboptimal—actually, very suboptimal. It causes much of the dysfunction we see in prestige media, among other things.
The reality of running a podcast with guests of various profiles is that you'll tend to overperform your baseline with guests who have larger profiles or who promote their episodes through their own channels. You'll underperform with people who are less discovered by the Internet at large.
Part of the reason to have a show that invites people to discuss interesting subjects is to raise the stature of people who aren't yet Internet famous. Ricki, for example, is—relative to her expertise about trading—less well known to daily Matt Levine readers than Matt Levine himself. If Matt were hypothetically on the podcast, we'd break our download records.
Sammy: We gotta get him on the podcast, man.
Patrick: That will probably happen at some point, if Bloomberg allows it. People who are "native internet creatures" with large Twitter presences they can use to share their episodes will tend to outperform versus those who aren't.
If you allow yourself to be ruled by the numbers, you'll tend to over-select for people who've invested heavily in becoming famous in some subculture, and anti-select for people doing the actual work who have genuine expertise. As someone who has invested some effort over the years in cultivating internet subcultural renown, I definitely don't mean this as criticism. But I want to feature people like typical published authors—who aren't actually well known by many people—experts in their fields, and startup founders doing wild new things at the phase where they're still doing those wild new things, rather than unicorn founders everyone wants to associate with.
Sammy: That makes sense.
Patrick: This is an iterated game. Something observed across various venues and channels as they grow is a matching function between guests and programs, where the subjectively most desirable guests get sorted to the programs with the largest following.
If you're a U.S. presidential candidate—apparently there's this guy named Joe something who has a podcast. I had no idea who this person was prior to this year.
Sammy: Are you kidding? Wow.
Patrick: I've been in Japan for 20 years and haven't invested much in following mixed martial arts. The big update for me this year was that the United States has moved away from having three major television networks, and now apparently Joe Rogan is the primary platform for U.S. presidential candidates.
Sammy: You know what? That's fair.
Patrick: A presidential candidate won't go to someone with typical viewership of 100,000 because that doesn't move the needle.
Similarly, there's no single ladder for status or audience size, but at every rung on these multiple ladders, there's a matching function happening. As shows grow, they tend to elevate the center of gravity of their guest population, with increasing mutual professionalization.
I'm happy that we largely don't take calls from PR firms attempting to place guests, which is absolutely a thing that exists. A significant percentage of guests on very professionalized media outlets come through this route. It's extremely incentive-compatible for all parties to have producers and PR professionals—who are repeated players specialized in this game—negotiate between themselves, versus having a host and individual make these decisions directly.
Thankfully, we're not there yet. As someone who worked in communications, I have no objection in principle to the media-industrial complex, but it does cause some downstream effects in how media is produced and consumed.
So we've talked quantitatively—qualitatively speaking, what were your favorite episodes?
Sammy: The Ricki episode was amazing. As someone who really enjoys listening to Ricki discuss trading, I loved hearing both of you compare notes on your respective teaching opportunities. I really enjoyed how she immediately solved the Starfighter puzzle. That was impressive.
Patrick: That was wild. We should try to make it possible for more guests to demonstrate expertise live on air like that in an unplanned fashion. For those who didn't listen to the episode, we discussed comparative design in video game-adjacent experiences that model real-world trading.
She has produced one such game, and I produced one called Stockfighter. I gave her a verbal description of the final level, and she was silent for about 15 seconds before immediately solving it four ways. This level was very challenging for most players—of the roughly 60 people who completed it, most took many hours to explore its contours before finding solutions.
Sammy: That episode also had the quote, "on a level playing field, professionals destroy amateurs," which… you can see why.
Patrick: That quote is complicated—we could discuss it at length. Expertise is real, returns to expertise are real, and expertise is extremely unevenly distributed in the world. Some people have looked at our recent history and assumed the institutions are rotting from within, that no one who understands anything exists anymore.
That's abundantly untrue. While institutions are having repeated failures in many ways—something I reluctantly agree with, though one can oversell that thesis—experts definitely exist, and those are arbitrarily deep wells of expertise. One of my shadow goals for the show is to introduce the broader world to these experts. If you were hypothetically in the room with some of these people, you'd feel confident that there's actually a field here with real results, not just random number generators, and that deep intellectual work is being performed. So, what were your other favorites?
Sammy: The most recent episode with Yatharth was great. It was an unusual format idea—I'd met Yatharth once, and I really enjoy the questions he asks on Twitter. I got the idea because he frequently tags Patrick on Twitter with insightful questions about finance. I thought, "Clearly he should do this on the podcast," and he did, and it was great. I also really enjoyed hearing about your personal life on the podcast.
Patrick: I always feel somewhat reticent about discussing my personal life. Parasocial relationships are an unfortunate reality of doing work in this fashion and in the way that I do it. That's a deep topic to explore.
People are buying packages of goods from anyone they have an ongoing relationship with. Part of that package might be expertise in narrow areas. Part is some sense of good taste, for lack of a better word. I think part of the job you're hiring me to do in listening to the podcast is to continue identifying guests who have something interesting to say and drawing that out of them in an hour to 90 minutes. And part is this parasocial relationship thing where I play this patio11 character on the Internet. Some people vibe with the patio11 character.
[Patrick notes: I have some nuanced thoughts on whether I am actually the character I play on the Internet. I’m the same person who was a Japanese salaryman or who saw my kids baptized in church, but the range of expected and appropriate behaviors for those roles is contextually dependent, right? It would be grossly inappropriate for me to Kool-Aid man into a religious rite and say “Have you ever wondered how donations flow through the Catholic Church? It’s a fascinating story with interlocking cultural, religious, and legal layers. We begin with the notion of a ‘corporation sole’...”
And so, yes, it is a performance, but when is oneself not a performance. Which is a thing we’re socialized out of taking explicit notice of very early in life, but another thing I have substantial experience with performing is not internalizing all social cues we make available to seven year olds, partially due to my own difficulties and partially due to considered choices.]
Patrick:They want to learn more about that character the same way some people want to learn what kind of pizza Thor would like. I'm most comfortable with the concrete deliverables in the package and least comfortable with the "invite you into the broader life" part. But it is part of the package, for better or worse.
Talking about production functions, intellectual interests, and the path-dependent way we find ourselves thrust into career positions—I fell into my job in many ways, and that wasn't completely random. There were some explicable events over the years which, while not 100% planned, exerted gravitational influence on the trajectory I ended up taking. Perhaps examining those is somewhat interesting, and perhaps people will tolerate my occasional stories of interactions with my wife for their cultural or comedy value.
Beyoncé
Sammy: I am very glad that the Single Ladies story is on the internet now. That brings me great joy.
Patrick: I still owe Beyoncé a thank you note. Got to send that one of these days. That's not a joke, by the way. I'm very literally going to send one—I've committed to my wife to doing so for many years, so that's on the list.
Sammy: That's hilarious. Invite her to the podcast! We could use some guests for 2025. Unironically, it would be really interesting to listen to you specifically interview Beyoncé about the music industry.
Patrick: We shall see if that happens. [Patrick notes: This is salaryman deadpan; I do not believe it is likely to happen, and do not plan on investing much effort in making it happen.]
I think there are various people in the music industry who combine being superstar performers with understanding the business far better than the typical CEO understands their business. Beyoncé is one of those people. Taylor Swift is frequently identified as one of the performers who has their head most properly oriented toward dealing with a sometimes dysfunctional industry and its dynamics. There are probably another five or ten you could name—probably more if you've spent significant time thinking about that industry. I'm thinking of potential guests who are perhaps more likely to pick up the phone than if I were to—
Sammy: Than if you were to ask Beyoncé.
Patrick: Beyoncé did sing "Telephone," right? "Boy why you blowin’ up my phone?" [Patrick notes: See, authenticity is complicated. I am simultaneously someone who fibbed about his degree of appreciation of her music to impress a young lady, and also someone who sings Beyoncé songs at karaoke with the young lady who is now my wife. (Ruriko does the Gaga parts.)]
More subjectively favorite episodes
Patrick: My subjectively favorite episodes include our interview with Kelsey Piper, a journalist at Vox among other places. I think the intersection of the tech industry and media is poorly understood outside a relatively narrow community of practice. The sometimes fraught relationship there is multifaceted—not just regarding the narrow view of various large tech companies' communications strategies, but also how it colors tech's relationship with other centers of power, like the government.
I appreciated being able to have a polite discussion between people from two camps often portrayed as opposed—the tech industry and tech journalism—and explain the incentives and path dependencies that caused this relationship to evolve from the more typical mutually synergistic industry-journalism relationship to quite antagonistic in recent years.
So check out that episode if people want to hear more about that topic. Another one I liked was our interview with Ross Rheingans-Yoo about the drug development industry and pipeline. I love learning new things, and I occasionally have the same chip on my shoulder about having an engineering degree that many CS people do, which is like, “I'm not a real engineer, I just did the computer stuff.” [Patrick notes: This self-deprecation is against my interests and our interests, fellow engineers, but it is also a real streak in our collective culture.]
Drug development, very definitely outside my wheelhouse, and yet relevant to all of our interests the last couple of years. If we project for the next 40 years, for what technological advances we see, we would be extremely unhappy if drug development kind of paused in 2025.
We would be extremely unhappy if Ozempic was still the most recent blockbuster success in the pharmaceutical industry, so understanding the pipeline, the different players, and how drugs get to market is interesting. People often assume you need a PhD and years of devotion to have a decent mental model of it.
But no—you really can spend an hour with a smart person who has devoted part of their career to something and get a much better mental model than you'd have as a well-read college graduate who's followed pharmaceutical companies through the New York Times and Wall Street Journal over the years.
Sammy: And then we had the episode with Joshua Morrison afterward.
Patrick: Yes, focused largely on Operation Warp Speed, the crash effort during the pandemic to bring multiple candidate vaccines to production. With a bonus discussion of what I think will be a breakout success over the next 10 years—the cluster of technologies for modernizing HVAC (heating, ventilation, and air conditioning) systems to modernize our built environment in offices, schools, and other pockets of trapped air to decrease the spread of respiratory viruses and other infectious agents.
We had a series of episodes on energy economics that I intend to continue—Casey Handmer on solar, Austin Vernon on fracking. I've been doing some pro bono work for an NGO in the geothermal space for the last year and would love to do a geothermal episode. We'll see if I can find someone knowledgeable about geothermal who's willing to join the podcast. That will likely happen. And we'll continue curating and finding interesting guests on interesting topics.
Thesis of Complex Systems
Sammy: So we've recorded episode 26 this week. There's been a couple of recurring themes throughout our podcast. Any of those you want to call out specifically?
Patrick: I think there's something like a general factor of infrastructure—a topic I probably want to explore in writing at more depth. After you've worked in any field of infrastructure for a while, you start developing mental models for how supply chains typically work, how incentives align or misalign, and how the interplay between multiple competing or cooperating organizations tends to create the glorious, functional mess we see in various parts of our world.
I've enjoyed seeing in Complex Systems episodes further evidence for this general factor of infrastructure—we can sometimes predict what guests will say before they say it. Often guests will bring it up themselves, riffing on something you've said before: "We see that in our industry as well..."
That suggests there's something learnable here. (At one level, obviously infrastructure is learnable. We have operations research departments for a reason.)
There seem to be generalizable lessons across many of the areas we examine. That feels both edifying and useful.
Sabbatical mode
Sammy: I'd love a list of books and other media about this. Because while you learn a lot by doing, some of this you can probably learn from reading. There's Factorio, there's The Wire—
Patrick: Unironically yes.
Sammy: Yeah, seriously. One's about the people, one's about the systems.
Patrick: There does exist a canon in operations research.
There's any amount of writing about Toyota, for example. Oh man, there's a bit of a corporate cult of personality around Toyota, explicitly, due to various factors, including the existence of a consulting class that will tell people, "For the low, low price of our invoices, we can make your company as successful as Toyota."
Well, we shall see. I do think there is some DNA from that collection of corporate entities and broader ecosystem in Japan that we didn't have in the United States but have more of these days—partly due to the efforts of that consulting class, partly due to generalized competition under capitalism, and partly due to other factors.
Speaking of interesting cultural quirks, this recovering Japanese salaryman is re-acclimating to American culture and the American professional-managerial class (PMC). It has some quirks, to put it mildly.
As we mentioned in the Yatharth episode, I am something of an insider who's also an outsider frequently over the course of my life. I feel like a weird insider-outsider with respect to the class that calls most of the shots in policy-oriented discussions and high-level corporate discussions in the United States.
The class lens on these things—particularly communication styles and micro-political realities—seems to be an interesting one to bring up, partly because while we spend a lot of time in academic settings talking about class, we then promptly purge any knowledge of that when we go into the workforce, which seems suboptimal. There are a number of lenses that we exhaustively train the PMC in having for the world, but through a combination of emergent behavior, social sanction, and other processes, we don't train those lenses on our own actions.
Sammy: I'm glad that one's going in the transcript.
Patrick: More discussion about what I'm subtweeting in the transcript if people want it.
[Patrick notes: This is such a recurring theme in Bits about Money and Complex Systems at this point that I can’t point to a single thesis statement. It frequently comes up in e.g. the people who create, write about, regulate, and advocate regarding complex systems are often not representative of the users of the systems. They frequently mispredict users' concerns and usage patterns.
But, by way of specific example, I’ve frequently observed that the American PMC is vocally in favor of diversity, and yet is utterly befuddled when it encounters cultures which disagree with the American PMC on any issue of substance. When it encounters them at e.g. its workplace, it sometimes becomes volcanically angry.
For example, for a few years, it was simply good manners in the American PMC to nod along when at a mandatory training where a highly paid professional described veneration of the written word as an act of white supremacy. A maximally sympathetic member of the American PMC would quietly tell you that voicing disagreement was deviant in this time and place, and strongly counsel that you not.
Anyhow, your friendly neighborhood Japanese salaryman, reporting from his travels in an exotic land filled with inscrutable rituals.]
Patrick: Another theme: "systems are understandable by mortal minds" seems like a vacuous observation about the world. Your city has a water department. None of the actions of that water department are random. The system works. Go to your faucet, turn it on—the water will work and it won't kill you. That's a very important fact about the world.
There isn't a secular priesthood of "water guys" who have some specialized knowledge inaccessible to mortal minds that causes the water to turn on every day. By asking the right people the right questions, you could learn more than you might think you want to know about water treatment. And you'd probably feel pretty edified knowing what percentage is chemical, what percentage is built infrastructure we've inherited from prior generations, what the day-to-day activity of the water department looks like, how software impacts it, and what the top topics are at water management professional conferences this year.
[Patrick notes: Of course water treatment professionals have conferences. Everyone has conferences. Theirs presumably sound a lot like yours, except with more scintillating water treatment topics, between the hallway track that everyone is actually there for.]
Sammy: I wonder, do you know the shibboleths for people who think a lot about water management?
Patrick: I don't know them. Given the general factor of infrastructure, I could make some high-percentage guesses, but maybe I'll put some in the show notes and then be appropriately chagrined when we eventually talk about water filtration on this podcast, which should definitely happen at some point.
[Patrick notes: Predictions for future topics: our talent pool is aging and we’re having difficulty replacing it, our physical plant is aging and we feel underresourced for maintenance, climate change is far more relevant to water systems than to most professionals (even though most professionals will make a low-effort guess that climate change probably matters to most professionals), and customer usage patterns for water have begun to change far faster than we can change the built infrastructure of our water systems.
I know rounds-to-nothing about water treatment; we’ll see how close knowing nothing got to actual concerns.]
There are only so many primal elements to go around. We've done the fire episode—well, is fracking a fire episode?
Sammy: Yeah, and Casey Handmer's stuff. We've done fire, we've done the sun. We haven't done wind. Have we done earth? I feel like fracking is both fire and earth. We got two birds with one stone. Anyways, what other themes do we tend to touch on here?
Patrick: There's a John Collison line I really like: "The world is a museum of passion projects." Much of the built infrastructure we see in our cities and artifacts of our culture, both great and small, took a singularly dedicated individual a large portion of their life or professional career to bring to fruition.
Perhaps that's obvious when you're looking at a cathedral. It's less obvious when you're looking at an individual hotel. Yet that hotel was once a great expenditure of effort from an often very small team for a very long period.
I have two riffs on the Museum of Passion Projects. One is that society advances when things that once required consuming effort from a small team for a very long time can be clone-stamped out by Capitalism Incorporated in a deterministic fashion—needing less of the stars to align and more just a Q3 budgeting process. Many of the great hotels of American cities were passion projects, while a copy of a Holiday Inn Express is just a real estate investment at the margin. That's actually a fun stack of the real estate development community—see the episode with my father, Jim McKenzie, for more about real estate development. We should do more real estate episodes.
[Patrick notes: The second riff? I think I was going to say "But the frontier is still rife with passion projects, with things that humanity doesn't yet know how to do, and which it requires people to recruit a few underqualified friends and claw those out of the ether by main force. And then a few years from now the story will be retconned. Qualified professionals assessed a very high likelihood of success for that merely incremental undertaking from the jump, naturally. And there will be some built artifacts that many are a bit embarrassed about, because we certainly wouldn't ever choose to build their like again, but which exude beauty tinged with madness."]
Sammy: We should also have someone on to talk about franchising.
Patrick: Yes. Being able to talk about the in-depth history of projects that give rise to the built environment of our cities and our lives is an ongoing research brief for this program.
Sabbatical mode
Sammy: So, it's been a bit of a long year for us, hasn't it?
Patrick: Yeah, it feels like a bit of a too-long year for me. I kind of divide the seasons of my professional life by what my main thing was at any given time. I left full-time employment at Stripe in January 2023, and it's now January 2025. So I've been in this semi-sabbatical mode for two years.
I jokingly say that one of my great things to show for it is beating Space Exploration and Factorio. I haven't beaten Space Age yet, but I'm getting there—800 hours invested into the two of those.
Sammy: You really haven’t beaten Space Age yet?! [Patrick notes: Mutual context which listeners don’t have: I told Sammy that when Space Age came out in late October he was to expect me to take one to two weeks hard offline. I was really, really anticipating that game, and felt I deserved a vacation.]
Patrick: It's been a very busy two months for me.
Professionally speaking, I'm kind of in an interstitial period between what happened last and whatever happens next. But Complex Systems was a great portion of my professional effort in 2024 and is, knock on wood, bearing some fruit, though I've continued to do other things as well.
Sammy: Yeah, we had that New York Magazine piece at the start of this year that really caught your attention.
Patrick: "$50,000 in a shoebox." For those who missed it, in February 2024, there was a piece in The Cut, which is a Vox publication. I refer to it as the New York Magazine piece because New York Magazine is a high-status publication with a print edition, and it was co-published by The Cut and the print edition of New York Magazine.
That has occasioned an investigative project from the two of us. Not to tell the entire story here, but the piece is a first-person account of the author being the victim of a scamming operation. One step in the narrative is that they made a $50,000 withdrawal from a bank in New York with very few questions being asked.
We have some questions about those questions not being asked, so we've been independently re-reporting some of that.
Sammy: I think the first two major things I did for you after the work trial were getting you a physical copy of that magazine issue and then going to New York to do some in-person investigative journalism.
Patrick: Yeah, we sent you into every Bank of America branch in Brooklyn to put some shoe leather into it.
Sammy: Not every one—all the ones we couldn't obviously rule out from Google Street View, which is great. Google Street View is such an amazing tool.
Patrick: Google and similar tools are available to talented amateurs—though I use "talented amateurs" auto-minimizingly. [Patrick notes: The cobbler’s children have no shoes, and no matter how many times I tell people not to minimize themselves, I still fall back to my old speech patterns.]
There's a very real sense in which this is our job. Bits About Money really does have paying subscribers, and I am something of an industry leader on the topic of banking regulation and proper procedure if someone walks into your bank branch attempting to withdraw $50,000 in cash. [Patrick notes: I am not saying this to sound self-important but this is also not a joke.]
We don't yet have the results of that investigation tied with a bow, but we're still working on it.
Sammy: Yeah, we're still working on it. Thank you for your patience, loyal audience.
Patrick: I did give a talk about it at Manifest. That's the conference associated with Manifold Markets, which is, full disclosure, a small angel investment of mine. [Patrick notes: I interviewed one of the founders, Stephen Gruett, for this podcast.] I'm a long-term fan of prediction markets, which had something of a rise in the sun this year, partly connected to the presidential election—always great for prediction markets—and partly connected to widespread uptake among cognoscenti.
At Manifest, we did something that's not part of most media production functions: we presented a story in progress, shared what we'd learned so far and how—
Sammy: And created a prediction market on what people think our conclusion to the story will be.
Patrick: Yes. The prediction market has resolved because it timed out at the end of 2024, but I still hope to get the investigation results up on Bits About Money at some point in 2025, hopefully sooner rather than later. [Shakes fist at various Freedom of Information Act respondents who haven't acted within statutory timelines]. More on that in a different place when the time comes.
Sammy: There was also a meetup we hosted. We should do more meetups this year. That was a fun event to host at Lighthaven, which is where I'm currently recording this because someone was jackhammering directly outside my house. Lighthaven is a really great venue for many things, including meetups—it's where Scott Alexander does his in Berkeley.
Patrick: It's funny how I've orbited the Silicon Valley ecosystem for a very long time at a large remove. Then after being an ocean away, I'm currently located in Chicago, which is far from Berkeley/SF/Silicon Valley—which are not actually the same place. I learned that only recently. How long was I in the tech industry before I understood that when people say "Silicon Valley," they don't mean San Francisco? I think I was in the tech industry for more than ten years before I realized those were physically distinct locations.
Sammy: That it used to refer more generally to different specific physical locations. And then I guess over the '90s and 2000s, it grew into largely San Francisco, is my understanding.
Patrick: I don't know if people would necessarily agree with that. I think "Silicon Valley" has achieved something of a metonym status with the SF Bay Area—well, first as a metonym for the larger tech industry, and then it got kind of retconned into the SF Bay Area, partly as the center of gravity of various firms and activities has shifted more toward San Francisco than the peninsula. I didn't even know it was called "the peninsula" until 15 years into the industry—shows what I know.
Made a few appearances on other people's programs, notably Conversations with Tyler. Really enjoyed that one—though it's a high-stress production function. Tyler Cowen is whip-smart and prepares by reading large samples of everything their guest has ever published, which for me is five million-plus words. He hits you with random questions from your entire published corpus, and you learn about them in real time during the recording and have to tap dance. I feel like I gave a relatively good accounting of things I'd written, in some cases from Hacker News comments from 2012. I don't know what his research process is, but wow—deeply talented person and team.
I was also on Dwarkesh's podcast and discussed some of what we learned during VaccinateCA days in 2021. That presaged some of the political winds that have blown through the tech industry in the last six months or so. (Obviously, we just went through an election season in the US, and the election caused various political winds to blow.)
The particular interactions of the tech industry and the government-adjacent misinformation apparatus that we discussed on that podcast have come to increasing fruition, including this morning when Mark Zuckerberg released a video essentially pre-committing to a very different level of engagement with respect to "misinformation" than had prevailed at some points in the recent past.
Sammy: What shocked me most from that was the decision to physically move their content moderation team to Texas instead of keeping them in California. I would have loved to be a fly on the wall when that was being decided.
Bits about Money and other writing in 2024
Patrick: Among standout things I've written this year, I think my best piece in the last 18 months was about debanking. For anyone following up on this episode years after the fact, there was something of a discourse, started by Marc Andreessen et al., regarding debanking, particularly through a crypto-related lens.
"Debanking" is intentionally an advocacy word. They use it to cover banks terminating accounts of existing customers, refusing to open accounts for particular disfavored individuals or businesses, and a grab bag of other complaints about the activities of banks and banking supervisors.
I wrote what I feel—I don't know if this is a modest feeling or not—was a somewhat definitive piece covering the history of this issue, the broader picture outside of a very crypto-focused lens, and some of the procedural history of the various actors people are worried about regarding actions taken against crypto firms and the crypto industry specifically.
I think I was pretty fair to crypto advocates and gave grace to places where they have a point, while also discussing the difference in perspectives that individuals of good faith in the crypto industry and banking supervision might have. For instance, regarding what degree of transparency they expect in routine communications between supervisors and banks that implicate a third-party industry. The piece is 24,000 words long. People are welcome to read it—I think it holds up pretty well.
Sammy: Yeah, I really enjoyed reading that.
On paywalls and not implementing them
Patrick: I've continued writing for Bits About Money and need to keep doing so. January will feature, as always, a subscription drive. If you have an education budget or want to learn more about financial infrastructure, visit bitsaboutmoney.com. Feel free to sign up or just read all the issues for free. [Patrick notes: If BAM is relevant to your professional interests, you should buy a membership. More paying members means more time to write on that beat.]
One of the things I like about my quirky position in life is that I can produce things that aren't paywalled, getting enough subsidization from advertisers for Complex Systems and paying Bits About Money members to keep paying the mortgage while doing this work.
Sammy: Do you think if large portions of Bits About Money were paywalled, you'd have more total paying subscribers?
Patrick: By default, that's a very good model for the world. People tend to make more money when they ask for more money. They also tend to sell more services when those services aren't public goods. At the same time, there are various goals for Bits About Money and my broader Internet participation—earning money is only one of them.
For example, the debanking piece is currently making the rounds in various corridors of power in Washington DC. It's much easier for something to circulate when you can simply go to a URL and immediately read it without a popup or paywall, or needing someone to download a PDF and attach it to emails to congressional staffers. Having content accessible on the Internet makes it easier to impact the policy discussion.
There's some compounding advantage to the industry—or industries broadly—to have some of this thought exist on the open Internet for people who find it professionally relevant, but not relevant enough to subscribe to what's termed in the trade as a "trade rag."
There's a trade rag for many industries—actually, multiple trade rags for each industry. For example, there are specialized publications for running a community bank. [Patrick notes: American Banker springs to mind.] These cost a couple hundred dollars a year [Patrick notes: a few thousand in some cases] and are largely consumed by e.g. the C-suite at community banks. Very plausibly, there are congressional staffers or people in banking regulators who should have subscriptions to various community banking trade rags, but might not because of irrational resource allocation in U.S. government or simply because they haven't gone through the required steps to get them. So they can't understand what the financial industry, or this subset of it, is telling itself on any given day.
People shouldn't necessarily model me as a mouthpiece for the financial industry, but to the extent that I'm a reliable weathervane for how some sub-segment of the industry thinks about various issues, it's probably better that this weathervane is publicly distributed rather than completely firewalled behind a paywall. [Patrick notes: As always, thoughts in my own spaces are my own, rather than necessarily representing past employers, etc etc etc.]
Sammy: Yeah, that makes sense. Anyways—so this is year three of the sabbatical?
Patrick: That's a terrifying way to phrase it. We've passed almost two years of the sabbatical at this point. I think I'd be unhappy if I didn't have a major change in situation at some point in 2025. But I don't know if that's a commitment—it's more something I'm committing to explore very much this year.
LLM plans for 2025
Sammy: That makes sense. You mentioned hoping to do more experiments with LLMs while you had this time. What particular directions are you interested in exploring?
Patrick: I think intensifying my use of them is one direction. People have said pretty persuasively that LLMs are transformative for writing code. I've written very little code during my sabbatical, partly due to that being a smaller portion of my life these last couple years, and partly because a major purpose of taking the sabbatical was to disconnect a bit from the somewhat all-consuming nature of professional life and spend more cycles on family issues. [Patrick notes: We relocated from Tokyo to Chicago, bought a house and extensively renovated, have blitzed through a bunch of childhood milestones, etc.]
I've done that and consequently done rather little programming.
I want to test LLMs for these emerging use cases. I also want to try this fascinating new frontier opening up—I'd like to explore some frontiers within the frontier versus simply consuming the most easily accessible consumer-oriented version via ChatGPT or similar, and reading others' research results. For people similarly trying to identify interesting research results, Simon Willison, who worked at VaccinateCA with me, has a blog where he frequently discusses interesting experiments he's done in LLM-land recently, or tools he's published. I've gotten a lot of value from following that, in addition to doing my own experiments.
We'll see if I manage to publish things. Some of the things I'd most like to try are difficult to publish for various reasons. One thing people have reported being successful is that apparently "patio11" is a useful token to LLMs for prompting them, which is wild to me. You can literally take a photo of a letter from a bank and say, "Write a response to this letter in the style of Patio11's Dangerous Professional," and you'll get shockingly good output.
People have sent me examples. A long time ago, I used to ghostwrite letters to banks for a few years. I no longer do that because I don't have time to ghostwrite hundreds of letters. It turns out that for one ten-thousandth the cost of having me ghostwrite letters, you can have a computer do it. As something of a subject matter expert in this, having read many of the outputs people have shared—yeah, that'll work. That's exactly what the doctor ordered.
Sammy: I wonder if you could do a blog post just collecting prompts people have used to get value out of LLMs using patio11 as a token.
Patrick: We'll see about that. The first thing I'd love to do is publish a micro tool that would just do it, but there are some risk areas associated with that. For example, this relates to the Copenhagen theory of culpability—where some actors in society assume anyone who has touched a particular issue is responsible for all harm created by that issue in totality.
There's some risk in holding myself out as someone available to work on issues with banking and other bureaucracies that every time someone is adversely affected by a bank or bureaucracy—either after following my advice or even not following it—they might direct their anger at me. The bank is a big impersonal institution; I have a name and face that you can get mad at. The particular contour of issues with banks is that if you do consumer advocacy for banks, the people who have issues with their banks will frequently have many unhappy things happening in their lives.
You can understand why someone would say, "The sequence of events is: I had a problem with my bank. You promised to help. I did what you told me to. That didn't help. And now my husband is dead."
Sammy: Yikes.
Patrick: I'm not exaggerating—these are the processes people go through. People have unfortunate events happen all the time. Those events are frequently tied to money, or will be tied by the individual in their narrative to money in part. That makes publishing a microsite that says "bring your banking issues and this will attempt to solve them" into something of a complicated subject for me.
[Patrick notes: One of the things that medical providers purchase from the insurance industry is the presence of someone the medical provider can direct anger at. Some of the anger is over “doctors could treat you if you didn’t pay them, but mostly won’t, and prefer someone else delivers that message.” Some of the anger is over humans being frail and mortal.
Similarly, much anger over the banking system is anger at the rest of society, which has deputized the banking system to tell people "We understand that you want things, and will not give them to you, because you cannot afford them." (This is not merely figurative; the literal text of the interaction where a grocery will decline to give you food will include the words "You should talk to your bank." The bank will pick up that phone call on the second ring, and the grocery store pays interchange so that it does not have to be sympathetic but unwavering during that conversation.)]
But I will certainly write about the Dangerous Professional topic, which needs a URL, just to collect my scattered writings on this over the years. It's something of a performance of class and weaponization of how bureaucracies work to have them collaborate more effectively with you than they might if you couldn't present as this character of the Dangerous Professional. I want a single legible URL to point people to for that concept in mindspace.
I'm increasingly thinking about writing differently now. It seems obvious that I was primarily writing for a human audience for most of my career as a writer, aside from considerations for search engine optimization and similar. It's very possible that a plurality of readers of pieces in the future will not be human.
How should writers think about that? Are you making pointers to mindspace on behalf of relatively sophisticated humans? For people who will explore those pointers using tools? Are you making pointers without needing human invocation, just trying to tie concepts together for the benefit of a training run in 2028? I don't have great answers to those questions right now, but I'm spending time thinking through them.
Sammy: I would also imagine that if a dominant hypothesis is that in a single-digit number of years, the majority of readers will not be human, that has more important implications than just “what I should write essays about right now.”
Patrick: Humans use various prosthetic devices that aren't usually characterized as such to perceive the world. Your camera is a prosthetic device, and these screens showing our faces to each other as we record are prosthetic devices. Currently, in how LLMs are deployed as a topology, relatively few users are using them, and they're mostly actively aware when they're using an LLM. I would bet substantially against both those things being true in the future.
They'll increasingly be built into how systems and society work broadly and will often be executed behind the scenes. If you look at the early adopter curve right now, the most advanced early adopters are invoking one LLM at a time for a couple minutes or hours every day. Eventually, it will likely be the case that the typical human has many instances of LLMs invoked on their behalf without explicit actions. And by many, I don't mean five or ten or twenty—these cost curves are what they are. There could potentially be millions of LLM invocations on behalf of the typical American consumer on a typical day.
[Patrick notes: I am aware this sounds unlikely. There exists an entire adtech ecosystem which has had thousands upon thousands of, essentially, simulated high-frequency traders bid upon your attention today. Many readers don’t know that ecosystem exists, which is fine; that ecosystem judged you knowing it existing as a suboptimal allocation of your attention for today, and instead showed you different ads.
And so very plausibly your grocery shopping in 2030 implicates thousands of agents which understand enough about the world to read a prompt beginning “You are an agent tasked with helping Insert Name Here find eggs.”]
Patrick: That concert of AI operating in the background has interesting implications for how we perceive the future. And that's kind of the most boring version of the future too—we reach there almost inevitably given what's priced in during early 2025. I'm perhaps a little less doom-ish than many people who have spent a lot of time thinking about this. Though I don't know if "less convinced about superintelligence" is actually accurate—that might be a topic for another day's deeper discussion. The capabilities that have already been persuasively demonstrated are extremely robust.
Just projecting current capabilities forward plus the enterprise software adoption curve implies somewhat wild things about the world we'll experience over the next 10 years, even if the curves stop scaling—which I don't think the smart money is betting on. Well, no, that’s excessive understatement. We can literally simply observe the smart money placing bets. It thinks: "No, we think if we dump a hundred billion dollars into this, there will be positive ROI.”
Sammy: Current capability LLMs would make the world really weird, but current capability LLMs are going to get a lot better by the end of this year.
Patrick: It will be interesting revisiting this in a year and seeing how different our personal experience of LLMs is and how well or poorly calibrated we are regarding the progress we'll see in 2025. I don't think I'm nearly the best calibrated person on this topic, but like many others in the tech industry, if one doesn't have at least half an eye on this, what is one doing? Many of my better-calibrated friends would say it's difficult to have anything less than two and a half of your eyes plus as many artificial intelligences as one can afford looking at this topic. And I understand where they're coming from too.
So that's a bit of the year in review and maybe a little of the 2025 year in preview for where we expect to take things. Sammy, thanks for coming on the podcast. You can find us around the internet. I'm patio11 in all the places. Do you want to drop your contact info?
Sammy: My website is hath.blog, and you can reach me at hath (at) hath.blog.
Patrick: If you have suggestions for podcast guests, topics you'd like discussed, or want to discuss sponsorship opportunities, you can reach out to me directly at Patrick at—well, all of my domain names will get you to the same place, so try any of them. [Patrick notes: kalzumeus.com is the one which will involve the least extra work for a poor, benighted cron job at Google. Which I say partly as a joke, but the LLM revolution has caused me to spend much more time than I would have expected caring on an emotional level about the subjective experience of computer programs.]
Thanks very much, everybody. We'll see you next week for the next episode of Complex Systems.
