Bits, Bytes, and Burgers with Byrne Hobart
This week, I'm joined (again) by Byrne Hobart, author of The Diff. Byrne is, with Matt Levine, the author whose beat and style probably most influence my own financially-inflected writing. We've been Internet buddies for a very long time, and I thought the sort of casual conversation we might have at a dinner with no specific agenda might be interesting to listen to, so we tried that out as a format.
If you haven't encountered Byrne before, I'd recommend starting with The 30-Year Mortgage is an Intrinsically Toxic Product. It thoroughly earns the title, and is accessible at a variety of financial sophistication levels.
Sponsors
Check is the leading payroll infrastructure provider and pioneer of embedded payroll. Check makes it easy for any SaaS platform to build a payroll business, and already powers 60+ popular platforms. Head to checkhq.com/complex and tell them patio11 sent you.
Building an enterprise-ready SaaS app? WorkOS has got you covered with easy-to-integrate APIs for SAML, SCIM, and more. Start now at https://bit.ly/WorkOS-Turpentine-Network
Timestamps
(00:00) Intro
(00:45) Economics of video game currencies
(02:56) Pricing strategies in mobile gaming
(05:08) Monetization skew towards high-end players
(08:08) VIP systems and casino host analogy
(11:08) Whale behavior in casual games
(15:03) Hyper-consuming outliers in other industries
(19:09) Sponsors: WorkOS | Check
(21:25) Hobbies and opportunity costs
(23:01) Custom software for tech billionaires
(26:30) Evolution of website development
(29:55) Restaurant websites and delivery apps
(40:17) McDonald's take rates
(44:59) Restaurant groups
(53:34) Tech company cafeterias and employee benefits
(57:57) Google's business model and economic feedback loops
(1:00:57) Early Google investment anecdote
(1:02:16) Writing as a memory aid
(1:04:46) Using ChatGPT for memory assistance
(1:10:30) LLMs as writing and coding aids
(1:13:34) Children's interaction with ChatGPT
(1:18:11) Arguing with LLMs and using them for research
(1:03:00) Wrap
Transcript
Patrick McKenzie: Hi everybody, my name is Patrick McKenzie, better known as Patio11 on the Internet, and I'm here with my buddy Byrne Hobart.
Byrne: Hey everyone.
Patrick McKenzie
So, Byrne is the first repeat guest that we've had on this program. Both of us tend to write and talk about many of the similar things in our respective beats around the intersection of finance and technology, and every time we have a conversation, I feel like a lot of people would love if they could be flies on the wall here. So we're just going to have a casual fly-on-the-wall chat for something that's been on Byrne and my respective minds recently.
Byrne: Absolutely. Let's do it.
Economics of video game currencies
Patrick McKenzie: So kicking things off, you write a newsletter called The Diff, which everyone listening to this should subscribe to (pitch out of the way). One thing you mentioned recently was about the economics for video games that are monetized by having a split, free currency/real money currency, and how the pricing grid works for those real money currencies.
Do you want to chat a little bit about how the pricing grid is for people who have not been sucked into Zynga before? Then we'll go from there.
Byrne: Yeah, so with the caveat that I have never purchased strawberries for my Farmville farm [Patrick notes: I will admit that I did buy Farmville Ruriko a nice autumn dress once while courting real-life Ruriko; thankfully, this did not scare her away] or a cool hat for my Counter-Strike character – but I'm aware that this exists and you can't miss it if you look at the financial statements of some companies in the gaming space – there's a model for games where they are free to play, but they are generally not free to win. You can download the game, play around with it – that's what the vast majority of people do – and what they will typically do is they will structure the game to give you various kinds of virtual resources. Some of those are very cheap and you get them all the time, and then some of them tend to be resources where your access to different parts of the game will be gated by the availability of these resources.
You often end up in a situation where you can get to the next level, you can get the next hit of dopamine if you either spend a bunch of time grinding, or take out your credit card (or you usually have your payment information saved, have had it saved for a very long time), you can make an in-app purchase and then proceed to have more fun.
But when you make these purchases, in many cases, you are not directly exchanging dollars for virtual goods, even though the virtual goods are what you're using; you're usually exchanging dollars – often not always – for some kind of in-game currency, and then you're spending that currency on those goods.
What a lot of these games have is kind of the equivalent of the currency exchange stores you might see in airports and in Times Square or something, where they are not there to make a really competitive market in Euro-USD or something, they are there to amortize the cost of high rent by really taking a large margin on those currency transactions.
Pricing strategies in mobile gaming
In real-world currency, you sort of have a sense of what these currency units do and what they buy, so the chunking thing doesn't really apply – but in in-game currencies, they give you a set of specific transactions you can make. You're exchanging typically dollars for magical gems and you can spend – it's almost never a round number – usually 99 cents for a certain number of gems or $4.99 for more gems or $9.99 for even more gems. As you might assume, there is a bulk discount.
But one of the things that they do in this pricing grid, if you actually price it out – and this is actually one of these cases as a side note where it's one these tiny, tiny improvements that AI has made to my life; if I had been writing the post on this three years ago, what I would have done is I would have had Excel and I would have had my browser window and I would have been typing numbers into this grid because they were in some kind of graphic, but because I had ChatGPT I took a screenshot, dragged it into ChatGPT, said, “please convert this into a table” and it did it just fine… and it also did the next obvious thing, which is give me the gem-USD exchange rate at each price point.
What you find is that you do get a bulk discount, but the bulk discount isn't really linear, and it sort of looks more compelling for the low dollar amounts, and then for the higher dollar amounts your savings tend to trickle off. But also, they generally do not do really round numbers, and they certainly aren't going to give you a gems for dollars exchange rate.
You can think of different customer behaviors where, my guess is the first transaction is almost always 99 cents. It's almost always that someone has one little thing in the game where, if they had the 50 gems, this would not be a problem; they don't have the 50 gems, so it is a problem. They want to get to the next thing; they get their 50 gems.
At some point someone does that perhaps a couple of times and they realize they should get a bulk discount, so they calculate what their bulk discount is and start spending $5 or $10 at a time. And then my guess is, once you're at the point where you are periodically interrupting your gaming time in order to make these purchases, you just decide, “okay, I'm gonna buy a very large amount, like $50 or $100 worth of gems.
Monetization skew towards high-end players
That raises the question: if you read anecdotes about these games or if you happen to have had access to some kind of anonymized data set of consumer spending, you might notice that there is a pretty strong power law. There are people who play these games occasionally and spend a little bit of money, there are people who have what I would describe as like – I don't know these people, but I've done the SQL queries – I would describe it as a problem. There are people who are spending hundreds, thousands of dollars a month on these casual games. Those people, those huge whales are pretty rare, but it kind of raises the question of, “okay, why doesn't the game give you a chance to buy a thousand dollars worth of gems at a clip, or five thousand or ten thousand?” Like, surely the conversion rate on one big hefty transaction is going to be better than the conversion rate on trying to consistently get them to make a smaller transaction over and over again.
My guess is when they show that pricing page, they really don't want people thinking about the possibility that they will buy five thousand dollars worth of gems at a time, and so at some point bigger discounts for larger purchases actually scare enough people off from making that first purchase. We have to assume that all this stuff is A-B tested endlessly.
And when I say probably the first purchase is 99 cents, that's not just a behavioral claim about how someone would start spending money on a casual game on a digital service – that's also a game design feature, that when they set up these barriers within the game, they probably do some kind of titrating thing where it's like you have to grind for a minute to get your 50 gems, and then you have to grind for two minutes to get the next 50 gems, and then for 10 minutes to get the next 50 gems, and so on – and at some point, it's just easier to spend some money, but they probably want to make sure that that first transaction is most naturally 99 cents.
Patrick McKenzie: So I would short the estimate that the first transaction is likely 99 cents, unless you have data to the contrary.
[Patrick notes: Both Byrne and I are avid prediction market users and financial geeks, and both understand “short” here to mean “bet against the probability of … before the truth is revealed.” Importantly, both Byrne and I also understand this statement to not be a slight. Jargon and shared sets of experiences/values makes for very high-bandwidth conversation sometimes.]
Byrne: Okay, I don't!
Patrick McKenzie: My point of view on this is largely informed by spending a few years as a consultant for software pricing pages, which is importantly adjacent to but not the way that pricing pages are done in mobile gaming. But to keep up with the state of the art, because Zynga et al. were employing a lot of very smart people to do a huge amount of testing on this sort of thing, I listened to the conference talks, read the PowerPoints, etc. and so I have a little bit of accidental knowledge here. One bit of accidental knowledge.
To be clear, I like a thing that Tom Clancy said once, where someone asked him if he felt fulfilled in life in his chosen profession; he said, “I entertain people. It's an honest living.” I think video games broadly entertain people. It's an honest living. There is, however, a spectrum. And many of the mobile games which are designed to be Skinner's boxes from hell are on the less salubrious end of that spectrum.
[Patrick notes: A Skinner’s box is a laboratory apparatus made famous by the eponymous psychologist, who designed it. In brief, it allows the scientist to train a rat to hit a lever when a light turns on, receiving a food treat, then remove the food from the equation and still trigger the lever-pushing with a light. When “Skinner’s box” is used pejoratively about gaming, it is to suggest that a game has optimized out providing an entertainment experience in favor of directly hacking the player’s reward circuits, generally for commercial reasons.]
VIP systems and casino host analogy
Patrick McKenzie: And then you can look at things like say – I've used many of these products in anger, as it were – I currently play Magic the Gathering after a 30-year absence, and it has a strikingly similar monetization model in its MtG: Arena product, and that is used by a lot of smart engineers in Seattle, San Francisco, etc. who are trying to recapture nostalgia for their childhood, probably have $200 a month average value per user, and are no worse off for having a hobby – it's golf clubs, but updated for the millennial set. I say all of that to excuse the following:
[Patrick notes: Video and slides links to below-referenced presentation.]
Patrick McKenzie: If you listen to a presentation from the CEO of Kongregate that she gave at a number of game conferences, including Gamasutra's conference back in the day, they talk extensively about the monetization of these games, and how the monetization skews towards the higher end of the spectrum: finger to the wind, I don't have active recall of all the numbers, but the vast majority of players will never pay a dime. Of payers, the 2%, most will make one transaction and then stop. [Patrick notes: You'll have to dig deep into the presentation, but this is impressionistically accurate.] Some people do not stop at one transaction, and as Byrne mentioned, spend thousands of dollars a month or more on these products.
[Patrick notes: And, critically, these whales, this tiny fraction of the 2% of payers, drive the monetization of the entire game.]
One reason that there is not a $1,000 a month thing on the landing page is that there are humans assigned to those players. Their interaction with the gaming company is no longer simply of a person typing things into their mobile phone, but they've been assigned someone who is very analogous to a casino host. For people who are not familiar with that bit of the less salubrious economy, if you are in the habit of going to Las Vegas and renting a room for a few days and then pushing a lot of money in on your loyalty card into games where the casino has a great edge, such as slots, etc., someone will make your acquaintance and buy you dinner. It's called comp in the industry, but they'll buy you dinner and they'll have a conversation about, like, “What brings you to Vegas? Are you in town for a conference, etc? Do you come here often? Do you stay at our place often? Do you stay anywhere else?” And basically build a profile of you which will get saved into the CRM system. There's actually a wonderful blog post from before Big Data was Big Data, saying that MGM had an unheard of amount of information on its customers. They measured it in terabytes. (This is from, like, the early 2000s when terabytes were still cool.)
[Patrick notes: I’m increasingly concerned about my ability to remember great essays from 20 years ago accurately: it was Harrah’s, not MGM, 300 GB not terabyte-scale, and made it into the Harvard Business Review, though I do distinctly recall a version that was the classic blog post form factor.]
Whale behavior in casual games
Patrick McKenzie: Anyhow, the casino host, their job is to bring you back to capture more of your share of wallet with that particular casino relative to every other casino in Vegas, and to facilitate your interactions with the casino and bring you back more frequently to spend more. The casino host for, without loss of generality, Zynga, is pretty much doing the same thing. It's an inefficient way to buy a friend, but it is effectively buying a friend who shares your passionate interest in whatever the recent Guild War that you're engaged in is, and who will do things like facilitating transactions for, “You think you need $10,000 to win the next stage of the Guild War? We can definitely take a wire for that,” and quote you the number of gems that that buys, etc. etc. – and like many VIP systems in the world, there are things like you get a preview of upcoming content releases, you get some very titrated but real access to the developers, etc.
That's very common in the gaming industry. The people who are external to the company but important to the success of the game – like gaming journalists, well-known players on YouTube, streamers, etc. – will often get invited to events under the cover of, NDA, frieNDA, a handshake or nothing at all, in advance of a game launch to kind of build buzz and help get the word out there… but those folks are not typically directly monetized. These folks are being very directly, aggressively monetized. Anyhow, a little thing about the world that most people wouldn't know.
[Patrick notes: A frieNDA is a common device in Silicon Valley where, for mutual convenience and as a demonstration of trust, no actual non-disclosure agreement is signed, but both parties understand themselves to be committing to keeping information off the gossip circuit. Sometimes it is announced, sometimes it is implied. A common rite of passage in Silicon Valley is someone young and impressionable learning how presumed-frienNDAed information was cheerily passed along by someone in a position of authority, who effectively makes a business practice of doing that. (VCs and tech journalists are encouraged to believe I’m pointing the finger at the other group. You, of course, are merely ethically performing your job.)]
Patrick McKenzie: I think it's interesting, though, because this is one of those things where you, I, and most people listening to this podcast think we understand the artifact because we can download the artifact on our phones and look at it and make assumptions on how the industry economics must work based on the artifact presented to us. This is an intuition that people repeatedly have about industries: I have flown in an airplane, therefore I know how airlines must make their money. I have been in a hotel room, therefore I know how hotels make their money. I have received medical treatment, and therefore I understand the breakdown of medical expenses.
These are, generally speaking, poor intuitions to have, because of the way that those costs, while they feel to unsophisticated customers are distributed relatively evenly with a couple of levers associated with them – like maybe you've got a slightly bigger hotel room – are not in any way distributed evenly. There are whales in hotel rooms. There are whales in airline travel. There are whales in many industries; there are certainly whales in medical treatment. (Most would keenly wish to not be in that category, but they also keenly wish to not die, and so they are getting their money's worth, as it were.)
Anyhow, got on my soapbox for a moment.
Byrne: I think that's true. I think there's a whole set of companies, or a whole set of products, where the typical person who interacts with them is not necessarily the median interaction – the median dollar spent is not spent by the median spender of dollars. So depending on what the power law looks like in that industry, you can have just very, very different economics, economics that only make sense in the light of the existence of these really, really heavy spenders.
With airlines, I ended up doing more business travel right around the same time that part of my job was to read and interpret airline financial statements. It was a really interesting experience to understand that I had not understood how this business worked, using some of the intuitions that I had as just a leisure traveler. Like I didn't really understand why airlines other than Southwest could ever make money – I did know that there was this general competitive dynamic where you have these capital assets, they last for a really long time, you have a lot of fixed costs, and so if you could sell that additional seat for $5, you would – and if Southwest has a lower cost structure than a lot of its competitors as it did many years ago, then of course they make all the money.
But then it turns out that it isn't just that, and that not every customer is mostly sorting by cost, and not every customer's economic relationship with the airline ends when they step off the plane – once they've gotten home and then they could restart that economic relationship with Spirit or Frontier or anybody else, whoever the low bidder is.
So yeah, sometimes the intuitions you get are going to be wrong, because you are not the person who drives the economics of that business – and I think that probably works the other way too, where it may be the case that the really hardcore players of these casual games don't really recognize just how weird that behavior is and may assume that a lot of other people do this too.
My favorite anecdote on “you're very unlikely to understand an industry if you are just the typical consumer interacting with an industry” is this brief note from, I think it was his ghostwritten autobiography, where Arnold Schwarzenegger is talking about how he and his bodybuilder buddies loved to go to the gym and then go to an all-you-can-eat buffet. One of their ongoing topics of conversation at the all-you-can-eat buffet was, “How could these places possibly stay in business if everyone sits down and just eats two pounds of protein?”
Hyper-consuming outliers in other industries
Patrick McKenzie: Yep. So for those of you who haven't read your substantial writing on this topic, something that you've covered many times is that the airlines work because there is a lucrative class of business travelers who are not spending their own money, they're spending money from a budget at work and they are willing to, subjectively speaking, pay richly for maximal convenience to their travel schedule or maximal convenience for business and necessity, and then bank on essentially their personal accounts and loyalty status with that airline, which they are able to put away towards the annual vacation to Hawaii or similar.
One of the points that you made recently was that this makes it difficult for low-cost airlines like Southwest, which, even if you get a collection of desirable routes in your network such that you're able to capture business travel, if you cannot both capture business travel and also get that person to somewhere nice over Christmas, you will not sell for the job to be done of the core business spenders in America, who can spend $100,000++ on air flights per year.
The existence of that statistic, by the way, blows the minds of many people; it falls pretty straightforwardly out of the map of what happens if you really do go home every weekend from New York to San Francisco or similar – which as folks who have known consultants might attest, is not that infrequent of a schedule when summed over the economy. Anyhow.
I genuinely do think there are industries where the sort of hyper-consuming outlier does not understand that the rest of the world are not hyper-consuming outliers. I'm stealing that word, by the way, from an essay that I read once which explains the market structure of the Japanese anime, manga, and cultural industries – I wish I remembered how to Google for that essay because it was fantastic. It said essentially that a number of the reasons for the very niche proclivities in parts of the anime world are they might not be very popular per se, but they are popular with a certain strain of hyper-consuming outlier who will, in addition to watching all the episodes and getting them all on DVD individually, buy a lot of the merch associated with that anime, such as plastic figures of the main characters (who are almost invariably scantily-clad young women) and pay $300+ for those figurines, of which a generous licensing fee goes back to the production company.
[Patrick notes: This is far from the only way that particular cat is skinned, and indeed some methods have a certain form of dark genius, atop an industry which otherwise mostly monetizes in the fashion of a Disney: license the IP to everything under the sun and collect a single digit percent on otherwise non-brandable high-margin consumer spending in an age/sex range. Anpanman, for example, is a $X0 billion stream of rent from young Japanese parents because it practically defines being a toddler to have Anpanman on your clothes and utensils.]
Patrick McKenzie: Some hyper-consuming outliers in some industries realize that they are disordered, or realize that they are, if not disordered, a little bit different than the average member of the population. [Patrick notes: Notoriously, many hyper-consuming outliers do not understand this about themselves. Heavy drinkers are one notorious category. Heavy readers are far less notorious but we—and, let’s be real, if you’re reading these podcast notes, you and I share an affinity here—frequently assume that the median American probably reads books. (Note: not true.)]
Patrick McKenzie: If you can trust the CEO at Kongregate – who definitely has an incentive to talk her book here, but I would believe it from from knowing many people who play various competitive video games – apparently, the whales for casual games are, you know, “we picked up the phone and called them and they seem like regular people.” Doctors, dentists, small business owners, engineers, etc. How can they spend tens of thousands of dollars a year? Because they're well off. Why do they like this game? I don't know. She makes an extended analogy to non-competitive figure skating: “I spend tens of thousands of dollars a year on that because it's my thing. Deal with it.”
She said, “to the extent that you think that our customers are disordered, a lot of them are like, ‘I just really care about wars between my clan and someone else's clan more than you do. Deal with it.’” And while I think there are parts that are self-serving, and I disbelieve the illusion for some games that target particular player populations, I think there's probably more than more than zero truth to that bit of, I don't know, informal economically motivated sociological work.
[Patrick notes: As is typical, these are impressionistic pastiches of an ancient memory rather than verbatim quotes.]
Hobbies and opportunity costs
Byrne: I think that is probably true. People have hobbies, high income people have hobbies, and even if the hobby doesn't actually cost a lot of money, the opportunity cost of that hobby is really high. There was a tweet a couple of months ago where someone asked, “why isn't there really, really expensive custom software sold for tens of thousands of dollars to one person who wants exactly that software?” And my claim was that it does exist. If you are a mid-career software engineer at a big tech company and one of your hobbies is getting your Linux configuration on your home computer exactly perfect, probably the opportunity cost of that is getting into the six figures if you've been doing that for a long time. And it's fun. But if the way that you did a hobby like that was you worked slightly longer hours and you pushed a little bit more aggressively for a raise from time to time and you actually dropped six figures on making sure that the hotkeys are exactly the way you want them and that all the drivers are doing what they're supposed to do and that you can see your smart camera feed and get alerts if something shows up – if they paid for that, that would look weird. But if they pay for it in time instead of in money, it seems normal within that world.
Patrick McKenzie: I have a slightly different point of view on this one, actually, because I think that custom bespoke software with one user absolutely does exist in the world; I believe this because I have taken people out to dinner in Tokyo who are in the business of producing software with that description. [Patrick notes: And frequently enough that there is still plausible deniability in this anecdote! I was talking about some other guy, Bob!]
Custom software for tech billionaires
Without giving away trade secrets for anybody, it's been publicly reported in a number of profiles of lifestyles of the rich and famous that some, without loss of generality, tech billionaires have very complicated smart home setups. Those smart home setups are not something where it's just like, “OK, plug in an Ubuntu CD at the end of the day and you're done.” They require what is effectively a small team of engineers to do an initial implementation, and then ongoing maintenance work – at the end of the day, that small team of engineers has an address and a C corporation and sends out invoices, and those invoices cost things in line with what a tech billionaire expects a team of engineers to cost, employed in perpetuity.
While it probably strikes a lot of people in the economy that it is ludicrous to keep a team that probably invoices you $1 to $2 million per year on the payroll indefinitely, just to maintain your smart home automation, anyone who has real estate pays some amount of money for ongoing grounds-keeping work, etc. – this is just that number scaled up. Potentially this individual might think, “Well, I and people who are close to me, and whose opinions I care very dearly about, spend a lot more time in the parts of the home that are subject to automation than we do looking at the green, green grass outside. But it will be green, green grass – and whatever the number is that gets that accomplished, I will pay that number. (Or, someone at my family office will pay that number – I will never have been confronted with this decision after initially green lighting the, ‘This is the guy? OK, yeah, give him what he wants.’)”
Byrne: I guess there are little pockets of the software industry that have that kind of dynamic, where there's a small number of people who need it and they have very, very particular needs – and they can absolutely afford to pay a lot for them. At hedge funds, for example, there could be a team of five people who are trading some particular thing, like the volatility of electricity or prices or something like that. They have particular information they need displayed, and they need that displayed reliably; they're making very, very expensive decisions if they make the wrong decision on the basis of what showed up on their screen.
It does make sense for them to basically have someone working full time on just building and maintaining the software just for them. That person is partly paid to build high-performance software that rapidly charts the right things and displays the right headlines and that kind of thing, [but] they're also being paid a lot to get yelled at. If something breaks, then the time between when it breaks and when it's fixed is time that money is not being made, and potentially time that money is being lost; it's good to have someone feel like it was worth it to get screamed at for that kind of mistake.
A lot of those jobs can be incredibly high-stress, but it's also kind of cool: it is actually a craft rather than a manufacturing position in software. You are not stamping out, building something that they're going to be a hundred million copies of and which a lot of people are going to use in roughly the same way – no, it's something where you actually see the end product and it is exactly what one specific person or one small group of people really, really wanted. (Hopefully.)
Patrick McKenzie: It's interesting we go between those poles over time. Way back in the day it was literally the case that all software was an individual person taking out the punch cards and writing one program at a time. Those programs largely did not survive for very long times.
Evolution of website development
Then the way that software was delivered… evolved over the years, to put it mildly. Every website used to be clawed out of the ether by an individual engineer (or an individual middle school student like myself) deciding to take it upon themselves to create a website for a particular organization. After it became obvious to businesses that having an internet presence was very valuable, some rather small businesses ended up contracting out bespoke software development work for the first and only time in the existence of those businesses’ history. This was the dominant way that, say, local restaurants, for example, got their websites made for the first few years of the internet.
Then we had the technological substrate change. There was the introduction of WordPress, and so you would pay a designer to make a custom WordPress theme for you if you were maximally desirous of having that happen, and the cost of the website would decrease. Down from there were restaurants that were shelling out tens of thousands of dollars for websites back in the day and then it became more of a, “pay $500 for a designer to customize WordPress and perhaps another $500 to get it up and running for you and perhaps $50 a month to host it and listen to your panicked emails about ‘why isn't the website up’” – and then that has largely (but not entirely) been supplanted by large platforms which give people a far less custom-made experience, but which are leveraging software that can be amortized over millions of businesses, whether those are platforms that have built-in distribution like the DoorDash, etc. of the world, or whether they are large software businesses, “website builders,” which have an encoded method for stamping out unique-looking but fundamentally identikit websites onto the world. A fun little business to be around the edges of.
Byrne: Yeah, it is also… I guess we lose, we do lose something if they're not these bespoke custom made things. There used to this parental complaint about restaurant websites, that if you just really wanted a flash-based experience that would show you pictures of very happy people dining at the restaurant that you would absolutely want to go to their website – but if you actually wanted to know what the menu was, what the hours were, how to get a reservation, etc., their website was just not actually the place where that information was stored.
So it has been nice that just like, your universal basic restaurant website now has a place where you click to see the menu, and a place where you click to see the hours and location.
One of the forces there is that the group that is most incentivized to create really clean, usable websites for restaurants is the service that's going to do the delivery, whether that's the service that just handles the e-commerce side of it or does the actual fulfillment. So in some sense, the world of restaurant web design has gotten really, really good because now DoorDash and Uber Eats are A/B testing which designs work at massive scale, and they've learned a lot.
With them, because they control a lot more of the transaction, you get to see little details about what actually matters. For example, DoorDash will sometimes offer a discount if you upload a picture of your food. They've basically outsourced the fine dining photography part of the restaurant media presence to the end consumer of that food, which is a lot more possible now that we all have really excellent cameras in our pockets at all times.
I've always found stuff like that really interesting. The little tweak to the transaction in order to get the customer to do something that is value added for the seller, but very, very hard for them to scale.
Restaurant websites and delivery apps
Patrick McKenzie: I've also noted some interesting transactions where… people might not appreciate this until you've attempted to do a large data gathering operation at material scale, but when you are dealing with businesses where the owner or account representative might be a difficult person to get on the telephone, either because their hand is on a fryer or because they are not operationally involved in the business, – they have staff to do that – they might still be the only ones that can give you an authoritative answer with respect to what the menu is.
It is like pulling teeth to get some signal on, “okay, you sell spaghetti for $12.99, but what is the spaghetti really?” One of the things that the ordering platforms have recently started doing is producing AI-based descriptions of spaghetti for those four customers in the United States of America who just do not understand what spaghetti looks like. And I say that laughingly, but, you know, I have arbitrarily high confidence that it can be A/B tested: perhaps adding two sentences of copy to the word spaghetti increases the sales of spaghetti. I would not bet against that at most odds offered to me, contingent on the platforms deciding to make this product improvement.
Then there's an interesting escalation ladder you can go to, where if we get minimal information from the restaurant, we can impute details of what they offer and put in that AI placeholder, and then ask our customers to act as a distributed intelligence-gathering network to take a photo of the spaghetti as it is delivered. Then if you want, you can loop it right back to, “Hello, ChatGPT, this is a photo of spaghetti. Does it have a meatball on it? We expect meatballs, but we're not sure. Please tell us and do that at material scale without needing a team of people to tag photos internally” – which, by the way, is absolutely a thing that tech companies have done internally. That is the genesis for Amazon Mechanical Turk as a product. For those of you who haven't used Amazon Mechanical Turk professionally, the name harkens back to the Mechanical Turk, which was a machine that could play chess with an arbitrary level of artificial intelligence back before electricity was a thing. (Spoiler alert, the large language model that learned how to play chess was a human who was sitting inside the chess box.)
Amazon found itself routinely having a business problem of “we need arbitrarily large, burstable amounts of the smallest amount of human labor that can actually be called labor, the thinnest slice of cognition that is still cognition, and we need to throw it at a bunch of problems and the mix of those problems will change every day.” So Amazon built a tool for internal use. Then it was like, “hey, we are not the only business in capitalism that has exactly this problem,” and they did the typical Amazon thing of, “let's externalize the service and turn a cost center into a revenue center.” A fun thing to play with.
I know there's many things to speculate about in the future with regards to how LLMs get used for most workloads, but one of the things that a lot of philosophers on that subject haven't done is look at the things that real businesses are offering real money for right now: one penny to identify if the following photo is in the right physical orientation, because we didn't want to hire a team of vision researchers to write an algorithm to do that.
That job, thankfully, is going to be automated by a for loop in an LLM in the very near future; progress against certain monetary prizes is an implicit signal that we haven't had major breakthroughs in cryptography in the last short while. [Patrick notes: The historical art on this was the RSA Factoring Challenges but the cryptocurrency community has extended it, in a frankly cyberpunk fashion. Of course, if one really gets good at factoring, one can simply post a message on the Internet co-signed by google.com and nsa.gov saying that you’ll accept your Nobel Prize after the world economy migrates to a new encryption standard.]
The front page of Amazon TurkTest is implicitly a signal: “this is above the bar for commercially available large language models at the moment.”
Byrne: Yeah, I guess there's the temporary period – I don't know how temporary it is, but there is that period where the mechanical turk behaviors that used to be more of a cost of goods sold, turned into capital expenditures for the buyer because now, when you think of that category of problem where you do need some minimum quantity of intelligence – you really don't need much and you don't want to pay a ton for it – you can think of that as less, “we're going to continuously pay to solve this problem” and more as “we're going to buy the training data that we need, produce the training data that we need to automate the solution of this problem.” I guess my suspicion would be that for Mechanical Turk that, over time, more and more of the tasks will be reflective of that.
You can actually use that as a way to look maybe six months in the future at what AI will be capable of. I guess you have to discern a little bit – both from thinking about that use case, and also thinking about trying to use an LLM or trying to use some other AI tool to solve it yourself – you need to think about that a little bit to figure out which of those are just kind of permanent, but there is a large set of tasks where, if you can describe it clearly, an intern could do it, or Mechanical Turk could do it, or ChatGPT could do it. Over time, I think the set of things where “you don't just put it into ChatGPT, it's a little bit harder than that,” that set does dwindle. The way that it dwindles is that people get the training data from things like Mechanical Turk – and I know there are plenty of other services that also do that kind of training for exactly the same purpose.
I do want to rewind slightly back to the delivery apps business; I think it's still a really fascinating business and just lots of fun to think about. One of the things I think about with that is that if someone starts a restaurant, their one location, they are probably (hopefully) good at some combination of cooking and just hospitality generally, giving people a nice experience at dinner. Running a successful restaurant does involve a lot of other competencies, some of which you can perfectly outsource – you're not setting up your own electricity and plumbing, those things are provided by third-party service providers and the pricing is very transparent, very easy to understand – but there are some things like designing the menu where you'd have to be very, very fortunate to be a really, really good chef who also knows exactly how to price things on the menu, how to arrange your special offers and, you know, in what order do you put things on the wine list to maximize the amount of money someone is going to spend?
Larger chains can afford to invest a lot of money in that. I like to go to some of the larger, not super cheap chains sometimes and just look at the menu and think to myself that when the jalapeno poppers special got put on this menu, it was like, that was what someone bragged about when he came home and his significant other asked, “How was work today?” He gets to say, “Finally, after months of fighting for it, I got the price of the jalapeno poppers down to $13.49. And yes, they are zesty.”
That stuff, you know, it doesn't really make sense to think about that really hard if you have a single location. If people like the staff, if you’re actually able to make good food that people enjoy, you don't have to care that much about those details – but if you are someone like Red Lobster, you have to care very, very deeply about all these little nuances and all the cross-elasticities of different items on the menu and whether or not there's something on the menu where most people won't buy it, but a party of six might choose to go somewhere else because one person in that party does want an entree that's vegan, but that does not have tofu or whatever the weird constraint is.
When restaurants move to using larger platforms, the platform can often do some of that analysis for them. (This is going to show up a lot more at the DoorDash level than at the level of whoever is doing the QR code based menu app.) Over time, I think they will be collecting enough data that they can start making more recommendations to restaurants on things like, “you really should be charging more for this; you probably don't really need this on the menu at all.” Some of that does require knowledge that these providers wouldn't have – but increasingly as the provider of the menu is also someone providing the payments, they can actually look at scale and say, “Okay, your diner is your own special thing, but also there are many, many diners; there are enough diners that we have statistical confidence on whether there should be two hash browns or three when someone orders this particular special.”
McDonald's take rates
Patrick McKenzie: I think that in some ways the platforms here, or B2B SaaS largely, are recapitulating the role of the franchise in a franchise/franchisee arrangement. Historically you didn't necessarily have to be a genius at site selection because McDonald's had a department of people to do that for you; you were being selected to do quite a bit of restaurant managerial labor and provide capital on behalf of McDonald's Inc.
The history of franchising is long and wonky, but part of what you are buying in return for your franchise fee (and the commitment to only buy your groceries at McDonald's in the future [Patrick notes: and also accept McDonalds as your landlord—almost uniquely among franchisors they use real estate as the ultimate Sword of Damocles against franchisees]) with the hidden embedded margin for McDonald's Incorporated is you were buying their ability to employ the best expertise on the market, for a very long time where you could not afford the best expertise on the market. You might not even understand you needed expertise in particular fields. And even if you were able to afford it, you could only afford it in bursts – you could not continuously have someone in charge of menu development, whereas McDonald's corporate definitely can.
So this function gets redirected into platforms and B2B SaaS companies – platforms in the fashion that we've been describing, and B2B SaaS companies because they're largely in charge of encoding a method for doing a business process. Usually version 1.0 of the application is, “well, you probably have some method for doing, I don't know, travel expense reporting at your company; we have a jumped-up spreadsheet that will do that in a web application, and you won't have to share Excel files over email anymore.” But increasingly, the company will become experts at the problem domain and kind of eat around the edges of, you know, “If you were doing this seriously, you'd probably have someone who is in charge of negotiating corporate rates with the various vendors that you use the most. Oh wait, we are doing this seriously! We have someone who is in charge of the rates!” And now part of what you're paying for the SaaS price every month or every year is essentially hiring those humans on your behalf without having to employ them directly.
I think, by the way, that once you start spooling out those sort of nuances, the take rates of platforms look very different. When people compare a 15%, 20%, 30% take rate to, say, credit card fees and think that, at the margin, the only thing the platform is providing is a customer, and then they're absorbing the payment expenses, then the take rates look very large – but if you compare them to franchise fees, the take rates suddenly look quite normal. And they don't charge you what it costs to open a McDonald's.
McDonald's will tell you, “yeah, you will need $3 million upfront cash on the barrel to do this, of which we will accept up to $2 million of bank financing – but $1 million cash on the barrel has to come from you.” So capitalism has found a way to employ experts without requiring you to put down a million dollars when you open a DoorDash account. I think that's kind of interesting and fun in a lot of ways.
Byrne: I think there's an interesting balance sheet story there where, yeah, you can do a sort of capital-light version of the franchising model, because the capital's already there. The restaurant already existed, so you don't need that incremental investment, you just need to add the sort of layer of franchisingness on top of it. But there's also an interesting dynamic on the P&L side, if there are external service providers and they know that they get pretty decent margins from dealing with some of these small companies. The margin picture for dealing with the small companies is, “it is a pain to get that restaurant on board – it is not 10 times harder to get a 10-location franchise to sign an agreement than to just get a single location to sign an agreement – but once you have them on board, you can charge them a lot, and they just can't be that price sensitive; it's just not worth their time to be that price sensitive.”
Patrick McKenzie: It tends to be a very sticky product.
Byrne: Once that relationship is being routed through a platform, the platform can be in a position where they take away $10 of the supplier's gross margin that gets them enough pricing power to raise their prices by $1; that's pure margin for them, and it's eating into a business that, because of those sales and marketing costs and just because of the operational expense of dealing with lots of fragmented customers, it's a fundamentally lower margin business.
You do end up with a lot of intermediaries getting nervous about what their gross margins look like in the long run, because there's now someone very, very sharp implicitly negotiating on behalf of their customers. I could see that going in a lot of different directions. Maybe DoorDash decides at some point that the right strategic move for them is to acquire a big food distributor so that they actually own that piece of the relationship; then what they get to say is, “Ok, we can think about these cross-elasticities. Maybe if we cut the price of tomatoes for our customers, it actually leads to so much incremental spaghetti and pizza consumption that we actually capture more of the upside than the downside.”
I think one last point to make on the whole question of A/B testing menus, systematically analyzing every little bit of the unit economics in order to eke out more margin: I suspect that this explains a phenomenon that you pointed out in one of your Coming Back to America tweets about portion sizes. I bet when you A/B test it, very few people are annoyed that they left 300 calories on the plate, but some set of customers is pretty annoyed that they were still slightly hungry or could still have eaten another bite after they finished their burrito. The marginal cost of those ingredients is really, really low; your labor cost for making a burrito with 100 more calories is basically zero. So if you A/B test it, you actually A/B test your way into bigger portions. (Maybe also more food waste, probably also people just eating a whole lot more when they sit down at one of these restaurants.)
Patrick McKenzie: Yeah, I actually read a review of Chipotle recently that made exactly that point, and I remember reading the review being a little bit confused.
(Disclaimer, was a long time Chipotle shareholder, it worked out kind of well.)
Chipotle is Chipotle in the same fashion that McDonald's is McDonald's. What is the theory of the mind of a user entering a text into a text box about McDonald's? What social or emotional need does that fulfill for them?
The user's review was, “They were not generous with their portion of rice on my burrito, three stars.” I thought, you know, this is just going into a pool of reviews which are aggregated across all Chipotle organizations, etc.
Then I thought, well, possibly this is like the anthropology of using apps, where if you ask someone to leave a review, some percentage of people will answer that question – like, “I am compelled to use any text box put in front of me with a question above it.” So at any rate, it does seem – shockingly to me, as someone whose stomach was resized by 20 years in Japan – that there are people who do really want that marginal 300 calories.
Restaurant groups
One question to which I don't know the answer, because I spent 20 years in Japan: do we have an institution between the mom-and-pop restaurant on one end of the scale, and large franchise or corporate-owned chains on the other end of the scale? Is there an institution of restaurant groups where there is a brain trust that owns 12 or 15 locations, often in a variety of concepts, and they provide design, menus, pricing, maybe clientele selection and real estate selection, but they don't individually operate each restaurant?
It occurs to me, Lettuce Entertain You in Chicago is kind of that. They have a Japan concept and a China concept and an India concept and one-offs of all of them, as far as I know. [Patrick notes: Thoroughly enjoyable in the category of “reasonable middle class restaurant meals in downtown Chicago”, incidentally, if anyone ever wants to do culinary econoethnography to fact check this podcast. My kids particularly enjoyed the ramen place.]
Byrne: Union Square hospitality group is kind of similar to that – the Danny Meyer complex of non-Shake Shack restaurants might sort of qualify, where it is a bunch of different concepts and in that case, as far as I know, it is common ownership, but they aren't all part of the same thing; they're owned and managed by the same broader organization, but they're of different genres. I don't actually know how much that kind of thing exists, although I do get the sense that there is this ecosystem of restaurant experts who are familiar with some particular category of concepts, some general aspect of that business, and they will drop in when there is a restaurant idea that is turning into something that might scale nationally.
If you read the Cava S-1, the IPO prospectus, and look at the biographies of some of the managers, it is clear that there are people who have done this kind of thing before. They are just always on the hunt for a concept that has, maybe three locations and they're all very busy, it’s three locations in one city; you sort of do the math on Cavas per capita in the Washington DC area, and do the math on how big Washington DC is compared to the rest of the country, and suddenly you have some pretty exciting numbers.
I think that that is probably the closest thing to that. Maybe it actually says something good about the American consumer, our taste in general – that the skills don't transfer all that well. If you are taking a concept that works like it's Mediterranean fast casual, Mediterranean slightly premium fast food, that you probably need to spend all of your time thinking about that exact concept and not trying to think of, “Okay, what are, what are the other food categories that could also be put in this bucket?” You'll probably drive yourself crazy if you're trying to do Mediterranean fast casual over here and also trying to do Indian fast casual over here and Korean fried chicken fast casual over there.
So maybe, yeah, maybe that is kind of where it starts to break down. Then I guess like the Union Square hospitality groups at Daniel Meyer's set of restaurants, they are all just really, really well done. It is basically the McDonald's equivalent for upper middle class people in New York where, you know, it will be a generically good experience. It may be one of the better meals you had, but probably not. It's probably just a very, very good meal, and the service will be impeccable; you will get exactly what you expect.
I guess that gets into another piece of just restaurant economics: part of what you're selling is just certainty. You offer a lot of certainty to the people driving a minivan full of kids home. They know the kids, their blood sugar is plummeting, it will continue to plummet, and so the job to be done is just get them something, get some food in them before things get really, really bad.
The other category I think about a lot is with restaurants to have a business meal at. I used to obsess over this when I was younger, trying to figure out, like, “What am I signaling if I choose a particular restaurant? Does a chain say something? If it's not a chain, do I actually know if it's a good restaurant or a greasy spoon restaurant?”
Yelp was just a lifesaver there because they recognize that as a category that people search for, and they had landing pages for it and everything. But I think that is what companies are trying to solve for when they do that – I guess with business, part of what you're trying to solve for is, you do actually want to slightly signal that you are a prosperous person who doesn't really have to worry about money. You do want a place that slightly overcharges, given the quality of the food, because you're going to be picking up the check and you need to just casually not look at the numbers and just pay for things.
(I guess it depends on what kind of business you're doing. I suspect that if you are having a chat with someone who wants to be a client for your CPA business, you probably carefully scrutinize the receipt and make sure the numbers add up and things.)
But yeah, thinking about the job being done, it's not just a calorie delivery service.
Patrick McKenzie: Yeah, I used to have an opinion, a bit of advice, which I'll repeat for the benefit of anyone who can use it: if you are in business for yourself, it's good to find a local restaurant where you like the vibe and you like the people, and start making a habit of doing as many of your business dinners there as possible and tipping generously. That will typically get you formally added to the list of whales. Then you get the joy of being the person whose consumption is not obvious to everyone else in the restaurant, but there will always be a table available for you. That's a thing you can either explicitly ask for, or the culture that is hospitality workers and managers will simply arrange for you. I often think that it coheres with the aesthetics of many people in software and similar: “we get to be patrons of the local gastronomic arts by finding a restaurant we like and using some of the relatively abundant software dollars to subsidize the gastronomic arts.”
Zooming in on another thing you said, one of the jobs to be done of the QSR, quick-service restaurants, the fast food industry, has historically been decreasing the amount of variance associated with dinner – radically compared to the home experience, and then quite a bit with regards to the traditional restaurant experience. I think one thing that people don't appreciate is… there was a complaint that everything is getting suckier over time. We all went to McDonald's as children. McDonald's, observationally, does not deliver precisely the same thing that they did 40 years ago. And they did not 40 years ago deliver precisely the thing that they delivered 20 years before that. But partly it's just because of the good kind of inflation, inflation of expectations in the United States of America: we've become a richer nation, and part of becoming wealthy has been that a lot of people in many socioeconomic classes have kind of promoted themselves out of cooking (and for good or ill – there's many people who are enormously attached to the cultural ritual of food preparation). But much of America essentially does not do it anymore and I think that will be almost normative within our lifetimes. Be that as it may.
Rather than McDonald's being a once a month treat for middle class people, it is now a major source of calories for large portions of the population. Partly due to that fact specifically, middle class people have promoted themselves into a different class of restaurant for the Shake Shacks and Chipotles and similar of the world for the fast casual experience, which does not discriminate along some axes, but does discriminate along others, including willingness to pay a few dollars more than McDonald's – a rather small, but material number of dollars more than McDonald's.
Then we've built a gradient on top of that too. You can get Chipotle, or you can get Chipotle delivered to your door in 30 minutes. There is an opaque, but very real fee that is a difference between those two numbers. Some people decide, “You know, I'm more of a ‘stay here, continue working, and then get the Chipotle delivered to me by a courier’ person than I am a ‘take an hour out of my day to walk laboriously down to Chipotle so that my servants can prepare meals for me’ person.”
The optimistic upshot is that the amount of societal wealth that quick-service restaurants represent is vastly underestimated by most people. We live like medieval kings and look at that situation and think, “those miserable-off people at McDonald's who are only having servants spring into their whims at the touch of a button to bring them their next diet coke.”
Tech company cafeterias and employee benefits
Byrne: Yeah, I was very much familiar with that norm growing up where we did go to restaurants, we did get only Chinese or pizza delivered. (Those were the delivery categories of food, and they were kind of a treat.) I do remember feeling slightly emotional, slightly like I had made it the first time that I seamlessed a meal to my desk because I was working late and the company paid for it. I felt like I'd actually reached this different category of existence where feeding me was just economically profitable to the organization that employed me and was not like a little giveaway or perk.
There is this economic relationship where they are basically buying more hours of office time – and getting a really good deal. Like, the return on investment from something like the Google cafeteria is just phenomenal because for a lot of people, instead of “they're hungry and so they go home and have dinner,” it’s “they're hungry so they eat and they talk to their other Google friends and maybe get some ideas.”
Patrick McKenzie: I've seen arguments that the thing that distinguishes the tech cafeteria from a cafeteria in most of the nation is that food is of subjectively higher quality, and virtually universally free. The institution that is a tech cafeteria was reported on many, many times, often by people complaining (adjacent to that reporting) of “these entitled tech workers, they don't even expect to pay for their own food, etc.”
Some people have advanced the hypothesis that this was a zero interest rate phenomenon – that now that money has a cost again, that salaries will be slashed, people will be forced back into the office, and there will be prices on things in the cafeteria or they will discontinue that service. That is not a reality-entangled point of view at all. The economics are just so absurdly attractive. Every company has a number and few companies disclose it, but fingers to the wind, it's probably like $15 per beneficiary per day of what it costs to provide those services, and $15 next to the total compensation of an engineer – even at a place that is not paying Google rates for engineers – is not a lot of money.
Cynically, recruiters have documents that this is not an accident. There's a document describing the plan: if the candidate pushes back on salary, then tell them that the value of the free cafeteria food is worth $10,000 a year, and say, OK, so you wanted $15,000 more in salary; I'll give you free cafeteria food and $5,000 more. Many relatively sophisticated professionals do not see that as a line – and the reason you know that they don't see it as a line is because there are documents describing this strategy. It works all the time when you employ it at scale.
Sorry, there’s a little bit of labor activism still left in me.
Be that as it may, cafeterias are here to stay forever for the, descriptively speaking, the best-compensated employees in capitalism, and you should just expect the economics of that to continue pushing down as the borders of the upper middle class in America push up.
Byrne: Yeah, I think that's right. I think the way that employees think about free company-provided food is a really good indicator of where the company's culture is right now. The optimum for the company is, “we're paying $15 for you to work several hours, and there is just not a country we could outsource software development to where we would pay less than that per hour – and certainly not for the caliber of people we get.”
But then you can also do the same pattern that you would do without the employee subsidized cafeteria: you leave around dinnertime, and what you can do instead is you come to work with an empty lunchbox instead of a full lunchbox and you load up enough food for yourself and your family and then you roll out the door.
I think that is the kind of thing where it was probably more common in a zero interest rates, growth-focused world where there was a bigger risk from not having enough headcount than there was from having too much head count and letting the culture diverge a little bit from what it had been.
Google's business model and economic feedback loops
I think at mature companies, it's also harder to have people fully bought into the mission because a lot of the mission has been accomplished by that point. Google has organized a lot of the world's useful information. I'm sure there's more to do with that, but there's also a lot of the things that people do at Google which are not organizing the next piece of information; they are optimizing either the cost of delivering that information or the revenue that you can derive from that – which is important. The reason venture capitalists were very excited to put money into Google was not the mission. The mission helped, but the exciting thing was realizing this is a product with a lot of usage and figuring there was probably a good business model.
It would be fascinating to read the original investment memos for the Google Series A. I don’t know the extent to which they realized that Google actually had just probably the best economic model ever, where you have all kinds of these positive feedback loops: you're getting more data and so you can serve better results and better ads, and the more searchers you have, the more ads you can display; you have higher bid density, so everyone's incentive is to bid roughly what they think their incremental margin is.
Not only are you making more money, but you get just really, really good intelligence on the profitability of different sectors.
Sometimes that's irrelevant. Google is not going to open up a law firm to do mesothelioma class actions, but if they noticed that people are paying a lot of money for leads in, I don't know, the office software space, for example, that might on the margin affect their decision to offer such services.
So yeah, it is an amazing model and it is useful for the world to make sure, when you develop an amazing product, that it is extremely profitable to do so, so that people who are just dollar maximizers want to invest in amazing things.
But yeah, less inspiring. It is just more likely that you wouldn't feel especially bad as a Googler in 2024, taking advantage of some of the benefits in a way that's clearly not what they were designed to encourage – whereas in 2004, I suspect that a lot of people would have really frowned on blatant attempts to exploit the Google system.
Patrick McKenzie: So fun fact, at a variety of large tech companies, the tax department is frequently tearing their hair out about that specific exploitation – less because of what they think that it does to the corporate culture (although there are other people in the company that very much worry about the corporate culture) and more that the tax department specifically worries about what the IRS will think about it.
The expense of providing a cafeteria is a business expense if you are providing the meals for consumption, at the business's convenience, and on the premises only. If you violate either of those two prongs, then it's simply extra compensation that you've provided your employees, and you have now not paid social security tax, etc. on that compensation – or it gets disallowed as a business expense and you've underpaid taxes, yada, yada, yada.
Early Google investment anecdote
So jumping off another thing you said, I don't know if we've ever seen the original investment memos for companies like Google or Facebook – although that would sure make for fascinating reading. But there have been a few oral histories of those early investment rounds published on the internet. One factoid I remember from it was that Ron Conway had to arm-twist the VC firm – I'm forgetting which one – which wanted to be in the Google series, before it did the not too atypical thing in VC and wanted that a little less, and they were about to go out of the deal.
Ron Conway called them on the phone and threatened them, saying that if they did not make good on their handshake commitment to invest, Ron Conway would fill the entire round himself. It's one of the canonical Silicon Valley stories of, like, threatening someone with yourself having a very good time. But one could not have known that it would play out exactly that way on the day that he made that offer/threat.
But a fun bit of Silicon Valley history that recedes into lore and memory… and perhaps when one tells the story ex post it does not match exactly the events of the day, but such is history.
So is there anything else that has been on your mind recently, Byrne?
Byrne: What else been on my mind recently? I'm going to cheat and look at what I've posted recently and see…
Writing as a memory aid
Patrick McKenzie: This is one of the best reasons to write things. One of the reasons I have kept a blog for a number of years is that it effectively functions as a publicly visible journal for someone who has occasionally sporadic memory of what I was supposed to be working on this month. It's like, “okay, flip back six months ago to where I wrote down my priorities.”
Byrne: No, that's exactly what I do, yeah. For a while I wasn't writing much online and one of the places I started writing again in my current phase of writing all the time was Goodreads.
Part of the reason for that was that I would look at a bookshelf and remember having read all the books on the bookshelf, and I would also feel like I couldn’t name five discrete things I learned from that book. I could relate the plot to you if it's a novel, but I couldn’t tell you the details.
It felt like I was getting a more insightful experience in the process of reading it than than whatever I retained, and I know that's a perennial problem that anyone anyone who reads a lot has, but writing on Goodreads – part of what it did was it just it did force me to actually think about, “What did I get from this book? What do I agree with? What do I disagree with?”
Sometimes you will read something and just gradually misremember it until you've almost inverted what the original point was, and it's good to have some kind of reference point. I think it helps with memory to have not just ingested something, but to try to explain it. That is a very, very useful habit in my experience; I highly recommend just having some kind of written record of what you've been doing and what you were thinking about and when.
It's also that I've written enough that I have forgotten a lot of things that I wrote – I remember more, but I don't remember everything – so sometimes it's weird to be looking back at something unrelated and end up finding something just delightful. I will occasionally find things I said about AI in 2020 that I feel like some of them aged poorly, some of them aged really well. I think at one point in like 2020 I had some joking allusion to, “I don't think my job is going to be replaced by a computer just yet, but I'm gonna be keeping a very close eye on GPT-4.” (It hasn't replaced me yet, but it keeps getting closer.)
So yeah, it's a good habit.
Using ChatGPT for memory assistance
Patrick McKenzie: I definitely endorse the notion of “write notes on what you've been reading recently, write notes on what you've been doing, etc.” – even if you don't publish them, they'll be super useful to you in the future.
It has been surprisingly useful for me to ask ChatGPT (or one’s LLM of choice) to riff on what you would have written as notes for a particular book or similar. I was at a conference and a fellow attendee approached me and said, “I know you read the series, The Dagger and the Coin, because you wrote about it on Twitter and gave it a two thumbs up recommendation. It's my favorite of all time. Will you do a conference talk with me about this tomorrow”
I said, “In principle, yes, but due to the funny way my memory works, I remember most of the major plot points, themes, the twist, etc. – it has one of the best twists in fantasy literature – but I don't even remember the names of the characters anymore because I have the written equivalent of face blindness.” And I was like, “But this is a solvable problem these days, even if there is not a vast secondary literature about this relatively obscure fantasy series (that everyone listening to this podcast should absolutely read).” You can just go to ChatGPT and say, “Hey, who are the top 10 dramatis personae in The Dagger and the Coin? Give me a one-sentence description, a Twitter-length description of their events in the plot.” And it was like, “OK, Cithrin was the banker for the…” – Oh yeah, that's her name! She's the viewpoint character in the second chapter. And then it goes from there. Anyhow.
I'm far less worried about being replaced by LLMs and similar technology, far more interested in what the ceiling is for capabilities augmentation – assisting humans who are, knock on wood, the OG intelligence in the universe. Well, OG intelligence for local values of interesting intelligence. (Everyone can predict the caveats there.)
Byrne: Yeah. I've done it really useful for things, like if you know there are multiple examples of some phenomenon happening, but you don't remember what they were – you know that this has happened, you know, three times before, but you don't remember when and you don't remember who it happened to, ChatGPT is a really good intuition pump for that, although you do have to check every single thing it says because it will get things wrong, especially about really obscure topics.
I had a weird experience recently, where I was writing about buybacks and I knew the general history where they used to be rare and that there used to be these legal questions about, “Can you actually do a buyback or is that market manipulation?” The SEC eventually said, “Okay, here are the times when it is definitely not market manipulation, so just go to town.” I asked ChatGPT about earlier instances of this and it told me they really didn't happen before the 1950s. Then I really prodded it and it gave me some examples, and then I checked those examples and they just didn't actually seem to correspond to real things that had happened. But in doing that research, I ended up Googling and when I Googled just the concept of buybacks and just random dates in the 1930s and 1920s, I found a blog post that had a table that was showing high and low valuations and dividend yields and price to book and things like that for various oil companies in 1932 or something, and I realized that a book that has that table probably has many other similar tables and is probably compiled by someone who is very interested in giving an extremely detailed history.
So I bought the book and it was great. It was basically a history of US financial markets from 1929 to 1933, titled “The Crash and Its Aftermath: A History of Securities Markets in the United States, 1929-1933.” Most of the book is basically tables like that and explanatory text.
But throughout the book, the author does mention numerous times that companies were actually buying back stock, and talks about how… if you know your 1920s and 1930s stock market history, you know that there were these proto-mutual funds that would borrow a ton of money and speculate in stocks. They often traded at a premium to the value of their holdings. (Goldman Sachs had two of them, and those had cross-linked, levered ownership – and then they collapsed. If you love complaining about Goldman Sachs, you have to reach pretty far back to find really big scandals, but you do find those scandals.) A couple of those organizations were trading at a large premium in 1929, but by 1931 or so they were generally trading at a discount, so some of them were actually pretty aggressively buying back their own stock.
It was kind of exciting to see that this did happen. It's been written down, but also it's forgotten lore. Can’t even ChatGPT; ChatGPT mostly denies that this ever happened, but it does actually exist. You get to feel a little bit like Gandalf in the early scenes of the Fellowship of the Ring, where he's riding off to Minas Tirith and digging through the archives, looking at these ancient books that no one has looked at for hundreds of years. You get to do a little bit of that if there's something that's not in the ChatGPT corpus. It also makes me slightly more optimistic about the claim that these LLMs are not actually trained on just huge volumes of pirated works, because this… it might have been pirated by somebody; it doesn't seem to exist on the usual suspects – but it's definitely not in the training data.
You do want to be somewhat cautious. But I think as just a way to remind yourself of different events that you know happened and about which you're forgetting some of the details, it's great. It's the tip-of-your-tongue machine.
LLMs as writing and coding aids
If you think about the writing process: if you know what you're going to say and you have the facts on hand and whatever you're referencing is in a tab and you don't have hundreds of tabs open at the moment, the writing process can be really, really fast – it's not necessarily as fast as speaking, but it gets pretty close. But sometimes what throws you out of that flow state is that you have a question you know you want to answer and you have the meta-question of, “Will I get the answer in two minutes or two hours?” It's just frustrating to have to skip into a different level of abstraction and think about optimizing the writing process instead of, “what is the next word going to be?” I think ChatGPT helps keep you in a flow state.
I also noticed that with coding with an LLM, the way that it keeps you in a flow state is, you always know what the last problem you were trying to solve is – both in natural language terms and in terms of code. So it’s always a little bit faster to just jump right back into where you were, whereas if you remember the name of the file you're working on and you just open it, you are at the top of a file and you have to look through a bunch of things and try to figure out, “Okay, where did I get stuck? What are the linked dependencies to other things that got me stuck?”
Patrick McKenzie: Yeah. Also, just as a pro user tip for other users of these things, it is one of the first tools where you can ask the tool directly, “Hey, what should I do next?” Even a wrong answer to that question is much better than no answer to that question, if you’re staring at a blank screen, or you lost your train of thought, or are context switching between seven windows to chase down that the anecdote that you wanted from the congressional testimony in 2012 that has blown out your own context window for what you were doing – it's effectively having an infinitely patient coworker that you can bother just to have someone to be able to bother and not be, like, stuck in your own brain about something.
As someone who is aspirationally patient and has been a coworker before, I think one of the under-measured productivity increases that we'll get from LLMs is simply less people tapping a coworker on the shoulder to function as a proto-LLM for things that computers are now perfectly capable of doing, saving both the one minute of that interruption, but also the vast context-switching costs that engineers and similar creative workers go through when their train of thought gets derailed and they lose 30 minutes of state built up in their head.
Byrne: It's definitely good for that. It’s like code comments but for your thought process, where you have these periodic check-ins and you sort of explain to yourself what you're doing. I think that this phenomenon is a little bit rare with Google searches, but it does happen sometimes with ChatGPT: as I'm typing in the question, I will realize I know the answer and not have to ask the question. It’s been good for that.
Children's interaction with ChatGPT
On the topic of chatbots: One of things that's been very interesting to me is watching my kids use them, because to my kids, LLMs have, of course, always existed. My six-year-old once looked over my shoulder while I was Googling something and she asked, is that ChatGPT? And I've never heard something that would terrify Googlers more than that.
Patrick McKenzie:
“No, it's much worse and it has ads in it!”
[Patrick notes: I still use Google Search very frequently, but as someone who was almost certainly a hyperconsumer of it for decades, I find myself needing to remember it is an option. This is partly downstream of ongoing deterioration of the service for the sorts of searches I do most frequently and partly ChatGPT or YouTube or Twitter or Reddit being better entry points to Ask the Internet.]
Byrne: But it's interesting to look at how they interact with it. One of my kids wanted to have a babysitter and hadn't had one in a while; she asked ChatGPT what to do with a babysitter, and it gave her a couple suggestions and then she said “that's not enough,” and asked, “please tell me 100 things that I could do with a babysitter.” It gave her a numbered list of 100 activities that a kid could do with their babysitter, and then she responded to that with, “that is way too many things.” I think it is just a lot more natural to view them as kind of humanoid.
No one really talks to Excel in the same way. You don't think of Excel as a buddy, but the chatbots do have something of a personality. I was very appreciative of Anthropic's willingness to give us a chatbot that didn't just have a personality, but actually had a special interest in the Golden Gate Bridge and could steer any conversation in that direction. I felt, you know, represented.
Patrick McKenzie: That was a wild art piece in a lot of ways, but not just an art piece.
[Patrick notes: If you're unfamiliar with this: Anthropic discovered that it could tweak its LLM Claude to be obsessed with the Golden Gate Bridge, and released a demo of a Claude with this tweak applied. It was hilarious, it was a mic drop demonstration of a new scientific capability, and it was... a little bit unsettling. (LLMs certainly aren't people, and the characters they play are usually thin veneers, but the reaction people had to this character made me feel sympathy with it, as someone who needed a few decades of RLHF to avoid bending all conversations to my obsession of the week.)]
We fundamentally don't understand how this technology works under the hood. It's been, what's the phrase, discovered more than it's been invented. So some big picture questions: do we work like that under the hood? Is that why folks like us end up being a little bit too interested in credit card points, et cetera, et cetera?
Byrne: Right, is there just one overgrown neuron somewhere that has lots of knock-on effects?
I think that does get to a kind of deep question about how you think about these models, because one view is that they are getting very good at simulating something – in the way that increasingly detailed image simulates a real thing, where at some point it is sufficiently detailed that it's like, the main thing that's salient is still two dimensional, and adding more detail does not actually make it feel more real.
But in another sense – going to humanize them and call them mental shortcuts – a lot of their mental shortcuts are very endearingly human. When the New York Times sued OpenAI and they had this example of ChatGPT being able to reproduce paywalled articles (but it turned out that those articles had also been copied elsewhere and were just in broader corpuses), one of the interesting things about it was that the articles, when they were generated, would be almost word for word, and they would use years correctly but make other errors. At one point in one of the articles (it was about Uber and taxi medallions and things like that) – there was a sentence, I think about how much money taxi drivers had borrowed against their medallions, how much that number had changed, default rates or something like that – and the LLM just skipped it.
That felt incredibly relatable to me because you can anchor a lot of things to years – a year is a token. 2008 means something to a lot of people. But a number like $357 million is just not a token in the same way; it doesn't have the same salience. If you are trying to relate some anecdote, you might remember the year in which a law got passed because you remember, “that was the year I moved to this place or got this job or met this girl” or whatever, but you're not gonna remember that number; you might just drop that particular sentence from your attempt to relate this story to somebody else.
So that did make me feel like they are replicating – obviously it's a different implementation – but they are replicating some things that our brains do, or maybe our way of thinking has been so influenced by language and by writing that we've actually manipulated ourselves into having brains that are more similar to LLMs, that are more similar to taking a sequence of tokens and predicting the next token.
Patrick McKenzie: Yeah, it is somewhat convenient that we already had a word for hallucination before we saw the concept demonstrated by non-human intelligence.
Arguing with LLMs and using them for research
Be that as it may, I do think whether one is a six-year-old or a professional user of LLMs, understanding that it's a tool that you can argue with – which is not a mode of interaction with many tools (perhaps for programmers, but it's largely one-sided argument when you're screaming at the compiler saying, “won't you work?”). The phrase, “No you're hallucinating, try again” has worked into a prompt for me many, many times. Similarly, when I'm using it in sort of search engine mode for the tip-of-the-tongue – who was that trader who blew up? Wasn’t it a British bank? Not Sterling… but it was definitely from Singapore and vis-a-vis futures or options probably in like Tokyo or Osaka… that's the sort of thing that it is just preternaturally good at getting. But frequently it'll say yeah, you know, that time in India, blah blah blah. It's like, “No, the salient bit of information was…” and it goes “Oh, Nick Leeson. Yeah, you know Barings Bank” and then you have to Google for the Wikipedia article to verify the particulars because Wikipedia is – well, it's less likely to make catastrophic mistakes than ChatGPT is, in my experience, for those heavily detail oriented things.
Oddly enough, Google search used to be very good for tip-of-the-tongue searches, particularly in certain domains. If you described a movie plot, legendarily it would often nail the exact movie the first time. It's been much less good for tip-of-the-tongue searches, particularly tip-of-the-tongue searches that are seeking a particular document.
I'm finding it difficult to even find particular documents by the name of them these days like my usual go-to example from this, which keeps disappearing from the internet in the obvious places, is Shorting Home Equity Mezzanine Tranches.
Byrne
(Byrne here takes a stab at the title to assist me.)
Patrick McKenzie
Thank you. A famous document that was covered in The Big Short as the Jenga scene – but there's an actual PDF. It's amazing. And every few years, whatever combination of university web pages and Scribd uploaders, etc. that had it previously, decay into bitrot. But there's always someone who has it. And Google has been unable to find that someone recently, which… the times we’re living in!
[Patrick notes: Since one should be the change one wants to see in the world, Shorting Home Equity Mezzanine Tranches will be indefinitely available at that link. It really is a fascinating historical artifact, both for the direct content (mostly of interest if you’re thoroughly versed in the global financial crisis) and in what its level of analysis and polish suggests about the relative ability of bankers versus e.g. other esteemed groups in society. (This is sales collateral and it wipes the floor with almost all reporting or government analysis of the same issues.)]
Byrne
One of my use cases for Google, which Google is bad at for good reasons, is when I read a story about some event and I want to compare it to another similar event. Google wants to weigh the recency very highly because it's clearly a query that is similar to queries people are doing for this recent event. If it's something like, “this company got bought” and I want to read about when they first got funded or “this company raised their series C” and I want to read about the series B, Google will often very helpfully correct me and know that I surely meant to look for the most recent thing.
I think that actually makes sense if the median user who's doing a query like that, the reason you would do that query is you heard about this event and you want to learn more about it. So of course, they should show you the most recent instance. I think that assuming that use case is a little bit annoying. Then LLMs have almost the opposite problem, where they're very atemporal. I guess the RLHF actually adds some temporality to them – they're very bad at telling you what the median person would have thought at some point in the past on some issue. They sort of tell you what someone today would say they would have thought if it were 1950.
There is sort of an endless present for LLMs, except to the extent that years provide – the years are tokens that tell you something about the distribution of the next set of tokens. Because of that, you can get them to tell you about these historical events and to not over-weight the recent past (which is also outside of their training window).
What that amounts to is that there's a nice synergy between the two, where Google does have the actual document written by a human being – it's often a PDF of a scanned document that somebody printed out, and you can assume it's real – and then the LLM knows about the existence of this thing and can basically give you enough information to do a query where you put it in quotes and tell Google, “No, do not correct this. I am the correct one right now. I am not the median user. I am the weird user who has this special use case.”
So yeah – together, I do think that actually ChatGPT has improved my Google efficiency.
Patrick McKenzie
I do think I would definitely agree with that. I think one of the reasons why people like us complain that Google Search has declined in quality over the last couple of years is that we are experiencing our transition into middle age and extremely senior users in the Google ecosystem – where back in 2006, we were the whales that were being optimized for in one way or another. As Google has grown their business and the median user who gets waited for no longer looks or searches like us – I think if you had a marketing meeting at Google these days and started to talk about binary operators, the Google marketing department would shoot you and then throw your corpse out of the room. Middle age… it's a trip.
And for perhaps a brief window in time, we are the folks that LLM product teams are optimizing for – or are we? Who knows? Looking under the covers of these things is somewhat difficult.
Byrne
Well, I guess it might be a feature of LLMs that they can optimize for many different kinds of users – in fact, they almost can't help but optimize for different kinds of users. There was a wonderful tweet recently where someone was interacting with the chatbot built into, I think their health monitoring watch, smartwatch or something like that – they said, please build me a to-do app in React. The chatbot said, “Let's focus on fitness and exercise,” and the user said “no,” and the chatbot responded with React code.
So yeah, they are sufficiently general purpose that you can bully them into producing the JavaScript that you want. And that does mean that they are implicitly customized for whatever use case you happen to have, unless it's a use case that conflicts with their rules about what kinds of content they can and can't return. They’re customized for everything except instructions on building explosives.
Patrick McKenzie
Well, on that happy note, I think that's as good a point as anywhere to end the conversation. Byrne, for the relatively few people here who haven't encountered you before, where can we find you on the internet?
Byrne
Sure, so my full name, Byrne Hobart. @ByrneHobart is also my Twitter handle. That is a good way to know some of my thoughts. Probably the best way to read me regularly is the newsletter The Diff. There's a free version. If you enjoyed this, I invite you to sign up and you will – if you read the archives – see things that we riffed on in this conversation, talked about in more detail, but with fewer corrections. (laughs)
You will also get more thoughts on AI, financial markets – lots on financial markets, how those work, what is new and different in them, company profiles, and much, much more.
Patrick McKenzie
All right, thanks very much for joining me.
Byrne
Absolutely.
