That time we accidentally created national healthcare infrastructure, with Dave Kasten

I met my friend Dave Kasten for real, after being Twitter mutuals, when we mutually rolled up our sleeves to fix America's inability to remember where it had misplaced the covid vaccine back in 2021. That effort probably ended up saving a few thousand lives.

I've written (and done an interview or two) about VaccinateCA before, but usually at a relatively high level, and wanted to go into the nitty-gritty of the technical, operational, and occasionally moral decisions in getting important work done quickly. Dave also works in the intersection of tech and policy, and has an optimistic and helpful perspective on that engagement, which I think many of us can learn from.

[Patrick notes: I pepper transcripts with other-than-contemporaneous commentary, set out in this fashion. I continue to think takeaways from the vaccine rollout, including some uniquely generated by VaccinateCA, are very important for our nation to digest.

I am a little bit less than reserved about describing the truth of the 2021 situation. In some places, this is unavoidably political, though not particularly partisan. And thus here is a disclaimer: while I think I'm being true to my recollection (and notes/electronic records/etc) of a chaotic time, my opinions are not necessarily shared by Call The Shots, Inc. (the 501c3 which ran the VaccinateCA project), or by other board members, employees, donors, or volunteers to the effort, or by other organizations I have been or am affiliated with.

Also note that sometimes these notes will appear in the transcript in the middle of Dave talking and this is me reacting to what he is saying, not necessarily him agreeing with my reaction. It's not an improv game and he doesn't necessarily intend for me to "Yes and" him in a very different direction.]

Sponsors:

Check is the leading payroll infrastructure provider and pioneer of embedded payroll. Check makes it easy for any SaaS platform to build a payroll business, and already powers 60+ popular platforms. Head to checkhq.com/complex and tell them patio11 sent you.

Building an enterprise-ready SaaS app? WorkOS has got you covered with easy-to-integrate APIs for SAML, SCIM, and more. Start now at https://bit.ly/WorkOS-Turpentine-Network

Timestamps

(00:00) Intro
(00:23) Dave Kasten’s unique career path
(01:37) The beginning of VaccinateCA
(03:09) Early challenges and volunteer efforts
(07:24) Volunteer coordination and call center operations
(10:55) Navigating policies and procedures
(14:33) Navigating policies and prioritization
(21:00) Adapting to volunteer feedback
(22:50) Sponsors: WorkOS | Check
(25:07) Public reception and media involvement
(35:40) Government and institutional responses
(45:59) Differences between tech and government approaches
(47:52) Challenges in pandemic information dissemination
(49:10) Accuracy and information systems
(53:43) Government and agile development
(56:51) Bureaucratic constraints and historical practices
(01:08:55) Government workforce and technological challenges
(01:22:34) The importance of policy engagement
(01:33:42) Wrap

You can Ctrl-F many of these in the below transcript. Sorry, I can't link them, due to technical limitations on some platforms.

Transcript

Patrick: Hideho everybody, I'm Patrick McKenzie, better known as patio11 on the internet, and I'm here with my buddy Dave Kasten.

Dave: Hi Patrick, great to be here.

Dave Kasten's unique career path

Patrick: So, Dave is another person that has… let's say, a little bit of a weird career history. We won’t touch on most of it because there aren’t all that many common threads – you worked at Blizzard Entertainment, McKinsey, and now helping the tech community understand policy folks in DC, where you’ve been for the last 15 years or so – but we ended up on each other’s radar back in 2021 when we were doing VaccinateCA. 

VaccinateCA ended up being something like “a clearinghouse for vaccine location information in the United States.” You were a volunteer on that effort, and that’s how I got to know you; I ended up accidentally falling backward into the CEO chair. I just want to talk in very granular detail about what doing hard things looked like, in that we were not governmental exactly, but definitely rubbed elbows with some folks in those spheres.

[Patrick notes: I have written about the big picture story of VaccinateCA (warning and/or promise of depth: 27,000 words!), but don’t think we’ve covered the “so what did y’all actually do every day?” in enough depth for people to copy tactics/strategy into their own projects, and so thought it would be useful to talk to Dave about that prior to talking about the broader tech/government engagement question.]

Patrick: Then we can talk more at a 30,000-foot level of detail about what engagement with policy folks looks like from a “doing-things-in-the-world” perspective.

Dave: Yeah, sounds great. 

The beginning of VaccinateCA

So it would be helpful for me to start by talking about how I got involved with VaccinateCA and how that evolved over time?

Patrick: I would love to hear it, because honestly, I didn’t know anything about anybody until they showed up in Discord. [Patrick notes: Where all important public health infrastructure is built! Also, notably, community software / Slackalike most commonly used by gamers, for those audience members who have not used it.]

Dave: As is probably the case for a decent number of listeners to this podcast, you are someone who has been on my radar as a person on Twitter–I refuse to say X, it’s Twitter still–for quite some time. 

I think probably we had tweeted back and forth with each other maybe a very little bit prior to that–I certainly wouldn’t say that we were friends, or even really acquaintances at that point–and then I remember when you tweeted out, in January of 2021 I guess it was, “Hey, it sure would be a good idea if somebody was to call a bunch of pharmacies and write down on the internet where the places that have vaccines are, because it turns out nobody actually has that information.”

[Patrick notes: To restore now-deleted context, I quoted tweeted someone reacting to one of many news stories saying vaccine discovery in California was borked with a thread beginning thusly:

]

Dave: I remember seeing that as I was going to bed, and I thought, “I wonder what’s going to happen with that.” A little part of me was like, maybe I should say, “Yeah, sure, I can try to do a little of that.” Then I kind of forgot about it, because it was January 2021. I think we were all a little burned out at that point in time.

Then the next morning, I logged back online and – in between doing my day job, which was at McKinsey at the time – I saw, “hey, there’s something here”; there continued to be people doing stuff.

Early challenges and volunteer efforts

I actually remember that, at that point, it wasn’t even a Discord server, right? The very first step was a Google Doc or Google Sheet or something, where it was just a list of pharmacies, I think in the SF area – calling them, and as you did it, you’d just fill in what you had learned.

I did maybe five or ten of those—I forget, honestly—and put the results in. I think they were all "no's" at that point in time – I think it is important to remember, that although we ended up getting into cadence where there are a lot of yeses later on, early on it was basically just getting the metaphorical door slammed in your face very politely, over and over again.

[Patrick notes: I recall us as having had the Discord from within an hour or two of the tweet going live; it was where most of the communication happened on the first day. Most people in the Discord had been invited via text message and similar by Karl Yang, who found it on Twitter. This was an extremely ad hoc effort which used mixed online and offline social networks to recruit the dozen-ish “coordinators” who did most of the heavy lifting in the first week or two, and most of our early volunteers.]

Patrick: It’s worth pausing for the audience that wasn’t there in those early days: the reason this project needed to exist was that there were thousands of people across California—tens of thousands—who were either patients or, more commonly, caregivers for patients, often the children of folks who were advanced in years, who were sequentially doing this exact exercise, calling a bunch of pharmacies and getting told no.

[Patrick notes: Frequently they, like we for the first few hours, were calling healthcare providers other than pharmacies, out of the mistaken impression that that was likely to actually work. Most doctors had no allocation, were not scheduled to get an allocation, and were never going to be told they had no allocation. State and national planners who knew that fact did not consider it their job to get on the nightly news to say “When the governor said ask your medical professional, to be clear, unless they spontaneously offer it you’ll have a much better time asking your pharmacist than your doctor.”] 

Patrick: The thing that had motivated me to tweet was reports of people calling upwards of 50 or 60 places prior to finding one that had the vaccine available. I thought, “this would be much more efficient if someone could just centralize this.” The way you centralize it – that actually worked for centralizing it – was to get told the "no's," so other people didn’t have to hear those no's.

I remember there was an early discussion among us about “is this being abusive of the medical system?” because we were making phone calls into the system at a time when the system was getting an awful lot of calls. Our decision, which was heavily overdetermined but it’s worth noting that smart people actually had this discussion, was that “it’s better to get told no, and have one net new phone call have to get answered by a medical professional, when we can put that on the Internet and thus avoid many phone calls at the margin versus not making that one sorta-imposition call.”

[Patrick notes: This should be a pass/fail question for getting to work as a medical ethicist. You would think all ethicists would pass, right. Right.]

Dave: Yep. And I think, jumping ahead, related to that for a second in the story, one of the things we got quite sophisticated about at Vaccinate CA over the course of it was (1) how do you sort of take someone off of your list temporarily or permanently? There was a particular healthcare-related entity, I'll avoid mentioning them, that essentially said, “hey, could you never call us?” For a variety of reasons related to how they distributed stuff they said, “look, we're never going to have useful information for you and it's really annoying.” And we said, “Sure.”

There were a variety of other settings we had internally within the system that evolved over time based on what we were seeing, about like, "don’t bother to call us back for a certain number of days," "don't bother to call us back until some sort of situation change,” in one sort or another – either in free-form notes, or we had switches we could throw in our databases. Then, of course, there were famous examples of the times we called horse hospitals, which politely informed us that they in fact would never have the COVID vaccine unless and until the equine industry got into vaccinating those fine mares and steeds as well.

[Patrick notes: We also had a “despite ending up in the database this is not actually a medical provider and will never receive an allocation and so we should spend zero resources on it in the future” label. Because we were a team heavily steeped in Internet culture, that flag was sir-this-is-an-arbys. If you are less online than America’s vaccine location infrastructure mavens, see here.\]

Patrick: I will say – tying in a little bit of off-topic information – the California Veterinary Medicine Association did successfully get veterinarians prioritized, group 1A, for the vaccine distribution. I don’t remember if we had the conversation internally, but I would have been okay with giving at least one call to horse hospitals just to make sure they didn’t, accidentally or otherwise, have a box lying around – but be that as it may.

In those early days, we were a scrappy Internet-organized operation. Our hypothesis was basically that there’s no central source of this information which is currently published, and we can develop a central source by making a large number of volunteer-powered phone calls. This hypothesis evolved over time, and our activities evolved over time. 

In the beginning, we were just distributing a spreadsheet, then we put a nice UI on the spreadsheet with a map, and a search box, and “find stuff in your zip code,” etc. But in those early days, the core of our operation every day was making as many phone calls as we could convince our volunteers to make. 

Volunteer coordination and call center operations

Can you talk a little bit about some of the stuff we learned in spinning up that kind of Internet-enabled call center?

[Patrick notes: Various team members and volunteers had various lenses for what we were doing. Some of our team had worked in progressive advocacy organizations and for them this was “kinda like a political campaign.” For me, it was “kinda like working in a call center” since I paid for my college education in part by doing that, and also “kinda like a leading WoW raid guild” since that was my only prior management experience. Jesse, our COO, often analogized it to being kinda like an OSS project with maintainers and volunteers existing in a delicate symbiotic relationship with important power dynamics to be aware of. And for many of us, it was simultaneously kinda like a tech startup, since we were deeply, deeply steeped in the folkways and habitus of tech startups.

I say this partly because different people perceive or emphasize different parts of the elephant, as it were, and partly because the culture of how we build organizations is extremely important. I think VaccinateCA’s manifest outperformance compared to better-resourced actors in society owed a lot to quite a bit of accidental cultural decisions and a few key extremely intentional ones.]  

Dave: Yep. I think the thing that was probably my central takeaway from this experience is that expertise in this question is not necessarily where you’d expect it would be. Quite early on, I actually was looking back at some of the old chat messages prior to this podcast just to prep, and early on I actually was one of the main volunteers that when folks were coming into our Discord server – they've been invited one way or another, whether they had seen a post by one of the volunteers (probably most often you [Patrick]) or they had been been pulled in by a friend directly – I'd often be the person greeting them in the intro channel saying, “Hey, Betty Smith, welcome to the channel. Here's what we're trying to do. Here's what you should go read. Would love to help you start making some calls; let me know how I can help.”

I did what I think was a pretty good job – I’m someone who’s pretty good at being a utility infielder, pinch hitter at a variety of different things, and I was doing what I considered to be a good job; I’d written down good process stuff and things like that, and I felt like I understood it fairly well. 

We then – somehow, I forget exactly how – had a couple of folks show up who had experience running much larger volunteer phone banking efforts for various political campaigns (primarily), and they just immediately looked at what we were doing and said, “You sweet summer children, you thought this was what good looked like?” and really took it to a whole other level. 

One, they had a much more granular sense of the process involved. I think we frankly had a little bit of a scrappier, early days effort, flowing ad-hoc – you show up when you can, when you're between meetings or calls, when you're on your lunch break.

They very quickly pushed more towards, “hey, let’s try to sign up for shifts, so we know we have continuous coverage, we have some ability to predict how many new calls we're going to be able to make per day, and what that looks like.”

[Patrick notes: I think we were acutely aware of some of our limitations. I’d have to review the memos from Day 5 or so but we were extremely ahead of e.g. volunteer burnout and uncertain call capacity being risks, and with a million things to do did a roughly appropriate amount of keeping call capacity up via getting better at volunteer management, seeking to use external professional callers, and similar.]

Patrick: There was also a bit of a labor force change in that, in the first week of the project, probably more than 90% of our calls were made by someone earning six figures in the software industry – which is not the way that most call centers are staffed, to put it mildly. 

We got the classic volunteer base for volunteer-based operations in America, which eventually delivered most of our volunteer calls – which selected a little less highly for, like, membership in the professional managerial class necessarily, and definitely selected less heavily for tech.

[Patrick notes: Who shows up at a bake sale or PTA meeting? That was eventually the volunteer base, considered in bulk. Though the diversity was a bit wild, including e.g. middle schoolers, at least one Israeli-living-in-Israel, two coordinators being based physically or mentally in Asia as of Day Zero, etc.] 

Patrick: I remember when we got told we were all using Discord because it was the natural thing for Karl Yang to invite his 10 best buddies onto to try to open source the availability of the vaccine, but Discord is very gamer-coded, etc.

Even as someone who is very gamer-coded himself, I did not have Discord installed prior to this because I thought it was for a different generation. We got told by one of our volunteer coordinators relatively early in the effort, “hey, these people don't exactly pick up new software as much as gamers etc. do, they're comfortable with Slack, they use Slack. We're going to use Slack for them.” And so we ended up with this fun system where we were doing X percent of the operations in Discord and then had a volunteer management portion happening in Slack as well.

Navigating policies and procedures

And yeah, echoing your point, there was definitely a playbook that existed in progressive advocacy organizations, etc., etc., where people who had been there, done that, gotten the t-shirt, were like, “okay, we're going to schedule orientation hours, we are going to do classes of this, yadda, yadda, and then we'll do a finger on the pulse of the sort of flow of calls.” It was very fascinating to me watching it happen and in some ways influencing the machine.

Can you talk a little bit about our “call captain” role and related infrastructure?

Dave: Yeah, sure. So I think one of the things that became clear I think relatively quickly as we transitioned from, “Hey, this is a fun weekend project over a long winter weekend” into being, “actually, this is starting to get some traction” – not just in the sense of “we're having fun with our little project,” in the sense of “we are getting real responses from real humans we often know – sometimes not – on Twitter saying, ‘yes, I got a vaccine for my grandma, for my mom, my sister, my wife, my husband,’ on the back of this website” and realizing, this is real. We started professionalizing a lot of the rules. We said, “Okay, there's a thing called being a call captain.” I probably don't have as crisp a recollection of the way we defined it as when I was in the thick of it.

But I think roughly what we said was “If you are a call captain, you have responsibility for a certain number of time slots per week; you're going to sign up for them, you're going to show up for them – and crucially, here's the set of core expectations we have of what you're going to do, but also part of your job is to surface the weird to the rest of the organization.” Part of what I considered to be my most important job when I was in a call captain shift was being the person whose job was to look at Slack and make sure stuff was getting pasted over into Discord, because that was directly relevant to the people who weren't necessarily making calls at that point in time (or for that matter, if their comparative advantage wasn't really making calls at all) and trying to make sure that they understood it. 

I was everything from, “Hey, we're seeing this consistent thing where we're thinking that how such and such situation about how the Veterans Administration is distributing vaccines should be put in the database in such and such way, and in Slack we talked about it one way, we all came to what we thought was a shared understanding, we're inputting in one of a series of databases that evolved over time in order to record that. However, when I look over here in Discord, I'm seeing feedback from our users directly from the website saying, ‘Hey, I searched for vaccines near me and I thought the VA could give me one for such and such reasons, and then when I called them, they said they couldn't’, and it's because there was a subtle mismatch between how the different filters work [relative to] how we thought about them in the database, how they're getting displayed on the website.” 

It had something to do with the order of the age filter versus the other special conditions filter; the VA without loss of generality would roughly let anyone who was a veteran get a vaccine at a certain point in time, but if you were one of their spouses or family members or caregivers, there was a more restrictive age consideration – but the way we were displaying that on the VaccinateCA website didn't have that same order, so occasionally caregivers were getting understandably frustrated that they thought they could get vaccines to protect their families and then couldn't. Lots of things like that we had to walk information over.

Navigating policies and prioritization

Patrick: Yeah, there were, in the state of California at any given time, on the order of hundreds of individual policies for what the true ordering of priority was. Over the United States, minimally 3,000, because there are 3,000 counties in the United States; probably well upwards of 5,000. More if you count every individual medical provider's informal policy (or formal policy) as one of those things.

[Patrick notes: Also note that since many government departments could not agree with themselves what the policy was as of today, it was frequently the case that a particular government organization had multiple policies in production simultaneously. They often contradicted each other in a way which caused medical providers to “fail closed”, i.e., not vaccinate using available doses.

As I’ve written previously, the complexity of society’s preferences being pushed down a low-state-capacity pipe and overwhelming the bandwidth there was a contributing factor to a lot of the breakage we saw.

We, the societal we, were capable of designing a 51 step online questionnaire and asking 70 year olds to please go through it. That questionnaire was complex because our moral preferences were complex: we wanted to save the lives of vulnerable people, yes, but we had many stakeholder groups which considered carveouts for their favored patients to be absolutely mandatory. Administering the spoils system outcompeted administering the vaccine as a goal.

So, having made a glorious sausage of our policy preferences, we... lacked administrative capacity to implement them. Which is why we needed to spend months, and tens of millions of dollars, on RFPs to government contractors to implement 51 step online questionnaires. But that questionnaire is a tiny artifact in the constellation of artifacts ordered to implement these preferences.

So, having frequently overwhelmed ourselves, we did the sensible thing and… threatened anyone not following the flowchart with losing their medical license and/or criminal prosecution.

Kafkaesque and farcical.]

Patrick: Minimally, order of magnitude of 5,000 policies that rhymed, that were from the government, that were democratically rather blessed, but that organizations had to keep in their heads at once. That's one of the reasons why the more centralized efforts to do this at the e.g. California state level broke down so much, because they were like, “You have to be in compliance with the policy; also, the policy is this unbounded blob that none of us can describe in a single document and is not designed to be described in a single document.”

(In some ways because describing the policy in a single document will make someone accountable for doing politically controversial and bluntly illegal things.)

[Patrick notes: As I have mentioned several times previously, the political environment the U.S. professional-managerial class caused many policies across the nation, including in California, to explicitly favor some patients over others on the basis of race. There are responsible servants of the public working for the state who will tell you they did not, in fact, do this. Many of them are people of good will. They are lying, in the service of virtues which they believe outweigh candor.

I am genuinely undecided on whether there isn’t a thin reed of hope in the observation that they feel they have to lie about this. Some states were entirely explicit that racial discrimination in the provision of medical care was state policy.] 

Dave: Yeah, and I think one thing related to that is that a county isn't actually a consistent unit across the United States – or even within the state of California. So for as we've already sort of alluded to, we eventually scaled the effort beyond just California and it got even weirder there, but let's stick to California for simplicity’s sake. 

The smallest counties in California have single-digit thousands of people in them. 

[Patrick notes: Consider Alpine County, population 1,141. If one arbitrarily picks 50,000 as the threshold for a county being small, of the 58 counties, Alpine, Sierra, Modoc, Trinity, Mariposa, Inyo, Pluma, Colusa, Del Norte, Glenn, Lassen, Amador, Siskiyou, and Caladeras are under it. ] 

Dave: Even if you count all the people who don't literally live in the county but might be eligible to receive it from that county because they're medical professionals and things like that, you're still only 2Xing that number – at the very most. 

By contrast, LA County is literally millions of people. [Patrick notes: About 10 million.] It could be a state in and of its own. So any policy structure that begins with the statement, “we are going to have a county-based system and distribute vaccines by county and then go do stuff” is going to have a very weird experience because, among other things, when you try to break it up into distributions per county, the minimum available shipping unit early on to send to a county was in the hundreds of of doses – several vials of, basically in ultra cold storage together – which means that by definition, when you send it to a little tiny county, you're actually hitting a much larger percentage of that county in the minimum number you can send there than if you were sending it to Los Angeles.

[Patrick notes: This quickly left particularly small rural counties in a place where they were advancing through the tier list faster than large urban counties. There was an extremely strong opinion in the halls of power that Californians with advantage, who had access to transportation and information, would happily get into their cars to get doses not-local-to-them. This was considered maximally socially undesirable in the early months of the effort.

The stated reason was that this would misdirect doses allocated for vulnerable individuals living in rural areas.

An equally held, and equally spoken out loud, rationalization is that some people in California consider the notion of an an elderly well-off white San Franciscan driving across the state to get a dose in a community where everyone is not yet dosed to be morally risible. They would prefer that person die rather than do that. The community that is California medical ethicists was explicit as to this preference, documented it carefully, and will phrase it as equity-informed ethical decisionmaking if you ask them to justify it. But, pace Dan Davies, while many of them are extremely pleased at the documents they wrote about health equity, their hands were tied by some other actor—guidance from the CDC, for example—and they did not personally deny anyone lifesaving healthcare based on their race. Nobody did, in fact. They simply enforced the residency restrictions that the state had put it place, shipped vaccine using an equity-aware prioritization scheme, etc etc.

This moral preference was strongly determinative of policy in California and elsewhere. It would be interesting to go back and check whether it saved the lives of its intended beneficiaries on net. I strongly suspect it was actively counterproductive even on that narrow axis, due to complexity drag on the overall effort and various unintended consequences. But the policy is monstrous in its own terms, and the quirky alignment of the vaccine in U.S. politics means that nobody really gains from looking at either what we did or the results obtained too closely.]

Patrick: Yep. To be clear here, VaccinateCA was not the entity that for the United States of America decided the county is the right level of abstraction to do allocations on. Oh goodness – I don't know if the United States of America ever sat down and had a serious discussion on the right level of abstraction to do allocations on. 

But the decision was made to allocate to states and have the states choose to sub-allocate, and I believe every state virtually without exception decided to sub-allocate to county health departments and then (with an asterisk) also some community groups.

On the list of states, the city of Chicago ended up being the 51st state in the United States. And it is curious to know how that possibly happened.

There was a person named Emanuel who was previously the mayor of Chicago and there was also a person named Emanuel drawing up the list of states in the United States. [Patrick notes: They are brothers; Dr. Zeke was on the COVID-19 Advisory Board. Rahm was mayor of Chicago through 2019, among many other career accomplishments.]

Maybe someone should look into that correlation, but be that as it may.

[Patrick notes: It’s a bit impolite to say that the practice of politics sometimes includes corruption. Since I am not impolite, I will make the more polite claim that Chicago, NYC, and Philadelphia got direct allocations which skipped their respective states/counties as a result of robust engagement in the political process. You’re welcome to verify this happened by perusing obscure CSV files published by the CDC, since candid first-person accounts of that political process are difficult to come by.

It was extremely important, if one wanted to get early access to the COVID vaccine, to have “your guy” in the room where it happened. This was absolutely rampant, at many levels. Moral revulsion about the brazenness of e.g. getting scheduled early for shots because one was a friend of the governor or one’s cousin was in the FBI ironically lead to overreactive crackdowns that e.g. banned end-of-day shots.]

Patrick: So yeah, in the early days we thought the core job of a call captain was to make sure people show up at 9:30 AM and actually start making the phone calls. Morale management was huge too – 

Dave: Yep. Huge.

Patrick: – especially in those early weeks when… I think the longest I personally remember, after the model was fundamentally working we went 90 minutes and on the order of 100 phone calls without getting a Yes on a day. People were feeling kind of crushed.

So one of the things that we ended up doing was having a hypothesis for where we'd be extremely likely to get one marginal Yes and calling them early in the morning, so that we could get gongbot to record a “yes” on Discord early. We empirically got many more phone calls actually made when we had a “yes” five minutes into the calling session, versus e.g. two hours into the calling session.

[Patrick notes: Gongbot was a simple integration which slurped up new records in airtable and put a message in a channel, bolded in the case of a Yes, for every new call made. The name is a reference to a cultural practice of many sales teams, which keep a physical musical instrument on their sales floor to strike when a representative successfully gets a commitment from a customer. This is to motivate the rest of the sales floor and visibly celebrate the success of the striking sales rep.] 

Dave: Yeah. I remember there were days where we were – specifically you and I – having backchannel conversations saying, “Okay, we haven't had a yes yet. It's 30 minutes in. What can we do to find it?” And we’d start generating hypotheses quite quickly about where it would be likely to be, because sometimes we'd looked at all the places that were pretty high likelihood yesterday just by chance; we were doing more prospecting the next day: “Well, you really want to get something that feels like a good prospect early on. What can you do to stare at your data and try to find those little potential nuggets within it?”

Patrick: Yeah, and there is no right answer for how to prioritize the calls to make, given that we had limited resources relative to the number of calls we'd have to make at essentially every point of the project. We focused on pharmacies very early for a couple of reasons, but there's on the order of 6,000 pharmacies, plus or minus, in California. 

So, which ones to call today? Given that we had for a couple of weeks a capacity of 300 to 500 calls every morning, and we did not necessarily know whether it was closer to 300 or closer to 500 because [it was] completely volunteer based at the moment.

[Patrick notes: Oh goodness could I go for ages on the complexity here. Availability information goes stale quickly, particularly early in the vaccination effort when we were supply-limited as a system. Should we burn a call this morning on checking a Yes from yesterday, to see if they still have stock? Or should we burn it only after three days, and use the marginal saved calls to “prospect” for more new Yeses? Should we target the locations we think are most likely to be Yes? Really? At all margins? What if the next very probable Yes is less useful, for example because it in a less-populated area where we see less search traffic and close to existing Yeses? Should we not instead attempt to find at least one Yes in areas with heavy search traffic but no Yes discovered yet? And then throw in moral or aesthetic preferences of team members and/or the state of California into the prioritization logic and…

There is no right answer to this question and we chose to make calls anyway. The perfect was the enemy of the good far too often during the vaccination effort. In a very literal way, time spent talking about optimization of the calling strategy traded off directly with making phone calls. Many, many orgs across the United States suffered analysis paralysis; we spent some cycles optimizing but actually banged the phones.]

But before I get into that topic, I want to circle back with that tight loop between callers and “the rest of the organization” was one of the key takeaways of the experience for me.

[Patrick notes: I think almost everyone in the early days made calls, partly for cultural reasons and partly that was what we were here to do. I made one of the first ones myself, and relatively few over the course of the effort, but would make a point of emphasizing to people that calling a pharmacy is so easy that I could do it from central Japan live if they wanted to listen to one.

But the divide here was a pretty real divide: our volunteers made most of the calls (prior to bringing in external call centers and paid callers, a few weeks later), and our engineers and other tech professionals were building the machine that would track/prioritize/collate/curate/publish those calls live. (In the early days before we could employ anyone many of our engineers did their work after their day job had wrapped for the day, and so VaccinateCA had a second shift which structurally could not call pharmacies because pharmacies close early. One of the reasons I flew to California in week 4 was being awake for both California shifts timeshifted-to-Japan was threatening to put me in the hospital.)] 

The vaccine rollout was chaotic in a lot of ways: it was intensely hyperlocal; there was no model of it that was actually accurate.

[Patrick notes: Hyperlocal sounds like the buzzword a VC might substitute for actually understanding the problem. Here’s what hyperlocal means in the real world: the distance in time and space between the patient and the medical provider and vaccine vial had to decrease to effectively zero for successful injection to happen. If you’re off by 100 feet or 15 minutes you get zero vaccine molecules.

It is very important to understand that for Seeing Like A State reasons the system could not perceive this truth. County health departments sometimes said “Absolutely does not exist at the county” when it absolutely existed across the street from their own office. They told this to allocators and they told this to patients seeking the vaccine. Not because of stupidity, not because of malice, simply because they could not bring themselves to See, for structural reasons.]

Patrick: So every day as we improved our model by a little bit, we would very rapidly attempt to get the learnings from that back into like, “Don't wait for the next engineering sprint. Don't wait for the Q2 planning session. What can we do five minutes from now to get this to inform the decisions we are making for the rest of the day?” That was a mantra internally.

[Patrick notes: “Every day matters. Every dose matters.” was the usual phrasing. Many startups have some internal cultural practice to valorize prioritizing speed over other competing goods at margins that might be controversial in other places. I wanted to make it extremely transparent to people involved with us that we didn’t move fast because breaking things was an objective, we moved fast because dose-days delayed meant lives lost. (A useful approximation early, heavily sensitive to what population was being dosed, was 10,000 dose-days was worth a life in expectation. I sometimes wonder how many organizations really, really felt that anything like that was true.)] 

Dave: Yeah. Absolutely a mantra. 

Adapting to volunteer feedback

Dave: I think a couple of things to note here. One is that I think our Discord server had more mechanisms of internal feedback than any other organization I've ever been a part of. 

[Patrick notes: That might be the most terrifying statement I’ve heard so far in 2024. Viewed from a different vantage point of the same Discord server, I often worried that there were communications breakdown between organizers, staff, and volunteers (to say nothing of the external world) simply because we couldn’t be maximally internally transparent. This was partially due to necessity of keeping some secrets to enable partnerships (which were critical to our effort), partly strategic, and partly (embarrassingly, for us and for the polity that is the United States) because we couldn’t be sure we had no people on Discord who wouldn’t sabotage vaccine deployment for ideological reasons.] 

As Patrick already alluded to, we had a channel that we called #report-gong and had a gongbot, that every time we found a vaccine, it would say, “Yes, vaccine found at so and so,” like a gong on a sales force floor, like when you're trying to sell new cars or something.

[Patrick notes: Midway through the effort it looked something like this, bolded for a Yes.

]

People would actually, like, emoji the bot, as sort of like a good luck ritual. I still remember when the bot found something, announced we'd found one after a long dry spell of 60 minutes or whatever, I would give it a particular emoji that made me feel like it was lucky.

There was that, and there were several other channels as well. There was one [Patrick notes: #shots-in-arms] where we posted examples from Twitter, or just stories we heard in our personal lives of people finding vaccines that we knew definitively had happened because of VaccinateCA – our friend told us, or we were sitting with the person when they did it and could say it in a very confirmatory kind way.

Patrick: It was micro-tactical but extremely true. One of the reasons that Twitter has done such a number on the news media (in positive and negative ways) is that a public statement sourced to a person on Twitter is reportable in ways that private conversations that the reporter was not present for are not. So Mallrat9000 – don't know what that person's actual name is, but I know their parents were vaccinated as a direct result of the VaccinateCA efforts, before VaccinateCA even decided to call itself VaccinateCA, in fact. 

Kim-Mai Cutler had cited us on her Twitter feed, and then we were off to the races with the rest of the PR strategy. As someone who wore the PR hat for a few years, kind of important, but not the most important bit of the story so I'll break it up for the moment.

Public reception and media involvement

Dave: I think one of the biggest signals that we got early on that we were adding value is that health and journalism professionals who in theory should have been the best situated to have that info already were tweeting at us thanking us for giving them that information. And then shortly thereafter, pharmacists started telling us as well – we’d call them and we weren't disguising who we were, we just weren't necessarily foregrounding it, it was just a person calling to ask about a vaccine as any resident of the United States could. 

Very quickly, they were saying, “Well, I don't have them, but I've heard through my pharmacy network, I've heard from seeing a local news story, I've heard from my friend who got a vaccine that there's this website called VaccinateCA dot com. You might want to check it out” – at which point our custom was to say, “funny fact, we're actually from VaccinateCA. Great to hear that it's helpful. Please do let us know if it changes. We'll always take an email or a tweet or whatever. Best of luck. Really appreciate all that you're doing to keep people safe and healthy.”

Patrick: We scripted our callers to then say something at that point like, “...and please do recommend us to other people calling you.” We did a number of tweaks on the script, many hundreds of tweaks on the script over the time.

That was one of the things that we were able to lean into early with having a professional labor force that doesn't typically make large numbers of scaled calls. It's that, in the early days, use your best judgment. You in expectation are a high-salaried, white-collar worker who talks to high-salaried, white-collar workers at meetings frequently. Tell the pharmacist what you need to tell them. No explanation from us on what that is.

And then as we transitioned to using a broader volunteer base and then paid callers, it's like, you folks need a script to be able to work. Here's the script, here's the decision tree, etc. etc. [Patrick notes: I’ve written about this class divide in the context of financial industry CS departments before.]

Some of the things on those scripts were surprising, which is one of the reasons that no one just whiteboards this and gets it right the first time.

For example, like when you call Insert Well-known Pharmacy Here, you will get a recording that tells you that the pharmacist has no information about the vaccine. The recording is lying to you. Hit four and talk to the pharmacist. 

There were large national chains which as a matter of policy lied to all people who called the pharmacy about the COVID vaccine (at least once). That happened.

Dave: Yep. And it happens. I think it's important for folks who haven't perhaps contemplated this question in much detail to realize that there is on some level an underlying good that that pharmacy bureaucrat somewhere in the corporate leadership of that particular pharmacy chain was making a decision on – which is there are things that a pharmacy has to give out, that if you don't receive them the same day, you definitively die. If you're a diabetic who needs access to insulin or a variety of other similar severe health conditions like that, you just die if you don't have access to those things. 

So it's actually quite important to make sure that you don't DDoS your pharmaceutical capabilities when they have those very important obligations. We as a society made a choice to effectively add on top of all of pharmacists' other duties in the middle of an already stressful time, “Hey, do something that has to touch functionally every single American.” 

[Patrick notes: DDoS stands for “distributed denial of service” attack, such as when a hacker uses a network of suborned computers to knock a website off the Internet by interacting with it more than it has capacity to service. When it is unintentional we sometimes call it the “hug of death” in the tech industry. Your blog went viral? Hug of death; your blog is now down.

A terrifying prospect for us in the early days was causing a hug of death at a medical provider by accurately reporting they had the vaccine, potentially leading to literal fatalities. America mostly avoided vaccine riots, but the risk of vaccine riots was not zero, and there was at least one chaotic on-the-ground situation which caused some soulsearching for some team members.

Without speaking for their particular reactions to that particular event, one thing I told coordinators very early (the same day they made me CEO) was that we were absolutely certain to have some people who used our information and then died, potentially in a way we influenced directly.

I told the team that morally unserious people might blame us for those deaths, potentially explicitly and to our faces. And I told them that team members should make the call to proceed with the mission even in light of this and give themselves grace for anyone we were unable to save despite our best efforts.] 

Dave: So a lot of, call it suboptimal second order decisions all flow down from the top that we essentially gave a new job to be done to all of those pharmacists without necessarily scaling up magically and instantaneously the number of pharmacists in America by an appropriate factor.

Patrick: Yep. And given more than a year to consider this question, we could have either trained people in administering vaccines or simply decided to wave a magic wand and change the paperwork requirement for administering vaccines to include, I don't know, nurses or similar.

[Patrick notes: Many things we treat as physical facts about the vaccine are facts that paperwork created. For example, the vaccines went bad after 12 hours… well, not really. They were paperworked to be useful for 12 hours, as a result of clinical research decisions made when it was costly (in time and lives!) to experimentally nail down the exact decay curve.

Lacking paperwork that said they were good after 12 hours, we as a nation pretended that we had invented the world’s most accurate atomic-scale countdown timer, and that the 4,321st second caused an irreversible chemical reaction that the 4,315th had not.

Anyhow, that’s a vivid example of a fact which exists on paper but not in the actual physical world. Here’s another one: who is capable of safely injecting a vaccine? Ah, a simple question, to which the state of Illinois had a 13 page answer. Doctors yes, duh, unless they were doctors licensed in Indiana, everyone knows vaccines in Indiana are totally different, unless they were doctors licensed in Indiana who had filed for a temporary practice permit under the authority of… you get the idea.]

Patrick: But we did not choose to make that choice at scale, and then 2021 ran around and we “went to war with the infrastructure we had, not the infrastructure we wished we had.”

I think one of the, that's a repeated story across the pandemic interventions is that we did not see a path to creating new capacity and so decided to repurpose capacity that already existed regardless of whether that capacity had capacity or not, if that makes sense.

So county health departments were the obvious place to do vaccination efforts when county health departments had never encountered or prepped a situation in which they would have to do prioritization or demand generation, and they were not good at prioritization or demand generation. (Demand generation here by the way is, like, “convince people to actually come in and get the vaccine.” [Patrick notes: This is a term of art in tech marketing departments.])

Many people who were project planners assumed, like, “This problem will take care of itself because I and everyone else in my social class have wanted this more than anything in the history of ever. After I get the vaccine, I will be able to go outside again. This is like the iPhone and Windows 95 and every other product launch to the hundredth power. How could you possibly need to tell anyone about why they need to take the vaccine?”

And then not everybody took the vaccine and people in our social class said, “that's vaccine hesitancy.”

That made me very frustrated because no one says it's “Pell Grant hesitancy” or “free money hesitancy” when you need to explain to someone their options for funding their college education, including the options that are like literally “sign here, free money will pour down upon you.” But we were not institutionally capable of telling basic and true things to people.

Like, the vaccine is free. Almost every pharmacy you walked into in America, the ones that had signage about the vaccine – and they were already punching above weight because most pharmacies didn't even have signage that we had the COVID vaccine in stock – the ones that did, their signage would say The vaccine is free* and the asterisks would say something like “your insurance company will be billed for it but by law there will be no out of pocket expenditure.” 

But since Americans understand, particularly in the medical context, that free* means you're going to get billed a heck of a lot of money in a non-deterministic fashion, people didn't believe that. We could have done saturated bombardment TV commercials on, “The vaccine is effective, safe and free! Effective, safe and free! Effective, safe and free!” and gotten that message out, and chose, as a society, not to by default.

Dave: Yeah, and I think this is something that's probably hard for a lot of folks who were in the “I really want to get the vaccine now” kind of mentality to remember, but we actually didn't really have very many traditional boring ads for this at all. For example, I know as a matter of fact that 100% of the ads in my neighborhood, which is Capitol Hill in Washington, DC, 100% of the ads that were on the street for it were ads that I had printed up on my own on a piece of 8.5 by 11 paper and put on telephone poles and things saying, “hey, here's how to get a vaccine in Washington, DC.”

There were various sorts of, sure, tweets from the DC health people and things on the nightly news and things like that – but not a single bus ad at all, not a single sort of static ad at all, not a mail flyer (though to be fair, mail had its own separate supply chain issues at that time.)

[Patrick notes: To underline how terrible the communication with the public was, I’d like to recount a conversation that I had with a Lyft driver months into the vaccination drive. They asked if I knew if they were eligible. I said it was a complicated story and would depend on many factors. They said work had said they were eligible but that they had missed it. I asked what their other job was.

That driver worked in a pharmacy

I told them they could go anywhere they’d ever gotten medication and say “I work in a pharmacy” then get the vaccine. (As I said this, I thought “That’s a tiny fib but asking that question makes it infinity percent more likely he gets the vaccine and if he gets a roadblock he’s roadblocked closer to the goal than he is at this exact moment.”)

I feel like many people might be intensely frustrated with this individual, and would like to triple underline that a population-scale vaccination effort cannot rely on all individuals being high-agency, eidetic memory, very engaged health policy experts. People like that are rarer than we think… even in pharmacies.

Vaccine hesitancy is obviously not the driver here, right? This guy was willing to take not-medical-advice from a stranger simply because the stranger was the only person who ever ended up having a 1:1 conversation with him about what one would actually practically do to get the vaccine.] 

Dave: The DC website that you would go to in order to get a vaccine, until very late in the vaccination process, did not have anything other than an automated Google Translate function for any language other than English. For those less familiar with the DC area, there are several languages that are fairly hard target languages that at the time Google Translate didn't do great in translating – like Amharic for the Ethiopian community, there's a meaningful Korean community and of course there's a large Spanish-speaking community as well up in Columbia Heights here in DC. 

There was no translation even into Spanish, the easiest language for most Americans to translate into a second language for, other than a static PDF that some woman had like, bless her heart, put arrows with a translation written next to it on some website that some people were using. 

It was lacking even the basics that we consider necessary for a not great e-commerce website.

Patrick: Yeah, we got, to my recollection, Spanish and traditional Chinese up in the first 20 days or so of the effort. This was over my objections as the CEO, no less.

So here's my, “you can be a good person and tell things that other people will people of good faith will disagree with.” It was internally suggested very early: the state of California has lots of linguistic diversity; many of the people who most need the vaccine for medical reasons have the least ability to read medicalese in English, and we should translate the website yesterday. 

Having been professionally involved in localization efforts for much of my career, I said, “That is a two to six week tarpit. We are two weeks into this organization. We currently have very fractional data about the availability of the vaccine, and the fractional data that we do have we do not have a super high confidence in the accuracy of, because we literally have no process in place for that right now.”

As someone who was running the effort from Japan at the time, I was extremely on board with the notion of having non-English information available at some point, but put my foot down as a leader prioritizing and said, “Hey, we have to make our choices on prioritization for today. We are not prioritizing translations. We are prioritizing data.” 

[Patrick notes: Given a sufficiently long time to make an argument I think I could persuade many intelligent Californians in the tech industry that at many margins a Spanish speaker is better served by more information being available in English than by less information being available in Spanish, via pathways such as e.g. bilingual children and community organizations, even if this was initially aesthetically discomfiting to that individual. But a long career has convinced me that there are better things to do with limited hours than achieving consensus about moral aesthetics.]

Patrick: Then a volunteer did something I was not aware of and used new localization technology, which has been invented in the last couple of years, to produce a working proof of concept for Spanish – just add water, basically – overnight. I was like, “Oh, if it doesn't actually derail this entire organization for six weeks (six weeks would be the fastest translation project I'd ever been on in my 20-year software career) then obviously we should do it immediately.”

(I remember getting told early by various people in civil society that “techies don't understand that there is diversity in California.”There exist many beliefs about the tech industry from people in positions of authority or other social influence, like reporters, which frustrate me.)

[Patrick notes: We ended up supporting the top ten most spoken languages in California, and I eagerly applaud those diligent public servants and reporters who can say the same about products they’ve shipped.]

Government and institutional responses

Patrick: The single most terrifying conversation we had over the interval was very early, I remember it being five or six days into the effort, when someone in a position of authority was like, “Hey, we love that you're doing this thing. Could you please send us your list of where the vaccine is?” That was the point at which I realized, “Oh goodness, it isn't simply a communications problem, like databases not talking to each other, that is causing there to be no publicly accessible list of where the vaccine is. It's not simply that the state government is rubbish at shipping software, which was one of my dominant hypotheses (and it's true). They literally don't know.”

[Patrick notes: Discussed in much more detail in previous writing: the vaccine had been devolved into so many mutually unintelligible distribution channels that the allocators did not know where it was.]

Patrick: And then there was another point where – going to be a little vague on who we were talking to to protect the careers and professional reputations of people who collaborated with us, and to not get reputation for ending the careers of people who take my phone calls–for the first couple of weeks of the effort, something which I said internally a lot was, “We're a stopgap here. There is clearly some better-resourced organization which is going to roll out a much better version of this, at which point we will join up with them, give them what we have, and then we do what they tell us to.”

And then we met them and went back to the team and tried to say the following in a way that was not shaken: “Good news and bad news. We are the cavalry. There is nobody coming to save us.” 

So positive takeaway: individuals can have impact on the world through agency, and civil society can self-organize in ways that surprise even arbitrarily well-resourced organizations like Google or the United States of America – which has some resources available to it, including nuclear weapons, which are hopefully better tracked than its COVID vaccine. Arrgh!

Dave: Usually, but not always – but let's not talk about certain recent inspector general reports from the US Air Force.

Patrick: (Ugh!)

Dave: But I think there is something to be said there about, one, as a Midwestern guy just like you, the notion that like we were the best that there was and there was no cavalry coming was a deeply terrifying thought. On some deep level I always believe, like, “Somewhere there's some command center, you know, some really smart group of people who are gonna take over and do it.” Like, yeah, you gotta be a good citizen and shovel the sidewalk of your neighbor and all that, but at the end of the day, eventually there's gonna be a snowplow coming, right? 

Not only was this not especially true, but like, we have compared notes in various capacities, both during then and then afterwards with folks who were doing this elsewhere and having the same result – there were multiple different efforts that happened both in the United States, in other states, as well as overseas, where the individuals who were running their own local VaccinateCA equivalent in one sense or another were high school kids. 

This often did not come out until the vaccine became legal for people of their age in the jurisdiction – because as a reminder, it took a while to get down to the younger age brackets. They posted photos of, “just got my vaccination!” And it's like, it's because you're a teenager that you only just got your vaccination. 

Patrick: One of the infrastructure mavens for my home state of Illinois was literally in middle school. 

The public health initiatives in the United States depended on the goodwill of middle schoolers. That's an absolutely true statement and… America should simultaneously pat its back on one side, and stab it on the other, for that being true.

Dave: Yep. We should be very proud we have such wonderful middle schoolers. I mean this sincerely. This is something where, in all honesty, one of the things that Patrick and I occasionally talk about is that we have a to-do reminder of, when some of these folks get closer to college applicant age, to make sure that somebody writes them some very nice words that can go in their application file about this.

[Patrick notes: Dave and I share some Midwestern sensibility about modest understatement, but to be very explicit, I would not upper bound potential academic or career interventions at "nice notes."]

Dave: But beyond that, I think it's important to remember that, like, there are lots of other sort of spare capacity groups in the United States beyond just high school students that could have been used. I live in Washington, DC, let me tell you – if there was one thing that the Army National Guard can execute on successfully, it is having a bunch of people in like a hangar or a barrack somewhere with a large number of telephones and a large number of – frankly, we could have given them paper forms, not even spreadsheets, and just had them execute in a very boring, rigorous kind of way, right? 

Like we know the number of people it would take to do this. It's not that big of an Army National Guard call up to do it all. They could have very plausibly done it – and by the way, probably because it's California, a decent number of those folks also would have software expertise as well. 

And indeed, to California's credit, it's worth noting that they called up the National Guard to use in a variety of other capacities, particularly for running large vaccination super sites and doing everything from traffic control to actually administering shots there. And they did that within that domain of what they were requested to do; they did it by all accounts quite well. 

But there's a lot of other swing capacity that the United States could have theoretically invoked, and I think at best we could say it chose not to. Probably more honestly, at no point was an active decision actually made because everyone was kind of just reacting and reacting kind of randomly to whatever news crossed their desk. And I think that does us a lot of discredit as a society.

Patrick: Yeah, there were very few actors that thought that this was the job. This was a 20% project for a senior aide who in many cases was self-appointed because they looked around and saw no one else working on COVID and decided to work on COVID. 

There was a gentleman who reached out to me via Twitter – he was essentially the equivalent of us working for a large California-wide medical provider – who had promoted himself to that position, being their COVID czar, from being a random person in data analysis, because he saw no one else analyzing the data.

Great that he stepped up; a little less great that we had a failure to lead in many places across many organizations. I do also 100% agree that the default is to do nothing, and so in many cases we just saw the default.

I do think there were decisions to do nothing made in some places, and it's useful to understand, for the historical record, how some of those decisions to do nothing were made and what the political economy was. 

One, which is a bit speculative from a person with more DC experience than (I believe) both of us put together but which I give a high degree of credence to, is, there was eventually a federally-blessed site called Vaccines.gov. Vaccines.gov would show the national Federal Retail Pharmacy Program doses, but it would not show California or any other state's doses. 

The political economy decision there is, you need a governor or some other source of authority on the state side to agree to give the information on their doses, thumbs up you using it, and then actually instruct people to do that work every day. There was a concern from many governors on both sides of the political aisle that they didn't want the President of the United States stealing the credit that they were certainly hoping would get them on the nightly news. 

The second-order effect of that concern was like, hypothetically, if this breaks down just red states versus blue states, it's going to look like the blue states get twice as many doses as the red states do per unit of population. That news cycle will be terrible. And thus, how no state doses, which are half of the doses in the United States, on the map that everyone assumes is going to show all the doses. 

(I believe if you looked at vaccines.gov today, you will still not see state doses, although it's far less pressing these days because there are abundant doses and essentially everywhere.)

But a choice was actually made there at some meeting for reasons. 

I don't want to throw vaccines.gov under the bus. It’s useful for people to understand: there was eventually a federally-blessed effort, Vaccines.gov, which took over – well, it was a rebranding of VaccineFinder which was run out of Boston Children's Hospital. They were good people doing good work, extensively under-resourced, in the public sector of the United States for about 10 years. 

Then they got the tap by the White House, because the White House believes that the federal government of the United States is incapable of shipping software products [Patrick notes: trauma response to the healthcare dot gov rollout during the Obama administration] –so “pick someone with gravitas that has a software product that can be quickly repurposed to fill the gaps”–and I would say, perhaps cynically,, “the partner needs no credit if it works and the administration needs no blame if it doesn't.”

We ended up with something between those two poles.

I'm glad they existed. We redirected our website to Vaccines.gov after we shut down in August. But the United States underperformed its capabilities comprehensively. I don't want my desire to be charitable to individuals and understanding of the constraints they found themselves in, etc., etc. to avoid giving that underlining, we should give ourselves a B to a B+ on getting the vaccine out on the time frame it got out, and a D-minus on the vaccine effort as a whole.

[Patrick notes: I’m being pretty generous on the timeframe thing, since on the one hand it was the fastest vaccine development in history and on the other hand we intentionally slowed ourselves down for paperwork reasons (stupid) and because of political calculations (to avoid giving Trump a “win” prior to the election).] 

Differences between tech and government approaches

Dave: Yep. I think one thing that is worth saying is that, if you have ever spent your time as a government contractor or a management consultant that has to use government computers, you will suddenly get in a much more intuitive way why it is that people in the federal government think that it is so hard to do things that are at the intersection of government and technology.

I have more than once talked to someone in government about some new cool tech thing and they're like, “yeah, that's great, but it takes 30 minutes for me to boot up my computer in the morning.”

I want to be clear, that's not an exaggeration. That is a routine thing that you can hear from many people at many government agencies: they come in, they turn on the computers, they enter their passwords, and then they go and get a cup of coffee. It is quite a long line, [but] they're not worried about how quickly they're getting through the line because they're still getting back to their desk before the computer has finished booting up, and all the competing antivirus software and all the misconfigured services on the back end, all of that finishes grinding and actually gives them something they can [use].

Patrick: And there's a bit of learned helplessness here. If you are working in an organization like that, the decision to use this particular antivirus software etc. has been made six levels above you on the basis of a federal regulation that you have no power to change. You, your boss, your boss's boss and everyone you work with also puts up with 30 minutes of downtime in the morning and so that's just the way it ends up.

[Patrick notes: Sometimes you put up with the financial industry and airlines simultaneously turning off because ehhh Compliance. Crazy story at Bits about Money if you didn’t follow that already.]

Patrick: That bit of learned helplessness echoed through a lot of our conversations, not just with the government. There's different flavors of learned helplessness in AppAmaGooFaceSoft, which we dealt with quite a bit. 

Challenges in pandemic information dissemination

I remember one moment early on with someone in civil society who was working directly with a county-level health organization on its pandemic strategy: I emailed them to say, “Hey, we plan to put this product in the market ASAP. What are your thoughts?” And without ordering me to not do it, he said, “that's very low priority. If we thought it could do anything, we could probably ship something like that and probably faster than you think it would, but simply having a map available does not help patients or caregivers who can't read maps, and if there was mistaken information on it and you sent someone 30 minutes out of their way, that would be catastrophic.”

[Patrick notes: Note that spontaneous verbal recollections of written records from years ago are not likely to be literally accurate. I’ve previously quoted that email as saying this, while actually copy/pasting:

“It would probably help with public communication to have a map like the one you describe, and we could probably build (and verify) in less time than you would expect, especially given what I read as your sometimes-correct view of the glacial pace of government. However, it comes with a lot of ancillary considerations. For example, it would be very bad to send people to a location for a vaccine only to be turned away because a website was wrong, especially if those people are seniors, or front-line workers who only get limited time off of work. It also doesn’t help folks who don’t have smartphones or lack the technical savvy to understand this kind of information, who, demographically speaking, are more likely to be the ones who need the early vaccine doses anyway.”

I think I was reasonably fair with the paraphrase, as you can see.

]

Patrick: Like, “Hmm, that's a persuasive argument – to not ship wrong information. But that is not a persuasive argument to have there be an entire information vacuum.” 

It felt like that Copenhagen theory of moral culpability where “I – as the person whose job is literally to get information out to the public about this – am not responsible for there being no information available on Thursday. But if I do anything marginal to increase the number of people who are informed about this issue, then I'm personally responsible for every person who gets misinformed by the map.”

Accuracy and information systems

We were very serious about accuracy. We were very serious about being as comprehensive as possible. But we didn't let that stop us from doing anything at all – we just attempted to build in systems that would allow us to trend in the direction of more accurate as quickly as possible.

Maybe [it’s] useful going from the meta-level to the micro-tactical things: we had on our website, “if you're reporting an issue and you are a health professional, call or text [blah blah blah].” That was a Twilio script that I wrote in five minutes. The back end of it was like, if they texted (thank goodness) it would publish directly to one of our channels and then we would react to it after having dealt with it. 

If they called, it would literally route to my cell phone and I would say, “Yes, can I take down the name of your place; what's the correction you want to make, etc. – Yes, we will handle that immediately.” Then the backend for it was typing into Discord and someone would handle it immediately.

From the general public, we had a different thing where they got sent to an inbox; we would treat that as advisory and try to prioritize someone to call the place that the information was reported to be inaccurate at, but we didn't default to trusting updates from the public. 

That, by the way, [was a] minor stitch in my craw, but man, was it a stitch in my craw for several months – we got branded as crowdsourcing the information very early, like, no, this is not Yelp for vaccine reviews. We are creating this scalable machine to talk to health professionals about the care that those health professionals will offer with their own two hands. We are just aggregating the information that they give us – versus, “we are trying to get conflictory reports and make some sense of them, and we take no responsibility for information that you could get on this website.” 

We cared very much about the accuracy of things in a way which Google Maps, for example, cares about the accuracy of information. (We got quite a look at that process considering we got our information onto Google Maps.) Google institutionally does not feel that it publishes every bit of information that can be found by Google search. 

(There is a deep rabbit hole behind that statement. I will make that statement and just let it hang out there.)

[Patrick notes: But now that I’m in the comments and have more time, let's say that this matters from a Section 230 perspective, at a minimum! But from the perspective of the best resourced actors in capitalism, the PR shield is likely worth more than the legal shield. If a hypothetical search engine surfaces untrue facts about vaccine availability, that wasn’t misinformation. There was a Process applied to bless a group of non-profit dogooders to put their information on our website and if they were wrong, Congressman, well, I’m sure their heart was in the right place now wasn’t it. 

Relatedly, VaccinateCA in some ways allowed large tech companies and the various governments to pretend they didn’t work with each other, where those groups individually had reasons why that kabuki was locally incentive compatible. And if you think that sounds darkly comical now, you should have been in the briefing sessions where we gave our lineup of spokespeople sartorial and speaking advice so that they’d read less like techies to people watching them on TV.] 

Patrick: But we very much did feel like we were publishing this information, this is a product of VaccinateCA – every instance of a bug is a “drop it and address it immediately.” Anyhow, I'm rabbit-holing a little bit on that.

Dave: I have a slight nuanced disagreement with you on that Patrick, which is that, as with many things, I think the original label of us being crowdsourced was slapped on us because an editor put that as the headline of a otherwise quite (I thought) lovely piece that was written about us by a wonderful reporter early on. 

Patrick: Kelsey Piper at Vox. Let's give her some credit. [Patrick notes: Previously, on Complex Systems.]

Dave: (Okay, good. I thought it was Kelsey, I just didn't want to get the attribution wrong.)

I think Kelsey wrote a great piece, headline got slapped on, bad editor, nothing to do about it–for better or worse, that's the journalistic norm. (I think worse, but that's a digression.) 

But I also think it's worth noting that crowdsourcing at the time–probably less so now, because several years have passed–but at the time, that was still kind of a buzzword that like, when you said to people in the tech industry, they're like, “that's so 2012”; when you said that to people in government, that was still a new, exciting trend.

[Patrick notes: True enough. Governor Newsom, for example, used “geofencing” to describe a policy he was proud of. In the tech industry, “geofencing” is a particular thing ad tech companies due to microtarget delivery of ads based on the physical location of a mobile device. The more standard way to phrase his policy is redlining in the provision of medical care.

I am aware that that sounds like a pretty robust criticism so you’re welcome to watch the news conference and read my previous writing on it; that criticism was earned.] 

Dave: So even if it's the case, as I think it was the case, that it had an incorrect connotation of the quality and validity of the data and the amount of effort that we put into sometimes literally moving pins on a map half a block, to the point where users almost certainly didn't even notice – we were also reflecting the fact that there's a culture that policy and service delivery outside of the tech world is much more behind the times than you might think, and when they hear crowdsourcing they also kind of think “Oh, Wikipedia – that website is really good.”

We used to tell kids not to use it, not to cite it; now we're like, “hey, you just can't cite it,” but we all know that when my kid asks the question why something happens and I don't know the answer, I'm going to Wikipedia. 

So I think there is something revealing in that about how the tech industry's understanding of what's good and bad about a given trend may often be quite disconnected from how policymakers with a non-technical background might view the same thing.

Government and agile development

Patrick: There is definitely a usage of the same words in very different fashion. For example, both the tech industry and the government love Agile development. The tech industry calls the government's version of Agile development “Waterfall.” But if you read those RFPs that are creating systems like California MyTurn, etc., many of them at this point will mandate Agile development practices – and they're mandating a Waterfall practice, like, point blank.

You can't run a government RFP, in the current constellation of how the United States does software procurement, in a truly “Agile” way. You can't run it like any high status software company runs their software products simply because – and I'm stealing this observation from Dave, who I should probably have on the podcast at some point… not this Dave, other Dave – in the software industry, you ship software and then 90% of your budget is spent in that live service mode of continuing to update software in response to what your users actually do with it. 

In government-land, 90% of the budget and effort is, this is not something that a project manager has individual scope or authority to change. It's been budgeted; the RFP had a number in it; we gave a milestone payment, so 90% of the total budget of this has to be done before launch. Then you have 10% for bug fixes, maintenance, and operational issues – and operating software, man. If you thought software development was low status in the government, wait until you see operational roles. I think there are some decision makers that probably consider it a point of pride that they haven't talked to anyone who has actually talked to a constituent in six years. (That might be overly cynical, but there is a deep truth to it though.)

Dave: Yeah, there is a deep truth to it. I also think one of the things that's quite challenging about government – as someone who has over the years had to translate between, call it private sector lenses on it, versus sort of public sector lenses on it – is that, to some degree, government can't decide to start or stop doing certain things without various legal rituals. 

You can't pivot the Department of Defense to healthcare. (I mean, it is actually arguably the world's largest healthcare provider anyways – that's a separate discussion.) It can't stop operating tank divisions altogether without Congress saying, “Hey, US Marine Corps, we agree with your new plan. You don't need to have tank battalions anymore because you're pivoting to preparing to fight a conflict in the Western Pacific and so you're going to buy missiles instead.” Without that, it's actually a crime for the commandant of the Marine Corps to stop maintaining his tank battalion until he gets that permission. 

Bureaucratic constraints and historical practices

So in the same way, all of this, yes, like it is a problem of government, but to some degree, the call is coming from inside the House (and the Senate) in terms of where the actual problem is originally emerging – we've decided a set of budgetary practices and a set of governance practices that embed that problem all throughout our budgeting cycle. It's not just the fault of some mid-ranking government bureaucrat. It really is, unfortunately, a historic budgeting practice that we've been embedded for far too long, at least as far back frankly as the space program.

Patrick: Yeah, and this is not merely an abstract problem. One of the things with the federal effort for displaying this data was that it was built on top of system which was displaying vaccine information for the annual flu shot, which is an important drug for an important condition that kills a lot of people, but maybe a little less critical to the United States than the COVID vaccine was in 2021. 

When that system was getting rolled out back in the day, various pharmacies told the government, “Look, we are happy receiving flu shots from you. We're happy administering flu shots to people who come in and ask for the flu shot.”

“But vaccines are a bit of a live political issue in the United States, and you say for bureaucratic reasons, we need to have a licensed pharmacist sign off on every submission to you of the flu shot inventory data. That is a person who is our employee, who we have a duty of care for, and who reports to a predictable address at a predictable time every morning; we would rather not like that person getting abused as a result of participation in this program, and so, as a condition of our participation in this program, we would like the United States of America to promise that the information that we type into this web portal never leave the United States of America.” And the United States of America in like 2012 said, “Sounds reasonable.” 

Fast forward to 2021. We're on the same technical and legal substrate as the 2012 flu shot was. And we, VaccinateCA, told people on phone calls with us, “Hey, clearly, we would not have been okay with that if we had known that that agreement would affect the ability to publish information about the COVID vaccine in 2021. Why don't we just ignore that agreement?” 

Responsible people who are socialized to not break the law would say things like, “Well, I don't have the authority on behalf of the United States of America to override a contractual term.” Well, who does have that authority? It's not going to happen. We can start the ball rolling on that, and possibly it will be done by the time there is another worldwide pandemic – but you're looking at like an 18 month time frame to get contracting changed. 

It's like, “oh goodness,” and one has a flight of fancy like “Could you please look the other way for a moment? I'm going to type some stuff into the computer and then the United States of America will continue promising the thing it promised, but maybe like an if statement will be deleted.”

That was not a form of looking the other way that happened], although I think there were some net positive looking of the other way that happened in various places at various times.

We have rituals of decision-making. Some of those rituals of decision making are truly in our best interest, some of them are socially important, and some of them are… I think Jason Fried said it best when he said that bureaucracy is scar tissue based on previous traumatic events, and those previous traumatic events happened. 

Like, a lot of the craziness around NASA, for example, is downstream of the Challenger explosion and also downstream of many people lining up for pork for very many years and us wanting to cut down on the corruption and have more efficiency in government; all good things. [But] we found ourselves – the societal ‘we’ – we found ourselves very constrained by scar tissue that had piled up over the years.

[Patrick notes: Previously, on Complex Systems.]

Dave: Yep. I think the even worse part on top of that is, one of my very close friends who does a lot of work with the Department of Defense describes the situation there as sort of that everything has been broken – then you had to break other stuff on top of it in order to be able to make things work on a day-to-day basis; then over time, that level of brokenness has new brokenness built on top of it, and so you're seven layers of workarounds and emergency fixes and “God, we need to get this to the battlefield right away” kind of things. 

That means that when you actually need to do something that is new and novel, you're not just starting fresh. You're actually having to first deal with all of the trauma that has previously happened, both literal and metaphorical, in order to get people to the state where they're willing to be able to do something new. And I think that is dramatically underrated as a cause of all this. Another way of saying this is in the industrial safety world, they say like, “every safety rule is written in blood.” That is much more true for a lot of things in government.

Patrick: I think there are second- and third-order consequences of it too. It is not in fact the case that there are no smart, high-agency people in the United States government. 

They are required to turn it on every Tuesday and they are in firefighting emergency mode every Tuesday. It would be easy for me to say, as the person who made it in mission his mission in life for a while on this issue specifically, like, “there should be no higher priority for the United States than finding the freaking COVID vaccine for the next couple of weeks” – but from “true priorities of the United States” perspective that isn't actually the case.

[Patrick notes: I feel like a crazy person for saying this, but we literally did not know where it was. To the very limited extent we perceived this to be a problem, we commissioned Accenture and similar to make a magical system that would collate all the Excel files and databases and tell us where it was. We did this several times. Accenture cashed those checks and said, essentially, “Our conclusion is we don’t know where it is. Here’s some crappy software, late. So, what’s the status on the other RFP? We won, right?” And we still, as you are reading this, don’t know where it is.

The reason that is not an ongoing national emergency is that instead of not knowing where the nukes are, that is a problem like not knowing where the milk is. Who cares; milk is abundant in the United States. Create abundance through spending massively is a strategy, and a strength of the United States, but is cold comfort to the people who died of covid in H1 2021.] 

Patrick: And a lot of our system-wide capacity for doing hard things was bent towards like, “if the people who have the authority to work across boundaries and fudge things (only in the ways that democratic society would eventually bless etc.) were to put their eye on this project, then Social Security checks would not get delivered in Kansas this month and there would be food riots.” [Patrick notes: That one is my go-to example of a non-defense-related federal computer system which kills people if it goes down for a few days. There are others. It is likely not the case, for structural reasons, that the people who keep that system from going down could actually meaningfully contribute to an emergency in a different fiefdom.]

Patrick:So the amount of brokenness in the system constrains the slack capacity of the system – also burns people out and results in differential attachment to the government versus other places that one could work, which have their own flavors of brokenness but maybe not that particular flavor of brokenness.

Dave: I think one of the things that I probably most differ from, call it sort of the “Bay Area perspective on government,” is that, being in DC and having friends for whom this has literally been a process they've gone through at various points in their careers, I've seen a lot more how it is that certain institutions that don't really make sense outside of DC suddenly start making sense. 

For example, why is it that we have think tanks? We have think tanks as a place to put political appointees when their administrations, their party side is not in power, in order for them to get some sleep – maybe write down one or two good lessons learned, but mainly get some sleep, and remember the names of their kids, because that's the only way that we can possibly in the first instance have people who are willing to sprint for two years working 100 hour weeks for, without loss of generality, the National Security Council, the CDC, the FBI, all these organizations that require this very high tempo of operation. 

Patrick: I think another point that you made to me in the past, which I didn't think about before you mentioned it, was that the think tank is like a paycheck for people who are living paycheck-to-paycheck in a very high-cost city, and would otherwise be… captured is not quite the right word.

There are only a couple of jobs you can do in policy, and if you are on deck for the next National Security Council, you can't go into the lobbying world because that's basically a one-way door. Well, you can go into the lobbying world, but it is a one-way door. 

And given that we need people to go out of power because the democratic processes kick their party out of power, but to be ready to, with a couple of months of advance warning, run the civilized world, you need some place to park them.

Dave: Yep, that's absolutely right.

Patrick: That, by the way – phew! – I would love to hear from insiders more than from outsiders, but the fact that there was a presidential administration changeover in early 2021 had non-zero relevance for like why there was a lot of chicken running around with their head cut off on the policy side of things, because the people making the United States’ strategy for this were in some cases just getting their government issued email address days before needing to work on this problem. But be that as it may. 

So stepping back from the VaccinateCA experience and towards this broader question of policy engagement and understanding how the Complex Systems (bum bum bum!) that are the government work – us tech people and other broadly informed people who read the New York Times, etc., but we haven't lived in the DC ecosystem, what don't we understand?

Dave:

I mean the flip answer is, so much.

Patrick: (This is an acceptable answer too. We should understand that, just like tech journalists don't understand tech all that well, those of us who have read the New York Times for most of our lives don't actually understand how policy is conducted in Washington.)

Dave: Yeah, I think that's very fair. 

I think one place to start is just that when you think about the size of the population – just San Francisco proper, even ignoring sort of all the Bay Area, which I realize this isn't really apples to apples – but just San Francisco proper, about 800,000, 900,000 people; DC proper about 700,000, 800,000 people, and then sort of outlying suburbs of roughly equivalent size for each. 

I think the first thing that tech could do better is just recognizing that it's a world as big and as complex, at least, as tech is in the Bay Area – that there is at least much diversity of specialization, of career path, of motivation, of level of income, of level of experience, and more on some dimensions such as (frankly) distribution of age within the federal government, compared to the average tech workforce within SF (SF metonymously for sort of the broader Bay Area tech community.)

I think that complexity is where to start. 

Patrick: Can I just do a little microscopic view on the age thing? One thing that we see over and over in the tech industry is that the distribution of the company’s workforce tends to… like, your median age and the degree of dispersal increases as your company has existed longer. So without having seen internal stats, I can tell you pretty confidently that the median age at Google is higher than the median age at, without loss of generality, Stripe, simply because Google is older than Stripe is. 

Now, the United States of America: somewhat older than Google. (laughs) Many of the cabinet-level departments have existed for longer than, well, descriptively speaking, the entire tech industry, and so you get much greater diversity along that spectrum than you do in tech. Senior decision makers tend to be monotonically older, etc.

[Patrick notes: The median age of the U.S. Senate is 65. Bill Gates is still young in that crowd. Which informs many things about the life experiences, etc, available to senators relative to those who e.g. build tech products and need to make governance decisions for global customers using them.]

Patrick: Anyhow, that little digression out of the way.

Dave: Yeah. And I think what that means also is that, within the DC area’s sort of equivalent [to the] software engineering community, let's be clear: there are people within this community who are radically underpaid by the standards of the tech community for equivalent levels of performance. 

Government workforce and technological challenges

So to take an obvious example that almost everyone on this that listens to this podcast has probably interacted with in one sense or another, or at least is familiar with: the IRS successfully runs everyone's tax returns every year.

They have this thing called the individual master file. That is what I, when I'm filing my 1040s, am interacting with. It takes in my information, it has a bunch of processes that make sure it gets into the right databases, it processes it well – and at the output of it, the United States government correctly figures out if I've underpaid or overpaid or paid just right, and refunds or accepts payment from me accordingly. And we use it to buy things like schools, social security checks, meals on wheels, aircraft carriers, right? This works as a deterministic process reliably year after year. 

The average person who works for the IRS in this function – this is easily findable by looking at inspector general reports – is near or at retirement age.

So they're basically choosing to stay in their offices in order to keep America running and make sure that we have money to pay for everything, kind of out of the goodness of their hearts, and because they enjoy working there. The IRS is not unaware of [this]; they're trying to hire a new generation of software engineers and to modernize their infrastructure, as well as just keep the lights on – but at a first approximation, this workforce has delivered on a very complicated mission using very low power (by modern standards) mainframes, very, very well and gets basically no credit for achieving that technical feat. I think things like that are things you should probably foreground a little.

Patrick: They're also a much smaller team than one would expect given the importance of the mission. The total headcount of engineers in the IRS is what, three figures, if that? Low three figures?

Dave: I think it's probably in the thousands, all told, because there's not only just individual master file, there's also the business-equivalent functions, there are a bunch of other backend stuff.Someone who actually is deeply familiar with this technical architecture and hasn't just read a couple IG reports could speak about it in more detail, but there actually is a lot of planning on the backend for… call it ‘all the edge cases.’

Patrick: I will pop the real number in the show notes. So folks, as you're listening to this, my first question is, how big is the IRS in terms of headcount? Compare that to a large tech company of choice. Then two, how many engineers do they probably have? Then read the show notes for the actual answer.

[Patrick notes: 82,000 FTEs so approximately half a Google. Of which, 7,200 in IT, or a Stripe engineering team and some change. I’d caution that an apples to apples comparison is difficult because many of them report into IT-implicated functions but are… other than technical in terms of skillset and job duties.]

Another thing is that I feel like we in tech mention this a lot – and people who are in and around government say “that's not really all that important,” but I think that this is one of the things where we better calibrated than they are – is that government technologists don't earn that much money. 

The people who are what we would call Staff Engineers in similar positions in the tech industry (who do not have reports) who work for the IRS probably earn in DC like $80,000, $90,000 – where we would not countenance paying college interns that much.

[Patrick notes: Useful search term would be GS-11 and GS-12 if you want to fact check me. I think an important proviso is the United States, as an intentional strategy, lies to itself about what it actually pays public servants, for example by building a lot of the package in deferred comp and healthcare benefits then pointing at what you’ll find for a GS-11 and saying “They’re so downtrodden! We should award them more deferred comp!”, but there really is a very large gap here.] 

Patrick: Iterate that game over a period of decades; that does some very particular things to one's workforce composition. One reason that the workforce is heading into retirement is, which engineer who has graduated college in the last 20 years would you tell, “you should probably move to Washington and work on the master file, versus doing Postgres optimization for Google?”

…Google doesn't use Postgres that much, but you know, without loss of generality, anybody who uses Postgres pays more than the IRS does.

Dave: Yeah. A couple of things to note with regards to that. 

The first is that I think it is to the great credit of the United States that part of why it was able to attract that technical workforce was a couple of reasons. 

The first was – and thankfully this is much less the case than it was in earlier decades – the United States Civil Service has historically been quite good at selecting for talent that other parts of the economy might have discriminated against.

[Patrick notes: Hear hear.] 

Dave: Obviously, and of course unfortunately, race and gender was one big dynamic there. Another, particularly in the IRS context, for example, is that the world's largest deaf university, Gallaudet University, is in Washington, DC. A bunch of government agencies, the IRS among them, realized very early on that for the cost of hiring a relatively small full-time sign language interpreter and translator pool, they could effectively be the employer of choice for that workforce who actually could, as a result, fully participate in all workforce conversation in a time when IM was much less of a thing, when automated captioning was much less of a thing. (That was a bigger advantage than it is now.)

I also think it's worth noting that when you do – and I've done this in my professional life, when I was at McKinsey in particular – when you do workforce surveys of government employees (or look at existing workforce data, some of it's public) and ask them why they stay, the number one answer, consistently over and over and over again (depending on how you bucket it) [it’s] either one or two buckets: it's the mission and it's the people.

Sometimes people answer both as one sentence, mission and people, which is part of why I think of it as one bucket – because it is in their heads. They get to do something they can't do anywhere else, in many contexts at all, or at least not legally in the case of certain national security-related functions, and they're working with people who often are actually really interesting, fascinating people to get to know who are equally driven by that sense of mission and derive purpose from it.

And I think that dynamic and why it drives people is an area where I kind of wish that tech knew that generally more, because I think it's an area of commonality that isn't being explored to nearly the extent that it could be. The same people who want to put a dent in the universe in the Bay Area are the ones who want to put a dent in the universe here in DC – they should get more beers together and I think there would be more commonality in that regard.

Patrick: Yeah, I think one of the reasons that streams don't cross as much is that there are limited points of intersection between people who walk a path that leads to high status tech careers and people who walk a path that leads to government careers. For example, like you mentioned, the case of feeder universities – one near and dear to my heart: Notre Dame.

[Patrick notes: I didn’t go there, but two brothers did, and Notre Dame holds a certain cultural relevance for Catholic boys who grew up in Chicago, in the same way that Stanford holds a certain place in the hearts of many people who are not alumni.] 

Patrick: If you hypothetically had a lot of your social circle at Notre Dame, you'd know some high-intelligence, high-drive people and a truly disproportionate amount would end up in the FBI and similar. But most tech people, for obvious reasons, didn't graduate from there, and have no one in their social network that did – and so tech folks' impression of the median government employee becomes the impression of the median government employee that they've talked to. 

And for similar reasons that I wrote about in an essay called Seeing Like a Bank, the people who are retail-accessible in government… we don't send our best. That falls out of a model of costing for retail services, etc., which every financial institution, large tech companies, et cetera – everyone is operating under fundamentally the same constraints. You can't make “project manager talks to senior engineer lead” happen on most retail conversations with the government, and keep the cost model alive in a way that most people can actually speak to the government.

[Patrick notes: Actually you can via the side channel which is “if you have an issue with someone at an agency, speak to your Congressman’s Constituent Services staff, where you’ll discover people of a shockingly different educational level, social class, etc who perform retail customer service work at a high standard.” But I digress.]

Patrick: The government considers it pretty important that most people can speak to it in various capacities, and so the frontline experience is going to be rough – and if your only experience of the government has been the frontline experience of it, you're going to have a pretty low set point for the typical government employee without necessarily capturing some of the staff-level folks who work on keeping the world running.

Dave: Yeah, and I think on some level that's a blessing – it means that you haven't been in a room where they needed to bring the A-Team talent in, which, you know, if you're trying to avoid regulation or government action [that] might be what you prefer. But, I think it also means that the first time – and this has happened to me a few times in my career, I've had the privilege of being like in the room where like the A-Team has actually been like pulled in for one reason or another – and you get to just hear them talk and you're like, “Oh. This is a different level of capability and speed of movement that any organization on earth, including the highlights of the tech industry, would be absolutely thrilled to bring on board any day of the week.” There's this line from one of my favorite writers, Charlie Strauss, who writes a series of science fiction-y spy novels set in Britain.

At the end of one of the novels, after defeating yet another end of the world scenario, our main character's boss tells them that if they could be fairly bonused out for what they would receive, they would get their new luxury yacht – but insofar as they can't, all that they're getting is a little time off and a very small promotion. We are quite frankly lucky that there are people that have that level of performance who are willing to do so.

[Patrick notes: Pace Inadequate Equilibria, a properly functioning society should have set a $1 trillion bonus pool aside for the CDC delivering the best work America is capable of, given them a $1 trillion budget, and asked them to send a text message if they needed another trillion. We instead didn’t break with our traditions of compensating government employees. They didn’t break with their traditional expectation for the importance of achieving results.]

Dave: I think also one more thought on this is that this often causes people to think that when government individuals are moving into the private sector, it's kind of influence peddling – look, I'm not naive. That is indeed sometimes the case. But I think far more often than is commonly understood, it's like for 20 years now, their best friend from college has been saying, “Hey, let me know when you retire. I really would love to work with you.” And they retire, and have kids going to college and a mortgage payment that they need to make, and all of a sudden they're like, “You know what? Sure. I'm willing to go work in the private sector and bring some of my capability there” – and they often flourish, not always, but often.

Patrick: You would point yourself more in an influence-peddling direction if you found that the people employed were wildly under baseline expectations for high status employers in the private sector, or if they were simply uniquely providing introductions, etc., but these folks do actual work in many cases, and you can, if you've had the opportunity to work with them, you can observe the actual work. 

I had a colleague who had previously worked in the Obama White House working with me on Stripe Atlas and she was pretty exceptional. One of the things I learned in one of our conversations was that, the first time everyone does a trip with the president – meaning flies to a foreign nation with the president for his first international trip, which is the first for his staff – for literally everybody on that plane, it's the first time that they've flown with the president to a foreign nation. 

And so everything is like invented de novo with the possible exception of some sort of institutional culture in the Secret Service that helps with a tiny part of the puzzle – but everything else, like, we're making it up as we go along. Which is a bit terrifying about the knowledge management within the United States government, but is kind of impressive with regards to, “Wow! We've lost a total of zero presidents on those international tours. Good job guys!”

[Patrick notes: This is not nearly as serious as e.g. the traveling crew for globally recognized musicians takes this problem, as you’ll learn if you ever end up sitting next to the advance team on a flight to Tokyo, but plausibly this is because, for presidents, we could get another one of you in a minute so don’t you ever get to thinking you’re irreplaceable.] 

Dave: Yeah. There I think one of the things that is particularly impressive about that cadre of individuals, when you get to meet them, is that you realize that above a certain level of playing the game – which is well above the level that I have ever been at in terms of, like… one, as a government contractor or management consultant, you're on the outside anyways, but even in terms of who I'm interacting with, well above that level, call it the undersecretary or the secretary level, or the assistant to the president level – to some degree, it is dependent upon personal relationships. It is dependent upon a personalized level of trust that is non-transferable. So as a result, part of why it's being made up on the fly is that actually what that president needs is different than what the last president needed. 

There are many examples of this that are talked about, one that is public: you can go on the CIA's website right now, and they will have many detailed pages that they talk about how proud they are of how they have adapted the president’s daily brief, the most important document in the intelligence community, given every single day – what it looks like for each president has changed quite a lot, from at one point being a little tiny pocket guide small enough it could fit in a president's pocket, to being on all the different modern briefing technologies of the day, depending on what that particular president wanted. They're very proud of that, as well they should be. 

But also that means like, if they literally run the playbook that had worked for the last president on that overseas trip, the president might not have picked up on an important message like, “hey, such and such foreign leader you're about to meet with, it's really important you know, fact XYZ” – they might have tuned out when that was given, and those plausibly have significant consequences attached. So in order to do this well, on some level, at that level of play, you have to be fairly bespoke.

Patrick: For various reasons, including that I lived most of my life in Japan (one has the usual reticence of an immigrant to delve too much into one’s host nation’s politics), I am less than politically engaged in a way that many people in the tech community are less than politically engaged.

The importance of policy engagement

I don’t want to radically raise the level of partisanship in this conversation, but it's become apparent to me over the last couple of years that if we don't engage with the government, we will get engaged on by the government.

[Patrick notes: Many of us in tech were confused about the Cambridge Analytica kerfluffle, because believing it actually swung an election requires one to not understand how advertising actually works. Happily, two great career paths if you hate understanding Facebook policies are the national security state and political reporting. They RLHFed each other into passionately believing a hallucination. It was weird to watch on Twitter, but didn’t seem that important at the time.

Fast forward a few years and there is now an official at the White House who understands his job to be using the tech industry as a catspaw to create a flagrantly unconstitutional censorship apparatus. The Supreme Court case about it was bounced on procedural grounds, but I think Justice Alito gets the importance of this issue to our nation basically correct in his dissent.]

Patrick: What can we as a community be better at here, understanding our counterparties over there?

Dave: So I think this is something that clearly you and I agree on a great deal. I think the top thing that I could say is that I think there is a default posture of mutual distrust and contempt between tech and government. To some degree, this is probably what the founding fathers intended, sort of ambition balanced against ambition, sure – but I think there is a lot of room for dramatically improving the level of mutual conversation and trying to share information that's helpful in sharing perspectives.

For example, I work on, as we mentioned at the top, I work on AI policy. I consult for a little AI nonprofit called Control AI based out of London. One thing that I think is actually really positive and notable – and the tech industry should kind of give itself a pat on the back for – is that they have so far largely managed to keep AI policy in particular from having partisan polarization. 

We're starting to see a little smidgen of it here and there, I won't deny that – but by comparison to anyone's expectation two or three years ago, far less than expected, when you go to a congressional hearing on this, it is by large members of both parties trying to engage in a genuine search for truth and insight and figuring out what a good regulation or a good policy might look like. That is something that we should seek to preserve as much as possible.

There are lots of issues like this in American life – we are polarized, but there are many that don't need to be. For example, we don't especially have partisan political polarization on the question of, “Should we attempt to avoid nuclear war?” The US government, if you look at a congressional research service report, spends three quarters of a trillion dollars over a 10 year period on things related to preventing nuclear war in various ways and deterring nuclear war.

That is the level of holding off the apocalypse in a nonpartisan way, or least largely nonpartisan way, that the American people are actually capable of.

Building something new like that, versus just running the playbook that's existed in one form or another since 1945 – that's more difficult. I think the more that tech can help people in Washington, DC wrap their heads around it, the better. When I go to lots of meetings and think tank talks and things like that about AI policy-related stuff in DC, there's a big disconnect between even what you would call the more moderate expectations that exist in the Bay Area about where AI is going – or for that matter technology in general – versus what Washington DC expects to be the case on any tech topic, but particularly AI. 

You know, the average thing if you're walking around SF or Berkeley, it's like, “Hey, we're really concerned about the world-changing potential of this AI, and if it fails in its ambitions, it's only going to be as transformative as the internet was. If it succeeds, very plausibly, particularly if you listen to some of the frontier labs who are working on this, it could be the final technology that changes all of humankind's history forever and poses existential risk to our own survival.”

DC doesn't really get that yet. The National Security Advisor gave a talk at an event recently where he said he thinks that AI might be the most important technology in a hundred years or even a thousand years – and then the examples that he gave after it of the ways in which it might be important, frankly could sound like he was reading off the playbook of describing Washington's concerns about the tech industry from call it the early 2010s. 

Things like misinformation, things like bias – not saying those aren't important, not saying people can't have nuanced opinions on them, but ignoring some of the transformational power and potential.

Patrick: I think one of the things that's occasionally been thrown around is that there is a posting-to-policy pipeline where simply writing blog posts that explain something in a way that can be understood by a bright college graduate who does not have, say, a PhD in applied mathematics and knows what a tensor is, is hugely influential, particularly when you go one step beyond the blog post, although there are certainly blog authors who are on the org chart of the United States government in the same way that the New York Times is on the org chart of the United States government. (Bracketing that for a moment.) 

Going one step beyond the blog post, one step beyond the essay, call it a white paper, call it what you will, but get to actual concrete policy proposals with a menu of proposals, and “here are our organizations’ considered versions of the tradeoffs,” and then intentionally circulating that with people – the decision makers in various arms of the government are allowed to use the open internet. In fact, they have divisions upon divisions of people who do nothing but use the open internet all day. But there are rituals, let's say, for getting a particular white paper in front of a decision maker and getting that blessed by committees of decision makers, etc. – and we should get better at understanding that those rituals exist.

And also, write the freaking white paper if one thinks that the future of the world literally depends on the government adopting a particular strategy.

[Patrick notes: This is not a subtweet of the people who have had this conversation with me re: AI regulation, but it is a piece of pointed advice to people who think that AI regulation at some particular setting is very important for society.]

Dave: 100% agree. I think the one downside of the “move fast” culture is that I think it causes people to have a lot of good ideas sprinkled across seven forum posts, three tweets, and like eight Google docs that they wrote over the course of a half decade. That's great, that's awesome, like in my day job, I steal from that thinking (well usually I cite it) but I'm using all of that thinking as is everyone else I know in the policy space – but having more of an ability to say, “Hey, here's how that all fits together. Here's how thing A kind of conflicts with thing B, and we figured out a way to make them work together without having mutual interference with each other” – it's actually quite difficult. 

Patrick: I think the tech community largely has an oral culture, which is funny... we would not consider this to be true about ourselves, but an anthropologist visiting the tech community would say, “Well, you don't have foundational documents, but you do have a certain number of thinkers that have relatively informal spaces that are considered hugely influential. I think that's an oral culture, dude.” The tech community is in many ways an oral culture. 

DC in some ways is, and in some ways, very much is not. It coalesces around particular identifiable documents that have been blessed by a process, etc. and the price of admission for getting into those discussions is like, you need to get your documents into the pipeline. You need to start getting them stamped by people. So having documents that are stampable in a way that consensus among the smartest people you know communicated over Twitter is not stampable [is important]. The New York Times editorial page: more stampable than [consensus on Twitter] is, but still not stampable. 

Just playing the game a little bit better, being willing to say, like, “Messaging is messaging, but between friends, this seems like a game that is important, which we should be playing like we want to win. Let's play it like we want to win it” would be an improvement in our community's culture at the moment, to the extent that I can speak for a very large diverse community, yada yada.

Dave: And you know, as always, Patrick and I only speak for ourselves and not for anyone else.

Patrick: He made the disclaimer so I don't have to. (laughs) Thank you! 

Dave: Yes, I figured someone should get to it. I think one thing that I would say on this point is, “reflect on the fact that the New York Times gets way more excited when it has a document leak or it can point to a document that's been released than when it knows the same information from its own reporting.”

That's kind of a stunning social fact – that at the end of the day, a journalist would rather have a document in their hand that someone else did than have reported it out by the sweat of their own brow.

[Patrick notes: I’d note that there are, conversely, actors in government who’d consider the NYT saying something to be much more useful than them saying it, for example because that means then they can cite it without having said it. To say your organization has failed will cost you points; to say that your organization needs to take immediate action during this news cycle to avoid taking further hits is just good optics.]

Dave: Part of that is because it's more verifiable; part of it is because they know how much effort went into making a document, versus just someone saying something over a coffee or a cocktail. So I think the more that you can find ways to produce outputs that are metabolizable by Washington DC, the more impact that you're going to have.

DC equally has an obligation to, quite frankly, figure out how to make itself more metabolizable towards tech; I think hopefully over time, those two will get into a better place. If you think that certain policy trends are going to need very important decisions being made very soon now, particularly in AI, for example, you should try now, to start getting your reps in on doing that because you're going to be good at it by the time that it matters.

Patrick: And start making the relationships that, like all communities, DC eventually depends on.

Dave: Absolutely.

Patrick: I think it is surprisingly operationalizable, to turn into things that any individual is capable of doing: Write an essay – don't call it a blog post.

[Patrick notes: Situational Awareness was a great exploration in terms of form factor. Morally speaking, that’s a blog post, but nobody calls it a blog post, and I would bet all further returns to my zero shares of Nvidia stock that it is more effective at reaching decisionmakers as a result of that packaging decision.]

Get it in the hands of people. Get it in a place where people who make decisions for various policy organizations are known to read. The list of those blogs is not very hard to think up, and you can punch way above your weight with that. You can punch way above your weight by getting an editorial into a newspaper of a large American metropolitan area, which sounds like it's a high bar, and my neighbor did it like days ago. (laughs)

So there are those arbitrages. Get good at those arbitrages and understanding where the system has more points of leverage than you might naively expect. 

Get good at playing that game, if this is truly important to us – and I think the future of AI is very self-evidently important to everyone in the tech industry and outside of the tech industry. I also think that broadly preserving the ability for there to be positive relationships after many years of there being very negative relationships (for bad reasons) is just self-evidently important. I think there's often an us versus them dynamic and I talked with Kelsey about that the tech community vs. tech reporter community finger pointing – regardless of where one would draw the lines of fault or how we ended up in this weird situation that we find ourselves in, there are +EV strategies and tactics available to us, and we should independently decide to take those strategies and tactics that are plus-EV while trying to work towards our counterparties understanding us better than they do today. 

So that is my final thought. Do you have any final thoughts for folks, Dave?

Dave: Yeah. I think my final thought on this is simply, DC is a seemingly complicated place. It is a place where demonstrably we know, at scale every year, thousands of young Americans between the ages of like 21 and 30 move here, and figure this place out – and end up having a really big impact on it. I have a friend who plausibly is the reason that the Affordable Care Act covered a particular medical procedure; she did that at the age of 26, I believe it was.

You can do that too. So consider trying to do that, on whatever your issue is that matters to you – and please always feel free to reach out to people like me to ask for advice. It is a social ritual in DC that is much more important than you might imagine, to be someone who is giving advice and passing on the advice that you yourself received at a slightly earlier stage.

Patrick: Cool. So where can folks find you on the Internet, Dave?

Dave: Folks can find me in a couple of places on the Internet. 

The first way is my occasional irregular Substack where I write about these kinds of issues as well as whatever else catches my fancy, davekasten.substack.com.

The second way is on Twitter – confusingly, there I use the other form of my name, so I'm @David_Kasten on Twitter. Always feel free to reach out if you want to talk about AI, policy, how the government can be better, or just want to chat.

Patrick: And as always, I'm patio11 on Twitter, which will continue being Twitter until the heat death of the universe. I don't care about corporate renamings. 

Dave: Amen.

Patrick: Thanks very much, everyone, for your time and attention. We'll be back next week. And thanks very much, Dave, for coming on.

Dave: Thank you for having me.