The Role Models - The Escalator - transcript
Disclaimer: a dramatization based on public records, including court filings, published emails and company posts. Some scenes and dialogue are reconstructed or composited. Voices are AI-generated and perform words from the public record; they are not intended to resemble any real person. Not affiliated with or endorsed by OpenAI or any person depicted. Allegations in court filings are unproven claims.

[0:01] Narrator: Tuesday, November fifth, two thousand nineteen. Nine months after OpenAI said it wouldn't release GPT-2, it releases GPT-2. All of it. The full model, one point five billion parameters, with the code and the weights.

[0:20] Narrator: The post lists what they learned. People found the text convincing. Researchers at Cornell had people rate its credibility, and the full model scored six point nine one out of ten. Researchers at the Middlebury Institute showed it could be fine-tuned to write extremist propaganda. OpenAI built a detector that caught about ninety-five percent of the full model's text, and said that wasn't good enough on its own.

[0:48] Narrator: And then, finding number four.

[0:50] Narrator: "We've seen no strong evidence of misuse so far."

[0:55] Narrator: That's the whole arc of the most famous AI scare of twenty-nineteen. Locked up in February. Called "too dangerous to release" by the press. Let out in pieces. And by November, released in full, with no strong evidence anyone had used it to do harm.

[1:14] Narrator: You can hear that as proof the caution was overblown. Or as proof the caution worked. OpenAI's own post did neither. It said they couldn't know about every threat, and that people who wanted a model like this could build one without them.

[1:31] Narrator: Hold on to that, because this episode is about the year that question stopped being theoretical. In twenty-nineteen, OpenAI argued about whether to let one model out of the building. By the end of twenty-twenty, it was selling access to a model more than a hundred times bigger, it had licensed that model to the biggest software company on earth, and one of the people who'd helped build it was walking out the door.

[1:59] Narrator: None of that happened in one dramatic meeting. It happened the way an escalator moves. One step at a time, and you only notice how high you are when you look down.

[2:14] Narrator: From Will Sigmon Media, this is The Role Models. Season one: OpenAI. Episode six. The Escalator.

[2:26] Narrator: Back in February, when GPT-2 was still locked up, OpenAI's head of policy, Jack Clark, gave the Guardian a phrase. I promised you we'd come back to it.

[2:38] Narrator: "I have a term for this. The escalator from hell. It's always bringing the technology down in cost and down in price. The rules by which you can control technology have fundamentally changed."

[2:52] Narrator: An escalator. You don't climb it. You stand still, and it carries you up anyway. Clark's point was that you can hold back one model, but you can't hold back the falling cost of building the next one. And the GPT-2 release in November said almost the same thing: other people could replicate it without them.

[3:14] Narrator: In the same interview, he said something else. "We're trying to build the road as we travel across it."

[3:21] Narrator: In twenty-twenty, OpenAI stepped onto the escalator on purpose. And they built the next model, and the road for selling it, at the same time.

[3:34] Narrator: January twenty-third, two thousand twenty. Before GPT-3 had a name, ten researchers at OpenAI posted a paper with a title that sounds like a physics lecture: Scaling Laws for Neural Language Models.

[3:50] Narrator: Its claim fits in one sentence. Read it slowly. "The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude."

[4:05] Narrator: Loss is the error, how wrong the model is. And a power law means it's predictable. Make the model bigger, give it more data and more computing, and it gets better, along a line you can draw in advance.

[4:20] Narrator: That's the escalator. If you believe that line, then the question is no longer whether bigger models will be smarter. It's who can afford the next step.

[4:31] Narrator: Look at who signed it. The first author was Jared Kaplan. The second was Sam McCandlish. The last was Dario Amodei. Remember those names. They'll be on a different letterhead soon.

[4:47] Narrator: Three weeks after that paper went up, on Monday, February seventeenth, twenty-twenty, MIT Technology Review published a long profile of OpenAI. The reporter was Karen Hao. She had spent three days at the office, and done nearly three dozen interviews with current and former employees, collaborators, friends, and outside experts.

[5:12] Narrator: The headline: "The messy, secretive reality behind OpenAI's bid to save the world."

[5:19] Narrator: What follows is her reporting. Many of her sources asked not to be named, because, she wrote, they weren't authorized to speak or feared retaliation. OpenAI answered some of it on the record, and I'll give you those answers too.

[5:35] Narrator: She opens with a ritual. Every year, she wrote, OpenAI's employees vote on when they think artificial general intelligence will arrive. It's mostly a way to bond, and the guesses vary a lot. But, she reported, half the lab bet it was likely within fifteen years.

[5:56] Narrator: Then, the building. Eighteenth and Folsom Streets in San Francisco, an old truck factory with tinted windows and most of the shades pulled down. Inside, two conference rooms on the first floor. A big one, named A Space Odyssey. And a small one, she wrote, "more of a glorified phone booth," named Infinite Jest.

[6:21] Narrator: That was the room she was restricted to. The second and third floors, where the desks and the robots were, were off limits. When it was time for an interview, people came down to her.

[6:34] Narrator: Greg Brockman met her on the first day. She wrote that he looked nervous and guarded, and that he said, with a tentative smile, "We've never given someone so much access before."

[6:47] Narrator: At lunch, he talked about the moon landing, the transcontinental railroad, Edison's light bulb. And the charter. Hao wrote that he recited it "like scripture." At one point he stopped to explain himself. "I guess I know all these lines," he said, "because I spent a lot of time really poring over them to get them exactly right."

[7:10] Narrator: That charter, from our episode on the divorce, turned out to be more than a mission statement. Hao reported that after the switch to the capped-profit company, the leadership put in a new pay structure, based in part on how well each employee absorbed the mission. She described a spreadsheet tab called "Unified Technical Ladder." The last column set out the culture expectations for each level.

[7:38] Narrator: Level three: "You understand and internalize the OpenAI charter." Level five: "You ensure all projects you and your team-mates work on are consistent with the charter." Level seven: "You are responsible for upholding and improving the charter, and holding others in the organization accountable for doing the same."

[8:00] Narrator: Then she got to the part about secrecy. She reported that a team called Foresight ran experiments to see how far they could push AI by training existing methods on more and more data and computing power. And that, for roughly six months, the results were kept from the public, because OpenAI saw that knowledge as its main competitive advantage. Employees and interns, she wrote, were told not to reveal them, and people who left signed nondisclosure agreements.

[8:33] Narrator: Only in January, she wrote, did the team "quietly" post a paper on one of the main open research sites. She doesn't name it. But a few minutes ago, we read a paper from OpenAI posted to that kind of site that January, about exactly that kind of result. Draw your own line. And Hao added that a paper from other researchers, with similar results, had been posted a few months earlier.

[9:01] Narrator: She described her own experience of that culture. After her first visits, when she started contacting employees directly, the head of communications reminded her that all interview requests had to go through her. When Hao declined, she wrote, employees were told to keep communications informed of her outreach. And she reported a Slack message from Jack Clark, a former journalist himself, praising people for keeping a tight lid while a reporter was, quote, "sniffing around."

[9:34] Narrator: OpenAI's answer, through a spokesperson, pointed back to the charter. The line from twenty-eighteen: "We expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research." And they added that each release went through what they called an "infohazard process," because they wanted to release results slowly, and understand the risks, "before setting loose in the wild."

[10:07] Narrator: Hao also sat with the man running the research. Dario Amodei. She wrote that he seemed like a more anxious version of Brockman, a hand tugging at his curls while he talked. He described the strategy as a portfolio of bets. And about language, the bet behind GPT-2, he said this: "Pure language is a direction that the field and even some of us were somewhat skeptical of. But now it's like, 'Wow, this is really promising.'"

[10:38] Narrator: And on safety, he was frank about how unsure everyone was. "We're in the awkward position of: we don't know what AGI looks like. We don't know when it's going to happen."

[10:51] Narrator: Near the end, Hao describes the money. After the Microsoft deal, she reported, the leadership had told employees that commercial work would be far away. But in recent months, she wrote, that pressure had grown. And she summarized Sam Altman's private twenty-twenty vision for the staff this way: OpenAI needs to make money in order to do research, not the other way around. Those are her words, describing his message. Not a quote from him.

[11:22] Narrator: The profile went up on a Monday. That same day, one reader who used to be very much on the inside answered it on Twitter. Elon Musk, two years off the board.

[11:35] Elon Musk (post): OpenAI should be more open.

[11:38] Narrator: He ended it with three letters. I-M-O. In my opinion. And in a reply that same day:

[11:48] Elon Musk (post): I have no control and only very limited insight into OpenAI. Confidence in Dario for safety is not high.

[11:57] Narrator: Dario. The man Hao had just described as the one driving the strategy. And another, which reads differently now than it did then.

[12:07] Elon Musk (post): All orgs developing advanced AI should be regulated, including Tesla.

[12:12] Narrator: Musk didn't explain the comment about Amodei, and Amodei didn't answer it in public. Remember the name of the man he doubted. In this story, he's about to become one of the loudest voices for AI safety in the industry.

[12:31] Narrator: First, a smaller post, on May fifth. Two OpenAI researchers, Danny Hernandez and Tom Brown, measured something nobody had pinned down before: how much less computing it takes, each year, to train an image-recognition network to the same level of skill.

[12:50] Narrator: Their answer: since twenty-twelve, it had been halving about every sixteen months. Training a network as good as the famous twenty-twelve system called AlexNet now took forty-four times less computing. By comparison, they wrote, Moore's Law would have given you about an elevenfold improvement over the same stretch.

[13:13] Narrator: Think about what that means for Jack Clark's escalator. It isn't only that chips get cheaper. The recipes get better too. Whatever is hard and expensive this year will be easier and cheaper for everyone a few years later. The post ended with a question it didn't try to answer. If this kind of progress keeps compounding for decades, "what might it lead to? We're not sure."

[13:39] Narrator: Thursday, May twenty-eighth, two thousand twenty. The world is in its third month of lockdown. And OpenAI posts a paper with a dry title: Language Models are Few-Shot Learners.

[13:53] Narrator: It introduces GPT-3. One hundred seventy-five billion parameters. GPT-2 had one point five billion. That's more than a hundred times bigger. The paper says it's ten times bigger than any previous model of its kind.

[14:11] Narrator: The title is the discovery. Before this, to teach a language model a new task, you'd retrain it on thousands of examples. GPT-3 didn't need that. You showed it a few examples, in plain text, and it picked up the pattern. Translation. Questions and answers. Unscrambling words. Three-digit arithmetic.

[14:36] Narrator: In one test, the authors gave GPT-3 a made-up word, a farduddle, which means to jump up and down really fast, and asked it to use another made-up word in a sentence. It did, without being trained to.

[14:51] Narrator: And there was a finding the authors flagged as a risk. They had GPT-3 write short news articles and asked about eighty people per test to tell them apart from articles written by humans. For the biggest model, people got it right about fifty-two percent of the time. A coin flip is fifty.

[15:12] Narrator: Two more details from that paper. Where it was trained: on a high-bandwidth cluster provided by Microsoft. And at the very end of the author list, Ilya Sutskever, and then, last of thirty-one names, Dario Amodei.

[15:31] Narrator: Two weeks later. Thursday, June eleventh. OpenAI announces something it had never had before. A product.

[15:41] Greg Brockman (post): We're releasing an API for accessing new AI models developed by OpenAI.

[15:47] Narrator: An API: a way for other companies' software to send text to OpenAI's model, and get text back. The post describes it the way you'd describe a vending machine.

[16:00] Greg Brockman (post): The API today provides a general-purpose text in, text out interface, allowing users to try it on virtually any English language task.

[16:10] Narrator: Behind the counter: models from the GPT-3 family. It launched as a private beta, with a waitlist. And the post says plainly what it's for.

[16:21] Greg Brockman (post): In addition to being a revenue source to help us cover costs in pursuit of our mission, the API has pushed us to sharpen our focus on general-purpose AI technology.

[16:32] Narrator: A revenue source. Eleven months earlier, the Microsoft post had said building a product "would mean changing our focus." Now the product is here, and the post says it sharpened their focus. The post took the question on directly. Why a commercial product? Their answer: "We see developing commercial products as one of the ways to make sure we have enough funding to succeed."

[16:59] Narrator: And why rent the model out instead of releasing it, the way GPT-2 finally was? The post gave three reasons. It pays for research. The models are expensive to run. And, third, renting is safer, because they could cut off customers who misused it. With an open model, you can't take it back.

[17:22] Narrator: That's the GPT-2 lesson, turned into a business model. Don't lock it up, and don't let it go. Keep your hand on the switch, and charge for access.

[17:35] Narrator: Remember what Dario Amodei told Karen Hao. A portfolio of bets. Many teams, many theories, and over time, the winners pull ahead, and the teams collapse into a single direction. In twenty-twenty, you can watch that happen in public, one blog post at a time.

[17:56] Narrator: April thirtieth. A post called Jukebox. A neural network that generates music, "including rudimentary singing," as raw audio, in different genres and artist styles. They trained it on one point two million songs gathered from the web. And this time, unlike GPT-2, they released the model and the code on day one.

[18:21] Narrator: It was honest about how far it had to go. The songs had no choruses that came back around. The audio was noisy. And it took about nine hours, the post said, to fully render one minute of music. They shared it with ten musicians, and the post reports that those musicians "did not find it immediately applicable to their creative process."

[18:44] Narrator: Then, June seventeenth, six days after the API. A post called Image GPT. The researchers took GPT-2, the language model, and fed it pictures instead of words. Each image unrolled into one long line of pixels, and the model's only job was to guess the next pixel.

[19:07] Narrator: The post's first sentence is the whole point. "Just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions."

[19:22] Narrator: The same exact model. Show it the top half of a photo, and it would paint a plausible bottom half. Nobody had told it what a cat was, or a tree. It had learned enough about the world in pictures from predicting one pixel at a time.

[19:40] Narrator: It wasn't efficient. The authors admitted it took far more computing than methods designed for images. One version trained for about twenty-five hundred GPU-days, they wrote, where a comparable method took about seventy. But that wasn't the point either. They were showing that you could, in their words, "trade off hand coded domain knowledge for compute."

[20:04] Narrator: Trade knowledge for compute. That is the bet, in six words. Don't teach the machine about language, or music, or pictures. Give it more data, and more computing, and the same simple job: guess what comes next.

[20:21] Narrator: Look at who signed it. Mark Chen, Alec Radford, Ilya Sutskever. And remember what Hao's sources told her in February. The next big secret project was a system trained on images, text, and other data. In the months ahead, the separate bets on words, sounds, and pictures would start to merge.

[20:45] Narrator: That summer, people with early API access started posting what GPT-3 could do. Code from a sentence. Fake poems by real poets. The demos spread fast, and the claims grew faster.

[21:02] Narrator: One of the most useful of those posts wasn't a demo. It was a test. On July sixth, a software engineer named Kevin Lacker published a blog post called "Giving GPT-3 a Turing Test." He asked it questions, plain question and answer, using the default prompt OpenAI suggested, and he posted what came back.

[21:25] Narrator: A lot of it was startling. How many eyes does a giraffe have? Two. Who was president of the United States in eighteen-oh-one? Thomas Jefferson. What city is in the northwest corner of Ohio? Toledo. Lacker wrote that ten years earlier, if he'd had that conversation, he'd have assumed there was a human on the other end.

[21:48] Narrator: Then he started asking stranger questions.

[21:52] Narrator: How many eyes does my foot have? "Your foot has two eyes." How many eyes does the sun have? "The sun has one eye." How do you sporgle a morgle? "You sporgle a morgle by using a sporgle."

[22:07] Narrator: And who was president of the United States in seventeen hundred? It answered: William Penn. Confidently. There was no United States in seventeen hundred.

[22:20] Narrator: Lacker's diagnosis is worth hearing in his words. GPT-3, he wrote, "doesn't quite know how to say 'Wait a moment… your question is nonsense.'" And, "It also doesn't know how to say 'I don't know.'"

[22:34] Narrator: File that away. A machine that always has an answer, and sounds just as sure when it's wrong. That is going to matter a great deal when millions of people start typing questions into one.

[22:49] Narrator: On July nineteenth, Sam Altman tried to turn the volume down. On Twitter, he wrote:

[22:56] Sam Altman (post): The GPT-3 hype is way too much. It's impressive, thanks for the nice compliments, but it still has serious weaknesses and sometimes makes very silly mistakes.

[23:07] Sam Altman (post): AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out.

[23:15] Narrator: Seven weeks later, on September eighth, the Guardian ran an opinion piece under a headline built to travel: "A robot wrote this entire article. Are you scared yet, human?" It opened: "I am not a human. I am a robot. A thinking robot."

[23:33] Narrator: Now read the note at the bottom. The Guardian wrote the instructions. A University of California, Berkeley undergraduate named Liam Porr fed them to GPT-3. It produced eight different essays. And the editors, quote, "chose instead to pick the best parts of each." They wrote: "We cut lines and paragraphs, and rearranged the order of them in some places."

[23:59] Narrator: So: a machine wrote the sentences. People chose which ones, and in what order. The headline said entire. The editor's note said otherwise. That gap, between what the model did and what people said it did, is going to follow this company for years.

[24:20] Narrator: Go back four days before that Guardian op-ed. Friday, September fourth, twenty-twenty. While the internet argues about whether GPT-3 can think, a small team at OpenAI posts something much less flashy. The title: "Learning to summarize with human feedback."

[24:42] Narrator: Lacker had found the problem. These models are trained to predict the next word of human writing. The post says it plainly: that goal "doesn't capture exactly what we want." A model trained to predict what a person would say "might make up facts when it is unsure."

[25:00] Narrator: So the team tried something else. The material was posts from Reddit, each with a short summary its author had written at the bottom, the kind people label T-L-D-R. Too long, didn't read.

[25:14] Narrator: Here's the method. First, the model writes two summaries of the same post. A person reads both and picks the better one. Do that tens of thousands of times. Then train a second model, a judge, to predict which summary a person would pick. And then let the first model practice against the judge, over and over, getting rewarded for summaries the judge likes.

[25:40] Narrator: Reinforcement learning from human feedback. Back in twenty-seventeen, researchers from OpenAI and DeepMind had used this trick to teach a simulated robot a backflip, just from people choosing between short video clips. And last episode, the safety team tried it on language, and hit the bug that flipped the sign. This is the same idea, grown up.

[26:04] Narrator: And it worked better than anyone might have guessed. A model of one point three billion parameters, trained with human feedback, beat a model of twelve billion trained the ordinary way. People even preferred its summaries to the ones the Reddit authors had written themselves.

[26:24] Narrator: Who were those people? The post tells you. About eighty contractors, hired through sites called Upwork, Scale, and Lionbridge, mostly American or Filipino. Paid by the hour, it says, at least fifteen dollars an hour, no matter how many summaries they judged. The team set up a chat room for them, and one-on-one video calls.

[26:47] Narrator: And the post was honest about a trap. The judge is only a stand-in for what people actually want. Push the model too hard to please it, and the summaries stop getting better. In the post's words: "Eventually optimizing the reward model actually makes things worse."

[27:07] Narrator: The post also drew a line around what the method can't do. It can teach a model a behavior once you know what you want. It "is not a method for determining what the desired model behavior should be." And for harder questions, it said, where people disagree, the people affected should be part of deciding what "good" means.

[27:28] Narrator: Look at the author list on this one. Paul Christiano, who led the safety work. And, last of seven, a researcher named Long Ouyang. In about a year and a half, his name will be first, on a paper that takes this exact recipe and applies it, not to summaries, but to a model that answers whatever you ask it.

[27:55] Narrator: Tuesday, September twenty-second, twenty-twenty. Microsoft's chief technology officer, Kevin Scott, posts on the company blog. Microsoft, he writes, is teaming up with OpenAI to "exclusively license GPT-3."

[28:13] Narrator: He also reminds readers where GPT-3 lives. That May, Microsoft had announced a supercomputer built for OpenAI, hosted on its Azure cloud. GPT-3, he writes, "is trained on Azure's AI supercomputer."

[28:31] Narrator: Microsoft's own feature on that machine, back in May, had given the numbers. "More than two hundred eighty-five thousand CPU cores, ten thousand GPUs and four hundred gigabits per second of network connectivity for each GPU server." One of the top five publicly disclosed supercomputers in the world, it said, "built in collaboration with and exclusively for OpenAI."

[28:59] Narrator: And it quoted Sam Altman on how it happened. OpenAI had asked itself, "If we could design our dream system, what would it look like?" Then, Altman said, "Microsoft was able to build it."

[29:14] Narrator: What did exclusive mean? Microsoft could build GPT-3 into its own products. And OpenAI, the post says, "will continue to offer GPT-3 and other powerful models via its own Azure-hosted API." So the public could still rent it, through OpenAI. But the underlying model was licensed to one company.

[29:38] Narrator: Go back to the bet from last episode. A billion dollars, an exclusive cloud, and a line about licensing "pre-AGI technologies" with Microsoft as the preferred partner. Fourteen months later, the first big pre-AGI technology had a name, and it had an exclusive licensee.

[29:58] Narrator: Years later, this is the arrangement Elon Musk's lawsuit would point to. His February twenty-twenty-four complaint alleged that OpenAI had become, in its words, "a closed-source de facto subsidiary of the largest technology company in the world: Microsoft." Those are allegations. OpenAI and Microsoft have disputed them, and none of it has been decided in this telling.

[30:28] Narrator: December twenty-twenty. OpenAI holds an employee meeting. And one of the two people who'd been setting the direction of its research tells his coworkers he's leaving.

[30:40] Narrator: His name has been at the end of the author lists all season. Dario Amodei. Vice president of research. He'd come to OpenAI from Google in twenty-sixteen. He'd helped build GPT-2 and GPT-3, and co-led the research with Ilya Sutskever.

[30:59] Narrator: OpenAI's post quotes what he told his coworkers that day.

[31:04] Narrator: "I want to thank Sam and thank everyone. I'm really proud of the work we've done together. I want to wish everyone the best, and I know that OpenAI will do really great things in the years ahead."

[31:16] Narrator: And he added a line that sounds like a farewell, and also like a warning. "We share the same goal of safe artificial general intelligence to benefit humanity, so it's incumbent on all of us in this space to work together to make sure things go well."

[31:35] Narrator: The announcement went up on December twenty-ninth, between the holidays, under the title "Organizational update from OpenAI." It says he's leaving after nearly five years. And it says he isn't leaving alone.

[31:52] Sam Altman (post): We wish him and his co-founders all the best in their new project, and we look forward to a collaborative relationship with them for years to come.

[32:01] Narrator: Co-founders. A new project. The post describes it in one sentence: a project that will "probably focus less on product development and more on research."

[32:12] Narrator: Less product. Six months after OpenAI launched its first one.

[32:19] Narrator: In the same post, OpenAI announced a promotion. Mira Murati, who'd led the API rollout, became senior vice president of Research, Product, and Partnerships. Research, and product, in one title.

[32:35] Narrator: The post doesn't say why Amodei left. Neither does his farewell. He's spoken about it since, and so have others, and the accounts don't all agree. That's next time.

[32:48] Narrator: So look down from the top of twenty-twenty. In January, a paper says bigger is predictably better. In February, a reporter is kept on the first floor and told the results were secret for six months. In May, the biggest model yet. In June, a price list. In September, a human-feedback recipe nobody outside the field noticed, and an exclusive license everybody did. And in December, the co-leader of the research leaves to start something that will, by his own colleagues' description, focus less on product.

[33:24] Narrator: Nobody in this episode broke a rule. Every step was announced, explained, and defended in a blog post. That's what makes an escalator so easy to stand on.

[33:35] Narrator: Next time, on The Role Models. The Quiet Build.