Last Thursday’s Princeton Public Lecture on AI hype began with brief talks based on our respective books:
- AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference. Arvind Narayanan & Sayash Kapoor. Princeton University Press, 2024.
- Artificial Intelligence: A Guide for Thinking Humans. Melanie Mitchell. Farrar, Straus and Giroux, 2019.
The meat of the event was a discussion between the two of us and with the audience. A lightly edited transcript follows.
Photo credit: Floriaan Tasche
AN: You gave the example of ChatGPT being unable to comply with the instruction to draw a room without an elephant in it. It even reassured you that there was no elephant in the image. For anyone wondering, that’s because the chatbot is separate from the image generator. So it literally can’t “see” what it produced. That’s how it’s architected.
I’m curious how easy those problems are going to be to solve. It’s something I think a lot about, especially when I debate with AI boosters who say it will revolutionize the economy. One of the biggest sticking points for me is that these systems can’t see outside the box they’re in. That limits their common sense.
[Note: we mention multimodal image generation a bit later; it is a new technique that doesn’t suffer from the elephant problem. But this discussion is about the lack of context, which is a broader issue.]
Here’s another example. Every time a new chatbot comes out, I try playing rock-paper-scissors with it. I’ll say, “Let’s see what happens if you go first.” If it picks “rock”, I’ll say “paper,” and it replies, “Oh, you win!” Then I say, “Let’s go again,” and whatever it picks, I’ll pick the option that beats it.
After this happens a few times, I ask it, “How do you think I won each time?” And it’ll say something like, “Oh, you must be really good at reading robot minds.” Almost no chatbot so far has understood that turn-taking in an online interface is different from simultaneous rock-paper-scissors in real life. It doesn’t recognize the basic elements of its own user interface.
Is that a big issue? Can it be easily solved? What are the implications?
MM: I think it’s a big issue. Someone once said that current AI systems lack a frontal cortex—the part of the brain responsible for what we call metacognition. That’s a term from psychology. It refers to awareness of one’s own cognition—whether what you’re saying is true or false, how confident you are in it, that kind of thing.
In addition to metacognition, these systems also lack episodic memory—memory of their own experiences. So each conversation is essentially a clean slate unless the system has some way to access a persistent record.
That interaction between memory and metacognition is essential to human intelligence and to avoiding these kinds of problems. It’s something current AI systems don’t have.
AN: That’s really interesting. One thing I’ve noticed in your work—and a big reason I’ve learned so much from you—is that you study the internals of AI and connect them to the internals of human cognition. You look for similarities and differences. It’s deeply satisfying to explore that from a curiosity-driven perspective.
I wonder, though: to what extent do we need to understand the internals to make practical decisions about AI? I’m not an expert on the internals; I study the behavior of AI—its societal and economic impacts. Are these two perspectives complementary? What’s the role of understanding internals when it comes to decision-making around AI?
MM: That’s a really important question. One strange thing about AI is that we built it—we trained it—but we don’t understand how it works. It’s so complex. Even the engineers at OpenAI who made ChatGPT don’t fully understand why it behaves the way it does.
It’s not unlike how we don’t fully understand ourselves. I can’t open up someone’s brain and figure out how they think—it’s just too complex.
When we study human intelligence, we use both psychology—controlled experiments that analyze behavior—and neuroscience, where we stick probes in the brain and try to understand what neurons or groups of neurons are doing.
I think the analogy applies to AI too: some people evaluate AI by looking at behavior, while others “stick probes” into neural networks to try to understand what’s going on internally. These are complementary approaches.
But there are problems with both. With the behavioral approach, we see that these systems pass things like the bar exam or the medical licensing exam—but what does that really tell us?
Unfortunately, passing those exams doesn’t mean the systems can do the other things we’d expect from a human who passed them. So just looking at behavior on tests or benchmarks isn’t always informative. That’s something people in the field have referred to as a crisis of evaluation.
I was actually going to ask you—what do we do about this crisis?
AN: I want to start by reinforcing the point you made. We can’t read much into the fact that a model passed the bar exam or the medical licensing exam. A lawyer’s job isn’t just answering bar exam questions all day.
And when lawyers use these tools for real, non-trivial tasks—as opposed to something like translating documents, which older AI has been able to do for a decade—they run into problems.
Many lawyers have actually been sanctioned by courts for submitting briefs filled with fake, non-existent citations that were generated by chatbots. They didn’t realize these aren’t search engines and don’t have access to a reliable repository of truth.
There used to be a steady stream of media articles about this, but I think the stories have slowed down—not because it stopped happening, but because it’s so commonplace that it’s no longer news.
So, what do we do about the crisis of evaluation? I have both a technical answer and a structural one.
Let me start with the structural point. Right now, the state of evaluation in AI is like the auto industry before independent safety testing. It’s as if car makers were the only ones evaluating their own products—for crash safety, environmental impact, and so on.
It’s like we have no EPA doing independent tests, no Consumer Reports crash testing.
MM: We used to have that situation with cars, right? Didn’t end well.
AN: Exactly. I think we need a robust, independent third-party evaluation system. We—and many others—have been trying to build that. So that’s one structural change that would help: changing how evaluations are done.
The second point is more technical. Historically, AI evaluation has been like a one-dimensional hill-climbing contest. You define a task—say, image classification—and try to keep improving accuracy. A great example is the ImageNet dataset, which came out of Princeton and Stanford. It’s often credited as the root of the modern deep learning revolution.
That approach worked really well for a long time. But it’s very clear that it’s not working anymore, because today’s AI systems aren’t narrowly focused on a single task. They do a broad range of things and need to be accountable to real-world users. So we need to go beyond accuracy and incorporate many other factors.
We have a project called The Science of AI Evaluation—and many others are working in this area too. I think we need to completely reboot our understanding of what it means to evaluate AI systems. I’m cautiously optimistic that we’re getting there.
MM: It’s absolutely essential to figure out, as Turing asked, what is the nature of their intelligence? How much can we trust them?
I wanted to ask you: a month or two ago, Bill Gates was on The Tonight Show and said that within a decade, AI will replace doctors and teachers, and that we won’t need humans for most things. What do you think about that?
AN: In your talk you gave historical examples of overconfidence from AI experts, which I loved. One point you made, which I thought was spot on, is: are these really the right people to be making predictions?
I’ve felt strongly about that. Often, when AI researchers predict that AI will take over some job, the basis for that prediction is an incredibly narrow and shallow understanding of what the job actually involves. A famous example is Geoff Hinton predicting back in 2016 that radiologists were about to be obsolete.
What I’ve found is that people in various professions have a much better understanding of the limits of AI in their domains than AI researchers do. The overconfidence often stems from this process: the researcher defines a one-dimensional benchmark that captures a tiny aspect of the job, sees that AI performance improves rapidly over time on that benchmark, projects it forward, and concludes that AI will surpass humans and take over the job in three years.
But they haven’t thought about the hundred other things involved in the job that are hard for AI but trivial for humans—because those things require context, common sense, and a deeper understanding.
MM: Yes, exactly. That’s what happened in the 1980s with expert systems. People thought those systems would replace doctors and teachers. “Knowledge engineers” would interview professionals—say, a doctor diagnosing cancer—and try to extract all their expertise, turning it into a big set of rules.
But it turns out it’s very hard to articulate all your knowledge. Much of it is tacit or unconscious—things you don’t even realize you know. And that knowledge is often crucial to doing the job well.
So I’m also skeptical of predictions from people like Bill Gates or Sam Altman. They often don’t appreciate the complexity of our own intelligence—the messy, context-dependent, embodied ways we actually think and work.
AN: That leads to something I’ve been thinking about a lot: the growing disconnect between the AI community—both researchers and industry—and the general public. And this disconnect matters.
If tennis racket manufacturers are out of touch with customers, it’s not a big deal. We just don’t buy their rackets. But AI is different—it touches every part of society. Whether the companies building it are in sync with public needs and values really matters.
This communication gap goes both ways. One side of it is that AI researchers and companies often ignore people who actually understand the limitations of AI in specific domains.
At the same time, companies also aren’t doing a good job communicating to the public what’s actually happening—what progress is real, what’s hype, what’s driving their trillion-dollar bets on data centers.
If that social contract isn’t clear, it breeds mistrust and backlash. And I worry that we’re headed in an unhealthy direction. Industry is amassing more and more power, while public understanding and trust seem to be declining.
MM: Yeah, I don’t know where it’s all going. But this feels like the kind of situation that led to past “AI winters.” There’s a huge amount of investment in AI right now, but not yet a huge payoff—at least in terms of companies like OpenAI making sustainable profits. Their products cost more to run than users pay for them.
So who knows what’s coming. But I do think AI is much broader than just computer scientists and engineers. And those are the people currently in charge of creating it, marketing it, and deciding how it should be used.
One example I heard: Microsoft integrated AI into Word and other Office tools—and the first thing most users tried to do was figure out how to turn it off. That tells you something. These companies aren’t necessarily in touch with what people want or need.
AN: Exactly. I want to reemphasize that AI should be much broader than just computer science. People from all areas of expertise have something to contribute—and should consider careers in AI.
Ironically, that’s even more true now because of AI itself. A lot of the coding and modeling tasks have been partially automated, making space for people with different skill sets to contribute.
I want to switch topics and ask you a question that keeps coming up: Are these AI systems actually thinking or reasoning? Or are they just simulating those processes? Does the distinction matter? And why is this such a polarized conversation? People seem to have very strong opinions on both sides.
MM: Terms like “thinking,” “reasoning,” “beliefs,” “goals,” and “consciousness”—these are mental terms we use for humans. But they’re not scientific terms. Marvin Minsky, the founder of the MIT AI Lab, called them pre-scientific concepts.
We can’t point to a part of the brain and say, “This is what thinking is,” or “Here’s where reasoning happens.” So these words are vague, and people can use them however they want.
For instance, there are new AI models called “reasoning models” that generate chains of thought—step-by-step explanations. Even calling that a “chain of thought” is anthropomorphizing. The language we use shapes how we perceive these systems.
Are they really thinking? I’m not sure the question is even meaningful. Philosophers have grappled with this for a long time. Can animals think? Are babies conscious? Do they have self-awareness? We still don’t have clear answers.
So in some ways, AI is pushing us to get better at defining these concepts. Terms like “artificial general intelligence” get thrown around, but different people mean different things when they say it. That’s part of why this debate is so polarized.
Some people argue that these machines don’t think at all—they don’t understand anything. One term that’s gotten attention is “stochastic parrots”—the idea that AI just recombines training data without comprehension.
Others worry that we’re too quick to ascribe human-like qualities to these systems. And some feel threatened by the idea that non-human entities might have cognitive processes that resemble ours. So the disagreement isn’t just technical—it’s also emotional and philosophical.
AN: In the meantime, are there different terms we should be using?
MM: That’s a great question. Probably, yes. There was a book I read—The Promise of Artificial Intelligence by a philosopher—where the author tried to do exactly that. Every time he used a mental term like “think” or “believe,” he put special symbols around it.
But it made the book really awkward to read. It’s just more convenient to say, “My computer is thinking about something.” We just have to be careful not to take ourselves too literally when we use those terms.
AN: Yeah, that’s hard to do in regular speech or writing for a non-specialist audience. We actually tried to avoid all anthropomorphic language in our book, but gave up on it—it ended up feeling like more trouble than it’s worth.
MM: And the way chatbots present themselves is very anthropomorphic. They use first-person pronouns—they’ll say, “I think…” or “I want…” They describe themselves using those terms.
And it’s well known that any entity that speaks fluent, natural language will cause people to project things like understanding onto it. That was already evident in the 1960s with the ELIZA chatbot. It was extremely simple, but people felt like it understood them.
So, to some extent, it’s our own tendency to project. But the harder question is: when does it cross the line into actually understanding or actually thinking? I don’t know.
AN: A lot of the human tendencies you just described—like the ELIZA effect—are understandable, especially when people first encounter these systems. And I agree that companies should be pressured to change how they present AI, knowing this is how people respond.
But is there also a role for changing how we educate children about these technologies—starting that process earlier?
For example, one of the reasons I use AI with my kids is not just because it’s fun and educational, but also to teach them about AI. I’ve been able to have real conversations with my young kids about whether ChatGPT is a person, and try to help them understand how it can do what it does without having feelings or being alive.
So if we changed the way kids interact with technology—guiding them rather than leaving them to form their own metaphors or understandings—maybe that would reduce some of these projection problems in the future.
Can we train ourselves out of the ELIZA effect, or is it just too deeply ingrained?
MM: Yeah, I’m not sure. I suspect it’s pretty deeply ingrained—we tend to anthropomorphize everything, even before AI. We do it with pets, with inanimate objects.
But I do think it’s really important to get kids thinking and talking about these systems early on. They’re going to grow up with them, in ways we can’t fully predict. So starting those conversations early—like you’re doing with your kids—is essential.
Audience Q&A
Q: My question is about the elephant example you gave. If the AI generates the image, why can’t it see the image?
MM: Great question. As Arvind mentioned, that’s because there are actually two different systems: one that generates the image, and another that describes it. And they don’t communicate with each other very well.
A lot of multimodal systems—those that combine vision and language—struggle to coordinate across modalities. That’s very different from how humans work. But improving that integration is a big research area right now.
AN: I think that may be changing—just in the last week or so, both Google and OpenAI have released products where the language model itself does image generation.
They’re slower than older systems, but from what I’ve read—haven’t tested them myself—they don’t seem to have the “elephant problem.” So that’s promising.
MM: Yeah, I have to update my talk every three days now.
Q: Could you talk about the environmental impacts of AI development?
AN: It’s an important and often confusing topic. What helped me understand it is realizing that two very different issues are being lumped together.
First, there’s the global issue: the carbon footprint of all these massive AI data centers—how much they contribute to climate change. That’s one angle.
The second issue is local: when a mega data center is built in a small community, what does it do to the power grid, the water supply, the local environment? Similarly, there’s the environmental impact of mining the materials used to make AI hardware—especially on the communities where the mining happens.
According to experts like Hannah Ritchie, the local impacts are actually the more serious concern right now.
Globally, the energy use of AI data centers is still a small fraction of total data center energy use, though that may change in the future.
MM: I’ll just add: compared to humans, AI systems need vastly more data and scale. ChatGPT, for example, was reportedly trained on 5,000 times more words than a 10-year-old child has encountered. That’s staggering. There’s something deeply un-human about the way these systems are trained.
If you need to revive something like the Three Mile Island nuclear power plant just to run your AI—as Microsoft has discussed—you’re probably doing something wrong.
Q: Melanie mentioned earlier the idea of projecting understanding. A lot of my friends are new teachers trying to help students navigate AI. Do you know of any work focused on helping young people understand projection—not necessarily the dangers, but how to be aware of it?
MM: I think parents talking to young children—as Arvind is doing—is one of the best ways to start.
Many kids grow up with smart speakers like Alexa and assume it’s another person. So it’s important to talk to them early about what technology is and isn’t.
There’s also a growing amount of work now on how to integrate AI into education—how to teach kids about it, how teachers can use it responsibly, and how students should think about it. It’s a big and active area.
Q: A question on evaluation—you mentioned the crisis around it. Where do you see academia’s role in helping industries deal with the pressures of evaluating AI systems, especially when evaluations don’t align with specific use cases?
AN: When a company releases an AI model, they typically evaluate its general capabilities. These are upstream evaluations—they’re broad and not tailored to specific use cases.
But when another company integrates that model into a product, they have a very specific use case and user base. So they need downstream evaluations—focused, context-specific testing.
The confusion often arises when people take upstream evaluations and read too much into them. But two companies with the same use case might get very different results depending on user behavior and query distribution.
So some of the responsibility for evaluation has to fall on the companies deploying the AI—or even the end users.
This is very different from traditional software. If your company’s never used Zoom, you can ask another company how it worked for them and get a good sense. But with AI, that doesn’t work. It’s stochastic, highly sensitive to context.
So our view is that academia should step in to fill the gap with something we’re calling midstream evaluation. That means evaluations that are more context-specific than upstream ones, but not as narrowly tailored as downstream ones.
They’re focused on sectors or product types rather than specific deployments. It’s a balance between generalizability and specificity—and I think that’s what’s missing right now.
Q: I’ve heard a lot about DeepSeek, and you mentioned efficiency. Have they given up anything to become so efficient? And do we believe they’re as efficient as they claim?
MM: I think they are. They’ve been fairly transparent about what they did—they came up with some clever ways to improve efficiency. They’re definitely more efficient, though I wouldn’t say drastically more efficient.
But in general, with any technology, the trend is always toward greater efficiency and smaller models. That’s what’s happening here too—people are figuring out how to make these systems work more efficiently over time.
One reason I think the Chinese researchers were able to achieve this is that they had constraints—export restrictions on GPUs and so on. Working under constraints pushed them to innovate in ways that made their systems more efficient.
There’s a broader lesson there: having too many resources can make you inefficient. Constraints can force creativity.
Q: Do you worry about dependency on AI systems? Or about how they could empower bad actors?
Second, this seems like the only industry I’ve seen where people are actively afraid of their own innovations. What do you make of that? We’ve seen AI insiders themselves asking for regulation.
AN: Overreliance is definitely a concern. We’ve seen this with autopilots—when a human pilot has to suddenly take control, there’s often a moment of confusion. We’re starting to see the same issue with self-driving cars.
There’s also a longer-term concern: de-skilling. If we rely too much on AI, will we lose the core skills that are necessary for real expertise? I think that’s a valid concern, and I expect that the burden will largely fall on individuals.
When I use AI in my own work, I try to be conscious of using it in ways that augment my abilities rather than replace them.
As for bad actors—yes, AI can be used by malicious people. But that’s been true of many technologies. One example: you’ll often see headlines like “AI can autonomously hack websites.” And that’s true, especially with large language models and agentic systems.
But if our worry is that AI will help hackers find software vulnerabilities better than humans—that battle was lost 10 or 20 years ago. Automated tools have long been used for that.
What’s surprising is that such tools have often helped defenders more than attackers. Developers now use them to find and fix bugs before releasing software.
So yes, AI is double-edged, but the real question is: does it change the balance between attackers and defenders? In many cases, the answer seems to be no—we’re still on stable footing.
As for why so many in the AI industry are calling for regulation: I think there are two reasons.
First, some of this regulation—whether intentionally or not—would protect incumbents from upstarts. That creates an incentive for established companies to support it.
Second, I think there’s a selection effect. Many people who go into AI genuinely believe it’s a world-changing technology—capable of creating either utopia or dystopia. I’m not saying they’re wrong, but it’s natural for people with strong convictions like that to worry about existential risks.
MM: I agree with all of that. Just to add a couple of points—there’s a paradox when it comes to overreliance: the better a system gets, the more we trust it.
If a self-driving car is terrible, you keep your hands on the wheel. But as it gets better, you start to trust it more—right up until the moment it makes a mistake and, say, slams on the brakes at a billboard (editing note: this refers to an example from the talk of a self-driving car confusing a picture of a stop sign on a billboard for an actual stop sign). So the better the system, the more vulnerable we can become when it fails.
As for the “AI doomers”—I think you’re right. The people building these systems are often the same people who believe they could lead to existential threats. That’s not surprising when they already believe these systems are or will become extremely powerful.
Q: Are there any areas where the two of you disagree, in terms of how AI will impact society or its future?
AN: Here’s one. You’re the expert on this, but I don’t quite share your intuition about the importance of embodiment.
MM: Yes, it’s an intuition—I don’t have a proof. The idea behind embodiment is that humans are bodies embedded in the world. We actively seek out data that’s relevant to us—we’re not passively fed information like GPT models that ingest the internet.
So the question is: can a system learn just as well from passively receiving language, or does it need to have a body—doing things in the world—to gain the kind of intelligence humans have?
There’s a big debate about this. It’s definitely surprising how much language-only models can learn. Even I’ve been surprised. But I still have the intuition that some things—especially learning efficiently—require embodiment. That is, being active in the world and able to interact with it.
AN: Just to clarify—I completely agree with you on active versus passive learning. What I was trying to tease apart is whether embodiment specifically is necessary, or whether you can still be “active” in a purely digital form.
MM: Right, that’s a good distinction. It may be that there are different ways to reach the same endpoint in terms of capabilities. But I would still predict that being embodied—whatever form that takes—is a far more efficient route.
And from psychology, we have a lot of evidence that embodied learning is key to how humans learn quickly and effectively.
Q: Arvind, you and Benedikt and Sayash have argued that the real bottlenecks for AI’s social and economic impact aren’t model development—they’re implementation and adaptation.
So, who’s working on that, and why isn’t more happening—especially at places like Princeton?
AN: There are lots of reasons, but one that we’ve written about is this—
When general-purpose models like ChatGPT came out, many developers (from what we can tell) convinced themselves these models were so close to artificial general intelligence that they no longer needed to build products.
The model was the product. You just put it out there, and users will figure out what they want to do with it.
That approach hasn’t worked. What we’ve seen is that users often misuse the tools—or don’t use them at all—unless they’re built into streamlined, specific products. That mindset is slowly shifting, and over the past year or so, there’s been a growing focus on product development, not just model development.
Q: What role does mathematics play in AI?
AN: Math plays many roles. At a foundational level, all of machine learning is built on mathematics. To really understand these systems, you need a strong grasp of linear algebra, calculus, probability, and statistics. All the theoretical work on how models behave or how to improve them is grounded in those areas.
But there’s another direction too: now AI is helping with mathematics. Some AI systems are trained specifically on math and are being used to assist mathematicians.
They help prove theorems, verify proofs, and even suggest new ideas. I think mathematics is going to be one of the fields where AI has a big impact in the near future.
Q: In my company, we use Copilot and Claude—and they’re great. My question is, as we rely on these tools more and more, how will it impact privacy expectations and behavior. Will it be the end of privacy?
AN: As it happens, in a past life, I studied the history of privacy norms and privacy law. It turns out we’ve been asking variants of this same question for the last 150 years whenever a new type of technology comes out.
When photography was invented, there was a huge concern: what does it mean to lose our privacy in public spaces? Would it lead to scandal, to the exposure of intimate lives in newspapers?
And yes, it did happen for a while. A lot of modern privacy law was a response to that.
Sometimes we respond with law, sometimes with social norms. When Google Glass came out in 2012, for example, there was a massive backlash. People didn’t want to be recorded all the time. That backlash essentially killed the product.
Now, we’re starting to see similar devices return—but this time, society has had time to adjust, and new norms are forming.
And in other cases, we respond with technical protections. A lot of today’s access control mechanisms in computing—stuff we take for granted—were born from earlier crises around data confidentiality.
So yes, AI will change privacy. But I’m optimistic. Through a mix of law, norms, and technology, we’ll adapt. That may sound simplistic, but history suggests that we’ll come through okay.
Q: Alison Gopnik has argued that “general intelligence” is the wrong frame for understanding AI. Instead, she says, it’s a social and cultural technology—more like a printing press or writing than like a person.
I’d love to hear your thoughts on that.
MM: I think that’s a great framing. There are lots of metaphors we use to understand AI systems, and this is one of the more useful ones.
Some people say AI is like a library rather than a person. Others say it’s like a corporation or a crowd of minds. Each metaphor has some truth and some limits, but they really matter—especially when they influence policy.
Take copyright lawsuits, for example. The New York Times is suing OpenAI for using their material without permission. OpenAI defends it by saying: “It’s like a person reading the newspaper and being inspired—it’s transformative use.”
But that metaphor only works if we view the model like a person. If we see it as a database or library, then it feels more like theft.
So yes, metaphors matter—not just for understanding, but for shaping legal and policy frameworks.
AN: Yes, Alison Gopnik has made the point that general intelligence is not a meaningful concept even in humans. I don’t have the expertise to have an opinion on that, but here’s a point I feel confident about: AGI is not a property of software. To ask whether an AI system is AGI is a category error.
At best, AGI is a joint property—of software and all the social, legal, technical, and institutional adaptations we build around it.
Software alone isn’t enough. To make AI work in the real world, we need to adapt industries, laws, infrastructure, norms. That’s the real long-term process—and it’s going to take decades.
AI was used to transcribe the audio of the event and to clean up the transcript. It was then reviewed and edited by AN and MM. This discussion was held on Thursday, March 27, 2025.
Leave a Reply