Where AI Actually Stands in 2026: DeepSeek, Reasoning Models and the Scaling Debate

State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI. Lex Fridman Podcast number 490 with Sebastian Raschka and Nathan Lambert. Originally 4 hours and 25 minutes.

DeepSeek spent roughly five million dollars training their model. The Allen Institute for AI spent about two million renting GPU clusters. Meanwhile, frontier labs are burning billions. That gap, between what open labs achieve for pocket change and what closed labs spend for marginal gains, might be the single most important number in artificial intelligence right now. And it raises a question nobody in Silicon Valley wants to answer: what if the billion-dollar moats are already gone?

This is a sprawling four-and-a-half-hour State of AI conversation from February 2026, and five things from this episode will reshape how you think about where artificial intelligence is actually heading. First, why the companies building AI might be building their own coffins. Second, why the models you use every day are getting dramatically smaller, not bigger. Third, why the person who figures out memory for AI agents will trigger the next revolution. Fourth, why coding as a profession is about to transform beyond recognition. And fifth, why the cynical case for AI might be more realistic than the hype.

The Five Million Dollar Earthquake

The DeepSeek moment of January 2025 didnt just surprise the AI community. It rewrote the economics of the entire field. Sebastian Raschka, author of Build a Large Language Model from Scratch, frames it bluntly. In 2026, no company has access to technology that no other company can replicate. Researchers change jobs constantly, knowledge diffuses across borders, and the idea that any single lab holds a permanent advantage is, in his words, simply not how the field works anymore.

Nathan Lambert, post-training lead at the Allen Institute for AI, adds the critical business context. The cost of training these models is actually low relative to the cost of serving them to hundreds of millions of users. Companies like OpenAI and Anthropic are primarily LLM service providers. If AI becomes commodified, and the evidence suggests it is, these companies could simply die. The models themselves are converging in quality. What matters now is distribution, products, and finding defensible niches.

Nathan goes further with a startling observation. OpenAI and Anthropic have all the same products, and when you talk to people inside these companies, they are solving a lot of the same problems. Google and xAI have other businesses to fall back on, but companies whose entire existence depends on selling LLM access face an existential question. There could be five or six companies competing in the API market, Nathan says, and he compares it to the cloud wars between AWS, Azure, and GCP. The difference is that cloud infrastructure took decades to commodify. LLMs might do it in years.

The China question looms large throughout the conversation. Sebastian argues that knowledge containment is impossible in 2026. You can make the same argument for computers, he says. You can say we dont want the public to have them. But look at Huawei making chips now. It took a few years, but it happened. Nathan agrees, calling any Manhattan Project approach for AI both impractical and unhelpful. There is no civilizational risk that justifies it, Lex adds. The AI race isnt about having secrets. Its about executing faster.

The Models Are Shrinking and Thats the Point

Heres something counterintuitive that most people miss entirely. The frontier models are actually getting smaller. Lex asks directly whether pre-training has hit a plateau. Nathans answer is nuanced but revealing. Models like GPT-4 were rumored to be around one trillion parameters at their largest, but theres strong evidence theyve gotten smaller as training has become more efficient. You want the model smaller because then your serving costs drop proportionately.

This creates an interesting dynamic. Claude 4.5 Sonnet ships before the bigger model because smaller models train faster. You can try more experiments, iterate more quickly, and get to market sooner, even though the bigger model is technically better. The economics of AI development are pushing toward efficiency, not raw scale.

Pre-training dataset sizes are measured in trillions of tokens. Sebastian explains the scale: smaller research models use five to ten trillion tokens, Qwen has documented going up to fifty trillion, and there are rumors that closed labs push to a hundred trillion tokens. But the actual training data is a small fraction of what gets collected. Labs like DeepSeek built their own optical character recognition systems to extract text from PDFs and digital documents across the web, unlocking trillions of tokens of candidate data. This OCR pipeline is something almost every major lab has built independently, a quiet infrastructure race that rarely makes headlines.

The conversation also tackles synthetic data, which many people still think of as problematic. Nathan points out that ChatGPT now gives wonderful answers, and you can train on those best answers. Early ChatGPT produced hallucination-heavy outputs that would poison training data. Current models produce responses good enough to train the next generation. The quality of synthetic data has crossed a critical threshold.

The Reinforcement Learning Revolution

Sebastian highlights Reinforcement Learning with Verifiable Rewards, or RLVR, as a breakthrough that is quietly reshaping how models learn to reason. The setup is deceptively simple. You give the model a math question and the correct answer, then let it figure out how to get there with minimal constraints. The beautiful thing is what happens in practice, Sebastian explains. The model develops step-by-step reasoning entirely on its own.

The DeepSeek R1 paper documented what they called an aha moment, where the model recognized its own mistake mid-reasoning and said, in effect, wait, I did something wrong, let me try again. Sebastian calls it genuinely cool that this emergent behavior falls out of such a simple training setup. Its analogous to a student with scratch paper working through a complex math problem, crossing things out, self-correcting, and arriving at the answer through iteration.

Nathan pushes back slightly, noting that models have seen the entire internet during pre-training. They have definitely seen people write things like wait let me reconsider. The aha moments might be partially learned behavior rather than emergent reasoning. But the practical result is undeniable. Inference time scaling, spending more compute at generation time through extended reasoning, has become the dominant paradigm shift of the past year.

Sebastian draws an important distinction between pre-training and post-training. Pre-training is soaking up knowledge. Reinforcement learning post-training is skill learning, unlocking the knowledge the model already has. With RL, you dont really teach it new knowledge, he explains. Its more like helping it figure out how to use what it already knows. Three papers in 2025 explored using RL during pre-training itself, but Sebastian notes these remain toy examples rather than production techniques.

Tool Use and the Coding Revolution Nobody Predicted

Both researchers agree that the single biggest surprise of the past year wasnt a new architecture or training technique. It was tool use. Nathan describes how models learned to use CLI commands, handle Git, search for information, and organize code repositories. If we were sitting in these chairs a year ago, he says, this is something that we didnt really think of the models doing.

The key insight is that tool use was surprisingly easy for models to learn through reinforcement learning. The model tries a tool, looks at what it gets back, tries another API, checks the result, and iterates until it solves the problem. Models pick up these skills very easily, Nathan says. This simple loop unlocked enormous practical value. But he adds a sobering caveat: its not clear what the next avenue will be. There are a lot of buzzy areas in AI, but nobody knows when the next step function will come.

Lex pushes them on coding specifically, and both reveal fascinating personal workflows. Nathan uses Claude Opus with extended thinking for code and philosophical discussion, ChatGPT for everyday queries, Gemini for long context needle-in-haystack tasks, and Grok 4 Heavy specifically for hardcore debugging that other models cant solve. Sebastian mirrors this multi-model approach with slightly different preferences. The consensus is telling: you use a model until it breaks, then you switch. And as Sebastian notes, its exactly like how we use browsers or text editors. Nobody types the same query into three different browsers to compare them.

The conversation turns to what full automation of coding might look like. Nathan frames it as a ratio: lines of useful code written per human in the loop. The superhuman coder scenario assumes that ratio goes to near zero humans. What does that world look like when the number of humans in the loop is in the hundreds, not in the hundreds of thousands? Nathan thinks software engineering will be driven more toward system design and goal-setting. Sebastian goes further, comparing it to calculators solving arithmetic. At some point LLMs will solve coding the way calculators solved calculating. The question is whether there will always be a human saying build that website, or whether AI will independently decide what needs building.

But Nathan raises a critical distinction. The problem with websites is that the web is resilient to slop. It will display garbage and the user will never know. He wants to think about safety-critical systems, logistics management, autonomous fleet operations, things where AI-generated code could have real consequences. Thats where the hard problems remain.

The Burnout Machine and the Culture of AI Labs

The conversation takes a surprisingly human turn when Lex asks about the work culture at frontier AI labs. Nathan is candid: people are working six days a week, twelve hours a day at many labs. He describes Anthropic as culturally deeply committed and organized, noting that everyone at Anthropic seems very aligned. This tight culture combined with intense competition creates extraordinary progress but at the cost of human capital.

Nathan wrote a post on burnout as hes tread in and out of it himself, especially while trying to be a manager and full-mode training simultaneously. He references the book Apple in China by Patrick McGee, where engineers working on supply chains had saving marriage programs, and some people literally died from the work intensity. The AI industry, he suggests, is creating a similar environment: a perfect machine for generating progress based on human sacrifice.

Sebastian offers a counterpoint from academia. Professors work a lot too, juggling teaching, grants, and research. But they are so fulfilled by mentoring students and having a constant mission that the workload feels different. In an era of chaos and rapid change, the stability and human connection of academic work is actually very rewarding. Its a quiet argument for why not everyone should chase the frontier lab dream.

Why NVIDIAs Real Moat Isnt Hardware

The conversation turns to NVIDIA, and Sebastians analysis cuts through the standard narrative. NVIDIAs moat isnt the GPU chip itself. Its CUDA, the software ecosystem thats been built over two decades. When Sebastian was a graduate student doing molecular dynamics simulations fifteen years ago, they were already using NVIDIA Tesla GPUs. That accumulated ecosystem of tools, libraries, developer knowledge, and institutional momentum is what makes NVIDIA nearly impossible to displace.

But heres the twist that could change everything. Sebastian thinks LLMs themselves might eventually replicate CUDA. It took fifteen years because it was genuinely hard, but now that we have AI that can write and understand code, the timeline for a competitor to build an equivalent software stack could compress dramatically. If someone designs fundamentally different hardware and uses AI to build the software layer, NVIDIAs twenty-year advantage could erode faster than anyone expects.

Nathan adds the scale argument. Even if someone builds better chips, the problem is adoption. When youre operating at the scale of major AI labs, why would you go with something risky where there are only a few chips available per year? You go with the proven option. NVIDIAs advantage is self-reinforcing: scale breeds reliability, reliability breeds adoption, adoption breeds scale.

The Memory Problem That Will Define the Next Era

Perhaps the most forward-looking part of the conversation centers on what they call continual learning, or more practically, how to give AI agents persistent memory. Nathan frames the problem clearly. Current models process everything in a single context window. When that window fills up, information gets lost. For agents that need to operate over hours, days, or weeks, this is a fundamental bottleneck that no amount of scaling has solved.

The solutions being explored are genuinely fascinating. DeepSeek V3.2 introduced sparse attention, where instead of attending to all tokens in the context, a lightweight indexer selects only the relevant ones. Sebastian explains how this connects to the original insight behind attention mechanisms: being selective about what information matters. Right now, brute force attention that processes everything gives the best results because you never miss information. But its wasteful. This year will be about figuring out how to be smarter about it.

Nathan describes a potential reinforcement learning approach where the model learns to compact its own history. The optimization problem becomes elegant: keep maximum evaluation scores while compressing context to minimum length. The minimum number of tokens needed for effective autoregressive prediction. This is fundamentally different from how current models work. Instead of plowing forward through tokens, the model would learn to actively manage its own memory, deciding what to keep and what to forget.

Sebastian connects this to how humans actually learn. Quantity is not always better because you have to be selective. Mid-training is about being selective with quality content, ensuring the last thing the model sees is the best material. The parallel to human memory is striking: we dont remember everything, we remember what matters. Teaching AI to do the same thing might be the key to making agents truly useful.

The Cynical Case Nobody Wants to Make

In one of the most thought-provoking segments, Lex asks Nathan to make the cynical case for AI. Is it possible that AI capabilities are plateauing in terms of what they actually mean for human civilization? Nathan takes the challenge seriously. On the coding front, really nice websites will be built. Very nice autocomplete. A nice way to understand codebases. But really just a very nice helper. It can help research mathematicians do some math. It can help with shopping. He pauses, then delivers the punchline: Its Clippy on steroids.

He continues listing what might be the real ceiling: computer use turns out extremely difficult to solve. Even if the models get better at narrow tasks, the cost of training and serving them at every level, both pre-training and inference, is enormous. Is the economic impact actually proportional to the investment?

Sebastian responds thoughtfully. There are so many obvious things to improve that it will take multiple years to saturate current capabilities. But he agrees with a nuance that Nathan calls a big statement: the dream of a general system thats useful to everybody is kind of dying. Specialized models for specific tasks might be more realistic than one model that does everything brilliantly. Multimodal is often treated as one thing, but video generation is a totally different problem from text reasoning, which is different from code generation.

Nathan adds that the frontier labs are still rushing to get the next model out, but the gains will be felt more through improving everything around the model. Better engineering of context, better inference scaling, better tool integration. The era of just put the better model in there is giving way to an era of engineering excellence around increasingly commodified model capabilities.

Open Source, Meta, and the Future of Llama

Nathan drops a provocation midway through: RIP Llama. Metas flagship open model project, which did more than perhaps anything else to democratize AI, appears to be losing organizational support. Meta is signing licensing deals with image generation companies like Black Forest Labs and Midjourney, suggesting a pivot away from training their own frontier models.

Sebastian is more charitable, noting that Meta still has excellent researchers motivated by proximity to Zuckerberg, and its too early to tell what their consumer AI strategy will be. But the broader point stands. Llama was the most focused expression of Metas AI ambitions, and that focus has waned.

The Allen Institute for AI, where Nathan works, represents a different vision for openness. Their OLMo models are fully open, including training data, code, and intermediate checkpoints. Sebastian emphasizes how valuable this is for researchers who want to understand what happens during training, not just use the final product. He has used OLMo checkpoints in his own research because you can examine how the model evolves during training, something impossible with closed releases that only share the final weights.

The future of open AI might not be one giant foundation model provider but rather an ecosystem of specialized, fully transparent research models that advance the science, alongside commercial models that advance the products.

How to Actually Learn AI in 2026

Lex asks what might be the most practically useful question of the episode: if youre a smart person interested in programming and AI, where do you start? Sebastian, who literally wrote the book on building LLMs from scratch, has a clear answer. Start by implementing a simple model from scratch that runs on your computer. The goal is not to build something you use every day. Its to see what exactly goes into an LLM, what comes out, and how pre-training works.

The key insight is that you can self-verify your implementation. Take an existing model from Hugging Face, load the same weights into your from-scratch implementation, and check that the outputs match. Sebastian compares this to RLVR: the verifiable reward is whether your model produces the same output as the reference implementation.

But Sebastian is honest about the limits. At some point, you reach a ceiling because small models can only do so much. Making a model larger isnt just about adding parameters. You have to shard across multiple GPUs, optimize the key-value cache, handle distributed training. Each optimization adds twenty or thirty lines of code. The gap between understanding an LLM and engineering a production LLM is enormous.

Nathan adds that the practical learning path has shifted. A year ago, understanding model architectures was paramount. Now, understanding how to work with models, how to design systems that use LLMs effectively, how to engineer prompts and tool integrations, might be more valuable for most people. The models are becoming commodities. The systems built around them are where the value lies.

Key Takeaways

First, the AI cost curve is collapsing. What cost billions two years ago now costs millions, and frontier performance is achievable by small teams with modest budgets. This trend is accelerating, and it threatens the business models of companies built entirely around selling model access.

Second, models are getting smaller and smarter, not just bigger. The economics of serving hundreds of millions of users push toward efficiency. Expect frontier models to continue shrinking while capabilities grow through better training techniques and inference-time compute.

Third, tool use and coding ability emerged almost by accident through reinforcement learning, and nobody predicted it a year ago. The next surprise could be equally unexpected and equally transformative.

Fourth, NVIDIAs real advantage is its twenty-year CUDA software ecosystem, not its chips. But AI itself might compress the timeline for competitors to replicate that ecosystem, creating the ironic possibility that NVIDIAs own customers build its eventual replacement.

Fifth, the race to give AI persistent memory is the most important unsolved problem in the field. Current context windows are a fundamental bottleneck for agents. Whoever cracks memory, the ability for AI to selectively remember and forget, changes everything.

Sixth, the cynical case for AI deserves more attention. Even the researchers closest to the frontier acknowledge that the dream of one general AI useful to everyone might be dying, replaced by specialized systems that excel in narrow domains. The question isnt whether AI will transform the world. Its whether the transformation justifies the hundreds of billions being invested.