Data-Centric AI: Why Curated Data Now Beats Model Architecture
YC's Paper Club explains why expert data and RL environments, not model architecture, have become AI's biggest bottleneck to progress.
What is data-centric AI, and why does it matter now?
Data-centric AI is the idea that improving a model’s training data, its labels, its coverage of edge cases, its expert judgment, produces bigger performance gains than tweaking a model’s architecture. For most of the last decade, machine learning research put architecture first: new layers, new activation functions, new attention mechanisms. Once transformer-based architectures became good enough and widely available, the bottleneck moved. Now the limiting factor on how well an AI system performs is usually the quality, coverage, and expertise embedded in its data, not the design of the network itself.
TL;DR
- The bottleneck has flipped: researchers used to spend the bulk of their time on model architecture and a small fraction on data, but in production systems today that ratio has reversed almost entirely.
- Looking at your errors beats reading papers: the fastest way to improve a model that’s stuck at a mediocre score is to examine false positives and false negatives directly, not to swap activation functions or add layers.
- Data and RL environments are products, not files: building a genuinely useful dataset or reinforcement learning environment requires the same craftsmanship as building software, including edge-case handling, ontology decisions, and ongoing maintenance.
- Distribution shift is the real killer: models trained on a clean, curated dataset often break in production because real-world inputs don’t match the training distribution, and no amount of architecture tuning fixes that.
- Expert domains need expert data: to get AI systems that perform like competent doctors, lawyers, accountants, or traders, you need curated data and environments built by people who actually have that expertise, not generic web text.
- The market has already validated this thesis: data-focused companies were once dismissed by investors as having no defensible value, yet the category has generated well over $100 billion in market capitalization over the past decade.
- Preference data matters as much as verifiable rewards: subjective judgments, like whether code is maintainable or whether a medical judgment call was reasonable, require human preference labeling that can’t be reduced to a simple pass/fail check.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Why did the field assume data was a commodity?
In the mid-2010s, a common view in machine learning was that data had little standalone value. Public benchmark datasets like ImageNet were freely available, and the belief was that once you had access to a large labeled dataset, additional data or better curation added little. Investors reportedly shared this skepticism: proposals for data-focused startups were often assigned a terminal value near zero, on the theory that data was undifferentiated and abundant.
That assumption did not survive contact with production systems. Models trained on clean benchmark data frequently fell apart once deployed against real-world inputs that didn’t resemble the training distribution. The gap between benchmark performance and real-world reliability turned out to be a data problem, not an architecture problem, and companies that specialized in closing that gap have since built substantial businesses around it.
How does looking at data actually improve a model?
The practical workflow behind data-centric AI is straightforward, even if it’s labor intensive. Instead of treating a stuck accuracy score as a signal to change the model, practitioners are advised to inspect the actual errors: pull every false positive and false negative, group them into buckets by cause, and rank those buckets by how much they’re hurting performance.
A classic retail computer vision example illustrates this: a model built to detect whether a product is in stock or out of stock might struggle not because its architecture is weak, but because of specific, fixable data issues, like fog obscuring a camera’s view of a shelf, a person blocking the frame, or a stock clerk pushing products forward in a way that looks like an empty shelf but technically isn’t. None of those problems get solved by adding more layers to a neural network. They get solved by identifying the failure mode and either relabeling, reclassifying, or collecting more targeted data for that specific case.
This also explains why human-level benchmarking matters. Early efforts to measure human accuracy on ImageNet found that people struggle with the same fine-grained distinctions models do, such as telling apart visually similar dog breeds. That kind of analysis only comes from actually looking at the data, not from reading architecture papers.
Why do datasets and RL environments count as products?
A recurring theme in data-centric AI is that datasets and reinforcement learning environments are not static files you download once. They require ongoing design decisions, quality control, and maintenance, much like software products.
One coffee. One working app.
You bring the idea. Remy manages the project.
Building a genuinely comprehensive dataset means making judgment calls about ontology (does “product pushed to the front of the shelf” count as its own category, or does it get folded into “out of stock” even though that risks confusing the model?), handling ambiguous edge cases consistently, and revisiting those decisions as the underlying task evolves. RL environments raise the difficulty further: they require verifiable reward signals, realistic task simulation, and rubrics detailed enough to grade an agent’s behavior reliably, sometimes packaged into full software environments that can take from a few hours to hundreds of hours of expert time to build properly.
There’s also a durability problem. Even a company that has captured enormous volumes of real-world interaction data (screen recordings of people using a piece of software, for instance) can find that data obsolete overnight if the underlying interface changes. Having a lot of data doesn’t mean you’re done collecting data. The need is continuous, which undercuts the old idea that data has no lasting value once you’ve acquired a large enough pile of it.
Is expert supervision the real constraint on AI progress?
For domains where AI needs to match professional-level judgment, the honest answer is yes. Getting a model to behave like a competent doctor, lawyer, accountant, or carpenter requires data and RL environments built with input from people who actually have that expertise. Generic internet text doesn’t encode the nuanced judgment of a licensed professional, and there’s no way to fake that with better model design.
This is a harder problem than standard labeling because expert time is scarce and expensive. A cardiologist or a specialized attorney can’t realistically hand-label the volume of data a foundation model needs. Manual labeling approaches also don’t scale well: each new label requires fresh cognitive effort, results are noisy unless you introduce redundant relabeling, and if the task definition changes, previous labeling work often has to start over. One response to this constraint has been “data programming,” an approach where expert judgment is encoded into reusable software (labeling functions and heuristics) rather than captured one label at a time. That approach, associated with weak supervision research out of Stanford’s AI lab and commercialized by companies like Snorkel AI, treats supervision itself as something you can scale programmatically rather than purely through manual effort.
The same scarcity logic applies to preference data, the subjective judgments about quality, style, or usability that verifiable rewards can’t capture. Two doctors evaluating the same patient case may disagree, and two engineers may both call a piece of code “working” while disagreeing sharply on whether it’s maintainable. Capturing that kind of nuanced disagreement, at scale, is its own unsolved engineering problem.
What does this mean for how AI companies get built?
One useful framing treats expert data and RL environments the way you’d think about mobile apps. A phone manufacturer doesn’t try to build every possible app, like a flashlight, a grocery delivery service, or a ride-hailing platform, in house. Some functions are simple enough that the platform owner builds them directly. Others, like running a logistics network for grocery delivery, are complex enough that it makes far more sense for a specialized company to own that whole vertical.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
The same logic is emerging in AI. It probably doesn’t make sense for every foundation model lab to independently build deep, HIPAA-compliant expertise in medical data curation, complete with a network of physicians, parallel case trajectories, and preference judgments between different diagnostic approaches. It’s more efficient for a specialized company to build that once and serve it to multiple model providers, similar to how one grocery delivery company serves both major phone platforms rather than each platform building its own.
Frequently Asked Questions
What does “data-centric AI” mean?
It’s an approach to building AI systems that prioritizes improving the quality, coverage, and curation of training data and RL environments over changing model architecture, based on the observation that data quality is now the main lever for improving real-world performance.
Why did investors once undervalue data-focused startups?
Data was widely treated as an abundant, undifferentiated commodity, especially with free benchmark datasets like ImageNet publicly available, so many investors assumed there was little defensible value in building a company around data curation. That view has reversed as data-focused companies have generated large amounts of market value over the past decade.
What’s the difference between verifiable rewards and preference data in RL environments?
Verifiable rewards apply to tasks with a clear right answer that can be checked automatically, like whether code passes a test. Preference data covers subjective judgments, like which of two designs is more usable or which coding style is more maintainable, where two qualified experts might reasonably disagree.
Why do models trained on clean datasets fail in production?
Training data is often carefully curated and doesn’t reflect the messiness of real-world inputs. Once deployed, models encounter cases and distributions they never saw during training, and performance can degrade sharply even though the underlying architecture hasn’t changed.
What is weak supervision or data programming?
It’s a method for encoding expert judgment into reusable software (rules, heuristics, or labeling functions) rather than requiring experts to manually label every individual data point, allowing supervision to scale closer to the speed of software rather than one-at-a-time human labeling.
