What Is Data Programming? Snorkel AI's Scaling Method Explained
Snorkel AI's data programming turns expert knowledge into labeling functions written in code, scaling supervision beyond manual annotation.
What is data programming?
Data programming is a method for turning expert judgment into code instead of one-off manual labels. Rather than paying an expert to tag training examples one at a time, you ask them to encode their reasoning as a labeling function, a small piece of software that captures a heuristic or rule they use to make a decision. Snorkel AI, which grew out of the Stanford AI Lab’s Hazy Research group, built its entire company around this idea, arguing that expert time, not raw data volume, is the real constraint on building good AI systems.
TL;DR
- Data programming encodes expert knowledge as labeling functions (small programs) instead of one-off manual annotations, so a single piece of expert judgment can be applied across an entire dataset.
- Snorkel AI traces its roots to the Stanford AI Lab, where its data programming research began roughly a decade before the company became a go-to data partner for frontier AI labs.
- The core problem the approach solves is that manual labeling doesn’t scale: it costs new cognitive effort for every single data point, and it produces no reusable reasoning behind each label.
- Manual labeling is also fragile to noise and schema change. Fixing noisy labels usually means redundant relabeling, and any change to task definitions forces teams to start over from scratch.
- Snorkel’s framing treats supervision as a provenance problem: without a record of why an expert made a call, there’s no way to audit, refactor, or update the resulting dataset as requirements shift.
- The company’s stated thesis is that scaling expertise, not scaling compute or architecture, is the actual bottleneck for building AI that matches or exceeds human performance in specialized domains like medicine, law, and finance.
- As AI evaluation has moved from simple labels toward full task environments with rubrics and verifiers, the underlying need, capturing expert judgment efficiently, has only gotten more acute.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
How does data programming actually work?
The starting point is a basic observation about manual labeling: it’s an O(n) problem. Every data point that needs a label requires a fresh unit of human cognitive effort. If you want a million labeled examples, you need something close to a million individual judgments, and each one costs time from a person qualified to make it.
Data programming changes the unit of work. Instead of asking an expert to label examples directly, you ask them to write or specify a labeling function, a rule, heuristic, or small model that reflects a piece of their reasoning. A cardiologist reviewing MRI scans might encode “if this region shows pattern X, flag it as abnormal” as a function rather than manually tagging thousands of scans one by one. Multiple labeling functions, often written by different experts or capturing different heuristics, get combined statistically to produce a final label, effectively voting and reconciling their outputs to reduce individual noise.
This matters for three practical reasons. First, it scales like software: once written, a labeling function can run over arbitrarily large datasets. Second, it’s adaptable: when a task definition or schema changes, you edit the function rather than redo the entire labeling effort from zero. Third, and arguably most important, it’s auditable. The logic behind a label exists as inspectable code, so teams can have an actual conversation about why the model made a call, rather than treating each label as an opaque one-off decision.
Why does manual labeling break down for expert domains?
Manual labeling has always been expensive, but the cost compounds sharply once you move into specialized fields. Generic crowdworkers can label whether an image contains a hot dog. They can’t reliably read an EHR record, assess legal risk in a contract, or judge whether a trading strategy’s reasoning is sound. Those tasks require MDs, PhDs, lawyers, or domain veterans, and there are only so many of them, with limited hours in a day.
Three specific failure modes show up in manual annotation at this level. Noise is one: any single label reflects one person’s judgment on one pass, so reducing error typically means adding redundant relabeling from multiple annotators, which multiplies cost. Fragility to change is another: if a task’s schema or definition shifts, for instance a new category needs to be added or a rule gets refined, teams often have to restart labeling rather than adjust existing work. The third, and the one Snorkel’s research emphasizes most, is the absence of provenance. A raw label doesn’t tell you the rationale behind it. There’s no record of what the annotator was thinking, which makes it hard to debug disagreements, update the labeling policy, or hand the work off to someone else.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
These problems don’t show up much in a controlled benchmark setting where the labeling task is simple and the dataset is static. They become severe once you’re trying to build data for something like an AI system meant to operate at the level of a domain expert, where correctness is subjective, ambiguous, or contested even among human specialists.
Why is expert supervision described as the bottleneck?
The broader argument, one Snorkel’s team makes explicitly, is that architecture and compute are no longer the limiting factor in building capable AI systems. The limiting factor is getting well-curated, expert-grade data and evaluation environments into the training pipeline. This applies directly to domains where two qualified professionals might reasonably disagree, such as two doctors reaching different conclusions from the same patient case, or two lawyers weighing a contract differently.
That kind of subjective, preference-style judgment can’t be captured with a simple right-or-wrong label. It requires structured comparisons: this response versus that one, this diagnosis path versus another, graded by someone with the relevant expertise. Building datasets like that at scale is exactly the problem data programming was designed to address, letting a limited pool of experts encode their judgment in a form that can be replicated across far more examples than they could personally review one by one.
How has the shape of this problem evolved?
Early labeling work, sometimes described as a first generation of data work, mostly involved simple judgments: thumbs up or thumbs down on a response, or basic prompt-and-response pairs, each requiring maybe half a minute of human attention to produce. That’s a relatively low bar for what counts as “supervision.”
The nature of the data being built for AI systems has since shifted toward much richer artifacts. Instead of a single label, teams now build entire task environments: structured worlds with rubrics, verifiers, and grading logic, often packaged so an AI agent can be tested or trained against realistic scenarios. Producing one of these environments can take a human expert anywhere from a few hours to well over a hundred hours, compared to the seconds required for an old-style label.
This raises the stakes on the scaling problem rather than resolving it. If a single label used to require 30 seconds of expert time, and a single evaluation environment now requires potentially hundreds of hours, the underlying question, how do you get more value out of every hour of expert time, becomes more urgent, not less. That’s the throughline in Snorkel’s approach: whether the artifact being built is a simple label or a full evaluation environment, the underlying constraint is always the availability and efficient use of expert judgment.
Frequently Asked Questions
What is a labeling function in data programming?
A labeling function is a small piece of code or logic, often written by a domain expert, that encodes a rule or heuristic for assigning a label. Instead of manually labeling each data point, the function applies that rule programmatically across a dataset, and outputs from multiple labeling functions can be combined to produce a final, denoised label.
How is data programming different from traditional manual labeling?
Manual labeling requires new human effort for every single data point, has no built-in mechanism for handling noise besides redundant relabeling, and produces no record of the reasoning behind a label. Data programming instead captures expert reasoning as reusable, auditable code that can scale across large datasets and be adjusted as task definitions change.
Where did Snorkel AI’s data programming approach come from?
The technique originated in research from the Stanford AI Lab’s Hazy Research group roughly a decade before Snorkel AI matured into a company working with frontier AI labs and large enterprises on data and evaluation problems.
Why does expert supervision matter more for domains like medicine or law?
In fields like medicine and law, correctness is often subjective or contested even among qualified experts. Two doctors can disagree on a diagnosis, and two lawyers can weigh a contract differently. Encoding this kind of judgment at scale requires structured comparisons and rubrics built by domain experts, which is a harder problem than simple right-or-wrong labeling.
Is data programming still relevant now that AI evaluation involves full task environments?
Yes. As datasets have evolved from simple labels toward complex environments with rubrics and verifiers, the core challenge, extracting expert judgment efficiently and at scale, has become more important, not less, since building these environments requires significantly more expert time per artifact.
