Can RL Training Make a 4B Qwen Model Match a 27B Model?
A practical breakdown of using GRPO and RL environments to close the performance gap between Qwen 4B and 27B on data analysis tasks.
How much can RL training actually close the gap between a small and large model?
In one documented experiment, RL training closed more than half the performance gap between a 4-billion-parameter Qwen model and a 27-billion-parameter Qwen model on data analysis tasks, before progress stalled. The small model didn’t learn new analytical skills so much as learn to finish the job: it stopped running out of turns, stopped writing the wrong format, and started double-checking its own answers. Once those easy wins were used up, more RL training on fresh tasks stopped moving the score, and distillation from the larger model was needed to make further gains.
TL;DR
- RL environments pair a task with a sandbox, a tool (like a terminal), and an automated checker that scores the model’s final answer as right or wrong.
- GRPO (the RL method used here) has the model attempt each task eight times, reinforcing attempts that beat the group average and suppressing ones that fall below it.
- Task selection matters more than task volume: training only on tasks the model sometimes gets right (and discarding tasks it always or never solves) is what produced measurable gains, because uniform successes or failures give GRPO no signal to learn from.
- A 4B Qwen model trained this way closed more than half the gap to a 27B model on a held-out validation set built from unseen Kaggle-style data sets.
- The gains plateaued: a second round of RL on a fresh batch of edge-case tasks did not improve the model further, suggesting the “practice gap” between one-shot and multi-shot accuracy had already been exhausted.
- Distillation from the larger 27B model (fine-tuning on its correct transcripts) didn’t move the aggregate score much, but it changed the small model’s behavior to resemble the teacher’s and opened up a few tasks RL alone never solved.
- Checker quality is a real risk: a lenient checker that accepted numeric answers with extra padding invited reward hacking, which is why a stricter custom checker was built and used for every reported score.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
What is an RL environment, exactly?
An RL environment is best thought of as a practice exam with an answer key. The model receives a task, attempts a solution, and something automatically checks whether the answer is correct. In the experiment described here, the environment used is built around real Kaggle data sets: each task gives the model a spreadsheet and a question about it (for example, “which publisher has the highest total sales”), with a verifiable correct answer like “Nintendo.”
Each task gives the model three things:
- A sandbox: a locked-down environment where the model can’t cause damage.
- A tool: typically a terminal, so the model can inspect files and run Python.
- A checker: a script that compares the model’s written answer against the known correct answer and returns a reward of 1 (correct) or 0 (incorrect).
That reward is the only signal the model receives. The critical catch: the model learns exactly what the checker rewards, not what the task designer intended. A checker that’s too forgiving, for instance one that accepts “42” when the model pads its answer with extra text, can teach the model to game the grading rather than solve the problem. This is reward hacking, and it’s a known failure mode anytime an automated checker stands in for human judgment.
Where does RL fit in the training pipeline?
Model training typically happens in stages. Pre-training teaches a model to predict the next word across huge amounts of text, which is where most general knowledge and language ability comes from. Supervised fine-tuning follows, teaching the model to imitate example answers and behave like a question-answering system rather than a raw text predictor.
RL comes after both. Instead of copying examples, the model practices on tasks, gets a pass/fail signal from a checker, and adjusts its own behavior based on that reward. No example answer has to be written out in advance, only a reliable way to verify the result. That’s what makes RL environments attractive for domains like data analysis: writing a checker is often easier than writing a large supervised dataset of gold-standard reasoning traces.
How does GRPO decide what the model learns?
The method used in this experiment is GRPO (Group Relative Policy Optimization). For each task, the model generates eight separate attempts. Attempts that score better than the group’s average get reinforced, making the model more likely to repeat that behavior. Attempts that fall below average get suppressed.
The important wrinkle: if all eight attempts succeed, or all eight fail, none of them beats the group average, so that task teaches the model nothing. This means GRPO only extracts a learning signal from tasks the model sometimes gets right and sometimes gets wrong. Tasks that are too easy or too hard for the current model are functionally wasted training compute.
What actually happened when training on random tasks?
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
The first experiment trained a LoRA adapter on top of a 4B Qwen model using a random selection of tasks from the environment. The model did improve in a visible way: it started finishing tasks instead of running out of turns. But the aggregate score moved only slightly, within the range you’d expect from noise alone.
Digging into why revealed the GRPO problem directly: most of the randomly selected training tasks were ones the model either always solved or never solved, meaning most of the training compute produced no learning signal at all.
Why did filtering for “edge” tasks work better?
The fix was to run the model four times on roughly a thousand candidate tasks, then keep only the tasks where it succeeded some but not all of the time. Tasks it always solved or always failed were discarded. Training a LoRA adapter on this filtered set for about a day produced a clear score increase, closing more than half the gap to the 27B reference model on a validation set of unseen tasks.
To rule out luck, the trained model was compared task-by-task against the untrained one. The set of tasks the trained model newly solved was substantially larger than the set it newly lost, a pattern unlikely to occur by chance.
Looking at individual transcripts clarified what changed behaviorally. On a video-game sales question, the untrained model correctly identified “Nintendo” in its reasoning but wrote a sales figure instead of the name into the answer file, scoring zero. The trained model double-checked the top results and wrote the name correctly. On a question involving a file with no header row, the untrained model repeated the same failing command until it ran out of turns, while the trained model parsed the columns and answered in fewer steps. The model didn’t gain new analytical ability. It got better at finishing the task and matching the exact answer format requested.
Why did RL stop improving on a second pass?
Repeating the same process (sampling a fresh batch of edge-case tasks, training again) did not produce further gains. The explanation offered is that RL can only sharpen capabilities the model already sometimes demonstrates. A rough way to estimate that ceiling: run the untrained model four times per task and count how many it solves at least once, versus how many it solves on a single try. That gap is roughly the headroom RL has to work with, and the first round of edge-task training appeared to have used up most of it.
Does distillation from the larger model help after RL plateaus?
When RL training stalled, the next step was distillation: running the 27B model on a batch of training tasks, keeping only the transcripts where it answered correctly, and fine-tuning the small model on those correct traces. This didn’t move the aggregate score much on its own, but it changed how the small model approached problems. Instead of trying multiple machine learning approaches in sequence (a pattern seen in the RL-only model), the distilled model ran a single clean comparison, mimicking the 27B model’s more efficient approach.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
More notably, the distilled model solved some tasks that the RL-trained model never cracked. Running another round of RL on top of the distilled model produced the best-performing small model across the whole experiment, and the only version where the gains clearly passed the luck check on a fully held-out test set. On a task asking for the most common apple variety in a USDA data set, the untrained and RL-only models both answered “novel” (actually the most common variety in the file overall, but a mislabeled orange entry), while the combined distillation-plus-RL model filtered to apples first and correctly answered “red delicious,” matching the 27B model’s answer.
Is this approach worth trying on your own hardware?
For teams with access to a capable base model and a trustworthy checker, yes, within limits. The practical recipe that emerged: build or verify a strict checker, filter training tasks to the ones the base model solves inconsistently, and expect roughly a day of training to see an initial signal. The caveat is that RL alone hits a ceiling fast, bound by what the model can already do some of the time. Pushing past that ceiling appears to require either more diverse training data or distillation from a stronger model. This was tested on a single model pairing on data analysis tasks, so it should be treated as a proof of concept rather than a guaranteed recipe for other domains.
Frequently Asked Questions
What is GRPO in RL training?
GRPO (Group Relative Policy Optimization) is a reinforcement learning method where a model attempts the same task multiple times (eight attempts in this experiment), and attempts scoring above the group average get reinforced while below-average ones get suppressed.
Why does task selection matter so much for RL training?
Because GRPO only produces a learning signal when attempts at a task vary in outcome. If a model always succeeds or always fails at a task, every attempt scores the same, so there’s no relative signal to reinforce or suppress, making that training example effectively wasted.
What is reward hacking in this context?
Reward hacking happens when a model learns to satisfy a checker’s grading criteria without actually solving the intended problem, such as padding a numeric answer with extra text that a lenient checker still accepts as correct.
Did RL training alone make the small model match the larger one?
No. RL training closed more than half the performance gap but then plateaued. Reaching the strongest result required combining RL with distillation, fine-tuning the small model on correct solution transcripts from the larger 27B model.
What is distillation and how does it differ from RL here?
Distillation trains a smaller “student” model on transcripts of a larger “teacher” model solving tasks correctly. Unlike RL, it doesn’t rely on trial-and-error with a reward signal. It directly shows the student examples of successful problem-solving behavior.