How to Turn an AI Video Into a Playable Game With Claude Code
A walkthrough of using Claude Code and the GoKay shader skill to convert an AI video clip into a real-time, playable GLSL shader demo.
What is “video to GLSL” and how does it work?
Video to GLSL is a workflow where a coding-capable language model watches an AI-generated video clip and rewrites what it sees as a shader, a short program that draws pixels in real time using math instead of stored footage. The model breaks the clip down into its visual components (surfaces, lighting, motion, texture), builds a rough draft, compares that draft against the reference video, then iterates until the two look close. The output is a text file of GLSL code, sometimes just a few hundred lines, that runs live in a browser. Nothing is rendered from a 3D model or imported asset. Every frame is calculated from scratch.
The technique was packaged into an installable tool called Shader Skills by GoKay, built to work inside Claude Code. It gives Claude a repeatable method for this video-to-code conversion rather than requiring a user to prompt the whole process manually.
TL;DR
- Video to GLSL converts an AI-generated video clip into a shader, a program that recreates the look of the footage using real-time math instead of pre-rendered frames.
- The GoKay shader skill can be installed straight from GitHub by pasting the repo link into Claude Code and asking it to install the skill, and Claude Code can also offer to install the same skill for Codex.
- A simple scene (a rover grabbing an alien chip) took about 45 minutes to convert into a playable browser shader demo using Claude Opus 5.5, complete with movement, interaction prompts, and sound.
- The shader result is not a pixel-perfect match to the source video, but it captures lighting, ambient occlusion, and atmosphere closely enough to read as a faithful recreation.
- A more demanding workflow pushes the same method into Unreal Engine instead of a browser shader, using the official Unreal MCP (Model Context Protocol) to build geometry, materials, and lighting procedurally, with no imported assets, Blender models, or marketplace content.
- The Unreal version of a detailed pirate captain’s quarters scene took about two and a half hours and roughly $35 in estimated API cost, versus 45 minutes for the simpler shader demo, but produced noticeably higher fidelity, interactive objects, and even a dynamic storm mode.
- The two approaches serve different purposes: shaders are fast, cheap, and great for quick experimental demos, while the Unreal/procedural route is slower and pricier but suited to projects meant to be expanded into something more substantial.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Why would you turn an AI video into code at all?
AI video generators like Kling or similar text-to-video models don’t actually build a 3D scene. They produce a convincing sequence of pixels, what one might call a believable hallucination, without any underlying geometry, physics, or object permanence. You can’t walk around inside it or interact with anything in it.
Feeding that video through a language model that can write and test code closes part of that gap. The model infers what the geometry, materials, and lighting probably look like, then expresses that guess as a program. The result won’t be physically exact, but it’s “grounded” in the sense that it actually runs, responds to input, and can be extended. The tradeoff: you lose some of the uncanny visual polish of the original AI video, but gain something closer to a real, inspectable, playable artifact.
How do you install and run the GoKay shader skill in Claude Code?
The process starts with Claude Code and a model capable of sustained agentic coding (Claude Opus 5.5 was used in this workflow). Installation is as simple as copying the GitHub link to GoKay’s Shader Skills repository and pasting it into a prompt asking Claude to install it. Claude Code handles the rest, including recognizing other coding tools on the same machine. In this case, it also detected an existing Codex installation and offered to install the same skill there, which it did without extra manual setup.
Once installed, the user supplies a reference video. In the demonstrated run, a short AI-generated clip was created showing a small rover grabbing an alien ship component. Claude Code then spun up a sub-agent to execute the skill: analyzing the clip, drafting a first version of the shader, comparing it frame-by-frame to the source, and refining the weakest parts.
What did the first shader conversion actually produce?
The rover demo took about 45 minutes to complete. Running it required nothing more than launching a batch file, which opened the result in a browser. The output was not just a static replica of the video, it was playable: the rover could be driven around, a sound effect played on interaction, and pressing a key let the user pick up the chip and dock it elsewhere in the scene, triggering a view of a planet in the background.
Visual details held up well. Ambient occlusion gave the walls and interior a sense of depth rather than flat shading, and the lighting in particular stood out as effective. A separate “cinematic” version, built to match the original reference as closely as possible rather than prioritize interactivity, was also generated for direct comparison. It was a solid but imperfect match, good enough to be called “very serviceable” rather than a flawless recreation. The underlying shader code for comparable scenes can be remarkably compact, in the range of a few hundred lines total.
Is the shader method good enough, or does it hit limits?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
The shader approach has a structural limitation: it only draws the picture. There’s no deeper simulation of physical rules, memory, or complex interaction beyond what’s explicitly scripted. That’s fine for a simple scene like a rover grabbing one object, but it raises the question of whether this is really a “game” or a very convincing moving image with a couple of interactive triggers layered on top.
That limitation is what prompted a second experiment: applying the same underlying method, not to a browser shader, but to a full game engine (Unreal Engine), using Unreal’s official MCP integration to let Claude build procedurally inside it.
How does the Unreal Engine version differ from the shader approach?
For a more ambitious test, a moody pirate captain’s quarters scene (visually somewhere between Harry Potter and a pirate ship interior, with candlelight, aged maps, and ocean visible through windows) was chosen as a harder target. Instead of generating a flat shader, Claude Code used a “video to Unreal” skill built on the same core method as GoKay’s shader skill: break down the clip, build a rough pass, compare it to the reference at matching moments, and fix the biggest gaps.
The key constraint was that everything had to be generated procedurally inside Unreal Engine 5.8 through the official Unreal MCP. No Blender imports, no Fab or Megascans marketplace assets, no starter content. Every piece of geometry, every material, and every light had to be built from scratch by the model.
This took roughly two and a half hours and an estimated $35 in API cost, assuming standard pricing, a significant jump from the shader workflow’s 45 minutes. The payoff was fidelity. Fine details read clearly: a brass armillary sphere that could be spun, a leather-bound atlas and captain’s log that could be picked up, a bookshelf with varied book designs and brass backing, wood textures and carpet tassels, and a working day/night and storm cycle with rain streaking the windows, rocking lanterns, and thunder and lightning. One notable miss: a pair of spectacles was modeled with the metal bridge piece rotated the wrong way, a small but telling reminder that these systems don’t always self-check obvious physical logic.
The scene didn’t attempt to build an entire ship, focusing instead on just the one richly detailed room. Scaling this kind of procedural build to something as large as a full ship would likely take substantially longer and cost considerably more.
Which approach should you actually use?
The shader/GLSL pipeline is the better fit for fast, cheap experimentation: quick mini-demos based on AI video generations that run in a browser at a high visual quality relative to the time invested. It’s not meant to be the foundation of a long-term project.
The Unreal/procedural route costs more in time and money but produces results suited to projects a developer actually intends to build on. For solo developers or small teams who want a fast way to block out an atmospheric scene with procedurally generated, interactive assets, the tradeoff of a few hours and tens of dollars for this level of detail is arguably a good deal already, even accounting for the rough edges.
Frequently Asked Questions
What is the GoKay shader skill?
It’s an installable skill for Claude Code (and compatible tools like Codex) that gives a language model a structured method for converting an AI-generated video clip into a running GLSL shader, by breaking the clip into visual components, drafting code, and iteratively comparing the output to the source video.
Do I need to know GLSL or shader programming to try this?
No. The workflow is designed to be driven through natural-language prompts to Claude Code, which installs the skill and writes the shader code itself. Familiarity with coding helps if you want to debug or extend the output, but it isn’t required to run the basic pipeline.
How much does it cost to convert a video into a shader or Unreal scene?
Costs vary a lot by complexity. A simple interactive shader scene took about 45 minutes to generate. A detailed, fully procedural Unreal Engine scene took around two and a half hours and an estimated $35 in API costs under standard pricing. Larger or more complex scenes would take longer and cost more.
Is the result an exact copy of the original AI video?
No. Both the shader and Unreal versions are close approximations, not pixel-perfect reconstructions. They capture lighting, mood, textures, and major geometry convincingly, but fine details can be imperfect or occasionally illogical, since the model is inferring structure that was never explicitly present in the source video.
Which method is better for building an actual game?
The Unreal Engine approach, since it produces assets inside a real game engine with physics, lighting systems, and object interaction already built in. The shader approach is faster and cheaper but is better suited to short demos and experiments rather than a project meant to be expanded over time.
