Hermes Agent + ComfyUI: Auto-Generate an Illustrated Storyboard
How Hermes agent connects to ComfyUI to iteratively refine prompts and autonomously generate a full illustrated storyboard, hands off.
What is the Hermes agent’s ComfyUI skill?
Hermes agent is an AI agent that includes a built in skill for connecting directly to a local ComfyUI instance. Instead of a person manually typing a prompt, checking the output, and tweaking the wording, Hermes can drive that whole loop itself: send a prompt to ComfyUI, look at the resulting image, decide what to change, and resend an improved prompt. It runs from either a desktop app or a terminal (TUI), and because it talks to ComfyUI over your own machine, the entire pipeline (agent plus image model) can operate fully locally, with no cloud API calls for either the reasoning or the image generation.
TL;DR
- Hermes agent can operate ComfyUI on its own, sending prompts, evaluating the generated image, and revising the prompt without a human in the loop for each step.
- A creator demoed this using Qwen Image 2.1 as the underlying image model, running fully locally on a consumer GPU (a 4090 in the demo).
- The agent improved output iteratively: an early attempt at a poster style diagram had misspelled labels and nonsense text, but successive regenerations produced cleaner spelling and layout over multiple passes.
- Given a single instruction, “create a storybook and post it” at a given local address, Hermes independently wrote a nine-plate narrative, generated matching images for each plate, and published the result.
- The storyboard demo reframed a well known disaster-movie plot as a future space log narrated by a ship’s steward, and took about 30 minutes end to end.
- Prompting quality still matters: results ranged from very strong (a Great Barrier Reef dive scene, a K-pop street shoot) to weak (a Sphinx construction scene, ultra-high-resolution space shots), and the agent’s usefulness came from correcting for the human’s own weak prompting.
- The approach hints at a general pattern: use an agent to research prompting best practices and negative prompts, then let it iterate, rather than hand-tuning every prompt yourself.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does Hermes agent iterate on prompts automatically?
The core mechanic is a feedback loop. The agent sends an initial prompt to ComfyUI, which generates an image using the connected model (in this case Qwen Image 2.1). Hermes then reviews that output and decides how to adjust the prompt: fixing wording, adding detail, correcting structure, or addressing obvious defects. It resends the revised prompt and repeats.
In one demonstrated example, a poster style diagram (numbered steps, labels, callouts) started out rough: misnumbered sequences, misspelled words, and outright nonsense strings where text should have been. Across several iterations, the spelling and structure visibly improved, even though some artifacts persisted, like a numbering scheme that never quite lined up with the number of items shown. The person running the demo eventually stopped the loop manually because it was still trying to refine further and the process was taking a while. That detail matters: the agent doesn’t necessarily know when to stop on its own, so a human still sets the boundary on iteration count or time.
Before starting, the recommended approach is to have the agent first look up prompting guidance and known negative prompts for the specific image model it’s driving. Image models like Qwen have documented prompting patterns and common negative prompt lists that curb typical failure modes. Feeding the agent that context before it starts iterating produces meaningfully better first attempts and faster convergence.
Is Qwen Image 2.1 good for this kind of automated workflow?
Based on the demonstrated results, Qwen Image 2.1 handles a wide range of subjects reasonably well at typical resolutions. A 1024x1024 image generated in about 7.5 seconds on a 4090. Bumping to 2048x2048 took roughly 30 seconds, and a 2688x1536 image also landed around 30 seconds. A larger 3072x3072 spaceport scene took about 95 seconds. At the extreme end, a 4096x4096 space image took 213 seconds and showed clear texture artifacts and odd patterning, suggesting that resolution has a practical ceiling before quality degrades. The creator noted that a 3090 should roughly double these generation times compared to a 4090.
Quality varied a lot by subject. A canoe scene and two arctic fox images looked convincing, including fine details like snow debris. A Great Barrier Reef dive scene with a shark, rays, and sea turtles held up well, though the water color didn’t perfectly match expected depth-based light falloff. A K-pop street shoot scene handled crowd details, equipment, cables, and reflective surfaces convincingly. On the weaker side, a historical Sphinx construction scene mixed time periods incorrectly and left out the nose, and cat illustrations showed water and glass behaving in physically inconsistent ways (spilling in a direction that didn’t match the glass position). Fur rendering across multiple animal images was flagged as a recurring weak point.
The pattern that emerges: strong performance on outdoor scenes, wildlife-adjacent staging, and busy human environments; weaker performance on precise physical interactions, fine text/diagrams, and historically accurate detail, at least without heavy prompt engineering.
How does the automated storyboard workflow actually work?
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
The most notable demonstration wasn’t a single image, it was a full multi-image storybook generated with minimal human input. The instruction given to Hermes was essentially: act as director, producer, and executive; take a well known disaster movie premise with a twist; be creative; make something shareable; and post the finished storyboard to a specific local IP address once done.
Hermes then independently:
- Wrote a narrative framed as a transmission log from a future spacecraft, told from the perspective of the ship’s steward rather than a present-day setting.
- Broke the story into nine plates (a defined multi-image sequence) rather than generating a single unstructured batch of images.
- Generated matching illustrations for each story beat, including a mutiny scene and its aftermath, in a style consistent with a storyboard rather than a fully rendered scene.
- Iterated on individual plates that didn’t convey the intended story beat clearly, regenerating some of them before settling on a final version.
- Published the finished set to the specified address without further manual steps.
The whole process took about 30 minutes from instruction to finished, published storyboard. One side note worth flagging for anyone building similar workflows: image models frequently refuse to generate content involving violence or mutiny-style plotlines when asked directly. In this case, framing the request as “you are the creative director, make the call” appeared to change how the model approached the request, though this is anecdotal and not something to rely on as a guaranteed bypass.
Is this workflow worth setting up?
For anyone already running ComfyUI locally with a capable GPU, adding an agent layer like this changes the interaction model from “manually prompt and inspect” to “state a goal and review the output.” The value is clearest in two situations: iterative refinement of a single image where the operator isn’t confident in their own prompting skill, and multi-image sequential projects like storyboards, comics, or slide decks where manually prompting each frame individually would be tedious.
The tradeoffs are real. Generation times scale steeply with resolution, so an agent iterating dozens of times at high resolution could take a long time or consume significant compute. The agent also doesn’t inherently know when an image is “good enough,” so a person still needs to set limits or check in periodically. And baseline image quality issues (fur texture, physical consistency, fine text) aren’t solved by agentic iteration alone if the underlying image model has those weaknesses across the board.
Frequently Asked Questions
What hardware do you need to run Hermes agent with ComfyUI locally?
The demonstrated setup used a 4090 GPU for image generation, with generation times for a 2048x2048 image around 30 seconds. A 3090 would be expected to take roughly double that time. Exact requirements depend on the specific image model and resolution used.
Does Hermes agent replace manual prompt engineering entirely?
Not entirely. It automates the iteration loop (send prompt, review image, revise prompt) but still benefits from being pointed at model-specific prompting guides and negative prompt lists before it starts. A human also typically decides when to stop the iteration process.
Can Hermes agent generate multiple connected images, not just one?
Yes. In the storyboard demonstration, it generated a nine-plate illustrated narrative from a single high-level instruction, writing the story, breaking it into scenes, generating art for each, and publishing the result on its own.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
Which image model was used in this demonstration?
Qwen Image 2.1, run fully locally alongside the Hermes agent, with no cloud dependency for either the language reasoning or the image generation.
What kinds of images does this workflow struggle with?
Based on the demonstrated results, fine text and diagrams, precise physical interactions (like liquid or glass behavior), and historically specific detail were weaker areas. Outdoor scenes, wildlife, and busy human environments performed noticeably better.
