How to Use Claude Code Computer Use with Dispatch for Remote Desktop Automation

The Remote Desktop Gap That Claude Code Computer Use Fills

Most automation tools work well when there’s an API to call. But a lot of software doesn’t have one — legacy enterprise apps, local tools, proprietary systems, desktop-only workflows. For those, you’re usually stuck doing it manually or building brittle workarounds.

Claude Code Computer Use changes the equation. It gives Claude the ability to see your screen, click buttons, type text, and interact with any desktop application — no API needed. And when you pair it with Dispatch, you can trigger those automations remotely from your phone, turning your desktop into a headless automation engine you control from anywhere.

This guide walks through exactly how to set up Claude Code Computer Use with Dispatch for remote desktop automation: what each piece does, how they connect, and how to build your first real workflow.

How Claude Code Computer Use Actually Works

Claude Code is Anthropic’s terminal-based AI agent. Unlike a chat interface, it can read and write files, execute shell commands, navigate codebases, and take actions in your local environment autonomously.

The computer use capability extends that further: Claude can now see and control your graphical desktop.

The Screenshot-Reason-Act Loop

At the technical level, computer use runs through a continuous cycle:

1. Capture — Claude takes a screenshot of the current desktop state
2. Reason — It analyzes what’s on screen and decides what action to take next
3. Act — It executes that action: move the mouse, click a button, scroll, or type
4. Repeat — It captures another screenshot to verify the result and continues

This loop runs autonomously until the task completes or Claude encounters something it can’t resolve.

What It Can Automate

Because Claude operates at the visual layer, it can interact with anything rendered on a display:

• Filling forms in desktop applications that have no web version
• Navigating menus and UI elements in legacy or proprietary software
• Transferring data between applications that don’t share an integration
• Running local tools that require hands-on GUI interaction
• Testing desktop applications through realistic user flows

The main constraint is cost and speed — each screenshot is processed as an image by the Claude API, so computer use is best suited for tasks where the effort saved justifies the token cost.

What Dispatch Adds to the Picture

Claude Code Computer Use, by itself, requires you to be at the terminal on your desktop machine. That’s fine for scheduled or pre-configured tasks, but it’s limiting when you want to kick off automation on the fly from your phone or another device.

Dispatch solves this. It’s a coordination layer that sits between your remote device and Claude Code running locally on your desktop — routing commands in, returning results out.

The Core Problem It Solves

Think of Dispatch as the call-and-response layer for remote Claude automation:

1. You send a command from your phone: “File this invoice in QuickBooks under Travel, dated today.”
2. Dispatch receives it and routes it to Claude Code on your desktop
3. Claude Code executes the task using computer use
4. Dispatch delivers confirmation back to you

Without this layer, you’d need to SSH into your machine or manually queue tasks — which removes most of the value of automation.

Why Not Just Use a Remote Desktop App?

Remote desktop tools like TeamViewer or RDP let you control your desktop remotely. You still do all the clicking yourself. Dispatch plus Claude Code Computer Use lets Claude handle the work — you just describe what needs to happen.

Prerequisites Before You Start

Before setting this up, make sure you have everything in place.

On your desktop machine:

• Claude Code installed (requires Node.js 18+ and an Anthropic API key or Claude Pro/Max subscription)
• macOS, Linux, or Windows (macOS and Linux are best supported for computer use; Windows works with additional setup)
• Screen access permissions granted to the terminal process running Claude Code
• Dispatch agent installed and running locally

On your phone or remote device:

• Dispatch mobile client or web interface configured to connect to your desktop
• A stable network connection — same local network, or a secure tunnel like Tailscale for remote access outside your network

Access requirements:

• An active Anthropic API key with computer use access, or a Claude Max subscription
• Enough API quota for image-heavy tasks — computer use consumes more tokens than text-only interactions

Set Up Claude Code with Computer Use

Step 1: Install Claude Code

Install via npm:

npm install -g @anthropic-ai/claude-code

Verify the installation:

claude --version

Step 2: Authenticate

Run the interactive setup to connect your Anthropic account:

claude

Follow the prompts to log in or enter your API key. Once authenticated, Claude Code is ready.

Step 3: Enable Computer Use Mode

Computer use is available on models like claude-3-5-sonnet-20241022 and later releases with visual desktop support. When you run Claude Code and assign a task that requires screen interaction, it will use the computer use capability automatically if your model supports it.

Test that it’s working with a simple prompt:

Take a screenshot of my desktop and describe what you see.

If Claude describes what’s on screen, computer use is active.

Step 4: Grant Screen Permissions

On macOS:

1. Open System Settings → Privacy & Security → Accessibility
2. Add your terminal app (Terminal, iTerm2, Warp, etc.)
3. Navigate to Screen Recording and add the same app

On Linux: Ensure your user has access to the display. The DISPLAY environment variable should be set (echo $DISPLAY), and tools like xdotool should be available for input simulation.

Without these permissions, Claude won’t be able to capture screenshots or send input events.

Configure Dispatch for Remote Triggering

With Claude Code and computer use confirmed working, the next step is setting up Dispatch so your phone can trigger automation remotely.

Step 1: Install the Dispatch Agent

Install Dispatch and initialize it in your working directory:

npm install -g dispatch-agent
dispatch init

During initialization, Dispatch will:

• Start a local endpoint to receive incoming commands
• Generate authentication tokens for pairing your mobile device
• Create a dispatch.config.json file you can customize

Step 2: Pair Your Mobile Device

Open the Dispatch mobile app or web interface on your phone. Enter:

• Your desktop machine’s local IP address (find it with ifconfig or ipconfig)
• The auth token generated by dispatch init

If you’re accessing from outside your home or office network, activate a tunnel first. Tailscale is a reliable option that handles NAT traversal cleanly.

Step 3: Configure Command Routing

Open dispatch.config.json and define how incoming commands route to Claude Code:

{
  "routes": [
    {
      "trigger": "natural_language",
      "agent": "claude-code",
      "mode": "computer-use",
      "confirmBeforeExecute": true,
      "timeoutSeconds": 120
    }
  ]
}

Keep confirmBeforeExecute set to true while you’re testing. This prompts you to approve what Claude plans to do before it acts on your desktop — useful for catching misinterpretations early.

Step 4: Test the Full Connection

From your phone in Dispatch, send:

Take a screenshot of my desktop and send it back.

Dispatch should route this to Claude Code, which captures the screen via computer use and returns the image to your phone. If that works end-to-end, you’re set up and ready to build real workflows.

Build a Real Automation Workflow

Here’s what a practical use case looks like from start to finish.

Example: Logging Expenses from Your Phone

You’re out of the office and receive a receipt. You need to log it in a desktop accounting app that has no mobile version or API.

The full flow:

1. You open Dispatch on your phone and send: “Log a $47 expense to QuickBooks under ‘Travel’, dated today.”
2. Dispatch routes the command to Claude Code on your desktop
3. Claude Code uses computer use to:
   • Open QuickBooks if it’s not already open
   • Navigate to “New Expense”
   • Fill in the amount, category, and date fields
   • Save the entry
4. Claude Code returns a completion confirmation
5. Dispatch shows you the result on your phone

Example: Reorganizing Files with a Local Tool

You have invoices in a Downloads folder that need to be processed through a local file management app — not a script, but a real GUI tool with its own interface.

1. Send: “Use FileManager Pro to move all PDFs in Downloads/invoices to the archive folder and rename them with today’s date prefix.”
2. Claude Code navigates the app, runs the operation, and confirms completion.

Safeguards Worth Building In from the Start

• Confirmation prompts — Don’t disable these until you’ve run an automation several times without issues
• Audit logs — Both Dispatch and Claude Code support logging; use them
• Scope limits — Restrict which applications Claude Code is allowed to touch for a given task
• Timeout handling — Set maximum execution times so a stalled automation doesn’t run indefinitely

Where MindStudio Fits In

Claude Code Computer Use with Dispatch handles local, on-device automation well. But many workflows also need to interact with cloud services, run on a schedule without manual triggering, or push data into external tools like a CRM or messaging system.

That’s where MindStudio connects. MindStudio’s Agent Skills Plugin (@mindstudio-ai/agent) lets any AI agent — including Claude Code — call 120+ typed capabilities as direct method calls. Instead of navigating a browser to send a notification or log data somewhere, Claude Code can call agent.sendEmail(), agent.searchGoogle(), or agent.runWorkflow() directly.

A practical extension of the Dispatch setup might look like this:

1. Dispatch triggers Claude Code on your desktop based on your phone command
2. Claude Code pulls data from a local desktop application via computer use
3. It passes that data to a MindStudio workflow that routes it to Salesforce, Slack, or Airtable
4. MindStudio sends a confirmation back

This bridges the local-to-cloud gap that neither Dispatch nor Claude Code handles on its own.

You can build workflows on MindStudio for free — no API keys or setup required for most integrations.

Troubleshooting Common Issues

Claude Is Capturing the Wrong Monitor

On multi-monitor setups, Claude may screenshot the wrong display. On Linux, set the DISPLAY variable explicitly. On macOS, check your computer use configuration for a display index option and set it to match your primary screen.

Clicks Land in the Wrong Location

If display scaling is above 100%, screen coordinates can drift relative to the screenshot. Try setting scaling to 100% for testing, or check whether your computer use setup has a DPI scaling factor setting to adjust for this.

Dispatch Commands Time Out

Increase the timeoutSeconds value in dispatch.config.json. Multi-step tasks that involve opening applications, waiting for load screens, and filling multiple fields can take 60–120 seconds. The default timeout is often shorter than that.

Phone Can’t Connect to Desktop

Check that:

• The auth token in the Dispatch mobile app matches the one from dispatch init
• Your firewall isn’t blocking the Dispatch port (check dispatch.config.json for the port number)
• If using Tailscale or another tunnel, the connection is active before you try to pair

Automation Stalls Mid-Task

This usually happens when Claude encounters an unexpected UI state — a dialog it didn’t anticipate, a loading spinner that takes longer than expected. Add explicit instructions for edge cases in your command: “If you see any dialog boxes, dismiss them and continue.”

Frequently Asked Questions

What is Claude Code Computer Use?

Claude Code Computer Use is a capability in Anthropic’s Claude Code tool that allows Claude to visually interact with a desktop — taking screenshots, moving the mouse, clicking UI elements, and typing text. It works with any visible application regardless of whether that app has an API, which makes it useful for automating legacy software, local tools, and GUI-based workflows that standard automation tools can’t reach.

Is it safe to let Claude control my desktop?

The main risk is unintended actions — Claude misreading a UI state and interacting with something it shouldn’t. Keep confirmBeforeExecute enabled when you’re starting out, use audit logs, and restrict Claude’s scope to specific applications for each task. Treat it the same way you’d treat granting someone remote access to your machine: start with limited permissions and expand carefully.

Can Claude Code Computer Use run on a headless server?

Yes, using a virtual display like Xvfb on Linux, but results are inconsistent. Some applications don’t render correctly in virtual display environments, which can mislead Claude’s visual reasoning. It works best on a machine with an active, real display session.

Does this work for browser automation too?

It can, but it’s not always the right tool for the job. For standard web automation, Playwright or Puppeteer are faster, more reliable, and cheaper per task since they don’t require screenshot cycles. Use Claude Code Computer Use for native desktop apps, system settings, and local tools where programmatic browser automation doesn’t apply.

How much does running this cost?

Each screenshot is processed as an image by the Claude API, so computer use is more token-intensive than text-only tasks. A single multi-step automation might consume anywhere from a few thousand to tens of thousands of tokens depending on complexity and how many action cycles it takes. For occasional tasks, the cost is manageable. For high-frequency use, calculate expected consumption against Anthropic’s current pricing before scaling up.

Can Dispatch trigger Claude Code on a schedule rather than on demand?

Yes. Dispatch supports cron-style scheduling in addition to on-demand triggers from your phone. You can configure recurring automations — for example, extracting data from a local reporting app every morning at 7am — without needing to send a manual command each time.

---

Key Takeaways

• Claude Code Computer Use lets Claude interact with any desktop application visually — no API required — through a continuous screenshot-reason-act loop.
• Dispatch adds remote access to that capability, letting you send natural language commands from your phone and receive results back.
• The combination is most useful for local apps, legacy software, and GUI-only workflows that standard automation tools can’t reach.
• Setup involves installing Claude Code, granting screen permissions, running Dispatch locally, and pairing your mobile device.
• Add MindStudio to the stack when your local automation needs to connect with cloud services, sync with business tools, or run without manual triggers.

If you’re looking to extend what Dispatch and Claude Code do locally into broader automated workflows, MindStudio’s agent builder connects AI agents to 1,000+ business tools — no code required. Start free at mindstudio.ai.