Computer Use — Driving Apps Visually

~16 min · cowork, computer-use, screen, visual

Level 0🌱 Novice

0 XP0/70 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

When the API doesn't exist, drive the UI

Computer Use lets Cowork take screenshots, analyze the UI, click, type, scroll — visually operating any app on your Mac, just like a human. It's the fallback when there's no MCP connector for what you need: filling expense forms, navigating a desktop app with no API, copying between two GUIs.

The architecture: screenshot → identify elements → plan actions → execute → screenshot again to verify. After every action, Cowork looks at the new screen state before deciding the next move. That feedback loop is what makes it more reliable than blind scripting; it's also why it's slower than an API-backed tool.

Best practice: MCP first, Computer Use second. If Slack has an MCP, use it. Computer Use is for the apps where there is no other option. Anthropic's own guidance: scope screen-recording permission narrowly (only the apps that need it), enable Computer Use only for the duration of the task.

Code

Same job, two execution paths·text

Goal:  "Send a summary of today's analysis to #product-team on Slack"

Path A — MCP (preferred):
  1. Cowork uses Slack MCP → posts message via API
  2. Reliable, fast, auditable

Path B — Computer Use (fallback if no Slack MCP):
  1. Open Slack via Computer Use
  2. Screenshot, find #product-team in sidebar
  3. Click channel, click input box
  4. Type message, click send
  5. Screenshot to verify the message appears
  Slower, more failure modes, but works when API doesn't

Form-filling — a real Computer Use task·text

Goal:  "Fill out the expense report in Concur for my Jan trip.
        Details in /inbox/trip-expenses-jan.md, cost center CC-4521."

Cowork's plan:
  1. Read /inbox/trip-expenses-jan.md
  2. Open browser, navigate to Concur URL
  3. Screenshot, identify form fields
  4. Fill each field: date, amount, category, description, CC
  5. Screenshot to verify entries
  6. PAUSE before final submit: "Ready to submit. Confirm?"
  7. On confirm: click Submit, screenshot the success state

External links

Exercise

Pick a real desktop task that no MCP currently covers (a form, a config dialog, a small native app). Run it via Cowork's Computer Use. Watch the screenshot/action loop. Note the failure modes you hit and how Cowork recovered.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousMCP Connectors — Pull Data In, Push Results Back Next →Cowork vs Operator vs Custom GPTs

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.