Gemini on macOS used to be a panel beside your work: you typed, it answered, you copied the answer across yourself. Computer Control changes the second half of that loop. The panel now reads the visible screen and drives other applications, so a single request can open a document, pull figures into a spreadsheet, check a page in the browser and send the result by mail.
The shift is easy to underestimate. A chat assistant that gets something wrong produces a wrong sentence, which you notice before you use it. An agent that acts across four windows produces a wrong action, often after the file has been saved and the mail has gone out. Everything that was a text problem becomes an operations and permissions problem.
01From chat to cross-app execution
The capability upgrade rests on three moves that happen in sequence every time you give the agent a task.
- Observe. The agent reads what is on screen: the front window, the app that owns it, the controls it contains. This is the point where the shape of your desktop stops being private to you.
- Plan. The request is turned into a short sequence of steps, mapped onto specific apps. “Put these numbers in the sheet” becomes: activate the spreadsheet, find the active cell, enter the values, keep the formatting.
- Act. Clicks and keystrokes are sent to the target app, and the result is checked before the next step. When the plan depends on something being where it was a moment ago, that check is what keeps it on the rails.
A text assistant can be wrong and you lose a sentence. A desktop agent can be wrong and you lose a file, a booking, or an email you never meant to send.
The practical difference shows up in the size of the task. Asking for a summary is one step and needs one window. Asking to reconcile a spreadsheet against a document, then email the exceptions, is a chain of steps that crosses app boundaries, and every boundary is a place where the agent has to get the app identity, the window state and the target control right.
| Aspect | Chat assistant | Computer Control |
|---|---|---|
| Scope | Its own window | Every app it is allowed to see |
| Output | Text you copy across | Changes made inside your apps |
| Failure cost | A sentence to rewrite | A file, message or setting changed |
| Trust needed | Accuracy | Accuracy plus the right to act |
02Authorization risks: who said you could do that?
To act, the agent needs permissions that are broad by design. Screen and accessibility access let it read what is displayed and send clicks and keystrokes to any app that asks for them, and that breadth is where the risks concentrate.
Blanket access
A single toggle can expose every app on the desktop. The agent may only need two of them.
Wrong window, right click
A click can land in a different window than the one planned, and the agent carries on as if it did not.
Permission creep
Access granted for one task stays on. Later tasks inherit reach you never reconsidered.
What a mistake looks like
A click meant for a spreadsheet cell lands in the document behind it. A send button is found by position, not by meaning, and the message goes out before the draft is checked. An agent told to “tidy this up” deletes a row it read as blank. None of these are exotic cases; they are the ordinary consequences of giving a software process control of a surface built for a person.
03Privacy boundaries move when the screen becomes readable
Any window the agent can see is now material it can use. Most desktops have more of it open than their owners remember: a bank tab, a private message thread, a half-written email, a client folder in the background.
Three boundaries deserve attention before the first task.
- Which apps are in scope. The safest setup is a short, deliberate list, not the whole desktop.
- What is on screen when a task starts. The agent reads what it sees, so background windows count.
- What stays after the task ends. Whether screen content, screenshots or transcripts persist after the work is done matters as much as what was read during it.
The permissions that make this possible are the same ones that let a helpful workflow run. The useful discipline is to treat Computer Control like giving a colleague access to your machine for one job, then withdrawing it when the job is over.
04How to run it without regrets
Start with one chain of steps and one app. Watch the first pass rather than walking away. Revoke access when the task ends, and keep anything sensitive out of the windows that stay open during the run. That is not caution for its own sake: it is the difference between delegating a task and delegating a desktop.