rohan ganapavarapu

Hacking on Xorg for Fun and Computer Use

ChatGPT unfortunately does not distribute a Linux app. I have heard a lot of good things about computer use and wanted to try it with my sub, but unfortunately I was relegated to the Codex CLI.

So I built my own. It has since (today) booked my flights, renewed my license, and paid a parking ticket.

I had a few requirements in mind:

  1. Native like performance
  2. It can access my laptop just like my normal WM (no virt)
  3. I can take over from the AI etc
  4. Does not interfere with me using my laptop
  5. I can transfer windows directly between them

So 1. eliminates a whole class of approaches with VNC, 5. eliminates using a different X server. 4. eliminates using a “virtual monitor”.

I needed to somehow run another WM within the same X server, without disturbing anything on my actual screen. To explain why that’s hard, I have to explain how Xorg actually gets a pixel onto your screen.

how Xorg works

Xorg is a server that connects to applications over a socket. The applications ask Xorg to do things on their behalf.

Apps draw stuff into windows which Xorg keeps track of. It manages the window’s position, size, and parent window it lives inside.

Everything drawn lands in a big block of memory called the root framebuffer. The CRTC is a part of the GPU which turns the data into pixels on your panel by reading the rectangle out of memory and streaming it to your screen at 60fps. Since dwm doesn’t composite, apps really draw directly into that shared block of memory.

What we want to do is use the framebuffer to store our virtual desktop without the CRTC scanning it. We can restrict the CRTC to scan 2240x1400 while making the framebuffer 2240x2800. We use the bottom half of the framebuffer as a place to store the visual data from our WM.

I didn’t want a “virtual monitor” because dwm (updategeom() -> XineramaQueryScreens()) would adopt it. It would get tags, a bar, and become a focusmon/tagmon destination. It would stop being a separate place and just become more of my desktop.

two window managers, one X server

All a WM does is manipulate a tree of windows. The root window covers the entire framebuffer and every app is a child of this root. An app says “show this window” and X11 asks the WM “ok, where do you want to actually put this?”. Crucially, the app never knows a WM exists. It simply asks the server, and the server hands the request to whoever registered (the WM) to intercept that window’s parent. This is only ever about showing, moving, and resizing. Once the app has been handed a rectangle it draws into it directly, and the WM never hears about it again.

This is why dwm is so suckless. It’s just a very thin wrapper of X11. The problem it’s solving is that apps don’t know about where other apps are so they can’t coordinate where they get painted on the screen. It just mediates between all the apps and allocates space in a sane, user-controlled, way.

A WM is just like any other app or process, but you can’t have two WMs intercepting the children of the same window. So, how, if the host WM already owns the root window (which represents the entire framebuffer) and is in charge of its tiling, can we put another WM inside of it?

There are three tricks. The first is that we patch the inner dwm so that instead of asking X11 what its root window is it gets it from an env variable. We then create a window that occupies the bottom portion of the framebuffer and tell the inner dwm that it is its root node, even though it’s technically a child of the root window the outer dwm manages.

All of this is done by a small daemon. It creates that container window, owns the agent’s input devices, and is the only thing both sides talk to: me from a terminal, and the agent over MCP.

But then, how do you prevent the outer dwm from messing with and trying to tile/manage that container? By default, X11 will ask the outer dwm to manage/intercept it like any other window. We just want to let the application (the inner dwm) sit exactly where we put it without getting intercepted by the outer one. Well it turns out, notifications, pop ups, tool tips, etc. need this behavior. This is the second trick: X11 has an override_redirect flag which is used to bypass the window manager entirely so a window can be mapped exactly where it asks. The outer dwm is never even told the container exists.

So you can have two WMs, but they’re both tiling into the same framebuffer. How does the outer dwm know not to tile its own stuff in the space occupied by the inner one? dwm’s happy path gets its geometry and sizing from RandR/Xinerama (the list of monitors) and not from the framebuffer, so it only ever tiles stuff inside the visible portion. It never considers the space below because as far as it knows that space isn’t part of any monitor.

The third trick is the inner half. You can compile dwm without Xinerama, in which case it never asks for the monitor list at all and falls back to mons->mw = sw; mons->mh = sh, i.e. just fill the root window. The inner dwm takes this fallback path on purpose. Since its “root” container is only the bottom 2240x1400, it thinks that is the extent of the framebuffer. If you kept the Xinerama path it would read the real monitor list and try to manage the same visible area as the outer dwm.

moving a window from the human desktop to the agent’s

Because both desktops are just nodes in the same tree on the same server, all you are doing is reparenting the child nodes. X11 has a useful XReparentWindow function for this. The window keeps its ID, its pixels, its process, and its connection to the server.

One edge case to note is that if the daemon dies, the container goes with it, and destroying a window destroys its children. XAddToSaveSet tells the server to give the window back to the (true) root node instead of killing it off.

decoupling input

But we have another problem, how do we decouple our normal inputs and the AI’s? There’s one X server here, and by default that means one cursor and one focused window (where most key events go). If the AI moves the cursor, it moves my cursor.

Focus is Xorg’s record of which window your keystrokes go to (denoted by a blue outline in my dwm build). It’s initially confusing because we think of focus as something a window has, when really it’s something the keyboard points at. Key inputs work via a virtual keyboard living inside Xorg which is fed by your physical one. The virtual keyboard also keeps track of which window is in focus. There’s also a virtual mouse that owns the cursor you see on screen. However, if you plug five mice or keyboards into your machine, they all feed the same virtual mouse and keyboard.

Xorg was built assuming one person sat at the machine and one virtual keyboard and mouse, but was updated with XI2 (X Input Extension 2) in 2009 to let you make more. They get created in twos, a virtual mouse and a virtual keyboard together, because clicking a window to focus it has to set focus on some keyboard, so every virtual mouse needs a keyboard partner.

A pair is also allowed to have no physical devices attached at all. So the daemon makes a second pair for the agent. My trackpad and keyboard stay attached to the human’s. The agent’s pair has no hardware behind it at all. The daemon fabricates the events in software and feeds them in, the same way the physical trackpad feeds the human’s.

That should be the end of it, but Xorg clamps all pointing input devices to the region the CRTC is scanning. It assumes you don’t want to point at something not visible on your desktop.

This is where the hackiness comes in. In Xorg, there is a legacy feature called RandR panning. The intended purpose is for when you have a screen smaller than your framebuffer and need to scroll around inside it. Since there is no longer a fixed CRTC region being scanned (CRTC can pan to any part of the framebuffer), it doesn’t know where to clamp and instead disables it.

Panning has two numbers: the total, which is the area you can scroll through, and the tracking area, which is where the pointer is allowed to roam. Since you don’t want to actually be able to pan to the bottom half, you set the total to exactly your screen’s real resolution. There’s no room left to scroll, so nothing ever moves. Then you set the tracking area to the whole framebuffer. Xorg still thinks you’re panning, so it allows pointers into the non-visible part.

One catch: growing the framebuffer and turning on panning have to be the same xrandr invocation. Do them as two commands and xrandr shrinks the root back down to match the panning total, and you’re clamped all over again.

Well, you have a new problem. Your (the human’s) pointer can now wander off the screen into the bottom of the framebuffer off the visible desktop/CRTC region. What you do is you create an XFixes pointer barrier which is scoped to just your virtual mouse. Four of them, boxing in the physical region. Since the agent’s isn’t in the device list, it walks straight through.

making keybinds work

Generally, all key inputs go to the focused window (e.g. typing into chrome address bar, Ctrl-l). But dwm’s keybinds (e.g. Alt-Shift-q, which kills the active window) need to be hijacked and go to dwm instead.

The way this is done is through something called a passive grab. When dwm starts, for each of its keybinds, it tells Xorg to not deliver it to whatever’s focused, but intercept it and send it to the WM.

dwm asks to intercept on all of its keybinds and registers on the root window (or the window it thinks is the root). Since every window is somewhere under the root, the grab automatically intercepts inputs no matter what window is in focus.

And, since XGrabKey takes no device argument, it can’t tell the human’s key presses from the agent’s. XI2 later added XIGrabKeycode, which does allow you to filter by device. However, since I wanted to keep the host dwm stock (we don’t vendor it), the source still uses the old function, and the passive grab fires no matter which keyboard the event came from. (Note that the inner dwm uses the new function, since we vendor it). So if the agent runs an Alt-Shift-q or a similar command, it affects both the inner and outer WM, and kills whatever window you happen to have focused while the agent is working.

Instead of patching the outer/host dwm to ignore key inputs from the agent’s virtual keyboard, we have the MCP recognize when the agent uses a keybind that would be hijacked and send the daemon a command. Rather than forwarding a key press, the daemon sends the inner dwm an X11 ClientMessage (same mechanism taskbars use to activate a window) asking it to do whatever the binding corresponds to directly. Since no key was ever pressed, nothing is matched by the outer dwm’s passive grab.

viewing and taking over the agent’s desktop

The human needs a way to view the agent’s desktop and take over control. The workspace is already sitting in a portion of the framebuffer. It’s just that no CRTC scans it. Fundamentally, all we want to do is copy one part of memory to another.

The naive solution is to just ask the server for the pixels in the window and constantly redraw them. The function for this is XGetImage. There are two main problems.

  1. You’d be polling constantly, redrawing an idle desktop 60 times a second
  2. XGetImage copies the pixels through a socket, about 12MB a frame

XDamage is an X11 extension that lets applications track modified regions of drawables. The viewer subscribes to it and receives events when something changes in the agent’s desktop. On receiving an event, the viewer copies that region and calls XDamageSubtract to say it’s caught up, and Xorg starts accumulating changes again from empty. Instead of constantly polling, you only draw on changes.

Instead of using a socket, there is another X11 extension called MIT-SHM which allows you to allocate a System V shared memory segment and tell the server about it. The X server and the viewer having the same physical pages mapped into their address spaces allows you to call XShmGetImage which writes pixels directly into that segment.

Because the viewer’s size is variable (not always 2240x1400), you use GL to do scaling. glTexSubImage2D puts the segment’s bytes into a texture, and then you draw a rectangle the size of your window with the texture stretched over it. Texture coords are normalized to the closed unit square (0,0) to (1,1), regardless of how many pixels are in memory. The GPU figures out which screen pixels are covered by the rectangle and samples the texture for each one.

Now for taking over, which is a UX problem already solved with VMs. The behavior we want is when you click into the window it captures all your input until you press a predetermined escape key. We do this with “active grabs”. A passive grab, from earlier, is “tell me only when one of these combinations fire”. An active grab is “give me every key and every click until I say stop.” Active grabs outrank passive ones, and prevent the outer dwm’s bindings from triggering on your key inputs.

Once it has the human’s input, the viewer doesn’t inject anything itself into the agent’s desktop. When the human clicks inside the viewer’s window, it converts that position into one corresponding to the window in the bottom half of the framebuffer, and sends that to the daemon. The daemon injects the event into the agent’s virtual pointer/keyboard. The agent’s desktop only ever sees one source of (x)input, which generalizes between human and agent use.

Both requirements 3 and 2 are natural consequences of this approach. All windows are still real ones on our real X server, driven by real Xorg input devices. The agent’s apps are my apps, running as me, maintaining my sessions.

The viewer is strictly for human use, and the agent itself never uses it at all. Its screenshots are read straight off the framebuffer at the native 2240x1400, so nothing is ever scaled and the coordinate space it clicks in never moves under it. I can resize the viewer, hide it, kill it, or run three of them. We’ve fully decoupled viewing from the agent’s operations by treating the window occupying the bottom half of the framebuffer as immutable, and our source of truth.

practical utility

I have found that computer use (gpt-5.6-sol xhigh fast in codex-cli) has exceeded my expectations. It works autonomously, follows guidelines, and only blocks on my input when it absolutely needs to (2fa codes, organ donor status). All of this was done in a single day. These are all the things it has done:

Find the open source code/repo here (note, although this blog post is not generated by AI, everything in the repo is): https://github.com/Ocean-Moist/linux-computer