A kill switch with a name on it
When a social network changes something overnight, a publishing service can fire thousands of failing requests at it and collect strikes against your accounts. The Relay’s switches stop that — and every one records who threw it and when it clears.
· 3 min read
Here is the incident every publishing service eventually has. A social network changes something about its API late on a Friday. The adapter that talks to it starts failing — not once, but for every scheduled post, on every connected account, retrying as designed. By Saturday morning the service has sent a hundred thousand failed requests, and the network has started treating the accounts they came from as misbehaving.
The customer did nothing wrong, and their accounts are collecting strikes. The fix is an emergency release; the damage happens in the hours before it ships.
Switches, at the smallest scope that works
The Relay has had incident controls since its first operational increment: pause publishing, resume it, read how deep the queues are. They pause at four scopes:
- A connection — one account on one network.
- A workspace — one customer, every network.
- A platform — one network, every customer.
- Global — everything.
The Friday-night incident above wants the platform switch: stop everything going to the network that changed, and nothing else. The form that throws switches leans deliberately towards the smallest one, because the instinct under pressure is to reach for global — and a global pause turns one network’s bad night into every customer’s.
The instinct under pressure is the biggest switch. The form is built to argue for the smallest.
A name and a clock on every switch
Until now those controls were reachable only from a terminal. They now have a page in the operator console, and the page records two things a terminal command never did.
Who threw it. Every action carries the staff member’s address, so a switch thrown against one customer’s workspace lands on that customer’s audit trail naming a person — not "operator". A customer asking why their posts stopped on Saturday gets an answer with a name in it.
When it clears. A switch carries its own expiry. The most common failure of a kill switch is not throwing it too late; it is forgetting to turn it back off, and a paused workspace nobody remembers is an outage that started as a fix.
A credential that is only for emergencies
The page uses the Relay’s operator credential, and nothing else in the console touches it. That separation is deliberate: the operator credential is the one that gets rotated after an incident, so routine traffic running on it would make rotating it break things — at exactly the moment you most need to rotate.
