The Horizon
Secrets are the new target
On October 21 Railway, the deployment platform, published a security disclosure. Their investigation so far indicates that an unauthorized third party may have accessed the systems that store customer environment variables, and, for some variables, obtained the decrypted values. Environment variables often hold API keys, database passwords and registry credentials. Railway paused all deployments during the investigation and advised all users, as a precaution, to rotate every environment variable stored there and review their logs for unusual activity.
If you’re on Railway, stop reading and go rotate everything. I’ll wait.
Why this one stuck with me
Incidents happen to every platform, and in my view Railway’s communication was fast and clear. What stuck with me was one line in their disclosure: they believe it’s part of a series of attacks recently targeting platform companies, aimed at compromising AI inference providers. They’ve also notified upstream providers so those keys can be revoked.
My read on why: a model API key with a big quota is close to cash. Someone holding it could resell the access, run their own workloads on your bill, or burn through your budget fast. In my experience these keys also tend to get less care than database credentials.
Four questions I’d check this week
Whatever platform you’re on, these are worth an hour of your team’s time.
1. Where do your AI keys actually live? One secret store, or copied into ten places: CI variables, a few laptops, an old staging server, a notebook someone shared once. You can’t protect what you can’t list.
2. Do they have spend limits and alerts? Some providers let you set spend limits and alerts at the organization or project level (OpenAI’s guide, for example), so check what yours offers. A stolen key with no limit is a very expensive morning. A stolen key with a limit and an alert is an annoying one.
3. Can you rotate every secret in under an hour? This is the real test. If rotating means hunting through repos and asking “who set this up?”, the gap isn’t the leak, it’s the process. Write the runbook now, while nothing is on fire.
4. Do your agents and pipelines use short-lived tokens? As more agents run in CI and touch production, the old habit of one long-lived key for everything gets riskier. Short-lived tokens, scoped per job, limit the damage when something leaks.
The bigger point
In my view, we spent the last couple of years making it easy to plug AI into everything, and that was the right call. But every integration came with a key, and in a lot of setups those keys sit right next to the most sensitive things we own. Security for the AI era isn’t only about prompt injection and model behaviour. A lot of it is the boring stuff, done properly: knowing where your secrets are, limiting what they can do, and being able to change them fast.
When did you last rotate yours?
Sources
The Horizon
Engineering leadership for the AI era, every other week.
One honest piece every two weeks on AI in software delivery, platforms and leading teams. No spam, unsubscribe anytime.