What happened
On October 21, Railway disclosed that an unauthorized party had accessed systems that store customer environment variables, and that for some of them the attacker may have obtained the decrypted contents. Railway paused all deployments, notified upstream providers to revoke keys, and told every user to roll their environment variables, including private registry credentials.
Its preliminary report fills in more detail:
- The attack window ran from October 20 to early October 22 (UTC). It started with a privilege escalation attempt in build infrastructure.
- At most 1.85% of workspaces, roughly 100,000, had some variables potentially compromised. Because of how the keys were encrypted, Railway can't yet say exactly which secrets in an affected project were exposed.
- The attacker went looking for AI provider API keys and platform API keys. Railway flags workspaces with lots of environment variables, or lots of AI keys, as higher risk.
- Railway says it cannot refund AI spend made with compromised keys. You have to take that up with the provider.
TechCrunch reports that Railway believes this was part of a wider campaign against platforms, aimed at compromising AI inference providers. The same week, AI coding tool OpenCode disclosed that hackers had taken API keys and other secrets from more than 1% of its customers.
Why builders are the target
Stolen AI keys are worth real money. Attackers resell access or burn through your quota on their own workloads, a pattern researchers call LLMjacking. A hosting platform is an efficient place to harvest them, because thousands of apps keep keys for OpenAI, Anthropic, Stripe and their database in one place.
There's an uncomfortable detail in Railway's report. The variables were encrypted, but the platform has to be able to decrypt them to inject them into your builds and running services. Once the attacker had the right internal access, encryption at rest didn't stop them. That isn't unique to Railway. Any host that runs your code can read your secrets.
So the question isn't "is my host secure?" It's "if my host's secrets leaked tomorrow, how bad would it be, and how fast could I recover?"
How to check your exposure
Spend 20 minutes on this, even if you've already clicked "rotate" on a few things.
1. Inventory every secret. List the variable names in each project and environment, including staging and preview environments. If you keep a .env.example, it's a good starting point:
grep -oE '^[A-Z0-9_]+' .env.exampleFor each one, note what it unlocks: a database, a payment provider, an AI provider, email, a storage bucket, a GitHub token.
2. Check your AI provider usage and billing. Look for usage spikes, unfamiliar models, or requests at odd hours since October 20. This is the most likely place for abuse to show up first.
3. Check the audit logs you have. Railway specifically recommends reviewing GitHub audit logs and installed app access, and registry logs such as Docker Hub. Also look at your payment provider's API logs and your database's connection logs.
4. Check whether your database is reachable from the internet. Railway's report recommends removing public network access to databases. If a leaked connection string points at a database anyone can connect to, the leak is immediately usable:
nc -zv your-db-host.example 5432 # "succeeded" from your laptop = publicly reachable5. Check what your frontend already gives away. Breaches like this are a reminder to separate real secrets from values that are public by design. A key that ships in your JavaScript bundle was never secret to begin with.
How to rotate without breaking production
The mistake people make under pressure is revoking the old key first, then scrambling to deploy the new one while the app returns errors. Do it in this order instead:
- Create a new credential at the provider (new API key, new database password, new webhook secret). Keep the old one active for now.
- Update the variable on your host, and redeploy so the running app picks it up.
- Verify the app works with the new value. Hit an endpoint that actually uses that service.
- Revoke the old credential at the provider.
- Confirm the old one is dead, for example by making a test call with it and checking that it fails.
Go through your list in order of blast radius: payment keys, database credentials, cloud and storage keys, AI provider keys, GitHub and registry tokens, then everything else. Railway's own guidance focuses on variables set before October 21, and its follow-up advice adds reconnecting GitHub and recreating CLI and CI credentials.
A few secrets need extra care:
- Session signing or JWT secrets. Changing these logs every user out. That's usually acceptable after a breach, but warn users if you can.
- Webhook signing secrets. Update the receiving side and the sending provider together, or you'll reject legitimate events for a while.
- Database passwords. If your platform supports it, create a new database user with the same permissions, switch the app over, then drop the old user. That's cleaner than changing a password in place.
Make the next breach less painful
You can't stop your host from getting breached. You can make a leaked secret worth less.
Scope keys tightly. Use restricted keys where providers offer them. A Stripe restricted key that can only create checkout sessions is far less useful to an attacker than a full secret key.
Cap spend. Set monthly budgets and usage alerts on every AI provider. This turns a five-figure surprise into an email.
Separate environments. Don't reuse production keys in staging or preview deploys. Each environment should have its own credentials so a leak in one doesn't hand over the other.
Keep databases private. Use your platform's private networking, and if you're on Supabase, make sure Row Level Security is on so a leaked anon key doesn't expose your tables.
Never keep secrets in reachable files. A .env file deployed into your web root leaks without any breach at all.
Write down your rotation runbook now. The list from step 1, the order above, and where each key is managed. Next time, rotation takes an hour instead of a weekend.
Quick checklist
Run through this for every app you deploy:
- Every secret in every environment is listed, with what it unlocks.
- Each one was rotated in create, deploy, verify, revoke order.
- AI provider usage and billing reviewed since October 20, with spend caps set.
- GitHub, registry, payment and database logs checked.
- Databases are not publicly reachable.
- Production and staging use different keys.
- No real secrets ship in frontend code or sit in web-accessible files.
The takeaway
Railway did the right thing by disclosing quickly and pushing for rotation. But this keeps happening to hosting and developer platforms, because that's where the keys are. The builders who come through it calmly are the ones who know what secrets they have and can swap them in an afternoon.
While you're cleaning up, it's worth checking what your app exposes from the outside too. AppSafe scans your live site for common issues like API keys in JavaScript bundles, exposed .env and .git files, and missing security headers, with plain-language fixes. Our launch checklist for AI-built apps covers the other basics.
When you've rotated everything, run a free AppSafe scan to make sure nothing else is leaking.