Rendered at 12:10:42 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
RoboErectus 5 hours ago [-]
The timeline is wild.
It started on the 20th. When they paused builds again at 2am I was still awake from dealing with this and started migrating to a competitor.
There are about 11hrs between when they confirmed Very Bad reads and notifying customers. But it was ~24 hours after they realized someone had some privilege escalation from their build infra.
My mind is boggled by their response.
If someone is trying to get in, literally anyone can point three or four agents at every attack surface and get your shit locked down. To be just fully under attack for... 48 hours? I can't imagine the situation to just be sitting there and taking it.
I told someone half joking that this seemed like they'd been compromised because it was the only thing that made sense. I wonder why google deleted their whole account earlier this year but all the trouble I've had with them I start to wonder less.
internetter 17 hours ago [-]
This was an awful write up. I want to know how, over multiple days worth of mitigations, the attacker remained in the system. I want to know exactly, from a detailed technical perspective, what flaws there were, what actions have been taken to fix them, and how you can be positive an attacker won’t gain access in the same manner. This post is so high level that it sounds like railway doesn’t even know the answer to them.
realityagent 15 hours ago [-]
That article's tl;dr seems to be: "we recommend you migrate off of Railway".
Happy to oblige, already taken care of.
antimatter 15 hours ago [-]
Just curious, where have you migrated to? Plenty of competitors.
realityagent 10 hours ago [-]
For my single simple service, EC2 for now as I need it running reliably and don't have time to mess around with more optimization.
TBD where it lands in the future. Railway was supposed to keep this simple and less messy, but you get what you pay for, as always.
l5870uoo9y 19 hours ago [-]
> Forensic investigation evidence indicates the attacker was likely looking for AI and platform API keys, for distilling model weights.
It’s impressive that AI API keys are so valuable that attackers are willing to devote resources to hacking a hosting company just to gain access to them.
Another thing: based the actions they have undertaken, it doesn’t sound like they’ve figured out how they were compromised.
khalidkhair 15 hours ago [-]
> Why did Railway not notify customers of potentially decrypted keys sooner? ... we wanted to ensure the attacker was ejected from our running infrastructure before telling users to roll their credentials, to prevent the credentials from becoming immediately compromised again.
The way this was handled is messed up to the point it's starting to sound hilarious, they could've easily let people know ASAP that the keys were compromised so they can disable them early and Railway could temporarily block them from updating secrets to keep them secure all around in the meantime
alpineman 16 hours ago [-]
What an absolute mess. Nothing but stress with railway over the past year.
It started on the 20th. When they paused builds again at 2am I was still awake from dealing with this and started migrating to a competitor.
There are about 11hrs between when they confirmed Very Bad reads and notifying customers. But it was ~24 hours after they realized someone had some privilege escalation from their build infra.
My mind is boggled by their response.
If someone is trying to get in, literally anyone can point three or four agents at every attack surface and get your shit locked down. To be just fully under attack for... 48 hours? I can't imagine the situation to just be sitting there and taking it.
I told someone half joking that this seemed like they'd been compromised because it was the only thing that made sense. I wonder why google deleted their whole account earlier this year but all the trouble I've had with them I start to wonder less.
TBD where it lands in the future. Railway was supposed to keep this simple and less messy, but you get what you pay for, as always.
It’s impressive that AI API keys are so valuable that attackers are willing to devote resources to hacking a hosting company just to gain access to them.
Another thing: based the actions they have undertaken, it doesn’t sound like they’ve figured out how they were compromised.
The way this was handled is messed up to the point it's starting to sound hilarious, they could've easily let people know ASAP that the keys were compromised so they can disable them early and Railway could temporarily block them from updating secrets to keep them secure all around in the meantime