Actually, we discovered the issue sometime last night: Turns out some douche canoe tried DDoSing Orain again. Thankfully, DigitalOcean wasn't asleep at the wheel half as long and nullrouted the retard not long into their reign of terror, and we found and plugged the leak.
Do note that even though we now have security measures against this sort of crap, doesn't make it impossible if some idiot with a grudge and way too much free time is really determined, but they didn't do even a fraction of the damage they were intending now that we have safeguards.
If it happens again, I'll let you guys know.
Update: It happened again, we're currently identifying the source of the attack so it can be shunted off. In the meantime, bookmark this link:
http://ganglia.orain.org/ganglia/
You can see the memory usage for servers via this. If the values are frightfully high for bandwidth (as in, spiking to the ceiling for no real reason) on multiple servers but fine on other and Orain is down, you'll know we had another attack. FYI, prod 8/9 are currently our MW servers.
Do note that even though we now have security measures against this sort of crap, doesn't make it impossible if some idiot with a grudge and way too much free time is really determined, but they didn't do even a fraction of the damage they were intending now that we have safeguards.
If it happens again, I'll let you guys know.
Update: It happened again, we're currently identifying the source of the attack so it can be shunted off. In the meantime, bookmark this link:
http://ganglia.orain.org/ganglia/
You can see the memory usage for servers via this. If the values are frightfully high for bandwidth (as in, spiking to the ceiling for no real reason) on multiple servers but fine on other and Orain is down, you'll know we had another attack. FYI, prod 8/9 are currently our MW servers.