Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
GethN7 Wrote:robkelk Wrote:Good to hear.
Orain's been on a yo-yo string today - up, down, up, down, up, down ... Is there another DDoS going on?
We had to go back in time, so the new and old settings are clashing because all the servers haven't gotten used to the new (old) loadbalancer settings, givee it a day or two. That would explain the "server not found" errors I'm getting...
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1
Threads: 0
Joined: Feb 2018
Reputation:
0
Any status update on resolving the server errors?
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
MarqFJA87 Wrote:Any status update on resolving the server errors?
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
Our situation has evened out somewhat. We're still looking into long term ways to prevent it from happening again, but the connection issues should be resolved.
Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
Orain was running fine this morning, but it's down this afternoon...
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
Still working on it, discovered more flies in the ointment.
Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
Well. somebody's getting in (I saw an email notifying me of a reply to a topic post I made last week) ... but that somebody isn't me.
Ah, well; my page of offline work keeps growing.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
robkelk Wrote:Well. somebody's getting in (I saw an email notifying me of a reply to a topic post I made last week) ... but that somebody isn't me.
Ah, well; my page of offline work keeps growing.
From what I understand, the servers HHVM settings are utterly out of whack thanks to an auto-upgrade that caused our production cluster to crap the bed and this especially crippled our IPv4 connections. John Lewis even came back to help us fix things (with Dusti's knowledge) due to the severity of the situation, and that is still ongoing.
Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
GethN7 Wrote:robkelk Wrote:Well. somebody's getting in (I saw an email notifying me of a reply to a topic post I made last week) ... but that somebody isn't me.
Ah, well; my page of offline work keeps growing.
From what I understand, the servers HHVM settings are utterly out of whack thanks to an auto-upgrade that caused our production cluster to crap the bed and this especially crippled our IPv4 connections. John Lewis even came back to help us fix things (with Dusti's knowledge) due to the severity of the situation, and that is still ongoing. Any idea when this might be resolved? A number of the freebie wiki's editors are asking...
(And any idea why there doesn't seem to be any problem accessing Orain in the mornings Eastern time, but we can't connect in the afternoons and evenings Eastern time?)
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
The fire should be out
05-27-2015, 03:25 AM
Updates: We seem to have finally gotten all the servers humming the same tune again.
We have also gotten a new Orain staff member, NDKilla.
Also, MW 1.25 has been released, but we plan to wait until the upcoming maintenance release before the farms are upgraded to it.
Finally, I've been looking into the possibility of upgrading ATT's LiquidThreads to Flow, which has more support and is reasonably stable, and I willl put the option to do so up to a vote with information on the pros and cons ocne I have discovered if it is technically feasible for our needs.
Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
Still getting timeout messages, on both ATT and the freebie wiki (which IIRC are hosted on two different servers). And that's after clearing my cache.
EDIT: http://downforeveryoneorjustme.com/orain.org also says Orain is still down.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
robkelk Wrote:Still getting timeout messages, on both ATT and the freebie wiki (which IIRC are hosted on two different servers). And that's after clearing my cache.
EDIT: http://downforeveryoneorjustme.com/orain.org also says Orain is still down.
You may need to flush your web cache. It said Orain is down, but I can get in just fine. If you have old login cookies, it will not come up since we just refreshed all the servers. If you still have issues, let me know.
Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
Orain's gone bye-bye again. I tried a completely different browser that's never been to Orain to check.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
Fire not out, flared back up again
05-28-2015, 01:34 AM
robkelk Wrote:Orain's gone bye-bye again. I tried a completely different browser that's never been to Orain to check.
I just got an update, and while I'm not going to give out detailed specifics for security reasons, I can say the following:
We have determined the attack is definitely one on Orain in general, and based on the information we gathered, it was deliberately malicious (Orain must have pissed off someone). We're currently looking into alternative hosting setups that can shunt this sort of attack off at the knees at the moment, more when I have more information.
Apparently, whoever initiated it is still trying to screw with us, and for the protection of our servers IPv4 access is going to suck until we make alternative arrangements, else we give whoever is doing this another chance to completely crap out the servers with another DDoS.
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
Things have gotten better
05-28-2015, 06:41 PM
Orain has moved its DNS servers to Cloudflare, and will be soon rotating all of our server IPs to ensure this cannot happen again.
We are finally recovering, and we expect complete restoration of all service very soon.
Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
I assume we'll get brought back to a more recent version of MediaWiki at the same time ... (Currently, ATT is advertising the Indiegogo campaign in the all-pages bulletin, sortable tables aren't sortable any more, the User Preferences show up on one list instead of a set of tabs, Recent Changes doesn't auto-update, etc.)
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
robkelk Wrote:I assume we'll get brought back to a more recent version of MediaWiki at the same time ... (Currently, ATT is advertising the Indiegogo campaign in the all-pages bulletin, sortable tables aren't sortable any more, the User Preferences show up on one list instead of a set of tabs, Recent Changes doesn't auto-update, etc.) Yeah. Core dumping our caching to Cloudflare as fast as we did caused some weirdness, give it a few days to settle.
Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
It's been a week, so I'm guessing I can start making bug reports again...
ERROR 1: The search box isn't auto-completing, across all of Orain.
Examples:
1) I type in Buffy on ATT and I don't get a drop-down listing " Buffy the Vampire Slayer".
2) I type in Buffy the Vampire Slayer on ATT and I'm sent to the search results page instead of the Buffy the Vampire Slayer page.
3) I type in Paola on poserdazfreebies.orain.org and I don't get a drop-down listing " Paola Character"
4) I type in Paola Character on poserdazfreebies.orain.org and I'm sent to the search results page (which starts with the line " There is a page named "Paola Character" on this wiki. See also the other search results found.") instead of being sent to the Paola Character page.
ERROR 2: We're getting "bare" edit boxes - no edit help bar across the top.
Example:
![[Image: edit-box-error-2015-06-07.jpg]](http://web.ncf.ca/fm536/tempstuff/edit-box-error-2015-06-07.jpg)
ERROR 3: The cached Special Pages on ATT haven't been updated since May 30.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
Noticed. If you do leave a report, link them to the last couple of posts here. I'm having issues with shell access right now, so I can't work on this myself.
Update: Filed issue report:
http://github.com/Orain/ansible-playbook/issues/614
Posts: 27,611
Threads: 2,271
Joined: Sep 2002
Reputation:
21
We're getting a lot of "Lua error: Cannot create process." instead of header menus tonight.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
Bob Schroeck Wrote:We're getting a lot of "Lua error: Cannot create process." instead of header menus tonight.
Not surprised. Due to how we had to throw up Cloudflare as fast as we did and given how ATT is the largest wiki database we have, it's probably trying to update the template tables and due to our size is doing so at a glacial pace.
IMO, Cloudflare kinda blows because I suspected we might have this issue, but we needed the DDoS protection ASAP, so, lesser of two evils and all.
If I can ever get my shell access working, I'm forcing a tables update to make ATT refresh itself faster, which should flush this issue.
Posts: 25,578
Threads: 2,060
Joined: Feb 2005
Reputation:
12
(goes and looks)
Getting Lua errors on the freebie wiki, too...
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."
- Michael Ignatieff, addressing Stanford University in 2012
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
robkelk Wrote:(goes and looks)
Getting Lua errors on the freebie wiki, too...
We probably need to make sure Scribunto is updated as well.
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
Good news, everyone.
We're slowly but surely getting everything back in order. Cloudlfare's caching, combined with the slightly differing extension/skin loading schema of MW 1.25 has been causing some issues, but we are gradually fixing them as soon as we can get them working right.
We have purged most of the old cache and most of what should be working should be back online, but bear with us, the above mentioned has thrown Orain seriously out of whack.
Posts: 1,745
Threads: 7
Joined: Oct 2013
Reputation:
3
WARNING!
06-14-2015, 10:59 PM
Orain may have to go down again shortly, due to what appears to be a threat of a DDoS, likely by the same parties behind the first one, and for security reasons we may soon have to take Orain's servers offline for an unspecified period in the near future to reevalute our current configuration, since while we do not know the exact scope of the threat we may face, we are fairly certain the same party who threatened us last time may try again, and we have no desire to experience downtime thanks to them attempting another security breach.
|
