Icelandic power outage: 15 Dec 2014

[cryptostorm_admin]

This afternoon, we had a multi-hour outage in our Icelandic datacentre, Datacell. Everything is back up & running fine currently.

Because Iceland has very limited subsea cable connectivity, and because there's really only two serious datacentres on the island (one of which we strongly prefer, i.e. Datacell), it's difficult to enable meaningful redundancy in this exitnode cluster. We're always looking for an approach to accomplish Icelandic redundancy, but at this point it's simply not something we can provide in production context. Thus, from time to time there are these hiccups in node availability in the cluster there.

Here's the report we received from Datacell, this evening:

FYI, there was a huge powercut in Reykjavik which was over 2h. so our UPS didnt hold that long.
Servers are coming backup but the DHCP did mix a few things up so I'm booting the servers one by one.
Your servers might be up already but have a different ip if it got it via DHCP originally

Thank you,

cryptostorm_admin

[parityboy]

@OP

One thing that might be of use is to relocate your IRC server. After I noticed the website being offline, the next place I headed was the IRC channel, until I remembered that it's hosted on the same network (and possibly on the same machine). Granted, if the website goes down users won't be able to get to IRC from the website anyway, but this incident might encourage them to use a dedicated IRC client for out-of-band and (unofficial) assistance, or at least a news update.

So yeah, move the IRC server to a different datacentre.

[cryptostorm_support]

That's a pretty good suggestion, parityboy. I'll definitely be bringing that up during our next staff dorkfest... I mean, "meeting"

[DesuStrike]

I second that!

[Pattern_Juggled]

On and off we've been seeing this pattern of packet loss several hops upstream from our Icelandic network infrastructure:

Not sure what's causing it, but we're asking around to see what's up. It looks like it's not even in our datacentre (the super-colo DataCell), so there's probably an interesting story to be told.

More news as we can track it down.

Cheers,

~ pj