Redundancy in website, email, & IRC infrastructure (etc.)

Looking for a bit more than customer support, and want to learn more about what cryptostorm is , what we've been announcing lately, and how the cryptostorm network makes the magic? This is a great place to start, so make yourself at home!
User avatar
Site Admin
Posts: 495
Joined: Thu Jan 01, 1970 5:00 am

Redundancy in website, email, & IRC infrastructure (etc.)

Post by df » Thu Jun 04, 2015 3:20 am

[i]{merged several related threads into one, for ease of access ~admin}[/i]

Our server in Iceland that hosts this forum and went down last night, as did the server hosting the Iceland exit node.

The remote logs showed no sign of any intrusions or attempts, and after a lengthy email conversation with a few of the good people at Datacell (who host our Iceland servers), it turns out there were routing issues that arose when a new link was added to the network.

During the downtime, we temporarily loaded our backup onto the same server running, which is why started resolving to (and was giving an SSL cert warning because the SSL cert in use was for not

We were also in the process of setting up some more redundant backups using VMs on a dedicated server in Moldova, but those VMs weren't completely provisioned in time for this Iceland downtime.

Anyways, Datacell has fixed the routing issues, so this server (and the Iceland VPN node) is back up and the DNS for has been switched back to

User avatar
Site Admin
Posts: 495
Joined: Thu Jan 01, 1970 5:00 am

Re: Iceland went down last night, back up now

Post by df » Thu Jun 11, 2015 11:55 pm

We haven't noticed any problems with Montreal. Frankfurt's ISP hasn't mentioned any downtime in the links that control our VPN node, and they're pretty good at notifying all customers about every little hiccup going through the server (even for links that have nothing to do with our server).

As for Iceland, I can confirm from several different networks that the website is now up.
If you can access then you can definitely access the other parts of the website.
Maybe your browser is reading from cache the error it saw when Iceland was down?
A good way to verify is to use a command-line thing like wget or curl to verify you can access

Even if your router/OS is caching the old DNS record of, that should work too since I left the backup server running on the nginx handling, so if Iceland goes down again all I have to do is change one A record to have the traffic go to the backup system.

Posts: 84
Joined: Sat Jan 10, 2015 5:14 pm

Re: Iceland went down last night, back up now

Post by DudeOfLondon » Sat Jun 13, 2015 1:25 am

Frankfurt node is very picky since yesterday. Every 15-20 minutes I have massive timeouts for about 2 minutes and then it mostly gets back to normal.
When the timeouts while browsing appear, I made ping tests.
Pinging the LAN-router is 1ms.
Pinging my own External IP: gives ~55ms
But pinging for example gives timeout.

Posts: 20
Joined: Thu Mar 27, 2014 8:22 am

Re: Iceland went down last night, back up now

Post by gbj » Mon Jun 29, 2015 4:43 pm

Fernrir has been down for the last five days for me and I get no reply when contacting cryptostorm support :(


Redundancy in website, email, & IRC infrastructure

Post by cryptostorm_team » Mon Jun 29, 2015 10:06 pm

During the past several weeks, we've accelerated a longer-running project to add redundancy and resilience to our websites and other non-network resources (we call these "non-network" because these items are not part of deliving cryptostorm's secure network itself, which is entirely separate from any websites or other single-point-of-failure components).

In the first two years of our existence, we didn't judge the need for such capacity to be mission critical; a small bit of downtime here and there with, for example, might be a minor inconvenience for all of us but would not be critical path. Of course, we retain rolling backups of files (and most of our website source is already hosted at github and thus is on independent infrastructure), so in the event of a sustained outage we could - and several times, did - switch over to secondary server capacity with the backup images.

Most companies handle this issue by outsourcing their hosting to a "content delivery network" like Cloudflare. For a basket of reasons too long to list here, this is not an approach with which we are comfortable, though it is "easier" and for less technically centred project teams it will in many cases be too tempting to pass up.

So, as cryptostorm has grown and evolved since 2013, we've known that the need for redundant website (and email, and IRC... we'll just say "website" and assume all that is included, as well) capacity would eventually be something we'd need to address. As we discuss in a bit more detail in a parallel blog post at, recent attacks on Iceland's internet infrastructure have caused access to our websites (which have always been hosted there, with our colleagues at Datacell) to become, in a word, sporadic (through no fault of Datacell's, to be clear).

Given that, we pushed forward to complete our internal effort to provide redundant, distributed, failsafe website access - we'd been making steady progress but with no deadline in sight, it naturally slipped behind critical tasks and was in some senses sleepwalking. Issues in Iceland got things into fast gear, and we set a tight timeline to get things in place.

Two days ago, on Saturday, we did our first production cut-over test of the new model we've put in place. Most went smoothly, and our security procedures held together comfortable. However, there were the (if we're being candid) expected hiccups here and there: the database powering this forum was intermittently refusing to stay up on Sunday evening, for example. Those issues are all now resolved and we're fine-tuning the details.

In this thread, we'll post a bit more technical detail on how we've approached this infrastructure redundancy project - some of it's a bit routine and boring, but other components are perhaps novel and even somewhat elegant in final form. It's worth nothing that the overall project is not complete; what we've done is the first cut-over test. Now, we're layering in the automated redundancy itself (in technical terms, the first step was actually more of a challenge than the redundancy itself).

Finally, it appears that our automated 'tokenbot' delivery of newly-purchased tokens was inactive from early Sunday through Monday morning. We'd concluded this was merely the result of cached DNS data in email delivery systems, but that conclusions was not accurate and in fact the tokenbot was simply not delivering tokens. Since then, we've manually confirmed all tokens not delivered timely during that period have now been delivered. Further, we've provided complimentary 66-day tokens to all those members affected by the delay. This was a genuine screw-up on our part - timely token delivery is a big deal to us, and to many members - and we offer our apologies for not being aware of the issue, and resolving it, sooner.

If there's additional questions or reports of transitional bugs, please do feel free to post them here - we'll do our best to stay current with replies. Through today, we've invested substantially all available team effort in completing the first step of this project, and thus haven't posted much data here on what's been in process. Now that's complete, we're able to do a better job of keeping the membership informed as to ongoing developments.

Best regards,

~ cryptostorm_team

Posts: 20
Joined: Thu Mar 27, 2014 8:22 am

Icelandic server

Post by gbj » Wed Jul 01, 2015 9:04 pm

Has the Icelandic server been taken down. I dont even see the fenrir exit node on And this is after a week of downtime :problem:

User avatar
Posts: 612
Joined: Sun Dec 16, 2012 6:34 am

re: Iceland &

Post by Pattern_Juggled » Mon Jul 06, 2015 11:45 am

We have been integrating a new, less technically intense platform over at [nb][/b], and to be honest we're still learning how to coordinate information posted there with threads here.

In this case, we provided an update on Fenrir and associated Icelandic infrastructure at cryptohaven last week... but failed to provide an echo or reference to that information here. Seems obvious, in hindsight. It's not clear if we'll be automating that coordination, or simply manually echoing - either way, it's something that will be done.

Meanwhile, here's an old-fashioned copy paste of the relevant data from cryprohaven's post on the subject:

This process has been underway for more than a month, as we saw the need to provide redundant capability to serve our websites... and it was more or less on track when, in recent weeks, problems with the internet connectivity coming and going from the island of Iceland started to become noticeable, then common, then almost overwhelming... in the past couple days, "overwhelming" is the description best suited. A visit to our network status page shows all the gory details, which impact both our websites and our Icelandic cluster (anchor node: fenrir).

Because these problems are 'upstream' from both our servers themselves, and from the datacenter in which our servers are housed, there's little or nothing any of us can do to resolve them. It's like having construction on a highway between one's house and one's intended destination: no amount of driveway sweeping or cleaning will help with the highway's crash site, and until that bottleneck clears there's not going to be much happy motoring to be had.

We're not leaving Iceland, and we're not leaving our current datacentre there! However, reality is that availability there is taking a hit lately - being an island, that's a risk. Word 'on the street' (i.e. amoungst well-connected colleagues in the deeper parts of the security tech ecosystem) is that these attacks relate to certain governments trying to "break" the anonymity of visitors to some sites within the Tor privacy network. We'll write a bit more about that in a separate post, but if that's why Iceland is being hit so hard lately, it's doubly tragic: both for the targets of the attack on Tor anonymity, and because the entire country of Iceland is being impacted so one vendetta can be acted out.

We'll update this post once the sites roll back to their normal selves... meanwhile, feel free to read the couple of posts here at cryptohaven. Not much, yet, but we're happy so far with how the project is progressing.
There's not much more to add, in terms of Iceland, meanwhile - we've prioritised the infrastructure redundancy effort as critical path since then, and largely focussed on ensuring it was completed with minimal drama. That process, although still a couple steps away from its final state, largely in-hand (knock wood) and having completed the gnarly chunks of it, the admin team has been provisioning several new nodes and an entirely new mechanism for secure session routing, this holiday weekend.

Once that's in hand and rolled out early this week, we'll be circling back to see what we can do to re-launch our icelandic cluster in a way that maximises performance, resilience, and security. It's too early to say with certainty, but we're cautiously optimistic that we can do this without either spending gratuitously for very little member-supporting capacity and without sacrificing session security in the process.


~ pj
...just a scatterbrained network topologist & crypto systems architect……… ҉҉҉

[list]✨ ✨ ✨[/list]
pj@ðëëþ.bekeybase pgpmit pgpðørkßöt-on-consolegit 'er github

User avatar
Posts: 6
Joined: Mon Jun 01, 2015 4:55 am

Re: Icelandic server

Post by sin » Tue Jul 07, 2015 11:59 am

I vote to booby trap iceland datacentre with thousands and thousands of tacs placed strategically around the compound.