US-central exitnode cluster | anchor node = | updates & "crazy shit"

Looking for a bit more than customer support, and want to learn more about what cryptostorm is , what we've been announcing lately, and how the cryptostorm network makes the magic? This is a great place to start, so make yourself at home!

US-central exitnode cluster | anchor node = | updates & "crazy shit"

Post by cryptostorm_team » Thu Dec 25, 2014 11:58 am

note: folks seeking the most current client configuration files need not wade through this entire discussion thread! The current versions are always posted in a separate, dedicated thread, and will be continuously updated there. Continue reading this thread if you're curious about the details of the config files, want to see earlier versions of them, or have comments/feedback to provide - thanks! :thumbup:


As we've mentioned elsewhere, we have recently completed an upgrade network-wide of our resource management process, known as the Hostname Assignment Framework, or HAF. One benefit of the HAF is dramatically enhanced flexibility in making fast, flexible shifts in server/node resources when circumstances require it.

In recent weeks, that flexibility has been crucial in our response to issues relating to one of our U.S.-based exitnode clusters. Briefly, we've moved to a three-zone U.S. coverage model: useast, uswest, and uscentral. Having downed our useast anchor node due to DMCA-related issues (that cluster will be reborn during Q1 2015, with new datacentre providers & redundant capacity early on), we shifted focus to creation of a strong uscentral cluster, with the deployment of a well-provisioned dedicated machine in Chicago, IL: mishigami.

However, circumstances required us to quickly shift member traffic from that node. Those circumstances (referred to as "crazy shit" more than once, both in twitter & during internal team discussions, if we may be blunt) is being written up separately and will be linked-to from this post when ready for publication.
craaazy_small.png (34.62 KiB) Viewed 34720 times
On an interim basis, we load-shifted uscentral sessions to uswest as we provisioned a new 'mishigami' in the central U.S.

That process is now complete, and is reborn. Folks curious about such things will note that uscentral HAF lookups now resolve to this anchor node; we expect to have redundant/expansion nodes in place within two weeks, as this cluster continues to grow.

Meanwhile, the prior mishigami has been reborn as - a donated Tor relay (pingdom uptime stats). It carries no member traffic and any cryptostorm-related data has long since been rm'd from the box. Now it stands as a sort of testament to... well, that's best discussed in the "crazy shit" post.

We're quite pleased with the performance of our new uscentral cluster after this series of transitions and adjustments, and we look forward to adding to this cluster as member awareness of its capacity and reliability continues to grow.
  • (HAF entries: |

Finally, we've been asked few times about the naming of our new node. Here's a bit of backstory:

Thank you,

~ cryptostorm_team


US-central exitnode cluster:

Post by cryptostorm_ops » Sat Dec 27, 2014 5:16 am

We have completed the transition to the newly-architected central US exitnode cluster, anchored by the new node in Chicago.

All prior HAF mappings pointing to 'chili' are now remapped at the TLD resolver to the relevant mishigami instances. Additionally, mishigami is now in the resource pools for all relevant balancers of whatever OS flavour, having replaced chili's IP-space in that regard.

Further, mishigami is fully HAF 1.4 compliant. Thus, for those who want to get a jump on the use of long-term-stable HAF connection settings, the newly-added resolvers for central US are:


The HAF 1.4 pooled resolvers (which currently only include Lisbon and central US-based instances, but which will grow to include all nodes and instances as the 1.4 HAF upgrade continues across the network), are and will remain as:

  • Sticky (TTL-based, DNS randomised) balancer:

    - - - -

    Sticky (TTL-based, DNS randomised) balancer:

Finally, here's a pre-release 1.4 version of the linux config file with the central US connection parameters already included:

And here's the text of the document, for those who prefer not to download a separate file:

Code: Select all

# this is the client settings file, versioning...
# cstorm_linux-uscentral_3.conf
# last update date: 28 Dec 2014

# it is intended to provide connection solely to the central USA exitnode cluster
# DNS resolver redundancy provided by TLD-striped, randomised lookup queries
# Chelsea Manning is indeed a badassed chick: #FreeChelsea!
# also... FuckTheNSA - for reals. W00d!

dev tun
resolv-retry 16

txqueuelen 686
# expanded packet queue plane, to improve throughput on high-capacity sessions

sndbuf size 1655368
rcvbuf size 1655368
# increase pre-ring packet buffering cache, to improve high-throughput session performance

# randomizes selection of connection profile from list below, for redundancy against...
# DNS blacklisting-based session blocking attacks

remote 443 udp

remote 443 udp

remote 443 udp

remote 443 udp

comp-lzo no
# specifies refusal of link-layer compression defaults
# we prefer compression be handled elsewhere in the OSI layers
# see forum for ongoing discussion -

# runs client-side "down" script prior to shutdown, to help minimise risk...
# of session termination packet leakage

# allows client to pull DNS names from server
# we don't use but may in future leakblock integration

explicit-exit-notify 3
# attempts to notify exit node when client session is terminated
# strengthens MiTM protections for orphan sessions

hand-window 37
# specified duration (in seconds) to wait for the session handshake to complete
# a renegotiation taking longer than this has a problem, & should be aborted

mssfix 1400
# congruent with server-side --fragment directive

# passes up, via bootstrapped TLS, SHA512 hashed token value to authenticate to darknet

# auth-retry interact
# 'interact' is an experimental parameter not yet in our production build.

ca ca.crt
# specification & location of server-verification PKI materials
# for details, see


ns-cert-type server
# requires TLS-level confirmation of categorical state of server-side certificate for MiTM hardening.

auth SHA512
# data channel HMAC generation
# heavy processor load from this parameter, but the benefit is big gains in packet-level...
# integrity checks, & protection against packet injections / MiTM attack vectors

cipher AES-256-CBC
# data channel stream cipher methodology
# we are actively testing CBC alternatives & will deploy once well-tested...
# cipher libraries support our choice - AES-GCM is looking good currently

replay-window 128 30
# settings which determine when to throw out UDP datagrams that are out of order...
# either temporally or via sequence number

# implements 'perfect forward secrecy' via TLS 1.x & its ephemeral Diffie-Hellman...
# see our forum for extensive discussion of ECDHE v. DHE & tradeoffs wrt ECC curve choice

key-method 2
# specification of entropy source to be used in initial generation of TLS keys as part of session bootstrap

log devnull.txt
verb 0
mute 1
# sets logging verbosity client-side, by default, to zero
# no logs kept locally of connections - this can be changed...
# if you'd like to see more details of connection initiation & negotiation
Thank you,


EDIT: removed out-of-date config file
- cryptostorm_support



Post by cryptostorm_admin » Sat Jan 17, 2015 6:32 pm

{archival version of the original post in this thread, now edited to bring current ~admin}
We are currently in process of making some changes and upgrades to the network-side server resources that support our Eastern United States exitnode cluster. Our cluster is anchored by a machine in Virginia, but that DC has been having some issues and we're striping in additional capacity to handle demand for that geographic region.

Under cryptostorm's Hostname Assignment Framework, exitnode clusters are (or more properly, will be - we are in the midst of a full upgrade to HAF compliance, and many pre-HAF associations can still be found in existing configuration files and so forth) are generally identified based on the city in which they are located. This provides a useful balance between geographic specificity, and needless profusions of micro-precise cluster locations that simply make things complex for members.

However in the case of the USA, we're moving towards more of a regional model. We will have West, Central, and Eastern US clusters - thus folks can choose a cluster and not need to lock down to excess detail. Thus far, only the USA has looked to us like it is best served by this approach, although in the future that may change.

Thus, we've taken the opportunity provided by the issues relating to our Virginia-based node (chili) to step forward to this regional model. The first of our central USA nodes - - will be the first node to embody this evolved approach to cluster-based resource management. Next will come machines to establish a proper Eastern USA cluster, and an expansion of our existing West coast USA presence to ensure redundant network capacity.

To smooth the transition, within the next 24 hours HAF entries associated with chili (and the underlying instances they represent) will see their respective HAF IP-mappings roll forward to the new Central node, mishigami. In terms of functional efficacy - pingtimes, essentially - the geographic shift from Virginia to Chicago is quite small. Still, we wanted to ensure that folks who watch such matters closely understand our thinking in how this is being rolled into production.

Inevitably, there's a bit of turbulence as we move our legacy, pre-HAF structures into full compliance with the HAF model. We are really confident that the benefits provided by consistent HAF mappings vastly outweigh the short-term bumps along the way. And, of course, we're always open to suggestions and critique as we move forward - this is, above all else, a collaborative project.

Already we've been asked as few times about the naming of our new node. Here's a bit of backstory:

Thank you,

~ cryptostorm_team
fixed missing quote tag ~parityboy


Re: US-central exitnode cluster: | updates & "crazy shit"

Post by Guest » Wed Jan 21, 2015 5:55 am

log devnull.txt
umm- won't that just create a text file named devnull.txt and log to it? would the 'verb 0' stop the loging- man says the command will 'supercede'?
did you mean, log /dev/null? seams it wouldn't be nessasary with the verb 0 setting?

man openvpn:

--log file
Output logging messages to file, including output to std‐
out/stderr which is generated by called scripts. If file
already exists it will be truncated. This option takes effect
immediately when it is parsed in the command line and will
supercede syslog output if --daemon or --inetd is also speci‐
fied. This option is persistent over the entire course of an
OpenVPN instantiation and will not be reset by SIGHUP, SIGUSR1,
or --ping-restart.

Note that on Windows, when OpenVPN is started as a service, log‐
ging occurs by default without the need to specify this option.

User avatar
Posts: 612
Joined: Sun Dec 16, 2012 6:34 am


Post by Pattern_Juggled » Wed Jan 21, 2015 6:23 am

Guest wrote:
log devnull.txt
umm- won't that just create a text file named devnull.txt and log to it? would the 'verb 0' stop the loging- man says the command will 'supercede'?
did you mean, log /dev/null? seams it wouldn't be nessasary with the verb 0 setting?.
Yah, it's kind of an inside joke: devnull.txt... perhaps less funny in the light of day than during long development sessions. :-)

And, yes, verbosity 0 supersedes... we could name the file to which the logs aren't written anything we want. The devnull.txt name has just become something of a tradition with us, fwiw.


~ pj
...just a scatterbrained network topologist & crypto systems architect……… ҉҉҉

[list]✨ ✨ ✨[/list]
pj@ðëëþ.bekeybase pgpmit pgpðørkßöt-on-consolegit 'er github