In the last month or so, we've been drawn into a series of overlapping projects involving deep structural problems in the VPN service market. Beginning with the webRTC leak, running through as-yet unpublished findings that many "leak testing" websites are aggressively gathering data during tests to pass to adware schemes, and directly into the superfish/komida investigations in which ssl session "kneecapper" programs have been discovered in wide deployment, it has been a busy time for many researchers.
For us, it has also been a sobering experience. As we shared recently in a somewhat frazzled-sounding series of short statements on twitter, we have been presented with data that remove many of our previous assumptions about both https-based "secure" web browsing, and the fundamental nature of the VPN industry (we don't like that term - "VPN service" - and rarely if ever use it to describe our service, but for now we're just using it to get past distractions in this post). Most all of what we've found isn't good.
In short, we've seen - and collected - data that (to us) document the practice of VPN services including trojans, adware, keyloggers, and other overt malware in their closed-source client installation packages. This started with a deep dive into one particular situation, and then widened exponentially as we began checking other (even more clearly) ugly installer packages. Review of network activity during these installations ("pcaps") confirms the installation of specific binaries that are known by malware researchers to be, in a word, dirty code. Some of it is hidden with incredible cleverness; some is right there for anyone to see who looks. But it's there.
This has left us with something of a conundrum at cryptostorm. On the one hand, ignoring these findings and simply going about our business is very tempting: as we've spent weeks helping with these various community-based research projects, we've let many things slide a bit in our own operations (mostly marketing and public outreach, and never security matters, to be clear). We've been lax in returning correspondence, and we've generally been a bit distracted. That's just reality: as a small, focussed team there's no way we can do this sort of highly intensive forensic analysis without taking that time from other tasks.
Further, to be blunt, the whole thing feels wrong. And that matters to us.
By the end of this week, it had devolved into a process of cycling through VPN service installers, unpacking and scanning them to see if there was badness inside. Often there was, requiring deeper analysis to confirm. And then what? The idea of publishing "hit pieces" including our data, as one-offs, has left our team cold. It's not who we are - we do best when striving for improvements, not when attacking others. But, to sit on these data is also not possible: these packages are being installed by many people trusting in the integrity of the VPN industry, and not thinking for a minute that they're opening themselves up to serious security problems as a result of installer-dropped malware.
By the end of the work week, we'd talked as a team and agreed to sit with the question for a bit.
Now, we feel we've come to a constructive path forward: cleanVPN.org
Rather than go hunting for dirty VPN installers and services, we choose to create a space in which clean operations can be highlighted, rewarded, and covered by ongoing independent review of their software integrity. Reward the good, and the bad will become less rewarding. This is the choice we as a team, at cryptostorm, have made.
To be clear, we see no benefit in creating yet another "VPN review" list. There are already far too many with far too little to offer in constructive results (indeed, often they are overtly evil themselves), in addition to being utterly subjective in nature. Nor do we seek to impose our own standards, as cryptostorm, on what is considered good or bad. These are questions for other venues and other discussions.
Rather, cleanVPN will focus on overt, independently-verifiable markers of code and service integrity including:
- - scans of all currently- and previously-distributed pre-compiled installers to verify absence of any known malware activity
- publication of full source code for all client-side applications and installers
- independent builds from source to verify that source is in fact resulting in binaries as distributed
- independent publication of hashed fingerprints of compiled installers, to ensure fake/infected versions can be spotted and removed
- test connections with default settings from distributed installers, to confirm actual VPN sessions result
- monitoring of test connections to confirm no proxy- or hijack-style out-of-band traffic is taking place
- websites are free of any aggressive, script-based adware or data grabbing schemes
- DNS records are properly propagates, and subhosts/hostnames/vhosts are well-enumerated (more on that in a separate post)
- provisioning of identity-validated https-based website service to ensure sites are not being MiTM'd themselves during installer download
We will help to prime the pump of cleanVPN.org by contributing our findings thus far, in raw form, via a newly-initialised github repository. We'll loop in any and all researchers who choose to make commits to that repo, which is intended as an open public resource for data gathering, analysis, and publication. We hope to see the project graduate out from our forum here, and into an standalone effort... but we didn't want to stall the process itself by waiting for those steps before getting things started. So we'll do an iterative rollout, and seek to hand off as much of the infrastructure element to the cleanVPN project as we can, as quick as it can be done.
Our hope is that the cleanVPN project can produce a published list of "known-clean" providers, who themselves can then make use of this seal of approval in their own public pronouncements. We leave the details of this process to the community for development as things progress from here, but it shouldn't be a terribly complex decision given what we've seen in the analyses so far: the dirty ones are dirty top to bottom, and the clean ones are clean from the first scan to the last. There seems to be little middle ground.
We're not stepping away from this work, as a team. Yes, we've considered it. Being in the maelstrom of public statements that "most VPN services install malware" (not our words specifically, but inevitably that will be the tl;dr version in some places) opens us up to attacks on our team, our service, and our operations far beyond the normal levels: the dirty shops, of course, know they're dirty and if they could only shut us up, their dirty (and, one infers, quite highly profitable) operations can continue unhindered. We do expect to have smears and attacks launched our way - it's happened before any time we've gone anywhere near these issues, and surely it will happen here.
But that is not reason enough to walk away from this.
Once the sunlight is allowed to shine on the darker corners of the VPN industry, we hope that the old practices of dirty smears and underhanded personal attacks will be left in the past. We prefer to see a future where network security service competes based on quality, competence, and features... not based on who bribed the review sites best, or which can cram the most malware into installers without them crashing entirely. Further, we see our "competition" as bareback network access: the vast majority of the world's billions of online citizens who have no network security whatsoever. That's where we focus our own efforts, not on trying to one-up someone else in our field.
That is our choice, as a team.
We hope other researchers, both within the VPN industry and crucially from beyond its stifled confines in the broader security tech community, will share of their time and expertise to help get cleanVPN off on a good strack. This project requires such generosity to succeed, and it's worth it. People deserve to have confidence that well-known VPN services aren't installing backdoors in their computers when they pay for privacy service, and everyone deserves to have a wide selection of network privacy tools from which to choose. These things matter, and we need to create a playing field on which good actions and good deeds are rewarded.
Reward the good and the temptation to do bad becomes less. Simple, but we feel effective.
Let's do it.
- ~ cryptostorm team