protocol obfuscation: Dust, mangle, etc.

Looking for a bit more than customer support, and want to learn more about what cryptostorm is , what we've been announcing lately, and how the cryptostorm network makes the magic? This is a great place to start, so make yourself at home!

protocol obfuscation: Dust, mangle, etc.

Post by cryptostorm_dev » Tue Jan 28, 2014 9:52 pm

{direct link:}

Dust: A Censorship-Resistant Internet Transport Protocol
Brandon Wiley | The University of Texas at Austin School of Information


Dust is an Internet protocol designed to resist a number of attacks currently in active use to censor Internet communication. While adherence to the theoretical maxims of cryptographic security are observed where possible, the focus of Dust is on real solutions to real attacks.

Use Case

The use case for Dust assumes that there is an area of the Internet surrounded by a packet filtering firewall. All normal Internet traffic passing between the inside and outside of the zone is filtered using standard packet filtering techniques, disallowing the transmission of certain banned content. It is also assumed that there is a secure, but perhaps expensive and high latency, channel into the filtered zone. This channel is considered to be out of band, i.e. it is not over the Internet (phone, letter, messenger, etc.). Given this scenario, Dust creates an inexpensive, unfiltered channel from inside the firewall to outside.

Packet Filtering Techniques

The packet filtering techniques currently in use attempt to keep as little state as possible in order to be scalable. Filtering can happen either at the individual packet level (drop banned packets) or at the stream level (block or throttle streams containing banned packets). For stream level filtering it is common to sample only the initial packets of the stream, or to do random statistical sampling of packets. Filters do not keep persistent state about streams other than whether they have been marked as banned. Technique for defeating filtering can therefore concentrate on not sending packets which will be marked as banned.

There are two general classificiations of techniques for determining of a whether a packet is banned. Shallow packet inspection (SPI) uses just the headers of the packet. This is less expensive because the headers need to be examined anyway in order to route the packet. Deep packet inspection (DPI) simply refers to examining the packet contents as well as the headers. DPI used to be considered too expensive to be practical, but is now in widespread use by some filters.

SPI techniques in active use for marking packets are as follows: source IP and port, destination IP and port, and packet length.
DPI techniques in active use for marking packets are as follows: examing packet for connection headers and handshakes of known protocols, examining packet contents for banned static strings, and examining packet contents for banned string patterns.

How Dust Circumvents Filters

Dust is an Internet protocol used to send packets which defeat the various filtering techniques currently in use.

IP and port blacklists - This is outside of the scope of the protocol. Communication over the Internet requires a destination IP and port which has not been blacklisted by the filter. For the source IP and port, packet spoofing could be used. For more information on how to circumvent IP blacklists, see the Arcadia paper.

Packet length - Dust packets have randomized lengths.

Connection headers and handshakes - Unlike SSH and SSL (both now filtered but some filters), Dust contains no plaintext handshake.
Static strings and string patterns - Dust packets are encrypted, so the contents cannot be examined.

In a sense, each Dust packet is unique in that the contents are entirely encrypted using a random initialization vector and the length is randomized as well. Dust packets can therefore not be classified by techniques that examines static characteristics of a packet.



The Dust protocol is a unidirectional UDP-based protocol. A Dust conversation consists of a single intro packet followed by any number of data packets. The minimal Dust conversation therefore consists of two packets: one intro packet, and one data packet. While the use of a unidirectional, unreliable protocol is unconventional, it was chosen because Dust has one purpose: to get data past filters. Special care must be taken when adding additional features to the protocol as every packet which is added provides another opportunity for information to be leaked which could be used in fingerprinting the protocol. Future work will consider ways to add additional desirable features to the protocol without compromising its core purpose.

The Key Exchange

Protocols such as SSL and SSH initiate a public key exchange using a plaintext handshake. They are therefore susceptible to protocol fingerprinting and filtering. Dust also requires a public key exchange, but not utilize a plaintext handshake. Instead, Dust uses a novel half in-band, half out-of-band key exchange protocol..

In order to accept a connection from an unknown host, a Dust server must first complete a key exchange with the client. The Dust server first creates an id-secret pair, which is stored securely on the server. The server then creates an invite, which contains the server's IP, port, public key, and the id-secret pair. The invite can be optionally encrypted using a password.

The server operator then distributes the invite and optional password to the client out-of-band. This constitutes the first half of the public key exchange.

The client uses the IP and port information from the invite to send an intro packet to the server. The intro packet is encrypted with the secret from the invite and contains the public key of the client. The id portion of the id-secret pair is prepended unencrypted to the encrypted packet. The server retrieves the id and uses it to lookup the associated secret. It uses the secret to decrypt the packet and retrieves the public key of the client. It adds the public key to its list of known hosts, associated with the IP and port from which the intro packet was sent. This completes the second phase of the public key exchange.

Sending Data

Once the key exchange has taken place, data packets can be sent. To send a data packet, the client generates an encryption key using its public key and the public key of the server. The client also generates a random initialization vector. This encryption key and the initialization vector are used to encrypt the data packet. The intialization vector is prepended, unencrypted, to the encrypted data packet. A random number of random bytes are appended to the encrypted data packet in order to give it a random length.

When the server receives a packet, it looks at the source IP address and port to determine the identity of the client. It then looks to see if it has a public key stored for that IP address and port. If so, it assumes the packet to be a data packet and decodes it. Otherwise, it assumes it to be an intro packet and decodes it as specified in the Key Exchange section. To decode a data packet, an encryptionl key is generated using the server's public key and the client's public key. The initialization vector is taken from the front of the packet. The remaining packet is decrypted. The extra random bytes that were added by the client are discarded. The result is the original unecrypted data.


A key consideration in the Dust protocol is that the only information in a Dust packet should always be effectively random to an observer. There are only two types of information in a Dust packet: encrypted (should appear random) and single-use random bytes (actually is random). The apparent randomness is essential to the protocol's ability to avoid detection. Therefore, care should be taken to use a good random number generator and to never reuse random bytes. For instance, the id from an invite id-secret pair, which is sent in an intro packet is intended for a single use and should never be reused. Similarly, the initialization vector used in data packets should always be randomly generated and never reused between data packets.

Packet Format

There are three types of Dust packets: invite, intro, and data packets. All three types of packets build upon the basic Dust packet format.

Dust Packet

Any dust packet has the following format:


dustPacket = mac(key, ciphertext) ciphertext padding
ciphertext: iv encrypt(key, iv, payload)
payload = timestamp length(data) length(padding) data


32 bytes - mac(key, ciphertext)
16 bytes - iv
4 bytes - timestamp
2 bytes - length(data)
1 byte - length(padding)
variable - data
0-31 bytes - padding (random)

mac(key, ciphertext) is the message authentication code of the ciphertext, used to ensure that the ciphertext is not tampered with or corrupted
iv is the initialization vector, random bytes used in the encryption to ensure that encrypting the same payload twice will yield a different packet
encrypt(key, iv, payload) encrypts the payload with the key and iv
padding is random bytes used to randomize the length of the packet
timestamp is a timestamp used to ensure that old packets are not replayed later by an attacker
length(data) is the length of the data, used to distinguish the actual data from the padding
length(padding) is the length of the padding, used to determine the total length of the packet when multiple packets are in the same byte stream
data is the actual data of the packet

Packet Chaining

Multiple Dust packets can be chained together inside of a single UDP packet. After decoding a Dust packet, the length of the headers, data, and padding should be summed to determine the total length of the Dust packet. If this is less than the length of the UDP packet then this is a packet chain and the additional packet should be parsed as well. As many packets can be chained as can fit inside a UDP packet.

The practical reason for allowing packet chaining is so that, if the message is small enough, a single UDP packet can be set carrying both an intro packet and a data packet. The smallest Dust conversation is therefore one packet. The other useful effect of packet chaining is that it allows for Dust packets to be transported over other channels, such as TCP.

Invite Packet

Invite packets are not sent directly over the Internet between the client and server like Intro and Data packets. They are sent out-of-band and so are in an ASCII-compatible format for ease of use in email, IM, or for printing. However, they do reuse the Data packet format as a way to securely encode them for transportation. Invite packets use a hashed password for encryption. After the packet is encrypted, it's converted to an ASCII-friendly hex encoding for transportation. Multiple invites can be sent in the same file by separating them with newlines.

A multiple invite package containing three invites would look like this:

Code: Select all

Note that you cannot decode the invites without the password.

An invite packet has the following format:


invitePacket = salt dataPacket(pbkdf(password, salt, iterations), inviteMessage)
inviteMessage = pubkey flags ip port id secret


32 bytes - salt
... # data packet headers precede data
32 bytes - pubkey
1 byte - flags
16 bytes - ip
2 bytes - port
16 bytes - id
32 bytes - secret
... # data packet padding follows data

salt is random bytes used in the PBKDF function
pbkdf(password, salt) is a Password-Based Key Derivation Function which produces a key from a password and random salt
pubkey is the public key of the server
flags is 8 bits of flags for different protocol options. The flag bits are defined as followed:

* 0 - IP version: 0 is IPv4, 1 is IPv6
* 1 - transport protocol: 0 is UDP, 1 is TCP
* 2-7 - reserved for future flags

ip is the IP of the server, either IPv4 or IPv6 depending on the IP version flag. If IPv4, only the first 4 bytes are used and the renaming 12 are zeros
port is the port that the server is listening on
id is an id that can be sent in an intro packet
secret is the corresponding encryption key to use to encrypt an intro packet with a given id

Intro Packets

An intro packet has the following format:


introPacket = identifier dustPacket(secret, sender-pubkey)


16 bytes - identifier
variable - dustPacket (sender-pubkey is 32 bytes)
identifier is the first part of the identifier-secret pair from an invite
dustPacket(secret, sender-pubkey) is a Dust packet, using secret as the encryption key and sender-pubkey as the data
secret is the second part of the identifier-secret pair from an invite
sender-pubkey is the public key of the sender

Data Packet

A data packet has the following format:


dataPacket = dustPacket(sessionKey, data)


variable - dustPacket (sessionKey is 32 bytes)
dustPacket(sessionKey, data) is a Dust packet, using sessionKey as the encryption key
sessionKey is the encryption key used for the Dust packet encryption. It's defined hash(session(sender-privkey, receiver-pubkey))


The basic Dust protocol is designed to counteract current Internet filters techniques. With extensions, the Dust protocol can be expanded to deal with a larger range of filtering situations.

Dust Lite

The Dust protocol creates a secure channel. This is a desirable quality because it means that Dust packets cannot be decoded by listeners and therefore cannot be flagged by filters. However, there are some situations where a secure channel is not necessary. In a situation where the goal is to bypass filters in the short-term only, the lightweight version of the Dust Lite can be used.

Dust Lite uses only Data packets, skipping the Invite and Intro steps. Dust Lite packets differ from normal Data packets in how the sessionKey is generated. A Dust Lite sessionKey is defined as xor(hash(sender-address), hash(receiver-address)) where sender-address and receiver-address are formatted in the following ways:

* for an IPv4 address: x.x.x.x:port
* for an IPv6 address: [x:x:x:x:x]:port

Since Dust List packets are encrypted and padded like any Dust packet, they will bypass any filters that are not specifically looking for Dust Lite packets. Additionally, the filter will need to calculate different sessionKey values for every pair of communicating addresses. This will increase the cost of filtering Dust Lite packets as the number of users increases. Unfortunately, once the sessionKey has been computed traffic can be blocked as well as forged. At that point, it's time to switch to the full Dust protocol.


The mechanics and format of Dust are presented in a general manner such that any suitable public key cryptography system and cipher could be used. However, actual implementation requires specific choices and choices for implementation of Dust have been made which are small, fast, and have libraries for which it is easy to create bindings.

* Public key system: Curve25519
* Session key exchange: Curve25519 implementation of ECDH
* MAC: Skein-MAC(key, data)
* Internal state: 256 bits
* Output size: 256 bits
* MAC Input: key
* Message Input: data
* Personalization Input: "1978-10-26 Dust/MAC"
* Hash: Skein(data)
* Internal state: 256 bits
* Output size: 256 bits
* Message Input: data
* Personalization Input: "1978-10-26 Dust/hash"
* PBKDF: Skein-PBKDF(password, salt)
* Internal state: 256 bits
* Output size: 256 bits
* Iterations: 13000
* Message Input: (salt+password)*iterations
* Personalization Input: "1978-10-26 Dust/PBKDF"
* PRNG: Skein-PRNG.init(seed); Skein-PRNG.getBytes(n)
* Internal state: 256 bits
* Output size: SEED_SIZE+n
* init: set S to seed
* getBytes: Skein(S) -» R; S=R[0..SEED_SIZE]; bytes=R[SEED_SIZE+1...]; return bytes;
* SEED_SIZE: 256 bits
* Personalization Input: "1978-10-26 Dust/PRNG"
* Cipher: Skein-Cipher(key, iv, data)
* Internal state: 256 bits
* Output size: len(data)
* Cipher mode: OFB
* IV size: 128 bits
* Key size: 256 bits
* Block size: 256 bits
* Personalization Input: "1978-10-26 Dust/cipher"

Open Questions
* Justification of cipher algorithm
* Justification of hash algorithm
* Failure modes and recovery
* When to discard key-IP bindings
* Appropriate windows for timestamps

Recent Changes
* Switched invite format to use data packets as a wrapper
* Password-based encryption of invites made mandatory
* Added flags byte to invite packet with IP version and protocol flags defined
* Added newline-delimited invite package format
* Switched hash from skein-256-256 to skein-512-256
* Use of Skein-PRNG as PRNG algorithm
* Switched from using skein512-256 hash for passwords to using Skein-PBKDF with random salt and 13000 iterations
* Added explicit paddingLength field
* Specified chainable packet semantics
* Switched cipher from Threefish to Skein-Cipher
* Removed filler field
* Added personalization
* Changed cipher mode to OFB
* Changed MAC from MAC of plaintext to MAC of ciphertext and IV, moved MAC before IV
* * TCP compatibility
* Switched hash from skein-512-256 to skein-256-256 for Java compatiblity
* Java port

Future Work
* Bidirectional extensions: acks and key exchange for forward secrecy
* Steganographic extension
* C port
* Key rotation
* Separate cipher and MAC keys
* Certificates


Special thanks goes to Drake Wilson for all of his excellent feedback regarding the Dust mechanics and algorithms, including the suggestion of both Skein and Curve25519, as well as the initial idea for Dust Lite.

Thanks to Greg Hazel for his contributions on the original version of the protocol and his continuing suggestions.

User avatar
Posts: 612
Joined: Sun Dec 16, 2012 6:34 am

Re: protocol obfuscation: Dust, mangle, etc.

Post by Pattern_Juggled » Sat Nov 22, 2014 5:35 pm

Bumping & pinning this thread, as it's a dev priority that's now becoming tangible and, as such, a good place to work out the approach we're going to take.

Also, I have this dream of using --mangle at the NIC to do amazing things with subtle attack surfaces presented by the as-implemented details of the TCP stack itself... it may well be a totally dumb idea, but so far I can't quite convince myself of that. So I'll put my sketchy ideas on the table here, and smart folks can tell us what's wrong with them.


  • ~ pj


Re: protocol obfuscation: Dust, mangle, etc.

Post by cryptostorm_dev » Thu Dec 11, 2014 4:08 am

Evading Internet Censorship

This research project by Brandon Wiley -- the tool is called "Dust" -- looks really interesting. Here's the description of his Defcon talk:
Abstract: The greatest danger to free speech on the Internet today is filtering of traffic using protocol fingerprinting. Protocols such as SSL, Tor, BitTorrent, and VPNs are being summarily blocked, regardless of their legal and ethical uses. Fortunately, it is possible to bypass this filtering by reencoding traffic into a form which cannot be correctly fingerprinted by the filtering hardware. I will be presenting a tool called Dust which provides an engine for reencoding traffic into a variety of forms. By developing a good model of how filtering hardware differentiates traffic into different protocols, a profile can be created which allows Dust to reencode arbitrary traffic to bypass the filters.

Dust is different than other approaches because it is not simply another obfuscated protocol. It is an engine which can encode traffic according to the given specifications. As the filters change their algorithms for protocol detection, rather than developing a new protocol, Dust can just be reconfigured to use different parameters. In fact, Dust can be automatically reconfigured using examples of what traffic is blocked and what traffic gets through. Using machine learning a new profile is created which will reencode traffic so that it resembles that which gets through and not that which is blocked. Dust has been created with the goal of defeating real filtering hardware currently deployed for the purpose of censoring free speech on the Internet. In this talk I will discuss how the real filtering hardware work and how to effectively defeat it.

Brandon Wiley

Brandon Wiley (@blanu) is a peer-to-peer pioneer who creates tools to circumvent Internet censorship. In 1999 he co-founded the Freenet project to create a censorship-resistant publishing platform. He is also known for the Curious Yellow superworm design. When working for BitTorrent, Inc. he was given the difficult task of trying to reason with the Internet service providers that were engaging in BitTorrent throttling. More recently he has been working for the Tor project on their next generation blocking-resistant protocols such as pyobfsproxy and obfs3. He is currently in the final stages of his PhD, where he is studying all of the most popular Deep Packet Inspection hardware and figuring out how to defeat it. His interests include Bayesian statistics, polymorphic encodings, and chiptune music.

User avatar
Posts: 612
Joined: Sun Dec 16, 2012 6:34 am

Re: protocol obfuscation: Dust, mangle, etc.

Post by Pattern_Juggled » Thu Dec 11, 2014 4:16 am

Dust: A Blocking-Resistant Internet Transport Protocol
Brandon Wiley
School of Information, University of Texas at Austin


Censorship of free speech on the Internet has been an increasing problem as the methods of identifying and filtering traffic have become more sophisticated. A number of approaches to counteract Internet censorship have been implemented, from censorship-resistant publishing systems to anonymizing proxies. While existing systems provide protection against some attacks, they fail to provide resistance to blocking of the transport protocol by modern techniques such as Deep Packet Inspection. Dust is proposed as a blocking-resistant Internet protocol designed to be used in conjunction with existing systems to add resistance to a number of attacks currently in active use to censor Internet communication.
(106.98 KiB) Downloaded 1414 times

User avatar
Posts: 612
Joined: Sun Dec 16, 2012 6:34 am

Using git to circumvent censorship of access to the Tor network

Post by Pattern_Juggled » Thu Dec 11, 2014 5:35 am

Using git to circumvent censorship of access to the Tor network
July 24, 2013

Bjorgvin Ragnarsson ¨
Pieter Westein

Code: Select all


We present git-pull-tor, a pluggable transport for Tor that obfuscates the network
traffic to resemble usage of the Git version control system. We compare the download
speed when using git-pull-tor to downloading files without an obfuscation proxy
and find the current version to have considerable performance issues. We find the
strength of the system to be its accuracy in imitating Git, since it uses the Git
program to perform the obfuscation. The weakness with regard to traffic analysis is
that is produces traffic flow patterns that are unusual for Git.
(362.96 KiB) Downloaded 1458 times

User avatar
Posts: 612
Joined: Sun Dec 16, 2012 6:34 am

"Candidate Indistinguishability Obfuscation and Functional Encryption for all circuits"

Post by Pattern_Juggled » Thu Jan 15, 2015 5:53 am

...just a scatterbrained network topologist & crypto systems architect……… ҉҉҉

[list]✨ ✨ ✨[/list]
pj@ðëëþ.bekeybase pgpmit pgpðørkßöt-on-consolegit 'er github

Posts: 2
Joined: Sat Jan 16, 2016 12:24 pm

Re: protocol obfuscation: Dust, mangle, etc.

Post by subatomic » Sat Jan 16, 2016 12:32 pm

This seems to be lingering for awhile. Why not bundle obfsproxy with the cstorm client to get obfuscation available more quickly (like the TBB does) then add Dust at a later date?