ZeroTier 1.6.0

Adam Ierymenko

Today ZeroTier is proud to release version 1.6.0, a major new release that contains a number of improvements and new features that are mostly back-ported from our still-in-progress version 2.0 tree.

Here’s a quick summary:

  • **Multipath engine **allowing multiple network connections to be used simultaneously. See the knowledge base for instructions.
  • **DNS **configuration management, a long-requested feature that will be adored by any system administrator using Active Directory or who wants to run any kind of internal DNS server to resolve names on their virtual network. Endpoint nodes must allow this on a per-network basis for security reasons. (Currently only supported on MacOS, Windows, Android, and iOS.)
  • **Faster and more secure encryption **using our AES-GMAC-SIV mode of operation (see below for details).
  • **Linux performance improvements **on multi-core systems, sometimes yielding up to 30% better throughput under high load.
  • **Apple Silicon **(MacOS ARM64) native support via universal binaries.
  • **Lots of bug fixes **in things like managed routes, MacOS network setup and tear-down, problems that caused the service to hang on exit, and more.

Read on for details about the big stuff.

DNS Management

DNS integration is a long-requested feature. In 1.6 it is now possible to set DNS servers at the network controller that will be applied to the host upon joining the network, provided the host approves this by allowing managed DNS. This will allow networks to push things like Active Directory or other intranet DNS servers to members of a network.

This still doesn’t let you simply address hosts by their name as configured at the controller, but we’re aware of this. We plan on adding a feature to allow the controller itself to be a DNS server too if one desires in a future ZeroTier version (likely post-2.0).

This feature is currently only supported on Windows, MacOS, Android, and iOS. Linux support is forthcoming but may be limited to common Linux DNS resolver configurations such as those found in Debian and CentOS/RHEL.

Multipath

After over a year of work by the very talented Joseph Henry, we are proud to merge a beta version of our multipath engine into ZeroTier 1.6! It’s off by default and is still subject to change, but we wanted to get it into your hands so you can see how it works in the real world.

Multipath means that ZeroTier is now able (with some configuration prodding) to use more than one Internet connection concurrently when communicating with other nodes. The concepts and terminology of our multipath implementation are based on the Linux Ethernet bonding driver as this is something we think many of you might have seen.

The multipath section of the normal manual is out of date, so for now (as of November 2020) please see the multipath page in our knowledge base for documentation.

ZeroTier multipath supports the following modes of operation:

  • active-backup, which is like the current behavior but with faster fail-over. In 2.0 we plan on eliminating the old path selection code and making this the default.
  • **broadcast **sends all packets over all paths, which provides better reliability and lowest possible latency at the cost of wasted overall bandwidth. (Packets are de-duplicated by the receiver.)
  • **balance-rr **alternates between network links and works best if both links are of similar speed and quality. It will stop using a dead link quickly, so it also offers redundancy and fast fail-over.
  • **balance-xor **uses an XOR of some select fields of each packet to decide which link to use, and on link failure will rapidly move traffic to a working link.
  • **balance-aware **is the most advanced mode and is one you won’t find in the Linux bonding driver documentation. This mode dynamically measures link quality and proportionally allocates traffic to maximize overall throughput and efficiency.

We are immensely proud of this work. It’s ground-breaking and we are not aware of anyone else who has anything like it outside of proprietary (and very expensive) SD-WAN box vendors. Please test it out and let us know how it fares in practice!

AES-GMAC-SIV

This section assumes some familiarity with block cipher modes of operation, stream ciphers, authenticated encryption, and other encryption concepts. Non-crypto-nerds are welcome to just skim or skip this part.

1.6 and later versions will now communicate with one another using our new AES-GMAC-SIV encryption mode, which replaces the old Salsa20/Poly1305 authenticated encryption used in previous versions. AES-GMAC-SIV is a slightly novel mode of cipher operation that we will explain below, and yes we did retain the help and peer review of professional cryptographers in designing and implementing it.

So why go through all the bother of a novel construction?

One of the hurdles on the road to widespread deployment in the enterprise is FIPS, NIST, and NSA certification. While Daniel Bernstein’s latin dance ciphers (Salsa and ChaCha) are fast, secure, and generally quite respectable, they are not yet in any of those certification specs. The poly1305 authenticated encryption mode isn’t either. That means we can’t use them if we want adoption in industries like finance, government, military, anything subject to ITAR regulations, and so on. We are aware that some of these constructs are under NIST review, but NIST moves at a glacial pace and we don’t feel like waiting years when we have potential large customers asking for FIPS right now.

The problem is that the usual FIPS-approved mode of choice for authenticated encryption, AES-GCM, is weaker in at least one important way than Salsa/ChaCha paired with Poly1305.

For nearly all stream ciphers and most other modes of operation, each message to be encrypted must include something called a nonce. Nonce is an abbreviation of number used once, and is just that: a number that should only be used once and never again. If a nonce is used more than once, the result is usually that both messages that were sent with the colliding nonce can now be decrypted and read. That’s not good, but AES-GCM is even worse than this. With AES-GCM it also becomes possible to attack the GMAC (GCM) portion of the construction, which is the part responsible for ensuring that messages were not modified. If an attacker sees a duplicated nonce with AES-GCM they can not only read the colliding messages but attack the authentication method and potentially *forge *messages, opening you up to all kinds of attacks including “oracle” attacks that can result in compromise of the secret key.

ZeroTier uses a 64-bit nonce combined with a message-dependent key randomization technique in its Salsa/Poly1305 mode, but if you transfer terabytes of data with the exact same key it’s still possible to have a duplicate nonce. Duplicating a nonce with Salsa/Poly1305 is not as catastrophic as it is with AES-GCM, but it is still bad… and remember that because of FIPS we want to get away from latin dances.

There are a class of stream cipher modes that fare much better against nonce duplication. These are called SIV or synthetic IV modes. Synthetic IV modes still use a nonce, but they also take the content of the message into account when computing the *real *nonce used in stream encryption. In SIV modes the duplication of the nonce means very little. It only means that if one sends two copies of the same exact message that also happen to have the same exact nonce, the attacker can see that you duplicated a message… but that’s it. They can’t decrypt anything.

A SIV mode called AES-GCM-SIV based on the AES-GCM construction exists, but unfortunately the designers of this mode decided to slightly alter the GMAC message authentication construct. The change has no security implications. It’s just a minuscule optimization for common CPUs, but it has the side effect of rendering the GMAC construction used in this mode non-standard. AES-GCM-SIV can’t be FIPS, NIST, or NSA certified.

We were therefore faced with the choice between AES-GCM and doing something creative. We decided to do something creative. More specifically, we decided to try to smuggle better cryptography into FIPS than what FIPS normally allows.

The result is AES-GMAC-SIV. The AES-GMAC-SIV mode is built entirely out of standard and FIPS-certifiable building blocks: AES-256, GMAC (the message authentication part of AES-GCM), and the CTR (counter) mode of AES operation, and could therefore be described as merely a form of GMAC-authenticated AES-CTR. This would be entirely acceptable under FIPS. It could also be implemented using AES, GMAC, and CTR implementations in pre-existing FIPS-certified cryptographic libraries and APIs. (We have our own faster implementation, but that’s just for speed and because we hate dependencies.)

Here’s a simple block diagram of our take on a SIV mode for AES-CTR:

The source code to this mode can be found in the AES files in the node/ subfolder of the source tree.

We retained Trail of Bits to audit the design of ZeroTier version 2.0, and in the process they took a look at AES-GMAC-SIV. They proved that our AES-GMAC-SIV mode had *worst case *security bounds at least equivalent to those of AES-SIV, the original SIV mode proposed for AES, and that these bounds were within the requirements for AES-CTR under FIPS. The audit can be found here (PDF).

Other than bureaucratic compliance AES and GMAC (the MAC part of GCM) have another very practical advantage. The vast majority of 64-bit chips made in the past 10 years (and even longer for X86-64) have hardware AES acceleration. While Salsa/ChaCha are very fast, hardware based AES at least equals and usually exceeds their performance on chips that have such acceleration. Even if performance is more or less the same, hardware AES will usually consume less energy than ARX-type ciphers like Salsa/ChaCha that more heavily use the CPU’s arithmetic and logic units. The fastest ChaCha and Salsa implementations leverage wide vector modes like AVX2 and AVX512 and while these are fast they consume a lot of power and can even cause some chips to frequency throttle.

Getting all the way to FIPS and other certifications will require us to also replace Curve25519/Ed25519 in identities. This won’t happen until version 2.0, which supports a new “type 1” identity containing both a Curve25519/Ed25519 key pair and NIST P-384 key pair. The 25519 pair allows interoperation with older identities, while *both *are used for key agreement between newer nodes. By using both for key agreement those who are suspicious of the security of NIST P-384 can rest assured that the security of the link can be no worse than the stronger of the two algorithms as the secrets are hashed together. For FIPS documentation purposes we can just consider the Curve25519 shared secret a “salt” included in key derivation after NIST P-384 ECDH key agreement.

Getting all the way to FIPS and other certifications will require us to also replace Curve25519/Ed25519 in identities. This won’t happen until version 2.0, which supports a new “type 1” identity containing both a Curve25519/Ed25519 key pair and NIST P-384 key pair. The 25519 pair allows interoperation with older identities, while *both *are used for key agreement between newer nodes. By using both for key agreement those who are suspicious of the security of NIST P-384 can rest assured that the security of the link can be no worse than the stronger of the two algorithms as the secrets are hashed together. For FIPS documentation purposes we can just consider the Curve25519 shared secret a “salt” included in key derivation after NIST P-384 ECDH key agreement.