DISCLAIMER. This guide comes "as is". The author is no expert in the topic. The guide aims to take a practical viewpoint on conducting experiments, explaining enough for a developer to get started.

Please note: this guide assumes Linux as the operating system! (It was made with Ubuntu 20.04.4 LTS desktop.) However, the same tricks likely work in Windows too although some adaptation is necessary. The HiveMQ server version was 2021.3 community edition.

Introduction

This guide shows how to experiment with TLS encryption in HiveMQ MQTT server. For a quick experiment, local self-signed certificates are assumed, although the same principles should apply to any certificate. The most burden seemed to come not from HiveMQ but certificates and their setup in the Java environment.

Prerequisites

You should already have set up HiveMQ and ensured it works. For more information, please see this post.

TLS certificates and domain name

In this example, the TLS connection is made to the local machine. If you use an actual remote domain instead, you can skip the setup of the local domain name.

That is, there is no actual domain name, but our host is called myhost.local. The desired routing can be implemented by adding the following line into /etc/hosts:

127.0.0.1 myhost.local

Before setting up TLS, you should try if this custom routing works by connecting to HiveMQ server over myhost.local.

To get your certificates and learn the basics of TLS encryption, see this post. As you generate self-signed certificates, you should set myhost.local as the domain name.

Enabling TLS in the server

Keystore setup for the server

You must bundle the private key and certificate of the server into a Java keystore. Assuming your certificate is called server.crt and the private key server.key respectively, bundling (into PKCS #12) happens as follows:

openssl pkcs12 -export -name servercert -in server.crt -inkey server.key -out server.p12

Then, create a Java keystore from the PKCS #12 file:

keytool -importkeystore -srckeystore server.p12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype JKS

Next, copy the keystore to HiveMQ root folder. In our case, the folder is as below:

sudo cp keystore.jks /opt/hivemq-ce-2021.3/

Server configuration

Next, take TLS into use. Add the following tls-tcp-listener element into the listeners element in HiveMQ config file located at conf/config.xml in HiveMQ directory (this is based on https://www.hivemq.com/docs/hivemq/4.7/user-guide/security.html):

<listeners>
  ...
  <tls-tcp-listener>
    <port>8883</port>
    <bind-address>0.0.0.0</bind-address>
    <tls>
        <keystore>
            <path>keystore.jks</path>
            <password>(PUT YOUR KEYSTORE PASSWORD HERE)</password>
            <private-key-password>(PUT YOUR PRIVATE KEY PASSWORD HERE)</private-key-password>
        </keystore>
        <client-authentication-mode>NONE</client-authentication-mode>
    </tls>
  </tls-tcp-listener>
  ...
</listeners>

About the code above, please note that:

• Unencrypted access is still possible if you have kept the default TCP listener in the config
• Client authentication is NONE, which means that client certificates are not validated

Please note that HiveMQ server comes with configuration examples, including a TLS example. In the current version, these are located in the directory conf/examples.

Next, restart the server so that the changes take effect. In HiveMQ log (located in log in HiveMQ directory), you should see something as follows to indicate your configuration has succeeded:

2022-03-15 12:08:08,815 INFO  - Started TCP Listener with TLS on address 0.0.0.0 and on port 8883

Client connections with TLS

Make the client trust your certificate

This section assumes that:

• your client is implemented in Java and uses HiveMQ client libraries
• your certificates are self-signed
• you use Eclipse as the IDE

If your certificates aren't self-signed, you can likely skip the rest of this section.

To enable your client to trust the server certificate, you must install the certificate of your self-made certificate authority (CA) into the client machine. If you don't do this, you Java-based client would give an error, such as (this may depend on your Java version):

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

To convert the CA certificate into DER format (assuming the CA cert is called rootca.crt):

openssl x509 -inform PEM -in rootca.crt -outform DER -out rootca.der

Please note that you may have multiple Java installations in the machine. The commands in this section assume the following installation path, which resulted when the "default-jdk" was installed with Apt. Make sure to adapt the path in the following commands as needed! If the certificate doesn't work later, the potential reason is that you have chosen the keystore of another Java installation.

/usr/lib/jvm/default-java

Furthermore, the following assumes that you have preserved the default keystore password, which is "changeit".

Next, install the CA certificate into Java keystore as below. This produces a warning about the missing switch -cacerts, but so it did with other commands too and adding the proposed switch didn't change anything. Still, the command seemed to succeed even with this. The alias "petrica202203" is supposed to be unique (see https://security.stackexchange.com/questions/123944/what-is-the-purpose-role-of-the-alias-attribute-in-java-keystore-files), therefore a selected my name (as I was the CA) followed by year and month as the certificate will eventually expire.

sudo keytool -importcert -alias petrica202203 -keystore /usr/lib/jvm/default-java/lib/security/cacerts -storepass changeit -file rootca.der

After adding, your newly added certificate should appear last in the list as you give this command:

keytool -keystore /usr/lib/jvm/default-java/lib/security/cacerts -storepass changeit -list

If you run or debug client code in Eclipse (which operates its specific Java virtual machine by default), you should tell it to use the system-wide keystore. Go to project properties and navigate to Run/Debug settings and choose to edit your launch configuration (see the figure below). In the dialog that opens, select the tab Arguments. There, add the following lines into VM arguments:

-Djavax.net.ssl.trustStore="/usr/lib/jvm/default-java/lib/security/cacerts"
-Djavax.net.ssl.trustStorePassword="changeit"

Now, as you launch the client, it supposedly uses the CA store of your system.

Configure client for TLS

Next, you should try if TLS access works.

Let us assume you have already implemented a client for experiments. If not, you can see this post.

You must add some more code to connect with TLS:

• Define the host as myhost.local (default: localhost)
• You must explicitly set the port to 8883 (default: 1883)
• Enable TLS/SSL (this guide uses the "default config" that appeared to work)

The items above have been considered in the code below.

import com.hivemq.client.mqtt.mqtt5.Mqtt5Client;
// ...
String host = "myhost.local";
int port = 8883;
Mqtt5Client client = Mqtt5Client.
    builder().
    serverHost(host).
    serverPort(port).
    sslWithDefaultConfig().
    build();

With the changes above, your client should be able to connect so you can verify if the TLS configuration works.