Solved — Is another HiveMQ instance running?

RocksDB

Recently, when trying to run HiveMQ to facilitate a test using an MQTT IOT stream the following error was presented:

ERROR - An error occurred while opening the publish_payload_store persistence. Is another HiveMQ instance running?

Looking at the error message one would be thinking that another HiveMQ instance is running. However, as this was not the case, I turned the HiveMQ logging into debug mode. Changing the log level for HiveMq is done by making a change to the logback.xml file in ./conf/

you have to change the below shown line:

    <root level="${HIVEMQ_LOG_LEVEL:-INFO}">

Into the below to hard code the log level to debug.

<root level="DEBUG">

When setting the log level to debug the system produced some interesting insights, primarily pointing to the RocksDB sub-system of HiveMQ.

2023-06-26 09:03:14,924 DEBUG - Original Exception:
org.rocksdb.RocksDBException: While fsync: a directory: Invalid argument
 at org.rocksdb.RocksDB.open(Native Method)
 at org.rocksdb.RocksDB.open(Source.java:252)
 at hmq.hz.a.a(Source.java:169)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:829)

RocksDB and file systems

As it turns out, in my case, I tried to start HiveMQ from a special mount-point on my filesystem. As I was running a vagrant/oracle virtualbox combination I placed HiveMQ in /vagrant/ as this is shared with the host operating system and makes it easy to swap files in and out of the virtual machine.

While in a lot of cases this is a good way of doing things this run yoin into a specific issue with RocksDB. RocksDB is not good in handling (for some reason unknown to me) file systems that are being actively synced. In case of Vagrant the /vagrant/ mountpoint is synced between the host OS and the virtual machine OS.