Recently, when trying to run HiveMQ to facilitate a test using an MQTT IOT stream the following error was presented:
ERROR - An error occurred while opening the publish_payload_store persistence. Is another HiveMQ instance running?
Looking at the error message one would be thinking that another HiveMQ instance is running. However, as this was not the case, I turned the HiveMQ logging into debug mode. Changing the log level for HiveMq is done by making a change to the logback.xml file in ./conf/
you have to change the below shown line:
Into the below to hard code the log level to debug.
When setting the log level to debug the system produced some interesting insights, primarily pointing to the RocksDB sub-system of HiveMQ.
2023-06-26 09:03:14,924 DEBUG - Original Exception:
org.rocksdb.RocksDBException: While fsync: a directory: Invalid argument
at org.rocksdb.RocksDB.open(Native Method)
RocksDB and file systems
As it turns out, in my case, I tried to start HiveMQ from a special mount-point on my filesystem. As I was running a vagrant/oracle virtualbox combination I placed HiveMQ in /vagrant/ as this is shared with the host operating system and makes it easy to swap files in and out of the virtual machine.
While in a lot of cases this is a good way of doing things this run yoin into a specific issue with RocksDB. RocksDB is not good in handling (for some reason unknown to me) file systems that are being actively synced. In case of Vagrant the /vagrant/ mountpoint is synced between the host OS and the virtual machine OS.