When we run a java process it is very convenient to connect through JMX to diagnose some JVM behaviour (memory consumption, thread activities, MBean attributes...), but JMX over RMI this not always possible through firewalls or tunnels, and especially through Kubernetes port forwarding.
The solution: JMXMP
Fortunately, as describe in the followings posts, JMXMP, JMX over a simple TCP connection may help us:
Enable JMXMP for Elassandra
Since the version 188.8.131.52, Elassandra implements a JMXMP server to allow remote connection over JMXMP. Just run Elassandra with the cassandra.jmxmp system property:
JVM_OPTS="$JVM_OPTS -Dcassandra.jmxmp" /usr/bin/cassandra -e -f
And run nodetool with the --jmxmp option as follow:
nodetool --jmxmp status Datacenter: DC1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID UN 172.17.0.2 70.86 KiB 8 100.0% 4c7bc0c1-4354-49c4-964b-f2f4ad4315b9
Connect VisualVM through JMXMP
To connect with VisualVM, you have to download the JMDK library from maven,
and execute visualvm by appending the downloaded jar in the classpath.
visualvm -cp:a ./jmxremote_optional-repackaged-5.0.jar
Create a local connection using the URI "service:jmx:jmxmp://localhost:7199/" (Uncheck the "Do not require SSL connection", unfortunately visualVM does not currently support SASL plain text client authentication).
And "voila" you can access to the Elassandra internals !
You can now use nodetool or visualvm through a Kubernetes port forwarding.
Secure your JMXMP connection
Elassandra support TLS/SSL encryption and SASL plain text login/password authentication with the same settings as JMX over RMI, but unfortunately, few tools support-it right now and a https://github.com/oracle/visualvm/issues/162 is open on visualVM.