Monitoring WSO2 products with Prometheus

Hi folks,

The following document explains how to integrate WSO2 products with Prometheus which is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. You can read more about Prometheus from the following link [1].

Apart from the main Prometheus server which scrapes and stores time series data Prometheus contains a set of special-purpose exporters. This post focuses on the JMX Exporter which is a collector that can be configured to scrape and expose mBeans of a JMX target. The exporter is intended to be run as a Java Agent, exposing an HTTP server and serving metrics of the local JVM.

WSO2’s JMX Mbeans service

Before we move on to the configurations related to the Prometheus server and its JMX exporter, a brief description on WSO2 JMX Mbeans will be as follows.

As you might already know, all WSO2 products expose JMX Mbeans during the startup and the related service URL as shown below will be logged on the wso2carbon.log during the startup.

service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi

JMX Mbeans service URL

The ports related to the above service are configured in the carbon.xml file (<Product_Home>/repository/conf) as follows.

<!-- The JMX Ports -->
        <JMX>
            <!--The port RMI registry is exposed-->
            <RMIRegistryPort>9999</RMIRegistryPort>
            <!--The port RMI server should be exposed-->
            <RMIServerPort>11111</RMIServerPort>
        </JMX>

It’s possible to check the above Mbeans using a tool such as visual vm or jconsole via the above jmxUrl.

JMX Mbeans via jconsole

The default credentials to connecting to the above service is admin:admin

Starting the main Prometheus server

Integrating the JMX exporter starts with starting the main Prometheus server which can be done by following the simple steps below.

1. Download the main Prometheus server related to your environment form the following location [2].
2. Unzip the downloaded file.
3. Start the server with a command as follows.

./prometheus — config.file=prometheus.yml

Starting the main Prometheus server

Please note that the above prometheus.yml is shipped with the server and by default it’s pointed to monitor the server it self which consumes the port 9090. Please read more on the YAML configurations related to Prometheus from the below link [3].

Once the server is started you should be able to access the Prometheus server via a web browser using the following URL.

http://localhost:9090

The monitoring target of the service (the server itself with the default YAML file) will be shown in following URL. It will also indicate if there are any errors in the connection.

http://localhost:9090/targets

Prometheus server monitoring itself

An endpoint will be exposed as /metrics allowing to consume the JMX Mbeans to be processed for purposes such as graphical display.

Running JMX exporter as a java agent for WSO2 products

Now let’s look into integrating the JMX exporter with WSO2 products. As it was explained above, the targets which are expected to expose the JMX Mbeans to the Prometheus server are configured via YAML files. In order to use the WSO2’s JMX URL as a target you will require to create a simple YAML file as follows.

startDelaySeconds: 30
jmxUrl: service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
ssl: false
lowercaseOutputName: true
username: admin
password: admin

Following are the steps to send the JMX Mbeans updates to the Prometheus server using the JMX exporter.

1. Create a YAML file (e.g. wso2config.yaml) with the above content and save it. The path to the file will be used in the configurations as described below.
2. Download the JMX exporter’s JAR from the following link [4].
3. Add the following line under $JAVA_OPTS, on the wso2server.sh file which is available in <Product_Home>/bin

-javaagent:<Path to the exporter's JAR file>/jmx_prometheus_javaagent-0.1.0.jar=1234:<Path to the YAML config file>/wso2config.yaml \

Please note the port given as 1234 which will be used to expose the JMX Mbeans via the Prometheus server as the /metrics endpoint.

4. Add another entry as follows to the Prometheus server’s YAML file (prometheus.yml) directing to listen to the above port (1234) given for the JMX exporter.

static_configs:
      - targets: ['localhost:9090']
      - targets: ['localhost:1234']

5. Restart the Prometheus server with the same command issued above.

./prometheus — config.file=prometheus.yml

6. Copy the following jars from <Product_Home>/repository/components/plugins to <Product_Home>/bin

org.wso2.carbon.tomcat_4.4.3.jar
tomcat_7.0.59.wso2v3.jar

The requirement for the above action is explained in the notes below.

7. Start the WSO2 server as usual with a command as follows.

./wso2server.sh

8. Check the Prometheus server’s targets via a web browser.

Target metrics endpoint for the WSO2 server on the given port

9. You can check the data exposed by the /metrics endpoint by accessing it via a browser. These data will be updated for each refresh, publishing the latest stats available for the WSO2 server.

Data exposed via the metrics endpoint

I hope the above setup provides you the fundamental configurations required to integrate WSO2 products with Prometheus and will lead you to play around with the settings and monitor the servers via Prometheus.

Have a great time with WSO2 products and Prometheus.

Cheers.

Notes:

1. Add the correct version of the JAR file (jmx_prometheus_javaagent-0.1.0.jar in the above case) to the wso2server.sh file.
2. Be mindful to provide an independent port (1234 in the above case) for the JMX exporter.
3. When the JAR files related to the tomcat server are not copied to <Product_Home>/bin where some of the tomcat JARs reside, it will throw the following error stating it can’t find the org.wso2.carbon.tomcat.jndi.CarbonJavaURLContextFactory class.

ERROR - JmxCollector JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NoInitialContextException: Failed to create InitialContext using factory specified in hash table. [Root exception is java.lang.ClassNotFoundException: class org.wso2.carbon.tomcat.jndi.CarbonJavaURLContextFactory not found]

4. On a Windows environment the above integration can be done by adding the following line in the wso2server.bat file under CMD_LINE_ARGS.

-javaagent:<Path_to_the_exporter_jar>\jmx_prometheus_javaagent-0.1.0.jar=1234:jmx_exporter_config.yaml

Since the exporter uses the “:” sign as a regular expression to split the arguments [5], there might be conflicts while resolving the path to its YAML config file because file paths in Windows contain “:” signs. Therefore place the exporter’s YAML file in side the <Product_Home> folder.

5. The above configurations were tested with API-Manager-1.10.0.

[1]https://prometheus.io/docs/introduction/overview/

[2]https://prometheus.io/download/

[3]https://prometheus.io/docs/prometheus/latest/configuration/configuration/

[4]https://github.com/prometheus/jmx_exporter

[5]https://github.com/prometheus/jmx_exporter/blob/master/jmx_prometheus_javaagent/src/main/java/io/prometheus/jmx/JavaAgent.java