Weblogic Monitoring with Prometheus and Grafana

Since quite some time Oracle is providing in their GitHub Repository the weblogic-monitoring-exporter https://github.com/oracle/weblogic-monitoring-exporter.

This weblogic-monitoring-exporter is basically a war file, which uses the WebLogic Server RESTful Management API to collect defined metrics of your WebLogic Server and exports them in a Prometheus compatible format.

At first you will need to download the get[Version-Number].sh from the GitHub Repository https://github.com/oracle/weblogic-monitoring-exporter/releases. Next step is to create a YAML file in which you define which metrics you want to collect from your WebLogic Server.
Place both files in one directory and execute the get[Version-Number].sh with your YAML file as input parameter. The get[Version-Number].sh will simple download from GitHub the required war file, so make sure that your system on which you execute the script got internet connection, and merge your YAML file into the war file.

Example YAML file named exporter-config.yml

metricsNameSnakeCase: true
queries:
- key: name
  keyName: location
  prefix: wls_server_
  applicationRuntimes:
    key: name
    keyName: app
    componentRuntimes:
      prefix: wls_webapp_config_
      type: WebAppComponentRuntime
      key: name
      values: [deploymentState, contextRoot, sourceInfo, sessionsOpenedTotalCount, openSessionsCurrentCount, openSessionsHighCount]
      servlets:
        prefix: wls_servlet_
        key: servletName
- JVMRuntime:
    prefix: wls_jvm_
    key: name
- executeQueueRuntimes:
    prefix: wls_socketmuxer_
    key: name
    values: [pendingRequestCurrentCount]
- workManagerRuntimes:
    prefix: wls_workmanager_
    key: name
    values: [stuckThreadCount, pendingRequests, completedRequests]
- threadPoolRuntime:
    prefix: wls_threadpool_
    key: name
    values: [executeThreadTotalCount, queueLength, stuckThreadCount, hoggingThreadCount]
- JMSRuntime:
    key: name
    keyName: jmsruntime
    prefix: wls_jmsruntime_
    JMSServers:
      prefix: wls_jms_
      key: name
      keyName: jmsserver
      destinations:
        prefix: wls_jms_dest_
        key: name
        keyName: destination

- persistentStoreRuntimes:
    prefix: wls_persistentstore_
    key: name
- JDBCServiceRuntime:
    JDBCDataSourceRuntimeMBeans:
      prefix: wls_datasource_
      key: name
- JTARuntime:
    prefix: wls_jta_
    key: name

Now let's merge the exporter-config.yml into the wls-exporter.war

 #
./get1.1.1.sh exporter-config.yml 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   607  100   607    0     0   2872      0 --:--:-- --:--:-- --:--:--  2876
100 2018k  100 2018k    0     0  1165k      0  0:00:01  0:00:01 --:--:-- 2247k
created /tmp/ci-21tVFvZiR4
/tmp/ci-21tVFvZiR4 /u00/app/oracle/install
in temp dir
  adding: config.yml (deflated 64%)
/u00/app/oracle/install

As next we can deploy the generated wls-exporter.war on your WebLogic Server, just connect with a browser to your WebLogic Console (http://<servername>:<AdminServerPort>/console)

Navigate under the Domain Structure to the Option Deployments, click the "Lock & Edit" button in the Change Center, if you are running your WebLogic Domain in Production Mode and click the "Install" button under the Configuration tab in the Deployments.

Now navigate to the location of the wls-exporter.war file and select it and hit the button "Next"

Make sure that the option "Install this deployment as an application" is marked and hit the button "Next"

Target the deployment of the wls-exporter.war to your Servers and hit the button "Next"

Leave the default and hit the button "Next"

Leave the default and hit the button "Next"

Click the button "Save" and finally click the button "Activate Changes" in the Change Center

After the successful activation you should see as follows

Now we are done with the deployment of the wls-exporter.war and we can start the application, just go to the tab "Control" and click under "Start" the option "Servicing all requests"

Confirm the startup of the application with "Yes"

After that you should see under the Control tab that the wls-exporter application is now in state Active

The next step is to configure your Prometheus environment to collect the metrics from the wls-exporter application. Just add under the scrape_configs section in your prometheus.yml configuration file the necessary entries for the WebLogic Servers.
Simply add under the scrape_config section in the prometheus.yml a job_name for each WebLogic Managed Server (including your AdminServer). If you are adding multiple WebLogic Servers, its advisable to use the metric_relabel_configs option, to prefix each metric collected by Prometheus with for example the name of the WebLogic Server:

 
# Align the job_name with your settings
# Align the IP address and Ports for the targets with your settings
# Align the metric_relabel_configs with your settings

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  scrape_timeout: 10s
. . .
. . .
scrape_configs:
. . .
. . .
  - job_name: 'WebLogicAdminServer'
    scrape_interval: 10s
    metrics_path: /wls-exporter/metrics
    static_configs:
      - targets: ['172.17.0.1:7001']
    basic_auth:
       username: weblogic
       password: Oracle12c
    metric_relabel_configs:
      - source_labels: [__name__]
        target_label: __name__
        replacement: "DemoAdminServer_${1}"

  - job_name: 'WebLogicDEMOMS1'
    scrape_interval: 10s
    metrics_path: /wls-exporter/metrics
    static_configs:
      - targets: ['172.17.0.1:7003']
    basic_auth:
       username: weblogic
       password: Oracle12c
    metric_relabel_configs:
      - source_labels: [__name__]
        target_label: __name__
        replacement: "DEMOMS1_${1}"

  - job_name: 'WebLogicDEMOMS2'
    scrape_interval: 10s
    metrics_path: /wls-exporter/metrics
    static_configs:
      - targets: ['172.17.0.1:7004']
    basic_auth:
       username: weblogic
       password: Oracle12c
    metric_relabel_configs:
      - source_labels: [__name__]
        target_label: __name__
        replacement: "DEMOMS2_${1}"
. . .
. . .

fter this changes, just restart your Prometheus and check under the Prometheus Console - Targets the status of your job_name.

You can see that we have now 3 Endpoints, one for each WebLogic Server (one AdminServer and two Managed Servers).

Under the Prometheus Console - Graph you can see all your defined metrics for your WebLogic Servers.

As next you can create a Dashboard in your Grafana environment.

So with a little bit of work you can easily monitor your WebLogic Environment with Prometheus and Grafana. In addition you can configure thresholds for your metrics and trigger alerts either via the alertmanager from Prometheus or via Grafana.

One point you should take care of with the weblogic-monitoring-exporter is the usage of the configuration option "domainQualifier: true". As documented this configuration is the metrics with the domain name, but currently the option "domainQualifier: true" will cause a non-Prometheus-conform formatted metric.

name{domain="demo_domain"} demo_domain
DemoAdminServer_wls_server_activation_time{domain="demo_domain",location="DemoAdminServer"} 1581419617765
DemoAdminServer_wls_server_admin_server_listen_port{domain="demo_domain",location="DemoAdminServer"} 7001
DemoAdminServer_wls_server_open_sockets_current_count{domain="demo_domain",location="DemoAdminServer"} 4
DemoAdminServer_wls_server_state_val{domain="demo_domain",location="DemoAdminServer"} 2

As you can see from above listing, the wls-exporter is generating the first line wrong, as the metric value is a string and therefor it can not be parsed by Prometheus.

I have currently opened an issue in the GitHub Repository for the weblogic-monitoring-exporter and I hope to get soon a feedback and/or solution https://github.com/oracle/weblogic-monitoring-exporter/issues/82

But nevertheless, the weblogic-monitoring-exporter is a great add-on for the Oracle WebLogic Server in order to monitor your environment with Prometheus and visualise your metrics with Grafana.

UPDATE 2nd April 2020: the above mentioned problem with the non-conform Prometheus formatted metric is resolved by the Oracle Development Team for the WebLogic-Monitoring-Exporter, just get the release 1.1.2 (https://github.com/oracle/weblogic-monitoring-exporter/releases)