Monitor and analyse SOA Suite data using ElasticSearch and Kibana

In this article I am going to create a light-weigh dasboard build on ElasticSearch and Kibana which can monitor and analyse SOA Suite data. It is fast and easy to create. It is also extendible if you would like to do more analysis (Apache Spark), transform data before storing it (Logstash) or get data from other sources (Beats) and so on.

In this example I will setup a SOA Suite 12C with a custom policy which gathers all the data and pushes it into ElasticSearch. As ElasticSearch is very nicely integrated with Kibana as a visualization tool, we will then create a Kibana dashboard on top of the data which we are logging. Let’s get started.

Setup

First of all we need a SOA Suite 12C instance. You can either use a docker image or install it….which ever you prefer. Next to that we need ElasticSearch and Kibana. I am using ElasticSearch 6.1.2 together with Kibana 6.1.1. You can download ElasticSearch here and Kibana here. You can start them both using executables in the bin folder. Once you started them you can check if they both are up and running. First run ElasticSearch

You can check if it is successfully up and running by making a GET Postman call to

http://localhost:9200/?pretty

This should result in something like this:

{
    "name": "REDROCK",
    "cluster_name": "elasticsearch",
    "cluster_uuid": "Bd5tesgAQf2xC7PwkZxXgw",
    "version": {
        "number": "6.1.2",
        "build_hash": "5b1fea5",
        "build_date": "2018-01-10T02:35:59.208Z",
        "build_snapshot": false,
        "lucene_version": "7.1.0",
        "minimum_wire_compatibility_version": "5.6.0",
        "minimum_index_compatibility_version": "5.0.0"
    },
    "tagline": "You Know, for Search"
}

Next lets start up Kibana:

Check

http://localhost:5601

in your browser and you should get your Kibana app.

Custom policy for sending data to ElasticSearch

Next is how are we going to gather all the info we want to log and get it into ElasticSearch? One of the ways is to make a custom policy and let this one put the data into ElasticSearch. Is this example we will do that. In a real production environment I would go for a more solid approach and have the custom policy put a message on a queue to decouple. A very high performance solution would be a Kafka server for transport but would also add to the complexity. There are a number of solutions here, take a good look at the requirements and choose your best fit.

Lucas Jellema wrote and excelent article about creating a custom policy here. I am going to re-use the code to create my own policy which sends the data to ElasticSearch.

As I said before, I will not be using a queue but I will just make a HTTP PUT right into ElasticSearch. You can find my code here if you want to have a look at it. I added the zip to the server and added the custom policy to the domain.

Then I configured the policy for an example OSB service I have running on my domain.

After saving and applying it all I fire of a simple soap request of my example service.

This comes back successful. Now have a look at the logging part in Weblogic:

As you can see I extract the service name, the ECID, the date, whether it was successful or not and the response time. This I then put to the ElasticSearch server on the index /logs. To get some data in there I ran a load test in SoapUI.

Create a Kibana dashboard

Now go to the Kibana app by browsing to http://localhost:5601. The first thing you have to do is make an Index Pattern. Go to Management and then to Index Patterns. Now enter here the index to which you put the document so mine would be under logs*.

Click Next Step. Now we can also select a Time Filter field name as we send dates in. I choose not to use it and click Create Index Pattern. Now you will get an overview of the pattern. To see the data click on Discover.

Here you can see all your data which you have put in and can query and filter if you want using the Lucene query syntax.

Now for getting the data into a dashboard. Click Visualize and the Create a visualization. I choose a Vertical Bar.

In the next screen you can choose an index or a Saved Search. I choose my previous created Logs* index. In the next screen you can configure how the graph should work and look like. I would like to see the amount of calls grouped by second for example. First I set the Custom Label of the vertical bar. Next I choose X-Axis as Buckets.

Next I select a Date Histogram as Aggregation. For Field I select my date field. For Interval I choose seconds and last I set my custom label to Time. Then you can press the Preview button for a preview on the right. If it looks good you can save it top right and give it a name. Now you can use it on a dashboard.

Click on Dasboard and choose Add. Now choose the visualization we just made…in my case Calls per second and voila. It shows the graph we have just configured. You can also configure it has to refresh every x seconds top right so you can actually see the calls near realtime. I also added a simple pie graph based on the amount of calls for each service.

Also a cool feature is that when you select data in a graph, it will act as a filter and affects then all graphs on the dasboard. This way you can drill down into data visually.

Conclusion

I have just scratched the surface here of the power of ElasticSearch and Kibana. I am not a BigData/Business Intelligence guy but I found it quite intuitive to build visualizations. As the elastic stack is offered by several cloud vendors…..AWS, Azure and Elastic itself, it is easy to start. Decide which data would be interesting, make a loosely coupled sidestep for your data, stream it into ElasticSearch and visualize fast.

References:

• Oracle SOA Suite 12c – Create, Deploy, Attach and Configure a Custom OWSM Policy – to report on service execution
• https://www.elastic.co/cloud