Creating a Log Dashboard with WSO2 DAS - Part I

Wrote by 3 Comments

WSO2 Data Analytics Server (DAS) can be used to do various kinds of batch data analytics and create dashboards out of those data. In this blog, I will be discussing on how can you create a simple dashboard using the data read from a log file. As the first part, lets read the log file, load and the store the data inside WSO2 DAS.


Prerequisites:

  • Download WSO2 DAS 3.0.1 [1].
  • A log file. (I will be using a log dumped from WSO2 ESB)

Unzip the WSO2 DAS server, and start the sever by running the <DAS_HOME>/bin/wso2server.sh script. Then loggin to the management console (Default URL https://localhost:9443/carbon) using "admin" as both username and password.


Create Event Stream

First we need to create an event stream to which we push the data we get from the log files. To create an event stream, in the management console, navigate to the Manage→Event→Streams. Then click on Add Event Stream. You will get form like below.



Give a desired name, version and a description for your event stream. Under "Stream Attributes" section add payload attributes, which you wish to populate with the data coming from the logs.
For example, I want to store the time-stamp, class name, and the log message. Therefore, I create three attributes to store the above three information.



Once the attributes were added, we need to persist this stream to the database. otherwise the data coming to this stream will not be able to use in future. To do that click on "Next[Persist Event]" icon. In the resulting form, tick on Persist Event Stream and also tick on all the three attributes, as below.



Once done, click on "Save Event Stream".


Create Event Receiver

Our next step would be to create an event receiver to read the log file and push the data in to the event stream we just created. For that, navigate to Manage→Event→Receivers and click on Add Event Receiver .



Give a desired name to the receiver. Since we are reading a log file, select the "Input Event Adapter Type" as "file-tail" and set "false" for "Start From End" property. For "Event Stream", pick the event stream "LogEventsStream" we created earlier, and message format as "text".

Next we need to do a custom mapping for the events. In order to do so, click on Advanced. Then create three regex expressions to match the fields you want to extract from the log, and assign those three regex expressions to the three fields we defined in the event stream (see below).


Here the first regex match to the timestamp in the log file, and the second regex matches to the class name in the log file, and so on. Once done, click on "Add event Receiver".  Then DAS will read you file and store the fields we defined earlier in the Database.


View the Data

To see the data stored by DAS, navigate to Manage→Interactive Analytics→Data Explorer in management Console. Pick "LOGEVENTSSTREAM" (the event stream we created earlier),  and click on "search". This should show a table with the information from the log file, as below.




First phase of creating a log dashboard is done. In the next post I will discuss on how can you create the dashboard out of this data.


References

[1] http://wso2.com/products/data-analytics-server/

Tags:

Share:

3 comments

  1. FreelancerApril 14, 2016 at 7:38 AM

    I have follow that instruction step by step an i get that error : Failed to update event receiver, Exception: java.lang.NullPointerException.
    If i disable customMapping event receiver is runnig. please help

    ReplyDelete
  2. UnknownMay 18, 2016 at 4:58 AM

    Thanks Alot

    ReplyDelete
  3. TriamFebruary 25, 2019 at 8:45 AM

    This news is important. private prostitute hooker

    ReplyDelete