Analytics for WSO2 ESB : Architecture in a Nutshell

Wrote by 4 Comments

ESB Analytics Server is the analytics distribution for the WSO2 ESB, which is built on top of WSO2 Data Analytics Server (DAS).  Analytics for ESB consists of an inbuilt dashboard for Statistics and Tracing visualization for Proxy Services, APIs, Endpoints, Sequence and Mediators. Here I will discuss on  the architecture of the Analytics Server, and how it operates behind the scenes, to provide this comprehensive dashboard.

Analytics Server can operates in three modes:

  • Statistics Mode
  • Tracing Mode
  • Offline Mode

For all the three modes, data are published from ESB Server  to the Analytics server via the data bridge. In doing so ESB server uses the "publisher" component/feature, while Analytics server uses the "receiver" component/feature of the data bridge. ESB triggers an event for a single message flow, to the Analytics Server. Each of these events contains the information about all the components that were invoked during the message flow.

If the statistics are enabled for a given Proxy/API at the ESB side, then the Analytics server will operates in "Statistics Mode". If the Tracing and capturing Syanpse properties are also  enabled at the ESB side, then the Analytics server will operates on the "Tracing Mode". Analytics server will switch between these modes on the fly, depending on the configurations set at the ESB side.


Statistics Mode

In this mode, ESB server will be sending information regarding each mediator, for each message to Analytics Server. Analytics Server will be calculating the summary statistics out of these information, and will store only summary statistics, but will not store any raw-data coming from the ESB. This is a hybrid solution of both siddhi (WSO2 CEP) and Apache Spark. This mode generate statistics in real time.



Pros: Can handle much higher throughput. Statistics are available in real time.
Cons: No tracing available. Hence any message related info will not be available in the dashboard.


Tracing Mode

Similar to the Statistics mode, ESB server will be sending information regarding each mediator, for each message to Analytics Server. Analytics Server will be calculating the summary statistics out of these information,. But unlike previous case, it will store both statistics as well as component wise data. This enables the user to trace any message using the dashboard. More importantly, this mode also allows a user to view statistics and trace messages in real time.



 Pros: Statistics and Tracing info are available in real time. Message level details are also available.
Cons: Throughput is limited. Can handle upto around 7000/n events per second, where n is the number of mediators/components available in the message flow of the event sent from ESB.


Offline Tracing

This mode also allows a user to get statistics as well as tracing, similar to the previous "Tracing Mode". But this operates in an offline/batch analytics mode, unlike previous scenario. More precisely, Analytics Server will store all the incoming events/data from ESB, but will not process them on the fly. Rather a user can collect data for any period of time, and can run a predefined spark script in-order to get the stats and tracing details.


 Pros: Users can trace messages, and message level details are available. A much high throughput can be achieved compared to "Tracing Mode"
Cons: No realtime processing.

Tags:

Share:

4 comments

  1. AnonymousOctober 7, 2016 at 12:22 AM

    How can I enable Offline Tracing mode? I have selected "Enable Statistics" and "Enable Tracing" in my ESB Api, but i can't see message data in ESB Analytics 5.0. Statistic data is working good, but no message related info in message gadget.
    Thanks!

    ReplyDelete
    Replies
    1. Supun SetungaOctober 7, 2016 at 1:14 AM

      To switch to offline mode, you need to undeploy/remove the existing ESB_ANALYTICS_HOME/repository/deployment/server/carbonapps/org.wso2.analytics.esb.realtime-1.0.0.car and deploy/copy the ESB_ANALYTICS_HOME/capps/org.wso2.analytics.esb.offline-1.0.0.car to ESB_ANALYTICS_HOME/repository/deployment/server/carbonapps directory. (Or do the same via the management console UI).

      For your second concern, make sure you have enabled both statistics, as well as payloads, in the ESB_HOME/repository/conf/synapse.properties file. Please refer [1].

      [1] https://docs.wso2.com/display/ESB500/Prerequisites+to+Publish+Statistics#PrerequisitestoPublishStatistics-Enablingstatisticsforthemediationflow

      Regards,
      Supun

      Delete
    2. AnonymousOctober 7, 2016 at 4:47 AM

      I finally have tracing data. The problem was in my CAR artifact. If you use a deployable CAR artifact you have to set 'statistics="enable" trace="enable"' in EVERY esb artifact included in CAR file. Checking Enable statistics and tracing in console GUI won't work.
      I'm trying your first indication now so I can use offline tracing and improve throughput.
      Thanks a lot, your answer was much help! (I discarded ESB configuration errors, so i focused artifacts)

      Delete
    3. AnonymousOctober 10, 2016 at 1:55 AM

      I used offline mode, but it didn't work. First the name of the xml file inside org.wso2.analytics.esb.offline-1.0.0.car wasn't correct. Inside Sparkscripts_1.0.0/artifact.xml there is a reference to EsbAnalytics-SparkScript-Offline-Statistic.xml, but the file included in CAR is named esb_stat_analytics_offline.xml.
      I changed the file name and repacked. Deployment worked good, but when the script failed when executed. I'm not an expert in Spark SQL, so I moved back to realtime mode. I'm waiting for a new release fixing this issue.
      Thanks anyway.

      Delete