High-volume event streams are becoming widespread in the telecom industry: traditional network data, IoT sensor data, activity events on social media, etc. In particular, live analysis of telco log files and performance metrics allows network operators to observe the status of the system and identify possible problems using online aggregations and incremental machine learning algorithms. Offline batch analysis of streams using tools like MapReduce is often too slow to respond to things happening right now. Traditional event processing frameworks can be challenging to set up and might not be scalable enough to handle the onslaught of data.
In this talk we will show an analytics pipeline setup for a telco use case that processes an unbounded data set of logs and performance metrics. The aggregated information and insights are stored and made visually available to the operator. The pipeline uses Kafka as message broker gathering the events from different input sources. Data is then consumed and analyzed by a Flink streaming job. The output is then stored in Elasticsearch, a scalable real-time capable data store, and finally visualized to the Operator using Kibana, an intuitive visualization platform that integrates seamlessly with Elasticsearch.
We will discuss some of the challenges and benefits of building an analytics pipeline using aforementioned tools and conclude with a demo of the system.
About the speaker
Ignacio Mulas Viela
Ignacio is a researcher working in the area of cloud analytics at Ericsson Research. His main focus is to look into what possibilities analytics techniques offer to the cloud and what possibilities the cloud can offer to the existing the existing analytics environments.
The work presented here is part of the collaboration between Ericsson Research and SICS.