Juggling with Bits and Bytes – How Apache Flink operates on binary data

Many popular open source data processing systems are implemented in JVM-based languages. Storing and operating on large amounts of data in-memory is a common challenge for this kind of data analysis systems. The most straight-forward approach to process data in a JVM is to put it as objects on the heap and directly work with these objects. However, this approach has several notable drawbacks including controlling the memory consumption and reducing the garbage collection overhead.

This talk presents Apache Flink’s approach to address this challenge. We discuss Flink’s active memory management, its custom data serialization framework, and its techniques to efficiently operate on binary data. The talk concludes with benchmark results that compare the straight-forward object-on-heap and Flink’s approach.