Session starts - 17:30
Static vs Dynamic Stream Processing
In early 2014 the Otto Group BI department decided to implement their own stream processing framework. From the very beginning
SPQR was designed to do real stream processing (instead of mini batching) and provide support for ad hoc queries & dynamic
topology reconfiguration. Especially the last two aspects were crucial for us to go with our own implementation as all known
frameworks thitherto could not supply us with any comparable feature.
After a second framework evaluation in 2015, we figured out that SPQR was still on the right track since ad hoc’ness continues
to be an open issue. But on the other hand we found a number of heavily required features being nicely integrated into existing
frameworks. Since Apache Flink was closest to our own concepts we finally decided to adopt the framework and find a way to add
our notion of flexibility into the setup: either as an abstraction layer or fully integrated.
The intention of this talk is to introduce the concept of ad hoc’ness and give some examples on why we consider it to be relevant
for a modern stream processing framework. Additionally it introduces possible integration points for this feature and put them
up for discussion.
About the speaker
Christian Kreuzfeldt
Christian Kreutzfeldt works as senior software developer & architect with focus on real-time streaming at the Business Intelligence department of the Otto Group in Hamburg.