We use Apache Flink extensively to implement streaming processing pipelines on the LINE data platform. This session presents two recent projects for high-performance and stable processing within these pipelines.
The first is a redesign of our Kafka-to-Elasticsearch pipeline using Flink. We’ll explain the background behind the project while digging into the differences between Kafka Streams and Flink architecture such as their threading models. In addition, we’ll show the results of the project in the form of a performance evaluation.
The second is an auto-scaling system for Flink jobs. We’ll explain how we designed the auto-scaling system to be generally useful for many types of Flink jobs on our platform as well as how we implemented it as a Kubernetes overlay.
