How I wish these operators had existed a few years ago when I was setting up Flink…

https://github.com/GoogleCloudPlatform/flink-on-k8s-operator

https://www.ververica.com/blog/google-cloud-platforms-flink-operator-for-kubernetes

The DA Platform is a huge step forward for running Flink at scale. I was lucky enough to see a demo and was really impressed. Far more advanced that the what can be achieved with Dataflow at the moment.

I’m currently bench-marking Flink against Google Cloud Dataflow using the same Apache Beam pipeline for quantitative analytics. One observation I’ve seen with Flink is the tail latency associated with some shards. 

Google Cloud Dataflow can optimise away stragglers in large jobs using “Dynamic Workload Rebalancing".  As far as I know, Flink is currently unable to perform similar optimisations.

Apache Beam vs. Apache Flink: Choosing the Right Distributed Processing Framework

Apache Beam and Apache Flink are both powerful open-source frameworks for distributed data processing, enabling efficient handling of massive datasets. While they share the common goal of parallel data processing, they differ significantly in their architecture, programming model, and execution strategies. Understanding these differences is crucial for choosing the right tool for your specific needs. This article will help you navigate the decision-making process.