getSummary
Written in English from the original postingSenior Data Engineer role focused on building and running real-time streaming pipelines with Apache Flink and Kafka. The role involves event-time processing, checkpointing, state management, lakehouse data modeling, and monitoring pipeline health. It is listed as remote or based in Cairo, Egypt.
Job description
Builds and runs real time streaming pipelines, keeping latency low and jobs recoverable at high
volume.
- Strong hands
on Apache Flink across DataStream API and Flink SQL. - Event time processing, checkpointing, savepoints, and state management in
production. - High volume Kafka streaming.
- Data modelling for analytics and streaming into a lakehouse.
- Monitors pipeline health and handles recovery when jobs fail.
Requirements
Requirements
pache Flink
(DataStream, Flink SQL), Kafka, Java, Scala, Python, SQL, Docker, Kubernetes
- Strong hands-on experience with Apache Flink, including DataStream API and Flink SQL.
- Experience with high-volume Kafka streaming and real-time data processing.
- Strong knowledge of event time processing, checkpointing, savepoints, and state management.
- Experience with data modelling for analytics and streaming into a lakehouse.
- Experience monitoring pipeline health and handling job recovery and failures.
- Apache Iceberg or other open table formats.
- Airflow.
- On premises or self managed cluster experience.
