Lead Data Engineer

Role: Lead Data Engineer Experience: 6–8 years Location: Bengaluru / Hybrid Employment: Full-time Role Summary We are looking for a Lead Data Engineer with strong hands-on experience in real-time data streaming, event processing, CDC, data integration, and modern data engineering. The Lead Data Engineer will be responsible for designing, developing, and maintaining highperformance data pipelines using Apache Flink, Apache Kafka, Debezium, CDC, ClickHouse, and Apache Airflow. This is a hands-on engineering role requiring strong practical experience in building productiongrade streaming applications, developing data pipelines, troubleshooting distributed systems, and optimizing data processing workloads. The ideal candidate should be comfortable working across the full data pipeline—from source systems and CDC ingestion through Kafka and Flink processing to analytical storage, APIs, dashboards, and downstream integrations. Key Responsibilities Real-Time Data Engineering Design, develop, and maintain real-time data pipelines using Apache Flink and Apache Kafka. Develop production-grade streaming applications for high-volume and low-latency workloads. Implement data transformation, filtering, enrichment, aggregation, and event processing. Build reliable event-processing pipelines with appropriate error handling and recovery mechanisms. Consume and publish events across Kafka topics. • Implement appropriate partitioning, consumer groups, offsets, and delivery mechanisms. Troubleshoot streaming pipeline failures and performance issues. Apache Flink Develop and maintain Apache Flink jobs forreal-time data processing. Implement: Stream transformations Filtering Mapping Aggregations Joins Windows Event-time processing Watermarks State management Implement Flink checkpointing and recovery mechanisms. Optimize Flink jobs for performance, scalability, and resource utilization. Monitor Flink jobs for latency, throughput, failures, and resource consumption. Troubleshoot state, checkpointing, backpressure, and processing issues. Kafka Develop Kafka-based ingestion and streaming pipelines. Create and manage Kafka topics and event streams. Workwith partitions, offsets,consumer groups, replication, and retention. Implement reliableproducer and consumer applications. Handle message ordering, retries,duplicate events, and replay scenarios. Monitor Kafka performance and troubleshoot consumerlag and throughput issues. Work with Kafka schemasand serialization formats. CDC & Debezium BuildCDC-based ingestion pipelines using Debezium . Configure and maintain Debeziumconnectors. Capture source-system inserts, updates,and deletes. Publish CDC events into Kafka. Handle initial snapshots and incremental CDC processing. Manage schemaevolution and changesin source systems. Implement data reconciliation and consistency checks. Troubleshoot CDC failures and source-to-target data issues. Data…