Senior Data Engineer - Databricks & Streaming - Healthcare AI (Onsite, Evening Shift, Lahore, PKR Salary)

Requirements: 4+ years of experience in data engineering, with substantial production experience in Databricks. Strong experience with Spark SQL, PySpark, Delta Lake, Medallion Architecture, and Delta Live Tables (DLT). Hands-on experience with Structured Streaming or equivalent production-grade streaming ingestion using Azure Event Hubs, Kafka, or Kinesis. Strong understanding of checkpoint recovery, watermarking, and deduplication strategies. Demonstrated experience debugging source-to-warehouse data discrepancies. Ability to walk through a real-world incident involving mismatched record counts and explain how the root cause was identified and resolved. Proven experience integrating third-party REST APIs in production. Experience handling pagination edge cases, rate and row limits, retries, and schema drift. Experience with entity resolution or data matching involving messy, real-world text data. Experience with a metrics or semantic layer such as Holistics AML/AQL, dbt Metrics, or LookML. Working understanding of why non-additive measures cannot be reliably calculated from pre-aggregated rollups. Strong SQL and Python skills, with the ability to own data pipelines end-to-end with minimal oversight. Strong written and spoken English, with the ability to collaborate effectively with a US-based team asynchronously. Healthcare data experience, including referrals, payer taxonomy, claims/eligibility, or other PHI-adjacent datasets. Familiarity with HIPAA handling expectations. Experience with voice-agent, call-center, or telephony/conversation data. Familiarity with call transcripts and containment or outcome metrics. Hands-on experience with Holistics, specifically AML/AQL modeling. Experience with the broader Azure ecosystem beyond Event Hubs, including ADLS, ADF, and Key Vault. Responsibilities: Build and maintain resilient ingestion pipelines for third-party vendor REST APIs. Work primarily with voice-AI observability and telephony platforms. Handle different pagination schemes, including offset/limit and page/cursor models. Manage row and rate limits through time-windowing and adaptive bisection. Implement robust schema-drift handling through contract and column-presence checks. Ensure alerts are triggered when fields are renamed, moved, or removed rather than silently propagating null values. Own streaming ingestion from Azure Event Hubs into Databricks using Structured Streaming and/or Auto Loader. Manage checkpoints and offsets, watermarking, and at-least-once deduplication. Perform source-parity reconciliation across ingested and production data. Investigate row counts, dropped or duplicated events, late-arriving data, and schema mismatches. Identify and resolve the root cause when ingested data does not match production sources. Develop and maintain Delta Lake pipelines using a Medallion Architecture (Bronze Silver Gold). Use Spark SQL and PySpark to build and maintain production data pipelines. …