Data Engineer

Role Overview We are seeking an experienced Data Engineer with strong hands-on expertise in Databricks, Delta Lake, Databricks SQL, Workflows, Unity Catalog, advanced SQL, and Python . The candidate will be responsible for designing, developing, and operationalizing production-grade data pipelines, with a strong focus on integrating data from REST APIs and building scalable, reliable data solutions. The ideal candidate should have experience developing enterprise data pipelines, implementing dimensional data models, establishing data quality frameworks, and working with governed data environments. Experience with the Azure data ecosystem and AI/engineering-tool telemetry will be an added advantage. Key Responsibilities Databricks Data Engineering Design, develop, and maintain production-grade data pipelines using Databricks . Develop scalable data processing solutions using Delta Lake and Databricks SQL . Build and manage Databricks Workflows for pipeline orchestration, scheduling, monitoring, and dependency management. Implement reliable and reusable data ingestion and transformation frameworks. Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency. Implement appropriate error handling, logging, monitoring, and recovery mechanisms for production pipelines. REST API Data Integration Develop data pipelines that ingest data from REST APIs and external enterprise systems. Design reusable API ingestion frameworks capable of handling authentication, pagination, rate limits, retries, incremental extraction, and error handling. Transform API responses into structured datasets suitable for downstream analytics. Implement mechanisms for incremental and historical data ingestion. Troubleshoot API connectivity, data availability, schema changes, and ingestion failures. Data Modelling & SQL Design and implement dimensional data models for analytical workloads. Develop fact and dimension tables and establish appropriate relationships for reporting and analytics. Write advanced SQL for data transformation, validation, aggregation, and analytical processing. Optimize complex SQL queries and Databricks workloads for performance. Ensure data models are scalable, maintainable, and aligned with business reporting requirements. Python Development Develop robust data engineering applications and pipeline components using Python . Build reusable Python libraries and utilities for ingestion, transformation, validation, and automation. Implement exception handling, logging, configuration management, and testing practices. Use Python to automate operational and data engineering activities. Data Quality & Reliability Design and implement data quality frameworks across ingestion and transformation pipelines. Establish automated checks for data completeness, accuracy, consistency, uniqueness, and validity. Implement data reconciliation and validation mechanisms…