Senior Databricks Data Engineer

Interview - Face to face Position Overview: Build and maintain scalable data pipelines using Azure Databricks to support DATA & AI ACDP projects. Apply 3-4 years of experience to deliver reliable ETL processes and collaborate on data-driven insights. Key Responsibilities: Design, build, and optimize data pipelines in Azure Databricks for ingestion, ETL/ELT, and transformations, implementing Medallion Architecture (Bronze, Silver, Gold) with Delta Lake for data quality and versioning. Utilize Databricks Spark (PySpark, SQL), Delta Live Tables, and Unity Catalog for pipeline development, governance, and basic streaming. Integrate data from ADLS Gen2, Azure SQL Database, and SQL Pools; support orchestration via Databricks workflows. Implement CI/CD in Azure DevOps for Databricks deployments, including Bicep templates and ADF integration. Apply data quality measures (expectations, constraints), security (RBAC, encryption), and monitoring for performance/cost efficiency using auto-scaling and Photon. Work with analysts and stakeholders to refine requirements into functional workflows; handle data cleansing, modeling, and Python/SQL scripting for structured/unstructured data. Troubleshoot pipelines and contribute to documentation/best practices. Required Qualifications: 8 years in data engineering, focused on Azure Databricks pipelines and ETL Proficiency in PySpark, SQL, Python; experience with cloud storage and basic orchestration Familiarity with Azure fundamentals and collaborative tools