Senior Databricks Engineer

Job Title: Senior Databricks Engineer Experience Required: 8+ Years Role Summary We are seeking an experienced Senior Databricks Developer with 8+ years of experience in Data Engineering, Big Data, and Cloud Analytics solutions, including extensive hands-onexpertise in Databricks, Apache Spark, PySpark, SQL, and Delta Lake. The ideal candidate will be responsible for leading end-to-end data engineering projects, engaging directly with clients and business stakeholders, mentoring development teams, and delivering scalable, secure, and high-performance data platforms across Azure and AWS environments. This role requires strong technical leadership, solution design capabilities, stakeholder management skills, and the ability to drive data transformation initiatives from requirements gathering through deployment and production support. Technical Expertise Databricks (PySpark, SQL, Notebooks, Workflows) , Apache Spark , Delta Lake , Python, Azure Data Factory (ADF) ,Azure Synapse Analytics ,AWS S3, Glue, Lambda , Unity Catalog MLflow ,Databricks Job Clusters ,CI/CD Pipelines ,GitLab / GitHub ,Data Governance & Data Quality Frameworks, Power BI / Tableau , Data Cataloging ,Lakehouse Architecture ,Medallion Architecture, Performance Tuning & Optimization Key Responsibilities • Lead the design, development, and implementation of enterprise-scale data engineering solutions using Databricks, PySpark, SQL, and Delta Lake. • Own end-to-end project delivery, including requirement gathering, solution design, development, testing, deployment, and production support. • Collaborate directly with clients, business stakeholders, architects, and product owners to understand business requirements and translate them into scalable technical solutions. • Conduct client discussions, solution workshops, effort estimations, technical presentations, and architecture reviews. • Design and implement scalable Lakehouse architectures and data platforms across Azure and AWS cloud environments. • Lead the development of high-performance ETL/ELT pipelines supporting both batch and real-time data processing workloads. • Drive best practices around coding standards, architecture governance, version control, CI/CD implementation, and operational excellence. • Architect and optimize Databricks Workflows, Job Clusters, Delta Tables, and Spark applications to maximize performance and minimize infrastructure costs. • Implement data governance frameworks using Unity Catalog, ensuring proper access controls, lineage tracking, metadata management, and compliance standards. • Mentor and guide junior and mid-level data engineers through code reviews, technical coaching, and knowledge-sharing initiatives. • Lead technical teams and coordinate project activities to ensure timely and successful project delivery. • Collaborate with Data Scientists, BI Teams, and Analytics stakeholders to enable advanced analytics and machine learning use cases. •…