Senior Data Engineer (APAC Region)

Data Pipeline Development & Operations • Design, build, and operate scalable and reliable data pipelines on the Databricks platform • Develop end-to-end data workflows from ingestion through transformation to consumption • Implement robust error handling, monitoring, and alerting mechanisms • Ensure data pipeline reliability, performance, and maintainability • Optimize pipeline performance through efficient Spark job design and cluster configuration • Manage and orchestrate complex data workflows using Databricks Jobs and workflows Legacy Code Modernization • Refactor legacy code and data pipelines to PySpark for improved performance and scalability • Migrate traditional ETL processes to modern ELT patterns on Databricks • Assess existing codebases and identify opportunities for optimization and modernization • Ensure backward compatibility and data integrity during migration processes • Document refactoring approaches and create migration playbooks • Collaborate with stakeholders to minimize disruption during code transitions Data Engineering Excellence • Implement data quality checks and validation frameworks • Design and maintain Delta Lake tables with appropriate optimization strategies • Develop reusable code libraries and frameworks for common data engineering tasks • Follow software engineering best practices including version control, testing, and CI/CD • Participate in code reviews and provide constructive feedback to team members • Troubleshoot and resolve data pipeline issues in production environments Collaboration & Knowledge Sharing • Work closely with data architects, analysts, and business stakeholders • Collaborate with Infrastructure (Infra), Applications (Apps), and Cyber teams • Share knowledge and best practices with Team ***** • Mentor junior data engineers on PySpark and Databricks technologies • Document technical solutions and maintain comprehensive documentation Essential Technical Skills • Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns • PySpark: Proven hands-on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization • Databricks Platform: Practical experience with Databricks workspace, cluster management, notebooks, and job orchestration • Workspace AI Agent: Knowledge of Databricks Workspace AI Agent capabilities and integration • Data Modelling: Experience implementing data models including dimensional modeling, data vault, or lakehouse architectures • Delta Lake: Understanding of Delta Lake features including ACID transactions, schema evolution, and optimization techniques • Python: Strong Python programming skills for data processing and automation Additional Technical Skills • SQL proficiency for data querying and transformation • Experience with cloud platforms (Azure, AWS, or GCP) • Understanding of data governance and security best practices • Knowledge of streaming data…