Senior Data Engineer - Databricks

Overview We are looking for a seasoned Senior Data Engineer to work with our team and clients to develop enterprise-grade data platforms, data lakes and lakehouses, services, and pipelines using Databricks. We are looking for more than just a Senior Data Engineer; we are seeking a technologist with excellent communication and customer service skills and a passion for data and problem solving. Contributions Lead and support the design, development, and migration of data lake and lakehouse environments using Databricks, with a focus on performance, reliability, and scalability. Assess and understand existing ETL jobs, workflows, data marts, BI tools, reports, and associated data dependencies. Design, develop, and optimize scalable data pipelines and data processing solutions using Databricks. Address technical inquiries concerning customization, integration, enterprise architecture, and the features and functionality of data products. Develop and support database, data warehouse, data mart, data lake, and lakehouse solutions in cloud environments such as AWS, Azure, or GCP. Apply hands-on expertise with Databricks, SQL, PySpark/Python, and cloud technologies to support data engineering solutions. Support an Agile software development lifecycle. Contribute to the continued growth of our AI & Data Exploitation Practice. Qualifications Ability to obtain and maintain a U.S. government security clearance. Bachelor's degree in computer science, information systems, engineering, or a related technical discipline. 5–7 years of industry experience developing production software and solving complex technical problems. 5–7 years of direct experience in Data Engineering, including experience with tools and technologies such as: Big data technologies: Databricks, Apache Spark, Delta Lake, or similar. Relational SQL: preferably T-SQL; alternatively PostgreSQL or MySQL. Data pipeline and workflow management tools: Databricks Workflows, Airflow, AWS Step Functions, or similar. Cloud services: Databricks on AWS, S3, EC2, RDS, or Azure equivalents. Object-oriented or scripting languages: PySpark/Python, Java, C++, Scala, or similar. Data lakehouse architecture and technologies such as Delta Lake or Apache Iceberg. Advanced SQL knowledge and experience working with relational databases, including query development and optimization, as well as familiarity with a variety of database technologies. Experience manipulating, processing, and extracting value from large, disparate datasets. Ability to assess existing data pipelines, understand their purpose and functionality, and efficiently reimplement or modernize them using Databricks. Experience working with structured and unstructured data. Experience architecting data systems, including transactional systems and data warehouses. Experience with the software development lifecycle (SDLC), CI/CD practices, and development, test, and production environments. …