Lead Databricks Platform Engineer

Job Title Lead Databricks Platform Engineer (Contract) Location: Remote (UK-based) Day Rate: Up to £550 p/d (Outside IR35) Contract: 6-Month initially No sponsorship offered - Applicants must have RTW in the UK and their own Ltd company We’re looking for a hands-on Lead Databricks Platform Engineer to join a global software vendor and take ownership of a modern data ecosystem, driving data engineering excellence, platform optimisation, and AI enablement. You’ll lead a team of 6+ engineers, set best practices, and ensure the Databricks platform is reliable, scalable, and cost-effective.

Key Responsibilities

• Lead, coach, and mentor 6+ data engineers in modern, AI-enabled practices.
• Own and optimise Databricks (Delta Lake, Unity Catalog, MLflow, Databricks SQL).
• Redesign data pipelines for maximum scalability, performance, and cost efficiency.
• Champion Python engineering best practices (CI/CD, testing, code reviews).
• Drive DataOps and MLOps to ensure strict observability, governance, and seamless lifecycles.

What We’re Looking For

• 5+ years in data engineering with 2+ years scaling Databricks environments.
• Deep expertise in Apache Spark, PySpark, Unity Catalog, and MLflow.
• Strong production Python background with robust CI/CD experience.
• Proven track record of optimising Databricks environments for cost and performance.
• Experience leading engineering teams and establishing modern best practices.

Robert Walters Operations Limited is an employment business and employment agency and welcomes applications from all candidates