Lead Data Engineer

We are seeking an experienced Lead Data Engineer with deep expertise in Python, Databricks and Spark to lead the delivery of large-scale data engineering initiatives. This is a unique opportunity for a technical leader who enjoys combining hands-on engineering with project delivery, team leadership and stakeholder engagement in a complex enterprise environment. Client Details Our client is a leading international financial institution with a long-established presence across major global markets. Serving a diverse client base that includes corporates, financial institutions and investors, the organisation delivers a broad range of banking, financing and capital markets services. With significant investment in digital transformation and data-led innovation, the organisation is modernising its technology estate and expanding its enterprise data capabilities. Data plays a critical role in supporting business operations, regulatory obligations, analytics and strategic decision-making, making this an exciting opportunity to contribute to large-scale, business-critical data initiatives within a complex global environment. Description Lead the end-to-end delivery of data engineering projects, ensuring successful outcomes against business objectives, timelines and quality standards Design, develop and optimise scalable data pipelines using Python, Databricks and PySpark Build and enhance ETL/ELT workflows, reusable frameworks and modern data engineering solutions Provide technical leadership and architectural guidance across data platform initiatives Design and govern Lakehouse architectures leveraging Databricks and Delta Lake Manage sprint planning, backlog prioritisation and delivery tracking within Agile teams Identify and mitigate delivery risks, dependencies and technical challenges Troubleshoot complex data, platform and performance-related issues Drive engineering best practices across coding standards, testing, performance optimisation and code reviews Act as a key interface between business stakeholders and engineering teams, translating requirements into scalable technical solutions Collaborate with architects, product owners and senior stakeholders to shape delivery roadmaps and technical direction Lead, mentor and develop a team of data engineers, fostering a culture of ownership and engineering excellence Ensure data quality, governance, reliability and security across data platforms Implement monitoring, logging and alerting capabilities to support operational excellence Champion modern engineering practices, including CI/CD, DevOps and automation Support production environments and drive continuous improvement across data solutions Profile You will bring: 10+ years' experience in data or software engineering Proven experience as a Lead Data Engineer, Technical Lead, Engineering Lead, or similar hands-on leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark / PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture and Lakehouse concepts Advanced SQL and distributed data systems expertise Experience with cloud platforms such as AWS, Azure or GCP Experience working within Agile delivery environments Excellent stakeholder management and communication skills Ability to balance technical leadership with hands-on delivery responsibilities Experience delivering data engineering solutions within Banking, Financial Services, Capital Markets, Insurance, or other highly regulated enterprise environments would be highly advantageous Desirable experience includes: Databricks certifications Kafka or Structured Streaming CI/CD and DevOps practices Airflow or Databricks Workflows Delta Lake optimisation techniques Docker and Kubernetes Exposure to machine learning pipelines Job Offer Competitive day rate of £650-880 per day Inside IR35 (Umbrella) Initial 6-month contract Hybrid working arrangement - 2/3 days onsite Opportunity to lead strategic data transformation initiatives Access to modern cloud and data engineering technologies Significant influence over technical direction and delivery Collaborative, high-performing engineering environment