Skip to content
getdataengineeringjobs
Back to results

Senior/Lead Data Engineer (Databricks)

Remote Lead Posted

Summary

Written in English from the original posting

Senior/Lead Data Engineer responsible for shaping a Databricks-based data platform and production data pipelines for industrial data. The role includes building reusable frameworks, automation, data modeling, testing, CI/CD, observability, and platform reliability. It requires Python, SQL, Azure, Terraform, and technical leadership, with fluent English and Polish.

Job description

Join our team to shape the technical direction of a data platform that makes complex industrial data usable, reliable, and scalable. Build reusable frameworks, internal tools, and automation that simplify customer onboarding and help engineering teams deliver safer, more consistent data pipelines. Work closely with engineers and domain experts to strengthen data quality, platform reliability, and developer experience while applying AI pragmatically to improve engineering productivity.

Our Client was established with the mission to fundamentally transform the execution of capital projects and operations. Designed by industry experts for industry experts, our Client’s platform empowers users to digitally search, visualize, navigate, and collaborate on assets. Drawing on 30 years of software expertise and 180 years of industrial legacy as part of the renowned Scandinavian business group, Client plays an active role in advancing the global energy transition. The company operates from Norway, the UK, and the U.S.

Key Responsibilities:

  • Shape the technical direction of the data platform, tooling, and engineering standards
  • Design and improve reusable frameworks, internal tools, and automation for data pipeline development
  • Define standards for testing, CI/CD, quality gates, and platform reliability
  • Develop reusable approaches to data modeling, schemas, data contracts, validation, and schema evolution
  • Improve observability, data quality, lineage, and developer experience across the platform
  • Turn recurring delivery challenges into scalable platform capabilities that reduce customer onboarding effort
  • Apply AI to engineering workflows, establishing clear guardrails and measuring its impact on productivity and quality
  • Lead technical design discussions, mentor engineers, and guide architectural decisions and technical trade-offs
  • Document best practices and support the adoption of standard platform patterns across teams
  • Evaluate architectural decisions against time-to-value, cost-to-serve, and delivery predictability

Requirements:

  • Data Platform Engineering: Extensive experience designing, building, and evolving scalable data platforms and production data pipelines using Databricks, Spark, and Spark Lakehouse on Azure (Azure SQL Database, Azure Blob Storage), working with MSSQL databases and infrastructure provisioning via Terraform 
  • Technical Leadership: Experience leading technical design, mentoring engineers, and establishing engineering standards across teams
  • Programming: Strong software engineering skills, with experience developing reusable frameworks, tooling, and automation for data processing (Python, SQL), maintaining high code quality with ruff and pytest 
  • Data Architecture: Strong understanding of data modeling, schema evolution, data contracts, and validation across multiple source systems (MSSQL, Spark Lakehouse), including experience working within and evolving monolithic architectures 
  • Testing and CI/CD: Practical experience implementing automated testing (pytest), linting and code quality checks (ruff), integration testing, quality gates, infrastructure automation (Terraform), and deployment pipelines using GitLab Actions 
  • Platform Reliability: Experience improving observability and monitoring with Grafana, diagnosing operational issues, and building reliable data services on Azure 
  • Data Quality and Lineage: Experience implementing approaches in Databricks and Spark that detect data issues early and make data flows easier to understand and troubleshoot
  • Developer Experience: Ability to simplify engineering workflows (including project and issue tracking in Linear) and turn repeated manual work into reusable, maintainable tools
  • AI-Assisted Engineering: Ability to identify practical AI use cases leveraging tools such as GitHub Copilot and ChatGPT, evaluate their effectiveness, and define safeguards for quality and reliability
  • Architectural Judgment: Ability to balance technical quality, delivery speed, operational cost, and long-term maintainability
  • English Proficiency: C1+ with the ability to communicate technical concepts and architectural decisions clearly
  • Polish Proficiency: C1+ / Native with the ability to collaborate effectively within the local Polish-speaking engineering team 
  • Interpersonal Skills: Strong collaboration skills and the ability to build alignment across engineering teams and domain experts

We offer*:

  • Flexible working format - remote, office-based or flexible
  • A competitive salary and good compensation package
  • Personalized career growth
  • Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
  • Active tech communities with regular knowledge sharing
  • Education reimbursement
  • Memorable anniversary presents
  • Corporate events and team buildings
  • Other location-specific benefits

*not applicable for freelancers

The weekly Digest

The week's most transparent data engineering jobs in your inbox once a week.

Once a week, a few new jobs that state their salary, location and stack, plus links to the rest. No open tracking; clicks are counted per issue, not per person. Unsubscribe with one click.

Similar jobs

N-iX Apply