Skip to content
getdataengineeringjobs
Back to results

Senior Databricks Consultant / Architect

Unison Group
Singapore, Singapore, Singapore Senior Posted

Summary

Written in English from the original posting

The role designs and implements enterprise-scale Databricks Lakehouse data engineering and analytics solutions on Azure. It involves building batch and streaming ETL/ELT pipelines, applying Medallion Architecture, and using Unity Catalog for governance. The position also provides technical leadership, CI/CD deployment, and mentoring for data engineering teams.

Job description

Role Overview

We are looking for an experienced Senior Databricks Consultant / Architect to lead the design, architecture, development, and implementation of enterprise-scale data engineering and analytics solutions using Azure Databricks. The ideal candidate should have strong expertise in Lakehouse architecture, PySpark, Delta Lake, cloud data platforms, and data governance.

Key Responsibilities

  • Design and architect scalable, secure, and high-performance Lakehouse solutions using Databricks.
  • Lead end-to-end data engineering and data platform implementation.
  • Define enterprise data architecture, standards, best practices, and technical roadmaps.
  • Design and implement data pipelines using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
  • Develop and optimize ETL/ELT pipelines for batch and streaming data processing.
  • Implement Medallion Architecture (Bronze, Silver, Gold) and Delta Live Tables (DLT).
  • Work extensively with Unity Catalog for data governance, security, access control, and lineage.
  • Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Event Hubs, and Azure Key Vault.
  • Optimize Spark workloads, cluster configurations, SQL queries, and overall platform performance.
  • Implement CI/CD pipelines, automated deployments, and DevOps practices.
  • Provide technical leadership, conduct design reviews, and mentor data engineering teams.
  • Collaborate with business stakeholders, architects, and engineering teams to translate business requirements into technical solutions.

Mandatory Technical Skills

  • Strong hands-on experience with Azure Databricks and Apache Spark.
  • Expertise in PySpark, Spark SQL, Python, and SQL.
  • Strong knowledge of Delta Lake and Lakehouse architecture.
  • Experience implementing Medallion Architecture.
  • Expertise in Unity Catalog, Delta Live Tables (DLT), and Auto Loader.
  • Strong understanding of ETL/ELT design and data pipeline development.
  • Experience with Azure Data Lake Storage (ADLS Gen2) and Azure Data Factory.
  • Knowledge of Databricks Workflows, cluster management, and performance tuning.
  • Experience with data security, governance, and access management.
  • Hands-on experience with Git, Azure DevOps, and CI/CD.

Good to Have

  • Experience with AWS Databricks or GCP Databricks.
  • Knowledge of Kafka, Event Hubs, and real-time data streaming.
  • Experience with Databricks SQL and Lakehouse Monitoring.
  • Knowledge of Terraform and Infrastructure as Code.
  • Experience with MLflow and the Databricks Machine Learning ecosystem.
  • Databricks Professional-level certifications.

Candidate Profile

  • Strong experience in enterprise data architecture and large-scale data engineering implementations.
  • Ability to independently drive architecture decisions and technical delivery.
  • Excellent communication, stakeholder management, and problem-solving skills.
  • Ability to work with cross-functional teams and provide technical direction.

The weekly Digest

The week's most transparent data engineering jobs in your inbox once a week.

Once a week, a few new jobs that state their salary, location and stack, plus links to the rest. No open tracking; clicks are counted per issue, not per person. Unsubscribe with one click.

Similar jobs

Unison Group Apply