Amazon Web Services Databricks Engineer

Overview 

Join a team where innovation meets mission. Our AI, cloud, cyber, and modernization solutions save agencies thousands of hours, safeguard national security, and strengthen health and humanitarian missions worldwide. With 1,700+ team members, 1,500+ AI/data experts, and 100+ prime contracts, we deliver at scale and with purpose. 

We’ve been recognized as a Top Workplace by the Washington Post for six straight years and named to the Inc. 5000 Fastest Growing Private Companies 13 of the past 14 years. Credence is a welcoming home for those looking to grow and contribute to positive change. We encourage all employees to expand beyond their boundaries and dive into important, world-changing federal challenges. 

Position Summary 

Credence has an immediate need for an AWS Databricks Engineer to join our growing AI and Automation practice. You will serve as a key technical contributor responsible for designing, building, and optimizing scalable cloud-based data solutions using AWS and Databricks. You’ll develop robust data pipelines, implement modern data lakehouse architectures, and collaborate across engineering, data science, and stakeholder teams to deliver secure, high-performance data capabilities that advance federal missions. 

Responsibilities include, but are not limited to, the duties listed below: 

• Databricks Engineering 
  Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake. 
• Data Lakehouse Architecture 
  Design and implement modern lakehouse solutions using Databricks and AWS services. Apply medallion architecture (Bronze/Silver/Gold layers) to improve data quality, governance, lineage, performance, and accessibility. 
• AWS Data Integration 
  Build and maintain ETL/ELT pipelines integrating structured, semi-structured, and unstructured data across AWS services such as S3, Glue, RDS, Redshift, Lambda, and related cloud data platforms. 
• Data Transformation & Optimization 
  Develop efficient data transformation workflows using SQL, Python, and PySpark. Optimize Spark workloads, queries, clusters, and storage configurations for performance, reliability, and cost efficiency. 
• Data Governance & Quality 
  Implement data quality, cataloging, lineage, governance, and access-control practices across Databricks and AWS environments, including the use of Unity Catalog where applicable. 
• Cloud Automation & DevOps 
  Automate data platform deployments and workflows using Infrastructure as Code (IaC), CI/CD pipelines, Git-based development practices, and AWS-native deployment tools. 
• Collaborative Engineering 
  Work closely with data engineers, software engineers, data scientists, cloud engineers, and mission stakeholders to translate business and technical requirements into production-ready data solutions. 
• Technical Rigor & Documentation 
  Develop clean, maintainable, and well-documented code while following engineering, security, and federal development standards. Support code reviews, testing, troubleshooting, and technical design documentation. 

Requirements

• U.S. citizenship with the ability to obtain and maintain a Secret clearance is required. 
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical field. 
• Hands-on experience in data engineering, cloud engineering, or related technical roles. 
• Hands-on experience building and maintaining production data solutions using Databricks, Apache Spark, and PySpark. 
• Strong experience with AWS data services such as S3, Glue, RDS, Redshift, Lambda, IAM, and CloudWatch. 
• Strong understanding of ETL/ELT processes, data lake and lakehouse architectures, data modeling, and large-scale data processing. 
• Hands-on experience with Delta Lake, SQL, Python, and Spark-based data processing. 
• Experience developing, scheduling, monitoring, and troubleshooting production data pipelines and workflows. 
• Experience with Git and CI/CD platforms such as GitLab, GitHub Actions, Jenkins, or AWS CodePipeline. 
• Understanding of cloud security, identity and access management, encryption, and secure data-handling practices. 
• Strong communication skills and a client-oriented mindset with the ability to collaborate across technical and non-technical teams. 

Preferred 

• Experience implementing Databricks Lakehouse Architecture and medallion architecture using Bronze, Silver, and Gold data layers. 
• Experience with Databricks Unity Catalog, Delta Live Tables, Databricks Workflows, or similar data orchestration and governance capabilities. 
• Strong Python proficiency and experience with data engineering libraries and frameworks such as PySpark, Pandas, and Pydantic. 
• Experience with Infrastructure as Code tools such as Terraform, OpenTofu, AWS CDK, or CloudFormation. 
• AWS and/or Databricks certifications, such as AWS Certified Data Engineer - Associate, AWS Certified Solutions Architect, or Databricks Certified Data Engineer credentials. 
• Knowledge of federal cybersecurity, RMF, FedRAMP, or other government regulatory and compliance frameworks. 
• Experience delivering cloud-native data solutions within federal government or other highly regulated environments. 

Salary Range: $155,000 - $185,000 annually. Actual compensation will be determined based on the selected candidate's experience, education, certifications, skills, and overall qualifications.

Benefits

• Health Care Plan (Medical, Dental & Vision)
• Retirement Plan (401k, IRA)
• Life Insurance (Basic, Voluntary & AD&D)
• Paid Time Off (Vacation, Sick & Public Holidays)
• Family Leave (Maternity, Paternity)
• Short Term & Long Term Disability
• Training & Development
• Wellness Resources