Lead - Data Engineer

• Lead the design, development, and delivery of core platform components, including the Constellation Lakehouse in Databricks, Connect API (GraphQL) layer, and Lakebase-backed applications.
• Build and maintain scalable batch and incremental data pipelines using Databricks, Apache Spark, PySpark, SQL, and Delta Lake.
• Integrate data from transactional databases, APIs, files, and external partners into governed datasets that support analytics and operational applications.
• Design and maintain unified healthcare data models spanning membership, providers, claims, encounters, clinical data, and other business domains.
• Develop reliable ingestion and transformation processes, including change data capture, schema evolution, deduplication, historical tracking, and reconciliation.
• Build and optimize data access layers and APIs in partnership with software engineering teams, supporting consistent business definitions and reliable downstream consumption.
• Develop and support Lakebase-backed applications and PostgreSQL data structures, including integration with lakehouse datasets and application services.
• Maintain and optimize Databricks workspaces, compute resources, job orchestration, and environment configurations in collaboration with infrastructure and security teams.
• Implement data governance and access controls through Unity Catalog and established organizational standards, including permissions, lineage, and auditability.
• Establish automated data quality checks, source-to-target reconciliation, freshness monitoring, and alerting to ensure datasets are accurate, complete, and usable.
• Apply DevOps practices, including Git, automated testing, CI/CD, infrastructure as code, and repeatable deployment across development, testing, and production environments.
• Tune pipelines, Spark workloads, SQL queries, and compute configurations to improve performance, scalability, and cost efficiency.
• Own the investigation and resolution of platform incidents and pipeline failures; perform root-cause analysis and implement preventive improvements.
• Partner with infrastructure teams on backup, recovery, and business continuity procedures, and maintain operational documentation and runbooks.
• Conduct code and design reviews, establish engineering standards, and mentor engineers across internal and vendor teams.

• Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field.
• 5+ years of data engineering experience building and supporting enterprise data platforms and pipelines.
• 3+ years of hands-on experience with Databricks in production environments.
• Strong proficiency in Python, PySpark, SQL, and Apache Spark.
• Experience with Delta Lake and modern lakehouse architectures.
• Hands-on experience with Azure data services and cloud-based data platforms.
• Strong knowledge of data modeling, relational databases, and ETL/ELT best practices.
• Experience integrating enterprise databases and APIs into scalable data solutions.
• Experience with Git, CI/CD pipelines, and automated deployment practices.
• Proven ability to monitor, troubleshoot, and optimize data pipelines and platform performance.
• Demonstrated leadership in solution design, code reviews, mentoring, and cross-functional collaboration.
• Strong problem-solving and communication skills.

Preferred Qualifications

• Experience administering and optimizing Databricks environments, including Unity Catalog.
• Experience with PostgreSQL, GraphQL APIs, and operational applications leveraging lakehouse data.
• Healthcare data experience, including claims, eligibility, provider, or clinical datasets.
• Familiarity with healthcare interoperability standards such as HL7, FHIR, and X12.
• Experience with Terraform or other infrastructure-as-code tools.
• Experience with streaming data pipelines and event-driven architectures.
• Experience supporting AI, machine learning, or NLP data workloads.
• Familiarity with regulated data environments and governance best practices.
• Experience with cloud data platform migrations and modernization initiatives.
• Azure, Databricks, or AWS certifications preferred.

• This position is remotely based in the U.S.
• The total compensation target pay range for this role is: $170,000 - $240,000. Actual compensation will be determined based on geographic location (current or future), experience, and other job-related factors.