Principal Data Engineer

Summary

We are looking for an experienced Principal Data Engineer to lead the design, development, and optimization of enterprise-scale data pipelines and modern data platforms supporting Finance and Revenue Data Management Platform (RDMP).

The ideal candidate will have strong hands-on expertise in Snowflake, Delta Lake, Databricks, Delta Live Tables (DLT), Unity Catalog, dbt, and PySpark, along with deep knowledge of data engineering architecture, ETL/ELT pipelines, data modeling, cloud platforms, and performance optimization.

This role requires a combination of hands-on technical expertise, architectural leadership, and the ability to establish engineering standards, mentor senior engineers, and deliver scalable, secure, and cost-efficient data solutions for finance and revenue data.

Key Responsibilities

• Architect, design, and lead the development of enterprise-scale ETL/ELT data pipelines for finance and revenue data, including billing, general ledger (GL), operating expenses, and revenue reporting.
• Establish standards for batch, incremental, and streaming data ingestion using Change Data Capture (CDC), watermarking, and event-driven processing.
• Design scalable, idempotent, fault-tolerant, and high-performance data pipelines across enterprise platforms.
• Lead the architecture and optimization of Snowflake and Databricks platforms for large-scale data processing and analytics.
• Implement and optimize Delta Lake, Delta Live Tables (DLT), and Unity Catalog for data processing, governance, and secure data access.
• Develop modular, reusable, and maintainable data transformation models using dbt, including testing and CI/CD integration.
• Build and optimize distributed data processing solutions using Python, PySpark, and advanced SQL.
• Establish orchestration standards using Apache Airflow and/or Azure Data Factory (ADF).
• Design and implement real-time and near-real-time data processing solutions using Kafka, Azure Event Hubs, and Spark Structured Streaming.
• Establish enterprise frameworks for data quality, validation, observability, reconciliation, and SLA monitoring.
• Optimize query performance, compute utilization, storage, partitioning, clustering, and overall platform costs.
• Implement CI/CD pipelines and engineering best practices using GitHub Actions and/or Azure DevOps.
• Ensure data security, governance, lineage, and auditability through RBAC, row-level and column-level security, and appropriate protection of PII and CPNI.
• Support enterprise data modeling, including star schemas, snowflake schemas, Data Vault, and Slowly Changing Dimensions (SCD Type 1/2).
• Collaborate with finance and business stakeholders to implement reliable data solutions for revenue recognition, reconciliation, financial reporting, and period-end close processes.
• Lead architecture and design reviews, establish reusable engineering frameworks, and ensure adherence to technical standards and non-functional requirements.
• Mentor senior engineers and technical leads, resolve complex technical challenges, and drive modernization and continuous improvement initiatives.

Required Qualifications

• 7+ years of experience with Snowflake and Delta Lake.
• 7+ years of experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• 5+ years of experience with dbt and PySpark.
• Strong hands-on experience designing and developing enterprise-scale ETL/ELT pipelines and distributed data processing solutions.
• Advanced proficiency in SQL, Python, PySpark, and data transformation techniques.
• Strong understanding of data lakehouse architecture, data modeling, incremental processing, CDC, and pipeline optimization.
• Experience with data orchestration, workflow scheduling, and dependency management.
• Strong knowledge of data quality frameworks, automated testing, observability, and production monitoring.
• Experience implementing CI/CD, version control, code reviews, and automated deployment practices.
• Understanding of data security, access controls, governance, lineage, and compliance requirements.
• Demonstrated ability to lead technical design decisions, establish engineering standards, and mentor engineering teams.
• Strong analytical, problem-solving, communication, and stakeholder management skills.
• Willingness to work onsite 3 days per week in Bellevue, WA, or Frisco, TX.

Preferred Qualifications

• Experience with Azure Data Factory, ADLS Gen2, Azure Event Hubs, and Azure Key Vault.
• Experience with Apache Airflow, Kafka, and Spark Structured Streaming.
• Experience with Terraform or Bicep for infrastructure as code.
• Experience with GitHub Actions and Azure DevOps for CI/CD automation.
• Familiarity with data quality tools such as Great Expectations and dbt testing frameworks.
• Experience with financial and revenue data domains, including billing, general ledger, revenue recognition, and reconciliation.
• Experience optimizing cloud data platforms for performance, scalability, reliability, and cost efficiency.
• Experience defining enterprise data engineering architecture, reusable design patterns, and platform standards.
• Experience mentoring senior engineers and collaborating with architects, product managers, and business stakeholders.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.o lead the design, development, and optimization of enterprise-scale data pipelines and modern data platforms supporting Finance and Revenue Data Management Platform (RDMP).

Job Description

We are looking for an experienced Principal Data Engineer to lead the design, development, and optimization of enterprise-scale data pipelines and modern data platforms supporting Finance and Revenue Data Management Platform (RDMP).

The ideal candidate will have strong hands-on expertise in Snowflake, Delta Lake, Databricks, Delta Live Tables (DLT), Unity Catalog, dbt, and PySpark, along with deep knowledge of data engineering architecture, ETL/ELT pipelines, data modeling, cloud platforms, and performance optimization.

This role requires a combination of hands-on technical expertise, architectural leadership, and the ability to establish engineering standards, mentor senior engineers, and deliver scalable, secure, and cost-efficient data solutions for finance and revenue data.

Key Responsibilities

• Architect, design, and lead the development of enterprise-scale ETL/ELT data pipelines for finance and revenue data, including billing, general ledger (GL), operating expenses, and revenue reporting.
• Establish standards for batch, incremental, and streaming data ingestion using Change Data Capture (CDC), watermarking, and event-driven processing.
• Design scalable, idempotent, fault-tolerant, and high-performance data pipelines across enterprise platforms.
• Lead the architecture and optimization of Snowflake and Databricks platforms for large-scale data processing and analytics.
• Implement and optimize Delta Lake, Delta Live Tables (DLT), and Unity Catalog for data processing, governance, and secure data access.
• Develop modular, reusable, and maintainable data transformation models using dbt, including testing and CI/CD integration.
• Build and optimize distributed data processing solutions using Python, PySpark, and advanced SQL.
• Establish orchestration standards using Apache Airflow and/or Azure Data Factory (ADF).
• Design and implement real-time and near-real-time data processing solutions using Kafka, Azure Event Hubs, and Spark Structured Streaming.
• Establish enterprise frameworks for data quality, validation, observability, reconciliation, and SLA monitoring.
• Optimize query performance, compute utilization, storage, partitioning, clustering, and overall platform costs.
• Implement CI/CD pipelines and engineering best practices using GitHub Actions and/or Azure DevOps.
• Ensure data security, governance, lineage, and auditability through RBAC, row-level and column-level security, and appropriate protection of PII and CPNI.
• Support enterprise data modeling, including star schemas, snowflake schemas, Data Vault, and Slowly Changing Dimensions (SCD Type 1/2).
• Collaborate with finance and business stakeholders to implement reliable data solutions for revenue recognition, reconciliation, financial reporting, and period-end close processes.
• Lead architecture and design reviews, establish reusable engineering frameworks, and ensure adherence to technical standards and non-functional requirements.
• Mentor senior engineers and technical leads, resolve complex technical challenges, and drive modernization and continuous improvement initiatives.

Required Qualifications

• 7+ years of experience with Snowflake and Delta Lake.
• 7+ years of experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• 5+ years of experience with dbt and PySpark.
• Strong hands-on experience designing and developing enterprise-scale ETL/ELT pipelines and distributed data processing solutions.
• Advanced proficiency in SQL, Python, PySpark, and data transformation techniques.
• Strong understanding of data lakehouse architecture, data modeling, incremental processing, CDC, and pipeline optimization.
• Experience with data orchestration, workflow scheduling, and dependency management.
• Strong knowledge of data quality frameworks, automated testing, observability, and production monitoring.
• Experience implementing CI/CD, version control, code reviews, and automated deployment practices.
• Understanding of data security, access controls, governance, lineage, and compliance requirements.
• Demonstrated ability to lead technical design decisions, establish engineering standards, and mentor engineering teams.
• Strong analytical, problem-solving, communication, and stakeholder management skills.
• Willingness to work onsite 3 days per week in Bellevue, WA, or Frisco, TX.

Preferred Qualifications

• Experience with Azure Data Factory, ADLS Gen2, Azure Event Hubs, and Azure Key Vault.
• Experience with Apache Airflow, Kafka, and Spark Structured Streaming.
• Experience with Terraform or Bicep for infrastructure as code.
• Experience with GitHub Actions and Azure DevOps for CI/CD automation.
• Familiarity with data quality tools such as Great Expectations and dbt testing frameworks.
• Experience with financial and revenue data domains, including billing, general ledger, revenue recognition, and reconciliation.
• Experience optimizing cloud data platforms for performance, scalability, reliability, and cost efficiency.
• Experience defining enterprise data engineering architecture, reusable design patterns, and platform standards.
• Experience mentoring senior engineers and collaborating with architects, product managers, and business stakeholders.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.o lead the design, development, and optimization of enterprise-scale data pipelines and modern data platforms supporting Finance and Revenue Data Management Platform (RDMP).