Principal Data Engineer

Summary

We are looking for an experienced Principal Data Engineer – Data & Intelligence to lead the design, development, and optimization of enterprise-scale data platforms and pipelines supporting finance and revenue data for Finance / RDMP initiatives.

The ideal candidate will have strong hands-on expertise in Snowflake, Delta Lake, Databricks, Delta Live Tables (DLT), Unity Catalog, dbt, and PySpark. The candidate should have extensive experience in data pipeline architecture, ETL/ELT development, cloud-native data platforms, data governance, data quality, and performance optimization.

This role requires technical leadership in designing scalable, secure, and reliable data solutions, establishing engineering standards, mentoring senior engineers, and driving platform modernization. Experience with financial data domains, including billing, general ledger (GL), revenue recognition, and reconciliation, is beneficial.

Key Responsibilities

• Architect and develop scalable ETL/ELT data pipelines for enterprise finance and revenue data using Snowflake, Databricks, Delta Lake, and PySpark.
• Establish data ingestion, transformation, orchestration, and real-time processing solutions using DLT, dbt, Kafka, Airflow, and Azure Data Factory.
• Optimize data platforms, SQL queries, pipeline performance, cloud infrastructure, and operational costs.
• Implement data quality, automated testing, monitoring, security, governance, and compliance standards.
• Design and optimize enterprise data models while ensuring data accuracy, reliability, and consistency.
• Establish CI/CD pipelines, reusable frameworks, coding standards, and deployment best practices.
• Perform production troubleshooting, root cause analysis, defect resolution, and continuous improvement.
• Lead technical architecture and design decisions, mentor senior engineers, and collaborate with cross-functional teams and business stakeholders.

Required Qualifications

• 7+ years of experience with Snowflake and Delta Lake.
• 7+ years of experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• 5+ years of experience with dbt and PySpark.
• Strong expertise in data engineering, ETL/ELT pipelines, and enterprise-scale data architecture.
• Advanced SQL skills, including query optimization and complex transformations.
• Experience with data modeling, data integration, and distributed data processing.
• Strong understanding of data quality, automated testing, monitoring, and observability.
• Experience with CI/CD, version control, deployment practices, and production support.
• Knowledge of data security, governance, access controls, and compliance requirements.
• Strong technical leadership, problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

• Experience with cloud platforms such as Microsoft Azure or AWS.
• Experience with Azure Data Factory, ADLS Gen2, Event Hub, and Key Vault.
• Experience with Apache Kafka and Spark Structured Streaming.
• Familiarity with Airflow, Terraform, and Bicep.
• Experience with GitHub Actions and Azure DevOps.
• Familiarity with Great Expectations and enterprise data observability tools.
• Experience in finance and revenue data domains, including billing, GL, reconciliation, and revenue recognition.
• Experience leading enterprise data platform modernization and mentoring senior engineers.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.

Job Description

We are looking for an experienced Principal Data Engineer – Data & Intelligence to lead the design, development, and optimization of enterprise-scale data platforms and pipelines supporting finance and revenue data for Finance / RDMP initiatives.

The ideal candidate will have strong hands-on expertise in Snowflake, Delta Lake, Databricks, Delta Live Tables (DLT), Unity Catalog, dbt, and PySpark. The candidate should have extensive experience in data pipeline architecture, ETL/ELT development, cloud-native data platforms, data governance, data quality, and performance optimization.

This role requires technical leadership in designing scalable, secure, and reliable data solutions, establishing engineering standards, mentoring senior engineers, and driving platform modernization. Experience with financial data domains, including billing, general ledger (GL), revenue recognition, and reconciliation, is beneficial.

Key Responsibilities

• Architect and develop scalable ETL/ELT data pipelines for enterprise finance and revenue data using Snowflake, Databricks, Delta Lake, and PySpark.
• Establish data ingestion, transformation, orchestration, and real-time processing solutions using DLT, dbt, Kafka, Airflow, and Azure Data Factory.
• Optimize data platforms, SQL queries, pipeline performance, cloud infrastructure, and operational costs.
• Implement data quality, automated testing, monitoring, security, governance, and compliance standards.
• Design and optimize enterprise data models while ensuring data accuracy, reliability, and consistency.
• Establish CI/CD pipelines, reusable frameworks, coding standards, and deployment best practices.
• Perform production troubleshooting, root cause analysis, defect resolution, and continuous improvement.
• Lead technical architecture and design decisions, mentor senior engineers, and collaborate with cross-functional teams and business stakeholders.

Required Qualifications

• 7+ years of experience with Snowflake and Delta Lake.
• 7+ years of experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• 5+ years of experience with dbt and PySpark.
• Strong expertise in data engineering, ETL/ELT pipelines, and enterprise-scale data architecture.
• Advanced SQL skills, including query optimization and complex transformations.
• Experience with data modeling, data integration, and distributed data processing.
• Strong understanding of data quality, automated testing, monitoring, and observability.
• Experience with CI/CD, version control, deployment practices, and production support.
• Knowledge of data security, governance, access controls, and compliance requirements.
• Strong technical leadership, problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

• Experience with cloud platforms such as Microsoft Azure or AWS.
• Experience with Azure Data Factory, ADLS Gen2, Event Hub, and Key Vault.
• Experience with Apache Kafka and Spark Structured Streaming.
• Familiarity with Airflow, Terraform, and Bicep.
• Experience with GitHub Actions and Azure DevOps.
• Familiarity with Great Expectations and enterprise data observability tools.
• Experience in finance and revenue data domains, including billing, GL, reconciliation, and revenue recognition.
• Experience leading enterprise data platform modernization and mentoring senior engineers.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.