Offer summary

Qualifications:

7+ years of experience in data engineering with a focus on cloud-native data architecture., Deep hands-on experience with Databricks architecture and workspace administration., Strong knowledge of Spark performance tuning and job optimization., Proven expertise in Databricks SQL, PySpark, and building reusable Python libraries..

Key responsabilities:

Lead the architecture and development of scalable ETL/ELT pipelines using Databricks and PySpark.

Collaborate with infrastructure teams to define Databricks architecture strategy and ensure secure implementation.

Drive DevOps best practices using Azure DevOps and CI/CD automation pipelines.

Mentor junior engineers and perform architectural reviews to ensure alignment with best practices.

Job description

APN Consulting, Inc. is a progressive IT staffing and services company offering innovative business solutions to improve client business outcomes. We focus on high impact technology solutions in ServiceNow, Fullstack, Cloud & Data, and AI / ML. Due to our globally expanding service offerings we are seeking top-talent to join our teams and grow with us.

Role: Lead/Architect Data Engineer
Location: Remote (US based-remote anywhere in US)
Duration: Contract

Job Summary

We are looking for a results-driven Lead Data Engineer (Contractor) to architect, develop, and guide the implementation of modern data pipelines and cloud-native analytics solutions. The ideal candidate will lead end-to-end delivery across engineering, analytics, and product teams, bringing deep experience in Databricks, PySpark, and Azure cloud platforms. This role also requires strong hands-on experience in Databricks architecture, administration, and performance optimization .

Key Responsibilities

Lead the architecture, design, and development of scalable ETL/ELT pipelines using Databricks, Pyspark, and SQL across distributed data environments.
Architect and manage Databricks workspaces, including provisioning and maintenance of clusters, cluster policies, and job compute environments in accordance with enterprise standards.
Collaborate with platform and infrastructure teams to define Databricks architecture strategy and ensure secure, scalable, and cost-effective implementation.
Define and enforce cluster policies to ensure proper resource utilization, cost control, and access control based on workload patterns and team requirements.
Lead performance tuning of Spark jobs, Databricks SQL queries, and notebooks, ensuring optimal execution and minimizing latency.
Build modular, reusable Python libraries using Pandas, NumPy, and PySpark for scalable data processing.
Develop optimized Databricks SQL queries and views to powerTableau dashboards
1. React and .NET-based applications
2. Ad-hoc and real-time analytics use cases
3. Work closely with frontend and backend development teams to deliver use-case-specific, query-optimized datasets.
Leverage Unity Catalog for fine-grained access control, data lineage, and metadata governance.
Drive DevOps best practices using Azure DevOps, Terraforms, and CI/CD automation pipelines.
Mentor junior engineers and perform architectural reviews to ensure consistency and alignment with best practices.

Required Skills & Qualifications

7+ years of experience in data engineering, with a strong background in cloud-native data architecture.
Deep hands-on experience with Databricks architecture, workspace administration, and cluster management.
Experience defining and managing cluster policies, pools, and autoscaling strategies.
Strong knowledge of Spark performance tuning and job optimization.
Proven expertise in Databricks SQL, PySpark, Delta Lake, and large-scale data pipelines.
Skilled in building reusable Python libraries with Pandas, Openpyxl, XLSXWriter, and PySpark.
Practical experience working with Unity Catalog for security and governance.
Strong collaboration experience with front-end/backend development teams and backend integration.
Strong SQL expertise and hands-on experience with PostgreSQL, SQL Server, or similar.
DevOps expertise with tools like Azure DevOps, Git, and pipeline automation.
Excellent communication skills with the ability to lead discussions with cross-functional teams and stakeholders.

Tools & Technologies

Cloud Platforms: Azure (preferred), AWS
Big Data & Analytics: Databricks, PySpark, Delta Lake, Databricks SQL, Spark Connect, Delta Live Tables
Programming & Frameworks: Python, Pandas, PySpark, Flask
Visualization & BI: Tableau
App Integration: React, .NET, REST APIs
DevOps & CI/CD: Azure DevOps, Git
Databases: Databricks SQL, Azure SQL DB, or similar

We are committed to fostering a diverse, inclusive, and equitable workplace where individuals from all backgrounds feel valued and empowered to contribute their unique perspectives. We strongly encourage applications from candidates of all genders, races, ethnicities, abilities, and experiences to join our team and help us build a culture of belonging.

Required profile