Offer summary

Qualifications:

3+ years of experience as a Data Engineer with a focus on data lake architecture and ETL pipeline development., Strong experience with AWS services including S3, Glue, Lambda, and Redshift., Hands-on experience with Airbyte or similar ETL tools and proficiency in Go for data pipeline automation., Solid understanding of SQL, data modeling, and familiarity with scripting languages like Python or Bash..

Key responsibilities:

Architect and maintain robust, scalable, and secure data infrastructure on AWS.

Design, develop, and maintain data pipelines using tools like Airbyte and custom-built services in Go.

Oversee the creation and maintenance of the data lake, ensuring high data quality and effective organization.

Implement best practices for data governance, security, and compliance in AWS, while collaborating with stakeholders to document processes.

Job description

Job Description

About the Role

As a Data Engineer, you will play a key role in building the data infrastructure that powers our data lake on AWS. You’ll be responsible for designing, deploying, and maintaining data pipelines and ensuring a seamless, scalable, and reliable data flow to support our analytics efforts. Leveraging your experience with AWS, Airbyte or similar ELT/ETL tool, and custom solutions in Go, you’ll work closely with our team to turn data into a strategic asset for our company.

Key Responsibilities

Data Infrastructure Setup: Architect and maintain robust, scalable, and secure data infrastructure on AWS.
Data Pipeline Development: Design, develop, and maintain data pipelines, primarily using tools like Airbyte and custom-built services in Go, to automate data ingestion and ETL processes.
Data Lake Management: Oversee the creation and maintenance of the data lake, ensuring efficient storage, high data quality, and effective partitioning, organization, performance, monitoring and alerting.
Integration and Customization: Integrate tools like Airbyte with various data sources and customize data flows to align with specific business needs. Where necessary, build custom connectors in Go to support unique data requirements.
Performance and Scalability: Optimize data pipelines and data lake storage for performance and scalability, ensuring low latency and high availability.
Data Governance and Security: Implement best practices for data governance, security, and compliance in AWS, including access control, encryption, and monitoring.
Collaboration and Documentation: Work closely with platform engineers, data analysts and other stakeholders to understand data requirements and document infrastructure, processes, and best practices.

Required Qualifications

Experience in Data Engineering: 3+ years of experience as a Data Engineer, with a focus on data lake architecture and ETL pipeline development.
AWS Proficiency: Strong experience with AWS services including but not limited to S3, Glue, Lambda, Redshift, and IAM.
ETL Expertise: Hands-on experience with Airbyte or similar ETL tools for data ingestion and transformation.
Proficiency in Go: Experience writing services and connectors in Go, particularly for data pipeline automation.
SQL and Data Modeling: Solid understanding of data modeling, SQL, and database concepts.
Strong Scripting Skills: Familiarity with Python, Bash, or other scripting languages for automation and data manipulation.
Data Governance and Security: Experience implementing security and governance best practices in cloud environments.

Preferred Skills

Containerization and Orchestration: Experience with Docker and Kubernetes is a plus.
Knowledge of Data Analytics Tools: Familiarity with data analytics tools such as Tableau, Looker, or QuickSight.
Experience with Data Warehouses: Familiarity with data warehouse solutions, particularly in the AWS ecosystem (e.g., Redshift, Athena).

Required profile