Job Description: Data Engineer Role Summary The Data Engineer role requires strong development and analytical skills. You are responsible for helping the team achieve the companys mission and vision by creating data pipelines that deliver near real time and accurate raw data, analytics and reports for our customers.
Roles and Responsibilities Implement high-quality data pipelines following industry standards and best practices using AWS services Consistently optimize application performance to ensure reports and analytics are always available in near real time Design and deliver highly available, scalable and fault tolerant data pipelines Build prototypes to prove out key design concepts and quantify technical constraints Ensure application security on every layer of the application Brainstorm and contribute ideas during product planning sessions and product solutioning Provide full documentation of delivered requirements Utilise Cloud and DevOps tools and technologies such as configuration management, automation, continuous integration, build, code repository, containerization, etc Communicate and collaborate with Product, QA and IT Ops teams to ensure quality, usability and performance of the applications Provide regular feedback with other team members for better collaboration Take ownership of the delivered features and applications Deploy patches and updates for platforms in production
Skills & Qualifications Must have production experience working with in-stream, big data processing and analytics using Apache Kafka, Apache Spark, Apache Spark Structured Streaming, Apache Flink, Kafka SQL or similar GIT, Design Patterns Familiarity with AWS cloud Infrastructure Experience with data exploration tools like Grafana, Jupyter Notebooks, Metabase, Amazon Quicksight, PowerBI or similar
Data Modelling and ERD Database technologies such as MySQL, MongoDB Good understanding of statistics, machine learning algorithms and frameworks NodeJS JIRA or similar Microservices Application performance monitoring tools Architecture Design
Strong problem solving skills with efficient ability to identify and troubleshoot problems Ability to interact, collaborate and prioritize with internal teams Highly self-motivated, goal-oriented and results-driven Proactive and can work independently with minimal supervision Excellent oral and written communication skills and fluent in English Bachelors Degree in Computer Science, IT, IS, data science, engineering, math 2+ Years of data engineering experience
M3 Wake Research
EPAM Systems
Autodesk
TecAlliance
SentinelOne