Beschreibung
Senior Data Engineer
Location: Munich
Duration: 6 months
I am working with a Global Company that is seeking a Data Engineer to architect and build a next-generation internal data platform for large scale data processing.
You are at the intersection of data, engineering, and product, and run the strategy and tactics of how we store and process massive amounts of performance metrics and other data we measure from our customers' database Servers.
Their platform is written in Java and hosted on the AWS cloud. It uses Kafka, Redis, and MySQL for data storage and analysis.
Responsibilities:
- Work with others to define, and propose for approval, a modern data platform design strategy and matching architecture and technology choices to support it, with the goals of providing a highly scalable, economical, observable, and operable data platform for storing and processing very large amounts of data within tight performance tolerances.
- Perform high-level strategy and hands-on infrastructure development for the data platform, developing and deploying new data management services both in our existing data centre infrastructure, and in AWS.
- Collaborate with engineering management to drive data systems design, deployment strategies, scalability, infrastructure efficiency, monitoring, and security.
- Discover, define, document, and design scalable Back End storage and robust data pipelines for different types of data streams.
- Write code, tests, and deployment manifests and artefacts, using CircleCI, Git and GitHub, pull requests, issues, etc. Collaborate with other engineers on code review and approval.
- Measure and improve the code and system performance and availability as it runs in production.
- Support product management in prioritizing and coordinating work on changes to our data platform, and serve as a lead on user-focused technical requirements and analysis of the platform.
- Help provide customer support, and you'll pitch in with other departments, such as Sales, as needed.
- Understand and enact our security posture and practices.
- Continually seek to understand and improve performance, reliability, resilience, scalability, and automation.
- Our goal is that systems should scale linearly with our customer growth, and the effort of maintaining the systems should scale sub-linearly.
- Contribute to a culture of blameless learning, responsibility, and accountability.
- Manage your workload, collaborating and working independently as needed, keeping management appropriately informed of progress and issues.
Key Words:
Data Engineer/Data Science/Java/Python/Cloud/AWS/Spark/Hadoop/Docker/ETL/SQL/MongoDB