Beschreibung
Freelance Data Engineer (AWS, Spark/Scala, Apache) *no agencies / consultancies*September - EOY
Location: Remote / 2 days per month Frankfurt
Language: German
# Project description
The services of the contractor (service provider) are provided for platforms of the CIO area.
This data lake platform allows the high-performance collection, enrichment, linking, refinement and evaluation of data from a wide variety of sources and opens up comprehensive improvement potential for data use in all areas.
With this platform, the basis for big data analysis and data science is created, which can create valuable insights or new business models. The platform is implemented on the basis of current standard technologies (AWS Serverless) and architectures and can be expanded as required and scalable as required.
# Service description
• Conception and data flows using AWS technologies and Apache
• Spark/Scala
• Description, coordination and documentation of functional and non-functional requirements in the form of epics, features and user stories
• Creation of technical documents (e.g. use cases / user stories, product backlog, functional system descriptions, technical service descriptions, technical interface descriptions)
• Consulting, implementation and conception of the technical requirements for the IT systems as well as derivation and development of technical implementation possibilities.
• Creation and evaluation of functional specifications
• Documentation of requirements in Jira
• Create, tune, and run test cases
• Organisation and implementation of workshops
•
# Must requirements:*
• The external specialist brings extensive project experience (at least 3 projects) with at least 3 ETL tools (Spark, Nifi, Jupyter,...), which he can prove within the framework of each reference project.
• The external specialist brings many years of architecture and implementation knowledge to AWS
• Stack (e.g. ECS, EMR, Athena, Glue, Redshift, Kinesis or MSK, Lambda, Python), which he can prove on the basis of the last three projects.
• The external specialist has many years of architecture and implementation knowledge of relational databases, SQL, ETL and BI tools, which he can prove on the basis of the last three projects.
• The specialist has very good knowledge of programming with Apache Spark and Scala, which he can prove on the basis of three projects in the last three years