Data Engineering / Azure Stack / REMOTE (448425)

Beschreibung

• Establish robust and performant pipelines using Python, Apache Spark within the Azure environment using Azure Databricks. The requirements for these are discussed in the sprint meetings.
• Independently implement a data pipeline using Azure Data Factory (ADF) and extract data from two different sources (cosmos DB and SAP Gigya) and merge these two to provide KPIs.
• Independently implement quality checks on incoming data and perform quality assurance tests on the datasets and code
• Independently analyse business requirements that come from Business colleagues participating in the sprint meetings. The analysis includes gathering the parameters from business and extracting this data from different databases and overlaying this to identify any patterns. 
• Consult the project team regarding what parameters can be retrieved and what cannot be retrieved for marketing reports. If any specific reporting cannot be done with existing data, this will be communicated to the Business colleagues who then have to adapt the product.
• Independently create maintenance documentation of the implementation in form of a handover document

• Deep knowledge of MS Azure Cloud Stack
• Several years / projects of experience in this field of activity
• Know how of Python & Apache Spark
• Understanding of the following topics in light of the assignment:  Key Vault, App Service, Container Registry, Container Instances, Kubernetes, MS SQL Server / database, Active Directory, Databricks, Azure Data Factory, Event hub
• Business fluent English, German a nice to have