Data Engineer / Architect

Realising a Hadoop Roadmap

• Responsible for the project budget and the successful realisation of a Hadoop Roadmap with a team of three Data Engineers.
• Developing and presenting a PoC together with a Principal Azure Cloud Architect from Microsoft as a blueprint for the planned migration to Azure.
• Design of the new data lake architecture, conception and implementation of Apache Spark (PySpark) ETL jobs.
• Lecturer of a company-wide Apache Spark 2.x (PySpark) training for the Data Engineering and Analytics team.
• Training the Analytics team with the Apache Spark ML module for large-scale ML scenarios.
• Reengineering of legacy Hadoop MapReduce Java processes by introducing a CI/CD pipeline based on Ansible and Bitbucket together with new build and test scripts.
• Presenting different options for data security on HDFS within the MapR distribution.

Building a data lake on Azure with Databricks 
During the project I was working ~ two days a week for this customer.

• Lead architect for the upcoming data lake in the Azure Cloud.
• Configuration and Setup of Databricks in Azure
• Training of the data team in python and Apache Spark (PySpark) within the Databricks development environment.
• Usage of Databricks delta format for performant update, insert, delete capabilities in the new data lake.
• Developing Apache Spark ETL jobs, Scheduling via Azure Data Factory
• Setting Azure Data Lake Storage Gen2 as Storage Layer for Databricks
• Configuration of an external Hive metastore with a new Azure SQL database.

Implementing a large-scale ML-Pipeline in AWS

• Reengineering of a consisting scikit-learn ML project with Apache Spark.
• Implementing Scala Spark and PySpark jobs for preprocessing, training and prediction.
• Deployment of the model to AWS by managing all related resources and handling the ML pipeline orchestration with Apache Airflow DAGs.
• Enhancing the overall performance by using Scala (User Defined Function) UDFs and sparse vectors.
• Identifying the best performing model via Sparks hyperparameter tuning (Model: random forest classifier).

Building a Hadoop Data Lake

• Implementing ETL jobs with Apache Spark (Scala) and Hive.
• Ensuring the integrity and availability of different data access layers.
• Orchestration and scheduling with Apache Airflow.
• Developing Java-based download clients for heterogeneous data sources.
• Building a Python-based crawler for thousands of websites and ingesting the data into Hive. Additionally returning the top k relevant websites for a given set of n keywords using spaCy and Spark MLlib (Algorithm: tf-idf).
• Construction of an Apache Spark Streaming and Apache Zeppelin prototype. Presenting the prototype while ingesting and transforming live app data in front of 150 attendees (talking in english).

PoC: Prediction of currency fluctuations

• Refactoring and debugging an existing R-based prediction project (Model: deep neural network).
• Conception of a Azure SQL database structure as a prediction sink.
• Implementing a Microsoft Azure Machine Learning pipeline including the data ingestion, R code execution and prediction storage.
• Offering the predictions as a web service for a given day.
• Cooperation with Microsoft as the Azure Machine Learning Studio was in an early release state.

Building a export pipeline to AWS

• Implementing the synchronisation between an on-premise SQL DB and Azure SQL DB with SSIS.
• Converting millions of SQL DB rows to a JSON representation.
• Building a custom SSIS component with C# for the upload to AWS SQS.
• Ensuring the future pipeline maintenance by implementing scheduling, monitoring and auto alarming.

Generating synthetic data

• Design of an automatic anonymisation tool and data security concept in a challenging big data context
  Implementing Apache Spark (Scala) ETL jobs including the randomisation and hashing of personal data to generate synthetic data for machine learning projects
  Ensuring the data integrity with different layers of Apache Hive tables and views
  Configuration of permissions with Apache Sentry for Hive and ACLs for HDFS

Enhancing a Cross-device analytics platform

• Developing ETL jobs with Pentaho Data Integration and Apache Hive.
• Ensuring the daily processing of heterogeneous sources and sinks like Google BigTable and MongoDB.
• Building an automatic validation of the analytical outputs