Big Data & Cloud Consultant / Developer

• Hands-On Migrate ETL Pipelines and Machine Learning Workloads from MySQL / Python to pySpark + Airflow to leverage scalability in the cloud
• Hands-On implement Microservices & APIs to internal Teams to leverage Cloud Data (Kubernetes, Spring Boot)
• Move SQL Datawarehouse to Cloud-based VDMs with Databricks Delta
• Hands-On Implement Java-based tooling to support Delta Tables on AWS S3
• Hands-On Implement SCD and timetravel on User Journey Data
• Provide Interfaces to Data to QlikSense & KPI Teams
• Report to Vice-President Data & Steering Committee
• Conduct Employee Trainings: Airflow, Spark, AWS

Key Skills: Airflow, Python, Java, Kafka, Spark, AWS,
Terraform, Gitlab CI

• Lead team of external contractors to migrate manual AWS setup to entirely Terraform-based scripts + deployment pipelines and adopt KMS based encryption
• Enable rapid Infrastructure deployment with Terraform, implement customized cluster autoscaling services for EMR in Python
• Hands-On Implement AWS Organization and switch from AD to AWS SSO
• Spearhead Adoption of Airflow on AWS MWAA
• Achieve AWS Cost reduction using transient clusters and Saving Plans
• Provide JupyterHub to Data Analysts
• Hands-On Implement KMS Encryption
• Present Project Progress and Cost Forecasts to C-Level Deciders
• Conduct Employee Trainings: Airflow, Spark, AWS

Key Skills: Airflow, Python, Java, Spark, Kafka, EMR, Redis, Docker, Kubernetes, Helm Charts, Gitlab, Gitlab CI

• Hands-on-implementation of Data Pipelines in Java + Spark and Python + pySpark
• Lead adoption of Airflow over Talend
• Hands-on-implementation of Airflow at Sovendus, provide Kubernetes based high-availability setup with scalable worker nodes
• Hands-on-implementation of custom Framework in Python + Java to support development of new ETL pipelines
• Migrate Machine Learning workloads to the cloud
• Introduce Confluent Enterprise cluster with Kafka Connect, implemnt custom SMTs and Partitioners in Java, introduce Schema registry for Data validation
• Conduct Employee Trainings: Python, Testing, Kafka

Key Skills: Spark, Scala, Java, Maven, Cassandra, SAP HANA, SAP Cloud

• Consulting to help improve processing times of an existing Data Pipeline to process streaming data and calculate aggregates on timeseries data using Apache Spark
• Deduct trainings for teams on Structured Streaming using Apache Spark
• Implement Metering / Billing for the existing ingestion pipeline and deploy to production
• Consult on topics such as multi-tenancy, high availability and hyperscaling archtectures for streaming data on AWS/Azure
• Hands-on-implementation for a Spark streaming ingestion to Cassandra PoC
• Hands-on-implementation for a Spark aggregate calculation to Hana PoC
• Use technologies such as AWS ElastiCache, AWS EMR, AWS Kinesis to build a streaming event processing pipeline PoC for the cloud
• Use technologies such as CloudFoundry, HANA and Spark Job Server for the current production landscape

Key Skills: Spark, Java, Maven, Azure, Kubernetes, Spring, Spring Boot

• Consulting on Big Data Architecture and Data Pipeline to cluster and transform large amounts of unstructured data to analyze  and process credit card transactions
• Design access patterns to Enterprise Data Lake based on Cloudera Enterprise Stack (Kafka, Spark, HBase, Hadoop, Solr) in the Azure Cloud.
• Design and implement ingestion pipeline using Spark Jobs for transaction processing of incoming credit card data (DataIn) including duplicate flagging and fraud detection.
• Definition of the REST-API Interface for the Frontend for retrieving billing information (eBilling)
• Design architecture and interfaces to invoicing (SAP), credit card handling (TSYS) and secure storage (nScale Vecto) for secure invoice download using the Frontend (Portal)

• Consulting on Atlassian JIRA, Workflow Design, Security/Permission Scheming and Plugin Development for the Atlassian Platform
• Implementation of heavily customized IT Security Process with custom Plugins and Workflow Scripting for 2000+ users
• Implement Preprocessing Platform based on Python/Spark/Kafka with filtering, ranking, clustering of Reports from several data sources and authoring a connector to store the security scans within JIRA as Tickets
• Customized Oracle SSL Connector for JIRA to use encrypted Oracle 11g w/ SSL as Backend
• LDAP & Identity IQ Connector for Enterprise-compliant IAM
• Rollout & JIRA Training for Departments in Germany, Luxembourg and Prague