28.08.2023 aktualisiert
SV
100 % verfügbar
Senior Data Engineer
Munich, Deutschland
Nur Remote
MCASkills
JavaApache AirflowData AnalysisMicrosoft AzureBigTable ImplementationsGoogle BigQueryCloud ComputingCloud-SpeicherComputerprogrammierungContinuous IntegrationInformation EngineeringData GovernanceETLData MigrationDatenmodell
Experienced Data Engineer with over 5 years of expertise in designing, building, and optimizing data pipelines and warehousing solutions using GCP and Azure technologies. With a strong background in Software Engineering and a total of 12+ years in the field, I have a proven track record of architecting efficient data systems and creating scalable solutions. My proficiency spans cloud services, containerization, ETL processes, and integration patterns. I am dedicated to implementing best practices in security, monitoring, and CI/CD to ensure the reliability and performance of data platforms.
- Proficient in creating and managing enterprise-scale data platforms with Azure, Prometheus, Grafana, Jenkins, and Airflow, emphasizing security, privacy, monitoring, and CI/CD.
- Extensive proficiency in Google Cloud Platform (GCP), encompassing expertise in BigQuery, Composer, Cloud Storage, Cloud Functions, Pub/Sub, Cloud Shell, GSUTIL, and the migration of legacy systems to the GCP environment.
- Skilled in developing scalable data solutions using BigQuery, Dataflow, Pub/Sub, and Kubernetes. Proficient in operationalizing Machine Learning projects.
- Expertise in designing ETL pipelines using tools like Dataflow, Dataproc, and Cloud Storage, along with building large-scale Data Lakes using Bigtable, BigQuery, and Cloud SQL.
- Experienced in data migration from BigQuery to Snowflake using Python scripting while prioritizing data security and protection.
- Proficient in GCP development tools such as Java, Python, Apache Beam, BigQuery, Spanner, and Dataflow.
- Solid understanding of architectural patterns, integration patterns, coding conventions, and best practices.
- Experienced Azure Data Engineer with expertise in Azure Data Factory, Azure Data Lake services, Azure Blob storage, Azure Event Hubs, Azure Synapse, Azure Cosmos DB, and Azure DevOps.
- Skilled in writing complex ETL code using Spark SQL and PySpark on Azure Databricks.
- Proficient in Azure Stream Analytics, ADF, Event Hubs, IoT, Azure Synapse Analytics, Azure Databricks, Snowflake, Airflow, and Kafka streaming.
- Experience in building Data Marts for analytical insights on portfolios, market trends, and opportunities.
- Proficient in Python, Pandas, NumPy, SparkSQL, and PySpark.
- Development experience in Delta Lakehouse architecture and associated technologies such as Databricks, Azure Data Lake, and Azure Data Factory.
- Knowledgeable in data governance practices, including data quality, data catalogue, data lineage, data modeling, data privacy, and reference data.
- Familiar with DevOps environments, Git, Continuous Integration, Docker, and Kubernetes.
Erfahrung
Berufserfahrungmehr als 10 Jahre
Freelancer seit2019
Sprachen
Englischverhandlungssicher
Zertifikate
Google cloud professional cloud architect
Google2023
Microsoft Certified Azure Data Engineer
Microsoft2022
Sun Certified Java Developer
Sun Certification2009