28.08.2023 aktualisiert

SV
100 % verfügbar

Senior Data Engineer

Munich, Deutschland
Nur Remote
MCA
Munich, Deutschland
Nur Remote
MCA

Profilanlagen

Shaikkh Valli_DataEngineer.pdf

Skills

JavaApache AirflowData AnalysisMicrosoft AzureBigTable ImplementationsGoogle BigQueryCloud ComputingCloud-SpeicherComputerprogrammierungContinuous IntegrationInformation EngineeringData GovernanceETLData MigrationDatenmodellDatenqualitätDatensicherheitDatensystemeDevOpsPythonMachine LearningBullen- und BärenmarktNumPyReference DataPrometheusAzure Data LakeSoftwareentwicklungSQLStreamingGoogle CloudAzure Data FactorySnowflakeGrafanaApache SparkInternet Of ThingsDatenschutzGitpandasContainerisierungData LakePySparkData LineageApache KafkaStream AnalyticsApache BeamDockerJenkinsDatabricks
Experienced Data Engineer with over 5 years of expertise in designing, building, and optimizing data pipelines and warehousing solutions using GCP and Azure technologies. With a strong background in Software Engineering and a total of 12+ years in the field, I have a proven track record of architecting efficient data systems and creating scalable solutions. My proficiency spans cloud services, containerization, ETL processes, and integration patterns. I am dedicated to implementing best practices in security, monitoring, and CI/CD to ensure the reliability and performance of data platforms.
 
  • Proficient in creating and managing enterprise-scale data platforms with Azure, Prometheus, Grafana, Jenkins, and Airflow, emphasizing security, privacy, monitoring, and CI/CD.
  • Extensive proficiency in Google Cloud Platform (GCP), encompassing expertise in BigQuery, Composer, Cloud Storage, Cloud Functions, Pub/Sub, Cloud Shell, GSUTIL, and the migration of legacy systems to the GCP environment.
  • Skilled in developing scalable data solutions using BigQuery, Dataflow, Pub/Sub, and Kubernetes. Proficient in operationalizing Machine Learning projects.
  • Expertise in designing ETL pipelines using tools like Dataflow, Dataproc, and Cloud Storage, along with building large-scale Data Lakes using Bigtable, BigQuery, and Cloud SQL.
  • Experienced in data migration from BigQuery to Snowflake using Python scripting while prioritizing data security and protection.
  • Proficient in GCP development tools such as Java, Python, Apache Beam, BigQuery, Spanner, and Dataflow.
  • Solid understanding of architectural patterns, integration patterns, coding conventions, and best practices.
  • Experienced Azure Data Engineer with expertise in Azure Data Factory, Azure Data Lake services, Azure Blob storage, Azure Event Hubs, Azure Synapse, Azure Cosmos DB, and Azure DevOps.
  • Skilled in writing complex ETL code using Spark SQL and PySpark on Azure Databricks.
  • Proficient in Azure Stream Analytics, ADF, Event Hubs, IoT, Azure Synapse Analytics, Azure Databricks, Snowflake, Airflow, and Kafka streaming.
  • Experience in building Data Marts for analytical insights on portfolios, market trends, and opportunities.
  • Proficient in Python, Pandas, NumPy, SparkSQL, and PySpark.
  • Development experience in Delta Lakehouse architecture and associated technologies such as Databricks, Azure Data Lake, and Azure Data Factory.
  • Knowledgeable in data governance practices, including data quality, data catalogue, data lineage, data modeling, data privacy, and reference data.
  • Familiar with DevOps environments, Git, Continuous Integration, Docker, and Kubernetes.

Erfahrung

Berufserfahrungmehr als 10 Jahre
Freelancer seit2019

Sprachen

Englischverhandlungssicher

Zertifikate

Google cloud professional cloud architect

Google

2023

Microsoft Certified Azure Data Engineer

Microsoft

2022

Sun Certified Java Developer

Sun Certification

2009


Kontaktanfrage

Einloggen & anfragen.

Das Kontaktformular ist nur für eingeloggte Nutzer verfügbar.

RegistrierenAnmelden