Beschreibung
Data Engineer with PySpark, AWS EMR and Airflow, experience is required for a leading financial services organisation based in London, to work on a greenfield project.
This role is FULLY REMOTE
inside IR35
£ per day
6 months +
Experience in the following;
- PySpark Data Engineering experience
- AWS Services, AWS EMR clusters
- Python, Spark, SQL & noSQL experience
- Hortonworks Hadoop
- Developing pipelines using PySpark and/or Scala Spark Streaming
- Developing pipelines using Jenkins
- Data Engineering tech
- Big Data, Apache Airflow workflow automation for data pipelines
- DevOps culture, jenkins, k8s, terraform, helm, docker
- TDD/BDD methodologies, Git
- Data ingestion, data pipelines (Apache Airflow workflow automation)