Beschreibung
Develop the Data Integration Solution for a Big Data Lake based on Cloudera, integrate and ingest various data sources (data streams, databases, files etc.)* Implementing file management, data integrity, and data quality checks and processes
* Work with Solution Designers to understand requirements and designs
* Coordinate with Systems Engineers to assess impact to technical solutions.
* Provide clear qualitative and quantitative information and knowledge to teams and stakeholders.
Several years of proven experience with data integration projects in Big Data Projects (Hadoop, Flume, Kafka, Sqoop, Spark, Hbase, HIVE, Impala, etc.) and mass data applications at Terabyte/Petabyte scale
* At least 5 years of experience with IT Projects in large Enterprises
* At least 3 years of experience developing ETL solutions utilizing Talend
* At least 3 years of experience with large hadoop clusters
* At least 3 years of experience with performance critical data processing at large scale using files, data streams, or databases
* Experience in developing Talend data intake, cleansing and transformation process in a Cloudera DataLake framework
* Experience with Talend Job optimization and integration into Oozie
* Experience with batch and streamed data flows in and out Cloudera (Spark, Kafka)
* Experience delivering J2EE Solutions
* Good DBMS knowledge : Oracle, MS SQL
* Good documentation skills
* Excellent oral and written communication skills in English (German nice to have)
* Team player, experience with agile development approacheand tools (SCRUM, KANBAN, JIRA)
* Must be able to effectively and professionally communicate to within the team
Beneficial:
* Experience with testing IT Systems and Data Integration systems
* Experience with working in internationally distributed cross-functional teams