Beschreibung
Our Customer is installing a high-throughput platform to support R&D and to speed up product development. Part of these activities is the generation of a suitable data repository that (1) is capable to handle the variety and not yet fully defined data derived from the platform, (2) is able to serve as a central repository for the different needs, and (3) provides capabilities to enhance general data transformations with advanced analytics (machine-learning / deep learning).In this context, our customer is searching for external support developing a first working version of such a data repository.
Outlook and strategic approach
Due to its innovative character, the project deals with several uncertainties regarding the data repository and analytics requirements. Therefore, our customer applies an iterative approach for the development where, at least initially, cloud capabilities are leveraged for data warehousing and analytics. At a later stage of the project, with the HTS platform delivering value, IT requirements and architecture will be reviewed and services and operations will be defined.
Work to be performed
From July 2019 to December 2019, we require agile development support creating a first working version of a data repository that enables different groups of internal customers to work with data generated by the high-throughput platform (e.g. platform operators, R&D lab specialists, ..).
Support the project team with development efforts. For selected workshops and focus sessions, we expect on-site activity, but remote support is envisioned.
The internal project team consists of overall project management, IT project management, product owner, data engineer and data scientist, cloud-ops specialist, chemical engineers and the team responsible for the platform once running.
The core development team consists of a cloud-ops specialist, data engineer and data scientist in addition to the external support. This team will perform the tasks described below:
All aspects of data warehouse development and implementation, including design, historization, ELT logic, dimensional modeling, testing and the design of data marts based on a single data source (MS SQL Server) with about 60-70 source tables and associated files containing experimental results.
Implement data flows for non-relational data using a data lake store.
Work together as part of the core development team to distribute and complete tasks.
Developers must employ industry best practices for technology project management and development documentation.
Integrate existing machine learning algorithms and existing advanced analytics components into data pipelines / ELT processes.
Required support period
We require support of one full time equivalent for 20 calendar weeks between July 2019 and December 2019. During this period, we will conduct four sprints with 5 weeks duraExperience working with agile software development processes in rapid iteration cycles