Data Engineer

Beschreibung

• Data engineering and manipulation. You can work with other technologists and analysts to integrate and separate data feeds in order to map, produce, transform and test new scalable data products that meet user needs. You have a demonstrable understanding of how to expose data from systems (for example, through APIs), link data from multiple systems and deliver streaming services. You know how to work with other technologists and analysts to understand and make use of different types of data models. You understand and can make use of different data engineering tools for repeatable data processing; you can compare between different data models
• Programming and build - You can design, write and iterate code from prototype to production-ready. You understand security, accessibility and version control. You can use a range of coding tools and languages.
• Technical understanding - You know about the specific technologies that underpin your ability to deliver the responsibilities and tasks of the role. You can apply the required breadth and depth of technical knowledge.
• Testing - You can plan, design, manage, execute and report tests, using appropriate tools and techniques, and work within regulations. You know how to ensure that risks associated with deployment are adequately understood and documented
• Problem resolution - You know how to log, analyse and manage problems in order to identify and implement the appropriate solution. You can ensure that the problem is fixed.
• Understanding analysis across the product life cycle (data science). You understand the different phases of product delivery and can plan and run the analysis for these. You can contribute to decision-making throughout the product life cycle. You know how to work in collaboration with user researchers, developers and staff in other roles throughout the product life cycle. You understand the value of analysis, how to contribute with impact and what data sources, analytical techniques and tools can be used at each point throughout the product life cycle.

• Strong at python, unit testing (pytest) and pep8 standards
• Writing robust data pipeline code that can run unattended
• Pandas data validation, manipulation, merging, joining and at times visualisation
• Unix environment, server health and management of ongoing running processes
• Github, git, pull requests, CI and code review
• Logging and reporting pragmatically
• Ability to troubleshoot and solve numerical and technical problems
• Data engineering experience using Python, SQL, Spark and AWS
• Hands on ETL development experience utilizing Microsoft enterprise stack/Azure
• Knowledge of data management platforms and development with SQLServer
• Experience with publishing data sets for visualisation and analysis
• Experience with supporting design of data models/data flows
• High attention to detail
• Excellent communication and facilitation skills evidenced through verbal and written means to a wide range of stakeholders
• Experience with Agile delivery

• Experience in delivering end-to-end BI solution from requirements
• Experience automating build of MS PowerPoint slide decks
• Experience of working with JIRA (or Azure DevOps or similar tools) within an Agile/Scrum environment
• Experience/understanding of product life cycle management