Data Engineer – Azure Data Factory – Partially remote
Oxford - Remote
We have an exciting opportunity for a Data Engineer to join a blue chip organisation on a permanent basis. Our clients offices are in Oxford however this role is majority remote with a trip to the office 1-2 times per week.
Azure Data Factory experience essential
Requirement gathering, solution design, and mentoring
• Engaging with business stakeholders to support the gathering of functional and non-functional requirements for data solutions
• Support the Data & Analytics Architect in designing data solutions aligned to the design principles and security standards as set by the EA function and the Cyber Security team
• Deputising for the Data & Analytics Architect as and when required such as in TDA sessions, Software Solution Design workshops and resource planning.
• Mentoring and supporting the technical development of the BI Developers in data engineering principles and practices
• Collaborate and support software development initiatives which involves insights and analytics reporting.
Create Azure based data storage solutions and data movements.
• Continuous Integration, continuous delivery, continuous deployment of the BI Solution such a Data Lake, Data Factory, Data Warehouse, Semantic Models using Azure DevOps.
• Continuous developing the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Azure and other ‘data’ technologies.
• Develop Data Factory and other data movement processes as required to support modern BI solutions and Software solutions, auditing, and other data needs in the business.
• Identify and implement suitable data quality management and meta data management solutions.
• Develop and apply data archiving, purging, and GDPR compliance processes.
• Ensure the Azure environment is proactively monitored, data flows from source systems are robust with adequate fault tolerance, the reporting layer is consistent, and new developments are designed in a modular fashion.
• Maintain and performance tune the cloud-based data storage solutions to support the operational as well as the BI application environment.
• Implement the technical aspects of data governance and access control policies.
• Ensure work is fully documented and referenced in line with agreed processes and formats
Build big data, data streaming, and AI infrastructure solutions.
• Translate business requirements into technical data solutions.
• Collaborate with and support the Data Scientist community and Software Delivery team with the development of AI solutions.
• Collaborate and support on implementation of pilots and proof of concepts using azure technologies.
• Work with the Data & Analytics Solution Architect and Cloud Solution Architect to ensure data elements of any solution are design and maintained to perform in the most cost effective and optimal level.
Education and experience
• Degree in computer science or equivalent level of work experience, information systems, or another related technical field
• A background in software and/or database development and engineering, data architecture, and/or developing data solutions
• Experience in DevOps and Agile methodologies
• Experience in preparing data for data science and ML.
• Experience of developing data pipelines using IoT/streaming data.
• Experience in Event Hub and Streaming Analytics
• Experience in DevOps CI/CD
• Experience in coding in Python / R / or equivalent languages.
• Experience deploying solutions to the Azure Cloud.
• Experience developing Modern Data Warehouses and Data Lakes using Azure.
• Design and development of best practice ETL flows, data stores, dimensional models, cubes
• Data preparation (sourcing, acquisition, integration)
• Knowledge of Serverless component like Logic Apps and Azure functions
• Infrastructure as Code technologies such as Microsoft ARM templates and the tools to automate their deployment.
• In-depth understanding of data management (e.g. permissions, recovery, security, and monitoring)
• Strong professional and ethical judgement in choosing methods that give rapid, high quality, actionable results.
• Ability to embrace and drive change, with a proactive and collaborative attitude.
• Proven ability to solve problems of varied technical backgrounds.
• Awareness of best practice, trends, and evolutions in the IT industry and most particularly MS Azure technology stack including intelligent edge technologies.
• Knowledge of MS Dynamics 365 technology stack (desirable)
• Knowledge of PolyBase (desirable)
• Knowledge of Data Bricks (desirable)
So if you are a data engineer looking for your next exciting opportunity please apply today!Back to Jobs