Data Engineer
£45,000 - £57,000
Hybrid WORKING
Location: United Kingdom (Greater London) Type: Permanent
An exciting opportunity has arisen for a skilled Data Engineer to support a government department in transitioning a successful proof-of-concept (PoC) business intelligence and data warehousing solution into a fully operational, production-ready platform.
Over a four-year period, a PoC was developed to explore technology options aligned with the department’s long-term digital strategy. Through agile delivery and close collaboration, the organisation significantly improved its understanding of internal data assets and the Microsoft technology ecosystem, enhancing its reporting capabilities and data-driven decision-making.
The next phase of this programme will involve expanding and maturing the platform into a scalable, secure, and well-supported system. This will be delivered through phased agile delivery cycles, with one or more specialist teams contributing to key technical milestones along the roadmap.
Key Responsibilities:
The successful candidate will be responsible for designing and implementing robust data engineering solutions within a Microsoft Azure environment. Key tasks will include optimising data pipelines, enhancing security configurations, and contributing to the overall architecture of the enterprise data platform.
Technical Skills:
Azure Data Factory (ADF):
- Incremental creation of data dimensions to optimise processing, reduce overheads, and improve cost efficiency across compute, storage, and networking.
- Enhancing pipeline security by replacing basic authentication with managed identities or service principals.
- Introducing parallelism into pipeline workflows to improve performance and scalability.
- Refactoring existing pipelines into reusable templates to support the onboarding of new datasets.
Azure SQL:
- Integrating database access with Microsoft Entra ID for improved identity and access management.
- Designing and implementing STAR schema models to support analytical workloads.
- Defining consistent naming conventions across all database objects.
- Migrating data processing logic from Power BI into Azure SQL for better scalability and maintainability.
- Segregating datasets into separate schemas or views to improve manageability and performance.
- Applying data protection
Desirable:
- Previous experience working on public sector or government data programmes.
- Familiarity with agile delivery in a multi-team or scaled agile environment.
Reference: AMC/RHU/DEBAE
Apply for this vacancy
Your CV will be sent to the selected department. At no time will your CV be sent outside of Anson McCade without your authorisation.
*Mandatory field.