https://hmrcdigital.blog.gov.uk/2017/11/08/data-virtualisation-why-we-moved-99-million-accounts/

Data virtualisation: why we moved 99 million accounts

NPS Programme Team
NPS Programme Team

I’m Ian Roberts and for the last year I’ve been managing a programme to build a flexible, scalable, cost effective and resilient hardware platform for all UK Tax & NI accounts on the National Insurance and PAYE System (NPS). If you’re interested in how we’re virtualising big legacy systems in government then my blog post today is for you.

I suspect that you, like me, are avid consumers of digital services both inside and outside of the workplace. Like most people my tolerance of poor performance and system availability is set very low.  We all expect our systems to work and be available when we need to use them. Our HMRC online customers are no different and that is really why the NPS programme started.

The move to cloud

NPS was first created back in 2009 when all PAYE and NI accounts were merged into one system. As was the IT orthodoxy of the day, it was hosted in large data centres on high-powered hardware using proprietary equipment and software. These systems were safe, secure and offered good system availability for the period. However they were difficult to scale, expensive and tied clients to single vendors.

So, in 2015 we decided to move NPS - one of our largest systems that supports up to 17,000 HMRC users on a busy day - to a more modern operating model. Over the last two years the presentation layer (December 2015) and then the application layer (June 2016) were migrated to a virtualised private cloud. And then in October 2017 things got really exciting when the largest and most complex part of the project was completed. In a single weekend we migrated more than 99 million accounts on 7,400 database tables, the largest of which contains 65 billion rows.

Planning was key

As you can imagine this took a lot of planning. As Programme Manager I had to coordinate the work of three external suppliers with our HMRC internal technical teams. We also worked with several other HMRC teams who had important inputs that reached far beyond the specialist knowledge of the core NPS delivery team. And, to add to the pressure, there were a multitude of security and service acceptance issues to carefully negotiate and navigate through because NPS is at the heart of not only HMRC systems but impacts the Department for Work & Pensions as well.

We ran the project along Agile principles with the technical solution worked through collaboratively and iteratively between HMRC and one of HMRC’s key hardware suppliers. This approach is not for the faint-hearted as solutions evolve and are not immediately clear, which can bring a bit of discomfort for people who grew up with the waterfall methodology.

Greatest achievement

Of course, none of the technical solution matters if the people that use it and run systems on it hadn’t understood and agreed to the plans I had developed. The creation of a collaborative “big-tent” of stakeholders and influencers was perhaps the greatest achievement of this Programme of work.

Oh, and moving 99 million accounts across in a single weekend - did I mention that bit?

Ian Roberts, NPS Programme Manager

Check out our current vacancies. They're updated regularly so worth keeping an eye on.

Now you can follow us on Twitter @HMRCdigital

And don't miss any of our blog posts by signing up for email alerts

Leave a comment

We only ask for your email address so we know you're a real person