An interview with Martin Laws, Information Systems Manager, Grampian Housing
With over 3,500 homes under management Grampian Housing Association is a leading provider of housing and related services across three local authority areas and a range of tenures throughout the North East of Scotland.
Grampian Housing Association needed to modernise its IT infrastructure in order to create a robust network and introduce more effective business continuity and disaster recovery arrangements.
Steady organisational growth had resulted in a physically sprawling server estate with 9 or 10 boxes. Some were three years old and others many years older. The server room was very noisy and space was running out. Cooling was also an issue, with considerable heat being generated in the room.
Backup arrangements were time consuming and cumbersome. A full backup on to tape for the entire server infrastructure would take about 18 hours, which was causing networking resourcing issues because the backup was still running when people came in the next the morning. Servers were often still busy being backed up and were consequently running slowly.
Martin Laws, Information Systems Manager at Grampian Housing Association, takes up the story:
“We couldn’t really react as fast as we’d like to if there was a need for new software or system add-on that required a new server. It would take about two to three weeks to procure a server and physically find space for it. Because of the lack of space and issue with heat, we would start to shoehorn multiple applications onto the one server to avoid adding another separate box.
“We had dipped our toe in the water of virtualisation by having a couple of servers virtualised on the VMware platform, so the strategy was to expand that and virtualise the entire infrastructure.
“We wanted to significantly beef-up our disaster recovery. Because we were backing-up to tape and it was taking so long to do, if there was a staff request to restore a file, it would take ages to process. We had to get the tape from our off-site location, catalogue it, index it, find the file and provide it to the user. It could take days to do. If there was a disaster we were aware that backing up onto media like tape would have really increased the time to restore significant amounts of data.
“You can’t restore a file while the system is engaged in backing up, so we would also have to wait for the back up to finish around lunchtime and then do the restore before 5pm, when it was time to start the backup again.”
The longer the situation remained, the more problems Grampian was experiencing with physical hardware. So they developed an ambitious IT strategy to invest in a new core strategy that would also support the impending introduction of a new housing management system.
Castle was awarded preferred IT partner status as part of an earlier tendering exercise to support the existing physical network structure.
Martin continued: “We knew Castle was VMware and Veeam accredited. We were happy with how VMware worked on the couple of virtualised servers that we had at the time. We considered a number of options but felt that the best way forward was to standardise the infrastructure on VMware.
“We had some performance issues with some of the existing grey badged servers and there were concerns about sourcing parts and spares. Standardising on IBM hardware for our infrastructure takes that worry away, as we know that if a part is required, Castle would be able to obtain it within their service level agreement.
“There used to be nine or ten servers and that has now been reduced to three IBM hosts. In terms of virtual servers, the current count is around 16, where previously we had planned around 13. Castle commissioned a few new servers for backup using Veeam and beefed-up our domain controllers at the same time.
“We are about to complete the implementation of Castle’s DataGuard, which was the second part of the project, We were backing up every night on our premises but now that is replicated in Castle’s data centres. That gives us peace of mind. Before, we used to put tapes offsite with a third party but we’re saving some money now, as we no longer need to do this.
“We will recycle and dispose of most of the old hardware but some of the newer boxes will be retained and used at another site, which could potentially act as another disaster recovery location.”
Since the upgrade, there has been a transformation in back-up and business continuity arrangements. Martin described the new environment:
“Previous 18 hour backups to tape have been replaced by full backup to disk in just under three hours. It can be as quick as 90 minutes, as we use incremental backups. It’s good to get that email every morning to say that the backup has completed successfully and in just a few hours.
“The Castle consultant set us up for incremental backups and took us through the process so that we can modify the job when needed. Off the back of the project we now have plenty of disk space for storage on the new SAN, so have implemented shadow copies four times a day on the network for the end users. So now if users delete standard Office files at any time through the day we can bring them back at four staging points taken during that day.”
According to Martin, the reduction of hardware that comes with virtualisation delivered further benefits:
“Because there is much less physical hardware to power in the data room, we are saving about two thirds off our energy consumption costs. We used Castle’s virtualisation calculator to work that out. Everything is now in a single cabinet and contained, so the heat is massively reduced. People have also noticed that the noise levels reduced significantly in that area of the office so they are very happy.
“Users have seen performance improvements for some applications but now that we are commissioning new servers and allocating them more processing resources from the hosts, along with migrating the applications to it, users will see more benefits in the long run.
“If there is any problem with backups or any other aspect of the system then the Castle Watchdog system sends an automatic high priority alert to Castle, which comes to us as well.
“It’s good to be safe in the knowledge that if anything is going wrong we will be alerted to it, as opposed to before when we were doing almost daily physical server checks. Now we don’t need to do that, as we know Castle will be on the phone and we’ll see alerts by email telling us if something’s up.
“The way VMware works, we now have the potential to do hardware maintenance during core hours without the users even noticing. So if there are any hardware upgrades on any of the hosts we can actually use VMware’s VMotion to move the servers on to another host live. Before, it would have taken a weekend or would have been an out-of-hours job. Our IT team welcomes anything that will reduce out-of-hours work!
“With the new housing management system coming in, along with some other major core system upgrades, we can put test servers in place, knowing that it’s not going to connect to our live network and cause any problems.
“We can commission and build a new server in a day, when it used to take three weeks. To be able to react that quickly is a huge benefit.”
A smooth implementation is vital for the organisation and Martin described his experience of working with Castle at Grampian Housing Association, where unforeseen challenges were quickly addressed:
“The implementation went according to plan. We did have a major hardware failure on one of our existing physical servers a couple of days into the project. It was on one of the servers that was due to move to the virtual world and be decommissioned. The Castle consultant was on-site at the time and immediately downed tools from the project to get it virtualised ahead of schedule. He moved it into the new virtual infrastructure, restored data to it and brought it back up. That was all done within Castle’s normal service level agreement and the project was still delivered on time.
“We’d used VMware before, but the Castle consultant took the time to take me through the process and keep me fully involved with what was happening. This allowed us to scope it and make adjustments as necessary and make tweaks as the project moved on.
“Because we don’t have to worry about our infrastructure any more, that frees up our internal IT resource, to focus on larger projects. The firefighting that we used to do on the network has been significantly reduced. As an IT team we’re a lot more proactive and strategic, as opposed to being constantly reactive and dealing with urgent problems.”
Summing up the whole project, Martin said:
“Over the long term, the project will save money on energy consumption and the weekly tape backups. The real saving is in our ability to react quicker and so not have to delay projects for weeks while we buy server hardware and commission it. IT and project time saved on that will pay for itself over the next five years.
“We’re very happy with the support we receive from Castle, both in general and on this specific project. Everything was built and tested and planned. We were fully informed throughout, there were no nasty surprises, the support was excellent and any questions were answered immediately. I don’t think there’s anything we could have changed to improve the project.”