Don’t talk to me about backups…

.. talk to me about high availability and data recovery

Blocked toilet

A few recent events have thrown into focus the ability for even large well established businesses to handle adverse IT events.

Firstly, the recent British Airways outage had many experienced IT professionals scratching their heads. The outage was blamed on a power issue which was, according to BA, “not an IT failure and had nothing to do with outsourcing of IT, it was an electrical power supply which was interrupted”. We were scratching our heads because the inability of an IT system to handle a power outage is very much an IT failure!

Secondly, for a client of ours, there was a recent issue with one of their databases where somehow some of the data within the database had been deleted. They had a high availability strategy in place ( the database was being replicated to another instance in case the primary went down ) but as this was data being deleted from the database, the deletion action had also been replicated to the other database instance. The data was gone. ( Luckily Roar IT had integrated another system into this database and this second system had a copy of a lot of the data that had been deleted, albeit in a different format. This enabled us to work with the client to reconstruct the majority of the lost data. )

The first event showed a lack of high availability planning, the second a lack of data recovery planning.

The language here is important, as too often I hear management staff referring to ‘backup’s. “Do we have a backup?” Can we put everyone on the backup system?” “It’s OK because we have a backup.”

In the BA incident, they may well have had a ‘backup’ strategy in that they were storing a historical record of their data, but they did not have a high availability strategy meaning that they could move their business activity to another set of systems if there was an outage.

In the second incident, the management were convinced they had a ‘backup’ strategy because they had another set of systems to switch to, but they weren’t keeping a historical record of their data, and so had no data recovery strategy.

For businesses that rely on their IT systems, then they need to consider both their high availability, and their data recovery. Let’s keep the talk of ‘backup’s to the fields of parking and drainage!

Walking into a doctor’s surgery and asking for penicillin

One of the problems I have found when adopting an agile approach to developing systems (and I mean truly agile, according to the rules, rather than a particular project management methodology), is that the high level of end user interaction in the development process can sometimes lead to the users, in effect, designing the systems.  They often come with an idea for a technical solution to their problem and ask us developers to build that, rather than allowing analysts to design the solution.

I often use the analogy of a visit to the doctor’s surgery to illustrate this.

In recent times the old role of systems analyst or analyst/developer seems to have evolved into two separate roles – the business analyst (often with little or no programming skills or experience) and the software developer.  In our doctor’s surgery analogy, the roles of user, business analyst and software developer are fulfilled by the patient, doctor and pharmacist.   

It is the doctor’s role to investigate the symptoms, diagnose the problem and come up with a treatment plan, which they agree with the patient. It is then the role of the pharmacist to provide that treatment.

In software development projects, it is frequently the case that the patient (user) walks into the surgery with their own treatment plan in mind. They enter the room and ask for penicillin.  It is too often, in my experience, that the doctor (analyst) then simply writes out a prescription for penicillin and hands that over to the pharmacist (developer).

Imagine the case, where the pharmacist spots a problem with the treatment plan (say the patient has a virus), doesn’t think it is wise, and says to the doctor, “I don’t think this is right”, only for the doctor to respond with: “That’s what the patient asked for, give it to them”.

I’ve seen this story play out in many a troubled software project!

Even worse, I’ve often seen the equivalent of the doctor simply leaving blank prescriptions at reception for patients to fill in themselves and hand to the pharmacist!

The business analyst role is not simply to liaise between users and developers (I’ve known business analysts whose time was mostly taken up being an email relay, or a chinese whisperer!). They need to be looking at the business needs behind any request and designing a solution that fits with any current systems that are in place, before bringing that to a developer.

In addition, it’s rare that an analyst with no technical or programming experience will have the ability to know the full extent of what a software system can achieve, or design solutions that fit into existing systems well.

Maybe it’s time to bring back the combined role of analyst/developer.