Take today as an example. I woke up at 2am to an email on my blackberry to say that the overnight change had not gone well and that they were rolling back (I’m sorry that I always tend to explain the semi-techie bits in “(” but I’m just not sure who is reading so I think it only fair I explain the terminology) (put back into a previous state that is known to be good.) Which at sleepy o’clock seemed to look like it was nothing to worry about.

This then lead to my fellow code redder (who also got the email and is here in Ipswich with me) to stay awake most of the night and thus changed our time to meet me at 7:30am for breakfast. This then changed us to doing back to back conference calls from the hotel rather than the office so we didn’t arrive til 10:45am.

As was stated in my earlier post today was supposed to be about going to the next level of detail in the plan and to get the monitoring started. As we arrived late to the office the sessions had already kicked off. No sooner had we walked through the door to the office did a service affecting issue start. Again see my earlier post for the No1 priority in any Code Red.  So we had to halt the sessions and change everyone (who happened to be onsite for the end to end review) onto troubleshooting the issue. (see post on the co-location benefits).

Once service had been restored (which wasn’t long- cos these guys are great at what they do!) and we identified the root cause we reviewed the plan and decided that due to the outage we would change the change schedule and we are not deploying some fixes we had planned to do tonight. When things calmed down I changed back onto the monitoring plan.

I had planned to go back to London tonight but as we have to now be here tomorrow. I changed my plans and am now staying tonight. From a blog perspective I was going to talk about monitoring but I decided to change it and discuss change

In case you didn’t get it one small little change, which coincidentally had nothing to do with me directly, at work has caused change in all areas of my life today.

So like I said

Changes come in all different sizes, different contexts and have different consequences.

Today they were big and small, they happened in my professional and personal life and most certainly the consequences were both big and small. – It’s the individual that works out which ones are big and which ones are small.

I’m still in Ipswich and we had the end to end review yesterday where we pulled together all the specialists to the same location see where the issues were. We made good progress but we need to go to the next level of detail today. Once that is complete we can start to see what the longer view looks like. But before that we need to make a few key decisions around the short term.

The short term in this context is from now until christmas (or more likely new year) . We need to agree what are the top priorities. Now this isn’t necessarily lots of changes because the No1 priority is service stability and that normally doesn’t lend itself to a raft of changes.  We have a few changes planned but the change I have worked really hard to get sorted (dealing with suppliers, techies, senior managers) isn’t going to happen until the new year now (that is unless there is a major problem!). It’s strange because I thought I might feel a little disgruntled that all my hard work has somewhat gone to waste but I don’t at all. I guess I realise that my contribution is still going to make a difference regardless of when it goes ahead and that I am still recognised for having delivered what I was asked to deliver so from that point of view its been a success.

I need to work on the next piece of the puzzle – monitoring. I’ll update with some progress tomorrow

However more recently I have found myself heavily embroiled in a new Code Red. It all started last Tuesday when we were asked to take a look at something that is quite high profile and could pose significant problems for BT if it went into meltdown – Especially over the christmas period) So as more of a preventative measure we had a conversation with the owner of the system and soon realised what should’ve been an assurance piece (making sure that everything was in hand and no problems were on the horizon) was something a lot more serious!

Needless to say that by Thursday of last week things were not looking good at all. So this meant a trip up the A12 to Ipswich to help the team out. The best way to help in a code red is be onsite working face to face with the team you are helping. That way there is no latency (of emails) when things need to be discussed and agreed upon. It also means that meaning and interpretation is not lost which sometimes can happen. We finally left the office in Ipswich at 11:30pm on Thursday went to the hotel checked in had a night cap and settled down for the night…. Til the 6:00am call to analyse the results of the traces we ran the previous night on the system.

The good news is we did on Friday isolate the problem and put measures in place to ensure service did not experience any outage over the weekend. That’s not to say I wasn’t on the phone all weekend making sure that the actions we had to improve things were followed through.

I’m now back in Ipswich for another 2 days to go through the system with a fine tooth comb and make sure there are no other “surprises” lurking as yet undiscovered. (I’m sure there will be something else!)

With all this said I did really enjoy the two days I spent working on this last week. It’s really what I signed up to do when I joined the Code Red team. Don’t get me wrong it is tiring and it is  . But the feeling of being there in the thick of it is indescribable – well I hope I’ve managed to give a little bit of insight to it.

I guess I’m still getting used to the peaks and troughs of this role but currently it’s all good!

The advantages are that I can make my role my own. I can choose within the scope of the role to do what I feel is appropriate. With the single caveat that it should add value to the customer, the business and to me. Other advantages allow me to be flexible with my work and location.

The disadvantages are that sometimes it’s not clear what it is that you can do to add the value. Having said that though for me what makes the difference in this situation is having a good network of people (managers, team members, other graduates, mentors) to turn to for support, advice and guidance.

I guess what makes a role/job good isn’t just the role/job itself but the people around you that help when you need it.

I’m currently waiting to find out the extent of the issues during the testing phase. I sat on 2 hourly conference calls all through the day on Saturday listening to the chatter. I think the front line teams assigned to handle the issues in the first instance were doing a good job so no reason to intervene. I’ll write again when I get some news.

…its not over yet though…

Seems to be resolved now so a bit of time to make up but noone really seems bothered.

I guess the only way I can describe it is to draw an analogy when installing a new piece of software on your own PC that in itself might take about 10 minutes end to end to install, shutdown and restart and then check its all gone ok when you’ve one processor and maybe a couple of gigabytes of ram. Imagine your system has 16 quad core processors and 90 gigabytes of ram. starting it takes a little bit longer.

The coffee has just brewed so that’s all for this update.

As it stands one current application is feeling under the weather so tonight/tomorrow morning (12am to 8am) I have the pleasure of sitting on a conference call listening to release go live that involves this sick application to make sure there are no major problems. I’m going to make a few more posts as the night goes on. But for now Im making the most of the time before clocking in by going on a Jack the Ripper walking tour in East London!