100 Degrees - Data Centre Implications

I picked up on a trend this morning from AppLogic’s Bert Armijo twitter post, that data centres are starting to consider running at 100 Fahrenheit (°F) according to Jon Brodkin. I found this little fact quite interesting and it lead me to convert this figure to a scale I understand, my frame of reference, Celsius.

It was very interesting to discover the Celsius value was 37.7777778, for a large number of reasons.

Having worked in Systems Administration since 1996, I have spent my fair spare of hours working in data centres. Indeed, in some case 36 hour stints at a time. I have to therefore consider this fact when thinking about future projects, not because I am lazy. Anyway that works intermittently in data centres (not sure on a daily basis, if you do let us know, leave a comment of your experience), will have found that in the past 4 years data centres temperatures have increased quite substantially. I would even go as far as to say that indeed some sysadmins will have stats to prove that. This situation may be correcting itself, but I think it was probably widespread as we introduced more and more 1u rack dense servers. Telecity in Harbour Exchange could not cope and indeed had a power failure in Harbour Exchange 8/9 due to the UPS’s last summer having to kick in around at around 01h30

1) On 3/18/2008 at 1:53:33 AM there was a power failure in the Telecity data centre in the TeliaSonera 2nd Floor Suite. A Telecity engineer explained to me that this was due to a power overload caused by the air conditioning ramping up and causing a power “dip” which DID NOT cause the UPS to kick in as the power never failed just dipped. In our racks the power failed, 50 servers rebooted, this in my opinion is a power failure. What is our recourse in this situation?
2) The heat in our area of the TeliaSonera Suite is excessive. I have thermal probes on some kit registering 50 degrees. I have numerous thermal shutdown alerts on a number of servers, obviously I have had to disable automatic thermal shutdown. I personally feel that this is a result (in our row) of the 2 Akamai racks. What is being done to resolve this?

So it if fair to say that our data centres’ ambient temperatures are already higher than 37.7 degrees at times. Strange that, at 01h53 in the morning, in London in March when the ambient air temperature was around 4 degrees Celsius (I found a decent use for wolframalpha, that and population figures). That is because contrary to conventional wisdom that one would expect, when we go to sleep, our machine work the hardest, not necessarily at the peak times. This is the time that every sysadmin schedules their full backs. Our servers often work more (or differently) at the low traffic times than they do at the peak traffic times. And although this may not be true of larger traffic sites, it is true for the millions of SME’s around the world. We read all (or a lot) of the data off the drives, we send it down the wire and we process reports and logs, etc, etc. At these times regionally, all our machines all concurrently start the data protection tasks. It would not surprise me if the power consumption increased on average between 00h00 and 06h00.

The 100 degree data centre move will have some hurdles to overcome first. The first being that older equipment is not always as heat tolerant as newer equipment and there is still old kit running in many data centres. True there is more newer kit, but there is still some old kit as well. This may force organisations to upgrade sure, but the current economic climate does not lend itself to SME’s doing bigger upgrades at the moment. Unless of course governments gave business’ incentives to upgrade and subsidised it. There would be more benefits to that than the UK’s current new vehicle incentives, which in a down economy are of questionable effectiveness in any case, but they are trying something at least. Subsidising SME businesses to upgrade there hardware would be beneficial for the economy as well. Indeed we would make massive savings in energy consumption and the competitiveness and productivity of our IT sector would be increased. Not to mention, the technical employees of the UK would all be more educated in the newer architectures of today and indeed it could put Britain at the forefront of IP v6 and take up. It would increase our base performance to 64-bit and Britain would the computing power to meeting the computing requirements of the next five years and stay at the front of the revolution. If indeed our infrastructure is up to the job of delivering the next five years of development. Could we see our large scale infrastructure not being able to keep up with Moore’s Law in the next few years and we probably need to find a way to deliver more through our existing infrastructure. Yesterday our flat was rewired with comms and I had to laugh when I saw they were only provisioning a two pair. No fiber, it would occur to me to put fiber in a building if you are rewiring at this point in time, two pair and fiber. Anyway, governments may want to consider the upsides of subsidising IT infrastructure in SME’s today, I am sure they could claim it as a carbon credit. If we where to act now at least we may not have to worry about an IP v4 credit scheme and all the hours of admin it will take to reclaim all those unused IP v4 allocations and who is going to want to give up their IP v4 addresses? Countries that start using IP v6. There is possibly gold in them hills if they do start an IP v4 trading scheme. But once again I digress.

The second problem that the 100 degree data centre presents is the poor Muppets that have to work in the 100 degree data centre. It is fairly common knowledge that human body does not work that well in high temperatures and many areas of the world at least schools close when the ambient temperature reaches a higher level. I am sure that this varies from place to place and time to time. However, this does not change the fact that the human body is not accustomed to carrying out labour at high temperatures. We may in the past have proved that with colonisation and slavery, etc, etc. Honestly who would want to work in these conditions? In fact billions of people around the world do. The human body is able to operate fairly effectively at a fairly wide range of temperatures. The comfort level is a much more variable factor. The TUC in the UK are currently trying to get a maximum working temperature in the workplace established. From their documentation, "there should be an absolute maximum temperature of 30°C (27°C for those doing strenuous work), at which point workers should not have to work and an employer should be liable for prosecution." That would lay waste to any UK data centres plans to run at 100 degrees. Possibly.

Perhaps we should consider building data centres of the future underground. We have enough empty mine tunnels surely, that the construction and civil engineering sectors could surely secure and renovate some of these to provide data centre footprints, indeed with the always present threat of nuclear weapons, perhaps data would be best hosted underground. In the event of a nuclear attack (or other more likely event) who is going to really care if the data is destroyed. Well, perhaps the people that start rebuilding. This may not work in London or Silicon Valley, due to flooding potential and earthquake potential. Indeed Silicon Valley and London are probably two of the most unsuitable and unsustainable areas to have the hub of our technology and financial systems, from a data hosting point of view. It is only a matter of time until Silicon Valley is hit by the big one (The problem with Google). Indeed there is possibly another reason why having data centres underground may have advantages in the future (see future post on solar weather threat).

The UK law currently does not have a meaningful figure at the upper end of the scale, which means that people working in hot environments are at the hands of for profit businesses, which means UK systems and network administrators are screwed :)

Makes one wonder why there are not more water cooled data centres, not air conditioned, water cooled. After all water has a much higher heat storage capacity than air. Ask anyone who has swum at night. Drill bores into the earth and radiate the heat into the rock strata.

That’s to say nothing about the fact that they have found that machines can operate perfectly well at the same temperature as mammalian life in general.

It is amazing what the number 100 can make you think about.