Reply from project about batch 890
----------------------------------------------

Hi all,

Sorry for the delayed response over Christmas. Just to let you know that I have stopped resends going out from this batch as the failures are very high.

The experiment setup and files etc are the same as we have used for other weather@home regions so I wasn’t expecting this degree of error. I know that the South Africa region is not as stable though as other regions and I am wondering if this is what is causing the problems in this case - it would be consistent with just an initial conditions difference causing an error as Ian reported.

I will check that I picked up the best restart file to use and also what the researchers want to do.

Best wishes and Happy New Year,

Sarah
------------------------------------------------------
Dr Sarah Sparrow
CPDN programme coordinator

Those who keep an eye on the server status page will have seen we have anew model type listed. These are Linux only tasks.

This experiment investigates the role of the ocean in the extreme events of the very wet 2013-14 winter in Europe. In the past these studies have been done by forcing the atmospheric model with observed sea surface temperatures and sea ice. In this experiment we use a more realistic representation of air-sea interactions in the form of a Slab ocean model, where the mixed layer is forced at the surface, and from heat convergence from the slow ocean circulation. The setup allows us to evaluate the impact of air-sea coupling on extreme events. This is a the first time this configuration is run on CPDN.

Posted here so more see it but it could as easily go in the science section if another moderator thinks that more appropriate.

Hi All,

Just to let you all know that there will be a period of downtime for the CPDN services tomorrow, Tuesday (3rd August). The CPDN services are expected to be disrupted for the majority of the working day. This downtime have been requested by the Engineering Department IT Support in order for them to conduct work on network switches. We apologies for any inconvenience.

Best regards,

Andy Bowery


Hi All,

The offending RAM has now been removed and the main project and dev projects have now been started again and are up and running.

Best regards,

Andy

Missed hearing this earlier in the week. The Royal Netherlands Meteorological Institute says that respected Dutch climate scientist Geert Jan van Oldenborgh has died at age 59 I am sure some working at CPDN would have known him.

Notification from the project team:

Hi All,

Just to let you know that the CPDN project is currently offline. Overnight there has been a power outage on the site where two key servers of the project are based. Engineering IT Support are aware of the issue and will be working to rectify it.

Best wishes,

Andy

Update from Andy at 1500 UTC today:

The dev site is back up and running. Engineer IT Support are currently working on restoring the power to the other server affected by the outage.

Best wishes,

Andy

And main site back up. I have one task currently uploading. I can't check download is working as I haven't managed to get OSX emulation working on my Linux box yet.

And main site back up. I have one task currently uploading. I can't check download is working as I haven't managed to get OSX emulation working on my Linux box yet.

Dave, I don't think it's back up. All the server processes are not running. It's still not accepting trickles or task completion updates. I've been able to upload the yearly files in my hadcm3s tasks, because they must be going to a different server.

Dave, I don't think it's back up. All the server processes are not running. It's still not accepting trickles or task completion updates. I've been able to upload the yearly files in my hadcm3s tasks, because they must be going to a different server.

I had two zips were uploading, they had reached about 40% but I am now getting transient upload errors on them. An out.zip has uploaded successfully. But I see all the servers listed in server status are saying not running. So not sure if things were OK for a short while and there has been a glitch or whether the uploads would have got to 100% and then failed anyway.

I guess it gives me more time to play with quemu and kvm

Edit: My zips from 216 models have all uploaded now but model not reporting so some progress.

All my hadcm3s zips went up fine, it's just the trickles and successful completion requests that are told the server is temporarily down. Perhaps the hadcm3s zips are going up to a different server (non-Oxford) than the hadam4 zips and that's why we are seeing a difference with them.

Looks like things are back up. Latest from Andy...

Hi All,

Just to let you know the main CPDN project and dev project are now back online.

Best wishes,

Andy

Uploads update Mon 12th

Have just heard from CPDN. There was a major failure in the cloud system they use at the weekend. It will take a day or two to move over to a new system before uploads will work again.

Testing site server software updated to 1.4.2 I will post again if not beaten to it when this is due to be rolled out to main site. Currently zips not showing on testing. Getting rid of obsolete task types on server status page has also been raised with Andy.

Hi All,

We are going to be moving the location of the main application/scheduler and database servers. For this we will need to take the project offline for 3.5 days. This works starts with a taking a copy of the database, during which we need to make sure no changes are made to the database, so we will be taking the project offline over that period. This work needs to start soon. So I will be starting this work in 1 hours time. Sorry for the short notice.

Best wishes,

Andy