I presume the server filled up again over the weekend?

Yes. Message in the News thread a couple of days ago.

---

Backups: Here

Thanks, sorry, not paying attention! Normally I spot the news posts.

Dave

Jonathan has been working on it.
But the biggest server for moving the data to has a disk problem now. And it takes a long time to 'chunk' the data for moving, verify that it's OK after the move, and then re-link each model to the research area.
There's terabytes to move, the university net isn't particularly fast, there was a network failure, and most of the IT people from all over Oxford took off for 'more interesting places' as soon as Long Vacation started.

---

Backups: Here

Les, I understand -- happens often enough over here - as soon as I spotted the upload problem, I suspended my Climate apps and let other applications cycle along. I long ago learned that one should have two or three applications running on a workstation for each the CPU apps and the GPU apps.

thanks Les,

is there anything we Crunchers should do with our BOINC client ?

I wish Jonathan well.

for those who missed Jonathan post.

<quote>

We suffered a brief network outage today, which prevented connections to or from various CPDN servers.
The fault developed at approximately 2 pm BST and continued for two hours.
The hardware responsible is due to be replaced imminently, but the project is 'at risk' until that has been done (probably for another 12 hours).

Jonathan Miller
CPDN SysAdmin

</quote>

http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=5447&nowrap=true#44898

25/09/2012 12:51:19 PM | climateprediction.net | Started upload of hadam3p_eu_wi36_1978_1_007215635_2_9.zip
25/09/2012 12:51:22 PM | climateprediction.net | [error] Error reported by file upload server: can't open file
25/09/2012 12:51:22 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_wi36_1978_1_007215635_2_9.zip: transient upload error
25/09/2012 12:51:22 PM | climateprediction.net | Backing off 3 min 1 sec on upload of hadam3p_eu_wi36_1978_1_007215635_2_9.zip
25/09/2012 12:54:18 PM | climateprediction.net | Started upload of hadam3p_eu_wkar_1963_1_007216497_1_9.zip
25/09/2012 12:54:20 PM | climateprediction.net | [error] Error reported by file upload server: can't open file

is there anything we Crunchers should do with our BOINC client ?

Set the project to No new tasks to stop polling for work
Suspend all climate models so as not to add more zips that won't upload.
Set Network activity suspended if possible to completely stop talking to the project.

---

Backups: Here

No more fun makes it. Constantly there are problems with the servers.
Also it does not get the team ready, finally, an application GPU to provide.
Climate was sometimes my favorite project.
Luckily there are still other scientific projects.

---

Wiki German Language, Wiki in deutscher Sprache
View

Hi,

We have issues on all three of our storage servers at the moment.

Currently Uploader1.atm is full, and the two machines who would normally receive her excess files are suffering from disk issues.

cpdn-upload2.oerc is one of the machines above, so she cannot currently receive uploads.

We are waiting on a fix - I suspect it is to do with the network outage that OeRC suffered yesterday afternoon (2 - 4 pm BST, 25 Sept 1012).

thanks for the update Jonathan. Best Wishes Byron.

Set the project to No new tasks to stop polling for work.
Done
Set Network activity suspended if possible to completely stop talking to the project.
Done
Suspend all climate models so as not to add more zips that won't upload.

I'm not sure on how I do this ?
Could you please provide details on how I do this ?

One way would be:
In BOINC Manager, select Projects tab
Select climateprediction.net project
Click Suspend button

Suspend all climate models so as not to add more zips that won't upload.

I'm not sure on how I do this ?
Could you please provide details on how I do this ?

If you're content to turn off network activity, as you have already done, then there is no need to suspend the models themselves, since the Zip files will simply accumulate until network activity is turned on again. Accumulation of Zip files is not normally a problem, it's 1000's of machines trying and failing to upload them to the affected server that's the problem.

If, however, you didn't want to turn network activity off because, for example, you are running other projects, then it might be a good idea to suspend the CPDN models in order to stop more Zips being generated and failing to upload. To do that, just select the model in the BOINC Manager 'Tasks' tab and press the 'Suspend button'; or select climateprediction.net in the 'Projects' tab and press the 'Suspend' button. The latter option will stop any CPDN tasks running, which may not be what you want, as it's only the HADAM3P EU models that are having upload problems: my PNW models have cleared without any problems.

I cannot suspend network activity, I have other 6 BOINC projects. I've put NNT.
Tullio

---

Thanks Iain and thanks Lockley for responding to my post.

but just a few minuets ago:

it looks like things are back up and running ?

26/09/2012 5:54:28 AM | climateprediction.net | Sending scheduler request: To send trickle-up message.
26/09/2012 5:54:28 AM | climateprediction.net | Not reporting or requesting tasks
26/09/2012 5:54:31 AM | climateprediction.net | Scheduler request completed
26/09/2012 5:54:34 AM | climateprediction.net | Started upload of hadam3p_eu_wkar_1963_1_007216497_1_11.zip
26/09/2012 5:56:04 AM | climateprediction.net | Finished upload of hadam3p_eu_wkar_1963_1_007216497_1_11.zip
26/09/2012 5:56:05 AM | climateprediction.net | Started upload of hadam3p_eu_wi36_1978_1_007215635_2_9.zip
26/09/2012 5:56:05 AM | climateprediction.net | Started upload of hadam3p_eu_wkar_1963_1_007216497_1_9.zip
26/09/2012 5:57:39 AM | climateprediction.net | Finished upload of hadam3p_eu_wi36_1978_1_007215635_2_9.zip
26/09/2012 5:57:39 AM | climateprediction.net | Finished upload of hadam3p_eu_wkar_1963_1_007216497_1_9.zip
26/09/2012 5:57:39 AM | climateprediction.net | Started upload of hadam3p_eu_wjrj_1991_1_007208963_2_9.zip
26/09/2012 5:59:06 AM | climateprediction.net | Finished upload of hadam3p_eu_wjrj_1991_1_007208963_2_9.zip
26/09/2012 6:00:10 AM | climateprediction.net | Started upload of hadam3p_eu_wi36_1978_1_007215635_2_11.zip
26/09/2012 6:01:43 AM | climateprediction.net | Finished upload of hadam3p_eu_wi36_1978_1_007215635_2_11.zip
26/09/2012 6:05:44 AM | climateprediction.net | Started upload of hadam3p_eu_wi36_1978_1_007215635_2_10.zip
26/09/2012 6:07:16 AM | climateprediction.net | Finished upload of hadam3p_eu_wi36_1978_1_007215635_2_10.zip
26/09/2012 6:29:14 AM | climateprediction.net | Started upload of hadam3p_eu_wjrj_1991_1_007208963_2_11.zip
26/09/2012 6:30:46 AM | climateprediction.net | Finished upload of hadam3p_eu_wjrj_1991_1_007208963_2_11.zip
26/09/2012 6:55:10 AM | climateprediction.net | Sending scheduler request: To send trickle-up message.
26/09/2012 6:55:10 AM | climateprediction.net | Not reporting or requesting tasks
26/09/2012 6:55:13 AM | climateprediction.net | Scheduler request completed
26/09/2012 6:56:44 AM | climateprediction.net | Started upload of hadam3p_eu_wkar_1963_1_007216497_1_10.zip
26/09/2012 6:58:14 AM | climateprediction.net | Finished upload of hadam3p_eu_wkar_1963_1_007216497_1_10.zip
26/09/2012 7:43:59 AM | climateprediction.net | Started upload of hadam3p_saf_0xoa_1969_1_006876818_2_12.zip
26/09/2012 7:44:22 AM | climateprediction.net | Finished upload of hadam3p_saf_0xoa_1969_1_006876818_2_12.zip
26/09/2012 7:53:30 AM | climateprediction.net | Started upload of hadam3p_saf_0xoa_1969_1_006876818_2_13.zip
26/09/2012 7:53:33 AM | climateprediction.net | Computation for task hadam3p_saf_0xoa_1969_1_006876818_2 finished
26/09/2012 7:53:33 AM | climateprediction.net | Starting task hadam3p_eu_w4nd_1985_1_007212256_2 using hadam3p_eu version 609 in slot 1
26/09/2012 7:55:50 AM | climateprediction.net | Sending scheduler request: To send trickle-up message.
26/09/2012 7:55:50 AM | climateprediction.net | Not reporting or requesting tasks
26/09/2012 7:55:56 AM | climateprediction.net | Scheduler request completed
26/09/2012 7:57:05 AM | climateprediction.net | Finished upload of hadam3p_saf_0xoa_1969_1_006876818_2_13.zip
26/09/2012 8:00:05 AM | climateprediction.net | Started upload of hadam3p_eu_wjrj_1991_1_007208963_2_10.zip
26/09/2012 8:01:34 AM | climateprediction.net | Finished upload of hadam3p_eu_wjrj_1991_1_007208963_2_10.zip
26/09/2012 8:03:44 AM | climateprediction.net | Started upload of hadam3p_saf_0z6f_1998_1_006888367_2_12.zip
26/09/2012 8:04:07 AM | climateprediction.net | Finished upload of hadam3p_saf_0z6f_1998_1_006888367_2_12.zip
26/09/2012 8:13:08 AM | climateprediction.net | Started upload of hadam3p_saf_0z6f_1998_1_006888367_2_13.zip
26/09/2012 8:13:12 AM | climateprediction.net | Computation for task hadam3p_saf_0z6f_1998_1_006888367_2 finished
26/09/2012 8:13:12 AM | climateprediction.net | Starting task hadam3p_pnw_z862_1985_1_006941106_2 using hadam3p_pnw version 609 in slot 3
26/09/2012 8:17:01 AM | climateprediction.net | Finished upload of hadam3p_saf_0z6f_1998_1_006888367_2_13.zip
26/09/2012 8:34:43 AM | climateprediction.net | Started upload of hadam3p_saf_13xn_1970_1_006904131_1_12.zip
26/09/2012 8:35:06 AM | climateprediction.net | Finished upload of hadam3p_saf_13xn_1970_1_006904131_1_12.zip
26/09/2012 8:44:08 AM | climateprediction.net | Started upload of hadam3p_saf_13xn_1970_1_006904131_1_13.zip
26/09/2012 8:44:12 AM | climateprediction.net | Computation for task hadam3p_saf_13xn_1970_1_006904131_1 finished
26/09/2012 8:44:12 AM | climateprediction.net | Starting task hadam3p_saf_110z_1994_1_006890763_1 using hadam3p_saf version 609 in slot 2
26/09/2012 8:44:39 AM | climateprediction.net | update requested by user
26/09/2012 8:44:44 AM | climateprediction.net | Sending scheduler request: Requested by user.
26/09/2012 8:44:44 AM | climateprediction.net | Reporting 2 completed tasks, requesting new tasks for CPU and NVIDIA, sending trickle-up message
26/09/2012 8:44:46 AM | climateprediction.net | Scheduler request completed: got 0 new tasks
26/09/2012 8:44:46 AM | climateprediction.net | Project has no tasks available
26/09/2012 8:47:57 AM | climateprediction.net | Finished upload of hadam3p_saf_13xn_1970_1_006904131_1_13.zip
26/09/2012 8:52:48 AM | climateprediction.net | update requested by user
26/09/2012 8:52:49 AM | climateprediction.net | Sending scheduler request: Requested by user.
26/09/2012 8:52:49 AM | climateprediction.net | Reporting 1 completed tasks, requesting new tasks for CPU and NVIDIA
26/09/2012 8:52:51 AM | climateprediction.net | Scheduler request completed: got 0 new tasks
26/09/2012 8:52:51 AM | climateprediction.net | Project has no tasks available

I think Byron, that you have filled up the server again with that lot.

Wed 26 Sep 2012 17:25:57 BST | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
Wed 26 Sep 2012 17:25:57 BST | climateprediction.net | Temporarily failed upload of hadam3p_eu_2qf2_1971_1_008173014_1_12.zip: transient upload error
Wed 26 Sep 2012 17:25:57 BST | climateprediction.net | Backing off 5 hr 43 min 9 sec on upload of hadam3p_eu_2qf2_1971_1_008173014_1_12.zip
Wed 26 Sep 2012 17:26:05 BST | climateprediction.net | Started upload of hadam3p_eu_2kj0_1962_1_008189170_0_3.zip
Wed 26 Sep 2012 17:26:06 BST | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space


Dave

I am still getting these messages. Any word on when it might be resolved?

---

I am still getting these messages. Any word on when it might be resolved?

There is a problem with the server to which the data would normally be moved. No doubt when that problem is fixed the moving process will resume.

I notice that some models are able to upload. My hadcm3n's seem to be uploading fine. From the 'other' board I think I read that pnw's also upload because they're going directly to sever at the Univ of WA where the project is located.

Eu mmodels, on the other hand, are completely backed up. I have 16 such files currently in the queue. However they're only 13 MB a piece; I have plenty of disk space; so I'm going to let those models continue to run.

CPDN seems clearly to be a 'set it and forget it' project. Where the contradiction comes in is that, on average, the people participating in the project are technical and it's natural that many of them would want to know more of what's going on. Of course, we do know that CPDN is chronically short-handed.

Even though I've tried to keep these remarks 'neutral', I expect someone will find something to take issue with. Such is human nature.

I notice that cpdnupload2.oerc is red now. That may mean it has been taken off line while the data is transferred however that in itself will take a while as it is several TB.

Note I am not suggesting they buy some as what little I know about how things are set up at Oxford is from my reading here but I saw on Tom's Hardware the other day that someone is now selling a reasonably speedy 4TB drive.

While it might not mean it at Oxford, it would certainly solve all my space problems for a while if it were a bit cheaper.