Hi
I'm trying to upload the final output files from two HADAM3P models - one is European and the other is Pacific North Western. I'm running on Windows but I'm not convinced this problem is Windows specific.

The output I get from BOINC is below. Both files upload to 100% complete then the upload fails as shown below. Is there a problem with this destination server?
thanks
P

Mon, 25 Nov 13 18:21:50 climateprediction.net [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_q6vh_2009_1_008332044_0_13.zip: No such file or directory
Mon, 25 Nov 13 18:21:50 climateprediction.net Temporarily failed upload of hadam3p_eu_q6vh_2009_1_008332044_0_13.zip: transient upload error
Mon, 25 Nov 13 18:21:50 climateprediction.net Backing off 13 min 50 sec on upload of hadam3p_eu_q6vh_2009_1_008332044_0_13.zip
Mon, 25 Nov 13 18:21:53 climateprediction.net [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_qag3_2043_1_008356940_0_13.zip: No such file or directory
Mon, 25 Nov 13 18:21:53 climateprediction.net Temporarily failed upload of hadam3p_pnw_qag3_2043_1_008356940_0_13.zip: transient upload error
Mon, 25 Nov 13 18:21:53 climateprediction.net Backing off 1 min 56 sec on upload of hadam3p_pnw_qag3_2043_1_008356940_0_13.zip

---

I'll talk to the project people about the server, but it'll be half a day or more before they're back at work.

In the meantime, both of your computers are crashing a lot/most of the longer hadcm3 models.
The error listings show a lot of "suspending", so possibly you have the setting: Suspend work if CPU usage is above set to the default of 25%.
Whatever the cause though, this type of climate model does NOT like being interrupted at certain critical points, and will crash. Suspending BOINC (and the model), at random moments, will sooner or later hit one of these critical points.
It could also be due to swapping a climate model for work from another project at the critical point.

Please look into why it's happening and do something about it. Otherwise your computers may get blocked from this project for wasting data sets.

Les,
Thanks for the reply - I'm in no urgency to upload as long as someone is looking at it.

I didn't expect to be accused of wasting data sets or be threatened with being blocked though. I joined in June 2003 as one of the Beta testers and I've been running as many models as possible since then. I'm sorry if my participation is not beneficial to the project.

P

---

There are a lot of computers which have a list of over a thousand data sets that have crashed, some within seconds of starting. It's obvious that the owners never look at the results. Sooner or later someone comes across them and they get blocked and an email sent to them.

In the Science section of this board there's a thread Misconfiguration e-mail where people are pointed to so they can ask for help.
There's a newer thread near it where people can report problem computers: Reporting misconfigured computers

So, I'm not "threatening", it's just a "heads up".
No point in continuing to waste electricity if the computer isn't running models to completion.

Hi Paul,

It looks like nearly all of the hadcm3/RAPIT crashes on your two computers are occurring at the "decadal" upload times, that is when the intermediate files are uploaded every 10 years. I notice in the stderr listing for your models a lot of "suspend request" lines. This can occur if, like Les says, the setting for "while processor usage is less than" is set to low values like the default 25%. I set mine to 0 for no restriction. Also, the "only after the computer has been idle" setting can cause suspensions. I also set this to 0 for no limit. And of the course the "Computing allowed while computer in use" checkbox should be checked, as well as the "Leave applications in memory when suspended".

If these PCs are multi-purpose and boinc slows the computer down with those settings, you may need to change some of those back.

The percent erroring out isn't bad for your 960. Your 660 has a higher percentage of errors but is still not as bad as most of those reported to the misconfigured computer thread. Hopefully playing around with the boinc preferences can result in a higher completion percentage.

I am seeing very much the same thing with my linux results.

Tue 26 Nov 2013 14:01:03 GMT climateprediction.net [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_oty2_2007_1_008288758_0_13.zip: No such file or directory
Tue 26 Nov 2013 14:01:03 GMT climateprediction.net Temporarily failed upload of hadam3p_eu_oty2_2007_1_008288758_0_13.zip: transient upload error
Tue 26 Nov 2013 14:01:03 GMT climateprediction.net Backing off 3 hr 55 min 16 sec on upload of hadam3p_eu_oty2_2007_1_008288758_0_13.zip

repeated periodically.

Perhaps I've come to the wrong place and there was a dearth of disk space on CPDN's servers which has also happened now. But certainly I don't recollect being able to upload any results recently, only trickle-ups. Agree with the mod's point that the CPDN models seem somewhat flakey compared to other projects. I take it as a hazard of donating my CPU time to this project, but it would be nice to know why it happens and how hard it would be to ameliorate, so to avoid wasting electricity. Crashes I tend to see also happen towards the latter half of the model (anecdotally - I've not measured). Have any of the models which are prone to crashing at critical moments been deliberately crashed on a debug version of the app to get a stack trace?

---

The 13.zip file not going through is almost certainly a server problem. It should get sorted out before too long. the 13.zip files go to a different server from the rest which explains why the others have gone through.

Hello Geophi,

Thank you for the informative and pragmatic reply. It is encouraging to know the problems exhibited by some of the models on my computers aren't as bad as first made out.

Some background which you may find useful (as may others looking at this thread). Both computers run BONIC when in use.

The 660 computer runs 24h/day, but is somewhat under powered and can churn/freeze briefly at times. BOINC is set to run when computer in use and suspend only when CPU usage is over 50%. I feel this is a balance between keeping BOINC running as much as possible but not impacting the machine performance too much. It has three active BOINC threads and I have set the keep model in memory when suspended.

The 960 computer runs from early evening to morning - it is hibernated during the day. It has plenty of RAM (12Gb) and usually has six active BOINC threads as I don't notice any performance impact. I have selected the keep in memory when suspended option as with the 660.

Your findings match what I had noticed - the hadcm3/RAPIT aborts tend to happen around the 10 year (25%/50%/75%) points. It is rare that I've noticed an abort at other points. I did look into it earlier this year as I was concerned about a computer issue causing models to fail despite it happening on two otherwise unrelated computers. The information I found (sorry I don't have the link to hand) noted that the 10 year (25%/50%/75%) points were the most likely abort points as that is when an unstable model is detected (i.e. due to a starting seed that sent the model unstable). I put my model failures down to that. A brief browse through failed hadcm3/RAPIT models on my computers shows they failed on other computers also. I think I would be more worried if models were aborting only on my computer while passing consistently on others.

I note the comment that the hadcm3/RAPIT models are sensitive to suspends.
When CPDN runs out of work I run an alternative project and if CPDN work becomes available again I may have to suspend the CPDN work temporarily to clear work for the other project. In my experience BOINC is rather poor at scheduling and with other projects having short (2 week) reporting deadlines their work units often go into panic to complete in time randomly bumping CPDN work units out the way. I prefer to suspend the CPDN work units and clear other work to avoid BOINC's haphazard scheduling.

BOINC projects target standard desktop computers where suspends are more likely to happen (as opposed to dedicated server farms where projects can run as the primary task uninterrupted). In that case, as noted also by Jonny below, perhaps the models need to be made more tolerant of these suspends to cope with the desktop computer environment where suspends are inevitable (but the processing time is free to the project). Since CPDN targets the desktop environment the models should be able to cope with suspends in a graceful way.

thanks
P

---

Jonny

Regarding the stability issue, it applies mostly to computers running multiple projects, and when certain BOINC options aren't set favourably to cpdn.
Most of the time, I only run climate models, and I don't have the problem.

As for fixing it, perhaps I should post The Spiel again:
The climate models used here are owned by the UK Met Office, and have been created over many decades by many climatologists for running on their supercomputers.
They consist of nearly a million lines of code, most of which is FORTRAN.
The cpdn project runs them under license, and doesn't have the source code for the main program.
The problem is when the many small output files created over the decade are converted from one format to another, and then zipped to be sent back to the server. (Not sure why the need for the conversion; perhaps a difference between supercomputer program and desktop programs needed to use the output?) However, it's while these files are being converted, that the whole setup is vulnerable to the various uses of the desktop computer that may occur during those few minutes.

So, it's up the the users here as to how they minimise the failures. Much advise has been posted over the years.
These failures are mostly a waste of electricity and data sets.

Unlike with some Linux and Mac computers, where failures occur in seconds,and by the hundreds, and are a definite waste of data sets. Their problems can be solved by following the instructions in the relevant sticky post at the top of the Linux and Macintosh sections of this board.

paulf and Jonny

The upload problem should be fixed now. Andy said about 7 hours ago:

I think this has been fixed now. The server which receives these files wasn't correctly mounting the directory where these files are received. It is mounted now so these should be coming in correctly now.

Dear Les,

I have the same problem than paulf and Jhonny:

2014.09.06. 18:24:03 | climateprediction.net | Started upload of hadam3p_eu_n7v4_2013_1_008804688_0_13.zip
2014.09.06. 18:25:42 | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_n7v4_2013_1_008804688_0_13.zip: No such file or directory
2014.09.06. 18:25:42 | climateprediction.net | Temporarily failed upload of hadam3p_eu_n7v4_2013_1_008804688_0_13.zip: transient upload error
2014.09.06. 18:25:42 | climateprediction.net | Backing off 04:00:31 on upload of hadam3p_eu_n7v4_2013_1_008804688_0_13.zip

Please ask to Andy for a checking regarding right directory mount.
Thanks a lot in advanced for your help.

Best wishes,

Zoltan

Hi,

I am getting the same type of error HTTP ERROR as of a few days ago. I was wanting it to work itself out. This is what is going on for Work unit 8941265.
name hadam3p_eu_jc77_2013_1_008795287.
application UK Met Office HadAM3P-HadRM3P Europe
created 4 Jul 2014 20:58:00 UTC

What I was wanting to know is if this is something new but with a different task or a different problem all together or just a continuation of an ongoing problem but with some task in general. This is the first real problem with this project that I had.

Any news of how to go about getting this HTTP ERROR uploaded to finish this work unit without going to the extreme of aborting it would be great.


Thanks Timothy Dickey




9/7/2014 12:06:59 AM | | Starting BOINC client version 7.4.8 for windows_intelx86
9/7/2014 12:06:59 AM | | log flags: file_xfer, sched_ops, task
9/7/2014 12:06:59 AM | | Libraries: libcurl/7.33.0 OpenSSL/1.0.1h zlib/1.2.8
9/7/2014 12:06:59 AM | | Running as a daemon
9/7/2014 12:06:59 AM | | Data directory: C:\Documents and Settings\All Users\Application Data\BOINC
9/7/2014 12:06:59 AM | | Running under account boinc_master
9/7/2014 12:06:59 AM | | No usable GPUs found
9/7/2014 12:06:59 AM | | Host name: xp-gateway-home
9/7/2014 12:06:59 AM | | Processor: 2 GenuineIntel Intel(R) Pentium(R) 4 CPU 3.20GHz [Family 15 Model 4 Stepping 1]
9/7/2014 12:06:59 AM | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss htt tm pni nx pbe
9/7/2014 12:06:59 AM | | OS: Microsoft Windows XP: Professional x86 Edition, Service Pack 3, (05.01.2600.00)
9/7/2014 12:06:59 AM | | Memory: 2.00 GB physical, 3.94 GB virtual
9/7/2014 12:06:59 AM | | Disk: 41.59 GB total, 16.14 GB free
9/7/2014 12:06:59 AM | | Local time is UTC -5 hours
9/7/2014 12:06:59 AM | | VirtualBox version: 4.2.16
9/7/2014 12:06:59 AM | climateprediction.net | URL http://climateprediction.net/; Computer ID 1304349; resource share 10
9/7/2014 12:06:59 AM | | Reading preferences override file
9/7/2014 12:06:59 AM | | Preferences:
9/7/2014 12:06:59 AM | | max memory usage when active: 1227.48MB
9/7/2014 12:06:59 AM | | max memory usage when idle: 1636.63MB
9/7/2014 12:06:59 AM | | max disk usage: 5.00GB
9/7/2014 12:06:59 AM | | max download rate: 92160 bytes/sec
9/7/2014 12:06:59 AM | | (to change preferences, visit a project web site or select Preferences in the Manager)
9/7/2014 12:06:59 AM | | Not using a proxy
9/7/2014 12:07:03 AM | | Contacting account manager at http://www.charityengine.com/
9/7/2014 12:07:03 AM | climateprediction.net | Started upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip
9/7/2014 12:07:21 AM | | Reading preferences override file
9/7/2014 12:07:21 AM | | Preferences:
9/7/2014 12:07:21 AM | | max memory usage when active: 1227.48MB
9/7/2014 12:07:21 AM | | max memory usage when idle: 1636.63MB
9/7/2014 12:07:21 AM | | max disk usage: 5.00GB
9/7/2014 12:07:21 AM | | max download rate: 92160 bytes/sec
9/7/2014 12:07:21 AM | | (to change preferences, visit a project web site or select Preferences in the Manager)
9/7/2014 12:07:25 AM | | Account manager contact succeeded
9/7/2014 12:07:38 AM | | Reading preferences override file
9/7/2014 12:07:38 AM | | Preferences:
9/7/2014 12:07:38 AM | | max memory usage when active: 1227.48MB
9/7/2014 12:07:38 AM | | max memory usage when idle: 1636.63MB
9/7/2014 12:07:38 AM | | max disk usage: 5.00GB
9/7/2014 12:07:38 AM | | max download rate: 92160 bytes/sec
9/7/2014 12:07:38 AM | | (to change preferences, visit a project web site or select Preferences in the Manager)
9/7/2014 12:09:49 AM | | Reading preferences override file
9/7/2014 12:09:49 AM | | Preferences:
9/7/2014 12:09:49 AM | | max memory usage when active: 920.61MB
9/7/2014 12:09:49 AM | | max memory usage when idle: 1329.77MB
9/7/2014 12:09:49 AM | | max disk usage: 5.00GB
9/7/2014 12:09:49 AM | | max download rate: 92160 bytes/sec
9/7/2014 12:09:49 AM | | (to change preferences, visit a project web site or select Preferences in the Manager)
9/7/2014 12:11:16 AM | climateprediction.net | update requested by user
9/7/2014 12:28:35 AM | climateprediction.net | Sending scheduler request: Requested by user.
9/7/2014 12:28:35 AM | climateprediction.net | Not requesting tasks: don't need (not highest priority project)
9/7/2014 12:28:57 AM | climateprediction.net | Scheduler request failed: Couldn't connect to server
9/7/2014 12:29:00 AM | | Project communication failed: attempting access to reference site
9/7/2014 12:29:33 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 12:30:29 AM | | Project communication failed: attempting access to reference site
9/7/2014 12:30:44 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 12:32:04 AM | | Project communication failed: attempting access to reference site
9/7/2014 12:32:17 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 12:54:42 AM | | Project communication failed: attempting access to reference site
9/7/2014 12:55:09 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 12:58:16 AM | | Project communication failed: attempting access to reference site
9/7/2014 12:58:33 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 1:03:45 AM | | Project communication failed: attempting access to reference site
9/7/2014 1:04:22 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 1:12:53 AM | | Project communication failed: attempting access to reference site
9/7/2014 1:13:26 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 1:29:35 AM | climateprediction.net | Sending scheduler request: To fetch work.
9/7/2014 1:29:35 AM | climateprediction.net | Requesting new tasks for CPU
9/7/2014 1:29:55 AM | | Project communication failed: attempting access to reference site
9/7/2014 1:30:13 AM | climateprediction.net | Scheduler request completed: got 1 new tasks
9/7/2014 1:30:15 AM | climateprediction.net | Started download of hadam3p_anz_rgzl_2012_1_008948596.zip
9/7/2014 1:30:15 AM | climateprediction.net | Started download of ic19611201_11_N96.gz
9/7/2014 1:30:27 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 1:30:43 AM | climateprediction.net | Finished download of hadam3p_anz_rgzl_2012_1_008948596.zip
9/7/2014 1:30:43 AM | climateprediction.net | Started download of atmos_restart_nfff.day.gz
9/7/2014 1:42:05 AM | | Project communication failed: attempting access to reference site
9/7/2014 1:42:32 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 1:42:38 AM | climateprediction.net | Finished download of ic19611201_11_N96.gz
9/7/2014 1:42:38 AM | climateprediction.net | Started download of region_restart_nfff.day.gz
9/7/2014 1:44:31 AM | | Project communication failed: attempting access to reference site
9/7/2014 1:44:49 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 1:52:47 AM | | Project communication failed: attempting access to reference site
9/7/2014 1:53:21 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 2:08:50 AM | | Project communication failed: attempting access to reference site
9/7/2014 2:09:21 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 2:26:11 AM | | Project communication failed: attempting access to reference site
9/7/2014 2:26:51 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 3:14:24 AM | | Project communication failed: attempting access to reference site
9/7/2014 3:15:07 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 3:39:14 AM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_jc77_2013_1_008795287_0_13.zip: No such file or directory
9/7/2014 3:39:14 AM | climateprediction.net | Temporarily failed upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip: transient upload error
9/7/2014 3:39:14 AM | climateprediction.net | Backing off 05:20:18 on upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip
9/7/2014 3:43:22 AM | climateprediction.net | Finished download of atmos_restart_nfff.day.gz
9/7/2014 3:43:22 AM | climateprediction.net | Started download of ancil_OSTIA_deltaSST_2013_mpip.gz
9/7/2014 4:02:54 AM | climateprediction.net | Finished download of ancil_OSTIA_deltaSST_2013_mpip.gz
9/7/2014 4:13:07 AM | climateprediction.net | Finished download of region_restart_nfff.day.gz
9/7/2014 7:00:57 AM | | Project communication failed: attempting access to reference site
9/7/2014 7:01:02 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 7:10:59 AM | climateprediction.net | Started upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip
9/7/2014 7:11:46 AM | | Project communication failed: attempting access to reference site
9/7/2014 7:11:58 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 7:28:16 AM | | Project communication failed: attempting access to reference site
9/7/2014 7:28:18 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 7:31:23 AM | | Project communication failed: attempting access to reference site
9/7/2014 7:32:17 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 7:33:20 AM | climateprediction.net | Temporarily failed upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip: transient HTTP error
9/7/2014 7:33:20 AM | climateprediction.net | Backing off 03:01:36 on upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip
9/7/2014 7:33:44 AM | | Project communication failed: attempting access to reference site
9/7/2014 7:33:46 AM | | Internet access OK - project servers may be temporarily down.
9/7/2014 7:35:29 AM | climateprediction.net | Started upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip
9/7/2014 7:41:43 AM | | Reading preferences override file
9/7/2014 7:41:43 AM | | Preferences:
9/7/2014 7:41:43 AM | | max memory usage when active: 920.61MB
9/7/2014 7:41:43 AM | | max memory usage when idle: 1329.77MB
9/7/2014 7:41:43 AM | | max disk usage: 5.00GB
9/7/2014 7:41:43 AM | | max download rate: 92160 bytes/sec
9/7/2014 7:41:43 AM | | (to change preferences, visit a project web site or select Preferences in the Manager)
9/7/2014 7:51:12 AM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_jc77_2013_1_008795287_0_13.zip: No such file or directory
9/7/2014 7:51:12 AM | climateprediction.net | Temporarily failed upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip: transient upload error
9/7/2014 7:51:12 AM | climateprediction.net | Backing off 03:46:16 on upload of hadam3p_eu_jc77_2013_1_008795287_0_13.zip

---

Hello Timothy

There's a problem with the servers at Oxford.
You'll have to wait until UK business hours on Monday for this problem to start being looked at.
There's nothing that you can do at your end to speed this up.

Dear Les,

it's great! My last part of hadam3p_eu_n7v4_2013_1_008804688_0_13.zip
was uploded :)
Thanks a lot for your help.

Best wishes,

Zoltan