I've got 3 zips from one of these models sitting in my transfer tab. Still very busy then.

The upload problem here is with
http://cpdn-upload2.oerc.ox.ac.uk/cgi-bin/file_upload_handler
which is not taking uploads from the "Moses something-or-other linux-only" models.
I have a few dozen files waiting upload. Fortunately, new work is available, but I hope this gets fixed before the weekend, because, if not, my slow uplink will take more than a day to upload the backlog of intermediate and final results.
I'll have to buy a smartphone to surf the web :)

---

I can confirm that I also have uploads for this model type stuck in the queue. I suggested given the temps here in UK have been somewhat above normal recently pouring a bucket of dry ice over the relevant server.

Edit: reported to those who have access to kick the server.

One of mine has now gone and another is well on it's way.

Edit: Another now gone.

Edit2: Am told the upload server has been reset.

All my backlog has now cleared.

---

Slow, but backlog clearing.
Thanks.

---

My backlog has cleared as well.

Good to see that backlogs are clearing or cleared. Apparently a gate was opened about 1012Z today.

(About 80 were hung on my queues but have been squeezing through my DSL bottleneck for ~10 hours. Queues are nearly clear.)

A few of the new tasks completed, with a few more within minutes. Two crashed. (Reported to staff.)

You might notice an unusual bit in task pages. Nothing to worry about at the moment; it's probably a timing & bookkeeping thing. (Observations also reported to staff.)

---

"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.

Not sure if this is the right place to post this but I have just had a compute error on a pnw model with the following info:

08/07/2015 09:06:02 | climateprediction.net | Finished upload of hadam3p_pnw_po83_2013_1_009979063_0_13.zip
08/07/2015 09:09:05 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
08/07/2015 09:09:05 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
08/07/2015 09:09:05 | climateprediction.net | Started upload of hadam3p_pnw_po83_2013_1_009979063_0_19.zip
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Info: Connection 949 seems to be dead!
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Info: Closing connection 949
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Info: timeout on name lookup is not supported
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Info: Hostname was NOT found in DNS cache
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Info: Trying 129.67.195.136...
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Info: Connected to cpdn-upload2.oerc.ox.ac.uk (129.67.195.136) port 80 (#950)
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.4.42)
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server: Host: cpdn-upload2.oerc.ox.ac.uk
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server: Accept: */*
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server: Accept-Encoding: deflate, gzip
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server: Content-Type: application/x-www-form-urlencoded
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server: Accept-Language: en_GB
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server: Content-Length: 295
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Sent header to server:
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Received header from server: HTTP/1.1 200 OK
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Received header from server: Date: Wed, 08 Jul 2015 08:08:58 GMT
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Received header from server: Server: Apache
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Received header from server: Transfer-Encoding: chunked
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Received header from server: Content-Type: text/plain
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Received header from server:
08/07/2015 09:09:05 | climateprediction.net | [http] [ID#158] Info: Connection #950 to host cpdn-upload2.oerc.ox.ac.uk left intact
08/07/2015 09:09:06 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Info: Found bundle for host cpdn-upload2.oerc.ox.ac.uk: 0x3f6cc60
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Info: Re-using existing connection! (#950) with host cpdn-upload2.oerc.ox.ac.uk
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Info: Connected to cpdn-upload2.oerc.ox.ac.uk (129.67.195.136) port 80 (#950)
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.4.42)
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: Host: cpdn-upload2.oerc.ox.ac.uk
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: Accept: */*
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: Accept-Encoding: deflate, gzip
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: Content-Type: application/x-www-form-urlencoded
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: Accept-Language: en_GB
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: Content-Length: 35478939
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server: Expect: 100-continue
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Sent header to server:
08/07/2015 09:09:06 | climateprediction.net | [http] [ID#158] Received header from server: HTTP/1.1 100 Continue
08/07/2015 09:09:08 | climateprediction.net | Message from task: 0
08/07/2015 09:09:08 | climateprediction.net | Computation for task hadam3p_pnw_po83_2013_1_009979063_0 finished
08/07/2015 09:09:08 | climateprediction.net | Output file hadam3p_pnw_po83_2013_1_009979063_0_14.zip for task hadam3p_pnw_po83_2013_1_009979063_0 absent
08/07/2015 09:09:08 | climateprediction.net | Output file hadam3p_pnw_po83_2013_1_009979063_0_15.zip for task hadam3p_pnw_po83_2013_1_009979063_0 absent
08/07/2015 09:09:08 | climateprediction.net | Output file hadam3p_pnw_po83_2013_1_009979063_0_16.zip for task hadam3p_pnw_po83_2013_1_009979063_0 absent
08/07/2015 09:09:08 | climateprediction.net | Output file hadam3p_pnw_po83_2013_1_009979063_0_17.zip for task hadam3p_pnw_po83_2013_1_009979063_0 absent
08/07/2015 09:09:08 | climateprediction.net | Output file hadam3p_pnw_po83_2013_1_009979063_0_18.zip for task hadam3p_pnw_po83_2013_1_009979063_0 absent
08/07/2015 09:09:08 | climateprediction.net | Starting task hadam3p_pnw_pnzv_2013_1_009978770_2
08/07/2015 09:10:47 | climateprediction.net | [http] [ID#158] Received header from server: HTTP/1.1 200 OK
08/07/2015 09:10:47 | climateprediction.net | [http] [ID#158] Received header from server: Date: Wed, 08 Jul 2015 08:08:59 GMT
08/07/2015 09:10:47 | climateprediction.net | [http] [ID#158] Received header from server: Server: Apache
08/07/2015 09:10:47 | climateprediction.net | [http] [ID#158] Received header from server: Content-Length: 64
08/07/2015 09:10:47 | climateprediction.net | [http] [ID#158] Received header from server: Content-Type: text/plain
08/07/2015 09:10:47 | climateprediction.net | [http] [ID#158] Received header from server:
08/07/2015 09:10:47 | climateprediction.net | [http] [ID#158] Info: Connection #950 to host cpdn-upload2.oerc.ox.ac.uk left intact
08/07/2015 09:10:47 | climateprediction.net | Finished upload of hadam3p_pnw_po83_2013_1_009979063_0_19.zip

I have checked and 13 zips have been registered but not the last one. Is there a bug in this one?
I've got 3 others running so will see what happens with those. They should all apparently have 19 zips!

Alan,

Failure to show #13-#18 .zips is an "unusual bit" mentioned in my last post in this thread. The problem was seen and reported in Beta test period. The result was staff found that "missing" .zip files were actually in the database. It was a bookkeeping exercise to set things right. Because some tasks now show all 18 files in our accounts, that problem was fixed. However, this is a different problem, one which seems similar to a few crashes I had. I have no explanation for that and have not received a reply from staff as to status of a "fix."

I'll send a link to your debug post to staff and hope for the best.

---

"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.

Thanks Astro. Got another one that failed after 2 zips:-

08-Jul-2015 19:28:09 [climateprediction.net] [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
08-Jul-2015 19:28:09 [climateprediction.net] [http] HTTP_OP::libcurl_exec(): ca-bundle set
08-Jul-2015 19:28:09 [climateprediction.net] Started upload of hadam3p_pnw_pnzv_2013_1_009978770_2_19.zip
08-Jul-2015 19:28:09 [climateprediction.net] [http] [ID#166] Info: Connection 1036 seems to be dead!
08-Jul-2015 19:28:09 [climateprediction.net] [http] [ID#166] Info: Closing connection 1036
08-Jul-2015 19:28:09 [climateprediction.net] [http] [ID#166] Info: timeout on name lookup is not supported
08-Jul-2015 19:28:09 [climateprediction.net] [http] [ID#166] Info: Hostname was NOT found in DNS cache
08-Jul-2015 19:28:09 [climateprediction.net] [http] [ID#166] Info: Trying 129.67.195.136...
08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Info: Connected to cpdn-upload2.oerc.ox.ac.uk (129.67.195.136) port 80 (#1037)
08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.4.42)

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Host: cpdn-upload2.oerc.ox.ac.uk

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Accept: */*

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Accept-Encoding: deflate, gzip

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Content-Type: application/x-www-form-urlencoded

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Accept-Language: en_GB

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Content-Length: 295

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server:

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Received header from server: HTTP/1.1 200 OK

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Received header from server: Date: Wed, 08 Jul 2015 18:28:02 GMT

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Received header from server: Server: Apache

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Received header from server: Transfer-Encoding: chunked

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Received header from server: Content-Type: text/plain

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Received header from server:

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Info: Connection #1037 to host cpdn-upload2.oerc.ox.ac.uk left intact
08-Jul-2015 19:28:10 [climateprediction.net] [http] HTTP_OP::libcurl_exec(): ca-bundle set
08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Info: Found bundle for host cpdn-upload2.oerc.ox.ac.uk: 0x3f75400
08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Info: Re-using existing connection! (#1037) with host cpdn-upload2.oerc.ox.ac.uk
08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Info: Connected to cpdn-upload2.oerc.ox.ac.uk (129.67.195.136) port 80 (#1037)
08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.4.42)

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Host: cpdn-upload2.oerc.ox.ac.uk

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Accept: */*

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Accept-Encoding: deflate, gzip

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Content-Type: application/x-www-form-urlencoded

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Accept-Language: en_GB

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Content-Length: 35494190

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server: Expect: 100-continue

08-Jul-2015 19:28:10 [climateprediction.net] [http] [ID#166] Sent header to server:

08-Jul-2015 19:28:11 [climateprediction.net] [http] [ID#166] Received header from server: HTTP/1.1 100 Continue

08-Jul-2015 19:28:11 [climateprediction.net] Message from task: 0
08-Jul-2015 19:28:11 [climateprediction.net] Computation for task hadam3p_pnw_pnzv_2013_1_009978770_2 finished
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_3.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_4.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_5.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_6.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_7.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_8.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_9.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_10.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_11.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_12.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_13.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_14.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_15.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_16.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_17.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent
08-Jul-2015 19:28:11 [climateprediction.net] Output file hadam3p_pnw_pnzv_2013_1_009978770_2_18.zip for task hadam3p_pnw_pnzv_2013_1_009978770_2 absent

Seems to be jumping from an early zip to zip 19.

Seems to be jumping from an early zip to zip 19.

That's a characteristic of these models: once they fail there's no more zips to upload, so they become "absent" to BOINC. But the final zip, with the re-start data, IS then created, which in this case is zip 19.

16.07.2015 04:55:27 | climateprediction.net | [file_xfer] http op done; retval -184 (transient HTTP error)
16.07.2015 04:55:27 | climateprediction.net | [file_xfer] file transfer status -184 (transient HTTP error)
16.07.2015 04:55:27 | climateprediction.net | Temporarily failed upload of hadam3p_pnw_phj7_2013_1_009970479_1_7.zip: transient HTTP error
16.07.2015 04:55:27 | climateprediction.net | Backing off 00:15:30 on upload of hadam3p_pnw_phj7_2013_1_009970479_1_7.zip


on more than 3 pnw_wu

---

16.07.2015 04:55:27 | climateprediction.net | [file_xfer] http op done; retval -184 (transient HTTP error)
16.07.2015 04:55:27 | climateprediction.net | [file_xfer] file transfer status -184 (transient HTTP error)
16.07.2015 04:55:27 | climateprediction.net | Temporarily failed upload of hadam3p_pnw_phj7_2013_1_009970479_1_7.zip: transient HTTP error
16.07.2015 04:55:27 | climateprediction.net | Backing off 00:15:30 on upload of hadam3p_pnw_phj7_2013_1_009970479_1_7.zip


on more than 3 pnw_wu

Yup - some server not reported on the server page is busted.
Usually I don't complain -- usually somebody somewhere in the complex net of what upload servers do what - fixes this type of thing.
I'm not even going to investigate my rapidly growing backlog of uploads. I'm not going to grep the error log and the status files to find out what server (maybe badc?) has a problem.

Not to say "dont report upload errors" please do.

CPDN ain't Google or the New York Stock Exchange. The results are more important than either of those ephemera.

Report errors. Thanks for reporting!! Don't expect an instant fix.

CPDN project is widely distributed on the upload side and the various projects and their various servers.

Could someone closer to the actual architecture explain the many-server many-project situation better than I can?

---

... many-server many-project situation ...

Latest policy is for a climate centre to host it's own data servers as part of the deal. This way Oxford doesn't need huge storage capacity, and the data is right at hand for each project. Sort of. Some of them use massive data centres to store the data, and these can be a few miles away.

... many-server many-project situation ...

Latest policy is for a climate centre to host it's own data servers as part of the deal. This way Oxford doesn't need huge storage capacity, and the data is right at hand for each project. Sort of. Some of them use massive data centres to store the data, and these can be a few miles away.

Except that, the new (and high profile) California Drought study is being handled centrally in Oxford, and as Alan's (chavk) log shows, the uploads are directed to Host: cpdn-upload2.oerc.ox.ac.uk (which resolves to aforgomon.oerc.ox.ac.uk).

That server is running (it responds to pings), but is currently not accepting connections. I suspect that the central Oxford infrastructure may not have been specified to handle the intense peaks of activity which occur when a new sub-project is launched and catches the public's imagination. Eirik's NYSE comparision is relevant: how many times do we hear about even well-resourced websites crashing when a major flotation/IPO is launched, or tickets go on sale for a major event or festival?

Yes, I currently have 5 x 75Mb hadam3prmpm2t_eu zip files waiting to upload and this number will grow over the coming days.

The uploads are backing off and retrying...without success.

Lets hope the upload server accepts them soon.

Digby

Yes I have the same here:

7/16/2015 4:45:56 AM | climateprediction.net | Started upload of hadam3p_pnw_pszg_2013_1_009984909_1_16.zip
7/16/2015 4:46:27 AM | | Project communication failed: attempting access to reference site
7/16/2015 4:46:27 AM | climateprediction.net | Temporarily failed upload of hadam3p_pnw_pszg_2013_1_009984909_1_16.zip: transient HTTP error
7/16/2015 4:46:27 AM | climateprediction.net | Backing off 00:03:42 on upload of hadam3p_pnw_pszg_2013_1_009984909_1_16.zip
7/16/2015 4:46:28 AM | | Internet access OK - project servers may be temporarily down.

uploads for pnw seem to be working :)

7/16/2015 9:03:21 AM | climateprediction.net | Starting task hadam3p_pnw_e3xy_2011_1_009608436_2
7/16/2015 9:04:47 AM | climateprediction.net | Finished upload of hadam3p_pnw_q0b4_2013_1_010006140_0_9.zip
7/16/2015 9:04:47 AM | climateprediction.net | Started upload of hadam3p_pnw_ps16_2013_1_009983744_1_10.zip
7/16/2015 9:05:03 AM | climateprediction.net | Finished upload of hadam3p_pnw_pnz4_2013_1_009978743_1_15.zip
7/16/2015 9:05:03 AM | climateprediction.net | Started upload of hadam3p_pnw_plbg_2013_1_009975334_0_11.zip
7/16/2015 9:06:24 AM | climateprediction.net | Finished upload of hadam3p_pnw_ps16_2013_1_009983744_1_10.zip
7/16/2015 9:06:24 AM | climateprediction.net | Started upload of hadam3p_pnw_pq36_2013_1_009981364_0_15.zip
7/16/2015 9:06:51 AM | climateprediction.net | Finished upload of hadam3p_pnw_plbg_2013_1_009975334_0_11.zip
7/16/2015 9:06:51 AM | climateprediction.net | Started upload of hadam3p_pnw_prtr_2013_1_009983491_1_15.zip
7/16/2015 9:08:25 AM | climateprediction.net | Finished upload of hadam3p_pnw_pq36_2013_1_009981364_0_15.zip
7/16/2015 9:08:25 AM | climateprediction.net | Started upload of hadam3p_pnw_q0fe_2013_1_010006294_0_9.zip
7/16/2015 9:08:32 AM | climateprediction.net | Finished upload of hadam3p_pnw_prtr_2013_1_009983491_1_15.zip
7/16/2015 9:08:32 AM | climateprediction.net | Started upload of hadam3p_pnw_q05k_2013_1_010005940_0_2.zip
7/16/2015 9:10:06 AM | climateprediction.net | Finished upload of hadam3p_pnw_q0fe_2013_1_010006294_0_9.zip
7/16/2015 9:10:06 AM | climateprediction.net | Started upload of hadam3p_pnw_prpo_2013_1_009983353_0_16.zip
7/16/2015 9:10:13 AM | climateprediction.net | Finished upload of hadam3p_pnw_q05k_2013_1_010005940_0_2.zip
7/16/2015 9:10:13 AM | climateprediction.net | Started upload of hadam3p_pnw_pwoa_2013_1_009989425_0_16.zip

Yes, mine started to upload about an hour ago and the backlog has now cleared :)