climateprediction.net home page
problem uploading

problem uploading

Message boards : Number crunching : problem uploading
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
hpitcher

Send message
Joined: 17 Jan 09
Posts: 1
Credit: 6,596,582
RAC: 0
Message 37126 - Posted: 9 Jun 2009, 11:58:56 UTC

i have four runs waiting to upload and am nearly finished with the next set of four runs. Just upgraded to latest version of boinc (6.6.31). Still failed upload. Internet connection works as normal.
ID: 37126 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 37127 - Posted: 9 Jun 2009, 13:22:30 UTC

This is a known problem.


Please read the recent posts in the News and Announcements thread, which is near the top of the Number Crunching section.

You can subscribe to the topic, (in fact, ANY topic), and get an email message whenever there is a new post. (Email notification must be ON to get these.)

There's another News thread at the top of the independent board, here.
This can also be subscribed to, and also needs email notification to be on for that board.

The news about the current problems starts with this post.


Basically, there are currently some serious server problems, and no uploads are being accepted by a couple of the upload servers.


******************

The Server Status page is accessible from the blue menu to the left.

ID: 37127 · Report as offensive     Reply Quote
Profile old_user3458

Send message
Joined: 30 Aug 04
Posts: 4
Credit: 663,454
RAC: 0
Message 37149 - Posted: 11 Jun 2009, 2:00:08 UTC

I have two units to upload and have been trying since 6/5/2009. I am posting to allow notification of the extent of the problem. Hope you get your servers fixed soon.
ID: 37149 · Report as offensive     Reply Quote
old_user120615

Send message
Joined: 28 Nov 05
Posts: 1
Credit: 1,174,336
RAC: 0
Message 37153 - Posted: 11 Jun 2009, 16:55:34 UTC

I have 10 uploads that have been failing all week. I just read the article about the server space being full and awaiting a fix.
ID: 37153 · Report as offensive     Reply Quote
old_user86059

Send message
Joined: 4 Jul 05
Posts: 1
Credit: 9,257
RAC: 0
Message 37170 - Posted: 12 Jun 2009, 6:59:29 UTC

Same here, 5 runs trying to upload since a few days now. I hope everything get fixed soon ! ;)
ID: 37170 · Report as offensive     Reply Quote
Profile old_user217043

Send message
Joined: 3 Jan 07
Posts: 10
Credit: 634,737
RAC: 0
Message 37179 - Posted: 12 Jun 2009, 18:02:43 UTC

Any news?

Good, bad or otherwise?

Thanks!
ID: 37179 · Report as offensive     Reply Quote
old_user541139

Send message
Joined: 10 Oct 08
Posts: 3
Credit: 62,715
RAC: 0
Message 37183 - Posted: 12 Jun 2009, 20:08:19 UTC

One of my 3 pending uploads apparently made it -- other 2 back to "retry" status.
ID: 37183 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 37188 - Posted: 12 Jun 2009, 21:44:02 UTC

Only bad news.

Milo managed to transfer a few terabytes of data off uploader.oerc so that one is working again, but the models have "distributed uploads", so each zip of a model goes to a different server.

***********************

As a reminder: once BOINC starts an upload attempt, it has 14 days to complete the transfer. After that, it aborts the file.

Also, please watch the News and Announcement thread at the top of Number Crunching. This is where any news on the problem will be posted.
It's possible to subscribe to that thread, and be sent an email to say that a new message has been posted. IF you have email notification ON.

Number Crunching is also the appropriate section for posts about problems; there are currently 2 threads there discussing it.


Backups: Here
ID: 37188 · Report as offensive     Reply Quote
Paul

Send message
Joined: 26 Feb 06
Posts: 10
Credit: 316,639
RAC: 0
Message 37208 - Posted: 13 Jun 2009, 20:41:04 UTC

Hi
with the message that if Boinc tries to upload files it aborts the transfer after 14 days if unsuccesful.
do we lose all the credits?

I have used 95 hours to do this so do you lose everything.
if so it is a lot of wasted computer time which could have been better used with other projects. I have 2 others just started with 122 hours to complete so should I start these?
paul
ID: 37208 · Report as offensive     Reply Quote
Andrew Mayne

Send message
Joined: 15 Nov 06
Posts: 2
Credit: 279,141
RAC: 0
Message 37209 - Posted: 13 Jun 2009, 20:57:29 UTC

Work doesn't need to be lost...I don't think.

For those who have the skills, maybe we can edit BOINC's client_state.xml file, under C:\ProgramData\BOINC for me

I just had a look and found this section:

<file_info>
    <name>hadam3p_n9v6_1985_2_006161276_0_3.zip</name>
    <nbytes>241912.000000</nbytes>
    <max_nbytes>50000000.000000</max_nbytes>
    <md5_cksum>bce3254ac0e10e90e02bc1f59313d51f</md5_cksum>
    <generated_locally/>
    <status>1</status>
    <upload_when_present/>
    <url>http://uploader1.atm.ox.ac.uk/cpdn_cgi/file_upload_handler</url>
    <persistent_file_xfer>
        <num_retries>50</num_retries>
        <first_request_time>1244562860.068153</first_request_time>
        <next_request_time>1244928762.174537</next_request_time>
        <time_so_far>1106.887008</time_so_far>
        <last_bytes_xferred>0.000000</last_bytes_xferred>
    </persistent_file_xfer>
    <signed_xml>
    <name>hadam3p_n9v6_1985_2_006161276_0_3.zip</name>
    <generated_locally/>
    <upload_when_present/>
    <max_nbytes>50000000</max_nbytes>
    <url> http://uploader1.atm.ox.ac.uk/cpdn_cgi/file_upload_handler </url>
    </signed_xml>
    <xml_signature>
8c8ad6ba3d0f1587907b825257bd5e098bf73a73e3e374711b97986178de4c9b
c985098cacb939cca07cce20a0091bc628b554cd6d084573d9b05637e68f2d84
bc83bdc756a3d21f0fc9056d45d0de5cdada829fa7541c57c4001d2cd4714fb5
28f33f526e39abd94f746deae4981e060ad9a31ba38c57c17cf3624420bc6037
.
    </xml_signature>
</file_info>


Notice the persistent_file_xfer bit. We could probably just alter the number of retries, request times and time-so-far don't you think?
ID: 37209 · Report as offensive     Reply Quote
Andrew Mayne

Send message
Joined: 15 Nov 06
Posts: 2
Credit: 279,141
RAC: 0
Message 37210 - Posted: 13 Jun 2009, 21:03:56 UTC - in response to Message 37209.  

right, I just reduced the number of retries, down to 5, though you could probably put 0, and moved the first request time closer to the next request time. Boinc seems to have accepted this, guess I bought that upload some more time!
ID: 37210 · Report as offensive     Reply Quote
Profile Iain Inglis

Send message
Joined: 9 Jan 07
Posts: 467
Credit: 14,549,176
RAC: 317
Message 37211 - Posted: 13 Jun 2009, 21:06:14 UTC

Andrew,

That's one of the possible methods, though it's simplest just to delete the whole persistent_file_xfer XML element - i.e. everything between and including the persistent_file_xfer start and end tags.

I did that for some affected machines and suspended network activity, but cpdn-upload1.comlab came back today anyway, so it wasn't necessary: everything uploaded as normal this morning.

Your file is destined for uploader1.atm, which is awaiting a power supply - which surely can't be long coming now.

Iain
ID: 37211 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 37212 - Posted: 13 Jun 2009, 21:33:56 UTC

Our files waiting in the Transfers tab should be OK, even if we've tried to upload them but failed. The BOINC 14-day limit on keeping files that have failed to upload allows one extra upload attempt after the 14-day limit. The only server still down, uploader1atm, went down on Friday 5 June. So the 14-day deadline for files to this server will be reached on Friday 19 June. I'd be very surprised indeed if Milo hasn't got it working before then. But if the worst happens, members can just suspend BOINC network activity before Friday then reallow it later after first checking on the server status page that all servers are up and running.

Members with HadAM3P files that earlier failed to upload to the other two problematic servers (now both working) could allow network activity now for long enough to let files for those servers upload.

This method avoids editing any BOINC files. It should be easy enough for our least experienced newbies.
Cpdn news
ID: 37212 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 37213 - Posted: 13 Jun 2009, 21:56:59 UTC

This thread really should be in the Number Crunching section.
This section is intended for discussions about the science of the modelling.

If you look in Number Crunching, you'll see several threads about the problem, as well as the News thread near the top of the list. You'll find that most of the questions here have already been answered in these other threads.

Also near the top of Number Crunching, is my post about editing client_state. This post is linked to from a post in the News thread.


Backups: Here
ID: 37213 · Report as offensive     Reply Quote
GlennMcC

Send message
Joined: 13 Dec 05
Posts: 2
Credit: 2,789,303
RAC: 0
Message 37245 - Posted: 17 Jun 2009, 1:36:41 UTC

Come on guys! You guys can't be that poorly organised that you have no failover plans for your servers, can you? If this lasts much longer many people may well reassess their participation. Storage is dirt cheap these days.
ID: 37245 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 37246 - Posted: 17 Jun 2009, 1:54:56 UTC

Storage is dirt cheap these days.

This IS a University environment!
Even locating a server cabinet with space is tricky, let alone the cost of electricity and airconditioning. It all has to be arranged and paid for, even if a lot of it is only paperwork.

And the matter of storage has been mentioned/brought up/discussed. None of which is going to get discussed here.

Those people who want to leave the project in disgust/anger/protest are free to do so.

ID: 37246 · Report as offensive     Reply Quote
GlennMcC

Send message
Joined: 13 Dec 05
Posts: 2
Credit: 2,789,303
RAC: 0
Message 37247 - Posted: 17 Jun 2009, 2:08:41 UTC - in response to Message 37246.  

Storage is dirt cheap these days.

This IS a University environment!
Even locating a server cabinet with space is tricky, let alone the cost of electricity and airconditioning. It all has to be arranged and paid for, even if a lot of it is only paperwork.

And the matter of storage has been mentioned/brought up/discussed. None of which is going to get discussed here.

Those people who want to leave the project in disgust/anger/protest are free to do so.



I was unaware of the University background. However, how about a creative temporary (at least) solution to the problem, it is nearly two weeks!
ID: 37247 · Report as offensive     Reply Quote
old_user507011

Send message
Joined: 14 Mar 08
Posts: 1
Credit: 3,224,340
RAC: 0
Message 37248 - Posted: 17 Jun 2009, 2:13:36 UTC

Hello, I've 17 waiting upload !!!
ID: 37248 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 37249 - Posted: 17 Jun 2009, 2:41:54 UTC

... how about a creative temporary (at least) solution ...

One WAS tried; transferring several gigabytes from Oxford Uni to some spare space in Canada.

But, as the latest post in News and Announcements says, it's not about storage now.


Backups: Here
ID: 37249 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 37251 - Posted: 17 Jun 2009, 3:10:38 UTC

There is a cunning way to delete posts here, although I think that for the average user there is a one hour limit.

1) Edit your post
2) Delete everything in it
3) Click Post reply

The post will now disappear.

ID: 37251 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : problem uploading

©2024 climateprediction.net