I have a problem because of new policies at work.
They limit addresses and package sizes during the day.
I \"wish\" there was a simple mech. for sending the results in smaller bunches (our limit is 3 megs during the day and 5 during off-hours)
The zip it is trying to send is 15+ megs and the server just cuts the transmission off.

The same limits apply to emails, but my wish would be that there were a way to configure that also - instead of direct network connections - break up the package to much smaller bunches and send them (manually to prevent robots / worms, etc.) to an email address
Anybody agree or have ideas?
Below is an edited sample of the \"messages\" I get - long endless chains of it.
What does the \"backing off\" part mean? The package is the same size and just endlessly fails.
...
06/03/2009 12:08:51 PM||Resuming network activity
06/03/2009 12:08:51 PM|climateprediction.net|Started upload of hadcm3ivolc_l1g4_2000_80_06011878_0_1.zip
06/03/2009 12:08:53 PM|climateprediction.net|Temporarily failed upload of hadcm3ivolc_l1g4_2000_80_06011878_0_1.zip: HTTP error
06/03/2009 12:08:53 PM|climateprediction.net|Backing off 2 hr 6 min 33 sec on upload of hadcm3ivolc_l1g4_2000_80_06011878_0_1.zip
06/03/2009 12:08:59 PM|climateprediction.net|Started upload of hadcm3ivolc_l1g4_2000_80_06011878_0_1.zip
06/03/2009 12:09:01 PM|climateprediction.net|Temporarily failed upload of hadcm3ivolc_l1g4_2000_80_06011878_0_1.zip: HTTP error
06/03/2009 12:09:01 PM|climateprediction.net|Backing off 25 min 52 sec on upload of ...
and on and on.

---

These climate models are big by the very nature of what they\'re doing. And the latest model about to be released has even larger files, even though its only 2 model years long.
None of this is likely to change.
The only way to get around this, is to copy the entire set of folders to some external device, (after COMPLETELY exiting from BOINC), and put it onto another computer where you DO have suitable access. Or else don\'t run these models. There\'s a list of a few other projects here.

\"Backing off\" means that BOINC is going to wait for a while and then try again, in the hope that by then the problem will be fixed.

---

Backups: Here

It is possible to limit network access to particular times of day, using \'Advanced | Preferences\' and selecting the \'network usage\' tab. The upload rate can be slowed as well, if that would help.

However, your off-peak limit of 5 MB looks to be too strict for CPDN. BOINC projects generally expect permission to be obtained for use of PCs and communications, particularly at work. The strict comms limit looks rather like permission would not be granted.

These climate models are big by the very nature of what they\'re doing. And the latest model about to be released has even larger files, even though its only 2 model years long.
None of this is likely to change.
The only way to get around this, is to copy the entire set of folders to some external device, (after COMPLETELY exiting from BOINC), and put it onto another computer where you DO have suitable access. Or else don\'t run these models. There\'s a list of a few other projects here.

\"Backing off\" means that BOINC is going to wait for a while and then try again, in the hope that by then the problem will be fixed.

Unfortunately this is also not an answer. New policies also exclude the use of flash drives or other \"portable devices\"
Too bad I didn\'t know this in advance - it looks like multiple hours of work has just gone up in smoke.

It is possible to limit network access to particular times of day, using \'Advanced | Preferences\' and selecting the \'network usage\' tab. The upload rate can be slowed as well, if that would help.

However, your off-peak limit of 5 MB looks to be too strict for CPDN. BOINC projects generally expect permission to be obtained for use of PCs and communications, particularly at work. The strict comms limit looks rather like permission would not be granted.

The time of day is immaterial also - we are a \"24/7\" shop, so the restrictions always apply. If I could email it, I would, but the email restrictions are for 4MB or less also. Some way to break up the big package into smaller ones is the only answer I see.
During \"off hours\" the size limit increases to 5mb so I could do it with 3 mailings if there were a way to berak it up and an address to send them to.
No such luck apparently.

Hi Wieber

When you\'ve finished these two models I think your best plan will be to attach to another BOINC project with small uploads and downloads. But let\'s see if we can first get the files from these two models to Oxford.

The models\' trickles will be uploading to the server without problems because they\'re so small. How many files will you accumulate until you complete both models? (HADSMs produce one at the end of each of the 3 phases; HADCM 80-year models generate one at the end of each decade.) Did you manage to send some files before the company changed its policy?

There\'s a CPDN manual file upload tool. I\'ve never used it to upload a complete model file, but on the CPDN Beta project we\'ve used it to select particular files and send them. So it may be possible to break up the complete files and send them in parts. (I need confirmation of that from someone who understands the files better than I do.)

If this is possible we would need to get confirmation from one of the programmers that manual partial uploads would be accepted and would be usable.

I must tell you in advance that I found using the upload tool quite time-consuming at first. But when I\'d understood how it worked it was quick and efficient.

---

Cpdn news

Hi and thanks for the response.

As far as I know, this was the first \"big\" package to be sent. Smaller messages are \"continuous\" and do not present a problem (I turn off network comms during \"normal hours, just to cooperate with new policies)

The company policy changed well after I started with BOINC/CLIMATE and they specifically let me do this with the understanding that the policies covering PC usage and communications may change in the future without notice.

They did.

I would have no trouble with using a manual tool, so long as package size could be controlled. That\'s the entire point of the policy, because of limitations on bandwidth and severly antiquated equipment (never mind the hidiously slow and unreliable connection).

Please let me know when you have something, as I would very much like to continue supporting this effort.

For the time being, I have suspended all activity.

I have also been a long time supporter of other projects, and will probably switch once this task is completed and submitted.

I\'d like to emphasize that I stick to my original idea: that either BOINC and/or the projects provide ways to submit smaller packages - user configurable in size - for those of us with similar limitations. I am sure that I am not the only one in the world with this problem.

Hi Wieber

When you\'ve finished these two models I think your best plan will be to attach to another BOINC project with small uploads and downloads. But let\'s see if we can first get the files from these two models to Oxford.

The models\' trickles will be uploading to the server without problems because they\'re so small. How many files will you accumulate until you complete both models? (HADSMs produce one at the end of each of the 3 phases; HADCM 80-year models generate one at the end of each decade.) Did you manage to send some files before the company changed its policy?

There\'s a CPDN manual file upload tool. I\'ve never used it to upload a complete model file, but on the CPDN Beta project we\'ve used it to select particular files and send them. So it may be possible to break up the complete files and send them in parts. (I need confirmation of that from someone who understands the files better than I do.)

If this is possible we would need to get confirmation from one of the programmers that manual partial uploads would be accepted and would be usable.

I must tell you in advance that I found using the upload tool quite time-consuming at first. But when I\'d understood how it worked it was quick and efficient.

Is this a 5MB limit per day ?
According to my rough calculation, you could set the preferences so that maximum upload/download rate is 0.0625 kbs per second.
I\'ve never tried this before, but this should effectively limit the amount of data uploaded per day.
As far as i know, the boinc client software would upload the file in chunks.
So even if you exit the client the next day, it resumes from where is stopped
and not try to upload the entire file again.

---

Hi, wieber. Any chance of linking your PC to a dial-up connection and thus by-passing the corporate network for the CPDN uploads? Slower, of course, but it would side-step the size limitation.

The files BOINC is trying to upload are just standard .ZIP archives of the data files. Provided you are allowed to download, install and run new software (and if you installed BOINC in the first place, that must have been possible in the past), you could get hold of a copy of 7-zip. This would let you decompress the original zip file into a temporary folder, and re-archive the contents using the \'Split to volumes\' option (7-zip supports floppy disk, 1.38MB volumes). You could then tunnel the component parts out of the building, say to your home email address, and re-assemble them to the original archive format at home. Remember to email yourself the original file name, too.

Then, if Tolu is agreeable, upload the file manually from a site without restrictions using the tool that mo.v mentions.

[I\'ve just tried this with a 21,608KB 40-year spinup file, and got 14 sections totalling 18.4 MB. The extra compression in the 7-zip format will be kinder on your corporate email-system - though not by much]

Hi, wieber. Any chance of linking your PC to a dial-up connection and thus by-passing the corporate network for the CPDN uploads? Slower, of course, but it would side-step the size limitation.

Unfortunately, no. 1) no phone access (policy), 2) no modem
(also no usb devices to \"take it home\" and send it (even if I had internet at home, which I do not (I live outside of the city - no cable, no satelite internet)

Is this a 5MB limit per day ?
According to my rough calculation, you could set the preferences so that maximum upload/download rate is 0.0625 kbs per second.
I\'ve never tried this before, but this should effectively limit the amount of data uploaded per day.
As far as i know, the boinc client software would upload the file in chunks.
So even if you exit the client the next day, it resumes from where is stopped
and not try to upload the entire file again.

No, the SIZE limit is the overall package. Somehow the server \"knows\" how big the package is and cuts the transmission after the first few seconds. Nothing to do with SPEED.
I am not trying to fight the policy - I understand and even sympathize - the equipment is old and the connection is pathetic. What I wish for from BOINC and/or the projects would be a way to adjust package SIZE.
If I read the instructions related to the transmission RATE correctly, this is about modem connection speed or somehow being able to tell the network server the rate it is supposed to use. This is not something I have control over, so far as I can tell. If I limit the transmission speed, it just gets cut off even faster, somehow sensing that i am not trying to \"push\" as fast as it can take. This is all \"magic\" to me, which is why \"I WISH...\"

Hi and thanks for the response.

As far as I know, this was the first \"big\" package to be sent. Smaller messages are \"continuous\" and do not present a problem (I turn off network comms during \"normal hours, just to cooperate with new policies)

The company policy changed well after I started with BOINC/CLIMATE and they specifically let me do this with the understanding that the policies covering PC usage and communications may change in the future without notice.

They did.

I would have no trouble with using a manual tool, so long as package size could be controlled. That\'s the entire point of the policy, because of limitations on bandwidth and severly antiquated equipment (never mind the hidiously slow and unreliable connection).

Please let me know when you have something, as I would very much like to continue supporting this effort.

For the time being, I have suspended all activity.

I have also been a long time supporter of other projects, and will probably switch once this task is completed and submitted.

I\'d like to emphasize that I stick to my original idea: that either BOINC and/or the projects provide ways to submit smaller packages - user configurable in size - for those of us with similar limitations. I am sure that I am not the only one in the world with this problem.

Hi Wieber

When you\'ve finished these two models I think your best plan will be to attach to another BOINC project with small uploads and downloads. But let\'s see if we can first get the files from these two models to Oxford.

The models\' trickles will be uploading to the server without problems because they\'re so small. How many files will you accumulate until you complete both models? (HADSMs produce one at the end of each of the 3 phases; HADCM 80-year models generate one at the end of each decade.) Did you manage to send some files before the company changed its policy?

There\'s a CPDN manual file upload tool. I\'ve never used it to upload a complete model file, but on the CPDN Beta project we\'ve used it to select particular files and send them. So it may be possible to break up the complete files and send them in parts. (I need confirmation of that from someone who understands the files better than I do.)

If this is possible we would need to get confirmation from one of the programmers that manual partial uploads would be accepted and would be usable.

I must tell you in advance that I found using the upload tool quite time-consuming at first. But when I\'d understood how it worked it was quick and efficient.

STILL HOPING FOR YOUR HELP.
WHERE CAN I FIND THE MANUAL TOOL YOU MENTION - THIS SEEMS TO BE MY ONLY HOPE,
UNLESS I FIND A WAY TO BREAK UP THE ZIP AND MAIL IT TO MYSELF A PIECE AT A TIME AND THEN ACCESS MY MAIL \"ON THE OUTSIDE\" (A LOCAL INTERNET CAFE)
I STILL DO NOT HAVE AN EMAIL ADDRESS TO SEND THE PIECES TO MORE DIRECTLY, SO THE UPLOAD TOOL YOU DESCRIBE SEEMS TO BE THE BEST BET SO FAR!
DOES IT STILL EXIST?
WHERE? GUESS I SHOULD \"GOOGLE\" AND SEE WHAT I CAN FIND.
THANKS FOR THE IDEA! :)

The upload tool mentioned is for the beta test site, and requires that one of the 2 project people move the data around once it arrives. And they need to know when to expect it to be used, and for what. It\'s only used occasionally, when there\'s a problem with the test models.
This is unlikely to be made available to the public in general because of the manual work involved. The same thing applies to using email, which is why there isn\'t any for this purpose.

If your computer circumstances have changed so much that you can\'t use the automated BOINC system to return files, then I think that you\'re out of options.

I have a similar problem and have worked out what the cause of the HTTP error failure is, well at least in my case.

In my situation all internet connectivity is via a Squid Proxy server which has limits on the size of HTTP upload requests, this limit is currently set to 15 MB.

All the CPDN trickle traffic works fine but when the final upload occurs it gets dropped immediately as the overall HTTP request (either HTTP PUT or POST) body size is > than the Squid request_body_max_size limit.

I would also really love to see a way for the final data to be uploaded in definable chunks, e.g. per meg settings, to get around this problem.

As the CPDN tasks have got bigger I have not been able to successfully upload results lately either and will also have to look at dropping out.

---

I have logged Boinc Ticket #873 - Allow spillting of result zip files into smaller segments to allow for network upload restrictions regarding this.