climateprediction.net home page
Trickles not showing.

Trickles not showing.

Message boards : Number crunching : Trickles not showing.
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · Next

AuthorMessage
Profile Pooh Bear 27
Avatar

Send message
Joined: 5 Feb 05
Posts: 465
Credit: 1,914,189
RAC: 0
Message 32760 - Posted: 27 Feb 2008, 16:46:00 UTC

There were several servers that got low on space, and that was why there was an outage. They moved some of the data and got the servers back up, but are still working on getting more data moved, which takes a lot of background time and bandwidth. Once they are at a comfortable level of move, they will turn the trickle crediting process back on (which also takes a lot of background time and bandwidth and why it is only run once a day).
ID: 32760 · Report as offensive     Reply Quote
old_user219190

Send message
Joined: 14 Jan 07
Posts: 52
Credit: 284,001
RAC: 0
Message 32761 - Posted: 27 Feb 2008, 17:01:57 UTC

Hi thanks for the info Pooh Bear.
If the situation is still ongoing everything falls into place.
The reason of my origional post was the quote from the news thread,

Many of you will have noticed Monday\'s CPDN server outage. Tolu worked all Monday and a good part of the night on unscheduled server maintenance, mostly moving large volumes of data. All the CPDN servers are again fully functional.

which clearly is not the case.
But why was this, if known, not reported earlier!
Thanks again




ID: 32761 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 32765 - Posted: 28 Feb 2008, 4:31:54 UTC

Hi Chris

When I posted that everything was fully-functional again, I thought it was. There are obviously additional functions that have to be enabled individually and don\'t show up as on or off on the server status page. I just can\'t trouble Tolu any more over this unless it\'s a problem affecting the running of the models (which it isn\'t) because he\'s holding the fort alone in Oxford while Milo\'s away all week helping with climate modelling in Italy.

There have been previous delays in trickles showing on our model web pages, or credit exports to the stats sites, and everything has always sorted itself out eventually. If the delay is longer than usual this time we\'ll just have to be patient. Hope everyone will understand.
Cpdn news
ID: 32765 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,778,421
RAC: 0
Message 32767 - Posted: 28 Feb 2008, 6:43:21 UTC - in response to Message 32765.  

By the way, another function not apparently working is the account updating process. One can change resource share, for example, and it shows the updated session on the account, but it does not reflect the change when you do an update from the client side.


Hi Chris

When I posted that everything was fully-functional again, I thought it was. There are obviously additional functions that have to be enabled individually and don\'t show up as on or off on the server status page. I just can\'t trouble Tolu any more over this unless it\'s a problem affecting the running of the models (which it isn\'t) because he\'s holding the fort alone in Oxford while Milo\'s away all week helping with climate modelling in Italy.

There have been previous delays in trickles showing on our model web pages, or credit exports to the stats sites, and everything has always sorted itself out eventually. If the delay is longer than usual this time we\'ll just have to be patient. Hope everyone will understand.


ID: 32767 · Report as offensive     Reply Quote
Profile John Hunt
Avatar

Send message
Joined: 5 Mar 05
Posts: 64
Credit: 790,577
RAC: 0
Message 32768 - Posted: 28 Feb 2008, 7:56:51 UTC
Last modified: 28 Feb 2008, 7:58:00 UTC

I won\'t quote mo.v\'s post again, but it is clear that Tolu has a lot on his plate at the moment with Milo being away!
This delay in credits granting has happened before but it always works out eventually - all that happens is you get a large award of credits on one day.

I\'ve been eagerly awaiting a milestone on CPDN - 100k credits - for the last couple of days now....
(it is reported here, but not on the stats sites yet!)

ID: 32768 · Report as offensive     Reply Quote
old_user219190

Send message
Joined: 14 Jan 07
Posts: 52
Credit: 284,001
RAC: 0
Message 32769 - Posted: 28 Feb 2008, 13:13:58 UTC

Hi Mo
Not universally known but there are three things in life that are certainties:
Death,Taxes and CPDN server problems.
Tis all part of the joys of crunching whether its end of model zip files bouncing around for a week or so or Killer 99 trickles.Thanks to the endeavors of those running the project, soon, all is well again.
But that was not my point.
I don\'t think it is over the top to say your devotion, the time, the effort you must spend helping others truly makes you one of the main assets of the project.[/brown nosing]
But that was not my point.
Yes credits and stat pages are fun, but with my meager resources and choices of projects (two of which are not the best sources of credit and I don\'t mean this one)excludes me from any race.
But that was not my point.
The point I was making in a thread titled \"Trickles not showing\" Was just that.
It was thought at the time the servers were up and running OK but just noticed
they were not.
But could I get this point across?could I heck.
Then out of the dark clouds came a shaft of light in the shape of Pooh Bear,and all became clear;)
Taken that it is very busy behind the scenes,still it only takes seconds to communicate,perhaps some encouragement can be given to those that turn off one of the processes to relay the info someway.
Thanks.
Chris.



ID: 32769 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,778,421
RAC: 0
Message 32823 - Posted: 3 Mar 2008, 8:01:16 UTC

Note, there is still one function which has not been recovered -- even after the servers have been reported as being up and running -- resource share changes, which appear to \'take\', are NOT being transmitted back to the workstations.

I would hope that the additional tweaks which are in view this coming week include that basic functionality.

ID: 32823 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,778,421
RAC: 0
Message 32824 - Posted: 3 Mar 2008, 8:04:17 UTC - in response to Message 32769.  

BOINC project server issues are generic to BOINC, SETI\'s validator right now is frozen and when SETI runs into problems, a LOT of folks go into panic mode -- and quickly. SETI has a LOT more resources to work with compared to Climate and still has problems.




Not universally known but there are three things in life that are certainties:
Death,Taxes and CPDN server problems.


ID: 32824 · Report as offensive     Reply Quote
Professor Desty Nova
Avatar

Send message
Joined: 19 Sep 04
Posts: 92
Credit: 1,926,211
RAC: 254
Message 32831 - Posted: 3 Mar 2008, 16:29:33 UTC

The trickles are catching up :-)))


Professor Desty Nova
Researching Karma the Hard Way
ID: 32831 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 32833 - Posted: 3 Mar 2008, 19:08:19 UTC
Last modified: 3 Mar 2008, 19:08:41 UTC

Hi Barry, you said

Note, there is still one function which has not been recovered -- even after the servers have been reported as being up and running -- resource share changes, which appear to \'take\', are NOT being transmitted back to the workstations.


You\'ve probably seen from the news thread that Milo has to do more data-shifting on the CPDN servers this week. If a couple of days after I (hope I can) post in the news that the server jobs are finished, you still notice this problem, could you please post again in this thread & mention it. One of us would then point this out to Milo.

I hope that seems a reasonable course of action.

Cpdn news
ID: 32833 · Report as offensive     Reply Quote
glaesum

Send message
Joined: 24 Feb 06
Posts: 47
Credit: 782,082
RAC: 0
Message 32836 - Posted: 3 Mar 2008, 22:30:27 UTC
Last modified: 3 Mar 2008, 22:32:25 UTC

I\'m in Oct 2079 on a model with two final trickles to complete it. Is it safer to hold back the final trickle of the model until the current problems are completely cleared up? My trickles are have caught up now and all the credits exported except one lot on each model. Plenty more to crunch meanwhile on the replacement model which is only at 1925!
ID: 32836 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 32841 - Posted: 4 Mar 2008, 0:12:47 UTC
Last modified: 4 Mar 2008, 0:20:36 UTC

Hi Glaesum

What you\'re suggesting is a good idea. In the case of CPDN, the server only classifies a model as abandoned if it hasn\'t trickled for I think 6 weeks. So you could safely suspend network activity and just let the final zip file languish in the Transfers window until Milo\'s finished all the server jobs. Or you could suspend the model before it reaches December of its last year.

Some members with computers they only have intermittent access to may allow network activity once per week or per month. Accumulating trickles and files like this does no harm at all.

The general opinion seems to be that there\'s a certain, albeit small, risk in attempting to upload a zip file and failing.

Another good idea is to back models up before they are due to create zip files. Then if necessary you can make them create the file again.
Cpdn news
ID: 32841 · Report as offensive     Reply Quote
glaesum

Send message
Joined: 24 Feb 06
Posts: 47
Credit: 782,082
RAC: 0
Message 32852 - Posted: 4 Mar 2008, 16:30:16 UTC

Ok thanks, since I\'m running various projects, suspending the task is the better option here although I did suspend network activity for a couple of days last week when the server probs were at their worst. Indeed I also backed up the model at 98% to protect the final lunge to the finish post.

I can see there is still quite a delay before trickles are showing in the model results. Keep us posted with Milo\'s progress - we\'re all cheering him on.

/pg
ID: 32852 · Report as offensive     Reply Quote
old_user428438

Send message
Joined: 1 Feb 07
Posts: 26
Credit: 885,216
RAC: 0
Message 32854 - Posted: 4 Mar 2008, 16:55:04 UTC - in response to Message 32852.  

Ok thanks, since I\'m running various projects, suspending the task is the better option here although I did suspend network activity for a couple of days last week when the server probs were at their worst. Indeed I also backed up the model at 98% to protect the final lunge to the finish post.

I can see there is still quite a delay before trickles are showing in the model results. Keep us posted with Milo\'s progress - we\'re all cheering him on.

/pg


I second the \"keep us posted\" sentiment - pretty please?

I\'ve just suspended a model that is a couple of trickles and the zip away from completion.

F.
ID: 32854 · Report as offensive     Reply Quote
glaesum

Send message
Joined: 24 Feb 06
Posts: 47
Credit: 782,082
RAC: 0
Message 32874 - Posted: 6 Mar 2008, 23:08:10 UTC

I\'ve run my 80yr coupled model to within 3hrs of completion and, with the end of the week coming, I wondered how things were getting on and whether it is deemed safe now to send the final trickle and result upload zips?
ID: 32874 · Report as offensive     Reply Quote
old_user428438

Send message
Joined: 1 Feb 07
Posts: 26
Credit: 885,216
RAC: 0
Message 32875 - Posted: 6 Mar 2008, 23:30:24 UTC - in response to Message 32874.  

I\'ve run my 80yr coupled model to within 3hrs of completion and, with the end of the week coming, I wondered how things were getting on and whether it is deemed safe now to send the final trickle and result upload zips?

Trickles are still not getting through promptly. I would hang off until the middle of next week and then assess the situation again if I were you.

F.
ID: 32875 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 32876 - Posted: 7 Mar 2008, 3:26:30 UTC


Well it\'s said that \"No news is good news\".
However, we haven\'t heard from Milo yet, so it this case it\'s \"No news is bad news\".

And, as old hands will know, the weekend is when Murphy likes to show up and drop a spanner in the works.


Backups: Here
ID: 32876 · Report as offensive     Reply Quote
glaesum

Send message
Joined: 24 Feb 06
Posts: 47
Credit: 782,082
RAC: 0
Message 32916 - Posted: 11 Mar 2008, 15:01:38 UTC
Last modified: 11 Mar 2008, 15:02:04 UTC

the latest trickle (a couple of hours ago) showed on the database within a minute and there was a general catch up of more trickles yesterday, the 10th. so I think that is a cue to run out the old model to completion. credits seem to follow within the day.

it wasn\'t impatience merely for the trickles, just wanting to follow up on the other thread as my new model is running even slower and I\'d like it to crunch on its own for a while first.

ID: 32916 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 32917 - Posted: 11 Mar 2008, 16:10:52 UTC - in response to Message 32916.  

I just had a huge jump in RAC, so they must have processed my trickles and zip files last night. :)
ID: 32917 · Report as offensive     Reply Quote
old_user428438

Send message
Joined: 1 Feb 07
Posts: 26
Credit: 885,216
RAC: 0
Message 32918 - Posted: 11 Mar 2008, 16:41:52 UTC - in response to Message 32917.  

I just had a huge jump in RAC, so they must have processed my trickles and zip files last night. :)

All my trickles caught up yesterday (but not in time for the overnight update of stats - that comes tonight :)

Perfect timing as my latest model is due to complete in about 20 hours, then I can let the one that I paused off the leash to finish its last couple of trickles and the zip. So, by my calculation, that has taken about 10 days to get things sorted out.

F.
ID: 32918 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Number crunching : Trickles not showing.

©2024 climateprediction.net