Task 14580730

Name	hadcm3n_o2ik_2020_40_007857005_3
Workunit	8012117
Created	23 Apr 2012, 13:00:49 UTC
Sent	23 Apr 2012, 13:00:54 UTC
Report deadline	23 Jul 2012, 20:28:05 UTC
Received	3 Jun 2012, 3:13:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	751945
Run time	10 days 13 hours 23 min 24 sec
CPU time	5 days 17 hours 35 min 34 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.51 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.21</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... 14:44:09 (29464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:02:55 (35962): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:58 (35962): No heartbeat from core client for 30 sec - exiting 18:02:59 (35962): No heartbeat from core client for 30 sec - exiting 18:03:00 (35962): No heartbeat from core client for 30 sec - exiting 18:03:01 (35962): No heartbeat from core client for 30 sec - exiting 18:03:02 (35962): No heartbeat from core client for 30 sec - exiting 18:03:03 (35962): No heartbeat from core client for 30 sec - exiting 18:03:04 (35962): No heartbeat from core client for 30 sec - exiting 18:03:05 (35962): No heartbeat from core client for 30 sec - exiting 18:03:06 (35962): No heartbeat from core client for 30 sec - exiting 18:03:07 (35962): No heartbeat from core client for 30 sec - exiting 18:03:08 (35962): No heartbeat from core client for 30 sec - exiting 18:03:09 (35962): No heartbeat from core client for 30 sec - exiting 18:03:10 (35962): No heartbeat from core client for 30 sec - exiting 18:03:14 (35962): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:06:53 (49569): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:11:39 (49755): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:11:43 (49755): No heartbeat from core client for 30 sec - exiting 16:11:44 (49755): No heartbeat from core client for 30 sec - exiting 16:11:45 (49755): No heartbeat from core client for 30 sec - exiting 16:11:46 (49755): No heartbeat from core client for 30 sec - exiting 16:30:51 (49798): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x104b004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x85ee04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x85ee00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x85ee04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x85ee00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x843804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x104b004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77620,0xa0ab8540) malloc: * error for object 0x1066600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation 12:57:32 (77620): No heartbeat from core client for 30 sec - exiting 12:57:37 (77620): No heartbeat from core client for 30 sec - exiting 12:57:38 (77620): No heartbeat from core client for 30 sec - exiting 12:57:39 (77620): No heartbeat from core client for 30 sec - exiting 12:57:40 (77620): No heartbeat from core client for 30 sec - exiting 12:57:41 (77620): No heartbeat from core client for 30 sec - exiting 12:57:42 (77620): No heartbeat from core client for 30 sec - exiting 12:57:43 (77620): No heartbeat from core client for 30 sec - exiting 12:57:44 (77620): No heartbeat from core client for 30 sec - exiting 12:57:45 (77620): No heartbeat from core client for 30 sec - exiting 12:57:46 (77620): No heartbeat from core client for 30 sec - exiting 12:57:47 (77620): No heartbeat from core client for 30 sec - exiting 12:57:48 (77620): No heartbeat from core client for 30 sec - exiting 12:57:49 (77620): No heartbeat from core client for 30 sec - exiting 12:57:50 (77620): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=77622, selfPID=77622, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(80213,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80213,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80213,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80213,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80213,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80213,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80213,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x841004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x103ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x1039e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x1039e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x1039e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x1039e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x1039e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x1039e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99620,0xa0ab8540) malloc: * error for object 0x1039e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation 05:28:09 (99620): No heartbeat from core client for 30 sec - exiting 05:28:11 (99620): No heartbeat from core client for 30 sec - exiting 05:28:12 (99620): No heartbeat from core client for 30 sec - exiting 20:51:24 (99620): No heartbeat from core client for 30 sec - exiting 14:18:49 (99620): No heartbeat from core client for 30 sec - exiting 14:18:51 (99620): No heartbeat from core client for 30 sec - exiting 14:18:52 (99620): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-apple-darwin(50341,0xa0ab8540) malloc: * error for object 0x843604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50341,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(156,0xa0ab8540) malloc: * error for object 0x83e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(156,0xa0ab8540) malloc: * error for object 0x83e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(156,0xa0ab8540) malloc: * error for object 0x83e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(156,0xa0ab8540) malloc: * error for object 0x83e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(156,0xa0ab8540) malloc: * error for object 0x83e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x85b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x1820604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x1820604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(46146,0xa0ab8540) malloc: * error for object 0x83e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(56741,0xa0ab8540) malloc: * error for object 0x847604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(56741,0xa0ab8540) malloc: * error for object 0x847604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x855004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x855004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84280,0xa0ab8540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug 19:25:44 (84280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:53 (84361): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:58 (84361): No heartbeat from core client for 30 sec - exiting 20:45:59 (84361): No heartbeat from core client for 30 sec - exiting 21:17:59 (84899): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:17 (84899): No heartbeat from core client for 30 sec - exiting 21:18:18 (84899): No heartbeat from core client for 30 sec - exiting 21:18:20 (84899): No heartbeat from core client for 30 sec - exiting 21:18:21 (84899): No heartbeat from core client for 30 sec - exiting 21:18:22 (84899): No heartbeat from core client for 30 sec - exiting 21:18:23 (84899): No heartbeat from core client for 30 sec - exiting 21:18:24 (84899): No heartbeat from core client for 30 sec - exiting 21:18:25 (84899): No heartbeat from core client for 30 sec - exiting 21:18:26 (84899): No heartbeat from core client for 30 sec - exiting 21:18:27 (84899): No heartbeat from core client for 30 sec - exiting 21:18:28 (84899): No heartbeat from core client for 30 sec - exiting 21:18:29 (84899): No heartbeat from core client for 30 sec - exiting 21:18:30 (84899): No heartbeat from core client for 30 sec - exiting 21:18:31 (84899): No heartbeat from core client for 30 sec - exiting 21:18:32 (84899): No heartbeat from core client for 30 sec - exiting 21:18:33 (84899): No heartbeat from core client for 30 sec - exiting 21:18:34 (84899): No heartbeat from core client for 30 sec - exiting 21:18:35 (84899): No heartbeat from core client for 30 sec - exiting 21:31:39 (85128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:31:52 (85128): No heartbeat from core client for 30 sec - exiting 21:31:53 (85128): No heartbeat from core client for 30 sec - exiting 21:31:54 (85128): No heartbeat from core client for 30 sec - exiting 21:31:55 (85128): No heartbeat from core client for 30 sec - exiting 21:31:56 (85128): No heartbeat from core client for 30 sec - exiting 21:31:57 (85128): No heartbeat from core client for 30 sec - exiting 21:31:58 (85128): No heartbeat from core client for 30 sec - exiting 22:17:18 (85221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:17:22 (85221): No heartbeat from core client for 30 sec - exiting 23:46:15 (85524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:02:00 (86114): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:19:04 (86218): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:19:07 (86218): No heartbeat from core client for 30 sec - exiting 00:19:08 (86218): No heartbeat from core client for 30 sec - exiting 00:19:09 (86218): No heartbeat from core client for 30 sec - exiting 00:19:10 (86218): No heartbeat from core client for 30 sec - exiting 00:19:11 (86218): No heartbeat from core client for 30 sec - exiting 00:19:12 (86218): No heartbeat from core client for 30 sec - exiting 00:19:13 (86218): No heartbeat from core client for 30 sec - exiting 00:19:14 (86218): No heartbeat from core client for 30 sec - exiting 00:19:15 (86218): No heartbeat from core client for 30 sec - exiting 00:19:16 (86218): No heartbeat from core client for 30 sec - exiting 00:19:17 (86218): No heartbeat from core client for 30 sec - exiting 00:19:18 (86218): No heartbeat from core client for 30 sec - exiting 00:19:19 (86218): No heartbeat from core client for 30 sec - exiting 00:19:20 (86218): No heartbeat from core client for 30 sec - exiting 00:19:21 (86218): No heartbeat from core client for 30 sec - exiting 00:19:22 (86218): No heartbeat from core client for 30 sec - exiting 00:19:23 (86218): No heartbeat from core client for 30 sec - exiting 00:19:24 (86218): No heartbeat from core client for 30 sec - exiting 00:19:25 (86218): No heartbeat from core client for 30 sec - exiting 00:19:26 (86218): No heartbeat from core client for 30 sec - exiting 00:19:27 (86218): No heartbeat from core client for 30 sec - exiting 00:52:15 (86329): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:28:03 (86557): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:04 (86799): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:31 (87340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:54 (87340): No heartbeat from core client for 30 sec - exiting 03:42:55 (87340): No heartbeat from core client for 30 sec - exiting 03:42:56 (87340): No heartbeat from core client for 30 sec - exiting 03:42:57 (87340): No heartbeat from core client for 30 sec - exiting 03:42:58 (87340): No heartbeat from core client for 30 sec - exiting 03:42:59 (87340): No heartbeat from core client for 30 sec - exiting 03:43:00 (87340): No heartbeat from core client for 30 sec - exiting 04:04:10 (87806): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:14:10 (87953): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:57:39 (88027): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:57:41 (88027): No heartbeat from core client for 30 sec - exiting 04:57:42 (88027): No heartbeat from core client for 30 sec - exiting 04:57:43 (88027): No heartbeat from core client for 30 sec - exiting 04:57:44 (88027): No heartbeat from core client for 30 sec - exiting 04:57:45 (88027): No heartbeat from core client for 30 sec - exiting 04:57:46 (88027): No heartbeat from core client for 30 sec - exiting 04:57:47 (88027): No heartbeat from core client for 30 sec - exiting 04:57:48 (88027): No heartbeat from core client for 30 sec - exiting 04:57:49 (88027): No heartbeat from core client for 30 sec - exiting 04:57:50 (88027): No heartbeat from core client for 30 sec - exiting 04:57:51 (88027): No heartbeat from core client for 30 sec - exiting 04:57:52 (88027): No heartbeat from core client for 30 sec - exiting 04:57:53 (88027): No heartbeat from core client for 30 sec - exiting 04:57:54 (88027): No heartbeat from core client for 30 sec - exiting 04:57:55 (88027): No heartbeat from core client for 30 sec - exiting 04:57:56 (88027): No heartbeat from core client for 30 sec - exiting 04:57:57 (88027): No heartbeat from core client for 30 sec - exiting 04:57:58 (88027): No heartbeat from core client for 30 sec - exiting 04:57:59 (88027): No heartbeat from core client for 30 sec - exiting 04:58:00 (88027): No heartbeat from core client for 30 sec - exiting 04:58:01 (88027): No heartbeat from core client for 30 sec - exiting 04:58:02 (88027): No heartbeat from core client for 30 sec - exiting 04:58:03 (88027): No heartbeat from core client for 30 sec - exiting 04:58:04 (88027): No heartbeat from core client for 30 sec - exiting 04:58:05 (88027): No heartbeat from core client for 30 sec - exiting 04:58:06 (88027): No heartbeat from core client for 30 sec - exiting 04:58:07 (88027): No heartbeat from core client for 30 sec - exiting 04:58:08 (88027): No heartbeat from core client for 30 sec - exiting 04:58:09 (88027): No heartbeat from core client for 30 sec - exiting 04:58:10 (88027): No heartbeat from core client for 30 sec - exiting 04:58:11 (88027): No heartbeat from core client for 30 sec - exiting 04:58:12 (88027): No heartbeat from core client for 30 sec - exiting 04:58:13 (88027): No heartbeat from core client for 30 sec - exiting 04:58:14 (88027): No heartbeat from core client for 30 sec - exiting 04:58:15 (88027): No heartbeat from core client for 30 sec - exiting 04:58:16 (88027): No heartbeat from core client for 30 sec - exiting 04:58:17 (88027): No heartbeat from core client for 30 sec - exiting 04:58:18 (88027): No heartbeat from core client for 30 sec - exiting 04:58:19 (88027): No heartbeat from core client for 30 sec - exiting 04:58:20 (88027): No heartbeat from core client for 30 sec - exiting 04:58:21 (88027): No heartbeat from core client for 30 sec - exiting 04:58:22 (88027): No heartbeat from core client for 30 sec - exiting 04:58:23 (88027): No heartbeat from core client for 30 sec - exiting 04:58:24 (88027): No heartbeat from core client for 30 sec - exiting 04:58:25 (88027): No heartbeat from core client for 30 sec - exiting 05:19:51 (88310): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:26:06 (2366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:26:10 (2366): No heartbeat from core client for 30 sec - exiting 12:26:11 (2366): No heartbeat from core client for 30 sec - exiting 12:26:12 (2366): No heartbeat from core client for 30 sec - exiting 12:26:13 (2366): No heartbeat from core client for 30 sec - exiting 12:26:14 (2366): No heartbeat from core client for 30 sec - exiting 12:26:15 (2366): No heartbeat from core client for 30 sec - exiting 12:26:16 (2366): No heartbeat from core client for 30 sec - exiting 12:26:17 (2366): No heartbeat from core client for 30 sec - exiting 12:26:48 (2366): No heartbeat from core client for 30 sec - exiting 12:26:49 (2366): No heartbeat from core client for 30 sec - exiting 12:26:50 (2366): No heartbeat from core client for 30 sec - exiting 12:26:51 (2366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:45:33 (3745): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:45:34 (3745): No heartbeat from core client for 30 sec - exiting 15:45:35 (3745): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 17:21:50 (4374): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:21:30 (5706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:21:31 (5706): No heartbeat from core client for 30 sec - exiting 23:21:32 (5706): No heartbeat from core client for 30 sec - exiting 23:21:33 (5706): No heartbeat from core client for 30 sec - exiting 23:21:34 (5706): No heartbeat from core client for 30 sec - exiting 23:21:35 (5706): No heartbeat from core client for 30 sec - exiting 23:21:36 (5706): No heartbeat from core client for 30 sec - exiting 00:14:23 (6883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:14:58 (6883): No heartbeat from core client for 30 sec - exiting 00:14:59 (6883): No heartbeat from core client for 30 sec - exiting 00:15:00 (6883): No heartbeat from core client for 30 sec - exiting 00:15:01 (6883): No heartbeat from core client for 30 sec - exiting 00:15:02 (6883): No heartbeat from core client for 30 sec - exiting 00:15:03 (6883): No heartbeat from core client for 30 sec - exiting 00:15:04 (6883): No heartbeat from core client for 30 sec - exiting 00:15:05 (6883): No heartbeat from core client for 30 sec - exiting 00:15:06 (6883): No heartbeat from core client for 30 sec - exiting 00:15:07 (6883): No heartbeat from core client for 30 sec - exiting 00:15:08 (6883): No heartbeat from core client for 30 sec - exiting 00:15:09 (6883): No heartbeat from core client for 30 sec - exiting 00:15:10 (6883): No heartbeat from core client for 30 sec - exiting 00:15:11 (6883): No heartbeat from core client for 30 sec - exiting 00:15:12 (6883): No heartbeat from core client for 30 sec - exiting 00:15:13 (6883): No heartbeat from core client for 30 sec - exiting 00:15:14 (6883): No heartbeat from core client for 30 sec - exiting 00:15:15 (6883): No heartbeat from core client for 30 sec - exiting 00:15:16 (6883): No heartbeat from core client for 30 sec - exiting 00:15:17 (6883): No heartbeat from core client for 30 sec - exiting 00:15:18 (6883): No heartbeat from core client for 30 sec - exiting 00:15:19 (6883): No heartbeat from core client for 30 sec - exiting 00:15:20 (6883): No heartbeat from core client for 30 sec - exiting 00:15:21 (6883): No heartbeat from core client for 30 sec - exiting 00:15:22 (6883): No heartbeat from core client for 30 sec - exiting 00:15:23 (6883): No heartbeat from core client for 30 sec - exiting 00:15:24 (6883): No heartbeat from core client for 30 sec - exiting 00:15:25 (6883): No heartbeat from core client for 30 sec - exiting 00:15:26 (6883): No heartbeat from core client for 30 sec - exiting 00:58:08 (7236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:11:17 (7525): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:29:00 (7747): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:31:13 (8164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:31:19 (8164): No heartbeat from core client for 30 sec - exiting 04:52:01 (8706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:54:07 (10346): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:10:04 (10472): No heartbeat from core client for 30 sec - exiting 08:10:16 (10472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 May 2012 16:35:28	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	259,200	448,437	1.7301
14 May 2012 00:22:50	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	233,280	402,172	1.7240
12 May 2012 18:16:25	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	207,360	356,588	1.7197
11 May 2012 19:18:31	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	181,440	312,817	1.7241
10 May 2012 19:43:55	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	155,520	268,800	1.7284
09 May 2012 19:31:46	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	129,600	224,923	1.7355
08 May 2012 03:41:50	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	103,680	180,564	1.7416
01 May 2012 17:14:21	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	77,760	135,368	1.7408
30 Apr 2012 16:05:46	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	51,840	90,759	1.7508
29 Apr 2012 08:57:18	751945	14580730	hadcm3n_o2ik_2020_40_007857005_3	25,920	46,042	1.7763