Task 13013520

Name	hadcm3n_t1s1_1940_40_007311376_1
Workunit	7508806
Created	27 Jun 2011, 22:55:10 UTC
Sent	27 Jun 2011, 22:55:29 UTC
Report deadline	27 Sep 2011, 6:22:40 UTC
Received	18 Aug 2011, 19:51:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1064289
Run time	6 days 21 hours 6 min 58 sec
CPU time	6 days 11 hours 6 min 32 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	2.38 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> ill try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... hadcm3n_6.07_i686-apple-darwin(86207,0xac2232c0) malloc: * error for object 0x8d1c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86207,0xac2232c0) malloc: * error for object 0x8d1c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86207,0xac2232c0) malloc: * error for object 0x8d1c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86207,0xac2232c0) malloc: * error for object 0x8d1c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86207,0xac2232c0) malloc: * error for object 0x8d1c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86207,0xac2232c0) malloc: * error for object 0x8d1c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86207,0xac2232c0) malloc: * error for object 0x8d1c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation 13:39:49 (86207): No heartbeat from core client for 30 sec - exiting 13:39:50 (86207): No heartbeat from core client for 30 sec - exiting 13:39:51 (86207): No heartbeat from core client for 30 sec - exiting 13:39:52 (86207): No heartbeat from core client for 30 sec - exiting 13:39:53 (86207): No heartbeat from core client for 30 sec - exiting 13:39:54 (86207): No heartbeat from core client for 30 sec - exiting 13:39:55 (86207): No heartbeat from core client for 30 sec - exiting 13:39:56 (86207): No heartbeat from core client for 30 sec - exiting 13:39:57 (86207): No heartbeat from core client for 30 sec - exiting 13:39:58 (86207): No heartbeat from core client for 30 sec - exiting 13:39:59 (86207): No heartbeat from core client for 30 sec - exiting 13:40:00 (86207): No heartbeat from core client for 30 sec - exiting 13:40:01 (86207): No heartbeat from core client for 30 sec - exiting 13:40:02 (86207): No heartbeat from core client for 30 sec - exiting 13:40:03 (86207): No heartbeat from core client for 30 sec - exiting 13:40:04 (86207): No heartbeat from core client for 30 sec - exiting 13:40:05 (86207): No heartbeat from core client for 30 sec - exiting 13:40:06 (86207): No heartbeat from core client for 30 sec - exiting 13:40:07 (86207): No heartbeat from core client for 30 sec - exiting 13:40:08 (86207): No heartbeat from core client for 30 sec - exiting 13:40:09 (86207): No heartbeat from core client for 30 sec - exiting 13:40:10 (86207): No heartbeat from core client for 30 sec - exiting 13:40:11 (86207): No heartbeat from core client for 30 sec - exiting 13:40:12 (86207): No heartbeat from core client for 30 sec - exiting 13:40:13 (86207): No heartbeat from core client for 30 sec - exiting 13:40:14 (86207): No heartbeat from core client for 30 sec - exiting 13:40:15 (86207): No heartbeat from core client for 30 sec - exiting 13:40:16 (86207): No heartbeat from core client for 30 sec - exiting 13:40:17 (86207): No heartbeat from core client for 30 sec - exiting 13:40:19 (86207): No heartbeat from core client for 30 sec - exiting 13:40:20 (86207): No heartbeat from core client for 30 sec - exiting 13:40:21 (86207): No heartbeat from core client for 30 sec - exiting 13:40:22 (86207): No heartbeat from core client for 30 sec - exiting 13:40:23 (86207): No heartbeat from core client for 30 sec - exiting 13:40:24 (86207): No heartbeat from core client for 30 sec - exiting 13:40:25 (86207): No heartbeat from core client for 30 sec - exiting 13:40:26 (86207): No heartbeat from core client for 30 sec - exiting 13:40:27 (86207): No heartbeat from core client for 30 sec - exiting 13:40:28 (86207): No heartbeat from core client for 30 sec - exiting 13:40:30 (86207): No heartbeat from core client for 30 sec - exiting 13:40:31 (86207): No heartbeat from core client for 30 sec - exiting 13:40:32 (86207): No heartbeat from core client for 30 sec - exiting 13:40:33 (86207): No heartbeat from core client for 30 sec - exiting 13:40:34 (86207): No heartbeat from core client for 30 sec - exiting 13:40:35 (86207): No heartbeat from core client for 30 sec - exiting 13:40:36 (86207): No heartbeat from core client for 30 sec - exiting 13:40:37 (86207): No heartbeat from core client for 30 sec - exiting 13:40:38 (86207): No heartbeat from core client for 30 sec - exiting 13:40:39 (86207): No heartbeat from core client for 30 sec - exiting 13:40:40 (86207): No heartbeat from core client for 30 sec - exiting 13:40:41 (86207): No heartbeat from core client for 30 sec - exiting 13:40:42 (86207): No heartbeat from core client for 30 sec - exiting 13:40:43 (86207): No heartbeat from core client for 30 sec - exiting 13:40:44 (86207): No heartbeat from core client for 30 sec - exiting 13:40:45 (86207): No heartbeat from core client for 30 sec - exiting 13:40:46 (86207): No heartbeat from core client for 30 sec - exiting 13:40:47 (86207): No heartbeat from core client for 30 sec - exiting 13:40:48 (86207): No heartbeat from core client for 30 sec - exiting 13:40:49 (86207): No heartbeat from core client for 30 sec - exiting 13:40:50 (86207): No heartbeat from core client for 30 sec - exiting 13:40:51 (86207): No heartbeat from core client for 30 sec - exiting 13:40:52 (86207): No heartbeat from core client for 30 sec - exiting 13:40:53 (86207): No heartbeat from core client for 30 sec - exiting 13:40:54 (86207): No heartbeat from core client for 30 sec - exiting 13:40:55 (86207): No heartbeat from core client for 30 sec - exiting 13:40:56 (86207): No heartbeat from core client for 30 sec - exiting 13:40:57 (86207): No heartbeat from core client for 30 sec - exiting 13:40:58 (86207): No heartbeat from core client for 30 sec - exiting 13:40:59 (86207): No heartbeat from core client for 30 sec - exiting 13:41:00 (86207): No heartbeat from core client for 30 sec - exiting 13:41:01 (86207): No heartbeat from core client for 30 sec - exiting 13:41:02 (86207): No heartbeat from core client for 30 sec - exiting 13:41:03 (86207): No heartbeat from core client for 30 sec - exiting 13:41:04 (86207): No heartbeat from core client for 30 sec - exiting 13:41:05 (86207): No heartbeat from core client for 30 sec - exiting 13:41:06 (86207): No heartbeat from core client for 30 sec - exiting 13:41:07 (86207): No heartbeat from core client for 30 sec - exiting 13:41:08 (86207): No heartbeat from core client for 30 sec - exiting 13:41:09 (86207): No heartbeat from core client for 30 sec - exiting 13:41:10 (86207): No heartbeat from core client for 30 sec - exiting 13:41:11 (86207): No heartbeat from core client for 30 sec - exiting 13:41:12 (86207): No heartbeat from core client for 30 sec - exiting 13:41:13 (86207): No heartbeat from core client for 30 sec - exiting 13:41:14 (86207): No heartbeat from core client for 30 sec - exiting 13:41:16 (86207): No heartbeat from core client for 30 sec - exiting 13:41:17 (86207): No heartbeat from core client for 30 sec - exiting 13:41:18 (86207): No heartbeat from core client for 30 sec - exiting 13:41:19 (86207): No heartbeat from core client for 30 sec - exiting 13:41:20 (86207): No heartbeat from core client for 30 sec - exiting 13:41:21 (86207): No heartbeat from core client for 30 sec - exiting 13:41:22 (86207): No heartbeat from core client for 30 sec - exiting 13:41:23 (86207): No heartbeat from core client for 30 sec - exiting 13:41:24 (86207): No heartbeat from core client for 30 sec - exiting 13:41:25 (86207): No heartbeat from core client for 30 sec - exiting 13:41:26 (86207): No heartbeat from core client for 30 sec - exiting 13:41:27 (86207): No heartbeat from core client for 30 sec - exiting 13:41:28 (86207): No heartbeat from core client for 30 sec - exiting 13:41:29 (86207): No heartbeat from core client for 30 sec - exiting 13:41:30 (86207): No heartbeat from core client for 30 sec - exiting 13:41:31 (86207): No heartbeat from core client for 30 sec - exiting 13:41:32 (86207): No heartbeat from core client for 30 sec - exiting 13:41:33 (86207): No heartbeat from core client for 30 sec - exiting 13:41:34 (86207): No heartbeat from core client for 30 sec - exiting 13:41:35 (86207): No heartbeat from core client for 30 sec - exiting 13:41:36 (86207): No heartbeat from core client for 30 sec - exiting 13:41:37 (86207): No heartbeat from core client for 30 sec - exiting 13:41:38 (86207): No heartbeat from core client for 30 sec - exiting 13:41:39 (86207): No heartbeat from core client for 30 sec - exiting 13:41:40 (86207): No heartbeat from core client for 30 sec - exiting 13:41:41 (86207): No heartbeat from core client for 30 sec - exiting 13:41:42 (86207): No heartbeat from core client for 30 sec - exiting 13:41:43 (86207): No heartbeat from core client for 30 sec - exiting 13:41:44 (86207): No heartbeat from core client for 30 sec - exiting 13:41:45 (86207): No heartbeat from core client for 30 sec - exiting 13:41:47 (86207): No heartbeat from core client for 30 sec - exiting 13:41:48 (86207): No heartbeat from core client for 30 sec - exiting 13:41:49 (86207): No heartbeat from core client for 30 sec - exiting 13:41:50 (86207): No heartbeat from core client for 30 sec - exiting 13:41:51 (86207): No heartbeat from core client for 30 sec - exiting 13:41:52 (86207): No heartbeat from core client for 30 sec - exiting 13:41:53 (86207): No heartbeat from core client for 30 sec - exiting 13:41:54 (86207): No heartbeat from core client for 30 sec - exiting 13:41:55 (86207): No heartbeat from core client for 30 sec - exiting 13:41:56 (86207): No heartbeat from core client for 30 sec - exiting 13:41:57 (86207): No heartbeat from core client for 30 sec - exiting 13:41:58 (86207): No heartbeat from core client for 30 sec - exiting 13:41:59 (86207): No heartbeat from core client for 30 sec - exiting 13:42:00 (86207): No heartbeat from core client for 30 sec - exiting 13:42:01 (86207): No heartbeat from core client for 30 sec - exiting 13:42:02 (86207): No heartbeat from core client for 30 sec - exiting 13:42:03 (86207): No heartbeat from core client for 30 sec - exiting 13:42:04 (86207): No heartbeat from core client for 30 sec - exiting 13:42:05 (86207): No heartbeat from core client for 30 sec - exiting 13:42:06 (86207): No heartbeat from core client for 30 sec - exiting 13:42:07 (86207): No heartbeat from core client for 30 sec - exiting 13:42:08 (86207): No heartbeat from core client for 30 sec - exiting 13:42:09 (86207): No heartbeat from core client for 30 sec - exiting 13:42:10 (86207): No heartbeat from core client for 30 sec - exiting 13:42:11 (86207): No heartbeat from core client for 30 sec - exiting 13:42:12 (86207): No heartbeat from core client for 30 sec - exiting 13:42:13 (86207): No heartbeat from core client for 30 sec - exiting 13:42:14 (86207): No heartbeat from core client for 30 sec - exiting 13:42:15 (86207): No heartbeat from core client for 30 sec - exiting 13:42:16 (86207): No heartbeat from core client for 30 sec - exiting 13:42:17 (86207): No heartbeat from core client for 30 sec - exiting 13:42:18 (86207): No heartbeat from core client for 30 sec - exiting 13:42:19 (86207): No heartbeat from core client for 30 sec - exiting 13:42:20 (86207): No heartbeat from core client for 30 sec - exiting 13:42:21 (86207): No heartbeat from core client for 30 sec - exiting 13:42:22 (86207): No heartbeat from core client for 30 sec - exiting 13:42:23 (86207): No heartbeat from core client for 30 sec - exiting 13:42:24 (86207): No heartbeat from core client for 30 sec - exiting 13:42:25 (86207): No heartbeat from core client for 30 sec - exiting 13:42:26 (86207): No heartbeat from core client for 30 sec - exiting 13:42:27 (86207): No heartbeat from core client for 30 sec - exiting 13:42:28 (86207): No heartbeat from core client for 30 sec - exiting 13:42:29 (86207): No heartbeat from core client for 30 sec - exiting 13:42:30 (86207): No heartbeat from core client for 30 sec - exiting 13:42:31 (86207): No heartbeat from core client for 30 sec - exiting 13:42:32 (86207): No heartbeat from core client for 30 sec - exiting 13:42:33 (86207): No heartbeat from core client for 30 sec - exiting 13:42:34 (86207): No heartbeat from core client for 30 sec - exiting 13:42:35 (86207): No heartbeat from core client for 30 sec - exiting 13:42:36 (86207): No heartbeat from core client for 30 sec - exiting 13:42:37 (86207): No heartbeat from core client for 30 sec - exiting 13:42:38 (86207): No heartbeat from core client for 30 sec - exiting 13:42:39 (86207): No heartbeat from core client for 30 sec - exiting 13:42:40 (86207): No heartbeat from core client for 30 sec - exiting 13:42:41 (86207): No heartbeat from core client for 30 sec - exiting 13:42:42 (86207): No heartbeat from core client for 30 sec - exiting 13:42:43 (86207): No heartbeat from core client for 30 sec - exiting 13:42:44 (86207): No heartbeat from core client for 30 sec - exiting 13:42:45 (86207): No heartbeat from core client for 30 sec - exiting 13:42:46 (86207): No heartbeat from core client for 30 sec - exiting 13:42:47 (86207): No heartbeat from core client for 30 sec - exiting 13:42:48 (86207): No heartbeat from core client for 30 sec - exiting 13:42:49 (86207): No heartbeat from core client for 30 sec - exiting 13:42:50 (86207): No heartbeat from core client for 30 sec - exiting 13:42:51 (86207): No heartbeat from core client for 30 sec - exiting 13:42:52 (86207): No heartbeat from core client for 30 sec - exiting 13:42:53 (86207): No heartbeat from core client for 30 sec - exiting 13:42:54 (86207): No heartbeat from core client for 30 sec - exiting 13:42:55 (86207): No heartbeat from core client for 30 sec - exiting 13:42:56 (86207): No heartbeat from core client for 30 sec - exiting 13:42:57 (86207): No heartbeat from core client for 30 sec - exiting 13:42:58 (86207): No heartbeat from core client for 30 sec - exiting 13:42:59 (86207): No heartbeat from core client for 30 sec - exiting 13:43:00 (86207): No heartbeat from core client for 30 sec - exiting 13:43:01 (86207): No heartbeat from core client for 30 sec - exiting 13:43:02 (86207): No heartbeat from core client for 30 sec - exiting 13:43:03 (86207): No heartbeat from core client for 30 sec - exiting 13:43:04 (86207): No heartbeat from core client for 30 sec - exiting 13:43:05 (86207): No heartbeat from core client for 30 sec - exiting 13:43:06 (86207): No heartbeat from core client for 30 sec - exiting 13:43:07 (86207): No heartbeat from core client for 30 sec - exiting 13:43:08 (86207): No heartbeat from core client for 30 sec - exiting 13:43:09 (86207): No heartbeat from core client for 30 sec - exiting 13:43:10 (86207): No heartbeat from core client for 30 sec - exiting 13:43:11 (86207): No heartbeat from core client for 30 sec - exiting 13:43:13 (86207): No heartbeat from core client for 30 sec - exiting 13:43:14 (86207): No heartbeat from core client for 30 sec - exiting 13:43:15 (86207): No heartbeat from core client for 30 sec - exiting 13:43:16 (86207): No heartbeat from core client for 30 sec - exiting 13:43:17 (86207): No heartbeat from core client for 30 sec - exiting 13:43:18 (86207): No heartbeat from core client for 30 sec - exiting 13:43:19 (86207): No heartbeat from core client for 30 sec - exiting 13:43:20 (86207): No heartbeat from core client for 30 sec - exiting 13:43:21 (86207): No heartbeat from core client for 30 sec - exiting 13:43:22 (86207): No heartbeat from core client for 30 sec - exiting 13:43:23 (86207): No heartbeat from core client for 30 sec - exiting 13:43:24 (86207): No heartbeat from core client for 30 sec - exiting 13:43:25 (86207): No heartbeat from core client for 30 sec - exiting 13:43:26 (86207): No heartbeat from core client for 30 sec - exiting 13:43:27 (86207): No heartbeat from core client for 30 sec - exiting 13:43:28 (86207): No heartbeat from core client for 30 sec - exiting 13:43:29 (86207): No heartbeat from core client for 30 sec - exiting 13:43:30 (86207): No heartbeat from core client for 30 sec - exiting 13:43:31 (86207): No heartbeat from core client for 30 sec - exiting 13:43:32 (86207): No heartbeat from core client for 30 sec - exiting 13:43:33 (86207): No heartbeat from core client for 30 sec - exiting 13:43:34 (86207): No heartbeat from core client for 30 sec - exiting 13:43:35 (86207): No heartbeat from core client for 30 sec - exiting 13:43:36 (86207): No heartbeat from core client for 30 sec - exiting 13:43:37 (86207): No heartbeat from core client for 30 sec - exiting 13:43:38 (86207): No heartbeat from core client for 30 sec - exiting 13:43:39 (86207): No heartbeat from core client for 30 sec - exiting 13:43:40 (86207): No heartbeat from core client for 30 sec - exiting 13:43:41 (86207): No heartbeat from core client for 30 sec - exiting 13:43:42 (86207): No heartbeat from core client for 30 sec - exiting 13:43:43 (86207): No heartbeat from core client for 30 sec - exiting 13:43:44 (86207): No heartbeat from core client for 30 sec - exiting 13:43:45 (86207): No heartbeat from core client for 30 sec - exiting 13:43:46 (86207): No heartbeat from core client for 30 sec - exiting 13:43:47 (86207): No heartbeat from core client for 30 sec - exiting 13:43:48 (86207): No heartbeat from core client for 30 sec - exiting 13:43:49 (86207): No heartbeat from core client for 30 sec - exiting 13:43:50 (86207): No heartbeat from core client for 30 sec - exiting 13:43:51 (86207): No heartbeat from core client for 30 sec - exiting 13:43:52 (86207): No heartbeat from core client for 30 sec - exiting 13:43:53 (86207): No heartbeat from core client for 30 sec - exiting 13:43:54 (86207): No heartbeat from core client for 30 sec - exiting 13:43:55 (86207): No heartbeat from core client for 30 sec - exiting 13:43:56 (86207): No heartbeat from core client for 30 sec - exiting 13:43:57 (86207): No heartbeat from core client for 30 sec - exiting 13:43:58 (86207): No heartbeat from core client for 30 sec - exiting 13:43:59 (86207): No heartbeat from core client for 30 sec - exiting 13:44:00 (86207): No heartbeat from core client for 30 sec - exiting 13:44:01 (86207): No heartbeat from core client for 30 sec - exiting 13:44:02 (86207): No heartbeat from core client for 30 sec - exiting 13:44:03 (86207): No heartbeat from core client for 30 sec - exiting 13:44:04 (86207): No heartbeat from core client for 30 sec - exiting 13:44:05 (86207): No heartbeat from core client for 30 sec - exiting 13:44:06 (86207): No heartbeat from core client for 30 sec - exiting 13:44:07 (86207): No heartbeat from core client for 30 sec - exiting 13:44:08 (86207): No heartbeat from core client for 30 sec - exiting 13:44:09 (86207): No heartbeat from core client for 30 sec - exiting 13:44:10 (86207): No heartbeat from core client for 30 sec - exiting 13:44:11 (86207): No heartbeat from core client for 30 sec - exiting 13:44:12 (86207): No heartbeat from core client for 30 sec - exiting 13:44:13 (86207): No heartbeat from core client for 30 sec - exiting 13:44:14 (86207): No heartbeat from core client for 30 sec - exiting 13:44:15 (86207): No heartbeat from core client for 30 sec - exiting 13:44:16 (86207): No heartbeat from core client for 30 sec - exiting 13:44:17 (86207): No heartbeat from core client for 30 sec - exiting 13:44:18 (86207): No heartbeat from core client for 30 sec - exiting 13:44:19 (86207): No heartbeat from core client for 30 sec - exiting 13:44:20 (86207): No heartbeat from core client for 30 sec - exiting Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Signal 15 received, exiting... Called boinc_finish 13:50:24 (397): No heartbeat from core client for 30 sec - exiting 13:50:25 (397): No heartbeat from core client for 30 sec - exiting 13:50:26 (397): No heartbeat from core client for 30 sec - exiting 13:50:27 (397): No heartbeat from core client for 30 sec - exiting 13:50:28 (397): No heartbeat from core client for 30 sec - exiting 13:50:29 (397): No heartbeat from core client for 30 sec - exiting 13:50:30 (397): No heartbeat from core client for 30 sec - exiting 13:50:31 (397): No heartbeat from core client for 30 sec - exiting 13:50:32 (397): No heartbeat from core client for 30 sec - exiting 13:50:33 (397): No heartbeat from core client for 30 sec - exiting 13:50:34 (397): No heartbeat from core client for 30 sec - exiting 13:50:35 (397): No heartbeat from core client for 30 sec - exiting 13:50:36 (397): No heartbeat from core client for 30 sec - exiting 13:50:37 (397): No heartbeat from core client for 30 sec - exiting 13:50:38 (397): No heartbeat from core client for 30 sec - exiting 13:50:39 (397): No heartbeat from core client for 30 sec - exiting 13:50:40 (397): No heartbeat from core client for 30 sec - exiting 13:50:41 (397): No heartbeat from core client for 30 sec - exiting 13:50:42 (397): No heartbeat from core client for 30 sec - exiting 13:50:43 (397): No heartbeat from core client for 30 sec - exiting 13:50:44 (397): No heartbeat from core client for 30 sec - exiting 13:50:45 (397): No heartbeat from core client for 30 sec - exiting 13:50:46 (397): No heartbeat from core client for 30 sec - exiting 13:50:47 (397): No heartbeat from core client for 30 sec - exiting 13:50:48 (397): No heartbeat from core client for 30 sec - exiting 13:50:49 (397): No heartbeat from core client for 30 sec - exiting 13:50:50 (397): No heartbeat from core client for 30 sec - exiting 13:50:51 (397): No heartbeat from core client for 30 sec - exiting 13:50:52 (397): No heartbeat from core client for 30 sec - exiting 13:50:54 (397): No heartbeat from core client for 30 sec - exiting 13:50:55 (397): No heartbeat from core client for 30 sec - exiting 13:50:56 (397): No heartbeat from core client for 30 sec - exiting 13:50:57 (397): No heartbeat from core client for 30 sec - exiting 13:50:58 (397): No heartbeat from core client for 30 sec - exiting 13:50:59 (397): No heartbeat from core client for 30 sec - exiting 13:51:00 (397): No heartbeat from core client for 30 sec - exiting 13:51:01 (397): No heartbeat from core client for 30 sec - exiting 13:51:02 (397): No heartbeat from core client for 30 sec - exiting 13:51:03 (397): No heartbeat from core client for 30 sec - exiting 13:51:04 (397): No heartbeat from core client for 30 sec - exiting 13:51:05 (397): No heartbeat from core client for 30 sec - exiting 13:51:06 (397): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x10c7c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x10c7c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x8fa004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x8fa000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x8fa004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x8fa000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x8fa004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x8fa000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x8fa004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x8fa000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x10cd004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x10db604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x10ea004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x10f8a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(397,0xac8bd2c0) malloc: * error for object 0x10f8a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133445) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=466, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133445) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=466, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133445) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=466, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133445) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=466, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133445) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=466, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133445) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=466, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Aug 2011 19:53:23	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	311,040	558,392	1.7952
04 Jul 2011 20:37:12	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	259,200	463,116	1.7867
04 Jul 2011 05:36:28	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	233,280	416,538	1.7856
03 Jul 2011 13:37:04	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	207,360	370,296	1.7858
02 Jul 2011 21:33:47	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	181,440	323,669	1.7839
02 Jul 2011 06:34:32	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	155,520	277,214	1.7825
01 Jul 2011 14:31:58	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	129,600	230,455	1.7782
30 Jun 2011 22:59:13	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	103,680	184,040	1.7751
30 Jun 2011 02:32:07	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	77,760	137,445	1.7676
29 Jun 2011 08:33:38	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	51,840	91,530	1.7656
28 Jun 2011 14:32:32	1064289	13013520	hadcm3n_t1s1_1940_40_007311376_1	25,920	46,032	1.7759