Name | hadcm3n_006o_1980_40_008000673_2 |
Workunit | 8155787 |
Created | 6 Aug 2012, 13:52:49 UTC |
Sent | 6 Aug 2012, 13:53:10 UTC |
Report deadline | 5 Nov 2012, 21:20:21 UTC |
Received | 22 Oct 2012, 7:46:51 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 22 (0x00000016) Unknown error code |
Computer ID | 1179185 |
Run time | 28 days 16 hours 34 min 56 sec |
CPU time | 28 days 16 hours 34 min 56 sec |
Validate state | Invalid |
Credit | 5,909.76 |
Device peak FLOPS | 1.43 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu |
Stderr | <core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09b699d8 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 14:34:19 (1273): No heartbeat from core client for 30 sec - exiting 14:35:29 (1273): No heartbeat from core client for 30 sec - exiting 14:35:30 (1273): No heartbeat from core client for 30 sec - exiting 14:35:31 (1273): No heartbeat from core client for 30 sec - exiting 14:35:32 (1273): No heartbeat from core client for 30 sec - exiting 14:35:33 (1273): No heartbeat from core client for 30 sec - exiting 14:35:34 (1273): No heartbeat from core client for 30 sec - exiting 14:35:35 (1273): No heartbeat from core client for 30 sec - exiting 14:35:36 (1273): No heartbeat from core client for 30 sec - exiting 14:35:37 (1273): No heartbeat from core client for 30 sec - exiting 14:35:38 (1273): No heartbeat from core client for 30 sec - exiting 14:35:39 (1273): No heartbeat from core client for 30 sec - exiting 14:35:40 (1273): No heartbeat from core client for 30 sec - exiting 14:35:41 (1273): No heartbeat from core client for 30 sec - exiting 14:35:42 (1273): No heartbeat from core client for 30 sec - exiting 14:35:43 (1273): No heartbeat from core client for 30 sec - exiting 14:35:44 (1273): No heartbeat from core client for 30 sec - exiting 14:35:45 (1273): No heartbeat from core client for 30 sec - exiting 14:35:46 (1273): No heartbeat from core client for 30 sec - exiting 14:35:47 (1273): No heartbeat from core client for 30 sec - exiting 14:35:48 (1273): No heartbeat from core client for 30 sec - exiting 14:35:49 (1273): No heartbeat from core client for 30 sec - exiting 14:35:50 (1273): No heartbeat from core client for 30 sec - exiting 14:35:51 (1273): No heartbeat from core client for 30 sec - exiting 14:36:13 (1273): No heartbeat from core client for 30 sec - exiting 14:36:14 (1273): No heartbeat from core client for 30 sec - exiting 14:36:15 (1273): No heartbeat from core client for 30 sec - exiting 14:36:16 (1273): No heartbeat from core client for 30 sec - exiting 14:36:17 (1273): No heartbeat from core client for 30 sec - exiting 14:36:18 (1273): No heartbeat from core client for 30 sec - exiting 14:36:19 (1273): No heartbeat from core client for 30 sec - exiting 14:36:20 (1273): No heartbeat from core client for 30 sec - exiting 14:36:21 (1273): No heartbeat from core client for 30 sec - exiting 14:36:22 (1273): No heartbeat from core client for 30 sec - exiting 14:36:23 (1273): No heartbeat from core client for 30 sec - exiting 14:36:24 (1273): No heartbeat from core client for 30 sec - exiting 14:36:25 (1273): No heartbeat from core client for 30 sec - exiting 14:36:26 (1273): No heartbeat from core client for 30 sec - exiting 14:36:27 (1273): No heartbeat from core client for 30 sec - exiting 14:36:28 (1273): No heartbeat from core client for 30 sec - exiting 14:36:29 (1273): No heartbeat from core client for 30 sec - exiting 14:36:30 (1273): No heartbeat from core client for 30 sec - exiting 14:36:31 (1273): No heartbeat from core client for 30 sec - exiting 14:36:32 (1273): No heartbeat from core client for 30 sec - exiting 14:36:33 (1273): No heartbeat from core client for 30 sec - exiting 14:36:34 (1273): No heartbeat from core client for 30 sec - exiting 14:36:35 (1273): No heartbeat from core client for 30 sec - exiting 14:36:36 (1273): No heartbeat from core client for 30 sec - exiting 14:36:37 (1273): No heartbeat from core client for 30 sec - exiting 14:36:38 (1273): No heartbeat from core client for 30 sec - exiting 14:36:39 (1273): No heartbeat from core client for 30 sec - exiting 14:36:40 (1273): No heartbeat from core client for 30 sec - exiting 14:36:41 (1273): No heartbeat from core client for 30 sec - exiting 14:36:42 (1273): No heartbeat from core client for 30 sec - exiting 14:36:43 (1273): No heartbeat from core client for 30 sec - exiting 14:36:44 (1273): No heartbeat from core client for 30 sec - exiting 14:36:45 (1273): No heartbeat from core client for 30 sec - exiting 14:36:46 (1273): No heartbeat from core client for 30 sec - exiting 14:36:47 (1273): No heartbeat from core client for 30 sec - exiting 14:36:48 (1273): No heartbeat from core client for 30 sec - exiting 14:36:49 (1273): No heartbeat from core client for 30 sec - exiting 14:36:50 (1273): No heartbeat from core client for 30 sec - exiting 14:36:51 (1273): No heartbeat from core client for 30 sec - exiting 14:36:52 (1273): No heartbeat from core client for 30 sec - exiting 14:36:53 (1273): No heartbeat from core client for 30 sec - exiting 14:36:54 (1273): No heartbeat from core client for 30 sec - exiting 14:36:55 (1273): No heartbeat from core client for 30 sec - exiting 14:36:56 (1273): No heartbeat from core client for 30 sec - exiting 14:36:57 (1273): No heartbeat from core client for 30 sec - exiting 14:36:58 (1273): No heartbeat from core client for 30 sec - exiting 14:36:59 (1273): No heartbeat from core client for 30 sec - exiting 14:37:00 (1273): No heartbeat from core client for 30 sec - exiting 14:37:01 (1273): No heartbeat from core client for 30 sec - exiting 14:37:02 (1273): No heartbeat from core client for 30 sec - exiting 14:37:03 (1273): No heartbeat from core client for 30 sec - exiting 14:37:04 (1273): No heartbeat from core client for 30 sec - exiting 14:37:05 (1273): No heartbeat from core client for 30 sec - exiting 14:37:06 (1273): No heartbeat from core client for 30 sec - exiting 14:37:07 (1273): No heartbeat from core client for 30 sec - exiting 14:37:08 (1273): No heartbeat from core client for 30 sec - exiting 14:37:09 (1273): No heartbeat from core client for 30 sec - exiting 14:37:10 (1273): No heartbeat from core client for 30 sec - exiting 14:37:11 (1273): No heartbeat from core client for 30 sec - exiting 14:37:12 (1273): No heartbeat from core client for 30 sec - exiting 14:37:13 (1273): No heartbeat from core client for 30 sec - exiting 14:37:14 (1273): No heartbeat from core client for 30 sec - exiting 14:37:15 (1273): No heartbeat from core client for 30 sec - exiting 14:37:16 (1273): No heartbeat from core client for 30 sec - exiting 14:37:17 (1273): No heartbeat from core client for 30 sec - exiting 14:37:18 (1273): No heartbeat from core client for 30 sec - exiting 14:37:19 (1273): No heartbeat from core client for 30 sec - exiting 14:37:20 (1273): No heartbeat from core client for 30 sec - exiting 14:37:21 (1273): No heartbeat from core client for 30 sec - exiting 14:37:22 (1273): No heartbeat from core client for 30 sec - exiting 14:37:23 (1273): No heartbeat from core client for 30 sec - exiting 14:37:24 (1273): No heartbeat from core client for 30 sec - exiting 14:37:25 (1273): No heartbeat from core client for 30 sec - exiting 14:37:26 (1273): No heartbeat from core client for 30 sec - exiting 14:37:27 (1273): No heartbeat from core client for 30 sec - exiting 14:37:28 (1273): No heartbeat from core client for 30 sec - exiting 14:37:29 (1273): No heartbeat from core client for 30 sec - exiting 14:37:30 (1273): No heartbeat from core client for 30 sec - exiting 14:37:31 (1273): No heartbeat from core client for 30 sec - exiting 14:37:32 (1273): No heartbeat from core client for 30 sec - exiting 14:37:33 (1273): No heartbeat from core client for 30 sec - exiting 14:37:34 (1273): No heartbeat from core client for 30 sec - exiting 14:37:35 (1273): No heartbeat from core client for 30 sec - exiting 14:37:36 (1273): No heartbeat from core client for 30 sec - exiting 14:37:37 (1273): No heartbeat from core client for 30 sec - exiting 14:37:38 (1273): No heartbeat from core client for 30 sec - exiting 14:37:39 (1273): No heartbeat from core client for 30 sec - exiting 14:37:40 (1273): No heartbeat from core client for 30 sec - exiting 14:37:41 (1273): No heartbeat from core client for 30 sec - exiting 14:37:43 (1273): No heartbeat from core client for 30 sec - exiting 14:37:44 (1273): No heartbeat from core client for 30 sec - exiting 14:37:45 (1273): No heartbeat from core client for 30 sec - exiting 14:37:46 (1273): No heartbeat from core client for 30 sec - exiting 14:37:47 (1273): No heartbeat from core client for 30 sec - exiting 14:37:48 (1273): No heartbeat from core client for 30 sec - exiting 14:37:49 (1273): No heartbeat from core client for 30 sec - exiting 14:37:50 (1273): No heartbeat from core client for 30 sec - exiting 14:37:51 (1273): No heartbeat from core client for 30 sec - exiting 14:37:52 (1273): No heartbeat from core client for 30 sec - exiting 14:37:53 (1273): No heartbeat from core client for 30 sec - exiting 14:37:54 (1273): No heartbeat from core client for 30 sec - exiting 14:37:55 (1273): No heartbeat from core client for 30 sec - exiting 14:37:56 (1273): No heartbeat from core client for 30 sec - exiting 14:37:57 (1273): No heartbeat from core client for 30 sec - exiting 14:37:58 (1273): No heartbeat from core client for 30 sec - exiting 14:38:02 (1273): No heartbeat from core client for 30 sec - exiting 14:38:03 (1273): No heartbeat from core client for 30 sec - exiting 14:38:04 (1273): No heartbeat from core client for 30 sec - exiting 14:38:05 (1273): No heartbeat from core client for 30 sec - exiting 14:38:06 (1273): No heartbeat from core client for 30 sec - exiting 14:38:07 (1273): No heartbeat from core client for 30 sec - exiting 14:38:08 (1273): No heartbeat from core client for 30 sec - exiting 14:38:09 (1273): No heartbeat from core client for 30 sec - exiting 14:38:10 (1273): No heartbeat from core client for 30 sec - exiting 14:38:11 (1273): No heartbeat from core client for 30 sec - exiting 14:38:12 (1273): No heartbeat from core client for 30 sec - exiting 14:38:13 (1273): No heartbeat from core client for 30 sec - exiting 14:38:14 (1273): No heartbeat from core client for 30 sec - exiting 14:38:15 (1273): No heartbeat from core client for 30 sec - exiting 14:38:16 (1273): No heartbeat from core client for 30 sec - exiting 14:38:17 (1273): No heartbeat from core client for 30 sec - exiting 14:38:18 (1273): No heartbeat from core client for 30 sec - exiting 14:38:19 (1273): No heartbeat from core client for 30 sec - exiting 14:38:20 (1273): No heartbeat from core client for 30 sec - exiting 14:38:21 (1273): No heartbeat from core client for 30 sec - exiting 14:38:22 (1273): No heartbeat from core client for 30 sec - exiting 14:38:23 (1273): No heartbeat from core client for 30 sec - exiting 14:38:24 (1273): No heartbeat from core client for 30 sec - exiting 14:38:25 (1273): No heartbeat from core client for 30 sec - exiting 14:38:26 (1273): No heartbeat from core client for 30 sec - exiting 14:38:27 (1273): No heartbeat from core client for 30 sec - exiting 14:38:28 (1273): No heartbeat from core client for 30 sec - exiting 14:38:29 (1273): No heartbeat from core client for 30 sec - exiting 14:38:30 (1273): No heartbeat from core client for 30 sec - exiting 14:38:31 (1273): No heartbeat from core client for 30 sec - exiting 14:38:32 (1273): No heartbeat from core client for 30 sec - exiting 14:38:33 (1273): No heartbeat from core client for 30 sec - exiting 14:38:34 (1273): No heartbeat from core client for 30 sec - exiting 14:38:35 (1273): No heartbeat from core client for 30 sec - exiting 14:38:36 (1273): No heartbeat from core client for 30 sec - exiting 14:38:37 (1273): No heartbeat from core client for 30 sec - exiting 14:38:38 (1273): No heartbeat from core client for 30 sec - exiting 14:38:39 (1273): No heartbeat from core client for 30 sec - exiting 14:38:40 (1273): No heartbeat from core client for 30 sec - exiting 14:38:41 (1273): No heartbeat from core client for 30 sec - exiting 14:38:42 (1273): No heartbeat from core client for 30 sec - exiting 14:38:43 (1273): No heartbeat from core client for 30 sec - exiting 14:38:44 (1273): No heartbeat from core client for 30 sec - exiting 14:38:45 (1273): No heartbeat from core client for 30 sec - exiting 14:38:46 (1273): No heartbeat from core client for 30 sec - exiting 14:38:47 (1273): No heartbeat from core client for 30 sec - exiting 14:38:48 (1273): No heartbeat from core client for 30 sec - exiting 14:38:49 (1273): No heartbeat from core client for 30 sec - exiting 14:38:50 (1273): No heartbeat from core client for 30 sec - exiting 14:38:51 (1273): No heartbeat from core client for 30 sec - exiting 14:38:52 (1273): No heartbeat from core client for 30 sec - exiting 14:38:53 (1273): No heartbeat from core client for 30 sec - exiting 14:38:54 (1273): No heartbeat from core client for 30 sec - exiting 14:38:55 (1273): No heartbeat from core client for 30 sec - exiting 14:38:56 (1273): No heartbeat from core client for 30 sec - exiting 14:38:57 (1273): No heartbeat from core client for 30 sec - exiting 14:38:58 (1273): No heartbeat from core client for 30 sec - exiting 14:38:59 (1273): No heartbeat from core client for 30 sec - exiting 14:39:00 (1273): No heartbeat from core client for 30 sec - exiting 14:39:01 (1273): No heartbeat from core client for 30 sec - exiting 14:39:02 (1273): No heartbeat from core client for 30 sec - exiting 14:39:03 (1273): No heartbeat from core client for 30 sec - exiting 14:39:04 (1273): No heartbeat from core client for 30 sec - exiting 14:39:05 (1273): No heartbeat from core client for 30 sec - exiting 14:39:06 (1273): No heartbeat from core client for 30 sec - exiting 14:39:07 (1273): No heartbeat from core client for 30 sec - exiting 14:39:08 (1273): No heartbeat from core client for 30 sec - exiting 14:39:09 (1273): No heartbeat from core client for 30 sec - exiting 14:39:10 (1273): No heartbeat from core client for 30 sec - exiting 14:39:11 (1273): No heartbeat from core client for 30 sec - exiting 14:39:12 (1273): No heartbeat from core client for 30 sec - exiting 14:39:13 (1273): No heartbeat from core client for 30 sec - exiting 14:39:14 (1273): No heartbeat from core client for 30 sec - exiting 14:39:15 (1273): No heartbeat from core client for 30 sec - exiting 14:39:16 (1273): No heartbeat from core client for 30 sec - exiting 14:39:20 (1273): No heartbeat from core client for 30 sec - exiting 14:39:21 (1273): No heartbeat from core client for 30 sec - exiting 14:39:22 (1273): No heartbeat from core client for 30 sec - exiting 14:39:37 (1273): No heartbeat from core client for 30 sec - exiting 14:39:38 (1273): No heartbeat from core client for 30 sec - exiting 14:39:39 (1273): No heartbeat from core client for 30 sec - exiting 14:39:40 (1273): No heartbeat from core client for 30 sec - exiting 14:39:41 (1273): No heartbeat from core client for 30 sec - exiting 14:39:42 (1273): No heartbeat from core client for 30 sec - exiting 14:39:43 (1273): No heartbeat from core client for 30 sec - exiting 14:39:44 (1273): No heartbeat from core client for 30 sec - exiting 14:39:45 (1273): No heartbeat from core client for 30 sec - exiting 14:39:46 (1273): No heartbeat from core client for 30 sec - exiting 14:39:47 (1273): No heartbeat from core client for 30 sec - exiting 14:39:48 (1273): No heartbeat from core client for 30 sec - exiting 14:39:49 (1273): No heartbeat from core client for 30 sec - exiting 14:39:50 (1273): No heartbeat from core client for 30 sec - exiting 14:39:51 (1273): No heartbeat from core client for 30 sec - exiting 14:39:52 (1273): No heartbeat from core client for 30 sec - exiting 14:39:53 (1273): No heartbeat from core client for 30 sec - exiting 14:39:54 (1273): No heartbeat from core client for 30 sec - exiting 14:39:55 (1273): No heartbeat from core client for 30 sec - exiting 14:39:56 (1273): No heartbeat from core client for 30 sec - exiting 14:39:57 (1273): No heartbeat from core client for 30 sec - exiting 14:39:58 (1273): No heartbeat from core client for 30 sec - exiting 14:39:59 (1273): No heartbeat from core client for 30 sec - exiting 14:40:00 (1273): No heartbeat from core client for 30 sec - exiting 14:40:01 (1273): No heartbeat from core client for 30 sec - exiting 14:40:02 (1273): No heartbeat from core client for 30 sec - exiting 14:40:03 (1273): No heartbeat from core client for 30 sec - exiting 14:40:04 (1273): No heartbeat from core client for 30 sec - exiting 14:40:05 (1273): No heartbeat from core client for 30 sec - exiting 14:40:06 (1273): No heartbeat from core client for 30 sec - exiting 14:40:07 (1273): No heartbeat from core client for 30 sec - exiting 14:40:08 (1273): No heartbeat from core client for 30 sec - exiting 14:40:09 (1273): No heartbeat from core client for 30 sec - exiting 14:40:20 (1273): No heartbeat from core client for 30 sec - exiting 14:40:21 (1273): No heartbeat from core client for 30 sec - exiting 14:40:22 (1273): No heartbeat from core client for 30 sec - exiting 14:40:23 (1273): No heartbeat from core client for 30 sec - exiting 14:40:24 (1273): No heartbeat from core client for 30 sec - exiting 14:40:25 (1273): No heartbeat from core client for 30 sec - exiting 14:40:26 (1273): No heartbeat from core client for 30 sec - exiting 14:40:27 (1273): No heartbeat from core client for 30 sec - exiting 14:40:28 (1273): No heartbeat from core client for 30 sec - exiting 14:40:29 (1273): No heartbeat from core client for 30 sec - exiting 14:40:30 (1273): No heartbeat from core client for 30 sec - exiting 14:40:31 (1273): No heartbeat from core client for 30 sec - exiting 14:40:32 (1273): No heartbeat from core client for 30 sec - exiting 14:40:33 (1273): No heartbeat from core client for 30 sec - exiting 14:40:34 (1273): No heartbeat from core client for 30 sec - exiting 14:40:35 (1273): No heartbeat from core client for 30 sec - exiting 14:40:36 (1273): No heartbeat from core client for 30 sec - exiting 14:40:37 (1273): No heartbeat from core client for 30 sec - exiting 14:40:38 (1273): No heartbeat from core client for 30 sec - exiting 14:40:39 (1273): No heartbeat from core client for 30 sec - exiting 14:40:40 (1273): No heartbeat from core client for 30 sec - exiting 14:40:41 (1273): No heartbeat from core client for 30 sec - exiting 14:40:42 (1273): No heartbeat from core client for 30 sec - exiting 14:40:43 (1273): No heartbeat from core client for 30 sec - exiting 14:40:47 (1273): No heartbeat from core client for 30 sec - exiting 14:40:48 (1273): No heartbeat from core client for 30 sec - exiting 14:40:49 (1273): No heartbeat from core client for 30 sec - exiting 14:40:50 (1273): No heartbeat from core client for 30 sec - exiting 14:40:51 (1273): No heartbeat from core client for 30 sec - exiting 14:40:52 (1273): No heartbeat from core client for 30 sec - exiting 14:40:53 (1273): No heartbeat from core client for 30 sec - exiting 14:40:54 (1273): No heartbeat from core client for 30 sec - exiting 14:40:55 (1273): No heartbeat from core client for 30 sec - exiting 14:40:56 (1273): No heartbeat from core client for 30 sec - exiting 14:40:57 (1273): No heartbeat from core client for 30 sec - exiting 14:40:58 (1273): No heartbeat from core client for 30 sec - exiting 14:40:59 (1273): No heartbeat from core client for 30 sec - exiting 14:41:00 (1273): No heartbeat from core client for 30 sec - exiting 14:41:01 (1273): No heartbeat from core client for 30 sec - exiting 14:41:02 (1273): No heartbeat from core client for 30 sec - exiting 14:41:03 (1273): No heartbeat from core client for 30 sec - exiting 14:41:04 (1273): No heartbeat from core client for 30 sec - exiting 14:41:05 (1273): No heartbeat from core client for 30 sec - exiting 14:41:06 (1273): No heartbeat from core client for 30 sec - exiting 14:41:07 (1273): No heartbeat from core client for 30 sec - exiting 14:41:08 (1273): No heartbeat from core client for 30 sec - exiting 14:41:15 (1273): No heartbeat from core client for 30 sec - exiting 14:41:26 (1273): No heartbeat from core client for 30 sec - exiting 14:41:27 (1273): No heartbeat from core client for 30 sec - exiting 14:41:28 (1273): No heartbeat from core client for 30 sec - exiting 14:41:29 (1273): No heartbeat from core client for 30 sec - exiting 14:41:30 (1273): No heartbeat from core client for 30 sec - exiting 14:41:34 (1273): No heartbeat from core client for 30 sec - exiting 14:41:35 (1273): No heartbeat from core client for 30 sec - exiting 14:41:36 (1273): No heartbeat from core client for 30 sec - exiting 14:41:37 (1273): No heartbeat from core client for 30 sec - exiting 14:41:38 (1273): No heartbeat from core client for 30 sec - exiting 14:41:39 (1273): No heartbeat from core client for 30 sec - exiting 14:41:40 (1273): No heartbeat from core client for 30 sec - exiting 14:41:41 (1273): No heartbeat from core client for 30 sec - exiting 14:41:42 (1273): No heartbeat from core client for 30 sec - exiting 14:41:43 (1273): No heartbeat from core client for 30 sec - exiting 14:41:44 (1273): No heartbeat from core client for 30 sec - exiting 14:41:45 (1273): No heartbeat from core client for 30 sec - exiting 14:41:46 (1273): No heartbeat from core client for 30 sec - exiting 14:41:47 (1273): No heartbeat from core client for 30 sec - exiting 14:41:48 (1273): No heartbeat from core client for 30 sec - exiting 14:41:49 (1273): No heartbeat from core client for 30 sec - exiting 14:41:50 (1273): No heartbeat from core client for 30 sec - exiting 14:41:51 (1273): No heartbeat from core client for 30 sec - exiting 14:41:52 (1273): No heartbeat from core client for 30 sec - exiting 14:41:54 (1273): No heartbeat from core client for 30 sec - exiting 14:41:55 (1273): No heartbeat from core client for 30 sec - exiting 14:41:56 (1273): No heartbeat from core client for 30 sec - exiting 14:41:57 (1273): No heartbeat from core client for 30 sec - exiting 14:41:58 (1273): No heartbeat from core client for 30 sec - exiting 14:41:59 (1273): No heartbeat from core client for 30 sec - exiting 14:42:00 (1273): No heartbeat from core client for 30 sec - exiting 14:42:01 (1273): No heartbeat from core client for 30 sec - exiting 14:42:02 (1273): No heartbeat from core client for 30 sec - exiting 14:42:03 (1273): No heartbeat from core client for 30 sec - exiting 14:42:04 (1273): No heartbeat from core client for 30 sec - exiting 14:42:05 (1273): No heartbeat from core client for 30 sec - exiting 14:42:06 (1273): No heartbeat from core client for 30 sec - exiting 14:42:07 (1273): No heartbeat from core client for 30 sec - exiting 14:42:08 (1273): No heartbeat from core client for 30 sec - exiting 14:42:09 (1273): No heartbeat from core client for 30 sec - exiting 14:42:10 (1273): No heartbeat from core client for 30 sec - exiting 14:42:11 (1273): No heartbeat from core client for 30 sec - exiting 14:42:12 (1273): No heartbeat from core client for 30 sec - exiting 14:42:13 (1273): No heartbeat from core client for 30 sec - exiting 14:42:14 (1273): No heartbeat from core client for 30 sec - exiting 14:42:15 (1273): No heartbeat from core client for 30 sec - exiting 14:42:16 (1273): No heartbeat from core client for 30 sec - exiting 14:42:17 (1273): No heartbeat from core client for 30 sec - exiting 14:42:18 (1273): No heartbeat from core client for 30 sec - exiting 14:42:49 (1273): No heartbeat from core client for 30 sec - exiting 14:42:50 (1273): No heartbeat from core client for 30 sec - exiting 14:42:51 (1273): No heartbeat from core client for 30 sec - exiting 14:42:52 (1273): No heartbeat from core client for 30 sec - exiting 14:42:53 (1273): No heartbeat from core client for 30 sec - exiting 14:42:54 (1273): No heartbeat from core client for 30 sec - exiting 14:42:55 (1273): No heartbeat from core client for 30 sec - exiting 14:42:56 (1273): No heartbeat from core client for 30 sec - exiting 14:42:57 (1273): No heartbeat from core client for 30 sec - exiting 12:27:23 (1273): No heartbeat from core client for 30 sec - exiting 12:31:21 (1273): No heartbeat from core client for 30 sec - exiting 12:31:22 (1273): No heartbeat from core client for 30 sec - exiting 12:31:23 (1273): No heartbeat from core client for 30 sec - exiting 12:31:24 (1273): No heartbeat from core client for 30 sec - exiting 12:31:25 (1273): No heartbeat from core client for 30 sec - exiting 12:31:26 (1273): No heartbeat from core client for 30 sec - exiting 12:31:27 (1273): No heartbeat from core client for 30 sec - exiting 12:31:28 (1273): No heartbeat from core client for 30 sec - exiting 12:31:29 (1273): No heartbeat from core client for 30 sec - exiting 12:31:30 (1273): No heartbeat from core client for 30 sec - exiting 12:31:31 (1273): No heartbeat from core client for 30 sec - exiting 12:31:32 (1273): No heartbeat from core client for 30 sec - exiting 12:31:33 (1273): No heartbeat from core client for 30 sec - exiting 12:31:34 (1273): No heartbeat from core client for 30 sec - exiting 12:31:35 (1273): No heartbeat from core client for 30 sec - exiting 12:31:36 (1273): No heartbeat from core client for 30 sec - exiting 12:31:37 (1273): No heartbeat from core client for 30 sec - exiting 12:31:38 (1273): No heartbeat from core client for 30 sec - exiting 12:31:39 (1273): No heartbeat from core client for 30 sec - exiting 12:31:40 (1273): No heartbeat from core client for 30 sec - exiting 12:31:41 (1273): No heartbeat from core client for 30 sec - exiting 12:31:42 (1273): No heartbeat from core client for 30 sec - exiting 12:31:43 (1273): No heartbeat from core client for 30 sec - exiting 12:31:44 (1273): No heartbeat from core client for 30 sec - exiting 12:31:45 (1273): No heartbeat from core client for 30 sec - exiting 12:31:46 (1273): No heartbeat from core client for 30 sec - exiting 12:31:47 (1273): No heartbeat from core client for 30 sec - exiting 12:31:48 (1273): No heartbeat from core client for 30 sec - exiting 12:31:49 (1273): No heartbeat from core client for 30 sec - exiting 12:31:50 (1273): No heartbeat from core client for 30 sec - exiting 12:31:51 (1273): No heartbeat from core client for 30 sec - exiting 12:31:52 (1273): No heartbeat from core client for 30 sec - exiting 12:31:53 (1273): No heartbeat from core client for 30 sec - exiting 12:31:54 (1273): No heartbeat from core client for 30 sec - exiting 12:31:55 (1273): No heartbeat from core client for 30 sec - exiting 12:31:56 (1273): No heartbeat from core client for 30 sec - exiting 12:31:57 (1273): No heartbeat from core client for 30 sec - exiting 12:31:58 (1273): No heartbeat from core client for 30 sec - exiting 12:31:59 (1273): No heartbeat from core client for 30 sec - exiting 12:32:00 (1273): No heartbeat from core client for 30 sec - exiting 12:32:02 (1273): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x083d2318 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09fd2318 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 00:26:18 (1183): No heartbeat from core client for 30 sec - exiting 22:57:05 (1183): No heartbeat from core client for 30 sec - exiting 22:57:06 (1183): No heartbeat from core client for 30 sec - exiting 22:57:07 (1183): No heartbeat from core client for 30 sec - exiting 22:57:08 (1183): No heartbeat from core client for 30 sec - exiting 22:57:09 (1183): No heartbeat from core client for 30 sec - exiting 22:57:10 (1183): No heartbeat from core client for 30 sec - exiting 22:57:11 (1183): No heartbeat from core client for 30 sec - exiting 22:57:12 (1183): No heartbeat from core client for 30 sec - exiting 22:57:13 (1183): No heartbeat from core client for 30 sec - exiting 22:57:14 (1183): No heartbeat from core client for 30 sec - exiting 22:57:15 (1183): No heartbeat from core client for 30 sec - exiting 22:58:16 (1183): No heartbeat from core client for 30 sec - exiting 22:58:17 (1183): No heartbeat from core client for 30 sec - exiting 22:58:18 (1183): No heartbeat from core client for 30 sec - exiting 22:58:19 (1183): No heartbeat from core client for 30 sec - exiting 22:58:20 (1183): No heartbeat from core client for 30 sec - exiting 22:58:21 (1183): No heartbeat from core client for 30 sec - exiting 22:58:22 (1183): No heartbeat from core client for 30 sec - exiting 22:58:23 (1183): No heartbeat from core client for 30 sec - exiting 22:58:24 (1183): No heartbeat from core client for 30 sec - exiting 22:58:25 (1183): No heartbeat from core client for 30 sec - exiting 22:58:26 (1183): No heartbeat from core client for 30 sec - exiting 03:45:07 (1183): No heartbeat from core client for 30 sec - exiting 03:45:52 (1183): No heartbeat from core client for 30 sec - exiting 03:45:53 (1183): No heartbeat from core client for 30 sec - exiting 03:45:54 (1183): No heartbeat from core client for 30 sec - exiting 03:45:55 (1183): No heartbeat from core client for 30 sec - exiting 03:45:56 (1183): No heartbeat from core client for 30 sec - exiting 03:45:57 (1183): No heartbeat from core client for 30 sec - exiting 03:45:58 (1183): No heartbeat from core client for 30 sec - exiting 03:45:59 (1183): No heartbeat from core client for 30 sec - exiting 03:46:00 (1183): No heartbeat from core client for 30 sec - exiting 03:46:01 (1183): No heartbeat from core client for 30 sec - exiting 03:46:02 (1183): No heartbeat from core client for 30 sec - exiting 03:46:03 (1183): No heartbeat from core client for 30 sec - exiting 03:46:04 (1183): No heartbeat from core client for 30 sec - exiting 03:46:05 (1183): No heartbeat from core client for 30 sec - exiting 03:46:06 (1183): No heartbeat from core client for 30 sec - exiting 03:46:07 (1183): No heartbeat from core client for 30 sec - exiting 03:46:08 (1183): No heartbeat from core client for 30 sec - exiting 03:46:17 (1183): No heartbeat from core client for 30 sec - exiting 03:46:40 (1183): No heartbeat from core client for 30 sec - exiting 03:46:53 (1183): No heartbeat from core client for 30 sec - exiting 03:46:54 (1183): No heartbeat from core client for 30 sec - exiting 03:46:55 (1183): No heartbeat from core client for 30 sec - exiting 03:46:56 (1183): No heartbeat from core client for 30 sec - exiting 03:46:57 (1183): No heartbeat from core client for 30 sec - exiting 03:46:58 (1183): No heartbeat from core client for 30 sec - exiting 03:46:59 (1183): No heartbeat from core client for 30 sec - exiting 03:47:02 (1183): No heartbeat from core client for 30 sec - exiting 03:47:03 (1183): No heartbeat from core client for 30 sec - exiting 03:47:04 (1183): No heartbeat from core client for 30 sec - exiting 03:47:05 (1183): No heartbeat from core client for 30 sec - exiting 03:47:06 (1183): No heartbeat from core client for 30 sec - exiting 03:47:07 (1183): No heartbeat from core client for 30 sec - exiting 03:47:08 (1183): No heartbeat from core client for 30 sec - exiting 03:47:09 (1183): No heartbeat from core client for 30 sec - exiting 03:47:10 (1183): No heartbeat from core client for 30 sec - exiting 03:47:11 (1183): No heartbeat from core client for 30 sec - exiting 03:47:12 (1183): No heartbeat from core client for 30 sec - exiting 03:47:13 (1183): No heartbeat from core client for 30 sec - exiting 03:47:14 (1183): No heartbeat from core client for 30 sec - exiting 03:47:15 (1183): No heartbeat from core client for 30 sec - exiting 03:47:16 (1183): No heartbeat from core client for 30 sec - exiting 03:47:17 (1183): No heartbeat from core client for 30 sec - exiting 03:47:18 (1183): No heartbeat from core client for 30 sec - exiting 03:47:19 (1183): No heartbeat from core client for 30 sec - exiting 03:47:20 (1183): No heartbeat from core client for 30 sec - exiting 03:47:21 (1183): No heartbeat from core client for 30 sec - exiting 03:47:22 (1183): No heartbeat from core client for 30 sec - exiting 03:47:23 (1183): No heartbeat from core client for 30 sec - exiting 03:47:43 (1183): No heartbeat from core client for 30 sec - exiting 03:47:44 (1183): No heartbeat from core client for 30 sec - exiting 03:47:45 (1183): No heartbeat from core client for 30 sec - exiting 03:47:46 (1183): No heartbeat from core client for 30 sec - exiting 03:47:47 (1183): No heartbeat from core client for 30 sec - exiting 03:47:48 (1183): No heartbeat from core client for 30 sec - exiting 03:47:49 (1183): No heartbeat from core client for 30 sec - exiting 03:47:50 (1183): No heartbeat from core client for 30 sec - exiting 03:47:51 (1183): No heartbeat from core client for 30 sec - exiting 03:47:52 (1183): No heartbeat from core client for 30 sec - exiting 03:47:53 (1183): No heartbeat from core client for 30 sec - exiting 03:47:54 (1183): No heartbeat from core client for 30 sec - exiting 03:47:55 (1183): No heartbeat from core client for 30 sec - exiting 03:47:56 (1183): No heartbeat from core client for 30 sec - exiting 03:47:57 (1183): No heartbeat from core client for 30 sec - exiting 03:47:58 (1183): No heartbeat from core client for 30 sec - exiting 03:47:59 (1183): No heartbeat from core client for 30 sec - exiting 07:17:03 (1183): No heartbeat from core client for 30 sec - exiting 07:42:27 (1183): No heartbeat from core client for 30 sec - exiting 08:33:36 (1183): No heartbeat from core client for 30 sec - exiting 08:33:37 (1183): No heartbeat from core client for 30 sec - exiting 08:33:38 (1183): No heartbeat from core client for 30 sec - exiting 08:33:39 (1183): No heartbeat from core client for 30 sec - exiting 08:33:40 (1183): No heartbeat from core client for 30 sec - exiting 08:33:41 (1183): No heartbeat from core client for 30 sec - exiting 08:33:42 (1183): No heartbeat from core client for 30 sec - exiting 08:33:43 (1183): No heartbeat from core client for 30 sec - exiting 08:33:44 (1183): No heartbeat from core client for 30 sec - exiting 08:33:45 (1183): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09ff6318 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called execv: No such file or directory </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
07 Sep 2012 15:01:11 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 492,480 | 2,478,896 | 5.0335 |
24 Aug 2012 10:26:35 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 259,200 | 1,291,170 | 4.9814 |
22 Aug 2012 07:31:04 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 233,280 | 1,152,457 | 4.9402 |
20 Aug 2012 10:44:51 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 207,360 | 1,016,525 | 4.9022 |
18 Aug 2012 20:51:35 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 181,440 | 882,506 | 4.8639 |
17 Aug 2012 01:53:35 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 155,520 | 752,044 | 4.8357 |
15 Aug 2012 05:04:17 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 129,600 | 622,146 | 4.8005 |
13 Aug 2012 16:11:49 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 103,680 | 494,378 | 4.7683 |
11 Aug 2012 21:02:35 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 77,760 | 365,880 | 4.7052 |
10 Aug 2012 05:01:09 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 51,840 | 235,774 | 4.5481 |
08 Aug 2012 07:56:24 | 1179185 | 15071465 | hadcm3n_006o_1980_40_008000673_2 | 25,920 | 107,279 | 4.1389 |
©2024 cpdn.org