Task 15760114

Name	hadcm3n_zbkv_1920_40_008281865_3
Workunit	8433000
Created	2 May 2013, 19:41:31 UTC
Sent	2 May 2013, 19:41:42 UTC
Report deadline	2 Aug 2013, 3:08:53 UTC
Received	29 May 2013, 20:59:06 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1279475
Run time	17 days 11 hours 27 min 33 sec
CPU time	15 days 22 hours 5 min 17 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	1.69 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.45</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_zbkv_1920_40_008281865/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D UnknowController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7181, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_zbkv_1920_40_008281865/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 00538CE6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7181, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_zbkv_1920_40_008281865/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 00538CE6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7181, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_zbkv_1920_40_008281865/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 00538CE6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7181, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_zbkv_1920_40_008281865/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 00538CE6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7181, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:57:26 (11832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:17 (13409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:59:40 (13420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:28 (13431): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:31:58 (13576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:32:44 (13971): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:36:14 (13991): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:37:03 (14006): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:37:45 (14021): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:39:56 (14039): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:40:44 (14055): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 19:47:03 (14068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:56 (24606): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:49:39 (24742): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:17 (24899): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:02 (25851): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:32 (26142): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:02:20 (26238): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:03 (26321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:50 (26404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:04:38 (26487): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:21 (26580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:06:33 (26664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:19:07 (26858): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:20:20 (28176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:21:03 (28306): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:23:26 (28456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:49 (28707): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:27:32 (28952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:29:57 (29134): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:30:42 (29424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:06:48 (29507): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:08:37 (918): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:09:34 (1110): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:22 (1212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:20 (1306): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:12:08 (1409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:02 (1502): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:50 (1606): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:30 (1692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:22:18 (2616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:50 (2779): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:02 (3847): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:55 (3994): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:27 (4100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:47:58 (5143): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:50:21 (5881): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:52:44 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:13:19 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:32 (17761): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:15:15 (17774): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:58 (17897): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:42 (17912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:20:25 (17925): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:35:29 (17941): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:13 (18117): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:55 (18130): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:59 (18142): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:43 (18155): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:26 (18170): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:42:29 (18180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:21:52 (25102): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:23:36 (26344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:18 (26357): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:25:21 (26368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:07 (26381): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:48 (26395): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:28:51 (26407): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:30:47 (26424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:31:29 (26553): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:35 (26564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:34:43 (26576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:36:56 (26592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:06 (26611): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:53 (26626): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:43:00 (26642): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:46:04 (26661): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:46:46 (26793): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:04 (26806): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:50:48 (26819): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:52:46 (26842): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:44 (26858): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:57:32 (27176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:59:46 (27326): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:02:14 (27344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:05:12 (27481): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:07:15 (27498): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:17 (27514): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:09:06 (27527): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:10:43 (27538): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:11:30 (27555): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:13:13 (27567): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:14:00 (27585): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:14:58 (27596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:16:22 (27719): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:17:10 (27730): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:34 (27741): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:21 (27752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:20:09 (27761): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:20:57 (27773): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:21:45 (27783): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:22:33 (27794): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:21 (27803): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:24:09 (27814): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:24:52 (27825): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:23 (27835): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:27:12 (27844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:28:00 (27856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:28:58 (27866): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:29:47 (27875): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:30:35 (27885): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:31:23 (28012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:32:21 (28022): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:33:10 (28034): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:33:58 (28045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:46 (28055): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:35:33 (28066): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:23 (28076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:24 (28076): No heartbeat from core client for 30 sec - exiting 11:36:25 (28076): No heartbeat from core client for 30 sec - exiting 11:37:10 (28086): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:04 (28102): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:51 (28113): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:39:40 (28123): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:40:38 (28134): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:41:27 (28148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:14 (28163): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:43:03 (28175): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:00 (28186): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:49 (28198): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:37 (28209): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:46:26 (28333): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:46:27 (28333): No heartbeat from core client for 30 sec - exiting 11:47:13 (28344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:48:02 (28356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:48:50 (28367): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:49:37 (28377): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:26 (28388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:14 (28403): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:52:03 (28413): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:01 (28425): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:49 (28438): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:54:38 (28448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:55:36 (28459): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:56:23 (28468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:57:10 (28478): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:57:58 (28487): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:58:45 (28495): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:59:33 (28502): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:00:20 (28510): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:01:08 (28634): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:01:56 (28645): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:43 (28653): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:31 (28661): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:04:18 (28669): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:05:06 (28677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:05:53 (28686): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:06:51 (28696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:08:14 (28707): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:09:02 (28719): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:09:49 (28728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:10:37 (28737): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:25 (28751): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:12:12 (28763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:13:00 (28774): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 May 2013 18:24:12	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	673,920	1,227,023	1.8207
22 May 2013 23:59:58	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	648,000	1,182,096	1.8242
22 May 2013 07:17:44	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	622,080	1,136,302	1.8266
21 May 2013 09:58:29	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	596,160	1,090,022	1.8284
20 May 2013 20:42:28	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	570,240	1,044,243	1.8312
20 May 2013 07:15:24	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	544,320	998,890	1.8351
19 May 2013 16:40:11	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	518,400	953,457	1.8392
19 May 2013 03:15:13	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	492,480	907,523	1.8428
18 May 2013 11:29:40	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	466,560	862,138	1.8479
17 May 2013 18:10:45	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	440,640	816,171	1.8522
16 May 2013 02:14:52	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	414,720	770,056	1.8568
14 May 2013 06:27:54	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	388,800	711,264	1.8294
13 May 2013 01:59:57	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	362,880	658,552	1.8148
12 May 2013 11:47:02	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	336,960	612,826	1.8187
11 May 2013 15:45:49	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	311,040	566,083	1.8200
10 May 2013 19:43:25	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	285,120	518,078	1.8171
09 May 2013 20:36:10	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	259,200	471,356	1.8185
09 May 2013 01:23:58	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	233,280	425,686	1.8248
08 May 2013 07:48:55	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	207,360	378,478	1.8252
07 May 2013 16:21:12	1279475	15760114	hadcm3n_zbkv_1920_40_008281865_3	181,440	332,661	1.8334