Task 12619994

Name	hadam3p_pnw_30cw_1984_1_007180616_0
Workunit	7378898
Created	22 Feb 2011, 12:14:40 UTC
Sent	8 Mar 2011, 11:39:14 UTC
Report deadline	18 Feb 2012, 16:59:14 UTC
Received	12 Mar 2011, 19:51:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID	570789
Run time	4 days 1 hours 49 min 44 sec
CPU time	3 days 15 hours 56 min 56 sec
Validate state	Invalid
Credit	1,754.30
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Pacific North West v6.08 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> Got ack for job that's till active </message> <stderr_txt> 08:12:35 (5560): No heartbeat from core client for 30 sec - exiting 08:12:36 (5560): No heartbeat from core client for 30 sec - exiting 08:12:37 (5560): No heartbeat from core client for 30 sec - exiting 08:12:38 (5560): No heartbeat from core client for 30 sec - exiting 08:12:39 (5560): No heartbeat from core client for 30 sec - exiting 08:12:40 (5560): No heartbeat from core client for 30 sec - exiting 08:12:41 (5560): No heartbeat from core client for 30 sec - exiting 08:12:42 (5560): No heartbeat from core client for 30 sec - exiting 08:12:43 (5560): No heartbeat from core client for 30 sec - exiting 08:12:44 (5560): No heartbeat from core client for 30 sec - exiting 08:12:45 (5560): No heartbeat from core client for 30 sec - exiting 08:12:50 (5560): No heartbeat from core client for 30 sec - exiting 08:12:51 (5560): No heartbeat from core client for 30 sec - exiting 08:12:52 (5560): No heartbeat from core client for 30 sec - exiting 08:12:53 (5560): No heartbeat from core client for 30 sec - exiting 08:12:54 (5560): No heartbeat from core client for 30 sec - exiting 08:12:55 (5560): No heartbeat from core client for 30 sec - exiting 08:12:56 (5560): No heartbeat from core client for 30 sec - exiting 08:12:57 (5560): No heartbeat from core client for 30 sec - exiting 08:12:58 (5560): No heartbeat from core client for 30 sec - exiting Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=2596, iMonCtr=1 08:12:59 (5560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:43 (5652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:15:11 (8020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:06 (4052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:45 (4052): No heartbeat from core client for 30 sec - exiting 11:57:49 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:17:42 (6756): No heartbeat from core client for 30 sec - exiting 14:17:43 (6756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:51:52 (6544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:55 (6736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:51 (7224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:52 (7224): No heartbeat from core client for 30 sec - exiting 22:11:53 (7224): No heartbeat from core client for 30 sec - exiting 22:11:54 (7224): No heartbeat from core client for 30 sec - exiting 22:11:55 (7224): No heartbeat from core client for 30 sec - exiting 22:14:41 (7424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:16:45 (7552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:17:41 (2400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:29:34 (6364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:31:23 (8012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:31:24 (8012): No heartbeat from core client for 30 sec - exiting 14:19:13 (6796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:06 (6684): No heartbeat from core client for 30 sec - exiting 18:18:07 (6684): No heartbeat from core client for 30 sec - exiting 18:18:08 (6684): No heartbeat from core client for 30 sec - exiting 18:18:09 (6684): No heartbeat from core client for 30 sec - exiting 18:18:10 (6684): No heartbeat from core client for 30 sec - exiting 18:18:11 (6684): No heartbeat from core client for 30 sec - exiting 18:18:32 (7116): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 18:18:38 (4156): start_timer_thread(): CreateThread() failed, errno 0 18:27:03 (2500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:42 (1528): start_timer_thread(): CreateThread() failed, errno 0 18:28:33 (1556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:26 (3276): start_timer_thread(): CreateThread() failed, errno 0 18:31:10 (2480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:19 (3996): start_timer_thread(): CreateThread() failed, errno 0 18:33:48 (692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:07 (4988): start_timer_thread(): CreateThread() failed, errno 0 18:35:30 (2312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:49 (3836): start_timer_thread(): CreateThread() failed, errno 0 18:37:19 (3896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not8:37:20 (3896): No heartbeat from core client for 30 sec - exiting 18:37:43 (7172): start_timer_thread(): CreateThread() failed, errno 0 18:38:26 (2916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:05 (5452): start_timer_thread(): CreateThread() failed, errno 0 18:39:35 (3152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:36 (3152): No heartbeat from core client for 30 sec - exiting 18:39:37 (3152): No heartbeat from core client for 30 sec - exiting 18:39:50 (5316): start_timer_thread(): CreateThread() failed, errno 0 18:40:34 (6688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4992, selfPID=4992, iMonCtr=2 18:42:32 (4848): start_timer_thread(): CreateThread() failed, errno 0 18:43:23 (8156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:24 (8156): No heartbeat from core client for 30 sec - exiting 18:44:48 (5116): start_timer_thread(): CreateThread() failed, errno 0 18:45:32 (7760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:46:36 (6704): start_timer_thread(): CreateThread() failed, errno 0 18:47:14 (3540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not run18:47:15 (3540): No heartbeat from core client for 30 sec - exiting 18:49:31 (2460): start_timer_thread(): CreateThread() failed, errno 0 18:50:15 (7452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:50:17 (7452): No heartbeat from core client for 30 sec - exiting 18:50:18 (7452): No heartbeat from core client for 30 sec - exiting 18:50:19 (7452): No heartbeat from core client for 30 sec - exiting 18:50:20 (7452): No heartbeat from core client for 30 sec - exiting 18:52:33 (6220): No heartbeat from core client for 30 sec - exiting 18:52:34 (6220): No heartbeat from core client for 30 sec - exiting 18:52:35 (6220): No heartbeat from core client for 30 sec - exiting 18:52:36 (6220): No heartbeat from core client for 30 sec - exiting 18:52:37 (6220): No heartbeat from core client for 30 sec - exiting 18:52:38 (6220): No heartbeat from core client for 30 sec - exiting 18:52:39 (6220): No heartbeat from core client for 30 sec - exiting 18:52:40 (6220): No heartbeat from core client for 30 sec - exiting 18:52:41 (6220): No heartbeat from core client for 30 sec - exiting 18:52:42 (6220): No heartbeat from core client for 30 sec - exiting 18:52:43 (6220): No heartbeat from core client for 30 sec - exiting 18:52:44 (6220): No heartbeat from core client for 30 sec - exiting 18:52:45 (6220): No heartbeat from core client for 30 sec - exiting 18:52:46 (6220): No heartbeat from core client for 30 sec - exiting 18:52:47 (6220): No heartbeat from core client for 30 sec - exiting 18:52:48 (6220): No heartbeat from core client for 30 sec - exiting 18:52:49 (6220): No heartbeat from core client for 30 sec - exiting 18:52:50 (6220): No heartbeat from core client for 30 sec - exiting 18:52:51 (7252): start_timer_thread(): CreateThread() failed, errno 0 18:52:51 (6220): No heartbeat from core client for 30 sec - exiting 18:52:52 (6220): No heartbeat from core client for 30 sec - exiting 18:52:53 (6220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:15 (5172): start_timer_thread(): CreateThread() failed, errno 0 18:53:34 (7228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4376, selfPID=4376, iMonCtr=2 18:55:25 (8064): start_timer_thread(): CreateThread() failed, errno 0 18:56:18 (6596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:19 (6596): No heartbeat from core client for 30 sec - exiting 18:57:43 (7444): start_timer_thread(): CreateThread() failed, errno 0 18:59:12 (7508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:00:10 (7352): start_timer_thread(): CreateThread() failed, errno 0 19:01:17 (1540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:01:54 (1444): start_timer_thread(): CreateThread() failed, errno 0 19:03:12 (7392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:03:59 (2916): start_timer_thread(): CreateThread() failed, errno 0 19:04:29 (8108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:04:43 (7408): start_timer_thread(): CreateThread() failed, errno 0 19:05:25 (4988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:37 (6680): start_timer_thread(): CreateThread() failed, errno 0 19:07:21 (7036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:09:29 (5456): start_timer_thread(): CreateThread() failed, errno 0 19:11:01 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:18 (6660): start_timer_thread(): CreateThread() failed, errno 0 19:12:47 (6212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:44 (4660): start_timer_thread(): CreateThread() failed, errno 0 19:15:27 (7236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:00 (4596): start_timer_thread(): CreateThread() failed, errno 0 19:17:44 (7932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:18:50 (6704): start_timer_thread(): CreateThread() failed, errno 0 19:19:25 (1376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:03 (6992): start_timer_thread(): CreateThread() failed, errno 0 19:21:49 (7652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:16 (7508): No heartbeat from core client for 30 sec - exiting 19:24:17 (7508): No heartbeat from core client for 30 sec - exiting 19:24:18 (7508): No heartbeat from core client for 30 sec - exiting 19:24:19 (7508): No heartbeat from core client for 30 sec - exiting 19:24:20 (6228): start_timer_thread(): CreateThread() failed, errno 0 19:24:20 (7508): No heartbeat from core client for 30 sec - exiting 19:24:21 (7508): No heartbeat from core client for 30 sec - exiting 19:24:22 (7508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:45 (6684): start_timer_thread(): CreateThread() failed, errno 0 19:25:25 (1724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:27:10 (8100): start_timer_thread(): CreateThread() failed, errno 0 19:27:39 (6316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:27:55 (7532): start_timer_thread(): CreateThread() failed, errno 0 19:29:36 (7232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:44 (3836): start_timer_thread(): CreateThread() failed, errno 0 19:32:14 (8056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:34 (1452): start_timer_thread(): CreateThread() failed, errno 0 19:34:03 (7748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:24 (7332): start_timer_thread(): CreateThread() failed, errno 0 19:36:12 (7820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:43 (3960): start_timer_thread(): CreateThread() failed, errno 0 19:39:11 (7020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:46 (6580): start_timer_thread(): CreateThread() failed, errno 0 19:40:16 (6776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:40:33 (4980): start_timer_thread(): CreateThread() failed, errno 0 19:41:36 (5752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:43:34 (7748): start_timer_thread(): CreateThread() failed, errno 0 19:44:14 (2520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3844, selfPID=3844, iMonCtr=2 19:46:13 (3960): start_timer_thread(): CreateThread() failed, errno 0 19:50:29 (7408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:30 (7408): No heartbeat from core client for 30 sec - exiting 19:50:31 (7408): No heartbeat from core client for 30 sec - exiting 19:51:39 (7880): start_timer_thread(): CreateThread() failed, errno 0 19:52:19 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:54:14 (6220): start_timer_thread(): CreateThread() failed, errno 0 19:54:44 (7252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:04 (4156): start_timer_thread(): CreateThread() failed, errno 0 19:55:33 (8156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:02 (4036): start_timer_thread(): CreateThread() failed, errno 0 19:58:58 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1800, selfPID=1800, iMonCtr=2 20:00:17 (7992): start_timer_thread(): CreateThread() failed, errno 0 20:01:00 (6836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:02 (6048): start_timer_thread(): CreateThread() failed, errno 0 20:03:31 (2128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:42 (5820): start_timer_thread(): CreateThread() failed, errno 0 20:04:11 (3224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:04:12 (3224): No heartbeat from core client for 30 sec - exiting 20:06:57 (3996): start_timer_thread(): CreateThread() failed, errno 0 20:07:27 (3876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:07:28 (3876): No heartbeat from core client for 30 sec - exiting 20:07:29 (3876): No heartbeat from core client for 30 sec - exiting 20:07:30 (3876): No heartbeat from core client for 30 sec - exiting 20:07:31 (3876): No heartbeat from core client for 30 sec - exiting 20:07:32 (3876): No heartbeat from core client for 30 sec - exiting 20:08:32 (5912): start_timer_thread(): CreateThread() failed, errno 0 20:09:19 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 220:11:05 (3960): No heartbeat from core c0ient for 30 sec - exiting :10:39 (2128): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 20:11:59 (6620): start_timer_thread(): CreateThread() failed, errno 0 20:12:29 (7644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:03 (3376): start_timer_thread(): CreateThread() failed, errno 0 20:16:32 (1720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:18:48 (5896): start_timer_thread(): CreateThread() failed, errno 0 20:19:27 (6224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:21:24 (7532): start_timer_thread(): CreateThread() failed, errno 0 20:21:53 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:23:24 (7256): start_timer_thread(): CreateThread() failed, errno 0 20:24:13 (8080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:26:12 (4516): start_timer_thread(): CreateThread() failed, errno 0 20:26:42 (1256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:26:52 (6596): start_timer_thread(): CreateThread() failed, errno 0 20:27:42 (7520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:29:57 (5456): start_timer_thread(): CreateThread() failed, errno 0 20:30:26 (3180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3808, selfPID=3808, iMonCtr=2 20:32:39 (3464): start_timer_thread(): CreateThread() failed, errno 0 20:34:06 (1548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:15 (1520): start_timer_thread(): CreateThread() failed, errno 0 20:35:48 (7300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:04 (3400): start_timer_thread(): CreateThread() failed, errno 0 20:36:45 (7580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:37:52 (8140): start_timer_thread(): CreateThread() failed, errno 0 20:38:18 (6228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:40:53 (2268): start_timer_thread(): CreateThread() failed, errno 0 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/xaakm.pipe_dummy 2048 Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 7 zip error: Output file write failure (write error on zip file) 20:49:29 (6920): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Mar 2011 14:26:27	570789	12619994	hadam3p_pnw_30cw_1984_1_007180616_0	80,736	307,159	3.8045
12 Mar 2011 00:45:36	570789	12619994	hadam3p_pnw_30cw_1984_1_007180616_0	69,216	261,244	3.7743
11 Mar 2011 11:36:13	570789	12619994	hadam3p_pnw_30cw_1984_1_007180616_0	57,696	217,427	3.7685
10 Mar 2011 21:54:13	570789	12619994	hadam3p_pnw_30cw_1984_1_007180616_0	46,176	173,673	3.7611
10 Mar 2011 09:24:13	570789	12619994	hadam3p_pnw_30cw_1984_1_007180616_0	34,656	129,810	3.7457
09 Mar 2011 19:03:27	570789	12619994	hadam3p_pnw_30cw_1984_1_007180616_0	23,136	85,731	3.7055
09 Mar 2011 03:11:40	570789	12619994	hadam3p_pnw_30cw_1984_1_007180616_0	11,616	42,857	3.6895