Task 13340263

Name	hadcm3n_o2hl_1900_40_007439391_2
Workunit	7636894
Created	6 Sep 2011, 10:10:40 UTC
Sent	6 Sep 2011, 10:24:01 UTC
Report deadline	6 Dec 2011, 17:51:12 UTC
Received	17 Sep 2011, 3:44:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-177 (0xFFFFFF4F) ERR_RSC_LIMIT_EXCEEDED
Computer ID	1167588
Run time	6 days 0 hours 59 min 5 sec
CPU time	4 days 5 hours 43 min 5 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	2.21 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> Maximum memory exceeded </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:20:44 (1740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:45 (1740): No heartbeat from core client for 30 sec - exiting 04:20:46 (1740): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:06:46 (3436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:06:48 (3436): No heartbeat from core client for 30 sec - exiting 14:06:49 (3436): No heartbeat from core client for 30 sec - exiting 14:06:50 (3436): No heartbeat from core client for 30 sec - exiting 14:06:51 (3436): No heartbeat from core client for 30 sec - exiting 14:06:52 (3436): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:14:35 (3852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:14:37 (3852): No heartbeat from core client for 30 sec - exiting 13:14:38 (3852): No heartbeat from core client for 30 sec - exiting 13:14:39 (3852): No heartbeat from core client for 30 sec - exiting 13:14:40 (3852): No heartbeat from core client for 30 sec - exiting 14:09:51 (528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:47:00 (1728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:47:02 (1728): No heartbeat from core client for 30 sec - exiting 16:45:54 (1312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:45:56 (1312): No heartbeat from core client for 30 sec - exiting 16:45:57 (1312): No heartbeat from core client for 30 sec - exiting 17:55:47 (2600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:48 (2600): No heartbeat from core client for 30 sec - exiting 17:55:49 (2600): No heartbeat from core client for 30 sec - exiting 21:38:33 (2384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:35 (2384): No heartbeat from core client for 30 sec - exiting 21:38:36 (2384): No heartbeat from core client for 30 sec - exiting 22:51:37 (3008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:51:39 (3008): No heartbeat from core client for 30 sec - exiting 22:51:40 (3008): No heartbeat from core client for 30 sec - exiting 22:59:04 (3588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:59:06 (3588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 23:36:04 (3548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:36:06 (3548): No heartbeat from core client for 30 sec - exiting 23:36:07 (3548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 01:02:03 (484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:02:05 (484): No heartbeat from core client for 30 sec - exiting 01:02:06 (484): No heartbeat from core client for 30 sec - exiting 01:54:36 (3464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:38 (3464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:36:13 (500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:36:15 (500): No heartbeat from core client for 30 sec - exiting 03:36:16 (500): No heartbeat from core client for 30 sec - exiting 03:36:17 (500): No heartbeat from core client for 30 sec - exiting 06:20:35 (260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:37 (260): No heartbeat from core client for 30 sec - exiting 06:20:38 (260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:57:44 (2460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:57:47 (2460): No heartbeat from core client for 30 sec - exiting 11:57:48 (2460): No heartbeat from core client for 30 sec - exiting 11:57:49 (2460): No heartbeat from core client for 30 sec - exiting 11:57:50 (2460): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:22:35 (2204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:22:38 (2204): No heartbeat from core client for 30 sec - exiting 15:22:40 (2204): No heartbeat from core client for 30 sec - exiting 15:22:41 (2204): No heartbeat from core client for 30 sec - exiting 15:36:38 (3952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:40 (3952): No heartbeat from core client for 30 sec - exiting 15:36:41 (3952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:11:18 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:21 (4072): No heartbeat from core client for 30 sec - exiting 21:11:22 (4072): No heartbeat from core client for 30 sec - exiting 21:11:23 (4072): No heartbeat from core client for 30 sec - exiting 21:47:26 (3208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:47:28 (3208): No heartbeat from core client for 30 sec - exiting 21:47:29 (3208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:54:48 (3304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:49 (3304): No heartbeat from core client for 30 sec - exiting 23:54:50 (3304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 01:08:49 (1320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:08:51 (1320): No heartbeat from core client for 30 sec - exiting 01:08:52 (1320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:30:12 (3284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:30:14 (3284): No heartbeat from core client for 30 sec - exiting 11:30:15 (3284): No heartbeat from core client for 30 sec - exiting 11:30:16 (3284): No heartbeat from core client for 30 sec - exiting 12:30:36 (3220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:30:39 (3220): No heartbeat from core client for 30 sec - exiting 12:30:40 (3220): No heartbeat from core client for 30 sec - exiting 12:40:26 (4048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:40:28 (4048): No heartbeat from core client for 30 sec - exiting 12:40:29 (4048): No heartbeat from core client for 30 sec - exiting 12:49:21 (3636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:49:23 (3636): No heartbeat from core client for 30 sec - exiting 12:49:24 (3636): No heartbeat from core client for 30 sec - exiting 12:49:25 (3636): No heartbeat from core client for 30 sec - exiting 12:58:11 (3164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:58:13 (3164): No heartbeat from core client for 30 sec - exiting 12:58:14 (3164): No heartbeat from core client for 30 sec - exiting 13:19:11 (3536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:19:12 (3536): No heartbeat from core client for 30 sec - exiting 14:21:21 (2584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:21:23 (2584): No heartbeat from core client for 30 sec - exiting 14:21:24 (2584): No heartbeat from core client for 30 sec - exiting 14:21:25 (2584): No heartbeat from core client for 30 sec - exiting 14:22:16 (3924): No heartbeat from core client for 30 sec - exiting 14:22:17 (3924): No heartbeat from core client for 30 sec - exiting 14:22:18 (3924): No heartbeat from core client for 30 sec - exiting 14:22:19 (3924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:52 (2992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:54 (2992): No heartbeat from core client for 30 sec - exiting 15:07:55 (2992): No heartbeat from core client for 30 sec - exiting 16:31:51 (2936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:31:54 (2936): No heartbeat from core client for 30 sec - exiting 16:31:55 (2936): No heartbeat from core client for 30 sec - exiting 16:32:44 (4064): No heartbeat from core client for 30 sec - exiting 16:32:46 (4064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:05 (1796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:07 (1796): No heartbeat from core client for 30 sec - exiting 17:00:08 (1796): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 13:35:05 (3348): No heartbeat from core client for 30 sec - exiting 13:35:07 (3348): No heartbeat from core client for 30 sec - exiting 13:35:08 (3348): No heartbeat from core client for 30 sec - exiting 13:35:09 (3348): No heartbeat from core client for 30 sec - exiting 13:35:10 (3348): No heartbeat from core client for 30 sec - exiting 13:35:11 (3348): No heartbeat from core client for 30 sec - exiting 13:35:12 (3348): No heartbeat from core client for 30 sec - exiting 13:35:13 (3348): No heartbeat from core client for 30 sec - exiting 13:35:14 (3348): No heartbeat from core client for 30 sec - exiting 13:35:15 (3348): No heartbeat from core client for 30 sec - exiting 13:35:17 (3348): No heartbeat from core client for 30 sec - exiting 13:35:18 (3348): No heartbeat from core client for 30 sec - exiting 13:35:19 (3348): No heartbeat from core client for 30 sec - exiting 13:35:20 (3348): No heartbeat from core client for 30 sec - exiting 13:35:21 (3348): No heartbeat from core client for 30 sec - exiting 13:35:22 (3348): No heartbeat from core client for 30 sec - exiting 13:35:23 (3348): No heartbeat from core client for 30 sec - exiting 13:35:24 (3348): No heartbeat from core client for 30 sec - exiting 13:35:25 (3348): No heartbeat from core client for 30 sec - exiting 13:35:26 (3348): No heartbeat from core client for 30 sec - exiting 13:35:27 (3348): No heartbeat from core client for 30 sec - exiting 13:35:29 (3348): No heartbeat from core client for 30 sec - exiting 13:35:30 (3348): No heartbeat from core client for 30 sec - exiting 13:35:31 (3348): No heartbeat from core client for 30 sec - exiting 13:35:32 (3348): No heartbeat from core client for 30 sec - exiting 13:35:33 (3348): No heartbeat from core client for 30 sec - exiting 13:35:34 (3348): No heartbeat from core client for 30 sec - exiting 13:35:35 (3348): No heartbeat from core client for 30 sec - exiting 13:35:36 (3348): No heartbeat from core client for 30 sec - exiting 13:35:37 (3348): No heartbeat from core client for 30 sec - exiting 13:35:38 (3348): No heartbeat from core client for 30 sec - exiting 13:35:40 (3348): No heartbeat from core client for 30 sec - exiting 13:35:41 (3348): No heartbeat from core client for 30 sec - exiting 13:35:42 (3348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o2hlko.pja3c10 Error converting file to netcdf: dataout/o2hlko.pia3c10 Error converting file to netcdf: dataout/o2hlko.pfa3c10 Error converting file to netcdf: dataout/o2hlka.pha3c10 Error converting file to netcdf: dataout/o2hlka.pga3c10 Error converting file to netcdf: dataout/o2hlka.pea3c10 Error converting file to netcdf: dataout/o2hlka.pda3c10 15:47:47 (3048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:47:50 (3048): No heartbeat from core client for 30 sec - exiting 15:47:51 (3048): No heartbeat from core client for 30 sec - exiting 15:47:52 (3048): No heartbeat from core client for 30 sec - exiting 15:47:53 (3048): No heartbeat from core client for 30 sec - exiting 15:47:54 (3048): No heartbeat from core client for 30 sec - exiting 16:59:34 (3836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:59:36 (3836): No heartbeat from core client for 30 sec - exiting 16:59:38 (3836): No heartbeat from core client for 30 sec - exiting 16:59:39 (3836): No heartbeat from core client for 30 sec - exiting 17:48:28 (2724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:48:30 (2724): No heartbeat from core client for 30 sec - exiting 17:48:31 (2724): No heartbeat from core client for 30 sec - exiting 20:27:06 (3228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:27:08 (3228): No heartbeat from core client for 30 sec - exiting 20:27:09 (3228): No heartbeat from core client for 30 sec - exiting 23:42:38 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:42:40 (3756): No heartbeat from core client for 30 sec - exiting 23:42:41 (3756): No heartbeat from core client for 30 sec - exiting 06:24:14 (408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:17 (408): No heartbeat from core client for 30 sec - exiting 06:24:18 (408): No heartbeat from core client for 30 sec - exiting 06:24:19 (408): No heartbeat from core client for 30 sec - exiting 06:38:29 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:38:32 (3824): No heartbeat from core client for 30 sec - exiting 06:38:33 (3824): No heartbeat from core client for 30 sec - exiting 06:38:34 (3824): No heartbeat from core client for 30 sec - exiting 09:25:39 (2216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:41 (2216): No heartbeat from core client for 30 sec - exiting 13:04:54 (300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:56 (300): No heartbeat from core client for 30 sec - exiting 13:04:58 (300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:27:31 (1752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:27:33 (1752): No heartbeat from core client for 30 sec - exiting 20:27:34 (1752): No heartbeat from core client for 30 sec - exiting 20:51:16 (3692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:18 (3692): No heartbeat from core client for 30 sec - exiting 22:16:03 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:16:05 (3456): No heartbeat from core client for 30 sec - exiting 22:16:06 (3456): No heartbeat from core client for 30 sec - exiting 23:17:16 (3764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:17:19 (3764): No heartbeat from core client for 30 sec - exiting 23:17:20 (3764): No heartbeat from core client for 30 sec - exiting 23:59:38 (1536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:59:40 (1536): No heartbeat from core client for 30 sec - exiting 23:59:41 (1536): No heartbeat from core client for 30 sec - exiting 04:38:39 (2812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:38:40 (2812): No heartbeat from core client for 30 sec - exiting 04:38:41 (2812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Abort request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Sep 2011 12:47:24	1167588	13340263	hadcm3n_o2hl_1900_40_007439391_2	77,760	277,127	3.5639
13 Sep 2011 07:42:49	1167588	13340263	hadcm3n_o2hl_1900_40_007439391_2	51,840	183,141	3.5328
10 Sep 2011 05:36:09	1167588	13340263	hadcm3n_o2hl_1900_40_007439391_2	25,920	90,828	3.5042