Task 14648192

Name	hadcm3n_103n_1980_40_007957198_1
Workunit	8112310
Created	9 May 2012, 11:58:33 UTC
Sent	9 May 2012, 11:59:02 UTC
Report deadline	8 Aug 2012, 19:26:13 UTC
Received	3 Jun 2012, 9:54:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1194303
Run time	15 days 12 hours 53 min 11 sec
CPU time	15 days 9 hours 7 min 47 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:52:17 (5768): No heartbeat from core client for 30 sec - exiting 07:52:19 (5768): No heartbeat from core client for 30 sec - exiting 07:52:20 (5768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:03:09 (4252): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5424, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:02:05 (4644): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/103nko.pjj9c10 Error converting file to netcdf: dataout/103nko.pij9c10 Error converting file to netcdf: dataout/103nko.pfj9c10 Error converting file to netcdf: dataout/103nka.phj9c10 Error converting file to netcdf: dataout/103nka.pgj9c10 Error converting file to netcdf: dataout/103nka.pej9c10 Error converting file to netcdf: dataout/103nka.pdj9c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:59:00 (2724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 19:25:17 (4240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:59:01 (1376): No heartbeat from core client for 30 sec - exiting 20:59:02 (1376): No heartbeat from core client for 30 sec - exiting 20:59:03 (1376): No heartbeat from core client for 30 sec - exiting 20:59:04 (1376): No heartbeat from core client for 30 sec - exiting 20:59:06 (1376): No heartbeat from core client for 30 sec - exiting 20:59:07 (1376): No heartbeat from core client for 30 sec - exiting 20:59:08 (1376): No heartbeat from core client for 30 sec - exiting 20:59:09 (1376): No heartbeat from core client for 30 sec - exiting 20:59:10 (1376): No heartbeat from core client for 30 sec - exiting 20:59:11 (1376): No heartbeat from core client for 30 sec - exiting 20:59:12 (1376): No heartbeat from core client for 30 sec - exiting 20:59:13 (1376): No heartbeat from core client for 30 sec - exiting 20:59:14 (1376): No heartbeat from core client for 30 sec - exiting 20:59:15 (1376): No heartbeat from core client for 30 sec - exiting 20:59:16 (1376): No heartbeat from core client for 30 sec - exiting 20:59:17 (1376): No heartbeat from core client for 30 sec - exiting 20:59:18 (1376): No heartbeat from core client for 30 sec - exiting 20:59:19 (1376): No heartbeat from core client for 30 sec - exiting 20:59:21 (1376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4596, iMonCtr=1 Model crash detected, will try to restart... 14:20:52 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jun 2012 21:57:57	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	777,600	1,328,864	1.7089
02 Jun 2012 04:01:54	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	751,680	1,284,096	1.7083
01 Jun 2012 11:46:23	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	725,760	1,238,999	1.7072
31 May 2012 12:10:48	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	699,840	1,194,547	1.7069
30 May 2012 17:19:20	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	673,920	1,150,610	1.7073
29 May 2012 21:18:13	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	648,000	1,105,103	1.7054
29 May 2012 01:25:06	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	622,080	1,060,966	1.7055
28 May 2012 08:44:22	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	596,160	1,016,614	1.7053
27 May 2012 12:46:16	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	570,240	971,978	1.7045
27 May 2012 00:23:21	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	544,320	927,497	1.7040
26 May 2012 07:33:47	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	518,400	883,552	1.7044
25 May 2012 06:19:58	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	492,480	839,635	1.7049
24 May 2012 07:07:10	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	466,560	795,277	1.7046
23 May 2012 06:13:38	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	440,640	751,493	1.7055
22 May 2012 08:02:45	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	414,720	705,608	1.7014
21 May 2012 15:59:52	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	388,800	661,309	1.7009
20 May 2012 15:20:21	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	362,880	617,005	1.7003
18 May 2012 23:53:53	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	336,960	572,913	1.7002
18 May 2012 11:48:47	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	311,040	529,297	1.7017
17 May 2012 23:35:05	1194303	14648192	hadcm3n_103n_1980_40_007957198_1	285,120	485,498	1.7028