Task 15557507

Name	hadcm3n_zj6i_1920_40_008255977_2
Workunit	8411101
Created	25 Jan 2013, 19:18:35 UTC
Sent	25 Jan 2013, 19:18:42 UTC
Report deadline	27 Apr 2013, 2:45:53 UTC
Received	18 Mar 2013, 21:34:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1126273
Run time	23 days 1 hours 58 min 14 sec
CPU time	18 days 17 hours 57 min 7 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	0.94 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:50:56 (3180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:50:57 (3180): No heartbeat from core client for 30 sec - exiting 14:50:58 (3180): No heartbeat from core client for 30 sec - exiting 14:50:59 (3180): No heartbeat from core client for 30 sec - exiting 14:51:00 (3180): No heartbeat from core client for 30 sec - exiting 14:51:01 (3180): No heartbeat from core client for 30 sec - exiting 14:51:02 (3180): No heartbeat from core client for 30 sec - exiting 14:54:06 (6524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:07 (6524): No heartbeat from core client for 30 sec - exiting 14:54:08 (6524): No heartbeat from core client for 30 sec - exiting 14:54:09 (6524): No heartbeat from core client for 30 sec - exiting 14:54:10 (6524): No heartbeat from core client for 30 sec - exiting 18:11:18 (4380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:08:25 (820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:08:26 (820): No heartbeat from core client for 30 sec - exiting 17:20:43 (3072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:45 (3788): No heartbeat from core client for 30 sec - exiting 17:23:46 (3788): No heartbeat from core client for 30 sec - exiting 17:23:47 (3788): No heartbeat from core client for 30 sec - exiting 17:23:48 (3788): No heartbeat from core client for 30 sec - exiting 17:23:49 (3788): No heartbeat from core client for 30 sec - exiting 17:23:50 (3788): No heartbeat from core client for 30 sec - exiting 17:23:51 (3788): No heartbeat from core client for 30 sec - exiting 17:23:52 (3788): No heartbeat from core client for 30 sec - exiting 17:23:53 (3788): No heartbeat from core client for 30 sec - exiting 17:23:54 (3788): No heartbeat from core client for 30 sec - exiting 17:23:55 (3788): No heartbeat from core client for 30 sec - exiting 17:23:56 (3788): No heartbeat from core client for 30 sec - exiting 17:23:57 (3788): No heartbeat from core client for 30 sec - exiting 17:23:58 (3788): No heartbeat from core client for 30 sec - exiting 17:23:59 (3788): No heartbeat from core client for 30 sec - exiting 17:24:00 (3788): No heartbeat from core client for 30 sec - exiting 17:24:01 (3788): No heartbeat from core client for 30 sec - exiting 17:24:02 (3788): No heartbeat from core client for 30 sec - exiting 17:24:03 (3788): No heartbeat from core client for 30 sec - exiting 17:24:04 (3788): No heartbeat from core client for 30 sec - exiting 17:24:05 (3788): No heartbeat from core client for 30 sec - exiting 17:24:06 (3788): No heartbeat from core client for 30 sec - exiting 17:24:07 (3788): No heartbeat from core client for 30 sec - exiting 17:24:08 (3788): No heartbeat from core client for 30 sec - exiting 17:24:09 (3788): No heartbeat from core client for 30 sec - exiting 17:24:11 (3788): No heartbeat from core client for 30 sec - exiting 17:24:12 (3788): No heartbeat from core client for 30 sec - exiting 17:24:13 (3788): No heartbeat from core client for 30 sec - exiting 17:24:14 (3788): No heartbeat from core client for 30 sec - exiting 17:24:15 (3788): No heartbeat from core client for 30 sec - exiting 17:24:16 (3788): No heartbeat from core client for 30 sec - exiting 17:24:17 (3788): No heartbeat from core client for 30 sec - exiting 17:24:18 (3788): No heartbeat from core client for 30 sec - exiting 17:24:19 (3788): No heartbeat from core client for 30 sec - exiting 17:24:20 (3788): No heartbeat from core client for 30 sec - exiting 17:24:21 (3788): No heartbeat from core client for 30 sec - exiting 17:24:22 (3788): No heartbeat from core client for 30 sec - exiting 17:24:23 (3788): No heartbeat from core client for 30 sec - exiting 17:24:24 (3788): No heartbeat from core client for 30 sec - exiting 17:24:25 (3788): No heartbeat from core client for 30 sec - exiting 17:24:26 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:24:27 (3788): No heartbeat from core client for 30 sec - exiting 17:24:28 (3788): No heartbeat from core client for 30 sec - exiting 17:24:29 (3788): No heartbeat from core client for 30 sec - exiting 17:24:30 (3788): No heartbeat from core client for 30 sec - exiting 17:24:31 (3788): No heartbeat from core client for 30 sec - exiting 17:30:03 (7608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:04 (7608): No heartbeat from core client for 30 sec - exiting 17:30:05 (7608): No heartbeat from core client for 30 sec - exiting 17:39:00 (7544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7544, iMonCtr=1 Model crash detected, will try to restart... 20:03:53 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:54 (5908): No heartbeat from core client for 30 sec - exiting 20:03:55 (5908): No heartbeat from core client for 30 sec - exiting 20:03:56 (5908): No heartbeat from core client for 30 sec - exiting 20:03:57 (5908): No heartbeat from core client for 30 sec - exiting 20:03:58 (5908): No heartbeat from core client for 30 sec - exiting 20:03:59 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 13:18:12 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:52:11 (5220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:52:12 (5220): No heartbeat from core client for 30 sec - exiting 17:52:13 (5220): No heartbeat from core client for 30 sec - exiting 17:52:14 (5220): No heartbeat from core client for 30 sec - exiting 17:52:15 (5220): No heartbeat from core client for 30 sec - exiting 17:52:16 (5220): No heartbeat from core client for 30 sec - exiting 17:57:04 (7988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:31 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:24:36 (7696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:00 (1688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:01 (1688): No heartbeat from core client for 30 sec - exiting 18:31:02 (1688): No heartbeat from core client for 30 sec - exiting 18:31:04 (1688): No heartbeat from core client for 30 sec - exiting 18:31:05 (1688): No heartbeat from core client for 30 sec - exiting 18:34:03 (5320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:04 (5320): No heartbeat from core client for 30 sec - exiting 18:40:11 (7576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:12 (7576): No heartbeat from core client for 30 sec - exiting 18:40:13 (7576): No heartbeat from core client for 30 sec - exiting 18:40:14 (7576): No heartbeat from core client for 30 sec - exiting 18:40:15 (7576): No heartbeat from core client for 30 sec - exiting 18:40:16 (7576): No heartbeat from core client for 30 sec - exiting 18:46:11 (7736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:46:12 (7736): No heartbeat from core client for 30 sec - exiting 18:46:13 (7736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zj6iko.pjd1c10 Error converting file to netcdf: dataout/zj6iko.pid1c10 Error converting file to netcdf: dataout/zj6iko.pfd1c10 Error converting file to netcdf: dataout/zj6ika.phd1c10 Error converting file to netcdf: dataout/zj6ika.pgd1c10 Error converting file to netcdf: dataout/zj6ika.ped1c10 Error converting file to netcdf: dataout/zj6ika.pdd1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:47:55 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:47:56 (3272): No heartbeat from core client for 30 sec - exiting 19:47:57 (3272): No heartbeat from core client for 30 sec - exiting 19:47:58 (3272): No heartbeat from core client for 30 sec - exiting 19:47:59 (3272): No heartbeat from core client for 30 sec - exiting 19:48:00 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:12:52 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:53 (5460): No heartbeat from core client for 30 sec - exiting 19:12:54 (5460): No heartbeat from core client for 30 sec - exiting 19:12:55 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zj6iko.pjd5c10 Error converting file to netcdf: dataout/zj6iko.pid5c10 Error converting file to netcdf: dataout/zj6iko.pfd5c10 Error converting file to netcdf: dataout/zj6ika.phd5c10 Error converting file to netcdf: dataout/zj6ika.pgd5c10 Error converting file to netcdf: dataout/zj6ika.ped5c10 Error converting file to netcdf: dataout/zj6ika.pdd5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5996, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5024, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3028, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3004, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:08:51 (4980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:08:52 (4980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... 05:49:06 (4240): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77315EAB read attempt to address 0x40646271 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zj6i_1920_40_008255977/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Mar 2013 20:34:47	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	518,400	1,619,825	3.1247
15 Mar 2013 18:57:54	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	492,480	1,536,897	3.1207
09 Mar 2013 20:11:33	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	466,560	1,456,125	3.1210
03 Mar 2013 17:11:13	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	440,640	1,376,817	3.1246
01 Mar 2013 18:49:13	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	414,720	1,293,080	3.1180
25 Feb 2013 20:09:39	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	388,800	1,213,255	3.1205
21 Feb 2013 16:30:55	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	362,880	1,133,336	3.1232
19 Feb 2013 22:25:36	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	336,960	1,051,277	3.1199
18 Feb 2013 15:50:42	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	311,040	968,523	3.1138
17 Feb 2013 05:56:25	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	285,120	886,518	3.1093
15 Feb 2013 18:39:45	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	259,200	806,781	3.1126
13 Feb 2013 23:43:38	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	233,280	725,776	3.1112
11 Feb 2013 02:29:52	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	207,360	647,220	3.1212
09 Feb 2013 01:14:56	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	181,440	565,067	3.1143
07 Feb 2013 17:22:25	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	155,520	484,983	3.1185
05 Feb 2013 04:10:58	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	129,600	404,286	3.1195
03 Feb 2013 04:36:44	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	103,680	321,478	3.1007
01 Feb 2013 04:25:55	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	77,760	239,773	3.0835
29 Jan 2013 22:31:49	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	51,840	160,223	3.0907
27 Jan 2013 19:40:02	1126273	15557507	hadcm3n_zj6i_1920_40_008255977_2	25,920	80,338	3.0995