Task 15937527

Name	hadcm3n_4f6m_1940_40_008302606_3
Workunit	8453741
Created	23 Aug 2013, 19:14:38 UTC
Sent	24 Aug 2013, 0:26:19 UTC
Report deadline	23 Nov 2013, 7:53:30 UTC
Received	30 Sep 2013, 20:50:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1212732
Run time	5 days 8 hours 26 min 23 sec
CPU time	4 days 19 hours 29 min 54 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:33:32 (8780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9876, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:12:05 (3508): No heartbeat from core client for 30 sec - exiting 20:12:06 (3508): No heartbeat from core client for 30 sec - exiting 20:12:07 (3508): No heartbeat from core client for 30 sec - exiting 20:12:08 (3508): No heartbeat from core client for 30 sec - exiting 20:12:09 (3508): No heartbeat from core client for 30 sec - exiting 20:12:10 (3508): No heartbeat from core client for 30 sec - exiting 20:12:11 (3508): No heartbeat from core client for 30 sec - exiting 20:12:12 (3508): No heartbeat from core client for 30 sec - exiting 20:12:13 (3508): No heartbeat from core client for 30 sec - exiting 20:12:14 (3508): No heartbeat from core client for 30 sec - exiting 20:12:15 (3508): No heartbeat from core client for 30 sec - exiting 20:12:16 (3508): No heartbeat from core client for 30 sec - exiting 20:12:17 (3508): No heartbeat from core client for 30 sec - exiting 20:12:18 (3508): No heartbeat from core client for 30 sec - exiting 20:12:19 (3508): No heartbeat from core client for 30 sec - exiting 20:12:20 (3508): No heartbeat from core client for 30 sec - exiting 20:12:21 (3508): No heartbeat from core client for 30 sec - exiting 20:12:22 (3508): No heartbeat from core client for 30 sec - exiting 20:12:23 (3508): No heartbeat from core client for 30 sec - exiting 20:12:24 (3508): No heartbeat from core client for 30 sec - exiting 20:12:25 (3508): No heartbeat from core client for 30 sec - exiting 20:12:26 (3508): No heartbeat from core client for 30 sec - exiting 20:12:27 (3508): No heartbeat from core client for 30 sec - exiting 20:12:28 (3508): No heartbeat from core client for 30 sec - exiting 20:12:29 (3508): No heartbeat from core client for 30 sec - exiting 20:12:30 (3508): No heartbeat from core client for 30 sec - exiting 20:12:31 (3508): No heartbeat from core client for 30 sec - exiting 20:12:32 (3508): No heartbeat from core client for 30 sec - exiting 20:12:33 (3508): No heartbeat from core client for 30 sec - exiting 20:12:34 (3508): No heartbeat from core client for 30 sec - exiting 20:12:35 (3508): No heartbeat from core client for 30 sec - exiting 20:12:36 (3508): No heartbeat from core client for 30 sec - exiting 20:12:37 (3508): No heartbeat from core client for 30 sec - exiting 20:12:38 (3508): No heartbeat from core client for 30 sec - exiting 20:12:39 (3508): No heartbeat from core client for 30 sec - exiting 20:12:40 (3508): No heartbeat from core client for 30 sec - exiting 20:12:41 (3508): No heartbeat from core client for 30 sec - exiting 20:12:42 (3508): No heartbeat from core client for 30 sec - exiting 20:12:43 (3508): No heartbeat from core client for 30 sec - exiting 20:12:44 (3508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:19:31 (5184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4f6mko.pjf0c10 Error converting file to netcdf: dataout/4f6mko.pif0c10 Error converting file to netcdf: dataout/4f6mko.pff0c10 Error converting file to netcdf: dataout/4f6mko.pcf0c10 Error converting file to netcdf: dataout/4f6mko.pbf0c10 Error converting file to netcdf: dataout/4f6mko.paf0c10 Error converting file to netcdf: dataout/4f6mka.phf0c10 Error converting file to netcdf: dataout/4f6mka.pgf0c10 Error converting file to netcdf: dataout/4f6mka.pef0c10 Error converting file to netcdf: dataout/4f6mka.pdf0c10 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77973AC3 read attempt to address 0x42D75F53 Engaging BOINC Windows Runtime Debugger... Cannot serialize file D:\DATA\Tomas\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4f6m_1940_40_008302606/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Sep 2013 20:54:47	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	259,200	415,794	1.6041
28 Sep 2013 18:59:17	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	233,280	374,773	1.6065
24 Sep 2013 00:34:51	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	207,360	335,197	1.6165
15 Sep 2013 13:50:33	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	181,440	293,894	1.6198
10 Sep 2013 20:57:38	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	155,520	251,861	1.6195
04 Sep 2013 22:05:39	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	129,600	209,496	1.6165
01 Sep 2013 16:17:17	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	103,680	166,408	1.6050
31 Aug 2013 13:57:17	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	77,760	124,002	1.5947
28 Aug 2013 22:52:08	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	51,840	82,028	1.5823
25 Aug 2013 21:25:43	1212732	15937527	hadcm3n_4f6m_1940_40_008302606_3	25,920	41,157	1.5878