Task 15449143

Name	hadcm3n_zats_1880_40_008249362_0
Workunit	8404486
Created	21 Nov 2012, 22:25:13 UTC
Sent	21 Nov 2012, 22:25:21 UTC
Report deadline	21 Feb 2013, 5:52:32 UTC
Received	30 Dec 2012, 16:10:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1092573
Run time	6 days 13 hours 43 min 22 sec
CPU time	4 days 19 hours 6 min 33 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.79 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 07:45:05 (4580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:45:07 (4580): No heartbeat from core client for 30 sec - exiting 07:45:08 (4580): No heartbeat from core client for 30 sec - exiting 07:45:09 (4580): No heartbeat from core client for 30 sec - exiting 07:45:10 (4580): No heartbeat from core client for 30 sec - exiting 07:45:11 (4580): No heartbeat from core client for 30 sec - exiting 07:45:12 (4580): No heartbeat from core client for 30 sec - exiting 07:45:13 (4580): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5104, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=748, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4232, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5708, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... 19:21:59 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4164, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4732, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5088, iMonCtr=1 Model crash detected, will try to restart... 21:01:37 (4776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:38 (4776): No heartbeat from core client for 30 sec - exiting 21:01:39 (4776): No heartbeat from core client for 30 sec - exiting 21:01:40 (4776): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5220, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7176, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5064, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5060, iMonCtr=1 Model crash detected, will try to restart... 21:30:13 (5428): No heartbeat from core client for 30 sec - exiting 21:30:14 (5428): No heartbeat from core client for 30 sec - exiting 21:30:15 (5428): No heartbeat from core client for 30 sec - exiting 21:30:16 (5428): No heartbeat from core client for 30 sec - exiting 21:30:17 (5428): No heartbeat from core client for 30 sec - exiting 21:30:18 (5428): No heartbeat from core client for 30 sec - exiting 21:30:19 (5428): No heartbeat from core client for 30 sec - exiting 21:30:20 (5428): No heartbeat from core client for 30 sec - exiting 21:30:21 (5428): No heartbeat from core client for 30 sec - exiting 21:30:22 (5428): No heartbeat from core client for 30 sec - exiting 21:30:23 (5428): No heartbeat from core client for 30 sec - exiting 21:30:24 (5428): No heartbeat from core client for 30 sec - exiting 21:30:25 (5428): No heartbeat from core client for 30 sec - exiting 21:30:26 (5428): No heartbeat from core client for 30 sec - exiting 21:30:27 (5428): No heartbeat from core client for 30 sec - exiting 21:30:28 (5428): No heartbeat from core client for 30 sec - exiting 21:30:29 (5428): No heartbeat from core client for 30 sec - exiting 21:30:30 (5428): No heartbeat from core client for 30 sec - exiting 21:30:31 (5428): No heartbeat from core client for 30 sec - exiting 21:30:32 (5428): No heartbeat from core client for 30 sec - exiting 21:30:33 (5428): No heartbeat from core client for 30 sec - exiting 21:30:34 (5428): No heartbeat from core client for 30 sec - exiting 21:30:35 (5428): No heartbeat from core client for 30 sec - exiting 21:30:36 (5428): No heartbeat from core client for 30 sec - exiting 21:30:37 (5428): No heartbeat from core client for 30 sec - exiting 21:30:38 (5428): No heartbeat from core client for 30 sec - exiting 21:30:39 (5428): No heartbeat from core client for 30 sec - exiting 21:30:40 (5428): No heartbeat from core client for 30 sec - exiting 21:30:41 (5428): No heartbeat from core client for 30 sec - exiting 21:30:42 (5428): No heartbeat from core client for 30 sec - exiting 21:30:43 (5428): No heartbeat from core client for 30 sec - exiting 21:30:44 (5428): No heartbeat from core client for 30 sec - exiting 21:30:45 (5428): No heartbeat from core client for 30 sec - exiting 21:30:46 (5428): No heartbeat from core client for 30 sec - exiting 21:30:47 (5428): No heartbeat from core client for 30 sec - exiting 21:30:48 (5428): No heartbeat from core client for 30 sec - exiting 21:30:49 (5428): No heartbeat from core client for 30 sec - exiting 21:30:50 (5428): No heartbeat from core client for 30 sec - exiting 21:30:51 (5428): No heartbeat from core client for 30 sec - exiting 21:30:52 (5428): No heartbeat from core client for 30 sec - exiting 21:30:53 (5428): No heartbeat from core client for 30 sec - exiting 21:30:54 (5428): No heartbeat from core client for 30 sec - exiting 21:30:55 (5428): No heartbeat from core client for 30 sec - exiting 21:30:56 (5428): No heartbeat from core client for 30 sec - exiting 21:30:57 (5428): No heartbeat from core client for 30 sec - exiting 21:30:58 (5428): No heartbeat from core client for 30 sec - exiting 21:30:59 (5428): No heartbeat from core client for 30 sec - exiting 21:31:00 (5428): No heartbeat from core client for 30 sec - exiting 21:31:01 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5316, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3252, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5140, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5180, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5056, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5672, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zatsko.pj88c10 Error converting file to netcdf: dataout/zatsko.pi88c10 Error converting file to netcdf: dataout/zatsko.pf88c10 Error converting file to netcdf: dataout/zatska.ph88c10 Error converting file to netcdf: dataout/zatska.pg88c10 Error converting file to netcdf: dataout/zatska.pe88c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4736, iMonCtr=1 Model crash detected, will try to restart... 21:13:36 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:37 (4804): No heartbeat from core client for 30 sec - exiting 21:13:38 (4804): No heartbeat from core client for 30 sec - exiting 21:13:39 (4804): No heartbeat from core client for 30 sec - exiting 21:13:40 (4804): No heartbeat from core client for 30 sec - exiting 21:13:41 (4804): No heartbeat from core client for 30 sec - exiting 21:13:42 (4804): No heartbeat from core client for 30 sec - exiting 21:13:43 (4804): No heartbeat from core client for 30 sec - exiting 21:13:45 (4804): No heartbeat from core client for 30 sec - exiting 21:13:46 (4804): No heartbeat from core client for 30 sec - exiting 21:13:47 (4804): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4500, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5488, iMonCtr=1 Model crash detected, will try to restart... 21:01:32 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:33 (4952): No heartbeat from core client for 30 sec - exiting 21:01:34 (4952): No heartbeat from core client for 30 sec - exiting 21:01:35 (4952): No heartbeat from core client for 30 sec - exiting 21:01:36 (4952): No heartbeat from core client for 30 sec - exiting 21:01:37 (4952): No heartbeat from core client for 30 sec - exiting 21:01:38 (4952): No heartbeat from core client for 30 sec - exiting 21:01:39 (4952): No heartbeat from core client for 30 sec - exiting 21:01:40 (4952): No heartbeat from core client for 30 sec - exiting 21:01:41 (4952): No heartbeat from core client for 30 sec - exiting 21:01:42 (4952): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77805EAB read attempt to address 0x40C688FC Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zats_1880_40_008249362/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Dec 2012 20:46:22	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	259,200	414,385	1.5987
26 Dec 2012 15:40:30	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	233,280	372,226	1.5956
22 Dec 2012 08:24:20	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	207,360	330,407	1.5934
18 Dec 2012 07:16:23	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	181,440	289,096	1.5933
14 Dec 2012 17:34:43	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	155,520	247,787	1.5933
05 Dec 2012 21:40:54	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	129,600	205,935	1.5890
03 Dec 2012 14:25:38	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	103,680	165,238	1.5937
30 Nov 2012 19:41:32	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	77,760	124,162	1.5967
28 Nov 2012 12:22:35	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	51,840	82,493	1.5913
25 Nov 2012 07:50:15	1092573	15449143	hadcm3n_zats_1880_40_008249362_0	25,920	41,146	1.5874