Task 13106357

Name	hadcm3n_ye6q_1900_40_007351244_0
Workunit	7548674
Created	6 Jul 2011, 14:12:08 UTC
Sent	16 Jul 2011, 11:01:45 UTC
Report deadline	15 Oct 2011, 18:28:56 UTC
Received	28 Aug 2011, 7:34:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1133662
Run time	6 days 1 hours 36 min 27 sec
CPU time	5 days 11 hours 48 min 34 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	3.13 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1952, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:08:53 (5628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4108, selfPID=4108, iMonCtr=1 20:18:05 (4276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:50 (2196): No heartbeat from core client for 30 sec - exiting 08:54:51 (2196): No heartbeat from core client for 30 sec - exiting 08:54:52 (2196): No heartbeat from core client for 30 sec - exiting 08:54:53 (2196): No heartbeat from core client for 30 sec - exiting 08:54:54 (2196): No heartbeat from core client for 30 sec - exiting 08:54:56 (2196): No heartbeat from core client for 30 sec - exiting 08:54:57 (2196): No heartbeat from core client for 30 sec - exiting 08:54:58 (2196): No heartbeat from core client for 30 sec - exiting 08:54:59 (2196): No heartbeat from core client for 30 sec - exiting 08:55:00 (2196): No heartbeat from core client for 30 sec - exiting 08:55:01 (2196): No heartbeat from core client for 30 sec - exiting 08:55:02 (2196): No heartbeat from core client for 30 sec - exiting 08:55:03 (2196): No heartbeat from core client for 30 sec - exiting 08:55:04 (2196): No heartbeat from core client for 30 sec - exiting 08:55:05 (2196): No heartbeat from core client for 30 sec - exiting 08:55:06 (2196): No heartbeat from core client for 30 sec - exiting 08:55:08 (2196): No heartbeat from core client for 30 sec - exiting 08:55:09 (2196): No heartbeat from core client for 30 sec - exiting 08:55:10 (2196): No heartbeat from core client for 30 sec - exiting 08:55:11 (2196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 18:01:45 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:31 (4916): No heartbeat from core client for 30 sec - exiting 09:54:32 (4916): No heartbeat from core client for 30 sec - exiting 09:54:33 (4916): No heartbeat from core client for 30 sec - exiting 09:54:34 (4916): No heartbeat from core client for 30 sec - exiting 09:54:35 (4916): No heartbeat from core client for 30 sec - exiting 09:54:36 (4916): No heartbeat from core client for 30 sec - exiting 09:54:37 (4916): No heartbeat from core client for 30 sec - exiting 09:54:38 (4916): No heartbeat from core client for 30 sec - exiting 09:54:39 (4916): No heartbeat from core client for 30 sec - exiting 09:54:41 (4916): No heartbeat from core client for 30 sec - exiting 09:54:42 (4916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:43 (4916): No heartbeat from core client for 30 sec - exiting BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 09:58:06 (4432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2011 10:35:49	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	285,120	438,908	1.5394
07 Aug 2011 07:25:29	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	259,200	402,577	1.5532
06 Aug 2011 19:52:37	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	233,280	365,641	1.5674
04 Aug 2011 16:28:48	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	207,360	325,095	1.5678
02 Aug 2011 14:06:42	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	181,440	284,778	1.5695
31 Jul 2011 04:08:39	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	155,520	243,054	1.5628
30 Jul 2011 16:14:00	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	129,600	203,033	1.5666
30 Jul 2011 04:04:49	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	103,680	163,174	1.5738
27 Jul 2011 11:05:26	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	77,760	122,027	1.5693
25 Jul 2011 21:03:22	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	51,840	81,180	1.5660
25 Jul 2011 20:31:54	1133662	13106357	hadcm3n_ye6q_1900_40_007351244_0	25,920	40,452	1.5606