Task 13464377

Name	hadcm3n_ymc0_1900_40_007361802_2
Workunit	7559232
Created	6 Oct 2011, 17:38:11 UTC
Sent	6 Oct 2011, 17:38:23 UTC
Report deadline	6 Jan 2012, 1:05:34 UTC
Received	11 Dec 2011, 0:45:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	830038
Run time	10 days 22 hours 56 min 46 sec
CPU time	10 days 2 hours 54 min 26 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:34:43 (16160): No heartbeat from core client for 30 sec - exiting 02:34:44 (16160): No heartbeat from core client for 30 sec - exiting 02:34:45 (16160): No heartbeat from core client for 30 sec - exiting 02:34:46 (16160): No heartbeat from core client for 30 sec - exiting 02:34:47 (16160): No heartbeat from core client for 30 sec - exiting 02:34:48 (16160): No heartbeat from core client for 30 sec - exiting 02:34:49 (16160): No heartbeat from core client for 30 sec - exiting 02:34:50 (16160): No heartbeat from core client for 30 sec - exiting 02:34:51 (16160): No heartbeat from core client for 30 sec - exiting 02:34:52 (16160): No heartbeat from core client for 30 sec - exiting 02:34:53 (16160): No heartbeat from core client for 30 sec - exiting 02:34:54 (16160): No heartbeat from core client for 30 sec - exiting 02:34:55 (16160): No heartbeat from core client for 30 sec - exiting 02:34:56 (16160): No heartbeat from core client for 30 sec - exiting 02:34:57 (16160): No heartbeat from core client for 30 sec - exiting 02:34:58 (16160): No heartbeat from core client for 30 sec - exiting 02:34:59 (16160): No heartbeat from core client for 30 sec - exiting 02:35:00 (16160): No heartbeat from core client for 30 sec - exiting 02:35:01 (16160): No heartbeat from core client for 30 sec - exiting 02:35:02 (16160): No heartbeat from core client for 30 sec - exiting 02:35:03 (16160): No heartbeat from core client for 30 sec - exiting 02:35:04 (16160): No heartbeat from core client for 30 sec - exiting 02:35:05 (16160): No heartbeat from core client for 30 sec - exiting 02:35:06 (16160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:48:40 (4936): No heartbeat from core client for 30 sec - exiting 22:48:42 (4936): No heartbeat from core client for 30 sec - exiting 22:48:43 (4936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:34:27 (8680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:28:37 (3092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:05:31 (8736): No heartbeat from core client for 30 sec - exiting 04:05:32 (8736): No heartbeat from core client for 30 sec - exiting 04:05:33 (8736): No heartbeat from core client for 30 sec - exiting 04:05:34 (8736): No heartbeat from core client for 30 sec - exiting 04:05:35 (8736): No heartbeat from core client for 30 sec - exiting 04:05:36 (8736): No heartbeat from core client for 30 sec - exiting 04:05:37 (8736): No heartbeat from core client for 30 sec - exiting 04:05:38 (8736): No heartbeat from core client for 30 sec - exiting 04:05:39 (8736): No heartbeat from core client for 30 sec - exiting 04:05:40 (8736): No heartbeat from core client for 30 sec - exiting 04:05:41 (8736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ymc0ko.pjb5c10 Error converting file to netcdf: dataout/ymc0ko.pib5c10 Error converting file to netcdf: dataout/ymc0ko.pfb5c10 Error converting file to netcdf: dataout/ymc0ka.phb5c10 Error converting file to netcdf: dataout/ymc0ka.pgb5c10 Error converting file to netcdf: dataout/ymc0ka.peb5c10 Error converting file to netcdf: dataout/ymc0ka.pdb5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=416, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Dec 2011 03:06:18	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	440,640	851,655	1.9328
07 Dec 2011 07:10:34	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	414,720	796,192	1.9198
05 Dec 2011 05:10:06	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	388,800	743,507	1.9123
15 Nov 2011 16:59:24	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	362,880	687,397	1.8943
15 Nov 2011 16:59:24	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	336,960	638,274	1.8942
10 Nov 2011 04:34:06	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	311,040	588,902	1.8933
08 Nov 2011 02:00:23	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	285,120	539,185	1.8911
05 Nov 2011 05:44:28	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	259,200	491,100	1.8947
03 Nov 2011 06:02:34	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	233,280	436,981	1.8732
17 Oct 2011 06:38:59	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	207,360	382,280	1.8436
15 Oct 2011 04:58:54	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	181,440	332,823	1.8343
14 Oct 2011 13:13:22	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	155,520	283,715	1.8243
13 Oct 2011 23:51:11	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	129,600	237,917	1.8358
12 Oct 2011 05:22:08	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	103,680	190,577	1.8381
11 Oct 2011 02:42:03	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	77,760	138,547	1.7817
09 Oct 2011 03:32:13	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	51,840	93,430	1.8023
07 Oct 2011 07:14:02	830038	13464377	hadcm3n_ymc0_1900_40_007361802_2	25,920	46,046	1.7765