Task 15937210

Name	hadcm3n_48xu_1940_40_008300973_3
Workunit	8452108
Created	23 Aug 2013, 13:58:50 UTC
Sent	24 Aug 2013, 0:54:42 UTC
Report deadline	23 Nov 2013, 8:21:53 UTC
Received	11 Sep 2013, 22:53:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1271432
Run time	6 days 4 hours 21 min 47 sec
CPU time	5 days 12 hours 13 min 21 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	2.41 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:36:43 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:40:08 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:17 (3520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:02:08 (7160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:13:27 (1040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:18:55 (2676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:34:40 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:38:30 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:09 (6220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:38 (6588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:04 (4172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:40:59 (6284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:58:47 (6856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:39:35 (5936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:49 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/48xuko.pjf5c10 Error converting file to netcdf: dataout/48xuko.pif5c10 Error converting file to netcdf: dataout/48xuko.pff5c10 Error converting file to netcdf: dataout/48xuka.phf5c10 Error converting file to netcdf: dataout/48xuka.pgf5c10 Error converting file to netcdf: dataout/48xuka.pef5c10 Error converting file to netcdf: dataout/48xuka.pdf5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6572, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6572, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6572, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Sep 2013 12:25:29	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	414,720	470,155	1.1337
06 Sep 2013 12:13:15	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	388,800	440,195	1.1322
03 Sep 2013 11:06:24	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	362,880	411,354	1.1336
03 Sep 2013 02:48:02	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	336,960	381,965	1.1336
02 Sep 2013 06:53:21	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	311,040	351,710	1.1308
01 Sep 2013 21:54:51	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	285,120	320,690	1.1248
28 Aug 2013 12:04:21	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	259,200	289,661	1.1175
27 Aug 2013 09:00:43	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	233,280	261,018	1.1189
27 Aug 2013 00:08:02	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	207,360	231,810	1.1179
26 Aug 2013 15:50:54	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	181,440	202,869	1.1181
26 Aug 2013 07:34:04	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	155,520	174,368	1.1212
25 Aug 2013 22:40:58	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	129,600	145,043	1.1192
25 Aug 2013 12:48:50	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	103,680	116,054	1.1193
25 Aug 2013 04:22:13	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	77,760	86,817	1.1165
24 Aug 2013 23:21:19	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	51,840	57,541	1.1100
24 Aug 2013 14:29:30	1271432	15937210	hadcm3n_48xu_1940_40_008300973_3	25,920	29,382	1.1336