Task 15604863

Name	hadcm3n_3j29_1940_40_008261089_3
Workunit	8416213
Created	11 Feb 2013, 13:46:51 UTC
Sent	11 Feb 2013, 13:46:54 UTC
Report deadline	13 May 2013, 21:14:05 UTC
Received	11 Mar 2013, 6:35:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1310449
Run time	15 days 3 hours 40 min 46 sec
CPU time	10 days 1 hours 50 min
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	2.61 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:35:29 (2592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:06:37 (8816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:06:38 (8816): No heartbeat from core client for 30 sec - exiting 21:06:39 (8816): No heartbeat from core client for 30 sec - exiting 21:06:40 (8816): No heartbeat from core client for 30 sec - exiting 21:06:41 (8816): No heartbeat from core client for 30 sec - exiting 21:06:42 (8816): No heartbeat from core client for 30 sec - exiting 21:06:43 (8816): No heartbeat from core client for 30 sec - exiting 21:06:44 (8816): No heartbeat from core client for 30 sec - exiting 21:06:45 (8816): No heartbeat from core client for 30 sec - exiting 21:06:46 (8816): No heartbeat from core client for 30 sec - exiting 21:06:47 (8816): No heartbeat from core client for 30 sec - exiting 21:06:48 (8816): No heartbeat from core client for 30 sec - exiting 21:06:49 (8816): No heartbeat from core client for 30 sec - exiting 21:06:50 (8816): No heartbeat from core client for 30 sec - exiting 21:06:51 (8816): No heartbeat from core client for 30 sec - exiting 21:06:52 (8816): No heartbeat from core client for 30 sec - exiting 21:06:53 (8816): No heartbeat from core client for 30 sec - exiting 21:06:54 (8816): No heartbeat from core client for 30 sec - exiting 21:16:29 (1948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:16:30 (1948): No heartbeat from core client for 30 sec - exiting 21:16:31 (1948): No heartbeat from core client for 30 sec - exiting 21:16:32 (1948): No heartbeat from core client for 30 sec - exiting 21:16:33 (1948): No heartbeat from core client for 30 sec - exiting 21:16:34 (1948): No heartbeat from core client for 30 sec - exiting 21:16:35 (1948): No heartbeat from core client for 30 sec - exiting 21:16:36 (1948): No heartbeat from core client for 30 sec - exiting 21:16:37 (1948): No heartbeat from core client for 30 sec - exiting 21:16:38 (1948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file I:\Boinc2/projects/climateprediction.net/hadcm3n_3j29_1940_40_008261089/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Mar 2013 20:27:38	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	570,240	860,413	1.5089
10 Mar 2013 07:56:18	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	544,320	822,167	1.5104
09 Mar 2013 11:53:00	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	518,400	784,623	1.5135
07 Mar 2013 15:25:33	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	492,480	747,434	1.5177
06 Mar 2013 13:55:14	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	466,560	708,697	1.5190
05 Mar 2013 12:56:58	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	440,640	669,507	1.5194
04 Mar 2013 07:33:08	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	414,720	631,834	1.5235
02 Mar 2013 13:55:21	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	388,800	592,288	1.5234
01 Mar 2013 22:33:45	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	362,880	554,442	1.5279
01 Mar 2013 09:07:34	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	336,960	516,923	1.5341
28 Feb 2013 06:02:04	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	311,040	477,944	1.5366
27 Feb 2013 09:01:53	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	285,120	439,414	1.5412
26 Feb 2013 09:50:39	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	259,200	398,672	1.5381
25 Feb 2013 09:40:02	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	233,280	358,616	1.5373
24 Feb 2013 09:00:27	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	207,360	317,121	1.5293
22 Feb 2013 03:05:45	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	181,440	277,795	1.5311
20 Feb 2013 23:47:38	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	155,520	238,913	1.5362
20 Feb 2013 01:06:31	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	129,600	203,184	1.5678
18 Feb 2013 15:50:42	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	103,680	161,452	1.5572
17 Feb 2013 00:20:17	1163938	15604863	hadcm3n_3j29_1940_40_008261089_3	77,760	122,125	1.5705