Task 15842491

Name	hadcm3n_o19b_2140_40_008270443_3
Workunit	8425567
Created	14 Jun 2013, 7:15:14 UTC
Sent	14 Jun 2013, 7:27:19 UTC
Report deadline	13 Sep 2013, 14:54:30 UTC
Received	18 Jul 2013, 11:13:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1156302
Run time	12 days 7 hours 45 min 27 sec
CPU time	11 days 4 hours 29 min 12 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.03 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1044, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 12:30:56 AM No files match the supplied pattern. MainError: 12:30:56 AM No files match the supplied pattern. MainError: 10:23:02 PM No files match the supplied pattern. MainError: 10:23:02 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 05:26:28 AM No files match the supplied pattern. MainError: 05:26:28 AM No files match the supplied pattern. MainError: 04:49:49 PM No files match the supplied pattern. MainError: 04:49:49 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 04:42:25 AM No files match the supplied pattern. MainError: 04:42:25 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 02:54:08 AM No files match the supplied pattern. MainError: 02:54:08 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 10:13:42 PM No files match the supplied pattern. MainError: 10:13:42 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 01:21:07 PM No files match the supplied pattern. MainError: 01:21:07 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:42:17 PM No files match the supplied pattern. MainError: 08:42:17 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 07:29:36 PM No files match the supplied pattern. MainError: 07:29:36 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/o19bka.ph11c10 Error converting file to netcdf: dataout/o19bka.pg11c10 Error converting file to netcdf: dataout/o19bka.pe11c10 MainError: 03:38:25 PM No files match the supplied pattern. MainError: 03:38:25 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jul 2013 14:43:45	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	777,600	983,491	1.2648
23 Jul 2013 14:43:45	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	751,680	950,870	1.2650
23 Jul 2013 14:43:45	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	725,760	919,870	1.2675
23 Jul 2013 14:43:45	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	699,840	888,591	1.2697
23 Jul 2013 14:43:45	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	673,920	856,861	1.2715
11 Jul 2013 02:58:28	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	648,000	824,355	1.2722
10 Jul 2013 04:44:37	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	622,080	791,792	1.2728
09 Jul 2013 16:53:32	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	596,160	760,151	1.2751
09 Jul 2013 05:30:27	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	570,240	724,715	1.2709
07 Jul 2013 22:24:45	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	544,320	691,097	1.2697
07 Jul 2013 12:31:46	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	518,400	657,657	1.2686
07 Jul 2013 00:26:26	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	492,480	624,208	1.2675
06 Jul 2013 14:29:00	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	466,560	590,688	1.2660
06 Jul 2013 05:00:07	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	440,640	557,610	1.2655
04 Jul 2013 14:25:02	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	414,720	523,983	1.2635
04 Jul 2013 14:03:03	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	388,800	491,546	1.2643
03 Jul 2013 09:10:30	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	362,880	459,408	1.2660
03 Jul 2013 00:13:25	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	336,960	428,377	1.2713
02 Jul 2013 12:04:25	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	311,040	396,403	1.2744
02 Jul 2013 11:51:50	1156302	15842491	hadcm3n_o19b_2140_40_008270443_3	285,120	362,528	1.2715