Task 15668979

Name	hadcm3n_zcdn_1920_40_008270911_3
Workunit	8426035
Created	15 Mar 2013, 23:42:36 UTC
Sent	15 Mar 2013, 23:42:48 UTC
Report deadline	15 Jun 2013, 7:09:59 UTC
Received	28 Mar 2013, 4:36:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1264094
Run time	11 days 4 hours 4 min 4 sec
CPU time	3 days 18 hours 38 min 11 sec
Validate state	Invalid
Credit	10,264.32
Device peak FLOPS	3.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 16:13:07 (4251): No heartbeat from core client for 30 sec - exiting 16:13:08 (4251): No heartbeat from core client for 30 sec - exiting 16:13:09 (4251): No heartbeat from core client for 30 sec - exiting 16:13:10 (4251): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:43:34 (3933): No heartbeat from core client for 30 sec - exiting 16:43:36 (3933): No heartbeat from core client for 30 sec - exiting 16:43:37 (3933): No heartbeat from core client for 30 sec - exiting 16:43:38 (3933): No heartbeat from core client for 30 sec - exiting 16:43:39 (3933): No heartbeat from core client for 30 sec - exiting 16:43:40 (3933): No heartbeat from core client for 30 sec - exiting 16:43:41 (3933): No heartbeat from core client for 30 sec - exiting 16:43:42 (3933): No heartbeat from core client for 30 sec - exiting 16:43:43 (3933): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:48:29 (3847): No heartbeat from core client for 30 sec - exiting 14:48:30 (3847): No heartbeat from core client for 30 sec - exiting 14:48:31 (3847): No heartbeat from core client for 30 sec - exiting 14:48:32 (3847): No heartbeat from core client for 30 sec - exiting 14:48:35 (3847): No heartbeat from core client for 30 sec - exiting 14:48:36 (3847): No heartbeat from core client for 30 sec - exiting 14:48:37 (3847): No heartbeat from core client for 30 sec - exiting 14:48:38 (3847): No heartbeat from core client for 30 sec - exiting 14:48:39 (3847): No heartbeat from core client for 30 sec - exiting 14:48:40 (3847): No heartbeat from core client for 30 sec - exiting 14:48:41 (3847): No heartbeat from core client for 30 sec - exiting 14:48:42 (3847): No heartbeat from core client for 30 sec - exiting 14:48:43 (3847): No heartbeat from core client for 30 sec - exiting 14:48:44 (3847): No heartbeat from core client for 30 sec - exiting 14:49:15 (3847): No heartbeat from core client for 30 sec - exiting 14:49:16 (3847): No heartbeat from core client for 30 sec - exiting 14:49:17 (3847): No heartbeat from core client for 30 sec - exiting 14:49:18 (3847): No heartbeat from core client for 30 sec - exiting 14:49:19 (3847): No heartbeat from core client for 30 sec - exiting 14:49:20 (3847): No heartbeat from core client for 30 sec - exiting 14:49:21 (3847): No heartbeat from core client for 30 sec - exiting 14:49:22 (3847): No heartbeat from core client for 30 sec - exiting 14:49:23 (3847): No heartbeat from core client for 30 sec - exiting 14:49:24 (3847): No heartbeat from core client for 30 sec - exiting 14:49:25 (3847): No heartbeat from core client for 30 sec - exiting 14:49:26 (3847): No heartbeat from core client for 30 sec - exiting 14:49:27 (3847): No heartbeat from core client for 30 sec - exiting 14:49:28 (3847): No heartbeat from core client for 30 sec - exiting 14:49:29 (3847): No heartbeat from core client for 30 sec - exiting 14:49:30 (3847): No heartbeat from core client for 30 sec - exiting 14:49:31 (3847): No heartbeat from core client for 30 sec - exiting 14:49:32 (3847): No heartbeat from core client for 30 sec - exiting 14:49:33 (3847): No heartbeat from core client for 30 sec - exiting 14:49:34 (3847): No heartbeat from core client for 30 sec - exiting 14:49:35 (3847): No heartbeat from core client for 30 sec - exiting 14:49:36 (3847): No heartbeat from core client for 30 sec - exiting 14:49:37 (3847): No heartbeat from core client for 30 sec - exiting 14:49:38 (3847): No heartbeat from core client for 30 sec - exiting 14:49:39 (3847): No heartbeat from core client for 30 sec - exiting 14:49:40 (3847): No heartbeat from core client for 30 sec - exiting 14:49:41 (3847): No heartbeat from core client for 30 sec - exiting 14:49:42 (3847): No heartbeat from core client for 30 sec - exiting 14:49:43 (3847): No heartbeat from core client for 30 sec - exiting 14:49:44 (3847): No heartbeat from core client for 30 sec - exiting 14:49:45 (3847): No heartbeat from core client for 30 sec - exiting 14:49:46 (3847): No heartbeat from core client for 30 sec - exiting 14:49:47 (3847): No heartbeat from core client for 30 sec - exiting 14:49:48 (3847): No heartbeat from core client for 30 sec - exiting 14:49:49 (3847): No heartbeat from core client for 30 sec - exiting 14:49:50 (3847): No heartbeat from core client for 30 sec - exiting 14:49:51 (3847): No heartbeat from core client for 30 sec - exiting 14:49:52 (3847): No heartbeat from core client for 30 sec - exiting 14:49:53 (3847): No heartbeat from core client for 30 sec - exiting 14:49:54 (3847): No heartbeat from core client for 30 sec - exiting 14:49:55 (3847): No heartbeat from core client for 30 sec - exiting 14:49:56 (3847): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:51:41 (4779): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:40:12 (3871): No heartbeat from core client for 30 sec - exiting 17:40:13 (3871): No heartbeat from core client for 30 sec - exiting 17:40:14 (3871): No heartbeat from core client for 30 sec - exiting 17:40:15 (3871): No heartbeat from core client for 30 sec - exiting 17:40:16 (3871): No heartbeat from core client for 30 sec - exiting 17:40:17 (3871): No heartbeat from core client for 30 sec - exiting 17:40:18 (3871): No heartbeat from core client for 30 sec - exiting 17:40:19 (3871): No heartbeat from core client for 30 sec - exiting 17:40:20 (3871): No heartbeat from core client for 30 sec - exiting 17:40:21 (3871): No heartbeat from core client for 30 sec - exiting 17:40:22 (3871): No heartbeat from core client for 30 sec - exiting 17:40:23 (3871): No heartbeat from core client for 30 sec - exiting 17:40:24 (3871): No heartbeat from core client for 30 sec - exiting 17:40:25 (3871): No heartbeat from core client for 30 sec - exiting 17:40:26 (3871): No heartbeat from core client for 30 sec - exiting 17:40:27 (3871): No heartbeat from core client for 30 sec - exiting 17:40:28 (3871): No heartbeat from core client for 30 sec - exiting 17:40:29 (3871): No heartbeat from core client for 30 sec - exiting 17:40:30 (3871): No heartbeat from core client for 30 sec - exiting 17:40:31 (3871): No heartbeat from core client for 30 sec - exiting 17:40:32 (3871): No heartbeat from core client for 30 sec - exiting 17:40:33 (3871): No heartbeat from core client for 30 sec - exiting 17:40:34 (3871): No heartbeat from core client for 30 sec - exiting 17:40:35 (3871): No heartbeat from core client for 30 sec - exiting 17:40:36 (3871): No heartbeat from core client for 30 sec - exiting 17:40:37 (3871): No heartbeat from core client for 30 sec - exiting 17:40:38 (3871): No heartbeat from core client for 30 sec - exiting 17:40:39 (3871): No heartbeat from core client for 30 sec - exiting 17:40:40 (3871): No heartbeat from core client for 30 sec - exiting 17:40:41 (3871): No heartbeat from core client for 30 sec - exiting 17:40:42 (3871): No heartbeat from core client for 30 sec - exiting 17:40:43 (3871): No heartbeat from core client for 30 sec - exiting 17:40:45 (3871): No heartbeat from core client for 30 sec - exiting 17:40:46 (3871): No heartbeat from core client for 30 sec - exiting 17:40:47 (3871): No heartbeat from core client for 30 sec - exiting 17:40:48 (3871): No heartbeat from core client for 30 sec - exiting 17:40:49 (3871): No heartbeat from core client for 30 sec - exiting 17:40:50 (3871): No heartbeat from core client for 30 sec - exiting 17:40:51 (3871): No heartbeat from core client for 30 sec - exiting 17:40:52 (3871): No heartbeat from core client for 30 sec - exiting 17:40:53 (3871): No heartbeat from core client for 30 sec - exiting 17:40:54 (3871): No heartbeat from core client for 30 sec - exiting 17:40:55 (3871): No heartbeat from core client for 30 sec - exiting 17:40:56 (3871): No heartbeat from core client for 30 sec - exiting 17:40:57 (3871): No heartbeat from core client for 30 sec - exiting 17:40:58 (3871): No heartbeat from core client for 30 sec - exiting 17:40:59 (3871): No heartbeat from core client for 30 sec - exiting 17:41:00 (3871): No heartbeat from core client for 30 sec - exiting 17:41:01 (3871): No heartbeat from core client for 30 sec - exiting 17:41:02 (3871): No heartbeat from core client for 30 sec - exiting 17:41:03 (3871): No heartbeat from core client for 30 sec - exiting 17:41:04 (3871): No heartbeat from core client for 30 sec - exiting 17:41:05 (3871): No heartbeat from core client for 30 sec - exiting 17:41:06 (3871): No heartbeat from core client for 30 sec - exiting 17:41:07 (3871): No heartbeat from core client for 30 sec - exiting 17:41:08 (3871): No heartbeat from core client for 30 sec - exiting 17:41:09 (3871): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_zcdn_1920_40_008270911/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822E07C Unknown Unknown Unknown hadcm3n_um_6.07_i 0818A767 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818D749 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75A7E46 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4351, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_zcdn_1920_40_008270911/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822F5BB Unknown Unknown Unknown hadcm3n_um_6.07_i 0818A767 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818D749 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75C3E46 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4351, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_zcdn_1920_40_008270911/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822F5BController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4351, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Mar 2013 23:20:14	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	855,360	895,356	1.0468
27 Mar 2013 14:50:31	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	829,440	868,228	1.0468
27 Mar 2013 06:46:37	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	803,520	841,104	1.0468
26 Mar 2013 22:52:02	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	777,600	813,996	1.0468
26 Mar 2013 14:30:14	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	751,680	786,877	1.0468
26 Mar 2013 06:15:06	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	725,760	759,744	1.0468
25 Mar 2013 22:01:38	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	699,840	732,647	1.0469
25 Mar 2013 13:50:08	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	673,920	705,561	1.0470
25 Mar 2013 05:36:27	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	648,000	678,481	1.0470
24 Mar 2013 21:28:34	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	622,080	651,379	1.0471
24 Mar 2013 13:23:29	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	596,160	624,262	1.0471
24 Mar 2013 05:41:12	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	570,240	597,138	1.0472
23 Mar 2013 21:22:35	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	544,320	570,033	1.0472
23 Mar 2013 13:20:02	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	518,400	542,910	1.0473
23 Mar 2013 05:32:06	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	492,480	515,835	1.0474
22 Mar 2013 21:27:11	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	466,560	488,760	1.0476
22 Mar 2013 12:45:15	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	440,640	461,776	1.0480
22 Mar 2013 04:57:48	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	414,720	434,692	1.0482
21 Mar 2013 21:09:20	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	388,800	407,602	1.0484
21 Mar 2013 13:25:22	1264094	15668979	hadcm3n_zcdn_1920_40_008270911_3	362,880	380,588	1.0488