Task 14793393

Name	hadcm3n_200d_1980_40_008000436_2
Workunit	8155550
Created	16 Jun 2012, 1:06:43 UTC
Sent	16 Jun 2012, 1:06:49 UTC
Report deadline	15 Sep 2012, 8:34:00 UTC
Received	18 Jul 2012, 8:08:35 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1211345
Run time	16 days 17 hours 3 min 59 sec
CPU time	16 days 17 hours 3 min 59 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	2.95 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.10.45</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... 10:43:09 (3392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:43:10 (3392): No heartbeat from core client for 30 sec - exiting 10:43:11 (3392): No heartbeat from core client for 30 sec - exiting 10:43:13 (3392): No heartbeat from core client for 30 sec - exiting 10:43:14 (3392): No heartbeat from core client for 30 sec - exiting 10:43:15 (3392): No heartbeat from core client for 30 sec - exiting 10:43:16 (3392): No heartbeat from core client for 30 sec - exiting 10:43:17 (3392): No heartbeat from core client for 30 sec - exiting 10:43:18 (3392): No heartbeat from core client for 30 sec - exiting 10:43:19 (3392): No heartbeat from core client for 30 sec - exiting 10:43:20 (3392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 02:36:47 (2380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:37:20 (2848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:21 (2848): No heartbeat from core client for 30 sec - exiting 06:03:14 (1116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Skipping gmts_generator due to netcdf error 28 - No space left on device Controller:: CPDN process is not running, exiting, bRetVal = 1BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/200dko.pjk4c10 Error converting file to netcdf: dataout/200dko.pik4c10 Error converting file to netcdf: dataout/200dko.pfk4c10 Error converting file to netcdf: dataout/200dka.phk4c10 Error converting file to netcdf: dataout/200dka.pgk4c10 Error converting file to netcdf: dataout/200dka.pek4c10 Error converting file to netcdf: dataout/200dka.pdk4c10 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 05:41:27 (1124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:28 (1124): No heartbeat from core client for 30 sec - exiting 05:41:29 (1124): No heartbeat from core client for 30 sec - exiting 05:41:30 (1124): No heartbeat from core client for 30 sec - exiting 05:41:31 (1124): No heartbeat from core client for 30 sec - exiting 05:41:32 (1124): No heartbeat from core client for 30 sec - exiting 05:41:33 (1124): No heartbeat from core client for 30 sec - exiting 05:41:34 (1124): No heartbeat from core client for 30 sec - exiting 05:41:35 (1124): No heartbeat from core client for 30 sec - exiting 05:41:37 (1124): No heartbeat from core client for 30 sec - exiting 05:41:38 (1124): No heartbeat from core client for 30 sec - exiting 05:41:39 (1124): No heartbeat from core client for 30 sec - exiting 05:41:40 (1124): No heartbeat from core client for 30 sec - exiting 05:41:41 (1124): No heartbeat from core client for 30 sec - exiting 05:41:42 (1124): No heartbeat from core client for 30 sec - exiting 05:41:43 (1124): No heartbeat from core client for 30 sec - exiting 05:41:44 (1124): No heartbeat from core client for 30 sec - exiting 05:41:45 (1124): No heartbeat from core client for 30 sec - exiting 05:41:46 (1124): No heartbeat from core client for 30 sec - exiting 05:41:48 (1124): No heartbeat from core client for 30 sec - exiting 05:41:49 (1124): No heartbeat from core client for 30 sec - exiting 05:41:50 (1124): No heartbeat from core client for 30 sec - exiting 05:41:51 (1124): No heartbeat from core client for 30 sec - exiting 05:41:52 (1124): No heartbeat from core client for 30 sec - exiting 05:41:53 (1124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 16:37:14 (1664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:37:16 (1664): No heartbeat from core client for 30 sec - exiting 16:37:17 (1664): No heartbeat from core client for 30 sec - exiting 16:37:18 (1664): No heartbeat from core client for 30 sec - exiting 16:37:19 (1664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Jul 2012 06:53:46	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	1,036,800	1,443,768	1.3925
17 Jul 2012 13:26:16	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	1,010,880	1,405,265	1.3901
16 Jul 2012 22:12:50	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	984,960	1,366,023	1.3869
16 Jul 2012 06:16:00	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	959,040	1,326,445	1.3831
15 Jul 2012 14:08:17	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	933,120	1,287,553	1.3798
14 Jul 2012 20:29:15	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	907,200	1,249,274	1.3771
14 Jul 2012 01:28:37	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	881,280	1,211,057	1.3742
13 Jul 2012 07:19:14	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	855,360	1,172,581	1.3709
12 Jul 2012 14:37:57	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	829,440	1,133,740	1.3669
11 Jul 2012 20:53:37	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	803,520	1,095,327	1.3632
11 Jul 2012 03:07:30	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	777,600	1,056,005	1.3580
10 Jul 2012 10:25:48	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	751,680	1,016,519	1.3523
09 Jul 2012 17:53:20	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	725,760	977,180	1.3464
09 Jul 2012 01:08:29	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	699,840	938,074	1.3404
08 Jul 2012 08:59:58	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	673,920	898,914	1.3339
07 Jul 2012 17:10:14	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	648,000	860,857	1.3285
07 Jul 2012 08:00:08	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	622,080	1,004,541	1.6148
06 Jul 2012 12:59:56	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	596,160	965,006	1.6187
05 Jul 2012 20:41:48	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	570,240	924,107	1.6206
05 Jul 2012 05:02:47	1211345	14793393	hadcm3n_200d_1980_40_008000436_2	544,320	882,413	1.6211