Task 11065687

Name	hadsm3dhet2_jsux_006600395_9
Workunit	6803768
Created	15 Mar 2010, 12:07:52 UTC
Sent	18 Jun 2010, 17:57:23 UTC
Report deadline	31 May 2011, 23:17:23 UTC
Received	14 Sep 2010, 8:46:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1374414
Run time	8 days 2 hours 49 min 5 sec
CPU time	5 days 16 hours 42 min 59 sec
Validate state	Invalid
Credit	2,183.35
Device peak FLOPS	1.74 GFLOPS
Application version	UK Met Office HadSM3 Slab Model v6.08 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25888, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25888, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=31691, selfPID=31691, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16801, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16801, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=18352, selfPID=18352, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18420, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18420, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=18813, selfPID=18813, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9704, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9704, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=10819, selfPID=10819, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24270, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24270, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=25008, selfPID=25008, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19933, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19933, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=20873, selfPID=20873, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20893, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20893, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=22293, selfPID=22293, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23407, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23407, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=23975, selfPID=23975, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30089, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30089, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=30840, selfPID=30840, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7104, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7104, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=7680, selfPID=7680, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21791, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21791, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=22906, selfPID=22906, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31560, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31560, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6692, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6692, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=8087, selfPID=8087, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21967, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21967, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=22897, selfPID=22897, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19911, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19911, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=20302, selfPID=20302, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26849, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26849, iMonCtr=1 Model crash detected, will try to restart... Can't acquire lockfile - exiting CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28513, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28513, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=30113, selfPID=30113, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=30113, selfPID=30113, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28513, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30135, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30135, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=31773, selfPID=31773, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19150, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19150, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=19998, selfPID=19998, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6024, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6024, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=6453, selfPID=6453, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=874, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=874, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=2416, selfPID=2416, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8830, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8830, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=10124, selfPID=10124, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27955, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27955, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=28597, selfPID=28597, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17579, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17579, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=18221, selfPID=18221, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31468, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31468, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=32332, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=32332, iMonCtr=1 Model crash detected, will try to restart... Can't acquire lockfile - exiting CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=32332, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=32332, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=3159, selfPID=3159, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=3159, selfPID=3159, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26260, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26260, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=885, selfPID=885, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7301, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7301, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=8831, selfPID=8831, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8847, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8847, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=16712, selfPID=16712, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16728, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16728, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=24576, selfPID=24576, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24592, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24592, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=6049, selfPID=6049, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=6280, selfPID=6280, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10986, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24982, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=26579, selfPID=26579, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26611, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=7010, selfPID=7010, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22900, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22900, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=23763, selfPID=23763, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=322, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=322, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=764, selfPID=764, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7989, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8999, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10609, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12255, iMonCtr=1 Model crash detected, will try to restart... Can't acquire lockfile - exiting CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12255, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15516, iMonCtr=1 Model crash detected, will try to restart... Can't acquire lockfile - exiting CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15516, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18913, iMonCtr=1 Model crash detected, will try to restart... Can't acquire lockfile - exiting CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18913, iMonCtr=1 Model crash detected, will try to restart... Can't acquire lockfile - exiting CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20541, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23922, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25694, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27306, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28947, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30558, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=517, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=517, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=13184, selfPID=13184, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=13211, selfPID=13211, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=3811, selfPID=3811, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=24761, selfPID=24761, iMonCtr=1 CPDN process is not running, exiting, bRetVal = 1, checkPID=31575, selfPID=31575, iMonCtr=1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: Ҋ BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: Ҋ BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: Ҋ BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: Ҋ BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: Ҋ BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: Ҋ Sorry, too many model crashes! :-( called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Sep 2010 04:01:37	496709	11065687	hadsm3dhet2_jsux_006600395_9	237,644	482,642	2.0309
12 Sep 2010 18:02:29	496709	11065687	hadsm3dhet2_jsux_006600395_9	226,842	460,547	2.0303
12 Sep 2010 10:07:49	496709	11065687	hadsm3dhet2_jsux_006600395_9	216,040	439,492	2.0343
09 Sep 2010 22:20:04	496709	11065687	hadsm3dhet2_jsux_006600395_9	205,238	418,471	2.0390
09 Sep 2010 16:19:39	496709	11065687	hadsm3dhet2_jsux_006600395_9	194,436	397,371	2.0437
08 Sep 2010 23:47:33	496709	11065687	hadsm3dhet2_jsux_006600395_9	183,634	375,899	2.0470
08 Sep 2010 17:06:03	496709	11065687	hadsm3dhet2_jsux_006600395_9	172,832	354,747	2.0526
08 Sep 2010 10:25:25	496709	11065687	hadsm3dhet2_jsux_006600395_9	162,030	333,777	2.0600
08 Sep 2010 04:20:52	496709	11065687	hadsm3dhet2_jsux_006600395_9	151,228	312,498	2.0664
07 Sep 2010 22:30:19	496709	11065687	hadsm3dhet2_jsux_006600395_9	140,426	291,699	2.0772
07 Sep 2010 08:41:21	496709	11065687	hadsm3dhet2_jsux_006600395_9	129,624	270,295	2.0852
06 Sep 2010 14:41:57	496709	11065687	hadsm3dhet2_jsux_006600395_9	118,822	248,982	2.0954
27 Aug 2010 08:17:21	496709	11065687	hadsm3dhet2_jsux_006600395_9	108,020	227,091	2.1023
27 Aug 2010 02:24:34	496709	11065687	hadsm3dhet2_jsux_006600395_9	97,218	205,675	2.1156
26 Aug 2010 04:53:40	496709	11065687	hadsm3dhet2_jsux_006600395_9	86,416	181,956	2.1056
14 Aug 2010 09:15:41	496709	11065687	hadsm3dhet2_jsux_006600395_9	75,614	160,476	2.1223
14 Aug 2010 01:38:22	496709	11065687	hadsm3dhet2_jsux_006600395_9	64,812	139,191	2.1476
12 Aug 2010 02:47:35	496709	11065687	hadsm3dhet2_jsux_006600395_9	54,010	118,141	2.1874
11 Aug 2010 15:36:11	496709	11065687	hadsm3dhet2_jsux_006600395_9	43,208	96,458	2.2324
10 Aug 2010 22:40:33	496709	11065687	hadsm3dhet2_jsux_006600395_9	32,406	73,364	2.2639