Task 22180862

Name	hadcm3s_18ra_200012_168_926_012127866_0
Workunit	12127866
Created	4 Jan 2022, 11:33:38 UTC
Sent	5 Jan 2022, 3:04:19 UTC
Report deadline	18 Dec 2022, 8:24:19 UTC
Received	30 Jan 2022, 5:39:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1503934
Run time	21 days 0 hours 32 min 29 sec
CPU time	16 days 10 hours 10 min 23 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	3.74 GFLOPS
Application version	UK Met Office HadCM3 short v8.36 i686-apple-darwin
Peak working set size	153.10 MB
Peak swap size	1,134.77 MB
Peak disk usage	55.84 MB
Stderr	<core_client_version>7.16.11</core_client_version> <![CDATA[ <message> process got signal 22</message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:10:23 (18230): BOINC client no longer exists - exiting 19:10:23 (18230): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:39 (11013): BOINC client no longer exists - exiting 21:57:39 (11013): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:14:14 (76381): BOINC client no longer exists - exiting 15:14:14 (76381): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:42:29 (98838): BOINC client no longer exists - exiting 11:42:29 (98838): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:15:46 (3769): BOINC client no longer exists - exiting 16:15:46 (3769): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:54 (98605): BOINC client no longer exists - exiting 16:30:54 (98605): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... OPEN: File Creation Failed: Operation timed out OPEN: Unable to Open File dataout/18rako.dal1co0 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... OPEN: File Creation Failed: Operation timed out OPEN: Unable to Open File dataout/18raka.dal2130 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy Model crashed: Error opening shortwave spectral file. IOSTAT= 121 tmp/pipe_dummy Model crashed: GETPPX: ERROR OPENING USER PPXREF tmp/pipe_dummy OPEN: File Open Failed: Operation timed out OPEN: Unable to Open File dataout/18rako.pal2c10 for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... OPEN: File Creation Failed: Operation timed out OPEN: Unable to Open File dataout/18rako.dal21b0 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy SIGABRT: abort called built using BOINC library version 7.7.0 Machine type Intel 80486 (32-bit executable) Sun Jan 30 02:43:14 2022 0 hadcm3s_um_8.36_i686-apple-darwin 0x00572b85 PrintBacktrace + 1009 1 hadcm3s_um_8.36_i686-apple-darwin 0x00566300 boinc_catch_signal + 241 2 libsystem_platform.dylib 0xa17f6e5b _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 libsystem_c.dylib 0xa167ca34 abort + 141 5 hadcm3s_um_8.36_i686-apple-darwin 0x001b3aa0 inactr + 4928 6 hadcm3s_um_8.36_i686-apple-darwin 0x003c446c stash_proc + 3068 7 hadcm3s_um_8.36_i686-apple-darwin 0x0049b36e um_shell + 3806 8 hadcm3s_um_8.36_i686-apple-darwin 0x004a8d78 main + 1120 9 hadcm3s_um_8.36_i686-apple-darwin 0x000018fd _start + 212 10 hadcm3s_um_8.36_i686-apple-darwin 0x00001828 start + 40 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbff91c5c edx: 0xa171491e edi: 0xbff91cc8 esi: 0x00000000 ebp: 0xbff91c98 esp: 0xbff91c5c ss: 0x00000023 efl: 0x00000292 eip: 0xa171491e cs: 0x0000000b ds: 0x00000023 es: 0x00000023 fs: 0x00000000 gs: 0x0000000f Binary Images Description: 0x1000 - 0x5eafff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3s_um_8.36_i686-apple-darwin 0xa0016000 - 0xa0017fff /usr/lib/libSystem.B.dylib 0xa01ad000 - 0xa0203fff /usr/lib/libc++.1.dylib 0xa0204000 - 0xa0227fff /usr/lib/libc++abi.dylib 0xa0d10000 - 0xa10d4fff /usr/lib/libobjc.A.dylib 0xa1375000 - 0xa13c5fff /usr/lib/libstdc++.6.dylib 0xa1534000 - 0xa1538fff /usr/lib/system/libcache.dylib 0xa1539000 - 0xa1543fff /usr/lib/system/libcommonCrypto.dylib 0xa1544000 - 0xa1549fff /usr/lib/system/libcompiler_rt.dylib 0xa154a000 - 0xa1552fff /usr/lib/system/libcopyfile.dylib 0xa1553000 - 0xa15b9fff /usr/lib/system/libcorecrypto.dylib 0xa15ba000 - 0xa15e7fff /usr/lib/system/libdispatch.dylib 0xa15e8000 - 0xa15edfff /usr/lib/system/libdyld.dylib 0xa15ee000 - 0xa15eefff /usr/lib/system/libkeymgr.dylib 0xa15fc000 - 0xa15fcfff /usr/lib/system/liblaunch.dylib 0xa15fd000 - 0xa1603fff /usr/lib/system/libmacho.dylib 0xa1604000 - 0xa1606fff /usr/lib/system/libquarantine.dylib 0xa1607000 - 0xa1608fff /usr/lib/system/libremovefile.dylib 0xa1609000 - 0xa1620fff /usr/lib/system/libsystem_asl.dylib 0xa1621000 - 0xa1621fff /usr/lib/system/libsystem_blocks.dylib 0xa1622000 - 0xa16b0fff /usr/lib/system/libsystem_c.dylib 0xa16b1000 - 0xa16b4fff /usr/lib/system/libsystem_configuration.dylib 0xa16b5000 - 0xa16b8fff /usr/lib/system/libsystem_coreservices.dylib 0xa16b9000 - 0xa16d1fff /usr/lib/system/libsystem_coretls.dylib 0xa16d2000 - 0xa16d8fff /usr/lib/system/libsystem_dnssd.dylib 0xa16d9000 - 0xa1700fff /usr/lib/system/libsystem_info.dylib 0xa1701000 - 0xa1722fff /usr/lib/system/libsystem_kernel.dylib 0xa1723000 - 0xa176ffff /usr/lib/system/libsystem_m.dylib 0xa1770000 - 0xa178afff /usr/lib/system/libsystem_malloc.dylib 0xa178b000 - 0xa17e1fff /usr/lib/system/libsystem_network.dylib 0xa17e2000 - 0xa17ebfff /usr/lib/system/libsystem_networkextension.dylib 0xa17ec000 - 0xa17f4fff /usr/lib/system/libsystem_notify.dylib 0xa17f5000 - 0xa17fbfff /usr/lib/system/libsystem_platform.dylib 0xa17fc000 - 0xa1805fff /usr/lib/system/libsystem_pthread.dylib 0xa1806000 - 0xa1809fff /usr/lib/system/libsystem_sandbox.dylib 0xa180a000 - 0xa180cfff /usr/lib/system/libsystem_secinit.dylib 0xa180d000 - 0xa1814fff /usr/lib/system/libsystem_symptoms.dylib 0xa1815000 - 0xa1827fff /usr/lib/system/libsystem_trace.dylib 0xa1828000 - 0xa182efff /usr/lib/system/libunwind.dylib 0xa182f000 - 0xa1855fff /usr/lib/system/libxpc.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=94573, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3s_se_8.36_i686-apple-darwin.dylib after 11 attempts forrtl: severe (121): Cannot access current working directory for unit 8, file "Unknown" Image PC Routine Line Source hadcm3s_um_8.36_i 004C4867 Unknown Unknown Unknown hadcm3s_um_8.36_i 004D7DA4 Unknown Unknown Unknown hadcm3s_um_8.36_i 0049AFBF Unknown Unknown Unknown hadcm3s_um_8.36_i 004A8D78 Unknown Unknown Unknown hadcm3s_um_8.36_i 000018FD Unknown Unknown Unknown hadcm3s_um_8.36_i 00001828 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=94573, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=94573, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=94573, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=94573, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( zip warning: Too many open files in system zip warning: Too many open files in system zip warning: Too many open files in system 18:44:02 (94573): called boinc_finish(22) </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Jan 2022 22:05:14	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	285,120	1,640,992	5.7554
27 Jan 2022 22:06:56	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	259,200	1,481,605	5.7161
25 Jan 2022 18:01:39	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	233,280	1,322,601	5.6696
23 Jan 2022 11:16:10	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	207,360	1,163,896	5.6129
20 Jan 2022 14:42:47	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	181,440	1,008,225	5.5568
18 Jan 2022 13:01:51	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	155,520	860,553	5.5334
16 Jan 2022 06:10:56	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	129,600	702,117	5.4176
14 Jan 2022 05:48:05	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	103,680	561,859	5.4192
12 Jan 2022 01:27:41	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	77,760	416,485	5.3560
10 Jan 2022 01:40:12	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	51,840	274,467	5.2945
07 Jan 2022 22:06:10	1503934	22180862	hadcm3s_18ra_200012_168_926_012127866_0	25,920	137,026	5.2865