climateprediction.net home page
Task 22247207

Task 22247207

Name oifs_43r3_ps_2103_2021050100_123_947_12165192_0
Workunit 12165192
Created 29 Nov 2022, 13:06:51 UTC
Sent 29 Nov 2022, 16:56:42 UTC
Report deadline 29 Dec 2022, 16:56:42 UTC
Received 30 Nov 2022, 11:16:22 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1532237
Run time 15 hours 10 min 37 sec
CPU time 27 min 25 sec
Validate state Invalid
Credit 12,096.00
Device peak FLOPS 7.27 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,413.87 MB
Peak swap size 4,978.16 MB
Peak disk usage 11,825.42 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
r/lib/boinc-client/slots/3/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3778733:3778733] [20221130:013511:1669797311:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:35:51 STEP 2040 H=2040:00 +CPU= 35.888
  01:36:12 STEP 2041 H=2041:00 +CPU= 20.705
  01:36:32 STEP 2042 H=2042:00 +CPU= 19.362
  01:37:02 STEP 2043 H=2043:00 +CPU= 29.414
  01:37:23 STEP 2044 H=2044:00 +CPU= 20.575
  01:37:42 STEP 2045 H=2045:00 +CPU= 18.427
  01:38:11 STEP 2046 H=2046:00 +CPU= 28.722
  01:38:29 STEP 2047 H=2047:00 +CPU= 17.753
  01:38:47 STEP 2048 H=2048:00 +CPU= 18.428
  01:39:19 STEP 2049 H=2049:00 +CPU= 31.260
  01:39:37 STEP 2050 H=2050:00 +CPU= 17.884
  01:39:55 STEP 2051 H=2051:00 +CPU= 17.269
  01:40:28 STEP 2052 H=2052:00 +CPU= 32.613
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002052
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002052
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002052
  01:40:44 STEP 2053 H=2053:00 +CPU= 16.214
  01:41:01 STEP 2054 H=2054:00 +CPU= 17.090
  01:41:34 STEP 2055 H=2055:00 +CPU= 32.154
  01:41:52 STEP 2056 H=2056:00 +CPU= 17.917
  01:42:08 STEP 2057 H=2057:00 +CPU= 16.069
  01:42:37 STEP 2058 H=2058:00 +CPU= 28.203
  01:43:00 STEP 2059 H=2059:00 +CPU= 22.692
  01:43:18 STEP 2060 H=2060:00 +CPU= 17.484
  01:43:46 STEP 2061 H=2061:00 +CPU= 27.650
  01:44:07 STEP 2062 H=2062:00 +CPU= 21.122
  01:44:28 STEP 2063 H=2063:00 +CPU= 19.888
  01:45:02 STEP 2064 H=2064:00 +CPU= 33.370
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002064
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002064
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 7430400
  01:45:27 STEP 2065 H=2065:00 +CPU= 23.813
  01:45:45 STEP 2066 H=2066:00 +CPU= 17.732
  01:46:11 STEP 2067 H=2067:00 +CPU= 26.153
  01:46:31 STEP 2068 H=2068:00 +CPU= 20.049
  01:46:51 STEP 2069 H=2069:00 +CPU= 19.017
  01:47:15 STEP 2070 H=2070:00 +CPU= 24.040
  01:47:32 STEP 2071 H=2071:00 +CPU= 16.894
  01:47:55 STEP 2072 H=2072:00 +CPU= 21.789
  01:48:23 STEP 2073 H=2073:00 +CPU= 27.623
  01:48:39 STEP 2074 H=2074:00 +CPU= 16.227
  01:48:57 STEP 2075 H=2075:00 +CPU= 17.656
  01:49:31 STEP 2076 H=2076:00 +CPU= 32.924
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002076
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002076
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002076
  01:49:47 STEP 2077 H=2077:00 +CPU= 15.969
  01:50:04 STEP 2078 H=2078:00 +CPU= 17.270
  01:50:36 STEP 2079 H=2079:00 +CPU= 31.412
  01:50:53 STEP 2080 H=2080:00 +CPU= 16.964
  01:51:09 STEP 2081 H=2081:00 +CPU= 15.800
  01:51:42 STEP 2082 H=2082:00 +CPU= 31.260
  01:52:02 STEP 2083 H=2083:00 +CPU= 19.600
  01:52:17 STEP 2084 H=2084:00 +CPU= 15.363
  01:52:44 STEP 2085 H=2085:00 +CPU= 26.805
  01:53:07 STEP 2086 H=2086:00 +CPU= 22.029
  01:53:28 STEP 2087 H=2087:00 +CPU= 20.620
  01:54:00 STEP 2088 H=2088:00 +CPU= 31.329
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002088
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002088
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_2.zip
Uploading the intermediate file: upload_file_2.zip
  01:54:21 STEP 2089 H=2089:00 +CPU= 20.892
Uploading trickle at timestep: 7516800
  01:54:41 STEP 2090 H=2090:00 +CPU= 18.802
  01:55:06 STEP 2091 H=2091:00 +CPU= 25.344
  01:55:25 STEP 2092 H=2092:00 +CPU= 19.188
  01:55:47 STEP 2093 H=2093:00 +CPU= 21.008
  01:56:14 STEP 2094 H=2094:00 +CPU= 27.223
  01:56:31 STEP 2095 H=2095:00 +CPU= 15.983
  01:56:51 STEP 2096 H=2096:00 +CPU= 19.861
  01:57:21 STEP 2097 H=2097:00 +CPU= 29.221
  01:57:37 STEP 2098 H=2098:00 +CPU= 16.526
  01:57:57 STEP 2099 H=2099:00 +CPU= 19.473
  01:58:34 STEP 2100 H=2100:00 +CPU= 36.136
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002100
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002100
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002100
  01:58:53 STEP 2101 H=2101:00 +CPU= 18.607
  01:59:13 STEP 2102 H=2102:00 +CPU= 20.046
  01:59:44 STEP 2103 H=2103:00 +CPU= 30.056
  02:00:01 STEP 2104 H=2104:00 +CPU= 17.615
  02:00:18 STEP 2105 H=2105:00 +CPU= 16.971
  02:00:49 STEP 2106 H=2106:00 +CPU= 29.708
  02:01:08 STEP 2107 H=2107:00 +CPU= 18.534
  02:01:26 STEP 2108 H=2108:00 +CPU= 17.694
  02:01:54 STEP 2109 H=2109:00 +CPU= 27.733
  02:02:16 STEP 2110 H=2110:00 +CPU= 21.743
  02:02:37 STEP 2111 H=2111:00 +CPU= 20.316
  02:03:10 STEP 2112 H=2112:00 +CPU= 32.123
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002112
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002112
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_3.zip
Uploading the intermediate file: upload_file_3.zip
  02:03:30 STEP 2113 H=2113:00 +CPU= 19.303
Uploading trickle at timestep: 7603200
  02:03:49 STEP 2114 H=2114:00 +CPU= 19.215
  02:04:18 STEP 2115 H=2115:00 +CPU= 28.602
  02:04:39 STEP 2116 H=2116:00 +CPU= 20.691
  02:04:58 STEP 2117 H=2117:00 +CPU= 18.235
  02:05:26 STEP 2118 H=2118:00 +CPU= 27.826
  02:05:45 STEP 2119 H=2119:00 +CPU= 18.759
  02:06:03 STEP 2120 H=2120:00 +CPU= 18.252
  02:06:32 STEP 2121 H=2121:00 +CPU= 28.341
  02:06:51 STEP 2122 H=2122:00 +CPU= 18.611
  02:07:12 STEP 2123 H=2123:00 +CPU= 20.833
  02:07:47 STEP 2124 H=2124:00 +CPU= 34.188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002124
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002124
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002124
  02:08:03 STEP 2125 H=2125:00 +CPU= 16.087
  02:08:22 STEP 2126 H=2126:00 +CPU= 18.374
  02:08:52 STEP 2127 H=2127:00 +CPU= 29.433
  02:09:11 STEP 2128 H=2128:00 +CPU= 18.708
  02:09:28 STEP 2129 H=2129:00 +CPU= 17.327
  02:09:59 STEP 2130 H=2130:00 +CPU= 29.653
  02:10:21 STEP 2131 H=2131:00 +CPU= 22.127
  02:10:39 STEP 2132 H=2132:00 +CPU= 17.426
  02:11:07 STEP 2133 H=2133:00 +CPU= 27.415
  02:11:27 STEP 2134 H=2134:00 +CPU= 20.365
  02:11:49 STEP 2135 H=2135:00 +CPU= 20.844
  02:12:23 STEP 2136 H=2136:00 +CPU= 33.428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002136
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002136
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_4.zip
Uploading the intermediate file: upload_file_4.zip
  02:12:42 STEP 2137 H=2137:00 +CPU= 19.156
Uploading trickle at timestep: 7689600
  02:13:03 STEP 2138 H=2138:00 +CPU= 20.017
  02:13:31 STEP 2139 H=2139:00 +CPU= 27.796
  02:13:48 STEP 2140 H=2140:00 +CPU= 16.947
  02:14:11 STEP 2141 H=2141:00 +CPU= 21.916
  02:14:40 STEP 2142 H=2142:00 +CPU= 29.127
  02:14:57 STEP 2143 H=2143:00 +CPU= 17.100
  02:15:16 STEP 2144 H=2144:00 +CPU= 18.845
  02:15:45 STEP 2145 H=2145:00 +CPU= 27.830
  02:16:04 STEP 2146 H=2146:00 +CPU= 19.012
  02:16:24 STEP 2147 H=2147:00 +CPU= 19.499
  02:16:57 STEP 2148 H=2148:00 +CPU= 32.820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002148
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002148
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002148
  02:17:16 STEP 2149 H=2149:00 +CPU= 18.329
  02:17:34 STEP 2150 H=2150:00 +CPU= 18.026
  02:18:03 STEP 2151 H=2151:00 +CPU= 28.145
  02:18:22 STEP 2152 H=2152:00 +CPU= 19.085
  02:18:41 STEP 2153 H=2153:00 +CPU= 18.871
  02:19:12 STEP 2154 H=2154:00 +CPU= 30.268
  02:19:31 STEP 2155 H=2155:00 +CPU= 18.981
  02:19:50 STEP 2156 H=2156:00 +CPU= 18.236
  02:20:20 STEP 2157 H=2157:00 +CPU= 29.443
  02:20:40 STEP 2158 H=2158:00 +CPU= 19.680
  02:20:59 STEP 2159 H=2159:00 +CPU= 19.209
  02:21:33 STEP 2160 H=2160:00 +CPU= 32.982
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002160
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002160
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_5.zip
Uploading the intermediate file: upload_file_5.zip
  02:21:52 STEP 2161 H=2161:00 +CPU= 18.363
Uploading trickle at timestep: 7776000
  02:22:13 STEP 2162 H=2162:00 +CPU= 21.151
  02:22:41 STEP 2163 H=2163:00 +CPU= 27.736
  02:23:00 STEP 2164 H=2164:00 +CPU= 18.270
  02:23:19 STEP 2165 H=2165:00 +CPU= 18.575
  02:23:49 STEP 2166 H=2166:00 +CPU= 29.563
  02:24:08 STEP 2167 H=2167:00 +CPU= 19.061
  02:24:27 STEP 2168 H=2168:00 +CPU= 19.148
  02:24:55 STEP 2169 H=2169:00 +CPU= 27.555
  02:25:13 STEP 2170 H=2170:00 +CPU= 18.092
  02:25:33 STEP 2171 H=2171:00 +CPU= 19.209
  02:26:02 STEP 2172 H=2172:00 +CPU= 28.885
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002172
  02:26:20 STEP 2173 H=2173:00 +CPU= 17.293
  02:26:42 STEP 2174 H=2174:00 +CPU= 21.803
  02:27:10 STEP 2175 H=2175:00 +CPU= 27.803
  02:27:26 STEP 2176 H=2176:00 +CPU= 15.569
  02:27:44 STEP 2177 H=2177:00 +CPU= 18.036
  02:28:15 STEP 2178 H=2178:00 +CPU= 30.511
  02:28:34 STEP 2179 H=2179:00 +CPU= 18.501
  02:28:52 STEP 2180 H=2180:00 +CPU= 17.291
  02:29:23 STEP 2181 H=2181:00 +CPU= 30.760
  02:29:43 STEP 2182 H=2182:00 +CPU= 19.206
  02:30:02 STEP 2183 H=2183:00 +CPU= 19.189
  02:30:39 STEP 2184 H=2184:00 +CPU= 35.701
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_6.zip
Uploading the intermediate file: upload_file_6.zip
  02:30:56 STEP 2185 H=2185:00 +CPU= 17.586
Uploading trickle at timestep: 7862400
  02:31:16 STEP 2186 H=2186:00 +CPU= 18.664
  02:31:47 STEP 2187 H=2187:00 +CPU= 30.769
  02:32:09 STEP 2188 H=2188:00 +CPU= 21.700
  02:32:28 STEP 2189 H=2189:00 +CPU= 18.753
  02:32:54 STEP 2190 H=2190:00 +CPU= 25.462
  02:33:12 STEP 2191 H=2191:00 +CPU= 18.304
  02:33:33 STEP 2192 H=2192:00 +CPU= 20.507
  02:34:02 STEP 2193 H=2193:00 +CPU= 28.194
  02:34:21 STEP 2194 H=2194:00 +CPU= 19.350
  02:34:43 STEP 2195 H=2195:00 +CPU= 21.178
  02:35:13 STEP 2196 H=2196:00 +CPU= 29.747
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002196
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2103
(argv4) batchid: 947
(argv5) wuid: 12165192
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_29713a8d9ee7a4c588dd416689af0464 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2103_2021050100_123_947_12165192.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2103_2021050100_123_947_12165192.zip
ic_ancil_file: ic_ancil_12165192
ifsdata_file: ifsdata_12165192
climate_data_file: clim_data_12165192
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_754855c5add0956c01ec90f3709d3a2e to: /var/lib/boinc-client/slots/3/ic_ancil_12165192.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12165192.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7e03231be0b2382fe875a091679a512f to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165192.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165192.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165192.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165192.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714
The child process has been launched with process id: 3780893
Executing the command: /var/lib/boinc-client/slots/3/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3780893:3780893] [20221130:023627:1669800987:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:37:03 STEP 2184 H=2184:00 +CPU= 32.130
  02:37:22 STEP 2185 H=2185:00 +CPU= 18.657
  02:37:43 STEP 2186 H=2186:00 +CPU= 20.449
  02:38:10 STEP 2187 H=2187:00 +CPU= 27.417
  02:38:29 STEP 2188 H=2188:00 +CPU= 18.568
  02:38:48 STEP 2189 H=2189:00 +CPU= 18.239
  02:39:20 STEP 2190 H=2190:00 +CPU= 30.987
  02:39:38 STEP 2191 H=2191:00 +CPU= 18.126
  02:39:56 STEP 2192 H=2192:00 +CPU= 17.753
  02:40:27 STEP 2193 H=2193:00 +CPU= 30.694
  02:40:48 STEP 2194 H=2194:00 +CPU= 20.170
  02:41:05 STEP 2195 H=2195:00 +CPU= 17.563
  02:41:34 STEP 2196 H=2196:00 +CPU= 28.915
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002196
  02:41:56 STEP 2197 H=2197:00 +CPU= 20.528
  02:42:15 STEP 2198 H=2198:00 +CPU= 19.230
  02:42:41 STEP 2199 H=2199:00 +CPU= 26.145
  02:43:01 STEP 2200 H=2200:00 +CPU= 19.749
  02:43:23 STEP 2201 H=2201:00 +CPU= 20.993
  02:43:52 STEP 2202 H=2202:00 +CPU= 29.337
  02:44:10 STEP 2203 H=2203:00 +CPU= 17.824
  02:44:31 STEP 2204 H=2204:00 +CPU= 20.032
  02:44:59 STEP 2205 H=2205:00 +CPU= 27.649
  02:45:17 STEP 2206 H=2206:00 +CPU= 17.702
  02:45:38 STEP 2207 H=2207:00 +CPU= 21.073
  02:46:13 STEP 2208 H=2208:00 +CPU= 34.048
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002208
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002208
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_1.zip
Uploading the intermediate file: upload_file_1.zip
  02:46:31 STEP 2209 H=2209:00 +CPU= 18.080
Uploading trickle at timestep: 7948800
  02:46:49 STEP 2210 H=2210:00 +CPU= 17.840
  02:47:20 STEP 2211 H=2211:00 +CPU= 30.295
  02:47:37 STEP 2212 H=2212:00 +CPU= 16.536
  02:47:55 STEP 2213 H=2213:00 +CPU= 18.280
  02:48:25 STEP 2214 H=2214:00 +CPU= 29.533
  02:48:44 STEP 2215 H=2215:00 +CPU= 18.152
  02:49:04 STEP 2216 H=2216:00 +CPU= 20.160
  02:49:36 STEP 2217 H=2217:00 +CPU= 31.469
  02:49:56 STEP 2218 H=2218:00 +CPU= 19.428
  02:50:14 STEP 2219 H=2219:00 +CPU= 17.770
  02:50:43 STEP 2220 H=2220:00 +CPU= 28.673
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002220
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002220
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002220
  02:51:04 STEP 2221 H=2221:00 +CPU= 20.770
  02:51:23 STEP 2222 H=2222:00 +CPU= 18.411
  02:51:48 STEP 2223 H=2223:00 +CPU= 24.652
  02:52:08 STEP 2224 H=2224:00 +CPU= 19.974
  02:52:28 STEP 2225 H=2225:00 +CPU= 19.942
  02:52:55 STEP 2226 H=2226:00 +CPU= 25.967
  02:53:13 STEP 2227 H=2227:00 +CPU= 17.897
  02:53:35 STEP 2228 H=2228:00 +CPU= 21.272
  02:54:06 STEP 2229 H=2229:00 +CPU= 30.786
  02:54:23 STEP 2230 H=2230:00 +CPU= 17.142
  02:54:43 STEP 2231 H=2231:00 +CPU= 19.573
  02:55:16 STEP 2232 H=2232:00 +CPU= 31.494
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002232
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002232
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_2.zip
Uploading the intermediate file: upload_file_2.zip
  02:55:34 STEP 2233 H=2233:00 +CPU= 17.795
Uploading trickle at timestep: 8035200
  02:55:55 STEP 2234 H=2234:00 +CPU= 20.901
  02:56:24 STEP 2235 H=2235:00 +CPU= 28.560
  02:56:41 STEP 2236 H=2236:00 +CPU= 16.832
  02:57:01 STEP 2237 H=2237:00 +CPU= 19.077
  02:57:33 STEP 2238 H=2238:00 +CPU= 31.417
  02:57:50 STEP 2239 H=2239:00 +CPU= 16.665
  02:58:08 STEP 2240 H=2240:00 +CPU= 18.018
  02:58:38 STEP 2241 H=2241:00 +CPU= 29.924
  02:58:57 STEP 2242 H=2242:00 +CPU= 18.216
  02:59:13 STEP 2243 H=2243:00 +CPU= 15.808
  02:59:46 STEP 2244 H=2244:00 +CPU= 32.310
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002244
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002244
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002244
  03:00:07 STEP 2245 H=2245:00 +CPU= 20.612
  03:00:27 STEP 2246 H=2246:00 +CPU= 19.546
  03:00:54 STEP 2247 H=2247:00 +CPU= 27.361
  03:01:13 STEP 2248 H=2248:00 +CPU= 18.469
  03:01:33 STEP 2249 H=2249:00 +CPU= 19.398
  03:02:05 STEP 2250 H=2250:00 +CPU= 31.431
  03:02:26 STEP 2251 H=2251:00 +CPU= 20.501
  03:02:45 STEP 2252 H=2252:00 +CPU= 18.974
  03:03:17 STEP 2253 H=2253:00 +CPU= 31.147
  03:03:38 STEP 2254 H=2254:00 +CPU= 20.972
  03:03:56 STEP 2255 H=2255:00 +CPU= 17.875
  03:04:29 STEP 2256 H=2256:00 +CPU= 31.985
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002256
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002256
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_3.zip
Uploading the intermediate file: upload_file_3.zip
  03:04:47 STEP 2257 H=2257:00 +CPU= 18.240
Uploading trickle at timestep: 8121600
  03:05:09 STEP 2258 H=2258:00 +CPU= 20.507
  03:05:37 STEP 2259 H=2259:00 +CPU= 28.147
  03:05:55 STEP 2260 H=2260:00 +CPU= 17.978
  03:06:16 STEP 2261 H=2261:00 +CPU= 20.739
  03:06:45 STEP 2262 H=2262:00 +CPU= 28.349
  03:07:03 STEP 2263 H=2263:00 +CPU= 18.394
  03:07:25 STEP 2264 H=2264:00 +CPU= 20.668
  03:07:55 STEP 2265 H=2265:00 +CPU= 30.397
  03:08:11 STEP 2266 H=2266:00 +CPU= 16.025
  03:08:30 STEP 2267 H=2267:00 +CPU= 18.201
  03:09:03 STEP 2268 H=2268:00 +CPU= 32.573
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002268
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002268
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002268
  03:09:24 STEP 2269 H=2269:00 +CPU= 20.096
  03:09:42 STEP 2270 H=2270:00 +CPU= 18.354
  03:10:13 STEP 2271 H=2271:00 +CPU= 30.056
  03:10:31 STEP 2272 H=2272:00 +CPU= 18.414
  03:10:51 STEP 2273 H=2273:00 +CPU= 18.890
  03:11:20 STEP 2274 H=2274:00 +CPU= 28.823
  03:11:39 STEP 2275 H=2275:00 +CPU= 18.405
  03:11:58 STEP 2276 H=2276:00 +CPU= 19.117
  03:12:29 STEP 2277 H=2277:00 +CPU= 30.191
  03:12:48 STEP 2278 H=2278:00 +CPU= 18.571
  03:13:07 STEP 2279 H=2279:00 +CPU= 19.025
  03:13:44 STEP 2280 H=2280:00 +CPU= 35.854
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002280
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002280
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_4.zip
Uploading the intermediate file: upload_file_4.zip
  03:14:01 STEP 2281 H=2281:00 +CPU= 16.942
Uploading trickle at timestep: 8208000
  03:14:20 STEP 2282 H=2282:00 +CPU= 18.943
  03:14:53 STEP 2283 H=2283:00 +CPU= 32.287
  03:15:13 STEP 2284 H=2284:00 +CPU= 19.810
  03:15:33 STEP 2285 H=2285:00 +CPU= 19.090
  03:16:01 STEP 2286 H=2286:00 +CPU= 28.605
  03:16:21 STEP 2287 H=2287:00 +CPU= 19.201
  03:16:38 STEP 2288 H=2288:00 +CPU= 16.564
  03:17:06 STEP 2289 H=2289:00 +CPU= 28.345
  03:17:29 STEP 2290 H=2290:00 +CPU= 22.237
  03:17:48 STEP 2291 H=2291:00 +CPU= 18.979
  03:18:19 STEP 2292 H=2292:00 +CPU= 29.502
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002292
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002292
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002292
  03:18:39 STEP 2293 H=2293:00 +CPU= 20.446
  03:18:59 STEP 2294 H=2294:00 +CPU= 19.297
  03:19:25 STEP 2295 H=2295:00 +CPU= 26.441
  03:19:45 STEP 2296 H=2296:00 +CPU= 19.527
  03:20:06 STEP 2297 H=2297:00 +CPU= 20.439
  03:20:36 STEP 2298 H=2298:00 +CPU= 29.326
  03:20:55 STEP 2299 H=2299:00 +CPU= 18.269
  03:21:14 STEP 2300 H=2300:00 +CPU= 19.124
  03:21:44 STEP 2301 H=2301:00 +CPU= 29.256
  03:22:03 STEP 2302 H=2302:00 +CPU= 18.964
  03:22:22 STEP 2303 H=2303:00 +CPU= 19.172
  03:22:58 STEP 2304 H=2304:00 +CPU= 34.710
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_5.zip
Uploading the intermediate file: upload_file_5.zip
  03:23:18 STEP 2305 H=2305:00 +CPU= 19.485
Uploading trickle at timestep: 8294400
  03:23:35 STEP 2306 H=2306:00 +CPU= 16.683
  03:24:05 STEP 2307 H=2307:00 +CPU= 29.730
  03:24:25 STEP 2308 H=2308:00 +CPU= 19.198
  03:24:42 STEP 2309 H=2309:00 +CPU= 17.635
  03:25:11 STEP 2310 H=2310:00 +CPU= 27.977
  03:25:29 STEP 2311 H=2311:00 +CPU= 18.543
  03:25:50 STEP 2312 H=2312:00 +CPU= 20.097
  03:26:17 STEP 2313 H=2313:00 +CPU= 26.868
  03:26:37 STEP 2314 H=2314:00 +CPU= 19.666
  03:26:55 STEP 2315 H=2315:00 +CPU= 17.798
  03:27:26 STEP 2316 H=2316:00 +CPU= 31.052
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002316
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002316
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002316
  03:27:46 STEP 2317 H=2317:00 +CPU= 19.320
  03:28:04 STEP 2318 H=2318:00 +CPU= 17.583
  03:28:32 STEP 2319 H=2319:00 +CPU= 27.449
  03:28:53 STEP 2320 H=2320:00 +CPU= 21.150
  03:29:13 STEP 2321 H=2321:00 +CPU= 19.394
  03:29:42 STEP 2322 H=2322:00 +CPU= 28.473
  03:30:01 STEP 2323 H=2323:00 +CPU= 18.404
  03:30:20 STEP 2324 H=2324:00 +CPU= 19.134
  03:30:46 STEP 2325 H=2325:00 +CPU= 25.274
  03:31:06 STEP 2326 H=2326:00 +CPU= 19.745
  03:31:27 STEP 2327 H=2327:00 +CPU= 20.741
  03:31:57 STEP 2328 H=2328:00 +CPU= 29.579
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002328
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002328
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_6.zip
Uploading the intermediate file: upload_file_6.zip
  03:32:16 STEP 2329 H=2329:00 +CPU= 18.043
Uploading trickle at timestep: 8380800
  03:32:37 STEP 2330 H=2330:00 +CPU= 20.794
  03:33:04 STEP 2331 H=2331:00 +CPU= 26.222
  03:33:21 STEP 2332 H=2332:00 +CPU= 17.337
  03:33:41 STEP 2333 H=2333:00 +CPU= 19.841
  03:34:10 STEP 2334 H=2334:00 +CPU= 28.338
  03:34:26 STEP 2335 H=2335:00 +CPU= 15.428
  03:34:45 STEP 2336 H=2336:00 +CPU= 18.944
  03:35:18 STEP 2337 H=2337:00 +CPU= 32.081
  03:35:36 STEP 2338 H=2338:00 +CPU= 18.625
  03:35:54 STEP 2339 H=2339:00 +CPU= 17.350
  03:36:27 STEP 2340 H=2340:00 +CPU= 32.397
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002340
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002340
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002340
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2103
(argv4) batchid: 947
(argv5) wuid: 12165192
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_29713a8d9ee7a4c588dd416689af0464 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2103_2021050100_123_947_12165192.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2103_2021050100_123_947_12165192.zip
ic_ancil_file: ic_ancil_12165192
ifsdata_file: ifsdata_12165192
climate_data_file: clim_data_12165192
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_754855c5add0956c01ec90f3709d3a2e to: /var/lib/boinc-client/slots/3/ic_ancil_12165192.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12165192.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7e03231be0b2382fe875a091679a512f to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165192.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165192.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165192.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165192.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714
The child process has been launched with process id: 3783067
Executing the command: /var/lib/boinc-client/slots/3/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3783067:3783067] [20221130:033733:1669804653:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:38:11 STEP 2328 H=2328:00 +CPU= 33.817
  03:38:31 STEP 2329 H=2329:00 +CPU= 19.597
  03:38:54 STEP 2330 H=2330:00 +CPU= 22.592
  03:39:22 STEP 2331 H=2331:00 +CPU= 27.755
  03:39:42 STEP 2332 H=2332:00 +CPU= 19.479
  03:40:00 STEP 2333 H=2333:00 +CPU= 17.614
  03:40:30 STEP 2334 H=2334:00 +CPU= 30.290
  03:40:48 STEP 2335 H=2335:00 +CPU= 17.376
  03:41:09 STEP 2336 H=2336:00 +CPU= 20.312
  03:41:42 STEP 2337 H=2337:00 +CPU= 32.888
  03:41:59 STEP 2338 H=2338:00 +CPU= 16.463
  03:42:17 STEP 2339 H=2339:00 +CPU= 17.959
  03:42:49 STEP 2340 H=2340:00 +CPU= 31.942
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002340
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002340
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002340
  03:43:08 STEP 2341 H=2341:00 +CPU= 18.712
  03:43:26 STEP 2342 H=2342:00 +CPU= 17.830
  03:43:58 STEP 2343 H=2343:00 +CPU= 30.886
  03:44:18 STEP 2344 H=2344:00 +CPU= 20.050
  03:44:37 STEP 2345 H=2345:00 +CPU= 18.331
  03:45:08 STEP 2346 H=2346:00 +CPU= 30.350
  03:45:25 STEP 2347 H=2347:00 +CPU= 17.099
  03:45:44 STEP 2348 H=2348:00 +CPU= 19.138
  03:46:15 STEP 2349 H=2349:00 +CPU= 30.268
  03:46:35 STEP 2350 H=2350:00 +CPU= 19.355
  03:46:51 STEP 2351 H=2351:00 +CPU= 15.874
  03:47:26 STEP 2352 H=2352:00 +CPU= 34.418
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002352
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002352
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_1.zip
Uploading the intermediate file: upload_file_1.zip
  03:47:48 STEP 2353 H=2353:00 +CPU= 21.395
Uploading trickle at timestep: 8467200
  03:48:05 STEP 2354 H=2354:00 +CPU= 17.003
  03:48:31 STEP 2355 H=2355:00 +CPU= 25.794
  03:48:54 STEP 2356 H=2356:00 +CPU= 21.793
  03:49:14 STEP 2357 H=2357:00 +CPU= 19.942
  03:49:38 STEP 2358 H=2358:00 +CPU= 23.735
  03:49:55 STEP 2359 H=2359:00 +CPU= 17.061
  03:50:16 STEP 2360 H=2360:00 +CPU= 20.865
  03:50:45 STEP 2361 H=2361:00 +CPU= 28.604
  03:51:00 STEP 2362 H=2362:00 +CPU= 14.390
  03:51:16 STEP 2363 H=2363:00 +CPU= 15.936
  03:51:52 STEP 2364 H=2364:00 +CPU= 35.361
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002364
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002364
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002364
  03:52:10 STEP 2365 H=2365:00 +CPU= 18.016
  03:52:27 STEP 2366 H=2366:00 +CPU= 16.813
  03:52:56 STEP 2367 H=2367:00 +CPU= 28.373
  03:53:17 STEP 2368 H=2368:00 +CPU= 20.473
  03:53:34 STEP 2369 H=2369:00 +CPU= 16.661
  03:54:01 STEP 2370 H=2370:00 +CPU= 26.833
  03:54:21 STEP 2371 H=2371:00 +CPU= 19.293
  03:54:40 STEP 2372 H=2372:00 +CPU= 18.576
  03:55:06 STEP 2373 H=2373:00 +CPU= 25.517
  03:55:25 STEP 2374 H=2374:00 +CPU= 18.913
  03:55:45 STEP 2375 H=2375:00 +CPU= 19.814
  03:56:19 STEP 2376 H=2376:00 +CPU= 32.436
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002376
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002376
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_2.zip
Uploading the intermediate file: upload_file_2.zip
  03:56:38 STEP 2377 H=2377:00 +CPU= 18.689
Uploading trickle at timestep: 8553600
  03:56:59 STEP 2378 H=2378:00 +CPU= 20.603
  03:57:27 STEP 2379 H=2379:00 +CPU= 28.290
  03:57:45 STEP 2380 H=2380:00 +CPU= 17.632
  03:58:06 STEP 2381 H=2381:00 +CPU= 20.154
  03:58:35 STEP 2382 H=2382:00 +CPU= 29.233
  03:58:54 STEP 2383 H=2383:00 +CPU= 19.009
  03:59:13 STEP 2384 H=2384:00 +CPU= 18.143
  03:59:40 STEP 2385 H=2385:00 +CPU= 27.243
  03:59:56 STEP 2386 H=2386:00 +CPU= 15.339
  04:00:17 STEP 2387 H=2387:00 +CPU= 20.276
  04:00:49 STEP 2388 H=2388:00 +CPU= 29.206
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002388
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002388
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002388
  04:01:09 STEP 2389 H=2389:00 +CPU= 19.167
  04:01:28 STEP 2390 H=2390:00 +CPU= 18.859
  04:01:55 STEP 2391 H=2391:00 +CPU= 26.894
  04:02:14 STEP 2392 H=2392:00 +CPU= 19.022
  04:02:34 STEP 2393 H=2393:00 +CPU= 19.154
  04:03:02 STEP 2394 H=2394:00 +CPU= 27.787
  04:03:21 STEP 2395 H=2395:00 +CPU= 18.363
  04:03:49 STEP 2396 H=2396:00 +CPU= 17.936
  04:04:24 STEP 2397 H=2397:00 +CPU= 32.418
  04:04:38 STEP 2398 H=2398:00 +CPU= 14.813
  04:04:54 STEP 2399 H=2399:00 +CPU= 15.128
  04:05:26 STEP 2400 H=2400:00 +CPU= 31.512
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002400
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002400
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMGGhpi1+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMSHhpi1+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165192/ICMUAhpi1+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2103_2021050100_123_947_12165192_0_r1564737714_3.zip
Uploading the intermediate file: upload_file_3.zip
Uploading trickle at timestep: 8640000
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:12:03 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 8,640,000 1,640 0.0002
09 May 2023 15:12:03 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 8,553,600 1,113 0.0001
09 May 2023 15:12:03 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 8,467,200 590 0.0001
09 May 2023 15:12:03 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 8,380,800 3,285 0.0004
09 May 2023 15:12:03 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 8,294,400 2,752 0.0003
09 May 2023 15:11:37 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 8,208,000 2,209 0.0003
09 May 2023 15:11:37 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 8,121,600 1,656 0.0002
09 May 2023 15:11:37 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 8,035,200 1,114 0.0001
09 May 2023 15:11:37 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,948,800 581 0.0001
09 May 2023 15:11:37 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,862,400 3,275 0.0004
09 May 2023 15:11:37 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,776,000 2,742 0.0004
09 May 2023 15:11:11 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,689,600 2,199 0.0003
09 May 2023 15:11:11 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,603,200 1,656 0.0002
09 May 2023 15:11:11 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,516,800 1,113 0.0001
09 May 2023 15:11:11 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,430,400 581 0.0001
09 May 2023 15:11:11 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,344,000 3,294 0.0004
09 May 2023 15:11:11 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,257,600 2,741 0.0004
09 May 2023 15:10:40 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,171,200 2,208 0.0003
09 May 2023 15:10:41 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 7,084,800 1,666 0.0002
09 May 2023 15:10:40 1532237 22247207 oifs_43r3_ps_2103_2021050100_123_947_12165192_0 6,998,400 1,123 0.0002


©2024 climateprediction.net