climateprediction.net home page
Task 22307558

Task 22307558

Name oifs_43r3_bl_a0gn_2016092300_15_991_12207386_0
Workunit 12207386
Created 13 Feb 2023, 13:07:09 UTC
Sent 13 Feb 2023, 13:10:07 UTC
Report deadline 14 Apr 2023, 13:10:07 UTC
Received 9 Mar 2023, 18:35:53 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1432159
Run time 14 hours 26 min 49 sec
CPU time 13 hours 21 min 37 sec
Validate state Valid
Credit 1,813.14
Device peak FLOPS 5.17 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,382.08 MB
Peak swap size 5,930.78 MB
Peak disk usage 1,555.48 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:20719:20719] [20230214:025414:1676339654:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:55:17 STEP 1104 H= 276:00 +CPU= 34.978
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001104
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001104
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001104
  02:55:49 STEP 1105 H= 276:15 +CPU= 28.523
  02:56:22 STEP 1106 H= 276:30 +CPU= 29.277
  02:56:56 STEP 1107 H= 276:45 +CPU= 30.744
  02:57:29 STEP 1108 H= 277:00 +CPU= 30.445
  02:58:04 STEP 1109 H= 277:15 +CPU= 30.969
  02:58:36 STEP 1110 H= 277:30 +CPU= 28.656
  02:59:11 STEP 1111 H= 277:45 +CPU= 31.548
  02:59:43 STEP 1112 H= 278:00 +CPU= 28.991
  03:00:18 STEP 1113 H= 278:15 +CPU= 30.466
  03:00:53 STEP 1114 H= 278:30 +CPU= 30.400
  03:01:23 STEP 1115 H= 278:45 +CPU= 27.883
  03:02:06 STEP 1116 H= 279:00 +CPU= 37.764
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001116
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001116
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001116
  03:02:38 STEP 1117 H= 279:15 +CPU= 28.770
  03:03:11 STEP 1118 H= 279:30 +CPU= 29.504
  03:03:46 STEP 1119 H= 279:45 +CPU= 31.109
  03:04:16 STEP 1120 H= 280:00 +CPU= 26.945
  03:04:51 STEP 1121 H= 280:15 +CPU= 30.865
  03:05:26 STEP 1122 H= 280:30 +CPU= 31.352
  03:06:00 STEP 1123 H= 280:45 +CPU= 30.251
  03:06:35 STEP 1124 H= 281:00 +CPU= 31.379
  03:07:07 STEP 1125 H= 281:15 +CPU= 29.009
  03:07:43 STEP 1126 H= 281:30 +CPU= 31.611
  03:08:15 STEP 1127 H= 281:45 +CPU= 28.130
  03:08:58 STEP 1128 H= 282:00 +CPU= 37.118
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001128
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001128
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001128
  03:09:34 STEP 1129 H= 282:15 +CPU= 31.260
  03:10:03 STEP 1130 H= 282:30 +CPU= 26.663
  03:10:40 STEP 1131 H= 282:45 +CPU= 31.802
  03:11:14 STEP 1132 H= 283:00 +CPU= 31.389
  03:11:47 STEP 1133 H= 283:15 +CPU= 29.228
  03:12:24 STEP 1134 H= 283:30 +CPU= 32.328
  03:12:58 STEP 1135 H= 283:45 +CPU= 30.419
  03:13:32 STEP 1136 H= 284:00 +CPU= 31.254
  03:14:09 STEP 1137 H= 284:15 +CPU= 32.465
  03:14:42 STEP 1138 H= 284:30 +CPU= 29.756
  03:15:17 STEP 1139 H= 284:45 +CPU= 30.548
  03:16:05 STEP 1140 H= 285:00 +CPU= 37.777
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001140
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001140
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001140
  03:16:35 STEP 1141 H= 285:15 +CPU= 27.132
  03:17:14 STEP 1142 H= 285:30 +CPU= 33.717
  03:17:47 STEP 1143 H= 285:45 +CPU= 29.500
  03:18:21 STEP 1144 H= 286:00 +CPU= 29.939
  03:18:56 STEP 1145 H= 286:15 +CPU= 31.234
  03:19:31 STEP 1146 H= 286:30 +CPU= 31.157
  03:20:05 STEP 1147 H= 286:45 +CPU= 31.070
  03:20:39 STEP 1148 H= 287:00 +CPU= 30.153
  03:21:11 STEP 1149 H= 287:15 +CPU= 29.130
  03:21:45 STEP 1150 H= 287:30 +CPU= 30.947
  03:22:18 STEP 1151 H= 287:45 +CPU= 29.330
  03:23:08 STEP 1152 H= 288:00 +CPU= 41.967
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001152
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001152
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001152
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001056
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001056
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001056
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001068
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001068
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001068
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001080
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001080
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001080
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001092
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001092
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001092
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001104
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001104
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001104
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001116
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001116
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001116
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001128
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001128
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001128
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001140
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001140
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001140
Zipping up the intermediate file: /BOINC/projects/climateprediction.net/oifs_43r3_bl_a0gn_2016092300_15_991_12207386_0_r86400562_11.zip
Uploading the intermediate file: upload_file_11.zip
  03:23:40 STEP 1153 H= 288:15 +CPU= 28.375
  03:24:13 STEP 1154 H= 288:30 +CPU= 29.837
  03:24:48 STEP 1155 H= 288:45 +CPU= 31.191
  03:25:19 STEP 1156 H= 289:00 +CPU= 28.317
  03:25:55 STEP 1157 H= 289:15 +CPU= 31.955
  03:26:28 STEP 1158 H= 289:30 +CPU= 29.365
  03:27:00 STEP 1159 H= 289:45 +CPU= 28.826
  03:27:39 STEP 1160 H= 290:00 +CPU= 33.524
  03:28:09 STEP 1161 H= 290:15 +CPU= 27.025
  03:28:42 STEP 1162 H= 290:30 +CPU= 30.204
  03:29:20 STEP 1163 H= 290:45 +CPU= 32.290
  03:30:00 STEP 1164 H= 291:00 +CPU= 35.611
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001164
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001164
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001164
  03:30:37 STEP 1165 H= 291:15 +CPU= 32.870
  03:31:10 STEP 1166 H= 291:30 +CPU= 29.782
  03:31:44 STEP 1167 H= 291:45 +CPU= 30.690
  03:32:22 STEP 1168 H= 292:00 +CPU= 32.998
  03:32:53 STEP 1169 H= 292:15 +CPU= 28.034
  03:33:28 STEP 1170 H= 292:30 +CPU= 31.242
  03:34:03 STEP 1171 H= 292:45 +CPU= 31.121
  03:34:40 STEP 1172 H= 293:00 +CPU= 32.564
  03:35:13 STEP 1173 H= 293:15 +CPU= 29.125
  03:35:50 STEP 1174 H= 293:30 +CPU= 32.113
  03:36:21 STEP 1175 H= 293:45 +CPU= 27.671
  03:37:03 STEP 1176 H= 294:00 +CPU= 36.730
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001176
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001176
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001176
  03:37:38 STEP 1177 H= 294:15 +CPU= 31.029
  03:38:10 STEP 1178 H= 294:30 +CPU= 28.875
  03:38:48 STEP 1179 H= 294:45 +CPU= 33.226
  03:39:23 STEP 1180 H= 295:00 +CPU= 31.867
  03:39:56 STEP 1181 H= 295:15 +CPU= 29.235
  03:40:31 STEP 1182 H= 295:30 +CPU= 31.373
  03:41:01 STEP 1183 H= 295:45 +CPU= 26.977
  03:41:41 STEP 1184 H= 296:00 +CPU= 32.729
  03:42:21 STEP 1185 H= 296:15 +CPU= 33.142
  03:43:01 STEP 1186 H= 296:30 +CPU= 32.746
  03:43:45 STEP 1187 H= 296:45 +CPU= 34.996
  03:44:34 STEP 1188 H= 297:00 +CPU= 41.305
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001188
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001188
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001188
  03:45:22 STEP 1189 H= 297:15 +CPU= 33.164
  03:46:08 STEP 1190 H= 297:30 +CPU= 32.618
  03:46:51 STEP 1191 H= 297:45 +CPU= 34.543
  03:47:34 STEP 1192 H= 298:00 +CPU= 34.585
  03:48:16 STEP 1193 H= 298:15 +CPU= 34.334
  03:48:56 STEP 1194 H= 298:30 +CPU= 32.725
  03:49:38 STEP 1195 H= 298:45 +CPU= 33.894
  03:50:20 STEP 1196 H= 299:00 +CPU= 34.295
  03:51:02 STEP 1197 H= 299:15 +CPU= 33.972
  03:51:45 STEP 1198 H= 299:30 +CPU= 34.552
  03:52:28 STEP 1199 H= 299:45 +CPU= 34.619
  03:53:31 STEP 1200 H= 300:00 +CPU= 45.014
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001200
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001200
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001200
  03:54:12 STEP 1201 H= 300:15 +CPU= 33.720
  03:54:51 STEP 1202 H= 300:30 +CPU= 32.599
  03:55:33 STEP 1203 H= 300:45 +CPU= 33.631
  03:56:15 STEP 1204 H= 301:00 +CPU= 34.069
  03:56:51 STEP 1205 H= 301:15 +CPU= 31.024
  03:57:28 STEP 1206 H= 301:30 +CPU= 31.489
  03:58:05 STEP 1207 H= 301:45 +CPU= 31.956
  03:58:42 STEP 1208 H= 302:00 +CPU= 32.175
  03:59:18 STEP 1209 H= 302:15 +CPU= 31.449
  03:59:55 STEP 1210 H= 302:30 +CPU= 31.843
  04:00:38 STEP 1211 H= 302:45 +CPU= 33.683
  04:01:26 STEP 1212 H= 303:00 +CPU= 40.754
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001212
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001212
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001212
  04:02:10 STEP 1213 H= 303:15 +CPU= 35.615
  04:02:47 STEP 1214 H= 303:30 +CPU= 31.614
  04:03:29 STEP 1215 H= 303:45 +CPU= 34.029
  04:04:09 STEP 1216 H= 304:00 +CPU= 34.044
  04:04:50 STEP 1217 H= 304:15 +CPU= 33.237
  04:05:31 STEP 1218 H= 304:30 +CPU= 33.303
  04:06:10 STEP 1219 H= 304:45 +CPU= 32.068
  04:06:51 STEP 1220 H= 305:00 +CPU= 33.859
  04:07:31 STEP 1221 H= 305:15 +CPU= 33.329
  04:08:11 STEP 1222 H= 305:30 +CPU= 33.656
  04:08:50 STEP 1223 H= 305:45 +CPU= 32.289
  04:09:40 STEP 1224 H= 306:00 +CPU= 41.479
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001224
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001224
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001224
  04:10:20 STEP 1225 H= 306:15 +CPU= 32.813
  04:10:52 STEP 1226 H= 306:30 +CPU= 27.959
  04:11:25 STEP 1227 H= 306:45 +CPU= 29.817
  04:12:03 STEP 1228 H= 307:00 +CPU= 31.682
  04:12:39 STEP 1229 H= 307:15 +CPU= 30.816
  04:13:14 STEP 1230 H= 307:30 +CPU= 30.908
  04:13:50 STEP 1231 H= 307:45 +CPU= 30.917
  04:14:28 STEP 1232 H= 308:00 +CPU= 32.141
  04:15:07 STEP 1233 H= 308:15 +CPU= 32.462
  04:15:43 STEP 1234 H= 308:30 +CPU= 30.719
  04:16:21 STEP 1235 H= 308:45 +CPU= 31.560
  04:17:10 STEP 1236 H= 309:00 +CPU= 39.930
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001236
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001236
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001236
  04:17:49 STEP 1237 H= 309:15 +CPU= 31.398
  04:18:28 STEP 1238 H= 309:30 +CPU= 31.533
  04:19:07 STEP 1239 H= 309:45 +CPU= 31.732
  04:19:47 STEP 1240 H= 310:00 +CPU= 32.534
  04:20:26 STEP 1241 H= 310:15 +CPU= 31.503
  04:21:05 STEP 1242 H= 310:30 +CPU= 31.590
  04:21:44 STEP 1243 H= 310:45 +CPU= 31.299
  04:22:25 STEP 1244 H= 311:00 +CPU= 32.313
  04:23:04 STEP 1245 H= 311:15 +CPU= 31.671
  04:23:43 STEP 1246 H= 311:30 +CPU= 31.918
  04:24:23 STEP 1247 H= 311:45 +CPU= 31.915
  04:25:22 STEP 1248 H= 312:00 +CPU= 42.589
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001248
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001248
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001248
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001152
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001152
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001152
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001164
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001164
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001164
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001176
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001176
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001176
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001188
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001188
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001188
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001200
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001200
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001200
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001212
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001212
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001212
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001224
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001224
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001224
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001236
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001236
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001236
Zipping up the intermediate file: /BOINC/projects/climateprediction.net/oifs_43r3_bl_a0gn_2016092300_15_991_12207386_0_r86400562_12.zip
Uploading the intermediate file: upload_file_12.zip
  04:26:03 STEP 1249 H= 312:15 +CPU= 31.584
  04:26:41 STEP 1250 H= 312:30 +CPU= 31.612
  04:27:21 STEP 1251 H= 312:45 +CPU= 31.917
  04:28:01 STEP 1252 H= 313:00 +CPU= 32.720
  04:28:40 STEP 1253 H= 313:15 +CPU= 32.027
  04:29:19 STEP 1254 H= 313:30 +CPU= 32.175
  04:29:59 STEP 1255 H= 313:45 +CPU= 32.302
  04:30:39 STEP 1256 H= 314:00 +CPU= 32.658
  04:31:19 STEP 1257 H= 314:15 +CPU= 31.734
  04:31:58 STEP 1258 H= 314:30 +CPU= 31.958
  04:32:38 STEP 1259 H= 314:45 +CPU= 32.270
  04:33:28 STEP 1260 H= 315:00 +CPU= 40.648
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001260
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001260
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001260
  04:34:09 STEP 1261 H= 315:15 +CPU= 33.130
  04:34:50 STEP 1262 H= 315:30 +CPU= 32.454
  04:35:31 STEP 1263 H= 315:45 +CPU= 32.891
  04:36:11 STEP 1264 H= 316:00 +CPU= 32.394
  04:36:51 STEP 1265 H= 316:15 +CPU= 32.214
  04:37:30 STEP 1266 H= 316:30 +CPU= 32.009
  04:38:10 STEP 1267 H= 316:45 +CPU= 31.836
  04:38:49 STEP 1268 H= 317:00 +CPU= 32.412
  04:39:29 STEP 1269 H= 317:15 +CPU= 31.653
  04:40:08 STEP 1270 H= 317:30 +CPU= 31.910
  04:40:47 STEP 1271 H= 317:45 +CPU= 32.228
  04:41:37 STEP 1272 H= 318:00 +CPU= 39.360
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001272
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001272
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001272
  04:42:17 STEP 1273 H= 318:15 +CPU= 32.008
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0gn
(argv4) batchid: 991
(argv5) wuid: 12207386
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /BOINC/slots/2
Project directory is: /BOINC/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386
..mkdir for temp folder for results failed
Copying: /BOINC/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /BOINC/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /BOINC/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e1b3dcb23b8ae782f44d8c773affb2c5 to: /BOINC/slots/2/oifs_43r3_bl_a0gn_2016092300_15_991_12207386.zip
Unzipping the namelist zip file: /BOINC/slots/2/oifs_43r3_bl_a0gn_2016092300_15_991_12207386.zip
ic_ancil_file: ic_ancil_12207386
ifsdata_file: ifsdata_12207386
climate_data_file: clim_data_12207386
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /BOINC/slots/2/ic_ancil_12207386.zip
Unzipping the IC ancils zip file: /BOINC/slots/2/ic_ancil_12207386.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5c006cc8e4ccace6dd86e1f878e5a5fb to: /BOINC/slots/2/ifsdata/ifsdata_12207386.zip
Unzipping the ifsdata_zip file: /BOINC/slots/2/ifsdata/ifsdata_12207386.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /BOINC/slots/2/159l_2/clim_data_12207386.zip
Unzipping the climate data zip file: /BOINC/slots/2/159l_2/clim_data_12207386.zip
Checking for progress XML file: /BOINC/slots/2/progress_file_12207386.xml
Opened progress file ok : /BOINC/slots/2/progress_file_12207386.xml
-- Model is restarting --
Adjusting last_iter, 1274, to previous model restart step.
Creating progress file: /BOINC/slots/2/progress_file_12207386.xml
last_cpu_time: 41054
upload_file_number: 13
last_iter: 1249
last_upload: 1123200
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a0gn_2016092300_15_991_12207386_0_r86400562
The child process has been launched with process id: 16841
Executing the command: /BOINC/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:16841:16841] [20230214:044333:1676346213:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:44:23 STEP 1248 H= 312:00 +CPU= 33.012
  04:45:01 STEP 1249 H= 312:15 +CPU= 32.168
  04:45:41 STEP 1250 H= 312:30 +CPU= 33.371
  04:46:17 STEP 1251 H= 312:45 +CPU= 31.073
  04:46:58 STEP 1252 H= 313:00 +CPU= 32.591
  04:47:37 STEP 1253 H= 313:15 +CPU= 31.932
  04:48:17 STEP 1254 H= 313:30 +CPU= 32.132
  04:48:57 STEP 1255 H= 313:45 +CPU= 32.228
  04:49:38 STEP 1256 H= 314:00 +CPU= 32.903
  04:50:17 STEP 1257 H= 314:15 +CPU= 32.441
  04:50:58 STEP 1258 H= 314:30 +CPU= 32.340
  04:51:37 STEP 1259 H= 314:45 +CPU= 32.105
  04:52:27 STEP 1260 H= 315:00 +CPU= 40.277
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001260
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001260
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001260
  04:53:06 STEP 1261 H= 315:15 +CPU= 32.323
  04:53:46 STEP 1262 H= 315:30 +CPU= 32.105
  04:54:25 STEP 1263 H= 315:45 +CPU= 32.221
  04:55:06 STEP 1264 H= 316:00 +CPU= 32.995
  04:55:45 STEP 1265 H= 316:15 +CPU= 32.211
  04:56:25 STEP 1266 H= 316:30 +CPU= 32.061
  04:57:05 STEP 1267 H= 316:45 +CPU= 32.286
  04:57:45 STEP 1268 H= 317:00 +CPU= 32.876
  04:58:23 STEP 1269 H= 317:15 +CPU= 31.982
  04:59:03 STEP 1270 H= 317:30 +CPU= 32.205
  04:59:42 STEP 1271 H= 317:45 +CPU= 31.916
  05:00:31 STEP 1272 H= 318:00 +CPU= 40.200
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001272
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001272
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001272
  05:01:11 STEP 1273 H= 318:15 +CPU= 31.994
  05:01:50 STEP 1274 H= 318:30 +CPU= 32.043
  05:02:29 STEP 1275 H= 318:45 +CPU= 31.347
  05:03:09 STEP 1276 H= 319:00 +CPU= 33.011
  05:03:49 STEP 1277 H= 319:15 +CPU= 31.852
  05:04:26 STEP 1278 H= 319:30 +CPU= 31.011
  05:05:06 STEP 1279 H= 319:45 +CPU= 31.919
  05:05:47 STEP 1280 H= 320:00 +CPU= 33.399
  05:06:29 STEP 1281 H= 320:15 +CPU= 33.415
  05:07:09 STEP 1282 H= 320:30 +CPU= 32.819
  05:07:50 STEP 1283 H= 320:45 +CPU= 32.798
  05:08:40 STEP 1284 H= 321:00 +CPU= 41.153
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001284
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001284
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001284
  05:09:21 STEP 1285 H= 321:15 +CPU= 32.895
  05:10:01 STEP 1286 H= 321:30 +CPU= 32.803
  05:10:41 STEP 1287 H= 321:45 +CPU= 32.732
  05:11:23 STEP 1288 H= 322:00 +CPU= 33.646
  05:12:03 STEP 1289 H= 322:15 +CPU= 32.412
  05:12:42 STEP 1290 H= 322:30 +CPU= 31.885
  05:13:20 STEP 1291 H= 322:45 +CPU= 31.191
  05:14:01 STEP 1292 H= 323:00 +CPU= 33.087
  05:14:41 STEP 1293 H= 323:15 +CPU= 32.328
  05:15:21 STEP 1294 H= 323:30 +CPU= 32.838
  05:16:01 STEP 1295 H= 323:45 +CPU= 32.210
  05:16:53 STEP 1296 H= 324:00 +CPU= 41.484
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001296
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001296
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001296
  05:17:28 STEP 1297 H= 324:15 +CPU= 31.150
  05:18:06 STEP 1298 H= 324:30 +CPU= 31.439
  05:18:45 STEP 1299 H= 324:45 +CPU= 32.037
  05:19:26 STEP 1300 H= 325:00 +CPU= 32.603
  05:20:07 STEP 1301 H= 325:15 +CPU= 32.788
  05:20:47 STEP 1302 H= 325:30 +CPU= 32.574
  05:21:27 STEP 1303 H= 325:45 +CPU= 32.521
  05:22:08 STEP 1304 H= 326:00 +CPU= 32.976
  05:22:47 STEP 1305 H= 326:15 +CPU= 32.086
  05:23:27 STEP 1306 H= 326:30 +CPU= 32.123
  05:24:11 STEP 1307 H= 326:45 +CPU= 32.634
  05:24:59 STEP 1308 H= 327:00 +CPU= 40.300
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001308
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001308
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001308
  05:25:36 STEP 1309 H= 327:15 +CPU= 32.179
  05:26:15 STEP 1310 H= 327:30 +CPU= 32.280
  05:26:53 STEP 1311 H= 327:45 +CPU= 32.414
  05:27:30 STEP 1312 H= 328:00 +CPU= 32.701
  05:28:08 STEP 1313 H= 328:15 +CPU= 32.031
  05:28:46 STEP 1314 H= 328:30 +CPU= 32.150
  05:29:23 STEP 1315 H= 328:45 +CPU= 32.084
  05:30:04 STEP 1316 H= 329:00 +CPU= 31.882
  05:30:42 STEP 1317 H= 329:15 +CPU= 32.807
  05:31:20 STEP 1318 H= 329:30 +CPU= 32.593
  05:31:58 STEP 1319 H= 329:45 +CPU= 32.341
  05:32:46 STEP 1320 H= 330:00 +CPU= 40.916
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001320
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001320
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001320
  05:33:23 STEP 1321 H= 330:15 +CPU= 32.530
  05:34:02 STEP 1322 H= 330:30 +CPU= 32.719
  05:34:41 STEP 1323 H= 330:45 +CPU= 32.994
  05:35:21 STEP 1324 H= 331:00 +CPU= 33.491
  05:36:00 STEP 1325 H= 331:15 +CPU= 32.912
  05:36:38 STEP 1326 H= 331:30 +CPU= 32.897
  05:37:16 STEP 1327 H= 331:45 +CPU= 32.826
  05:37:56 STEP 1328 H= 332:00 +CPU= 33.642
  05:38:34 STEP 1329 H= 332:15 +CPU= 32.868
  05:39:13 STEP 1330 H= 332:30 +CPU= 32.715
  05:39:52 STEP 1331 H= 332:45 +CPU= 32.796
  05:40:39 STEP 1332 H= 333:00 +CPU= 40.570
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001332
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001332
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001332
  05:41:17 STEP 1333 H= 333:15 +CPU= 32.250
  05:41:55 STEP 1334 H= 333:30 +CPU= 32.440
  05:42:33 STEP 1335 H= 333:45 +CPU= 32.324
  05:43:12 STEP 1336 H= 334:00 +CPU= 33.572
  05:43:50 STEP 1337 H= 334:15 +CPU= 32.367
  05:44:27 STEP 1338 H= 334:30 +CPU= 31.899
  05:45:04 STEP 1339 H= 334:45 +CPU= 32.472
  05:45:39 STEP 1340 H= 335:00 +CPU= 31.345
  05:46:15 STEP 1341 H= 335:15 +CPU= 31.509
  05:46:52 STEP 1342 H= 335:30 +CPU= 31.526
  05:47:30 STEP 1343 H= 335:45 +CPU= 31.770
  05:48:21 STEP 1344 H= 336:00 +CPU= 43.255
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001344
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001344
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001344
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001248
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001248
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001248
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001260
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001260
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001260
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001272
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001272
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001272
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001284
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001284
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001284
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001296
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001296
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001296
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001308
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001308
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001308
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001320
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001320
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001320
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001332
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001332
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001332
Zipping up the intermediate file: /BOINC/projects/climateprediction.net/oifs_43r3_bl_a0gn_2016092300_15_991_12207386_0_r86400562_13.zip
Uploading the intermediate file: upload_file_13.zip
  05:48:59 STEP 1345 H= 336:15 +CPU= 31.805
  05:49:39 STEP 1346 H= 336:30 +CPU= 31.564
  05:50:17 STEP 1347 H= 336:45 +CPU= 32.295
  05:50:55 STEP 1348 H= 337:00 +CPU= 32.825
  05:51:33 STEP 1349 H= 337:15 +CPU= 32.371
  05:52:11 STEP 1350 H= 337:30 +CPU= 32.009
  05:52:49 STEP 1351 H= 337:45 +CPU= 32.027
  05:53:27 STEP 1352 H= 338:00 +CPU= 32.598
  05:54:05 STEP 1353 H= 338:15 +CPU= 31.900
  05:54:43 STEP 1354 H= 338:30 +CPU= 32.226
  05:55:21 STEP 1355 H= 338:45 +CPU= 32.376
  05:56:07 STEP 1356 H= 339:00 +CPU= 39.615
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001356
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001356
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001356
  05:56:45 STEP 1357 H= 339:15 +CPU= 31.613
  05:57:21 STEP 1358 H= 339:30 +CPU= 31.580
  05:57:59 STEP 1359 H= 339:45 +CPU= 31.575
  05:58:37 STEP 1360 H= 340:00 +CPU= 32.508
  05:59:14 STEP 1361 H= 340:15 +CPU= 31.709
  05:59:51 STEP 1362 H= 340:30 +CPU= 31.541
  06:00:28 STEP 1363 H= 340:45 +CPU= 31.711
  06:01:07 STEP 1364 H= 341:00 +CPU= 32.433
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0gn
(argv4) batchid: 991
(argv5) wuid: 12207386
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /BOINC/slots/2
Project directory is: /BOINC/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386
..mkdir for temp folder for results failed
Copying: /BOINC/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /BOINC/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /BOINC/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e1b3dcb23b8ae782f44d8c773affb2c5 to: /BOINC/slots/2/oifs_43r3_bl_a0gn_2016092300_15_991_12207386.zip
Unzipping the namelist zip file: /BOINC/slots/2/oifs_43r3_bl_a0gn_2016092300_15_991_12207386.zip
ic_ancil_file: ic_ancil_12207386
ifsdata_file: ifsdata_12207386
climate_data_file: clim_data_12207386
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /BOINC/slots/2/ic_ancil_12207386.zip
Unzipping the IC ancils zip file: /BOINC/slots/2/ic_ancil_12207386.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5c006cc8e4ccace6dd86e1f878e5a5fb to: /BOINC/slots/2/ifsdata/ifsdata_12207386.zip
Unzipping the ifsdata_zip file: /BOINC/slots/2/ifsdata/ifsdata_12207386.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /BOINC/slots/2/159l_2/clim_data_12207386.zip
Unzipping the climate data zip file: /BOINC/slots/2/159l_2/clim_data_12207386.zip
Checking for progress XML file: /BOINC/slots/2/progress_file_12207386.xml
Opened progress file ok : /BOINC/slots/2/progress_file_12207386.xml
-- Model is restarting --
Adjusting last_iter, 1365, to previous model restart step.
Creating progress file: /BOINC/slots/2/progress_file_12207386.xml
last_cpu_time: 44934
upload_file_number: 14
last_iter: 1345
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a0gn_2016092300_15_991_12207386_0_r86400562
The child process has been launched with process id: 17513
Executing the command: /BOINC/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:fvk-015:1:1:17513:17513] [20230214:060459:1676351099:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:05:44 STEP 1344 H= 336:00 +CPU= 35.144
  06:06:19 STEP 1345 H= 336:15 +CPU= 31.832
  06:06:57 STEP 1346 H= 336:30 +CPU= 32.455
  06:07:35 STEP 1347 H= 336:45 +CPU= 32.620
  06:08:18 STEP 1348 H= 337:00 +CPU= 33.180
  06:08:56 STEP 1349 H= 337:15 +CPU= 31.946
  06:09:32 STEP 1350 H= 337:30 +CPU= 31.581
  06:10:11 STEP 1351 H= 337:45 +CPU= 32.595
  06:10:50 STEP 1352 H= 338:00 +CPU= 32.991
  06:11:28 STEP 1353 H= 338:15 +CPU= 32.739
  06:12:08 STEP 1354 H= 338:30 +CPU= 34.394
  06:12:47 STEP 1355 H= 338:45 +CPU= 32.823
  06:13:34 STEP 1356 H= 339:00 +CPU= 40.403
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001356
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001356
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001356
  06:14:12 STEP 1357 H= 339:15 +CPU= 32.693
  06:14:50 STEP 1358 H= 339:30 +CPU= 32.636
  06:15:27 STEP 1359 H= 339:45 +CPU= 32.198
  06:16:02 STEP 1360 H= 340:00 +CPU= 32.096
  06:16:41 STEP 1361 H= 340:15 +CPU= 33.244
  06:17:17 STEP 1362 H= 340:30 +CPU= 32.942
  06:17:52 STEP 1363 H= 340:45 +CPU= 31.380
  06:18:27 STEP 1364 H= 341:00 +CPU= 31.269
  06:19:04 STEP 1365 H= 341:15 +CPU= 32.071
  06:19:38 STEP 1366 H= 341:30 +CPU= 31.468
  06:20:15 STEP 1367 H= 341:45 +CPU= 32.674
  06:21:02 STEP 1368 H= 342:00 +CPU= 42.056
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001368
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001368
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001368
  06:21:37 STEP 1369 H= 342:15 +CPU= 31.582
  06:22:16 STEP 1370 H= 342:30 +CPU= 33.727
  06:22:51 STEP 1371 H= 342:45 +CPU= 31.482
  06:23:26 STEP 1372 H= 343:00 +CPU= 32.232
  06:24:01 STEP 1373 H= 343:15 +CPU= 30.990
  06:24:38 STEP 1374 H= 343:30 +CPU= 32.033
  06:25:11 STEP 1375 H= 343:45 +CPU= 29.777
  06:25:41 STEP 1376 H= 344:00 +CPU= 28.012
  06:26:17 STEP 1377 H= 344:15 +CPU= 31.858
  06:26:52 STEP 1378 H= 344:30 +CPU= 31.238
  06:27:27 STEP 1379 H= 344:45 +CPU= 31.764
  06:28:13 STEP 1380 H= 345:00 +CPU= 40.713
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001380
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001380
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001380
  06:28:48 STEP 1381 H= 345:15 +CPU= 31.616
  06:29:26 STEP 1382 H= 345:30 +CPU= 33.004
  06:30:01 STEP 1383 H= 345:45 +CPU= 31.426
  06:30:37 STEP 1384 H= 346:00 +CPU= 32.550
  06:31:13 STEP 1385 H= 346:15 +CPU= 31.316
  06:31:52 STEP 1386 H= 346:30 +CPU= 33.619
  06:32:27 STEP 1387 H= 346:45 +CPU= 31.620
  06:33:06 STEP 1388 H= 347:00 +CPU= 33.674
  06:33:42 STEP 1389 H= 347:15 +CPU= 32.073
  06:34:17 STEP 1390 H= 347:30 +CPU= 31.507
  06:34:54 STEP 1391 H= 347:45 +CPU= 32.738
  06:35:42 STEP 1392 H= 348:00 +CPU= 41.490
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001392
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001392
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001392
  06:36:18 STEP 1393 H= 348:15 +CPU= 32.314
  06:36:52 STEP 1394 H= 348:30 +CPU= 30.783
  06:37:28 STEP 1395 H= 348:45 +CPU= 31.670
  06:38:03 STEP 1396 H= 349:00 +CPU= 31.600
  06:38:42 STEP 1397 H= 349:15 +CPU= 31.969
  06:39:16 STEP 1398 H= 349:30 +CPU= 31.188
  06:39:51 STEP 1399 H= 349:45 +CPU= 31.158
  06:40:26 STEP 1400 H= 350:00 +CPU= 31.567
  06:41:03 STEP 1401 H= 350:15 +CPU= 32.180
  06:41:38 STEP 1402 H= 350:30 +CPU= 31.089
  06:42:13 STEP 1403 H= 350:45 +CPU= 31.423
  06:42:56 STEP 1404 H= 351:00 +CPU= 39.291
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001404
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001404
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001404
  06:43:32 STEP 1405 H= 351:15 +CPU= 31.610
  06:44:06 STEP 1406 H= 351:30 +CPU= 30.897
  06:44:41 STEP 1407 H= 351:45 +CPU= 31.366
  06:45:16 STEP 1408 H= 352:00 +CPU= 31.414
  06:45:52 STEP 1409 H= 352:15 +CPU= 31.338
  06:46:26 STEP 1410 H= 352:30 +CPU= 31.035
  06:47:02 STEP 1411 H= 352:45 +CPU= 31.578
  06:47:37 STEP 1412 H= 353:00 +CPU= 31.716
  06:48:14 STEP 1413 H= 353:15 +CPU= 32.436
  06:48:48 STEP 1414 H= 353:30 +CPU= 30.918
  06:49:24 STEP 1415 H= 353:45 +CPU= 31.513
  06:50:06 STEP 1416 H= 354:00 +CPU= 38.906
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001416
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001416
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001416
  06:50:43 STEP 1417 H= 354:15 +CPU= 32.395
  06:51:19 STEP 1418 H= 354:30 +CPU= 31.339
  06:51:53 STEP 1419 H= 354:45 +CPU= 31.012
  06:52:32 STEP 1420 H= 355:00 +CPU= 33.854
  06:53:06 STEP 1421 H= 355:15 +CPU= 30.960
  06:53:43 STEP 1422 H= 355:30 +CPU= 32.153
  06:54:17 STEP 1423 H= 355:45 +CPU= 30.947
  06:54:53 STEP 1424 H= 356:00 +CPU= 32.212
  06:55:31 STEP 1425 H= 356:15 +CPU= 32.460
  06:56:06 STEP 1426 H= 356:30 +CPU= 31.125
  06:56:43 STEP 1427 H= 356:45 +CPU= 32.041
  06:57:26 STEP 1428 H= 357:00 +CPU= 38.525
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001428
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001428
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001428
  06:58:04 STEP 1429 H= 357:15 +CPU= 32.665
  06:58:39 STEP 1430 H= 357:30 +CPU= 31.226
  06:59:13 STEP 1431 H= 357:45 +CPU= 31.204
  06:59:51 STEP 1432 H= 358:00 +CPU= 33.309
  07:00:25 STEP 1433 H= 358:15 +CPU= 31.225
  07:01:03 STEP 1434 H= 358:30 +CPU= 32.601
  07:01:37 STEP 1435 H= 358:45 +CPU= 30.815
  07:02:15 STEP 1436 H= 359:00 +CPU= 33.290
  07:02:50 STEP 1437 H= 359:15 +CPU= 32.126
  07:03:20 STEP 1438 H= 359:30 +CPU= 27.461
  07:03:54 STEP 1439 H= 359:45 +CPU= 31.119
  07:04:40 STEP 1440 H= 360:00 +CPU= 39.522
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.5919755575E-21      I       0.6846572012E-05      I
ICloud fractionI      0.0000000000E+00      I       0.3284733432E-01      I
I  Relat. Hum. I      0.3220469974E+00      I       0.3705915396E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.4453755059E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    315596785 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS788992.0 Mbytes/s, TOTAL TIME=    0.0147( 2.7%)
 07:04:40 STEP 1440 H= 360:00 +CPU= 39.522
 END CNT3
 NSTEP =  1440 CNT0   000000000  29.932  29.932  34.085   0.878               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN         2836/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     53    0       34      1044068     2836 ************     6.3     2.5     3.9     3.3    11.2
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 3581366.9 3581366.9    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 3576434.9 3576434.9     99.86      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  105    5132.6    5132.6     15.05      0.00
   9 SPCM     - SPECTRAL COMP.                      96    1088.7    1088.7      2.92      0.00
  10 SCAN2M - PHYSICS                               97   19210.7   19210.7     52.03      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  8     816.3     816.3      0.18      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  5      71.0      71.0      0.01      0.00
  17 GRIDFPOS IN CNT4                                8      43.6      43.6      0.01      0.00
  27 RESTART FILES - WRITING                         2    3254.2    3254.2      0.18      0.00
  28 RESTART FILES - READING                         1    4256.0    4256.0      0.12      0.00
  29 SU4FPOS IN CNT4                                 8       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 8    8287.4    8287.4      1.85      0.00
  31 POSDDH IN STEPO                                 1      30.1      30.1      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     97    6830.2    6830.2     18.50      0.00
  39 SU0YOMB                                         1     472.5     472.5      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                     97      21.9      21.9      0.06      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               96     473.5     473.5      1.27      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               96      85.3      85.3      0.23      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  5       7.6       7.6      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      212     535.0     535.0      3.17      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       200     336.2     336.2      1.88      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        200     104.6     104.6      0.58      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       212     202.4     202.4      1.20      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      56.0      56.0      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            212      34.2      34.2      0.20      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            200      22.7      22.7      0.13      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            212     114.2     114.2      0.68      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            200      35.6      35.6      0.20      0.00
 400 GSTATS                                      12155       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 11263       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     3581.4 CPU TIME    3163.6 VECTOR TIME     3163.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      362.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /BOINC/slots/2/ICMGGh7zg+001440
Moving to projects directory: /BOINC/slots/2/ICMSHh7zg+001440
Moving to projects directory: /BOINC/slots/2/ICMUAh7zg+001440
Adding to the zip: /BOINC/slots/2/NODE.001_01
Adding to the zip: /BOINC/slots/2/ifs.stat
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001356
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001356
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001368
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001368
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001368
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001380
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001380
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001380
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001392
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001392
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001392
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001404
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001404
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001404
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001416
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001416
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001416
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001428
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001428
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001428
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001440
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001440
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001440
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001344
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMSHh7zg+001344
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMUAh7zg+001344
Adding to the zip: /BOINC/projects/climateprediction.net/oifs_43r3_bl_12207386/ICMGGh7zg+001356
Zipping up the final file: /BOINC/projects/climateprediction.net/oifs_43r3_bl_a0gn_2016092300_15_991_12207386_0_r86400562_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
07:08:13 (17466): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:02 1432159 22307558 oifs_43r3_bl_a0gn_2016092300_15_991_12207386_0 1,295,100 48,097 0.0371


©2024 cpdn.org