climateprediction.net home page
Task 22268924

Task 22268924

Name oifs_43r3_ps_0085_1998050100_123_967_12183729_0
Workunit 12183729
Created 22 Dec 2022, 15:17:00 UTC
Sent 3 Jan 2023, 17:25:38 UTC
Report deadline 2 Feb 2023, 17:25:38 UTC
Received 5 Jan 2023, 7:33:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 14 hours 32 min 10 sec
CPU time 12 hours 3 min 13 sec
Validate state Invalid
Credit 2,419.20
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.39 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,769.89 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ss_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1188373:1188373] [20230105:112246:1672885366:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:23:28 STEP 1344 H=1344:00 +CPU= 37.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:24:49 STEP 1345 H=1345:00 +CPU= 21.159
  11:25:11 STEP 1346 H=1346:00 +CPU= 21.314
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:26:36 STEP 1347 H=1347:00 +CPU= 32.971
  11:26:58 STEP 1348 H=1348:00 +CPU= 21.439
  11:27:18 STEP 1349 H=1349:00 +CPU= 20.027
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:28:52 STEP 1350 H=1350:00 +CPU= 34.128
  11:29:13 STEP 1351 H=1351:00 +CPU= 20.485
  11:29:33 STEP 1352 H=1352:00 +CPU= 20.477
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:31:06 STEP 1353 H=1353:00 +CPU= 32.153
  11:31:26 STEP 1354 H=1354:00 +CPU= 20.628
  11:31:46 STEP 1355 H=1355:00 +CPU= 19.410
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:33:33 STEP 1356 H=1356:00 +CPU= 35.897
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001356
  11:33:53 STEP 1357 H=1357:00 +CPU= 20.420
  11:34:14 STEP 1358 H=1358:00 +CPU= 20.441
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:58 STEP 1359 H=1359:00 +CPU= 32.869
  11:36:19 STEP 1360 H=1360:00 +CPU= 20.790
  11:36:39 STEP 1361 H=1361:00 +CPU= 20.412
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:38:23 STEP 1362 H=1362:00 +CPU= 32.618
  11:38:44 STEP 1363 H=1363:00 +CPU= 20.602
  11:39:05 STEP 1364 H=1364:00 +CPU= 21.118
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:40:28 STEP 1365 H=1365:00 +CPU= 32.861
  11:40:50 STEP 1366 H=1366:00 +CPU= 21.337
  11:41:11 STEP 1367 H=1367:00 +CPU= 20.900
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:42:59 STEP 1368 H=1368:00 +CPU= 38.596
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_56.zip
Uploading the intermediate file: upload_file_56.zip
  11:43:20 STEP 1369 H=1369:00 +CPU= 20.200
  11:43:40 STEP 1370 H=1370:00 +CPU= 20.605
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:45:13 STEP 1371 H=1371:00 +CPU= 32.771
  11:45:35 STEP 1372 H=1372:00 +CPU= 20.994
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:48 STEP 1373 H=1373:00 +CPU= 21.870
  11:47:21 STEP 1374 H=1374:00 +CPU= 33.239
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:48:53 STEP 1375 H=1375:00 +CPU= 20.542
  11:49:15 STEP 1376 H=1376:00 +CPU= 21.770
  11:49:47 STEP 1377 H=1377:00 +CPU= 32.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:50:59 STEP 1378 H=1378:00 +CPU= 20.801
  11:51:21 STEP 1379 H=1379:00 +CPU= 21.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:52:48 STEP 1380 H=1380:00 +CPU= 35.725
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001380
  11:53:08 STEP 1381 H=1381:00 +CPU= 20.492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:54:20 STEP 1382 H=1382:00 +CPU= 20.951
  11:54:53 STEP 1383 H=1383:00 +CPU= 32.817
  11:55:14 STEP 1384 H=1384:00 +CPU= 20.769
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:56:36 STEP 1385 H=1385:00 +CPU= 20.388
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:50 STEP 1386 H=1386:00 +CPU= 33.380
  11:58:11 STEP 1387 H=1387:00 +CPU= 21.079
  11:58:33 STEP 1388 H=1388:00 +CPU= 21.137
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:00:06 STEP 1389 H=1389:00 +CPU= 32.422
  12:00:27 STEP 1390 H=1390:00 +CPU= 20.708
  12:00:48 STEP 1391 H=1391:00 +CPU= 21.384
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:02:38 STEP 1392 H=1392:00 +CPU= 37.828
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_57.zip
Uploading the intermediate file: upload_file_57.zip
  12:02:58 STEP 1393 H=1393:00 +CPU= 20.096
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:04:01 STEP 1394 H=1394:00 +CPU= 21.353
  12:04:34 STEP 1395 H=1395:00 +CPU= 33.022
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:05:56 STEP 1396 H=1396:00 +CPU= 21.121
  12:06:17 STEP 1397 H=1397:00 +CPU= 21.357
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:41 STEP 1398 H=1398:00 +CPU= 33.156
  12:08:02 STEP 1399 H=1399:00 +CPU= 20.850
  12:08:24 STEP 1400 H=1400:00 +CPU= 21.068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:09:58 STEP 1401 H=1401:00 +CPU= 32.561
  12:10:20 STEP 1402 H=1402:00 +CPU= 21.136
  12:10:40 STEP 1403 H=1403:00 +CPU= 20.000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:12:15 STEP 1404 H=1404:00 +CPU= 35.574
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001404
  12:12:36 STEP 1405 H=1405:00 +CPU= 20.174
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:13:59 STEP 1406 H=1406:00 +CPU= 21.266
  12:14:32 STEP 1407 H=1407:00 +CPU= 32.782
  12:14:53 STEP 1408 H=1408:00 +CPU= 20.791
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:16:15 STEP 1409 H=1409:00 +CPU= 21.432
  12:16:48 STEP 1410 H=1410:00 +CPU= 33.186
  12:17:08 STEP 1411 H=1411:00 +CPU= 20.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:18:42 STEP 1412 H=1412:00 +CPU= 21.416
  12:19:15 STEP 1413 H=1413:00 +CPU= 32.558
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:20:27 STEP 1414 H=1414:00 +CPU= 21.205
  12:20:49 STEP 1415 H=1415:00 +CPU= 21.553
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:21:59 STEP 1416 H=1416:00 +CPU= 38.584
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_58.zip
Uploading the intermediate file: upload_file_58.zip
  12:22:18 STEP 1417 H=1417:00 +CPU= 19.613
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:23:18 STEP 1418 H=1418:00 +CPU= 19.978
  12:23:51 STEP 1419 H=1419:00 +CPU= 32.886
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:25:05 STEP 1420 H=1420:00 +CPU= 21.286
  12:25:27 STEP 1421 H=1421:00 +CPU= 22.230
  12:26:00 STEP 1422 H=1422:00 +CPU= 32.584
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:27:21 STEP 1423 H=1423:00 +CPU= 20.217
  12:27:43 STEP 1424 H=1424:00 +CPU= 21.879
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:29:09 STEP 1425 H=1425:00 +CPU= 33.043
  12:29:29 STEP 1426 H=1426:00 +CPU= 20.610
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:30:40 STEP 1427 H=1427:00 +CPU= 20.267
  12:31:16 STEP 1428 H=1428:00 +CPU= 36.042
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001428
  12:31:36 STEP 1429 H=1429:00 +CPU= 20.397
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:32:59 STEP 1430 H=1430:00 +CPU= 21.437
  12:33:32 STEP 1431 H=1431:00 +CPU= 32.346
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:34:54 STEP 1432 H=1432:00 +CPU= 20.763
  12:35:17 STEP 1433 H=1433:00 +CPU= 21.925
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:36:52 STEP 1434 H=1434:00 +CPU= 33.987
  12:37:12 STEP 1435 H=1435:00 +CPU= 20.484
  12:37:33 STEP 1436 H=1436:00 +CPU= 20.684
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:39:17 STEP 1437 H=1437:00 +CPU= 32.610
  12:39:38 STEP 1438 H=1438:00 +CPU= 20.651
  12:39:59 STEP 1439 H=1439:00 +CPU= 21.164
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:41:49 STEP 1440 H=1440:00 +CPU= 38.359
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_59.zip
Uploading the intermediate file: upload_file_59.zip
  12:42:10 STEP 1441 H=1441:00 +CPU= 20.585
  12:42:31 STEP 1442 H=1442:00 +CPU= 21.044
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:44:14 STEP 1443 H=1443:00 +CPU= 32.020
  12:44:35 STEP 1444 H=1444:00 +CPU= 21.165
  12:44:56 STEP 1445 H=1445:00 +CPU= 20.635
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:46:21 STEP 1446 H=1446:00 +CPU= 33.867
  12:46:42 STEP 1447 H=1447:00 +CPU= 20.833
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:48:06 STEP 1448 H=1448:00 +CPU= 21.871
  12:48:39 STEP 1449 H=1449:00 +CPU= 32.825
  12:49:00 STEP 1450 H=1450:00 +CPU= 20.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:50:21 STEP 1451 H=1451:00 +CPU= 20.640
  12:50:57 STEP 1452 H=1452:00 +CPU= 35.357
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001452
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:52:17 STEP 1453 H=1453:00 +CPU= 20.364
  12:52:39 STEP 1454 H=1454:00 +CPU= 21.112
  12:53:10 STEP 1455 H=1455:00 +CPU= 31.420
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:55:05 STEP 1456 H=1456:00 +CPU= 22.062
  12:55:26 STEP 1457 H=1457:00 +CPU= 20.371
  12:56:00 STEP 1458 H=1458:00 +CPU= 33.718
  12:56:21 STEP 1459 H=1459:00 +CPU= 20.684
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:57:32 STEP 1460 H=1460:00 +CPU= 21.732
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:59:58 STEP 1461 H=1461:00 +CPU= 34.172
  13:00:24 STEP 1462 H=1462:00 +CPU= 24.601
  13:00:49 STEP 1463 H=1463:00 +CPU= 24.928
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:03:04 STEP 1464 H=1464:00 +CPU= 42.499
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_60.zip
Uploading the intermediate file: upload_file_60.zip
  13:03:26 STEP 1465 H=1465:00 +CPU= 21.821
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:05:19 STEP 1466 H=1466:00 +CPU= 22.304
  13:05:55 STEP 1467 H=1467:00 +CPU= 34.683
  13:06:19 STEP 1468 H=1468:00 +CPU= 23.856
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:18:24 STEP 1469 H=1469:00 +CPU= 22.767
  13:19:01 STEP 1470 H=1470:00 +CPU= 36.982
  13:19:24 STEP 1471 H=1471:00 +CPU= 22.345
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:21:20 STEP 1472 H=1472:00 +CPU= 22.807
  13:21:56 STEP 1473 H=1473:00 +CPU= 36.115
  13:22:20 STEP 1474 H=1474:00 +CPU= 23.399
  13:22:43 STEP 1475 H=1475:00 +CPU= 22.344
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0085
(argv4) batchid: 967
(argv5) wuid: 12183729
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/12
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_551a05142064b83051aa66f624140c1b to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0085_1998050100_123_967_12183729.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0085_1998050100_123_967_12183729.zip
ic_ancil_file: ic_ancil_12183729
ifsdata_file: ifsdata_12183729
climate_data_file: clim_data_12183729
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a3e0ba099eb8ef606d3329508d4dd08d to: /var/lib/boinc-client/slots/12/ic_ancil_12183729.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12183729.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c2fd7219f0753e8146ae972f8c779015 to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12183729.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12183729.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12183729.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12183729.zip
Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12183729.xml
Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12183729.xml
-- Model is restarting --
Adjusting last_iter, 1476, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12183729.xml
last_cpu_time: 39526
upload_file_number: 61
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519
The child process has been launched with process id: 1208189
Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1208189:1208189] [20230105:132649:1672892809:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:39:28 STEP 1464 H=1464:00 +CPU= 42.977
  13:39:51 STEP 1465 H=1465:00 +CPU= 22.616
  13:40:15 STEP 1466 H=1466:00 +CPU= 23.388
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:43:44 STEP 1467 H=1467:00 +CPU= 36.561
  13:44:07 STEP 1468 H=1468:00 +CPU= 23.351
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:49:43 STEP 1469 H=1469:00 +CPU= 23.635
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:52:31 STEP 1470 H=1470:00 +CPU= 36.762
  13:52:54 STEP 1471 H=1471:00 +CPU= 22.383
  13:53:17 STEP 1472 H=1472:00 +CPU= 22.315
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:54:51 STEP 1473 H=1473:00 +CPU= 33.199
  13:55:14 STEP 1474 H=1474:00 +CPU= 22.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:56:36 STEP 1475 H=1475:00 +CPU= 20.263
  13:57:12 STEP 1476 H=1476:00 +CPU= 35.590
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001476
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:58:53 STEP 1477 H=1477:00 +CPU= 20.406
  13:59:14 STEP 1478 H=1478:00 +CPU= 20.961
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:00:39 STEP 1479 H=1479:00 +CPU= 32.206
  14:01:00 STEP 1480 H=1480:00 +CPU= 21.610
  14:01:21 STEP 1481 H=1481:00 +CPU= 20.121
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:02:56 STEP 1482 H=1482:00 +CPU= 33.151
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:03:57 STEP 1483 H=1483:00 +CPU= 20.103
  14:04:18 STEP 1484 H=1484:00 +CPU= 20.742
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:05:43 STEP 1485 H=1485:00 +CPU= 32.673
  14:06:04 STEP 1486 H=1486:00 +CPU= 21.424
  14:06:25 STEP 1487 H=1487:00 +CPU= 20.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:08:15 STEP 1488 H=1488:00 +CPU= 39.214
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_61.zip
Uploading the intermediate file: upload_file_61.zip
  14:08:35 STEP 1489 H=1489:00 +CPU= 19.608
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:10:16 STEP 1490 H=1490:00 +CPU= 20.438
  14:10:48 STEP 1491 H=1491:00 +CPU= 32.155
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:12:12 STEP 1492 H=1492:00 +CPU= 21.724
  14:12:32 STEP 1493 H=1493:00 +CPU= 20.499
  14:13:05 STEP 1494 H=1494:00 +CPU= 32.564
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:14:38 STEP 1495 H=1495:00 +CPU= 21.085
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:16:10 STEP 1496 H=1496:00 +CPU= 21.154
  14:16:43 STEP 1497 H=1497:00 +CPU= 32.777
  14:17:03 STEP 1498 H=1498:00 +CPU= 20.691
  14:17:24 STEP 1499 H=1499:00 +CPU= 20.280
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:19:19 STEP 1500 H=1500:00 +CPU= 35.335
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001500
  14:19:40 STEP 1501 H=1501:00 +CPU= 20.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:21:11 STEP 1502 H=1502:00 +CPU= 20.900
  14:21:43 STEP 1503 H=1503:00 +CPU= 32.396
  14:22:04 STEP 1504 H=1504:00 +CPU= 20.530
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:23:36 STEP 1505 H=1505:00 +CPU= 20.477
  14:24:09 STEP 1506 H=1506:00 +CPU= 33.342
  14:24:29 STEP 1507 H=1507:00 +CPU= 20.050
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:26:01 STEP 1508 H=1508:00 +CPU= 21.015
  14:26:34 STEP 1509 H=1509:00 +CPU= 32.096
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:28:17 STEP 1510 H=1510:00 +CPU= 21.544
  14:28:39 STEP 1511 H=1511:00 +CPU= 21.414
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:30:18 STEP 1512 H=1512:00 +CPU= 37.892
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_62.zip
Uploading the intermediate file: upload_file_62.zip
  14:30:38 STEP 1513 H=1513:00 +CPU= 20.163
  14:30:59 STEP 1514 H=1514:00 +CPU= 20.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:32:41 STEP 1515 H=1515:00 +CPU= 32.090
  14:33:02 STEP 1516 H=1516:00 +CPU= 20.660
  14:33:22 STEP 1517 H=1517:00 +CPU= 20.325
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:35:06 STEP 1518 H=1518:00 +CPU= 33.191
  14:35:27 STEP 1519 H=1519:00 +CPU= 20.300
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:36:30 STEP 1520 H=1520:00 +CPU= 21.080
  14:37:03 STEP 1521 H=1521:00 +CPU= 32.681
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:38:26 STEP 1522 H=1522:00 +CPU= 21.202
  14:38:46 STEP 1523 H=1523:00 +CPU= 19.746
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:40:22 STEP 1524 H=1524:00 +CPU= 36.053
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001524
  14:40:43 STEP 1525 H=1525:00 +CPU= 20.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:15 STEP 1526 H=1526:00 +CPU= 20.494
  14:42:48 STEP 1527 H=1527:00 +CPU= 33.242
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:44:01 STEP 1528 H=1528:00 +CPU= 20.943
  14:44:22 STEP 1529 H=1529:00 +CPU= 20.420
  14:44:54 STEP 1530 H=1530:00 +CPU= 32.468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:46:15 STEP 1531 H=1531:00 +CPU= 20.133
  14:46:37 STEP 1532 H=1532:00 +CPU= 21.186
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:48:00 STEP 1533 H=1533:00 +CPU= 32.085
  14:48:21 STEP 1534 H=1534:00 +CPU= 21.371
  14:48:42 STEP 1535 H=1535:00 +CPU= 20.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:50:12 STEP 1536 H=1536:00 +CPU= 38.749
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_63.zip
Uploading the intermediate file: upload_file_63.zip
  14:50:31 STEP 1537 H=1537:00 +CPU= 19.464
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:51:43 STEP 1538 H=1538:00 +CPU= 20.578
  14:52:15 STEP 1539 H=1539:00 +CPU= 32.222
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:53:36 STEP 1540 H=1540:00 +CPU= 20.683
  14:53:57 STEP 1541 H=1541:00 +CPU= 20.446
  14:54:29 STEP 1542 H=1542:00 +CPU= 32.589
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:55:50 STEP 1543 H=1543:00 +CPU= 20.637
  14:56:12 STEP 1544 H=1544:00 +CPU= 21.181
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:57:46 STEP 1545 H=1545:00 +CPU= 32.045
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:59:07 STEP 1546 H=1546:00 +CPU= 20.970
  14:59:28 STEP 1547 H=1547:00 +CPU= 21.126
  15:00:04 STEP 1548 H=1548:00 +CPU= 35.442
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001548
  15:00:24 STEP 1549 H=1549:00 +CPU= 19.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:02:06 STEP 1550 H=1550:00 +CPU= 21.148
  15:02:38 STEP 1551 H=1551:00 +CPU= 31.911
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:03:50 STEP 1552 H=1552:00 +CPU= 20.923
  15:04:10 STEP 1553 H=1553:00 +CPU= 19.716
  15:04:42 STEP 1554 H=1554:00 +CPU= 32.537
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:06:14 STEP 1555 H=1555:00 +CPU= 20.334
  15:06:35 STEP 1556 H=1556:00 +CPU= 20.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:08:18 STEP 1557 H=1557:00 +CPU= 32.448
  15:08:38 STEP 1558 H=1558:00 +CPU= 20.580
  15:09:00 STEP 1559 H=1559:00 +CPU= 21.359
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:10:39 STEP 1560 H=1560:00 +CPU= 37.940
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_64.zip
Uploading the intermediate file: upload_file_64.zip
  15:10:59 STEP 1561 H=1561:00 +CPU= 19.837
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:12:13 STEP 1562 H=1562:00 +CPU= 21.497
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:13:48 STEP 1563 H=1563:00 +CPU= 32.298
  15:14:09 STEP 1564 H=1564:00 +CPU= 20.618
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:15:19 STEP 1565 H=1565:00 +CPU= 20.172
  15:15:52 STEP 1566 H=1566:00 +CPU= 33.228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:17:24 STEP 1567 H=1567:00 +CPU= 20.613
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:18:28 STEP 1568 H=1568:00 +CPU= 21.834
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:19:41 STEP 1569 H=1569:00 +CPU= 32.330
  15:20:03 STEP 1570 H=1570:00 +CPU= 21.114
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:21:16 STEP 1571 H=1571:00 +CPU= 20.948
  15:21:52 STEP 1572 H=1572:00 +CPU= 35.693
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001572
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:23:24 STEP 1573 H=1573:00 +CPU= 20.890
  15:23:45 STEP 1574 H=1574:00 +CPU= 21.146
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:25:40 STEP 1575 H=1575:00 +CPU= 32.817
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:27:03 STEP 1576 H=1576:00 +CPU= 20.797
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0085
(argv4) batchid: 967
(argv5) wuid: 12183729
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/12
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_551a05142064b83051aa66f624140c1b to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0085_1998050100_123_967_12183729.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0085_1998050100_123_967_12183729.zip
ic_ancil_file: ic_ancil_12183729
ifsdata_file: ifsdata_12183729
climate_data_file: clim_data_12183729
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a3e0ba099eb8ef606d3329508d4dd08d to: /var/lib/boinc-client/slots/12/ic_ancil_12183729.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12183729.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c2fd7219f0753e8146ae972f8c779015 to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12183729.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12183729.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12183729.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12183729.zip
Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12183729.xml
Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12183729.xml
-- Model is restarting --
Adjusting last_iter, 1577, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12183729.xml
last_cpu_time: 42397
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519
The child process has been launched with process id: 1226440
Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1226440:1226440] [20230105:152842:1672900122:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:29:21 STEP 1560 H=1560:00 +CPU= 35.854
  15:29:41 STEP 1561 H=1561:00 +CPU= 20.042
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:31:13 STEP 1562 H=1562:00 +CPU= 21.242
  15:31:45 STEP 1563 H=1563:00 +CPU= 31.700
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:33:06 STEP 1564 H=1564:00 +CPU= 20.263
  15:33:27 STEP 1565 H=1565:00 +CPU= 20.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:35:02 STEP 1566 H=1566:00 +CPU= 33.383
  15:35:23 STEP 1567 H=1567:00 +CPU= 20.512
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:36:55 STEP 1568 H=1568:00 +CPU= 20.746
  15:37:27 STEP 1569 H=1569:00 +CPU= 32.286
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:38:40 STEP 1570 H=1570:00 +CPU= 20.666
  15:39:01 STEP 1571 H=1571:00 +CPU= 20.965
  15:39:35 STEP 1572 H=1572:00 +CPU= 34.403
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001572
  15:40:57 STEP 1573 H=1573:00 +CPU= 20.554
  15:41:18 STEP 1574 H=1574:00 +CPU= 20.803
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:43:11 STEP 1575 H=1575:00 +CPU= 32.656
  15:43:32 STEP 1576 H=1576:00 +CPU= 20.778
  15:43:52 STEP 1577 H=1577:00 +CPU= 20.080
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:45:37 STEP 1578 H=1578:00 +CPU= 32.931
  15:45:57 STEP 1579 H=1579:00 +CPU= 19.957
  15:46:18 STEP 1580 H=1580:00 +CPU= 20.675
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:48:11 STEP 1581 H=1581:00 +CPU= 32.261
  15:48:32 STEP 1582 H=1582:00 +CPU= 20.588
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:50:05 STEP 1583 H=1583:00 +CPU= 21.234
  15:50:44 STEP 1584 H=1584:00 +CPU= 38.378
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183729/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0085_1998050100_123_967_12183729_0_r1669971519_65.zip
Uploading the intermediate file: upload_file_65.zip
  15:51:02 STEP 1585 H=1585:00 +CPU= 17.925
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:52:25 STEP 1586 H=1586:00 +CPU= 20.156
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:53:50 STEP 1587 H=1587:00 +CPU= 32.616
  15:54:12 STEP 1588 H=1588:00 +CPU= 21.764
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:55:33 STEP 1589 H=1589:00 +CPU= 20.395
  15:56:07 STEP 1590 H=1590:00 +CPU= 33.049
  15:56:26 STEP 1591 H=1591:00 +CPU= 19.752
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:58:07 STEP 1592 H=1592:00 +CPU= 20.890
  15:58:40 STEP 1593 H=1593:00 +CPU= 31.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:00:23 STEP 1594 H=1594:00 +CPU= 21.808
  16:00:43 STEP 1595 H=1595:00 +CPU= 19.838
  16:01:19 STEP 1596 H=1596:00 +CPU= 35.042
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001596
  16:02:41 STEP 1597 H=1597:00 +CPU= 21.079
  16:03:02 STEP 1598 H=1598:00 +CPU= 20.726
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:07:53 1519251 22268924 oifs_43r3_ps_0085_1998050100_123_967_12183729_0 1,728,000 12,095 0.0070
09 May 2023 16:07:49 1519251 22268924 oifs_43r3_ps_0085_1998050100_123_967_12183729_0 864,000 6,322 0.0073


©2024 cpdn.org