climateprediction.net home page
Task 22264722

Task 22264722

Name oifs_43r3_ps_0152_1994050100_123_963_12179796_0
Workunit 12179796
Created 22 Dec 2022, 4:59:51 UTC
Sent 16 Jan 2023, 20:27:02 UTC
Report deadline 17 Mar 2023, 20:27:02 UTC
Received 18 Jan 2023, 17:14:23 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1537174
Run time 11 hours 46 min 21 sec
CPU time 11 hours 34 min 17 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 2.98 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,409.00 MB
Peak swap size 4,978.13 MB
Peak disk usage 7,674.18 MB
Stderr 
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
e:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:752:752] [20230118:195959:1674043199:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:01:22 STEP  408 H= 408:00 +CPU= 75.842
  20:02:07 STEP  409 H= 409:00 +CPU= 43.671
  20:02:53 STEP  410 H= 410:00 +CPU= 45.556
  20:04:03 STEP  411 H= 411:00 +CPU= 67.951
  20:04:49 STEP  412 H= 412:00 +CPU= 45.227
  20:05:34 STEP  413 H= 413:00 +CPU= 43.372
  20:06:44 STEP  414 H= 414:00 +CPU= 69.840
  20:07:29 STEP  415 H= 415:00 +CPU= 43.847
  20:08:15 STEP  416 H= 416:00 +CPU= 45.277
  20:09:25 STEP  417 H= 417:00 +CPU= 68.156
  20:10:11 STEP  418 H= 418:00 +CPU= 45.396
  20:10:55 STEP  419 H= 419:00 +CPU= 44.010
  20:12:14 STEP  420 H= 420:00 +CPU= 76.224
Moving to projects directory: /root/slots/4/ICMGGhq0f+000420
Moving to projects directory: /root/slots/4/ICMSHhq0f+000420
Moving to projects directory: /root/slots/4/ICMUAhq0f+000420
  20:12:58 STEP  421 H= 421:00 +CPU= 43.861
  20:13:44 STEP  422 H= 422:00 +CPU= 45.057
  20:14:54 STEP  423 H= 423:00 +CPU= 68.074
  20:15:40 STEP  424 H= 424:00 +CPU= 45.440
  20:16:25 STEP  425 H= 425:00 +CPU= 43.728
  20:17:35 STEP  426 H= 426:00 +CPU= 69.483
  20:18:20 STEP  427 H= 427:00 +CPU= 43.732
  20:19:07 STEP  428 H= 428:00 +CPU= 45.425
  20:20:16 STEP  429 H= 429:00 +CPU= 67.771
  20:21:03 STEP  430 H= 430:00 +CPU= 45.194
  20:21:49 STEP  431 H= 431:00 +CPU= 45.094
  20:23:12 STEP  432 H= 432:00 +CPU= 79.977
Moving to projects directory: /root/slots/4/ICMGGhq0f+000432
Moving to projects directory: /root/slots/4/ICMSHhq0f+000432
Moving to projects directory: /root/slots/4/ICMUAhq0f+000432
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000408
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000408
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000408
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000420
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000420
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000420
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_17.zip
Uploading the intermediate file: upload_file_17.zip
  20:23:56 STEP  433 H= 433:00 +CPU= 43.750
  20:24:43 STEP  434 H= 434:00 +CPU= 45.443
  20:25:52 STEP  435 H= 435:00 +CPU= 68.136
  20:26:38 STEP  436 H= 436:00 +CPU= 45.543
  20:27:23 STEP  437 H= 437:00 +CPU= 43.869
  20:28:35 STEP  438 H= 438:00 +CPU= 70.113
  20:29:20 STEP  439 H= 439:00 +CPU= 43.816
  20:30:06 STEP  440 H= 440:00 +CPU= 45.365
  20:31:15 STEP  441 H= 441:00 +CPU= 68.091
  20:32:01 STEP  442 H= 442:00 +CPU= 45.275
  20:32:46 STEP  443 H= 443:00 +CPU= 43.755
  20:34:03 STEP  444 H= 444:00 +CPU= 76.041
Moving to projects directory: /root/slots/4/ICMGGhq0f+000444
Moving to projects directory: /root/slots/4/ICMSHhq0f+000444
Moving to projects directory: /root/slots/4/ICMUAhq0f+000444
  20:34:48 STEP  445 H= 445:00 +CPU= 43.839
  20:35:34 STEP  446 H= 446:00 +CPU= 45.382
  20:36:44 STEP  447 H= 447:00 +CPU= 67.912
  20:37:30 STEP  448 H= 448:00 +CPU= 45.266
  20:38:14 STEP  449 H= 449:00 +CPU= 43.667
  20:39:25 STEP  450 H= 450:00 +CPU= 69.787
  20:40:09 STEP  451 H= 451:00 +CPU= 43.775
  20:40:55 STEP  452 H= 452:00 +CPU= 45.041
  20:42:04 STEP  453 H= 453:00 +CPU= 67.826
  20:42:49 STEP  454 H= 454:00 +CPU= 45.100
  20:43:35 STEP  455 H= 455:00 +CPU= 44.873
  20:44:56 STEP  456 H= 456:00 +CPU= 79.368
Moving to projects directory: /root/slots/4/ICMGGhq0f+000456
Moving to projects directory: /root/slots/4/ICMSHhq0f+000456
Moving to projects directory: /root/slots/4/ICMUAhq0f+000456
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000432
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000432
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000432
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000444
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000444
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000444
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_18.zip
Uploading the intermediate file: upload_file_18.zip
  20:45:41 STEP  457 H= 457:00 +CPU= 43.831
  20:46:27 STEP  458 H= 458:00 +CPU= 45.045
  20:47:35 STEP  459 H= 459:00 +CPU= 67.610
  20:48:20 STEP  460 H= 460:00 +CPU= 44.962
  20:49:05 STEP  461 H= 461:00 +CPU= 43.729
  20:50:15 STEP  462 H= 462:00 +CPU= 69.300
  20:50:58 STEP  463 H= 463:00 +CPU= 43.347
  20:51:44 STEP  464 H= 464:00 +CPU= 45.290
  20:52:53 STEP  465 H= 465:00 +CPU= 67.771
  20:53:39 STEP  466 H= 466:00 +CPU= 45.288
  20:54:23 STEP  467 H= 467:00 +CPU= 43.366
  20:55:40 STEP  468 H= 468:00 +CPU= 76.075
Moving to projects directory: /root/slots/4/ICMGGhq0f+000468
Moving to projects directory: /root/slots/4/ICMSHhq0f+000468
Moving to projects directory: /root/slots/4/ICMUAhq0f+000468
  20:56:23 STEP  469 H= 469:00 +CPU= 43.382
  20:57:09 STEP  470 H= 470:00 +CPU= 45.050
  20:58:17 STEP  471 H= 471:00 +CPU= 67.387
  20:59:03 STEP  472 H= 472:00 +CPU= 44.920
  20:59:46 STEP  473 H= 473:00 +CPU= 43.342
  21:00:57 STEP  474 H= 474:00 +CPU= 69.783
  21:01:40 STEP  475 H= 475:00 +CPU= 43.277
  21:02:27 STEP  476 H= 476:00 +CPU= 45.566
  21:03:35 STEP  477 H= 477:00 +CPU= 67.325
  21:04:20 STEP  478 H= 478:00 +CPU= 44.823
  21:05:05 STEP  479 H= 479:00 +CPU= 44.522
  21:06:26 STEP  480 H= 480:00 +CPU= 79.283
Moving to projects directory: /root/slots/4/ICMGGhq0f+000480
Moving to projects directory: /root/slots/4/ICMSHhq0f+000480
Moving to projects directory: /root/slots/4/ICMUAhq0f+000480
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000456
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000456
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000456
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000468
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000468
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000468
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_19.zip
Uploading the intermediate file: upload_file_19.zip
  21:07:10 STEP  481 H= 481:00 +CPU= 43.373
  21:07:56 STEP  482 H= 482:00 +CPU= 45.261
  21:09:04 STEP  483 H= 483:00 +CPU= 67.514
  21:09:49 STEP  484 H= 484:00 +CPU= 45.122
  21:10:33 STEP  485 H= 485:00 +CPU= 43.527
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0152
(argv4) batchid: 963
(argv5) wuid: 12179796
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12179796
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_12febb48c58e3f5ac2e68ff919771df2 to: /root/slots/4/oifs_43r3_ps_0152_1994050100_123_963_12179796.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0152_1994050100_123_963_12179796.zip
ic_ancil_file: ic_ancil_12179796
ifsdata_file: ifsdata_12179796
climate_data_file: clim_data_12179796
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_bc754af98eb2a5b61c7e93e5b5603f37 to: /root/slots/4/ic_ancil_12179796.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12179796.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1bf8e0079ea47285962d79c8ee3214 to: /root/slots/4/ifsdata/ifsdata_12179796.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12179796.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12179796.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12179796.zip
Checking for progress XML file: /root/slots/4/progress_file_12179796.xml
Opened progress file ok : /root/slots/4/progress_file_12179796.xml
-- Model is restarting --
Adjusting last_iter, 486, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12179796.xml
last_cpu_time: 27718
upload_file_number: 20
last_iter: 479
last_upload: 1728000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749
The child process has been launched with process id: 779
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:779:779] [20230118:211217:1674047537:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:13:39 STEP  480 H= 480:00 +CPU= 75.305
  21:14:23 STEP  481 H= 481:00 +CPU= 43.455
  21:15:08 STEP  482 H= 482:00 +CPU= 45.012
  21:16:17 STEP  483 H= 483:00 +CPU= 67.857
  21:17:03 STEP  484 H= 484:00 +CPU= 45.273
  21:17:47 STEP  485 H= 485:00 +CPU= 43.425
  21:18:57 STEP  486 H= 486:00 +CPU= 69.591
  21:19:41 STEP  487 H= 487:00 +CPU= 43.429
  21:20:26 STEP  488 H= 488:00 +CPU= 44.950
  21:21:35 STEP  489 H= 489:00 +CPU= 67.617
  21:22:20 STEP  490 H= 490:00 +CPU= 45.114
  21:23:04 STEP  491 H= 491:00 +CPU= 43.325
  21:24:20 STEP  492 H= 492:00 +CPU= 75.572
Moving to projects directory: /root/slots/4/ICMGGhq0f+000492
Moving to projects directory: /root/slots/4/ICMSHhq0f+000492
Moving to projects directory: /root/slots/4/ICMUAhq0f+000492
  21:25:04 STEP  493 H= 493:00 +CPU= 43.492
  21:25:49 STEP  494 H= 494:00 +CPU= 45.038
  21:26:57 STEP  495 H= 495:00 +CPU= 67.471
  21:27:43 STEP  496 H= 496:00 +CPU= 44.828
  21:28:26 STEP  497 H= 497:00 +CPU= 43.385
  21:29:37 STEP  498 H= 498:00 +CPU= 69.556
  21:30:21 STEP  499 H= 499:00 +CPU= 43.572
  21:31:06 STEP  500 H= 500:00 +CPU= 44.999
  21:32:15 STEP  501 H= 501:00 +CPU= 67.693
  21:33:00 STEP  502 H= 502:00 +CPU= 44.822
  21:33:45 STEP  503 H= 503:00 +CPU= 44.610
  21:35:05 STEP  504 H= 504:00 +CPU= 79.194
Moving to projects directory: /root/slots/4/ICMGGhq0f+000504
Moving to projects directory: /root/slots/4/ICMSHhq0f+000504
Moving to projects directory: /root/slots/4/ICMUAhq0f+000504
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000480
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000480
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000480
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000492
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000492
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000492
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_20.zip
Uploading the intermediate file: upload_file_20.zip
  21:35:50 STEP  505 H= 505:00 +CPU= 43.377
  21:36:35 STEP  506 H= 506:00 +CPU= 44.922
  21:37:43 STEP  507 H= 507:00 +CPU= 67.407
  21:38:29 STEP  508 H= 508:00 +CPU= 44.908
  21:39:13 STEP  509 H= 509:00 +CPU= 43.341
  21:40:23 STEP  510 H= 510:00 +CPU= 69.465
  21:41:07 STEP  511 H= 511:00 +CPU= 43.473
  21:41:52 STEP  512 H= 512:00 +CPU= 45.051
  21:43:01 STEP  513 H= 513:00 +CPU= 67.659
  21:43:46 STEP  514 H= 514:00 +CPU= 44.908
  21:44:30 STEP  515 H= 515:00 +CPU= 43.607
  21:45:46 STEP  516 H= 516:00 +CPU= 75.511
Moving to projects directory: /root/slots/4/ICMGGhq0f+000516
Moving to projects directory: /root/slots/4/ICMSHhq0f+000516
Moving to projects directory: /root/slots/4/ICMUAhq0f+000516
  21:46:30 STEP  517 H= 517:00 +CPU= 43.563
  21:47:15 STEP  518 H= 518:00 +CPU= 45.164
  21:48:24 STEP  519 H= 519:00 +CPU= 67.614
  21:49:09 STEP  520 H= 520:00 +CPU= 44.988
  21:49:53 STEP  521 H= 521:00 +CPU= 43.244
  21:51:03 STEP  522 H= 522:00 +CPU= 69.429
  21:51:47 STEP  523 H= 523:00 +CPU= 43.369
  21:52:32 STEP  524 H= 524:00 +CPU= 45.002
  21:53:40 STEP  525 H= 525:00 +CPU= 67.476
  21:54:26 STEP  526 H= 526:00 +CPU= 44.920
  21:55:11 STEP  527 H= 527:00 +CPU= 44.628
  21:56:31 STEP  528 H= 528:00 +CPU= 79.030
Moving to projects directory: /root/slots/4/ICMGGhq0f+000528
Moving to projects directory: /root/slots/4/ICMSHhq0f+000528
Moving to projects directory: /root/slots/4/ICMUAhq0f+000528
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000504
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000504
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000504
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000516
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000516
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000516
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_21.zip
Uploading the intermediate file: upload_file_21.zip
  21:57:15 STEP  529 H= 529:00 +CPU= 43.372
  21:58:00 STEP  530 H= 530:00 +CPU= 45.009
  21:59:08 STEP  531 H= 531:00 +CPU= 67.697
  21:59:54 STEP  532 H= 532:00 +CPU= 44.991
  22:00:38 STEP  533 H= 533:00 +CPU= 43.479
  22:01:48 STEP  534 H= 534:00 +CPU= 69.446
  22:02:32 STEP  535 H= 535:00 +CPU= 43.441
  22:03:17 STEP  536 H= 536:00 +CPU= 45.002
  22:04:25 STEP  537 H= 537:00 +CPU= 67.624
  22:05:11 STEP  538 H= 538:00 +CPU= 45.220
  22:05:55 STEP  539 H= 539:00 +CPU= 43.493
  22:07:11 STEP  540 H= 540:00 +CPU= 75.731
Moving to projects directory: /root/slots/4/ICMGGhq0f+000540
Moving to projects directory: /root/slots/4/ICMSHhq0f+000540
Moving to projects directory: /root/slots/4/ICMUAhq0f+000540
  22:07:55 STEP  541 H= 541:00 +CPU= 43.401
  22:08:41 STEP  542 H= 542:00 +CPU= 45.069
  22:09:49 STEP  543 H= 543:00 +CPU= 67.585
  22:10:34 STEP  544 H= 544:00 +CPU= 44.905
  22:11:18 STEP  545 H= 545:00 +CPU= 43.363
  22:12:27 STEP  546 H= 546:00 +CPU= 69.215
  22:13:11 STEP  547 H= 547:00 +CPU= 43.328
  22:13:56 STEP  548 H= 548:00 +CPU= 44.885
  22:15:05 STEP  549 H= 549:00 +CPU= 67.476
  22:15:50 STEP  550 H= 550:00 +CPU= 44.990
  22:16:35 STEP  551 H= 551:00 +CPU= 44.517
  22:17:55 STEP  552 H= 552:00 +CPU= 79.217
Moving to projects directory: /root/slots/4/ICMGGhq0f+000552
Moving to projects directory: /root/slots/4/ICMSHhq0f+000552
Moving to projects directory: /root/slots/4/ICMUAhq0f+000552
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000528
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000528
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000528
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000540
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000540
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000540
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_22.zip
Uploading the intermediate file: upload_file_22.zip
  22:18:39 STEP  553 H= 553:00 +CPU= 43.417
  22:19:25 STEP  554 H= 554:00 +CPU= 44.988
  22:20:33 STEP  555 H= 555:00 +CPU= 67.678
  22:21:19 STEP  556 H= 556:00 +CPU= 45.015
  22:22:03 STEP  557 H= 557:00 +CPU= 43.366
  22:23:13 STEP  558 H= 558:00 +CPU= 69.431
  22:23:57 STEP  559 H= 559:00 +CPU= 43.407
  22:24:42 STEP  560 H= 560:00 +CPU= 44.918
  22:25:51 STEP  561 H= 561:00 +CPU= 67.591
  22:26:36 STEP  562 H= 562:00 +CPU= 44.988
  22:27:20 STEP  563 H= 563:00 +CPU= 43.480
  22:28:36 STEP  564 H= 564:00 +CPU= 75.406
Moving to projects directory: /root/slots/4/ICMGGhq0f+000564
Moving to projects directory: /root/slots/4/ICMSHhq0f+000564
Moving to projects directory: /root/slots/4/ICMUAhq0f+000564
  22:29:20 STEP  565 H= 565:00 +CPU= 43.406
  22:30:05 STEP  566 H= 566:00 +CPU= 45.017
  22:31:13 STEP  567 H= 567:00 +CPU= 67.519
  22:31:59 STEP  568 H= 568:00 +CPU= 44.932
  22:32:42 STEP  569 H= 569:00 +CPU= 43.345
  22:33:52 STEP  570 H= 570:00 +CPU= 69.335
  22:34:36 STEP  571 H= 571:00 +CPU= 43.388
  22:35:21 STEP  572 H= 572:00 +CPU= 45.188
  22:36:29 STEP  573 H= 573:00 +CPU= 67.372
  22:37:15 STEP  574 H= 574:00 +CPU= 44.843
  22:38:00 STEP  575 H= 575:00 +CPU= 44.657
  22:39:21 STEP  576 H= 576:00 +CPU= 79.496
Moving to projects directory: /root/slots/4/ICMGGhq0f+000576
Moving to projects directory: /root/slots/4/ICMSHhq0f+000576
Moving to projects directory: /root/slots/4/ICMUAhq0f+000576
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000552
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000552
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000552
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000564
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000564
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000564
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_23.zip
Uploading the intermediate file: upload_file_23.zip
  22:40:05 STEP  577 H= 577:00 +CPU= 43.412
  22:40:51 STEP  578 H= 578:00 +CPU= 45.020
  22:41:59 STEP  579 H= 579:00 +CPU= 67.531
  22:42:44 STEP  580 H= 580:00 +CPU= 44.942
  22:43:28 STEP  581 H= 581:00 +CPU= 43.441
  22:44:39 STEP  582 H= 582:00 +CPU= 69.504
  22:45:22 STEP  583 H= 583:00 +CPU= 43.370
  22:46:08 STEP  584 H= 584:00 +CPU= 45.033
  22:47:16 STEP  585 H= 585:00 +CPU= 67.569
  22:48:02 STEP  586 H= 586:00 +CPU= 44.967
  22:48:45 STEP  587 H= 587:00 +CPU= 43.434
  22:50:02 STEP  588 H= 588:00 +CPU= 75.701
Moving to projects directory: /root/slots/4/ICMGGhq0f+000588
Moving to projects directory: /root/slots/4/ICMSHhq0f+000588
Moving to projects directory: /root/slots/4/ICMUAhq0f+000588
  22:50:46 STEP  589 H= 589:00 +CPU= 43.429
  22:51:31 STEP  590 H= 590:00 +CPU= 45.059
  22:52:39 STEP  591 H= 591:00 +CPU= 67.595
  22:53:25 STEP  592 H= 592:00 +CPU= 44.974
  22:54:09 STEP  593 H= 593:00 +CPU= 43.395
  22:55:19 STEP  594 H= 594:00 +CPU= 69.426
  22:56:03 STEP  595 H= 595:00 +CPU= 43.401
  22:56:48 STEP  596 H= 596:00 +CPU= 45.014
  22:57:56 STEP  597 H= 597:00 +CPU= 67.367
  22:58:42 STEP  598 H= 598:00 +CPU= 44.910
  22:59:27 STEP  599 H= 599:00 +CPU= 44.998
  23:00:48 STEP  600 H= 600:00 +CPU= 79.272
Moving to projects directory: /root/slots/4/ICMGGhq0f+000600
Moving to projects directory: /root/slots/4/ICMSHhq0f+000600
Moving to projects directory: /root/slots/4/ICMUAhq0f+000600
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000576
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000576
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000576
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000588
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000588
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000588
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_24.zip
Uploading the intermediate file: upload_file_24.zip
  23:01:32 STEP  601 H= 601:00 +CPU= 43.487
  23:02:18 STEP  602 H= 602:00 +CPU= 44.925
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0152
(argv4) batchid: 963
(argv5) wuid: 12179796
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12179796
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_12febb48c58e3f5ac2e68ff919771df2 to: /root/slots/4/oifs_43r3_ps_0152_1994050100_123_963_12179796.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0152_1994050100_123_963_12179796.zip
ic_ancil_file: ic_ancil_12179796
ifsdata_file: ifsdata_12179796
climate_data_file: clim_data_12179796
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_bc754af98eb2a5b61c7e93e5b5603f37 to: /root/slots/4/ic_ancil_12179796.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12179796.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1bf8e0079ea47285962d79c8ee3214 to: /root/slots/4/ifsdata/ifsdata_12179796.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12179796.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12179796.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12179796.zip
Checking for progress XML file: /root/slots/4/progress_file_12179796.xml
Opened progress file ok : /root/slots/4/progress_file_12179796.xml
-- Model is restarting --
Adjusting last_iter, 603, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12179796.xml
last_cpu_time: 34280
upload_file_number: 25
last_iter: 599
last_upload: 2160000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749
The child process has been launched with process id: 857
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:857:857] [20230118:230802:1674054482:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:09:24 STEP  600 H= 600:00 +CPU= 75.393
  23:10:07 STEP  601 H= 601:00 +CPU= 43.322
  23:10:53 STEP  602 H= 602:00 +CPU= 45.042
  23:12:02 STEP  603 H= 603:00 +CPU= 67.831
  23:12:47 STEP  604 H= 604:00 +CPU= 44.866
  23:13:31 STEP  605 H= 605:00 +CPU= 43.342
  23:14:41 STEP  606 H= 606:00 +CPU= 69.412
  23:15:25 STEP  607 H= 607:00 +CPU= 43.279
  23:16:10 STEP  608 H= 608:00 +CPU= 45.060
  23:17:19 STEP  609 H= 609:00 +CPU= 67.995
  23:18:04 STEP  610 H= 610:00 +CPU= 44.916
  23:18:48 STEP  611 H= 611:00 +CPU= 43.681
  23:20:04 STEP  612 H= 612:00 +CPU= 75.327
Moving to projects directory: /root/slots/4/ICMGGhq0f+000612
Moving to projects directory: /root/slots/4/ICMSHhq0f+000612
Moving to projects directory: /root/slots/4/ICMUAhq0f+000612
  23:20:48 STEP  613 H= 613:00 +CPU= 43.423
  23:21:34 STEP  614 H= 614:00 +CPU= 44.983
  23:22:41 STEP  615 H= 615:00 +CPU= 67.359
  23:23:27 STEP  616 H= 616:00 +CPU= 44.907
  23:24:11 STEP  617 H= 617:00 +CPU= 43.427
  23:25:21 STEP  618 H= 618:00 +CPU= 69.545
  23:26:05 STEP  619 H= 619:00 +CPU= 43.203
  23:26:49 STEP  620 H= 620:00 +CPU= 44.755
  23:27:57 STEP  621 H= 621:00 +CPU= 67.367
  23:28:43 STEP  622 H= 622:00 +CPU= 45.083
  23:29:28 STEP  623 H= 623:00 +CPU= 44.489
  23:30:49 STEP  624 H= 624:00 +CPU= 79.280
Moving to projects directory: /root/slots/4/ICMGGhq0f+000624
Moving to projects directory: /root/slots/4/ICMSHhq0f+000624
Moving to projects directory: /root/slots/4/ICMUAhq0f+000624
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000600
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000600
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000600
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000612
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000612
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000612
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_25.zip
Uploading the intermediate file: upload_file_25.zip
  23:31:33 STEP  625 H= 625:00 +CPU= 43.341
  23:32:18 STEP  626 H= 626:00 +CPU= 45.188
  23:33:27 STEP  627 H= 627:00 +CPU= 67.900
  23:34:13 STEP  628 H= 628:00 +CPU= 45.486
  23:34:57 STEP  629 H= 629:00 +CPU= 43.437
  23:36:07 STEP  630 H= 630:00 +CPU= 69.426
  23:36:51 STEP  631 H= 631:00 +CPU= 43.680
  23:37:37 STEP  632 H= 632:00 +CPU= 45.176
  23:38:45 STEP  633 H= 633:00 +CPU= 67.752
  23:39:31 STEP  634 H= 634:00 +CPU= 45.210
  23:40:15 STEP  635 H= 635:00 +CPU= 43.398
  23:41:32 STEP  636 H= 636:00 +CPU= 76.087
Moving to projects directory: /root/slots/4/ICMGGhq0f+000636
Moving to projects directory: /root/slots/4/ICMSHhq0f+000636
Moving to projects directory: /root/slots/4/ICMUAhq0f+000636
  23:42:16 STEP  637 H= 637:00 +CPU= 44.013
  23:43:01 STEP  638 H= 638:00 +CPU= 44.967
  23:44:10 STEP  639 H= 639:00 +CPU= 68.175
  23:44:55 STEP  640 H= 640:00 +CPU= 44.836
  23:45:39 STEP  641 H= 641:00 +CPU= 43.457
  23:46:49 STEP  642 H= 642:00 +CPU= 69.290
  23:47:33 STEP  643 H= 643:00 +CPU= 43.310
  23:48:19 STEP  644 H= 644:00 +CPU= 45.105
  23:49:26 STEP  645 H= 645:00 +CPU= 67.318
  23:50:12 STEP  646 H= 646:00 +CPU= 45.161
  23:50:57 STEP  647 H= 647:00 +CPU= 44.893
  23:52:18 STEP  648 H= 648:00 +CPU= 79.744
Moving to projects directory: /root/slots/4/ICMGGhq0f+000648
Moving to projects directory: /root/slots/4/ICMSHhq0f+000648
Moving to projects directory: /root/slots/4/ICMUAhq0f+000648
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000624
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000624
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000624
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000636
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000636
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000636
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_26.zip
Uploading the intermediate file: upload_file_26.zip
  23:53:01 STEP  649 H= 649:00 +CPU= 43.567
  23:53:47 STEP  650 H= 650:00 +CPU= 45.301
  23:54:55 STEP  651 H= 651:00 +CPU= 67.976
  23:55:40 STEP  652 H= 652:00 +CPU= 45.165
  23:56:24 STEP  653 H= 653:00 +CPU= 43.465
  23:57:34 STEP  654 H= 654:00 +CPU= 69.453
  23:58:17 STEP  655 H= 655:00 +CPU= 43.659
  23:59:03 STEP  656 H= 656:00 +CPU= 45.265
  00:00:12 STEP  657 H= 657:00 +CPU= 67.910
  00:00:57 STEP  658 H= 658:00 +CPU= 45.166
  00:01:41 STEP  659 H= 659:00 +CPU= 43.752
  00:02:57 STEP  660 H= 660:00 +CPU= 75.989
Moving to projects directory: /root/slots/4/ICMGGhq0f+000660
Moving to projects directory: /root/slots/4/ICMSHhq0f+000660
Moving to projects directory: /root/slots/4/ICMUAhq0f+000660
  00:03:41 STEP  661 H= 661:00 +CPU= 43.600
  00:04:26 STEP  662 H= 662:00 +CPU= 45.069
  00:05:34 STEP  663 H= 663:00 +CPU= 67.839
  00:06:20 STEP  664 H= 664:00 +CPU= 45.144
  00:07:03 STEP  665 H= 665:00 +CPU= 43.508
  00:08:14 STEP  666 H= 666:00 +CPU= 69.834
  00:08:57 STEP  667 H= 667:00 +CPU= 43.534
  00:09:43 STEP  668 H= 668:00 +CPU= 45.138
  00:10:51 STEP  669 H= 669:00 +CPU= 67.800
  00:11:36 STEP  670 H= 670:00 +CPU= 45.176
  00:12:21 STEP  671 H= 671:00 +CPU= 44.893
  00:13:42 STEP  672 H= 672:00 +CPU= 79.947
Moving to projects directory: /root/slots/4/ICMGGhq0f+000672
Moving to projects directory: /root/slots/4/ICMSHhq0f+000672
Moving to projects directory: /root/slots/4/ICMUAhq0f+000672
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000648
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000648
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000648
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000660
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000660
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000660
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_27.zip
Uploading the intermediate file: upload_file_27.zip
  00:14:25 STEP  673 H= 673:00 +CPU= 43.399
  00:15:11 STEP  674 H= 674:00 +CPU= 45.066
  00:16:19 STEP  675 H= 675:00 +CPU= 67.860
  00:17:04 STEP  676 H= 676:00 +CPU= 45.110
  00:17:48 STEP  677 H= 677:00 +CPU= 43.533
  00:18:58 STEP  678 H= 678:00 +CPU= 69.746
  00:19:42 STEP  679 H= 679:00 +CPU= 43.577
  00:20:27 STEP  680 H= 680:00 +CPU= 45.312
  00:21:36 STEP  681 H= 681:00 +CPU= 68.069
  00:22:21 STEP  682 H= 682:00 +CPU= 45.227
  00:23:05 STEP  683 H= 683:00 +CPU= 43.711
  00:24:21 STEP  684 H= 684:00 +CPU= 75.982
Moving to projects directory: /root/slots/4/ICMGGhq0f+000684
Moving to projects directory: /root/slots/4/ICMSHhq0f+000684
Moving to projects directory: /root/slots/4/ICMUAhq0f+000684
  00:25:05 STEP  685 H= 685:00 +CPU= 43.505
  00:25:50 STEP  686 H= 686:00 +CPU= 45.162
  00:26:58 STEP  687 H= 687:00 +CPU= 67.894
  00:27:44 STEP  688 H= 688:00 +CPU= 45.121
  00:28:27 STEP  689 H= 689:00 +CPU= 43.593
  00:29:37 STEP  690 H= 690:00 +CPU= 69.673
  00:30:21 STEP  691 H= 691:00 +CPU= 43.490
  00:31:06 STEP  692 H= 692:00 +CPU= 45.149
  00:32:14 STEP  693 H= 693:00 +CPU= 67.230
  00:32:59 STEP  694 H= 694:00 +CPU= 44.976
  00:33:44 STEP  695 H= 695:00 +CPU= 44.782
  00:35:05 STEP  696 H= 696:00 +CPU= 79.890
Moving to projects directory: /root/slots/4/ICMGGhq0f+000696
Moving to projects directory: /root/slots/4/ICMSHhq0f+000696
Moving to projects directory: /root/slots/4/ICMUAhq0f+000696
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000672
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000672
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000672
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000684
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000684
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000684
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_28.zip
Uploading the intermediate file: upload_file_28.zip
  00:35:48 STEP  697 H= 697:00 +CPU= 43.453
  00:36:33 STEP  698 H= 698:00 +CPU= 45.042
  00:37:41 STEP  699 H= 699:00 +CPU= 67.785
  00:38:27 STEP  700 H= 700:00 +CPU= 45.074
  00:39:10 STEP  701 H= 701:00 +CPU= 43.522
  00:40:20 STEP  702 H= 702:00 +CPU= 69.593
  00:41:04 STEP  703 H= 703:00 +CPU= 43.407
  00:41:49 STEP  704 H= 704:00 +CPU= 45.085
  00:42:57 STEP  705 H= 705:00 +CPU= 67.853
  00:43:42 STEP  706 H= 706:00 +CPU= 45.125
  00:44:26 STEP  707 H= 707:00 +CPU= 43.594
  00:45:42 STEP  708 H= 708:00 +CPU= 75.980
Moving to projects directory: /root/slots/4/ICMGGhq0f+000708
Moving to projects directory: /root/slots/4/ICMSHhq0f+000708
Moving to projects directory: /root/slots/4/ICMUAhq0f+000708
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0152
(argv4) batchid: 963
(argv5) wuid: 12179796
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12179796
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_12febb48c58e3f5ac2e68ff919771df2 to: /root/slots/4/oifs_43r3_ps_0152_1994050100_123_963_12179796.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0152_1994050100_123_963_12179796.zip
ic_ancil_file: ic_ancil_12179796
ifsdata_file: ifsdata_12179796
climate_data_file: clim_data_12179796
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_bc754af98eb2a5b61c7e93e5b5603f37 to: /root/slots/4/ic_ancil_12179796.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12179796.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1bf8e0079ea47285962d79c8ee3214 to: /root/slots/4/ifsdata/ifsdata_12179796.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12179796.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12179796.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12179796.zip
Checking for progress XML file: /root/slots/4/progress_file_12179796.xml
Opened progress file ok : /root/slots/4/progress_file_12179796.xml
-- Model is restarting --
Adjusting last_iter, 709, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12179796.xml
last_cpu_time: 40119
upload_file_number: 29
last_iter: 695
last_upload: 2505600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749
The child process has been launched with process id: 891
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:891:891] [20230119:004712:1674060432:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:48:33 STEP  696 H= 696:00 +CPU= 75.746
  00:49:17 STEP  697 H= 697:00 +CPU= 43.643
  00:50:03 STEP  698 H= 698:00 +CPU= 45.171
  00:51:11 STEP  699 H= 699:00 +CPU= 68.060
  00:51:56 STEP  700 H= 700:00 +CPU= 45.108
  00:52:40 STEP  701 H= 701:00 +CPU= 43.660
  00:53:50 STEP  702 H= 702:00 +CPU= 69.937
  00:54:34 STEP  703 H= 703:00 +CPU= 43.507
  00:55:19 STEP  704 H= 704:00 +CPU= 45.366
  00:56:28 STEP  705 H= 705:00 +CPU= 68.226
  00:57:13 STEP  706 H= 706:00 +CPU= 45.224
  00:57:57 STEP  707 H= 707:00 +CPU= 43.508
  00:59:13 STEP  708 H= 708:00 +CPU= 76.456
Moving to projects directory: /root/slots/4/ICMGGhq0f+000708
Moving to projects directory: /root/slots/4/ICMSHhq0f+000708
Moving to projects directory: /root/slots/4/ICMUAhq0f+000708
  00:59:57 STEP  709 H= 709:00 +CPU= 43.670
  01:00:43 STEP  710 H= 710:00 +CPU= 45.350
  01:01:51 STEP  711 H= 711:00 +CPU= 68.256
  01:02:37 STEP  712 H= 712:00 +CPU= 45.126
  01:03:20 STEP  713 H= 713:00 +CPU= 43.592
  01:04:31 STEP  714 H= 714:00 +CPU= 70.011
  01:05:14 STEP  715 H= 715:00 +CPU= 43.443
  01:06:00 STEP  716 H= 716:00 +CPU= 45.197
  01:07:08 STEP  717 H= 717:00 +CPU= 68.312
  01:07:54 STEP  718 H= 718:00 +CPU= 45.270
  01:08:39 STEP  719 H= 719:00 +CPU= 44.940
  01:10:00 STEP  720 H= 720:00 +CPU= 80.402
Moving to projects directory: /root/slots/4/ICMGGhq0f+000720
Moving to projects directory: /root/slots/4/ICMSHhq0f+000720
Moving to projects directory: /root/slots/4/ICMUAhq0f+000720
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000696
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000696
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000696
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMGGhq0f+000708
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMSHhq0f+000708
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12179796/ICMUAhq0f+000708
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0152_1994050100_123_963_12179796_0_r1141160749_29.zip
Uploading the intermediate file: upload_file_29.zip
  01:10:44 STEP  721 H= 721:00 +CPU= 43.542
  01:11:29 STEP  722 H= 722:00 +CPU= 45.217
  01:12:37 STEP  723 H= 723:00 +CPU= 67.885
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net