climateprediction.net home page
Task 22270833

Task 22270833

Name oifs_43r3_ps_0861_1999050100_123_968_12185505_0
Workunit 12185505
Created 22 Dec 2022, 19:55:14 UTC
Sent 29 Dec 2022, 6:31:31 UTC
Report deadline 28 Jan 2023, 6:31:31 UTC
Received 29 Dec 2022, 20:48:57 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1537174
Run time 11 hours 59 min 29 sec
CPU time 11 hours 54 min 18 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 2.98 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,409.63 MB
Peak swap size 4,978.13 MB
Peak disk usage 7,320.48 MB
Stderr 
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
1229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2087:2087] [20221229:224308:1672324988:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:44:29 STEP  456 H= 456:00 +CPU= 74.899
  22:45:12 STEP  457 H= 457:00 +CPU= 42.958
  22:45:56 STEP  458 H= 458:00 +CPU= 44.537
  22:47:04 STEP  459 H= 459:00 +CPU= 67.265
  22:47:48 STEP  460 H= 460:00 +CPU= 44.450
  22:48:31 STEP  461 H= 461:00 +CPU= 42.939
  22:49:41 STEP  462 H= 462:00 +CPU= 69.105
  22:50:24 STEP  463 H= 463:00 +CPU= 43.038
  22:51:08 STEP  464 H= 464:00 +CPU= 44.544
  22:52:16 STEP  465 H= 465:00 +CPU= 67.266
  22:53:01 STEP  466 H= 466:00 +CPU= 44.589
  22:53:44 STEP  467 H= 467:00 +CPU= 42.966
  22:54:59 STEP  468 H= 468:00 +CPU= 75.239
Moving to projects directory: /root/slots/4/ICMGGhq0f+000468
Moving to projects directory: /root/slots/4/ICMSHhq0f+000468
Moving to projects directory: /root/slots/4/ICMUAhq0f+000468
  22:55:42 STEP  469 H= 469:00 +CPU= 43.060
  22:56:27 STEP  470 H= 470:00 +CPU= 44.649
  22:57:35 STEP  471 H= 471:00 +CPU= 67.199
  22:58:19 STEP  472 H= 472:00 +CPU= 44.580
  22:59:02 STEP  473 H= 473:00 +CPU= 42.932
  23:00:12 STEP  474 H= 474:00 +CPU= 69.071
  23:00:55 STEP  475 H= 475:00 +CPU= 42.905
  23:01:39 STEP  476 H= 476:00 +CPU= 44.530
  23:02:47 STEP  477 H= 477:00 +CPU= 67.264
  23:03:31 STEP  478 H= 478:00 +CPU= 44.486
  23:04:16 STEP  479 H= 479:00 +CPU= 44.222
  23:05:36 STEP  480 H= 480:00 +CPU= 79.183
Moving to projects directory: /root/slots/4/ICMGGhq0f+000480
Moving to projects directory: /root/slots/4/ICMSHhq0f+000480
Moving to projects directory: /root/slots/4/ICMUAhq0f+000480
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000456
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000456
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000456
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000468
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000468
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000468
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_19.zip
Uploading the intermediate file: upload_file_19.zip
  23:06:19 STEP  481 H= 481:00 +CPU= 43.014
  23:07:03 STEP  482 H= 482:00 +CPU= 44.497
  23:08:11 STEP  483 H= 483:00 +CPU= 67.310
  23:08:55 STEP  484 H= 484:00 +CPU= 44.521
  23:09:39 STEP  485 H= 485:00 +CPU= 43.032
  23:10:48 STEP  486 H= 486:00 +CPU= 69.200
  23:11:31 STEP  487 H= 487:00 +CPU= 42.960
  23:12:16 STEP  488 H= 488:00 +CPU= 44.607
  23:13:23 STEP  489 H= 489:00 +CPU= 67.322
  23:14:08 STEP  490 H= 490:00 +CPU= 44.626
  23:14:51 STEP  491 H= 491:00 +CPU= 43.014
  23:16:07 STEP  492 H= 492:00 +CPU= 75.419
Moving to projects directory: /root/slots/4/ICMGGhq0f+000492
Moving to projects directory: /root/slots/4/ICMSHhq0f+000492
Moving to projects directory: /root/slots/4/ICMUAhq0f+000492
  23:16:50 STEP  493 H= 493:00 +CPU= 43.005
  23:17:35 STEP  494 H= 494:00 +CPU= 44.619
  23:18:42 STEP  495 H= 495:00 +CPU= 67.271
  23:19:27 STEP  496 H= 496:00 +CPU= 44.545
  23:20:10 STEP  497 H= 497:00 +CPU= 42.921
  23:21:19 STEP  498 H= 498:00 +CPU= 69.025
  23:22:02 STEP  499 H= 499:00 +CPU= 42.891
  23:22:47 STEP  500 H= 500:00 +CPU= 44.475
  23:23:54 STEP  501 H= 501:00 +CPU= 67.188
  23:24:39 STEP  502 H= 502:00 +CPU= 44.465
  23:25:23 STEP  503 H= 503:00 +CPU= 44.160
  23:26:43 STEP  504 H= 504:00 +CPU= 78.854
Moving to projects directory: /root/slots/4/ICMGGhq0f+000504
Moving to projects directory: /root/slots/4/ICMSHhq0f+000504
Moving to projects directory: /root/slots/4/ICMUAhq0f+000504
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000480
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000480
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000480
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000492
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000492
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000492
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_20.zip
Uploading the intermediate file: upload_file_20.zip
  23:27:26 STEP  505 H= 505:00 +CPU= 42.889
  23:28:10 STEP  506 H= 506:00 +CPU= 44.477
  23:29:18 STEP  507 H= 507:00 +CPU= 67.081
  23:30:02 STEP  508 H= 508:00 +CPU= 44.528
  23:30:45 STEP  509 H= 509:00 +CPU= 42.930
  23:31:54 STEP  510 H= 510:00 +CPU= 68.938
  23:32:38 STEP  511 H= 511:00 +CPU= 43.062
  23:33:22 STEP  512 H= 512:00 +CPU= 44.562
  23:34:30 STEP  513 H= 513:00 +CPU= 67.187
  23:35:14 STEP  514 H= 514:00 +CPU= 44.529
  23:35:57 STEP  515 H= 515:00 +CPU= 43.031
  23:37:13 STEP  516 H= 516:00 +CPU= 75.300
Moving to projects directory: /root/slots/4/ICMGGhq0f+000516
Moving to projects directory: /root/slots/4/ICMSHhq0f+000516
Moving to projects directory: /root/slots/4/ICMUAhq0f+000516
  23:37:56 STEP  517 H= 517:00 +CPU= 42.939
  23:38:41 STEP  518 H= 518:00 +CPU= 44.623
  23:39:48 STEP  519 H= 519:00 +CPU= 67.208
  23:40:33 STEP  520 H= 520:00 +CPU= 44.537
  23:41:16 STEP  521 H= 521:00 +CPU= 42.924
  23:42:25 STEP  522 H= 522:00 +CPU= 68.971
  23:43:08 STEP  523 H= 523:00 +CPU= 42.938
  23:43:53 STEP  524 H= 524:00 +CPU= 44.482
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0861
(argv4) batchid: 968
(argv5) wuid: 12185505
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12185505
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_03024ee4c3f6568ca18d33402810c719 to: /root/slots/4/oifs_43r3_ps_0861_1999050100_123_968_12185505.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0861_1999050100_123_968_12185505.zip
ic_ancil_file: ic_ancil_12185505
ifsdata_file: ifsdata_12185505
climate_data_file: clim_data_12185505
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_517498dc4cf7dd16c7f04f870fb607e1 to: /root/slots/4/ic_ancil_12185505.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12185505.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2c7998878ed23906df8a45ca5209cedb to: /root/slots/4/ifsdata/ifsdata_12185505.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12185505.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12185505.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12185505.zip
Checking for progress XML file: /root/slots/4/progress_file_12185505.xml
Opened progress file ok : /root/slots/4/progress_file_12185505.xml
-- Model is restarting --
Adjusting last_iter, 525, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12185505.xml
last_cpu_time: 28490
upload_file_number: 21
last_iter: 503
last_upload: 1814400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318
The child process has been launched with process id: 2110
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2110:2110] [20221229:234530:1672328730:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:46:49 STEP  504 H= 504:00 +CPU= 74.142
  23:47:32 STEP  505 H= 505:00 +CPU= 42.659
  23:48:17 STEP  506 H= 506:00 +CPU= 44.274
  23:49:23 STEP  507 H= 507:00 +CPU= 66.465
  23:50:08 STEP  508 H= 508:00 +CPU= 44.284
  23:50:50 STEP  509 H= 509:00 +CPU= 42.631
  23:51:59 STEP  510 H= 510:00 +CPU= 68.327
  23:52:42 STEP  511 H= 511:00 +CPU= 42.705
  23:53:26 STEP  512 H= 512:00 +CPU= 44.340
  23:54:33 STEP  513 H= 513:00 +CPU= 66.642
  23:55:18 STEP  514 H= 514:00 +CPU= 44.428
  23:56:01 STEP  515 H= 515:00 +CPU= 42.722
  23:57:15 STEP  516 H= 516:00 +CPU= 74.495
Moving to projects directory: /root/slots/4/ICMGGhq0f+000516
Moving to projects directory: /root/slots/4/ICMSHhq0f+000516
Moving to projects directory: /root/slots/4/ICMUAhq0f+000516
  23:57:58 STEP  517 H= 517:00 +CPU= 42.787
  23:58:43 STEP  518 H= 518:00 +CPU= 44.436
  23:59:49 STEP  519 H= 519:00 +CPU= 66.598
  00:00:34 STEP  520 H= 520:00 +CPU= 44.518
  00:01:17 STEP  521 H= 521:00 +CPU= 42.818
  00:02:26 STEP  522 H= 522:00 +CPU= 68.438
  00:03:08 STEP  523 H= 523:00 +CPU= 42.663
  00:03:53 STEP  524 H= 524:00 +CPU= 44.270
  00:05:00 STEP  525 H= 525:00 +CPU= 66.533
  00:05:44 STEP  526 H= 526:00 +CPU= 44.250
  00:06:28 STEP  527 H= 527:00 +CPU= 43.957
  00:07:47 STEP  528 H= 528:00 +CPU= 78.360
Moving to projects directory: /root/slots/4/ICMGGhq0f+000528
Moving to projects directory: /root/slots/4/ICMSHhq0f+000528
Moving to projects directory: /root/slots/4/ICMUAhq0f+000528
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000504
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000504
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000504
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000516
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000516
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000516
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_21.zip
Uploading the intermediate file: upload_file_21.zip
  00:08:30 STEP  529 H= 529:00 +CPU= 42.815
  00:09:15 STEP  530 H= 530:00 +CPU= 44.288
  00:10:22 STEP  531 H= 531:00 +CPU= 66.591
  00:11:06 STEP  532 H= 532:00 +CPU= 44.373
  00:11:49 STEP  533 H= 533:00 +CPU= 42.858
  00:12:58 STEP  534 H= 534:00 +CPU= 68.568
  00:13:41 STEP  535 H= 535:00 +CPU= 42.859
  00:14:25 STEP  536 H= 536:00 +CPU= 44.473
  00:15:32 STEP  537 H= 537:00 +CPU= 66.700
  00:16:17 STEP  538 H= 538:00 +CPU= 44.446
  00:17:00 STEP  539 H= 539:00 +CPU= 42.857
  00:18:15 STEP  540 H= 540:00 +CPU= 74.710
Moving to projects directory: /root/slots/4/ICMGGhq0f+000540
Moving to projects directory: /root/slots/4/ICMSHhq0f+000540
Moving to projects directory: /root/slots/4/ICMUAhq0f+000540
  00:18:58 STEP  541 H= 541:00 +CPU= 42.852
  00:19:42 STEP  542 H= 542:00 +CPU= 44.453
  00:20:49 STEP  543 H= 543:00 +CPU= 66.563
  00:21:34 STEP  544 H= 544:00 +CPU= 44.371
  00:22:17 STEP  545 H= 545:00 +CPU= 42.768
  00:23:25 STEP  546 H= 546:00 +CPU= 68.401
  00:24:08 STEP  547 H= 547:00 +CPU= 42.738
  00:24:52 STEP  548 H= 548:00 +CPU= 44.297
  00:25:59 STEP  549 H= 549:00 +CPU= 66.561
  00:26:44 STEP  550 H= 550:00 +CPU= 44.305
  00:27:28 STEP  551 H= 551:00 +CPU= 43.942
  00:28:47 STEP  552 H= 552:00 +CPU= 78.403
Moving to projects directory: /root/slots/4/ICMGGhq0f+000552
Moving to projects directory: /root/slots/4/ICMSHhq0f+000552
Moving to projects directory: /root/slots/4/ICMUAhq0f+000552
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000528
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000528
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000528
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000540
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000540
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000540
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_22.zip
Uploading the intermediate file: upload_file_22.zip
  00:29:30 STEP  553 H= 553:00 +CPU= 42.892
  00:30:14 STEP  554 H= 554:00 +CPU= 44.303
  00:31:21 STEP  555 H= 555:00 +CPU= 66.669
  00:32:06 STEP  556 H= 556:00 +CPU= 44.352
  00:32:49 STEP  557 H= 557:00 +CPU= 42.780
  00:33:58 STEP  558 H= 558:00 +CPU= 68.610
  00:34:41 STEP  559 H= 559:00 +CPU= 42.881
  00:35:25 STEP  560 H= 560:00 +CPU= 44.456
  00:36:32 STEP  561 H= 561:00 +CPU= 66.763
  00:37:17 STEP  562 H= 562:00 +CPU= 44.379
  00:37:59 STEP  563 H= 563:00 +CPU= 42.795
  00:39:15 STEP  564 H= 564:00 +CPU= 74.808
Moving to projects directory: /root/slots/4/ICMGGhq0f+000564
Moving to projects directory: /root/slots/4/ICMSHhq0f+000564
Moving to projects directory: /root/slots/4/ICMUAhq0f+000564
  00:39:58 STEP  565 H= 565:00 +CPU= 42.900
  00:40:42 STEP  566 H= 566:00 +CPU= 44.510
  00:41:49 STEP  567 H= 567:00 +CPU= 66.786
  00:42:34 STEP  568 H= 568:00 +CPU= 44.431
  00:43:17 STEP  569 H= 569:00 +CPU= 42.827
  00:44:26 STEP  570 H= 570:00 +CPU= 68.607
  00:45:09 STEP  571 H= 571:00 +CPU= 42.889
  00:45:53 STEP  572 H= 572:00 +CPU= 44.313
  00:47:00 STEP  573 H= 573:00 +CPU= 66.752
  00:47:45 STEP  574 H= 574:00 +CPU= 44.319
  00:48:29 STEP  575 H= 575:00 +CPU= 44.044
  00:49:48 STEP  576 H= 576:00 +CPU= 78.552
Moving to projects directory: /root/slots/4/ICMGGhq0f+000576
Moving to projects directory: /root/slots/4/ICMSHhq0f+000576
Moving to projects directory: /root/slots/4/ICMUAhq0f+000576
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000552
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000552
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000552
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000564
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000564
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000564
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_23.zip
Uploading the intermediate file: upload_file_23.zip
  00:50:31 STEP  577 H= 577:00 +CPU= 42.702
  00:51:15 STEP  578 H= 578:00 +CPU= 44.332
  00:52:22 STEP  579 H= 579:00 +CPU= 66.653
  00:53:07 STEP  580 H= 580:00 +CPU= 44.305
  00:53:49 STEP  581 H= 581:00 +CPU= 42.721
  00:54:58 STEP  582 H= 582:00 +CPU= 68.474
  00:55:41 STEP  583 H= 583:00 +CPU= 42.808
  00:56:26 STEP  584 H= 584:00 +CPU= 44.444
  00:57:33 STEP  585 H= 585:00 +CPU= 66.780
  00:58:17 STEP  586 H= 586:00 +CPU= 44.416
  00:59:00 STEP  587 H= 587:00 +CPU= 42.788
  01:00:15 STEP  588 H= 588:00 +CPU= 74.613
Moving to projects directory: /root/slots/4/ICMGGhq0f+000588
Moving to projects directory: /root/slots/4/ICMSHhq0f+000588
Moving to projects directory: /root/slots/4/ICMUAhq0f+000588
  01:00:58 STEP  589 H= 589:00 +CPU= 42.861
  01:01:42 STEP  590 H= 590:00 +CPU= 44.450
  01:02:49 STEP  591 H= 591:00 +CPU= 66.683
  01:03:34 STEP  592 H= 592:00 +CPU= 44.438
  01:04:17 STEP  593 H= 593:00 +CPU= 42.805
  01:05:25 STEP  594 H= 594:00 +CPU= 68.481
  01:06:08 STEP  595 H= 595:00 +CPU= 42.877
  01:06:53 STEP  596 H= 596:00 +CPU= 44.354
  01:08:00 STEP  597 H= 597:00 +CPU= 66.726
  01:08:44 STEP  598 H= 598:00 +CPU= 44.409
  01:09:28 STEP  599 H= 599:00 +CPU= 44.082
  01:10:48 STEP  600 H= 600:00 +CPU= 78.696
Moving to projects directory: /root/slots/4/ICMGGhq0f+000600
Moving to projects directory: /root/slots/4/ICMSHhq0f+000600
Moving to projects directory: /root/slots/4/ICMUAhq0f+000600
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000576
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000576
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000576
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000588
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000588
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000588
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_24.zip
Uploading the intermediate file: upload_file_24.zip
  01:11:31 STEP  601 H= 601:00 +CPU= 42.886
  01:12:15 STEP  602 H= 602:00 +CPU= 44.379
  01:13:22 STEP  603 H= 603:00 +CPU= 66.714
  01:14:07 STEP  604 H= 604:00 +CPU= 44.390
  01:14:50 STEP  605 H= 605:00 +CPU= 42.836
  01:15:58 STEP  606 H= 606:00 +CPU= 68.540
  01:16:41 STEP  607 H= 607:00 +CPU= 42.913
  01:17:26 STEP  608 H= 608:00 +CPU= 44.520
  01:18:33 STEP  609 H= 609:00 +CPU= 66.702
  01:19:18 STEP  610 H= 610:00 +CPU= 44.494
  01:20:01 STEP  611 H= 611:00 +CPU= 42.843
  01:21:16 STEP  612 H= 612:00 +CPU= 74.740
Moving to projects directory: /root/slots/4/ICMGGhq0f+000612
Moving to projects directory: /root/slots/4/ICMSHhq0f+000612
Moving to projects directory: /root/slots/4/ICMUAhq0f+000612
  01:21:59 STEP  613 H= 613:00 +CPU= 42.822
  01:22:43 STEP  614 H= 614:00 +CPU= 44.436
  01:23:50 STEP  615 H= 615:00 +CPU= 66.641
  01:24:34 STEP  616 H= 616:00 +CPU= 44.385
  01:25:17 STEP  617 H= 617:00 +CPU= 42.796
  01:26:26 STEP  618 H= 618:00 +CPU= 68.465
  01:27:09 STEP  619 H= 619:00 +CPU= 42.792
  01:27:53 STEP  620 H= 620:00 +CPU= 44.325
  01:29:00 STEP  621 H= 621:00 +CPU= 66.753
  01:29:45 STEP  622 H= 622:00 +CPU= 44.331
  01:30:29 STEP  623 H= 623:00 +CPU= 44.054
  01:31:48 STEP  624 H= 624:00 +CPU= 78.820
Moving to projects directory: /root/slots/4/ICMGGhq0f+000624
Moving to projects directory: /root/slots/4/ICMSHhq0f+000624
Moving to projects directory: /root/slots/4/ICMUAhq0f+000624
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000600
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000600
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000600
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000612
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000612
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000612
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_25.zip
Uploading the intermediate file: upload_file_25.zip
  01:32:31 STEP  625 H= 625:00 +CPU= 42.845
  01:33:16 STEP  626 H= 626:00 +CPU= 44.435
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0861
(argv4) batchid: 968
(argv5) wuid: 12185505
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12185505
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_03024ee4c3f6568ca18d33402810c719 to: /root/slots/4/oifs_43r3_ps_0861_1999050100_123_968_12185505.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0861_1999050100_123_968_12185505.zip
ic_ancil_file: ic_ancil_12185505
ifsdata_file: ifsdata_12185505
climate_data_file: clim_data_12185505
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_517498dc4cf7dd16c7f04f870fb607e1 to: /root/slots/4/ic_ancil_12185505.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12185505.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2c7998878ed23906df8a45ca5209cedb to: /root/slots/4/ifsdata/ifsdata_12185505.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12185505.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12185505.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12185505.zip
Checking for progress XML file: /root/slots/4/progress_file_12185505.xml
Opened progress file ok : /root/slots/4/progress_file_12185505.xml
-- Model is restarting --
Adjusting last_iter, 627, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12185505.xml
last_cpu_time: 34969
upload_file_number: 26
last_iter: 623
last_upload: 2246400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318
The child process has been launched with process id: 2144
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2144:2144] [20221230:013457:1672335297:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:36:17 STEP  624 H= 624:00 +CPU= 74.279
  01:37:00 STEP  625 H= 625:00 +CPU= 42.612
  01:37:45 STEP  626 H= 626:00 +CPU= 44.261
  01:38:51 STEP  627 H= 627:00 +CPU= 66.533
  01:39:36 STEP  628 H= 628:00 +CPU= 44.243
  01:40:18 STEP  629 H= 629:00 +CPU= 42.667
  01:41:27 STEP  630 H= 630:00 +CPU= 68.468
  01:42:10 STEP  631 H= 631:00 +CPU= 42.654
  01:42:54 STEP  632 H= 632:00 +CPU= 44.270
  01:44:01 STEP  633 H= 633:00 +CPU= 66.555
  01:44:45 STEP  634 H= 634:00 +CPU= 44.350
  01:45:28 STEP  635 H= 635:00 +CPU= 42.822
  01:46:43 STEP  636 H= 636:00 +CPU= 74.462
Moving to projects directory: /root/slots/4/ICMGGhq0f+000636
Moving to projects directory: /root/slots/4/ICMSHhq0f+000636
Moving to projects directory: /root/slots/4/ICMUAhq0f+000636
  01:47:26 STEP  637 H= 637:00 +CPU= 42.865
  01:48:10 STEP  638 H= 638:00 +CPU= 44.386
  01:49:17 STEP  639 H= 639:00 +CPU= 66.716
  01:50:02 STEP  640 H= 640:00 +CPU= 44.383
  01:50:45 STEP  641 H= 641:00 +CPU= 42.755
  01:51:53 STEP  642 H= 642:00 +CPU= 68.481
  01:52:36 STEP  643 H= 643:00 +CPU= 42.800
  01:53:21 STEP  644 H= 644:00 +CPU= 44.320
  01:54:27 STEP  645 H= 645:00 +CPU= 66.599
  01:55:12 STEP  646 H= 646:00 +CPU= 44.270
  01:55:56 STEP  647 H= 647:00 +CPU= 43.921
  01:57:15 STEP  648 H= 648:00 +CPU= 78.575
Moving to projects directory: /root/slots/4/ICMGGhq0f+000648
Moving to projects directory: /root/slots/4/ICMSHhq0f+000648
Moving to projects directory: /root/slots/4/ICMUAhq0f+000648
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000624
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000624
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000624
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000636
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000636
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000636
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_26.zip
Uploading the intermediate file: upload_file_26.zip
  01:57:58 STEP  649 H= 649:00 +CPU= 42.670
  01:58:42 STEP  650 H= 650:00 +CPU= 44.280
  01:59:49 STEP  651 H= 651:00 +CPU= 66.669
  02:00:34 STEP  652 H= 652:00 +CPU= 44.296
  02:01:17 STEP  653 H= 653:00 +CPU= 42.785
  02:02:25 STEP  654 H= 654:00 +CPU= 68.562
  02:03:08 STEP  655 H= 655:00 +CPU= 42.803
  02:03:53 STEP  656 H= 656:00 +CPU= 44.436
  02:05:00 STEP  657 H= 657:00 +CPU= 66.674
  02:05:44 STEP  658 H= 658:00 +CPU= 44.351
  02:06:27 STEP  659 H= 659:00 +CPU= 42.783
  02:07:42 STEP  660 H= 660:00 +CPU= 74.736
Moving to projects directory: /root/slots/4/ICMGGhq0f+000660
Moving to projects directory: /root/slots/4/ICMSHhq0f+000660
Moving to projects directory: /root/slots/4/ICMUAhq0f+000660
  02:08:25 STEP  661 H= 661:00 +CPU= 42.875
  02:09:10 STEP  662 H= 662:00 +CPU= 44.482
  02:10:17 STEP  663 H= 663:00 +CPU= 66.676
  02:11:01 STEP  664 H= 664:00 +CPU= 44.315
  02:11:44 STEP  665 H= 665:00 +CPU= 42.798
  02:12:53 STEP  666 H= 666:00 +CPU= 68.546
  02:13:36 STEP  667 H= 667:00 +CPU= 42.699
  02:14:20 STEP  668 H= 668:00 +CPU= 44.391
  02:15:27 STEP  669 H= 669:00 +CPU= 66.579
  02:16:11 STEP  670 H= 670:00 +CPU= 44.320
  02:16:55 STEP  671 H= 671:00 +CPU= 44.046
  02:18:15 STEP  672 H= 672:00 +CPU= 78.443
Moving to projects directory: /root/slots/4/ICMGGhq0f+000672
Moving to projects directory: /root/slots/4/ICMSHhq0f+000672
Moving to projects directory: /root/slots/4/ICMUAhq0f+000672
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000648
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000648
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000648
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000660
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000660
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000660
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_27.zip
Uploading the intermediate file: upload_file_27.zip
  02:18:57 STEP  673 H= 673:00 +CPU= 42.718
  02:19:42 STEP  674 H= 674:00 +CPU= 44.307
  02:20:49 STEP  675 H= 675:00 +CPU= 66.669
  02:21:33 STEP  676 H= 676:00 +CPU= 44.415
  02:22:16 STEP  677 H= 677:00 +CPU= 42.764
  02:23:25 STEP  678 H= 678:00 +CPU= 68.529
  02:24:08 STEP  679 H= 679:00 +CPU= 42.797
  02:24:52 STEP  680 H= 680:00 +CPU= 44.343
  02:25:59 STEP  681 H= 681:00 +CPU= 66.687
  02:26:44 STEP  682 H= 682:00 +CPU= 44.415
  02:27:27 STEP  683 H= 683:00 +CPU= 42.831
  02:28:42 STEP  684 H= 684:00 +CPU= 74.892
Moving to projects directory: /root/slots/4/ICMGGhq0f+000684
Moving to projects directory: /root/slots/4/ICMSHhq0f+000684
Moving to projects directory: /root/slots/4/ICMUAhq0f+000684
  02:29:25 STEP  685 H= 685:00 +CPU= 42.865
  02:30:09 STEP  686 H= 686:00 +CPU= 44.418
  02:31:16 STEP  687 H= 687:00 +CPU= 66.770
  02:32:01 STEP  688 H= 688:00 +CPU= 44.325
  02:32:43 STEP  689 H= 689:00 +CPU= 42.783
  02:33:52 STEP  690 H= 690:00 +CPU= 68.555
  02:34:35 STEP  691 H= 691:00 +CPU= 42.802
  02:35:20 STEP  692 H= 692:00 +CPU= 44.433
  02:36:27 STEP  693 H= 693:00 +CPU= 66.705
  02:37:11 STEP  694 H= 694:00 +CPU= 44.264
  02:37:55 STEP  695 H= 695:00 +CPU= 43.931
  02:39:14 STEP  696 H= 696:00 +CPU= 78.502
Moving to projects directory: /root/slots/4/ICMGGhq0f+000696
Moving to projects directory: /root/slots/4/ICMSHhq0f+000696
Moving to projects directory: /root/slots/4/ICMUAhq0f+000696
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000672
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000672
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000672
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000684
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000684
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000684
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_28.zip
Uploading the intermediate file: upload_file_28.zip
  02:39:57 STEP  697 H= 697:00 +CPU= 42.634
  02:40:41 STEP  698 H= 698:00 +CPU= 44.331
  02:41:48 STEP  699 H= 699:00 +CPU= 66.662
  02:42:33 STEP  700 H= 700:00 +CPU= 44.306
  02:43:16 STEP  701 H= 701:00 +CPU= 42.715
  02:44:24 STEP  702 H= 702:00 +CPU= 68.626
  02:45:07 STEP  703 H= 703:00 +CPU= 42.746
  02:45:52 STEP  704 H= 704:00 +CPU= 44.365
  02:46:59 STEP  705 H= 705:00 +CPU= 66.736
  02:47:43 STEP  706 H= 706:00 +CPU= 44.325
  02:48:26 STEP  707 H= 707:00 +CPU= 42.780
  02:49:42 STEP  708 H= 708:00 +CPU= 75.452
Moving to projects directory: /root/slots/4/ICMGGhq0f+000708
Moving to projects directory: /root/slots/4/ICMSHhq0f+000708
Moving to projects directory: /root/slots/4/ICMUAhq0f+000708
  02:50:25 STEP  709 H= 709:00 +CPU= 43.461
  02:51:11 STEP  710 H= 710:00 +CPU= 45.168
  02:52:19 STEP  711 H= 711:00 +CPU= 67.786
  02:53:04 STEP  712 H= 712:00 +CPU= 45.025
  02:53:48 STEP  713 H= 713:00 +CPU= 43.429
  02:54:58 STEP  714 H= 714:00 +CPU= 69.499
  02:55:42 STEP  715 H= 715:00 +CPU= 43.405
  02:56:27 STEP  716 H= 716:00 +CPU= 44.882
  02:57:35 STEP  717 H= 717:00 +CPU= 67.398
  02:58:21 STEP  718 H= 718:00 +CPU= 44.886
  02:59:06 STEP  719 H= 719:00 +CPU= 44.514
  03:00:27 STEP  720 H= 720:00 +CPU= 79.073
Moving to projects directory: /root/slots/4/ICMGGhq0f+000720
Moving to projects directory: /root/slots/4/ICMSHhq0f+000720
Moving to projects directory: /root/slots/4/ICMUAhq0f+000720
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000696
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000696
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000696
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000708
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000708
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000708
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_29.zip
Uploading the intermediate file: upload_file_29.zip
  03:01:11 STEP  721 H= 721:00 +CPU= 43.390
  03:01:56 STEP  722 H= 722:00 +CPU= 45.113
  03:03:04 STEP  723 H= 723:00 +CPU= 67.672
  03:03:50 STEP  724 H= 724:00 +CPU= 45.015
  03:04:34 STEP  725 H= 725:00 +CPU= 43.296
  03:05:44 STEP  726 H= 726:00 +CPU= 69.294
  03:06:28 STEP  727 H= 727:00 +CPU= 43.598
  03:07:14 STEP  728 H= 728:00 +CPU= 45.217
  03:08:22 STEP  729 H= 729:00 +CPU= 67.525
  03:09:08 STEP  730 H= 730:00 +CPU= 44.999
  03:09:51 STEP  731 H= 731:00 +CPU= 43.393
  03:11:08 STEP  732 H= 732:00 +CPU= 75.999
Moving to projects directory: /root/slots/4/ICMGGhq0f+000732
Moving to projects directory: /root/slots/4/ICMSHhq0f+000732
Moving to projects directory: /root/slots/4/ICMUAhq0f+000732
  03:11:52 STEP  733 H= 733:00 +CPU= 43.853
  03:12:38 STEP  734 H= 734:00 +CPU= 45.017
  03:13:46 STEP  735 H= 735:00 +CPU= 67.680
  03:14:32 STEP  736 H= 736:00 +CPU= 45.154
  03:15:16 STEP  737 H= 737:00 +CPU= 43.691
  03:16:26 STEP  738 H= 738:00 +CPU= 69.617
  03:17:10 STEP  739 H= 739:00 +CPU= 43.397
  03:17:56 STEP  740 H= 740:00 +CPU= 45.075
  03:19:05 STEP  741 H= 741:00 +CPU= 67.713
  03:19:50 STEP  742 H= 742:00 +CPU= 45.100
  03:20:35 STEP  743 H= 743:00 +CPU= 44.528
  03:21:57 STEP  744 H= 744:00 +CPU= 79.732
Moving to projects directory: /root/slots/4/ICMGGhq0f+000744
Moving to projects directory: /root/slots/4/ICMSHhq0f+000744
Moving to projects directory: /root/slots/4/ICMUAhq0f+000744
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000720
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000720
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000720
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000732
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000732
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000732
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_30.zip
Uploading the intermediate file: upload_file_30.zip
  03:22:41 STEP  745 H= 745:00 +CPU= 43.365
  03:23:27 STEP  746 H= 746:00 +CPU= 45.364
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0861
(argv4) batchid: 968
(argv5) wuid: 12185505
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12185505
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_03024ee4c3f6568ca18d33402810c719 to: /root/slots/4/oifs_43r3_ps_0861_1999050100_123_968_12185505.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0861_1999050100_123_968_12185505.zip
ic_ancil_file: ic_ancil_12185505
ifsdata_file: ifsdata_12185505
climate_data_file: clim_data_12185505
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_517498dc4cf7dd16c7f04f870fb607e1 to: /root/slots/4/ic_ancil_12185505.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12185505.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2c7998878ed23906df8a45ca5209cedb to: /root/slots/4/ifsdata/ifsdata_12185505.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12185505.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12185505.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12185505.zip
Checking for progress XML file: /root/slots/4/progress_file_12185505.xml
Opened progress file ok : /root/slots/4/progress_file_12185505.xml
-- Model is restarting --
Adjusting last_iter, 747, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12185505.xml
last_cpu_time: 41469
upload_file_number: 31
last_iter: 743
last_upload: 2678400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318
The child process has been launched with process id: 2186
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:2186:2186] [20221230:032511:1672341911:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:26:33 STEP  744 H= 744:00 +CPU= 75.472
  03:27:17 STEP  745 H= 745:00 +CPU= 43.633
  03:28:03 STEP  746 H= 746:00 +CPU= 45.156
  03:29:11 STEP  747 H= 747:00 +CPU= 67.439
  03:29:56 STEP  748 H= 748:00 +CPU= 45.018
  03:30:40 STEP  749 H= 749:00 +CPU= 43.694
  03:31:50 STEP  750 H= 750:00 +CPU= 69.440
  03:32:34 STEP  751 H= 751:00 +CPU= 43.501
  03:33:20 STEP  752 H= 752:00 +CPU= 45.006
  03:34:28 STEP  753 H= 753:00 +CPU= 67.858
  03:35:14 STEP  754 H= 754:00 +CPU= 45.040
  03:35:58 STEP  755 H= 755:00 +CPU= 43.559
  03:37:15 STEP  756 H= 756:00 +CPU= 75.975
Moving to projects directory: /root/slots/4/ICMGGhq0f+000756
Moving to projects directory: /root/slots/4/ICMSHhq0f+000756
Moving to projects directory: /root/slots/4/ICMUAhq0f+000756
  03:37:59 STEP  757 H= 757:00 +CPU= 43.831
  03:38:45 STEP  758 H= 758:00 +CPU= 45.463
  03:39:54 STEP  759 H= 759:00 +CPU= 68.072
  03:40:39 STEP  760 H= 760:00 +CPU= 45.332
  03:41:24 STEP  761 H= 761:00 +CPU= 43.733
  03:42:34 STEP  762 H= 762:00 +CPU= 69.727
  03:43:18 STEP  763 H= 763:00 +CPU= 43.322
  03:44:04 STEP  764 H= 764:00 +CPU= 45.399
  03:45:13 STEP  765 H= 765:00 +CPU= 68.004
  03:45:58 STEP  766 H= 766:00 +CPU= 45.144
  03:46:44 STEP  767 H= 767:00 +CPU= 45.062
  03:48:05 STEP  768 H= 768:00 +CPU= 79.727
Moving to projects directory: /root/slots/4/ICMGGhq0f+000768
Moving to projects directory: /root/slots/4/ICMSHhq0f+000768
Moving to projects directory: /root/slots/4/ICMUAhq0f+000768
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000744
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000744
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000744
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMGGhq0f+000756
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMSHhq0f+000756
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12185505/ICMUAhq0f+000756
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0861_1999050100_123_968_12185505_0_r1488612318_31.zip
Uploading the intermediate file: upload_file_31.zip
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net