<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230119:205645:1674161805:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0168
(argv4) batchid: 960
(argv5) wuid: 12176812
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/3
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6a16b5d651f63ea81cc40e75f28e53c0 to: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_0168_1991050100_123_960_12176812.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_0168_1991050100_123_960_12176812.zip
ic_ancil_file: ic_ancil_12176812
ifsdata_file: ifsdata_12176812
climate_data_file: clim_data_12176812
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d772df049f9ad68133a38a669dfa02e8 to: /var/lib/boinc-client_40000/slots/3/ic_ancil_12176812.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/3/ic_ancil_12176812.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8cf7c335ccc28d09028eebe1ad075a86 to: /var/lib/boinc-client_40000/slots/3/ifsdata/ifsdata_12176812.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/3/ifsdata/ifsdata_12176812.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/3/159l_2/clim_data_12176812.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/3/159l_2/clim_data_12176812.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
last_cpu_time: 60425
upload_file_number: 97
last_iter: -985
last_upload: 8899200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598
The child process has been launched with process id: 1435
Executing the command: /var/lib/boinc-client_40000/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1435:1435] [20230119:213110:1674163870:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:33:13 STEP 2472 H=2472:00 +CPU= 36.210
21:33:31 STEP 2473 H=2473:00 +CPU= 17.963
21:33:51 STEP 2474 H=2474:00 +CPU= 19.864
21:34:25 STEP 2475 H=2475:00 +CPU= 33.681
21:34:41 STEP 2476 H=2476:00 +CPU= 16.322
21:35:00 STEP 2477 H=2477:00 +CPU= 18.417
21:35:34 STEP 2478 H=2478:00 +CPU= 34.029
21:35:55 STEP 2479 H=2479:00 +CPU= 20.832
21:36:11 STEP 2480 H=2480:00 +CPU= 16.495
21:36:46 STEP 2481 H=2481:00 +CPU= 34.496
21:37:05 STEP 2482 H=2482:00 +CPU= 19.075
21:37:22 STEP 2483 H=2483:00 +CPU= 17.630
21:38:03 STEP 2484 H=2484:00 +CPU= 40.195
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002484
21:38:24 STEP 2485 H=2485:00 +CPU= 21.622
21:38:44 STEP 2486 H=2486:00 +CPU= 19.934
21:39:16 STEP 2487 H=2487:00 +CPU= 31.830
21:39:39 STEP 2488 H=2488:00 +CPU= 23.355
21:40:02 STEP 2489 H=2489:00 +CPU= 22.294
21:40:36 STEP 2490 H=2490:00 +CPU= 34.739
21:41:00 STEP 2491 H=2491:00 +CPU= 23.871
21:41:17 STEP 2492 H=2492:00 +CPU= 17.214
21:41:49 STEP 2493 H=2493:00 +CPU= 32.074
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0168
(argv4) batchid: 960
(argv5) wuid: 12176812
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/3
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6a16b5d651f63ea81cc40e75f28e53c0 to: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_0168_1991050100_123_960_12176812.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_0168_1991050100_123_960_12176812.zip
ic_ancil_file: ic_ancil_12176812
ifsdata_file: ifsdata_12176812
climate_data_file: clim_data_12176812
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d772df049f9ad68133a38a669dfa02e8 to: /var/lib/boinc-client_40000/slots/3/ic_ancil_12176812.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/3/ic_ancil_12176812.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8cf7c335ccc28d09028eebe1ad075a86 to: /var/lib/boinc-client_40000/slots/3/ifsdata/ifsdata_12176812.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/3/ifsdata/ifsdata_12176812.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/3/159l_2/clim_data_12176812.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/3/159l_2/clim_data_12176812.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
-- Model is restarting --
Adjusting last_iter, 2494, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
last_cpu_time: 60990
upload_file_number: 97
last_iter: 2471
last_upload: 8899200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598
The child process has been launched with process id: 1425
Executing the command: /var/lib/boinc-client_40000/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:214823:1674164903:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:50:08 STEP 2472 H=2472:00 +CPU= 28.054
21:50:29 STEP 2473 H=2473:00 +CPU= 20.704
21:50:44 STEP 2474 H=2474:00 +CPU= 14.835
21:51:14 STEP 2475 H=2475:00 +CPU= 30.368
21:51:35 STEP 2476 H=2476:00 +CPU= 20.345
21:51:49 STEP 2477 H=2477:00 +CPU= 13.997
21:52:20 STEP 2478 H=2478:00 +CPU= 30.757
21:52:37 STEP 2479 H=2479:00 +CPU= 17.516
21:52:59 STEP 2480 H=2480:00 +CPU= 21.697
21:53:32 STEP 2481 H=2481:00 +CPU= 33.532
21:53:48 STEP 2482 H=2482:00 +CPU= 16.169
21:54:08 STEP 2483 H=2483:00 +CPU= 19.923
21:54:39 STEP 2484 H=2484:00 +CPU= 31.061
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002484
21:54:57 STEP 2485 H=2485:00 +CPU= 18.012
21:55:12 STEP 2486 H=2486:00 +CPU= 15.031
21:55:36 STEP 2487 H=2487:00 +CPU= 23.466
21:55:51 STEP 2488 H=2488:00 +CPU= 14.952
21:56:06 STEP 2489 H=2489:00 +CPU= 14.714
21:56:38 STEP 2490 H=2490:00 +CPU= 32.225
21:56:55 STEP 2491 H=2491:00 +CPU= 17.391
21:57:10 STEP 2492 H=2492:00 +CPU= 14.782
21:57:35 STEP 2493 H=2493:00 +CPU= 25.017
21:57:51 STEP 2494 H=2494:00 +CPU= 16.258
21:58:08 STEP 2495 H=2495:00 +CPU= 16.520
21:58:39 STEP 2496 H=2496:00 +CPU= 31.599
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_97.zip
Uploading the intermediate file: upload_file_97.zip
21:58:55 STEP 2497 H=2497:00 +CPU= 15.900
21:59:12 STEP 2498 H=2498:00 +CPU= 16.644
21:59:37 STEP 2499 H=2499:00 +CPU= 25.101
21:59:53 STEP 2500 H=2500:00 +CPU= 15.984
22:00:08 STEP 2501 H=2501:00 +CPU= 14.632
22:00:38 STEP 2502 H=2502:00 +CPU= 29.732
22:01:00 STEP 2503 H=2503:00 +CPU= 22.388
22:01:17 STEP 2504 H=2504:00 +CPU= 16.721
22:01:39 STEP 2505 H=2505:00 +CPU= 22.817
22:01:54 STEP 2506 H=2506:00 +CPU= 14.687
22:02:11 STEP 2507 H=2507:00 +CPU= 17.107
22:02:40 STEP 2508 H=2508:00 +CPU= 28.255
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002508
22:02:56 STEP 2509 H=2509:00 +CPU= 16.195
22:03:10 STEP 2510 H=2510:00 +CPU= 14.760
22:03:37 STEP 2511 H=2511:00 +CPU= 26.308
22:03:54 STEP 2512 H=2512:00 +CPU= 17.093
22:04:08 STEP 2513 H=2513:00 +CPU= 14.303
22:04:31 STEP 2514 H=2514:00 +CPU= 23.224
22:04:48 STEP 2515 H=2515:00 +CPU= 16.778
22:05:06 STEP 2516 H=2516:00 +CPU= 18.097
22:05:30 STEP 2517 H=2517:00 +CPU= 23.748
22:05:45 STEP 2518 H=2518:00 +CPU= 15.259
22:06:01 STEP 2519 H=2519:00 +CPU= 15.594
22:06:31 STEP 2520 H=2520:00 +CPU= 30.017
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_98.zip
Uploading the intermediate file: upload_file_98.zip
22:06:45 STEP 2521 H=2521:00 +CPU= 14.294
22:07:00 STEP 2522 H=2522:00 +CPU= 14.698
22:07:24 STEP 2523 H=2523:00 +CPU= 23.743
22:07:43 STEP 2524 H=2524:00 +CPU= 19.048
22:07:59 STEP 2525 H=2525:00 +CPU= 16.041
22:08:22 STEP 2526 H=2526:00 +CPU= 23.022
22:08:36 STEP 2527 H=2527:00 +CPU= 14.386
22:08:51 STEP 2528 H=2528:00 +CPU= 14.943
22:09:22 STEP 2529 H=2529:00 +CPU= 31.215
22:09:40 STEP 2530 H=2530:00 +CPU= 17.225
22:09:56 STEP 2531 H=2531:00 +CPU= 16.526
22:10:22 STEP 2532 H=2532:00 +CPU= 26.282
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002532
22:10:37 STEP 2533 H=2533:00 +CPU= 14.329
22:10:52 STEP 2534 H=2534:00 +CPU= 15.073
22:11:15 STEP 2535 H=2535:00 +CPU= 23.103
22:11:30 STEP 2536 H=2536:00 +CPU= 14.849
22:11:44 STEP 2537 H=2537:00 +CPU= 14.507
22:12:11 STEP 2538 H=2538:00 +CPU= 27.076
22:12:25 STEP 2539 H=2539:00 +CPU= 14.167
22:12:41 STEP 2540 H=2540:00 +CPU= 15.612
22:13:09 STEP 2541 H=2541:00 +CPU= 27.570
22:13:25 STEP 2542 H=2542:00 +CPU= 16.114
22:13:42 STEP 2543 H=2543:00 +CPU= 16.879
22:14:15 STEP 2544 H=2544:00 +CPU= 33.698
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_99.zip
Uploading the intermediate file: upload_file_99.zip
22:14:36 STEP 2545 H=2545:00 +CPU= 20.337
22:14:54 STEP 2546 H=2546:00 +CPU= 18.451
22:15:17 STEP 2547 H=2547:00 +CPU= 23.228
22:15:36 STEP 2548 H=2548:00 +CPU= 19.003
22:15:54 STEP 2549 H=2549:00 +CPU= 17.359
22:16:17 STEP 2550 H=2550:00 +CPU= 23.099
22:16:31 STEP 2551 H=2551:00 +CPU= 14.536
22:16:48 STEP 2552 H=2552:00 +CPU= 16.457
22:17:13 STEP 2553 H=2553:00 +CPU= 24.954
22:17:28 STEP 2554 H=2554:00 +CPU= 15.378
22:17:45 STEP 2555 H=2555:00 +CPU= 17.022
22:18:13 STEP 2556 H=2556:00 +CPU= 27.744
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002556
22:18:27 STEP 2557 H=2557:00 +CPU= 14.415
22:18:42 STEP 2558 H=2558:00 +CPU= 14.561
22:19:05 STEP 2559 H=2559:00 +CPU= 23.010
22:19:21 STEP 2560 H=2560:00 +CPU= 16.529
22:19:35 STEP 2561 H=2561:00 +CPU= 13.841
22:20:00 STEP 2562 H=2562:00 +CPU= 24.614
22:20:14 STEP 2563 H=2563:00 +CPU= 14.444
22:20:32 STEP 2564 H=2564:00 +CPU= 17.298
22:20:55 STEP 2565 H=2565:00 +CPU= 23.272
22:21:16 STEP 2566 H=2566:00 +CPU= 21.217
22:21:37 STEP 2567 H=2567:00 +CPU= 21.001
22:22:12 STEP 2568 H=2568:00 +CPU= 34.409
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_100.zip
Uploading the intermediate file: upload_file_100.zip
22:22:30 STEP 2569 H=2569:00 +CPU= 18.591
22:22:49 STEP 2570 H=2570:00 +CPU= 18.680
22:23:18 STEP 2571 H=2571:00 +CPU= 29.178
22:23:35 STEP 2572 H=2572:00 +CPU= 16.608
22:23:50 STEP 2573 H=2573:00 +CPU= 15.699
22:24:16 STEP 2574 H=2574:00 +CPU= 25.462
22:24:36 STEP 2575 H=2575:00 +CPU= 20.289
22:24:51 STEP 2576 H=2576:00 +CPU= 14.909
22:25:22 STEP 2577 H=2577:00 +CPU= 30.571
22:25:37 STEP 2578 H=2578:00 +CPU= 15.503
22:25:52 STEP 2579 H=2579:00 +CPU= 15.300
22:26:21 STEP 2580 H=2580:00 +CPU= 28.984
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002580
22:26:35 STEP 2581 H=2581:00 +CPU= 14.022
22:26:51 STEP 2582 H=2582:00 +CPU= 15.820
22:27:21 STEP 2583 H=2583:00 +CPU= 29.494
22:27:40 STEP 2584 H=2584:00 +CPU= 19.481
22:27:59 STEP 2585 H=2585:00 +CPU= 18.642
22:28:25 STEP 2586 H=2586:00 +CPU= 26.562
22:28:40 STEP 2587 H=2587:00 +CPU= 14.127
22:28:54 STEP 2588 H=2588:00 +CPU= 14.387
22:29:18 STEP 2589 H=2589:00 +CPU= 24.040
22:29:36 STEP 2590 H=2590:00 +CPU= 18.416
22:29:51 STEP 2591 H=2591:00 +CPU= 14.385
22:30:20 STEP 2592 H=2592:00 +CPU= 29.035
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_101.zip
Uploading the intermediate file: upload_file_101.zip
22:30:35 STEP 2593 H=2593:00 +CPU= 14.914
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0168
(argv4) batchid: 960
(argv5) wuid: 12176812
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/3
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6a16b5d651f63ea81cc40e75f28e53c0 to: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_0168_1991050100_123_960_12176812.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_0168_1991050100_123_960_12176812.zip
ic_ancil_file: ic_ancil_12176812
ifsdata_file: ifsdata_12176812
climate_data_file: clim_data_12176812
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d772df049f9ad68133a38a669dfa02e8 to: /var/lib/boinc-client_40000/slots/3/ic_ancil_12176812.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/3/ic_ancil_12176812.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8cf7c335ccc28d09028eebe1ad075a86 to: /var/lib/boinc-client_40000/slots/3/ifsdata/ifsdata_12176812.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/3/ifsdata/ifsdata_12176812.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/3/159l_2/clim_data_12176812.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/3/159l_2/clim_data_12176812.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
-- Model is restarting --
Adjusting last_iter, 2593, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
last_cpu_time: 63431
upload_file_number: 102
last_iter: 2591
last_upload: 9331200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598
The child process has been launched with process id: 1394
Executing the command: /var/lib/boinc-client_40000/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1394:1394] [20230119:223916:1674167956:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
22:40:23 STEP 2592 H=2592:00 +CPU= 22.538
22:40:36 STEP 2593 H=2593:00 +CPU= 12.481
22:40:50 STEP 2594 H=2594:00 +CPU= 13.899
22:41:12 STEP 2595 H=2595:00 +CPU= 22.151
22:41:26 STEP 2596 H=2596:00 +CPU= 14.201
22:41:40 STEP 2597 H=2597:00 +CPU= 14.009
22:42:02 STEP 2598 H=2598:00 +CPU= 22.021
22:42:16 STEP 2599 H=2599:00 +CPU= 13.620
22:42:31 STEP 2600 H=2600:00 +CPU= 15.451
22:42:53 STEP 2601 H=2601:00 +CPU= 21.575
22:43:13 STEP 2602 H=2602:00 +CPU= 19.981
22:43:29 STEP 2603 H=2603:00 +CPU= 16.429
22:43:54 STEP 2604 H=2604:00 +CPU= 24.652
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002604
22:44:08 STEP 2605 H=2605:00 +CPU= 13.831
22:44:24 STEP 2606 H=2606:00 +CPU= 15.966
22:44:46 STEP 2607 H=2607:00 +CPU= 21.900
22:45:00 STEP 2608 H=2608:00 +CPU= 14.220
22:45:14 STEP 2609 H=2609:00 +CPU= 13.734
22:45:36 STEP 2610 H=2610:00 +CPU= 22.460
22:45:54 STEP 2611 H=2611:00 +CPU= 18.374
22:46:08 STEP 2612 H=2612:00 +CPU= 13.890
22:46:34 STEP 2613 H=2613:00 +CPU= 25.597
22:46:48 STEP 2614 H=2614:00 +CPU= 14.048
22:47:02 STEP 2615 H=2615:00 +CPU= 14.383
22:47:38 STEP 2616 H=2616:00 +CPU= 35.640
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_102.zip
Uploading the intermediate file: upload_file_102.zip
22:47:53 STEP 2617 H=2617:00 +CPU= 14.801
22:48:11 STEP 2618 H=2618:00 +CPU= 18.032
22:48:15 (839): handle_upload_file_status: can't parse boinc_ufs_upload_file_101.zip
22:48:34 STEP 2619 H=2619:00 +CPU= 23.510
22:48:49 STEP 2620 H=2620:00 +CPU= 14.398
22:49:03 STEP 2621 H=2621:00 +CPU= 14.266
22:49:26 STEP 2622 H=2622:00 +CPU= 23.111
22:49:41 STEP 2623 H=2623:00 +CPU= 14.390
22:49:56 STEP 2624 H=2624:00 +CPU= 15.133
22:50:18 STEP 2625 H=2625:00 +CPU= 21.982
22:50:32 STEP 2626 H=2626:00 +CPU= 14.275
22:50:47 STEP 2627 H=2627:00 +CPU= 14.773
22:51:11 STEP 2628 H=2628:00 +CPU= 24.098
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002628
22:51:25 STEP 2629 H=2629:00 +CPU= 13.825
22:51:40 STEP 2630 H=2630:00 +CPU= 15.004
22:52:01 STEP 2631 H=2631:00 +CPU= 21.481
22:52:15 STEP 2632 H=2632:00 +CPU= 13.785
22:52:30 STEP 2633 H=2633:00 +CPU= 15.557
22:52:54 STEP 2634 H=2634:00 +CPU= 23.926
22:53:09 STEP 2635 H=2635:00 +CPU= 14.715
22:53:23 STEP 2636 H=2636:00 +CPU= 14.036
22:53:45 STEP 2637 H=2637:00 +CPU= 21.414
22:53:58 STEP 2638 H=2638:00 +CPU= 13.662
22:54:12 STEP 2639 H=2639:00 +CPU= 14.199
22:54:48 STEP 2640 H=2640:00 +CPU= 35.115
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_103.zip
Uploading the intermediate file: upload_file_103.zip
22:55:01 STEP 2641 H=2641:00 +CPU= 13.774
22:55:16 STEP 2642 H=2642:00 +CPU= 14.724
22:55:17 (839): handle_upload_file_status: can't parse boinc_ufs_upload_file_101.zip
22:55:40 STEP 2643 H=2643:00 +CPU= 23.935
22:55:59 STEP 2644 H=2644:00 +CPU= 18.701
22:56:14 STEP 2645 H=2645:00 +CPU= 15.415
22:56:40 STEP 2646 H=2646:00 +CPU= 26.067
22:56:55 STEP 2647 H=2647:00 +CPU= 14.735
22:57:10 STEP 2648 H=2648:00 +CPU= 15.369
22:57:38 STEP 2649 H=2649:00 +CPU= 27.894
22:57:57 STEP 2650 H=2650:00 +CPU= 18.531
22:58:12 STEP 2651 H=2651:00 +CPU= 15.319
22:58:40 STEP 2652 H=2652:00 +CPU= 27.612
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002652
22:58:59 STEP 2653 H=2653:00 +CPU= 18.729
22:59:14 STEP 2654 H=2654:00 +CPU= 15.859
22:59:40 STEP 2655 H=2655:00 +CPU= 25.630
22:59:59 STEP 2656 H=2656:00 +CPU= 18.731
23:00:15 STEP 2657 H=2657:00 +CPU= 15.898
23:00:45 STEP 2658 H=2658:00 +CPU= 30.534
23:01:03 STEP 2659 H=2659:00 +CPU= 17.782
23:01:19 STEP 2660 H=2660:00 +CPU= 16.231
23:01:43 STEP 2661 H=2661:00 +CPU= 24.219
23:02:08 STEP 2662 H=2662:00 +CPU= 24.285
23:02:29 STEP 2663 H=2663:00 +CPU= 21.129
23:03:00 STEP 2664 H=2664:00 +CPU= 31.122
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_104.zip
Uploading the intermediate file: upload_file_104.zip
23:03:23 STEP 2665 H=2665:00 +CPU= 22.566
23:03:40 (839): handle_upload_file_status: can't parse boinc_ufs_upload_file_101.zip
23:03:47 STEP 2666 H=2666:00 +CPU= 24.140
23:04:21 STEP 2667 H=2667:00 +CPU= 34.539
23:04:44 STEP 2668 H=2668:00 +CPU= 22.368
23:05:08 STEP 2669 H=2669:00 +CPU= 24.300
23:05:42 STEP 2670 H=2670:00 +CPU= 33.708
23:06:03 STEP 2671 H=2671:00 +CPU= 21.736
23:06:25 STEP 2672 H=2672:00 +CPU= 22.113
23:06:54 STEP 2673 H=2673:00 +CPU= 28.668
23:07:15 STEP 2674 H=2674:00 +CPU= 21.118
23:07:35 STEP 2675 H=2675:00 +CPU= 20.034
23:08:10 STEP 2676 H=2676:00 +CPU= 34.809
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002676
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0168
(argv4) batchid: 960
(argv5) wuid: 12176812
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/3
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6a16b5d651f63ea81cc40e75f28e53c0 to: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_0168_1991050100_123_960_12176812.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/3/oifs_43r3_ps_0168_1991050100_123_960_12176812.zip
ic_ancil_file: ic_ancil_12176812
ifsdata_file: ifsdata_12176812
climate_data_file: clim_data_12176812
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d772df049f9ad68133a38a669dfa02e8 to: /var/lib/boinc-client_40000/slots/3/ic_ancil_12176812.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/3/ic_ancil_12176812.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8cf7c335ccc28d09028eebe1ad075a86 to: /var/lib/boinc-client_40000/slots/3/ifsdata/ifsdata_12176812.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/3/ifsdata/ifsdata_12176812.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/3/159l_2/clim_data_12176812.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/3/159l_2/clim_data_12176812.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
-- Model is restarting --
Adjusting last_iter, 2677, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/3/progress_file_12176812.xml
last_cpu_time: 65122
upload_file_number: 105
last_iter: 2663
last_upload: 9590400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598
The child process has been launched with process id: 1387
Executing the command: /var/lib/boinc-client_40000/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1387:1387] [20230119:230945:1674169785:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
23:11:06 STEP 2664 H=2664:00 +CPU= 25.736
23:11:20 STEP 2665 H=2665:00 +CPU= 14.697
23:11:36 STEP 2666 H=2666:00 +CPU= 15.654
23:11:58 STEP 2667 H=2667:00 +CPU= 22.409
23:12:19 STEP 2668 H=2668:00 +CPU= 20.938
23:12:34 STEP 2669 H=2669:00 +CPU= 14.630
23:12:59 STEP 2670 H=2670:00 +CPU= 25.344
23:13:18 STEP 2671 H=2671:00 +CPU= 19.235
23:13:34 STEP 2672 H=2672:00 +CPU= 15.081
23:13:57 STEP 2673 H=2673:00 +CPU= 23.475
23:14:11 STEP 2674 H=2674:00 +CPU= 14.056
23:14:26 STEP 2675 H=2675:00 +CPU= 14.549
23:14:52 STEP 2676 H=2676:00 +CPU= 26.520
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002676
23:15:10 STEP 2677 H=2677:00 +CPU= 17.772
23:15:25 STEP 2678 H=2678:00 +CPU= 14.725
23:15:49 STEP 2679 H=2679:00 +CPU= 24.227
23:16:08 STEP 2680 H=2680:00 +CPU= 19.369
23:16:24 STEP 2681 H=2681:00 +CPU= 15.817
23:16:50 STEP 2682 H=2682:00 +CPU= 25.406
23:17:10 STEP 2683 H=2683:00 +CPU= 20.741
23:17:28 STEP 2684 H=2684:00 +CPU= 17.727
23:17:54 STEP 2685 H=2685:00 +CPU= 26.193
23:18:09 STEP 2686 H=2686:00 +CPU= 14.650
23:18:26 STEP 2687 H=2687:00 +CPU= 17.538
23:18:53 STEP 2688 H=2688:00 +CPU= 26.750
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/3/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176812/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0168_1991050100_123_960_12176812_2_r488357598_105.zip
Uploading the intermediate file: upload_file_105.zip
23:19:07 STEP 2689 H=2689:00 +CPU= 13.939
23:19:22 STEP 2690 H=2690:00 +CPU= 14.572
23:19:25 (827): handle_upload_file_status: can't parse boinc_ufs_upload_file_101.zip
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
