<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
H=2485:00 +CPU= 19.955
00:09:58 STEP 2486 H=2486:00 +CPU= 20.559
00:10:30 STEP 2487 H=2487:00 +CPU= 31.974
00:10:51 STEP 2488 H=2488:00 +CPU= 20.565
00:11:11 STEP 2489 H=2489:00 +CPU= 19.973
00:11:43 STEP 2490 H=2490:00 +CPU= 32.446
00:12:03 STEP 2491 H=2491:00 +CPU= 19.950
00:12:24 STEP 2492 H=2492:00 +CPU= 20.472
00:12:56 STEP 2493 H=2493:00 +CPU= 31.846
00:13:17 STEP 2494 H=2494:00 +CPU= 20.486
00:13:38 STEP 2495 H=2495:00 +CPU= 20.593
00:14:17 STEP 2496 H=2496:00 +CPU= 38.845
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_103.zip
Uploading the intermediate file: upload_file_103.zip
00:14:37 STEP 2497 H=2497:00 +CPU= 19.931
00:14:58 STEP 2498 H=2498:00 +CPU= 20.456
00:15:30 STEP 2499 H=2499:00 +CPU= 32.095
00:15:51 STEP 2500 H=2500:00 +CPU= 20.501
00:16:11 STEP 2501 H=2501:00 +CPU= 19.956
00:16:44 STEP 2502 H=2502:00 +CPU= 32.505
00:17:04 STEP 2503 H=2503:00 +CPU= 19.999
00:17:25 STEP 2504 H=2504:00 +CPU= 20.518
00:17:57 STEP 2505 H=2505:00 +CPU= 31.826
00:18:17 STEP 2506 H=2506:00 +CPU= 20.483
00:18:37 STEP 2507 H=2507:00 +CPU= 19.962
00:19:13 STEP 2508 H=2508:00 +CPU= 35.090
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002508
00:19:33 STEP 2509 H=2509:00 +CPU= 20.029
00:19:53 STEP 2510 H=2510:00 +CPU= 20.517
00:20:26 STEP 2511 H=2511:00 +CPU= 31.931
00:20:46 STEP 2512 H=2512:00 +CPU= 20.534
00:21:06 STEP 2513 H=2513:00 +CPU= 19.966
00:21:39 STEP 2514 H=2514:00 +CPU= 32.384
00:21:59 STEP 2515 H=2515:00 +CPU= 19.903
00:22:20 STEP 2516 H=2516:00 +CPU= 20.474
00:22:52 STEP 2517 H=2517:00 +CPU= 31.900
00:23:12 STEP 2518 H=2518:00 +CPU= 20.447
00:23:33 STEP 2519 H=2519:00 +CPU= 20.562
00:24:15 STEP 2520 H=2520:00 +CPU= 38.843
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_104.zip
Uploading the intermediate file: upload_file_104.zip
00:24:45 STEP 2521 H=2521:00 +CPU= 20.165
00:25:19 STEP 2522 H=2522:00 +CPU= 21.034
00:26:10 STEP 2523 H=2523:00 +CPU= 32.445
00:26:30 STEP 2524 H=2524:00 +CPU= 20.370
00:26:51 STEP 2525 H=2525:00 +CPU= 19.914
00:27:23 STEP 2526 H=2526:00 +CPU= 32.391
00:27:44 STEP 2527 H=2527:00 +CPU= 19.974
00:28:05 STEP 2528 H=2528:00 +CPU= 20.514
00:28:37 STEP 2529 H=2529:00 +CPU= 31.798
00:28:58 STEP 2530 H=2530:00 +CPU= 20.455
00:29:18 STEP 2531 H=2531:00 +CPU= 19.910
00:29:54 STEP 2532 H=2532:00 +CPU= 35.155
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002532
00:30:14 STEP 2533 H=2533:00 +CPU= 20.264
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0354
(argv4) batchid: 982
(argv5) wuid: 12198998
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_071f17ecaedca3358b3532192176d424 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0354_2013050100_123_982_12198998.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0354_2013050100_123_982_12198998.zip
ic_ancil_file: ic_ancil_12198998
ifsdata_file: ifsdata_12198998
climate_data_file: clim_data_12198998
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1e634b81de74fb90fdba7fe1ee97d1f1 to: /var/lib/boinc-client/slots/3/ic_ancil_12198998.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12198998.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3ce7a9bd6633bf4b7dc8c2073f6ab459 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12198998.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12198998.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12198998.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12198998.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12198998.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12198998.xml
-- Model is restarting --
Adjusting last_iter, 2534, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12198998.xml
last_cpu_time: 63466
upload_file_number: 105
last_iter: 2519
last_upload: 9072000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342
The child process has been launched with process id: 156142
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:156142:156142] [20230126:003138:1674693098:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
00:32:25 STEP 2520 H=2520:00 +CPU= 36.362
00:32:52 STEP 2521 H=2521:00 +CPU= 20.101
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
double free or corruption (top)
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0354
(argv4) batchid: 982
(argv5) wuid: 12198998
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_071f17ecaedca3358b3532192176d424 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0354_2013050100_123_982_12198998.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0354_2013050100_123_982_12198998.zip
ic_ancil_file: ic_ancil_12198998
ifsdata_file: ifsdata_12198998
climate_data_file: clim_data_12198998
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1e634b81de74fb90fdba7fe1ee97d1f1 to: /var/lib/boinc-client/slots/3/ic_ancil_12198998.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12198998.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3ce7a9bd6633bf4b7dc8c2073f6ab459 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12198998.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12198998.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12198998.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12198998.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12198998.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12198998.xml
-- Model is restarting --
Adjusting last_iter, 2522, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12198998.xml
last_cpu_time: 63534
upload_file_number: 105
last_iter: 2519
last_upload: 9072000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342
The child process has been launched with process id: 157582
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157582:157582] [20230126:003626:1674693386:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
00:37:15 STEP 2520 H=2520:00 +CPU= 36.549
00:37:35 STEP 2521 H=2521:00 +CPU= 19.968
00:38:02 STEP 2522 H=2522:00 +CPU= 20.403
00:39:05 STEP 2523 H=2523:00 +CPU= 31.814
00:39:42 STEP 2524 H=2524:00 +CPU= 20.335
00:40:11 STEP 2525 H=2525:00 +CPU= 19.795
00:40:44 STEP 2526 H=2526:00 +CPU= 32.155
00:41:05 STEP 2527 H=2527:00 +CPU= 19.762
00:41:26 STEP 2528 H=2528:00 +CPU= 20.356
00:41:58 STEP 2529 H=2529:00 +CPU= 31.548
00:42:19 STEP 2530 H=2530:00 +CPU= 20.373
00:42:40 STEP 2531 H=2531:00 +CPU= 19.832
00:43:15 STEP 2532 H=2532:00 +CPU= 34.724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002532
00:43:35 STEP 2533 H=2533:00 +CPU= 19.809
00:43:56 STEP 2534 H=2534:00 +CPU= 20.356
00:44:28 STEP 2535 H=2535:00 +CPU= 31.617
00:44:49 STEP 2536 H=2536:00 +CPU= 20.342
00:45:09 STEP 2537 H=2537:00 +CPU= 19.963
00:45:42 STEP 2538 H=2538:00 +CPU= 32.203
00:46:02 STEP 2539 H=2539:00 +CPU= 19.777
00:46:22 STEP 2540 H=2540:00 +CPU= 20.343
00:46:54 STEP 2541 H=2541:00 +CPU= 31.523
00:47:15 STEP 2542 H=2542:00 +CPU= 20.355
00:47:36 STEP 2543 H=2543:00 +CPU= 20.405
00:48:16 STEP 2544 H=2544:00 +CPU= 38.599
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_105.zip
Uploading the intermediate file: upload_file_105.zip
00:48:37 STEP 2545 H=2545:00 +CPU= 19.819
double free or corruption (!prev)
00:48:57 STEP 2546 H=2546:00 +CPU= 20.316
00:49:29 STEP 2547 H=2547:00 +CPU= 31.607
00:49:50 STEP 2548 H=2548:00 +CPU= 20.399
00:50:10 STEP 2549 H=2549:00 +CPU= 19.955
00:50:43 STEP 2550 H=2550:00 +CPU= 32.322
00:51:04 STEP 2551 H=2551:00 +CPU= 19.861
00:51:24 STEP 2552 H=2552:00 +CPU= 20.414
00:51:56 STEP 2553 H=2553:00 +CPU= 31.534
00:52:17 STEP 2554 H=2554:00 +CPU= 20.429
00:52:37 STEP 2555 H=2555:00 +CPU= 19.858
00:53:12 STEP 2556 H=2556:00 +CPU= 34.884
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002556
00:53:33 STEP 2557 H=2557:00 +CPU= 19.885
00:53:54 STEP 2558 H=2558:00 +CPU= 20.472
00:54:26 STEP 2559 H=2559:00 +CPU= 31.704
00:54:46 STEP 2560 H=2560:00 +CPU= 20.432
00:55:07 STEP 2561 H=2561:00 +CPU= 19.978
00:55:40 STEP 2562 H=2562:00 +CPU= 32.364
00:56:00 STEP 2563 H=2563:00 +CPU= 19.859
00:56:21 STEP 2564 H=2564:00 +CPU= 20.614
00:56:54 STEP 2565 H=2565:00 +CPU= 31.708
00:57:14 STEP 2566 H=2566:00 +CPU= 20.481
00:57:35 STEP 2567 H=2567:00 +CPU= 20.544
00:58:16 STEP 2568 H=2568:00 +CPU= 38.851
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_106.zip
Uploading the intermediate file: upload_file_106.zip
00:58:37 STEP 2569 H=2569:00 +CPU= 19.931
00:58:58 STEP 2570 H=2570:00 +CPU= 20.337
00:59:30 STEP 2571 H=2571:00 +CPU= 31.679
00:59:50 STEP 2572 H=2572:00 +CPU= 20.400
01:00:11 STEP 2573 H=2573:00 +CPU= 20.189
01:00:44 STEP 2574 H=2574:00 +CPU= 32.412
01:01:04 STEP 2575 H=2575:00 +CPU= 19.887
01:01:25 STEP 2576 H=2576:00 +CPU= 20.504
01:01:57 STEP 2577 H=2577:00 +CPU= 31.718
01:02:18 STEP 2578 H=2578:00 +CPU= 20.458
01:02:38 STEP 2579 H=2579:00 +CPU= 19.937
01:03:13 STEP 2580 H=2580:00 +CPU= 34.924
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002580
01:03:33 STEP 2581 H=2581:00 +CPU= 20.027
01:03:54 STEP 2582 H=2582:00 +CPU= 20.481
01:04:26 STEP 2583 H=2583:00 +CPU= 31.755
01:04:47 STEP 2584 H=2584:00 +CPU= 20.463
01:05:07 STEP 2585 H=2585:00 +CPU= 20.021
01:05:40 STEP 2586 H=2586:00 +CPU= 32.333
01:06:00 STEP 2587 H=2587:00 +CPU= 19.846
01:06:21 STEP 2588 H=2588:00 +CPU= 20.492
01:06:53 STEP 2589 H=2589:00 +CPU= 31.753
01:07:14 STEP 2590 H=2590:00 +CPU= 20.480
01:07:35 STEP 2591 H=2591:00 +CPU= 20.508
01:08:14 STEP 2592 H=2592:00 +CPU= 38.799
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_107.zip
Uploading the intermediate file: upload_file_107.zip
01:08:36 STEP 2593 H=2593:00 +CPU= 19.982
01:08:57 STEP 2594 H=2594:00 +CPU= 20.442
01:09:29 STEP 2595 H=2595:00 +CPU= 31.876
01:09:50 STEP 2596 H=2596:00 +CPU= 20.475
01:10:11 STEP 2597 H=2597:00 +CPU= 19.991
01:10:43 STEP 2598 H=2598:00 +CPU= 32.387
01:11:04 STEP 2599 H=2599:00 +CPU= 19.880
01:11:25 STEP 2600 H=2600:00 +CPU= 20.512
01:11:57 STEP 2601 H=2601:00 +CPU= 31.711
01:12:18 STEP 2602 H=2602:00 +CPU= 20.466
01:12:38 STEP 2603 H=2603:00 +CPU= 19.913
01:13:13 STEP 2604 H=2604:00 +CPU= 34.986
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002604
01:13:34 STEP 2605 H=2605:00 +CPU= 20.012
01:13:55 STEP 2606 H=2606:00 +CPU= 20.476
01:14:27 STEP 2607 H=2607:00 +CPU= 31.860
01:14:48 STEP 2608 H=2608:00 +CPU= 20.517
01:15:09 STEP 2609 H=2609:00 +CPU= 20.218
01:15:42 STEP 2610 H=2610:00 +CPU= 32.454
01:16:02 STEP 2611 H=2611:00 +CPU= 19.944
01:16:23 STEP 2612 H=2612:00 +CPU= 20.519
01:16:55 STEP 2613 H=2613:00 +CPU= 31.839
01:17:16 STEP 2614 H=2614:00 +CPU= 20.501
01:17:37 STEP 2615 H=2615:00 +CPU= 20.586
01:18:17 STEP 2616 H=2616:00 +CPU= 38.888
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_108.zip
Uploading the intermediate file: upload_file_108.zip
01:18:38 STEP 2617 H=2617:00 +CPU= 20.059
01:18:58 STEP 2618 H=2618:00 +CPU= 20.525
01:19:30 STEP 2619 H=2619:00 +CPU= 31.838
01:19:51 STEP 2620 H=2620:00 +CPU= 20.534
01:20:12 STEP 2621 H=2621:00 +CPU= 20.093
01:20:44 STEP 2622 H=2622:00 +CPU= 32.516
01:21:05 STEP 2623 H=2623:00 +CPU= 19.994
01:21:25 STEP 2624 H=2624:00 +CPU= 20.573
01:21:57 STEP 2625 H=2625:00 +CPU= 31.814
01:22:18 STEP 2626 H=2626:00 +CPU= 20.532
01:22:38 STEP 2627 H=2627:00 +CPU= 20.010
01:23:14 STEP 2628 H=2628:00 +CPU= 35.079
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002628
01:23:34 STEP 2629 H=2629:00 +CPU= 20.021
01:23:55 STEP 2630 H=2630:00 +CPU= 20.576
01:24:27 STEP 2631 H=2631:00 +CPU= 31.884
01:24:48 STEP 2632 H=2632:00 +CPU= 20.573
01:25:09 STEP 2633 H=2633:00 +CPU= 20.154
01:25:41 STEP 2634 H=2634:00 +CPU= 32.460
01:26:01 STEP 2635 H=2635:00 +CPU= 19.954
01:26:22 STEP 2636 H=2636:00 +CPU= 20.479
01:26:54 STEP 2637 H=2637:00 +CPU= 31.902
01:27:15 STEP 2638 H=2638:00 +CPU= 20.508
01:27:36 STEP 2639 H=2639:00 +CPU= 20.503
01:28:16 STEP 2640 H=2640:00 +CPU= 38.991
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_109.zip
Uploading the intermediate file: upload_file_109.zip
01:28:37 STEP 2641 H=2641:00 +CPU= 20.046
01:28:58 STEP 2642 H=2642:00 +CPU= 20.554
01:29:30 STEP 2643 H=2643:00 +CPU= 31.788
01:29:51 STEP 2644 H=2644:00 +CPU= 20.541
01:30:11 STEP 2645 H=2645:00 +CPU= 20.401
01:30:44 STEP 2646 H=2646:00 +CPU= 32.448
01:31:04 STEP 2647 H=2647:00 +CPU= 20.065
01:31:25 STEP 2648 H=2648:00 +CPU= 20.603
01:31:57 STEP 2649 H=2649:00 +CPU= 31.864
01:32:18 STEP 2650 H=2650:00 +CPU= 20.548
01:32:38 STEP 2651 H=2651:00 +CPU= 19.969
01:33:13 STEP 2652 H=2652:00 +CPU= 35.206
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002652
01:33:34 STEP 2653 H=2653:00 +CPU= 20.037
01:33:55 STEP 2654 H=2654:00 +CPU= 20.589
01:34:27 STEP 2655 H=2655:00 +CPU= 31.868
01:34:48 STEP 2656 H=2656:00 +CPU= 20.617
01:35:08 STEP 2657 H=2657:00 +CPU= 20.123
01:35:41 STEP 2658 H=2658:00 +CPU= 32.454
01:36:01 STEP 2659 H=2659:00 +CPU= 20.031
01:36:22 STEP 2660 H=2660:00 +CPU= 20.541
01:36:54 STEP 2661 H=2661:00 +CPU= 31.887
01:37:14 STEP 2662 H=2662:00 +CPU= 20.506
01:37:35 STEP 2663 H=2663:00 +CPU= 20.595
01:38:15 STEP 2664 H=2664:00 +CPU= 39.004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_110.zip
Uploading the intermediate file: upload_file_110.zip
01:38:36 STEP 2665 H=2665:00 +CPU= 20.022
01:38:57 STEP 2666 H=2666:00 +CPU= 20.539
01:39:29 STEP 2667 H=2667:00 +CPU= 31.925
01:39:50 STEP 2668 H=2668:00 +CPU= 20.861
01:40:11 STEP 2669 H=2669:00 +CPU= 21.112
01:40:44 STEP 2670 H=2670:00 +CPU= 33.054
01:41:05 STEP 2671 H=2671:00 +CPU= 20.024
01:41:25 STEP 2672 H=2672:00 +CPU= 20.340
01:41:57 STEP 2673 H=2673:00 +CPU= 31.297
01:42:17 STEP 2674 H=2674:00 +CPU= 20.201
01:42:37 STEP 2675 H=2675:00 +CPU= 19.697
01:43:12 STEP 2676 H=2676:00 +CPU= 34.622
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002676
01:43:32 STEP 2677 H=2677:00 +CPU= 19.710
01:43:52 STEP 2678 H=2678:00 +CPU= 20.172
01:44:24 STEP 2679 H=2679:00 +CPU= 31.432
01:44:44 STEP 2680 H=2680:00 +CPU= 20.247
01:45:04 STEP 2681 H=2681:00 +CPU= 19.910
01:45:37 STEP 2682 H=2682:00 +CPU= 32.083
01:45:56 STEP 2683 H=2683:00 +CPU= 19.691
01:46:17 STEP 2684 H=2684:00 +CPU= 20.242
01:46:48 STEP 2685 H=2685:00 +CPU= 31.431
01:47:09 STEP 2686 H=2686:00 +CPU= 20.233
01:47:29 STEP 2687 H=2687:00 +CPU= 20.322
01:48:09 STEP 2688 H=2688:00 +CPU= 38.557
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_111.zip
Uploading the intermediate file: upload_file_111.zip
01:48:29 STEP 2689 H=2689:00 +CPU= 19.760
01:48:50 STEP 2690 H=2690:00 +CPU= 20.174
01:49:21 STEP 2691 H=2691:00 +CPU= 31.450
01:49:42 STEP 2692 H=2692:00 +CPU= 20.278
01:50:02 STEP 2693 H=2693:00 +CPU= 19.869
01:50:35 STEP 2694 H=2694:00 +CPU= 32.120
01:50:54 STEP 2695 H=2695:00 +CPU= 19.764
01:51:15 STEP 2696 H=2696:00 +CPU= 20.341
01:51:47 STEP 2697 H=2697:00 +CPU= 31.491
01:52:07 STEP 2698 H=2698:00 +CPU= 20.346
01:52:27 STEP 2699 H=2699:00 +CPU= 19.817
01:53:02 STEP 2700 H=2700:00 +CPU= 34.896
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002700
01:53:22 STEP 2701 H=2701:00 +CPU= 19.817
01:53:43 STEP 2702 H=2702:00 +CPU= 20.334
01:54:14 STEP 2703 H=2703:00 +CPU= 31.655
01:54:35 STEP 2704 H=2704:00 +CPU= 20.376
01:54:55 STEP 2705 H=2705:00 +CPU= 19.976
01:55:28 STEP 2706 H=2706:00 +CPU= 32.446
01:55:48 STEP 2707 H=2707:00 +CPU= 19.881
01:56:08 STEP 2708 H=2708:00 +CPU= 20.371
01:56:40 STEP 2709 H=2709:00 +CPU= 31.639
01:57:01 STEP 2710 H=2710:00 +CPU= 20.366
01:57:22 STEP 2711 H=2711:00 +CPU= 20.467
01:58:01 STEP 2712 H=2712:00 +CPU= 38.812
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342_112.zip
Uploading the intermediate file: upload_file_112.zip
01:58:21 STEP 2713 H=2713:00 +CPU= 19.849
01:58:42 STEP 2714 H=2714:00 +CPU= 20.363
01:59:14 STEP 2715 H=2715:00 +CPU= 31.693
01:59:34 STEP 2716 H=2716:00 +CPU= 20.415
01:59:55 STEP 2717 H=2717:00 +CPU= 19.876
02:00:28 STEP 2718 H=2718:00 +CPU= 32.790
02:00:48 STEP 2719 H=2719:00 +CPU= 20.176
02:01:11 STEP 2720 H=2720:00 +CPU= 20.890
02:01:45 STEP 2721 H=2721:00 +CPU= 32.568
02:02:07 STEP 2722 H=2722:00 +CPU= 21.226
02:02:32 STEP 2723 H=2723:00 +CPU= 22.330
02:03:16 STEP 2724 H=2724:00 +CPU= 38.494
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002724
02:03:42 STEP 2725 H=2725:00 +CPU= 20.609
02:04:04 STEP 2726 H=2726:00 +CPU= 22.142
02:04:41 STEP 2727 H=2727:00 +CPU= 35.273
02:05:04 STEP 2728 H=2728:00 +CPU= 22.742
02:05:27 STEP 2729 H=2729:00 +CPU= 22.446
02:06:04 STEP 2730 H=2730:00 +CPU= 36.024
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0354
(argv4) batchid: 982
(argv5) wuid: 12198998
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198998
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_071f17ecaedca3358b3532192176d424 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0354_2013050100_123_982_12198998.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0354_2013050100_123_982_12198998.zip
ic_ancil_file: ic_ancil_12198998
ifsdata_file: ifsdata_12198998
climate_data_file: clim_data_12198998
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1e634b81de74fb90fdba7fe1ee97d1f1 to: /var/lib/boinc-client/slots/3/ic_ancil_12198998.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12198998.zip
02:06:28 STEP 2731 H=2731:00 +CPU= 21.812
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3ce7a9bd6633bf4b7dc8c2073f6ab459 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12198998.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12198998.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12198998.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12198998.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12198998.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12198998.xml
-- Model is restarting --
Adjusting last_iter, 2731, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12198998.xml
last_cpu_time: 68756
upload_file_number: 113
last_iter: 2711
last_upload: 9763200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0354_2013050100_123_982_12198998_1_r2055457342
The child process has been launched with process id: 157673
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020639:1674698799:0.034] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
02:06:56 STEP 2732 H=2732:00 +CPU= 23.263
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line =
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line =
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line =
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line =
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line =
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line =
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line =
02:08:13 STEP 2733 H=2733:00 +CPU= 35.122
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line =
02:08:36 STEP 2734 H=2734:00 +CPU= 21.273
02:09:00 STEP 2735 H=2735:00 +CPU= 21.372
ABORT! 1 !! *** WAVE MODEL HAS ABORTED
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.388] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3223 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.388] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.426] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : INITMDL
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020927:1674698967:168.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : USERIN
SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1
Process 1 thread 1 calling linux_trbk from intel_trbk()
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/3/oifs_43r3_model.exe' : sigcontextptr=0x7fff5fc326a0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/3/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020929:1674698969:170.486] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3223MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020929:1674698969:170.486] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020929:1674698969:170.486] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020929:1674698969:170.486] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x70000267e9], nsigs = 1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020929:1674698969:170.486] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020930:1674698970:171.486] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:heinlein:1:1:157673:157673] [20230126:020930:1674698970:171.486] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC903B Unknown Unknown Unknown
oifs_43r3_model.e 00000000008C2880 sdl_mod_mp_sdl_sr 100 sdl_mod.F90
oifs_43r3_model.e 000000000081D8AF abor1_ 41 abor1.F90
oifs_43r3_model.e 00000000015F8ACA mpabort_ 50 mpabort.F
oifs_43r3_model.e 00000000015D51C8 abort1_ 44 abort1.F
oifs_43r3_model.e 000000000166B07F userin_ 553 userin.F
oifs_43r3_model.e 00000000015D7FF0 initmdl_ 452 initmdl.F
oifs_43r3_model.e 00000000008711E0 wavemdl_ 393 wavemdl.F
oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90
oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
The child process terminated with status: 0
02:10:17 STEP 2736 H=2736:00 +CPU= 39.624
02:10:39 STEP 2737 H=2737:00 +CPU= 21.834
..Failed, model did not complete successfully
</stderr_txt>
]]>
