<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ts/0/progress_file_12175027.xml
-- Model is restarting --
Adjusting last_iter, 2, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12175027.xml
last_cpu_time: 1486
upload_file_number: 0
last_iter: -1
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762
The child process has been launched with process id: 114127
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:114127:114127] [20230120:021635:1674148595:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:19:03 STEP 0 H= 0:00 +CPU= 35.104
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000000
02:19:28 STEP 1 H= 1:00 +CPU= 24.681
02:19:50 STEP 2 H= 2:00 +CPU= 21.769
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:23:57 STEP 3 H= 3:00 +CPU= 34.119
02:24:19 STEP 4 H= 4:00 +CPU= 22.519
02:24:41 STEP 5 H= 5:00 +CPU= 21.417
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:26:57 STEP 6 H= 6:00 +CPU= 34.204
02:27:18 STEP 7 H= 7:00 +CPU= 20.977
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:29:50 STEP 8 H= 8:00 +CPU= 21.340
02:30:24 STEP 9 H= 9:00 +CPU= 33.063
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:33:17 STEP 10 H= 10:00 +CPU= 21.758
02:33:39 STEP 11 H= 11:00 +CPU= 22.092
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:35:47 STEP 12 H= 12:00 +CPU= 37.528
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000012
02:36:08 STEP 13 H= 13:00 +CPU= 20.563
02:36:30 STEP 14 H= 14:00 +CPU= 21.493
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:38:46 STEP 15 H= 15:00 +CPU= 34.075
02:39:08 STEP 16 H= 16:00 +CPU= 21.704
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:41:30 STEP 17 H= 17:00 +CPU= 21.276
02:42:03 STEP 18 H= 18:00 +CPU= 33.228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:44:14 STEP 19 H= 19:00 +CPU= 20.781
02:44:36 STEP 20 H= 20:00 +CPU= 21.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:46:51 STEP 21 H= 21:00 +CPU= 33.388
02:47:12 STEP 22 H= 22:00 +CPU= 20.952
02:47:33 STEP 23 H= 23:00 +CPU= 20.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:50:03 STEP 24 H= 24:00 +CPU= 38.044
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000012
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762_0.zip
Uploading the intermediate file: upload_file_0.zip
02:50:24 STEP 25 H= 25:00 +CPU= 20.640
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:52:32 STEP 26 H= 26:00 +CPU= 21.033
02:53:05 STEP 27 H= 27:00 +CPU= 32.325
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:55:27 STEP 28 H= 28:00 +CPU= 21.072
02:55:48 STEP 29 H= 29:00 +CPU= 20.726
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
02:58:14 STEP 30 H= 30:00 +CPU= 33.667
02:58:35 STEP 31 H= 31:00 +CPU= 20.584
02:58:56 STEP 32 H= 32:00 +CPU= 20.882
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:01:20 STEP 33 H= 33:00 +CPU= 32.879
03:01:41 STEP 34 H= 34:00 +CPU= 20.650
03:02:01 STEP 35 H= 35:00 +CPU= 20.440
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:04:19 STEP 36 H= 36:00 +CPU= 36.282
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000036
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:06:21 STEP 37 H= 37:00 +CPU= 20.734
03:06:42 STEP 38 H= 38:00 +CPU= 21.035
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:08:57 STEP 39 H= 39:00 +CPU= 33.771
03:09:19 STEP 40 H= 40:00 +CPU= 21.771
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:11:21 STEP 41 H= 41:00 +CPU= 21.067
03:11:56 STEP 42 H= 42:00 +CPU= 33.879
03:12:15 STEP 43 H= 43:00 +CPU= 19.756
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:14:48 STEP 44 H= 44:00 +CPU= 21.393
03:15:23 STEP 45 H= 45:00 +CPU= 34.032
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:18:06 STEP 46 H= 46:00 +CPU= 22.001
03:18:28 STEP 47 H= 47:00 +CPU= 21.441
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:20:59 STEP 48 H= 48:00 +CPU= 39.871
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000048
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000048
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000036
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000036
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000036
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762_1.zip
Uploading the intermediate file: upload_file_1.zip
03:21:20 STEP 49 H= 49:00 +CPU= 20.864
03:21:42 STEP 50 H= 50:00 +CPU= 21.326
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:24:08 STEP 51 H= 51:00 +CPU= 33.436
03:24:29 STEP 52 H= 52:00 +CPU= 21.393
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:26:32 STEP 53 H= 53:00 +CPU= 20.369
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:28:27 STEP 54 H= 54:00 +CPU= 33.497
03:28:48 STEP 55 H= 55:00 +CPU= 21.053
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:30:41 STEP 56 H= 56:00 +CPU= 21.139
03:31:14 STEP 57 H= 57:00 +CPU= 32.976
03:31:35 STEP 58 H= 58:00 +CPU= 21.124
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:34:08 STEP 59 H= 59:00 +CPU= 21.206
03:34:45 STEP 60 H= 60:00 +CPU= 36.758
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000060
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:36:57 STEP 61 H= 61:00 +CPU= 20.610
03:37:19 STEP 62 H= 62:00 +CPU= 21.475
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:39:24 STEP 63 H= 63:00 +CPU= 33.320
03:39:47 STEP 64 H= 64:00 +CPU= 22.274
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:42:40 STEP 65 H= 65:00 +CPU= 21.150
03:43:16 STEP 66 H= 66:00 +CPU= 34.741
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:45:18 STEP 67 H= 67:00 +CPU= 21.140
03:45:40 STEP 68 H= 68:00 +CPU= 21.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:48:05 STEP 69 H= 69:00 +CPU= 33.429
03:48:27 STEP 70 H= 70:00 +CPU= 22.332
03:48:49 STEP 71 H= 71:00 +CPU= 21.354
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:51:20 STEP 72 H= 72:00 +CPU= 39.551
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000072
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000072
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000060
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000060
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000060
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762_2.zip
Uploading the intermediate file: upload_file_2.zip
03:51:40 STEP 73 H= 73:00 +CPU= 20.624
03:52:02 STEP 74 H= 74:00 +CPU= 21.319
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:54:26 STEP 75 H= 75:00 +CPU= 33.069
03:54:48 STEP 76 H= 76:00 +CPU= 21.643
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:56:22 STEP 77 H= 77:00 +CPU= 21.413
03:56:57 STEP 78 H= 78:00 +CPU= 34.750
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:59:19 STEP 79 H= 79:00 +CPU= 20.982
03:59:41 STEP 80 H= 80:00 +CPU= 21.202
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:01:36 STEP 81 H= 81:00 +CPU= 33.621
04:01:58 STEP 82 H= 82:00 +CPU= 22.102
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:04:20 STEP 83 H= 83:00 +CPU= 20.668
04:04:58 STEP 84 H= 84:00 +CPU= 37.337
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000084
04:05:18 STEP 85 H= 85:00 +CPU= 20.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:07:10 STEP 86 H= 86:00 +CPU= 21.371
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:09:04 STEP 87 H= 87:00 +CPU= 32.918
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:11:07 STEP 88 H= 88:00 +CPU= 21.642
04:11:28 STEP 89 H= 89:00 +CPU= 20.563
04:12:02 STEP 90 H= 90:00 +CPU= 34.419
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:13:56 STEP 91 H= 91:00 +CPU= 21.750
04:14:17 STEP 92 H= 92:00 +CPU= 21.542
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:16:32 STEP 93 H= 93:00 +CPU= 33.641
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0383
(argv4) batchid: 958
(argv5) wuid: 12175027
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4ebd472324b9b7aa44216f4e5d0890e2 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0383_1989050100_123_958_12175027.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0383_1989050100_123_958_12175027.zip
ic_ancil_file: ic_ancil_12175027
ifsdata_file: ifsdata_12175027
climate_data_file: clim_data_12175027
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_09383d4610a953a6e7bb55b377e24cb8 to: /var/lib/boinc-client/slots/0/ic_ancil_12175027.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12175027.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_65622488adaccbc85e60a05eb4a4f993 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12175027.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12175027.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12175027.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12175027.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12175027.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12175027.xml
-- Model is restarting --
Adjusting last_iter, 93, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12175027.xml
last_cpu_time: 3909
upload_file_number: 3
last_iter: 71
last_upload: 259200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762
The child process has been launched with process id: 139123
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:139123:139123] [20230120:041817:1674155897:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
04:18:57 STEP 72 H= 72:00 +CPU= 37.408
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:20:29 STEP 73 H= 73:00 +CPU= 21.516
04:20:50 STEP 74 H= 74:00 +CPU= 21.054
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:22:46 STEP 75 H= 75:00 +CPU= 33.718
04:23:08 STEP 76 H= 76:00 +CPU= 21.556
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:24:51 STEP 77 H= 77:00 +CPU= 20.830
04:25:26 STEP 78 H= 78:00 +CPU= 34.529
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:27:19 STEP 79 H= 79:00 +CPU= 21.164
04:27:41 STEP 80 H= 80:00 +CPU= 22.085
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:29:25 STEP 81 H= 81:00 +CPU= 34.087
04:29:47 STEP 82 H= 82:00 +CPU= 21.455
04:30:08 STEP 83 H= 83:00 +CPU= 20.751
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:32:08 STEP 84 H= 84:00 +CPU= 36.997
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000084
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:33:30 STEP 85 H= 85:00 +CPU= 21.123
04:33:52 STEP 86 H= 86:00 +CPU= 21.582
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:35:36 STEP 87 H= 87:00 +CPU= 32.886
04:35:58 STEP 88 H= 88:00 +CPU= 21.801
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:37:41 STEP 89 H= 89:00 +CPU= 20.563
04:38:15 STEP 90 H= 90:00 +CPU= 34.325
04:38:36 STEP 91 H= 91:00 +CPU= 20.743
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:40:39 STEP 92 H= 92:00 +CPU= 21.566
04:41:12 STEP 93 H= 93:00 +CPU= 32.785
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:42:56 STEP 94 H= 94:00 +CPU= 21.115
04:43:18 STEP 95 H= 95:00 +CPU= 21.804
04:43:56 STEP 96 H= 96:00 +CPU= 38.259
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000096
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000096
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000084
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000084
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000084
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762_3.zip
Uploading the intermediate file: upload_file_3.zip
04:44:16 STEP 97 H= 97:00 +CPU= 19.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:45:41 STEP 98 H= 98:00 +CPU= 21.171
04:46:15 STEP 99 H= 99:00 +CPU= 33.521
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:47:49 STEP 100 H= 100:00 +CPU= 21.634
04:48:10 STEP 101 H= 101:00 +CPU= 20.832
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:49:56 STEP 102 H= 102:00 +CPU= 33.725
04:50:17 STEP 103 H= 103:00 +CPU= 20.640
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:51:50 STEP 104 H= 104:00 +CPU= 21.470
04:52:23 STEP 105 H= 105:00 +CPU= 33.060
04:52:45 STEP 106 H= 106:00 +CPU= 21.379
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:54:28 STEP 107 H= 107:00 +CPU= 21.203
04:55:05 STEP 108 H= 108:00 +CPU= 36.796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000108
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000108
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000108
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:56:58 STEP 109 H= 109:00 +CPU= 21.544
04:57:21 STEP 110 H= 110:00 +CPU= 22.043
04:57:54 STEP 111 H= 111:00 +CPU= 32.937
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:59:46 STEP 112 H= 112:00 +CPU= 22.090
05:00:08 STEP 113 H= 113:00 +CPU= 21.315
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:02:02 STEP 114 H= 114:00 +CPU= 33.980
05:02:23 STEP 115 H= 115:00 +CPU= 21.036
05:02:46 STEP 116 H= 116:00 +CPU= 21.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:04:49 STEP 117 H= 117:00 +CPU= 33.462
05:05:11 STEP 118 H= 118:00 +CPU= 21.467
05:05:32 STEP 119 H= 119:00 +CPU= 21.146
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:07:43 STEP 120 H= 120:00 +CPU= 39.173
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000108
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762_4.zip
Uploading the intermediate file: upload_file_4.zip
05:08:04 STEP 121 H= 121:00 +CPU= 20.845
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:09:49 STEP 122 H= 122:00 +CPU= 22.203
05:10:22 STEP 123 H= 123:00 +CPU= 33.467
05:10:44 STEP 124 H= 124:00 +CPU= 21.655
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:12:36 STEP 125 H= 125:00 +CPU= 20.619
05:13:10 STEP 126 H= 126:00 +CPU= 33.846
05:13:31 STEP 127 H= 127:00 +CPU= 20.575
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:15:04 STEP 128 H= 128:00 +CPU= 21.682
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:17:09 STEP 129 H= 129:00 +CPU= 33.644
05:17:31 STEP 130 H= 130:00 +CPU= 22.190
05:17:53 STEP 131 H= 131:00 +CPU= 21.087
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:19:50 STEP 132 H= 132:00 +CPU= 36.284
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000132
05:20:12 STEP 133 H= 133:00 +CPU= 20.972
05:20:34 STEP 134 H= 134:00 +CPU= 21.858
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:22:28 STEP 135 H= 135:00 +CPU= 33.009
05:22:49 STEP 136 H= 136:00 +CPU= 21.284
05:23:10 STEP 137 H= 137:00 +CPU= 20.735
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:25:05 STEP 138 H= 138:00 +CPU= 33.404
05:25:26 STEP 139 H= 139:00 +CPU= 20.700
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:27:09 STEP 140 H= 140:00 +CPU= 21.658
05:27:42 STEP 141 H= 141:00 +CPU= 33.026
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:29:26 STEP 142 H= 142:00 +CPU= 21.699
05:29:48 STEP 143 H= 143:00 +CPU= 21.505
05:30:27 STEP 144 H= 144:00 +CPU= 39.075
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000144
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000144
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000132
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000132
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000132
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762_5.zip
Uploading the intermediate file: upload_file_5.zip
05:32:11 STEP 145 H= 145:00 +CPU= 21.224
05:32:33 STEP 146 H= 146:00 +CPU= 21.936
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:34:38 STEP 147 H= 147:00 +CPU= 33.872
05:35:00 STEP 148 H= 148:00 +CPU= 21.556
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:36:42 STEP 149 H= 149:00 +CPU= 20.904
05:37:17 STEP 150 H= 150:00 +CPU= 34.306
05:37:38 STEP 151 H= 151:00 +CPU= 21.074
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:39:21 STEP 152 H= 152:00 +CPU= 21.469
05:39:54 STEP 153 H= 153:00 +CPU= 32.923
05:40:16 STEP 154 H= 154:00 +CPU= 21.263
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:42:19 STEP 155 H= 155:00 +CPU= 21.283
05:42:56 STEP 156 H= 156:00 +CPU= 36.778
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000156
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000156
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000156
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:44:40 STEP 157 H= 157:00 +CPU= 21.328
05:45:02 STEP 158 H= 158:00 +CPU= 21.661
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:46:57 STEP 159 H= 159:00 +CPU= 33.152
05:47:20 STEP 160 H= 160:00 +CPU= 22.586
05:47:41 STEP 161 H= 161:00 +CPU= 21.124
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:49:47 STEP 162 H= 162:00 +CPU= 34.291
05:50:09 STEP 163 H= 163:00 +CPU= 21.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:51:52 STEP 164 H= 164:00 +CPU= 21.659
05:52:25 STEP 165 H= 165:00 +CPU= 33.411
05:52:46 STEP 166 H= 166:00 +CPU= 20.847
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:54:19 STEP 167 H= 167:00 +CPU= 21.728
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:56:11 STEP 168 H= 168:00 +CPU= 39.079
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000168
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000168
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMGGhq0f+000156
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMSHhq0f+000156
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027/ICMUAhq0f+000156
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762_6.zip
Uploading the intermediate file: upload_file_6.zip
05:56:32 STEP 169 H= 169:00 +CPU= 20.968
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
05:58:05 STEP 170 H= 170:00 +CPU= 21.690
05:58:38 STEP 171 H= 171:00 +CPU= 33.111
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:00:22 STEP 172 H= 172:00 +CPU= 22.052
06:00:43 STEP 173 H= 173:00 +CPU= 21.053
06:01:17 STEP 174 H= 174:00 +CPU= 33.890
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:03:09 STEP 175 H= 175:00 +CPU= 20.832
06:03:31 STEP 176 H= 176:00 +CPU= 21.889
06:04:05 STEP 177 H= 177:00 +CPU= 33.387
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:06:19 STEP 178 H= 178:00 +CPU= 21.714
06:06:41 STEP 179 H= 179:00 +CPU= 21.154
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:08:28 STEP 180 H= 180:00 +CPU= 36.818
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000180
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000180
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000180
06:08:49 STEP 181 H= 181:00 +CPU= 21.353
06:09:11 STEP 182 H= 182:00 +CPU= 21.262
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:11:15 STEP 183 H= 183:00 +CPU= 33.659
06:11:37 STEP 184 H= 184:00 +CPU= 22.130
06:11:58 STEP 185 H= 185:00 +CPU= 20.316
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:13:54 STEP 186 H= 186:00 +CPU= 34.110
06:14:15 STEP 187 H= 187:00 +CPU= 20.886
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:15:49 STEP 188 H= 188:00 +CPU= 21.631
06:16:23 STEP 189 H= 189:00 +CPU= 33.671
06:16:45 STEP 190 H= 190:00 +CPU= 21.389
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:18:17 STEP 191 H= 191:00 +CPU= 21.783
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0383
(argv4) batchid: 958
(argv5) wuid: 12175027
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175027
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4ebd472324b9b7aa44216f4e5d0890e2 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0383_1989050100_123_958_12175027.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0383_1989050100_123_958_12175027.zip
ic_ancil_file: ic_ancil_12175027
ifsdata_file: ifsdata_12175027
climate_data_file: clim_data_12175027
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_09383d4610a953a6e7bb55b377e24cb8 to: /var/lib/boinc-client/slots/0/ic_ancil_12175027.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12175027.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_65622488adaccbc85e60a05eb4a4f993 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12175027.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12175027.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12175027.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12175027.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12175027.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12175027.xml
-- Model is restarting --
Adjusting last_iter, 191, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12175027.xml
last_cpu_time: 6996
upload_file_number: 7
last_iter: 167
last_upload: 604800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0383_1989050100_123_958_12175027_0_r1006138762
The child process has been launched with process id: 156941
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062042:1674163242:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ABORT! 1 !! *** WAVE MODEL HAS ABORTED
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3520 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : INITMDL
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : USERIN
SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1
Process 1 thread 1 calling linux_trbk from intel_trbk()
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/0/oifs_43r3_model.exe' : sigcontextptr=0x7ffcc1877120
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/0/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.097] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3520MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.097] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.097] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.097] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x920002650d], nsigs = 1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062117:1674163277:35.097] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062118:1674163278:36.097] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:p500:1:1:156941:156941] [20230120:062118:1674163278:36.097] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC903B Unknown Unknown Unknown
oifs_43r3_model.e 00000000008C2880 sdl_mod_mp_sdl_sr 100 sdl_mod.F90
oifs_43r3_model.e 000000000081D8AF abor1_ 41 abor1.F90
oifs_43r3_model.e 00000000015F8ACA mpabort_ 50 mpabort.F
oifs_43r3_model.e 00000000015D51C8 abort1_ 44 abort1.F
oifs_43r3_model.e 000000000166B07F userin_ 553 userin.F
oifs_43r3_model.e 00000000015D7FF0 initmdl_ 452 initmdl.F
oifs_43r3_model.e 00000000008711E0 wavemdl_ 393 wavemdl.F
oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90
oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully
</stderr_txt>
]]>
