<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
129fe0
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21732:21732] [20230123:115913:1674471553:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
12:00:10 STEP 1008 H=1008:00 +CPU= 50.319
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_41.zip
Uploading the intermediate file: upload_file_41.zip
12:00:40 STEP 1009 H=1009:00 +CPU= 29.328
12:01:14 STEP 1010 H=1010:00 +CPU= 31.603
12:01:54 STEP 1011 H=1011:00 +CPU= 39.564
12:02:27 STEP 1012 H=1012:00 +CPU= 31.590
12:02:53 STEP 1013 H=1013:00 +CPU= 25.215
12:03:42 STEP 1014 H=1014:00 +CPU= 47.488
12:04:14 STEP 1015 H=1015:00 +CPU= 30.229
12:04:41 STEP 1016 H=1016:00 +CPU= 26.011
12:05:27 STEP 1017 H=1017:00 +CPU= 45.204
12:05:55 STEP 1018 H=1018:00 +CPU= 26.340
12:06:24 STEP 1019 H=1019:00 +CPU= 28.499
12:07:16 STEP 1020 H=1020:00 +CPU= 49.728
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001020
12:07:42 STEP 1021 H=1021:00 +CPU= 25.727
12:08:14 STEP 1022 H=1022:00 +CPU= 31.063
12:08:55 STEP 1023 H=1023:00 +CPU= 40.012
12:09:31 STEP 1024 H=1024:00 +CPU= 33.921
12:09:57 STEP 1025 H=1025:00 +CPU= 25.988
12:10:41 STEP 1026 H=1026:00 +CPU= 42.798
12:11:11 STEP 1027 H=1027:00 +CPU= 29.102
12:11:39 STEP 1028 H=1028:00 +CPU= 26.249
12:12:24 STEP 1029 H=1029:00 +CPU= 43.479
12:12:51 STEP 1030 H=1030:00 +CPU= 26.386
12:13:23 STEP 1031 H=1031:00 +CPU= 30.386
12:14:18 STEP 1032 H=1032:00 +CPU= 53.770
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_42.zip
Uploading the intermediate file: upload_file_42.zip
12:14:45 STEP 1033 H=1033:00 +CPU= 26.162
12:15:18 STEP 1034 H=1034:00 +CPU= 31.434
12:15:59 STEP 1035 H=1035:00 +CPU= 39.960
12:16:34 STEP 1036 H=1036:00 +CPU= 33.494
12:17:01 STEP 1037 H=1037:00 +CPU= 26.626
12:17:49 STEP 1038 H=1038:00 +CPU= 46.089
12:18:19 STEP 1039 H=1039:00 +CPU= 28.249
12:18:46 STEP 1040 H=1040:00 +CPU= 26.343
12:19:33 STEP 1041 H=1041:00 +CPU= 45.272
12:20:00 STEP 1042 H=1042:00 +CPU= 26.321
12:20:32 STEP 1043 H=1043:00 +CPU= 29.401
12:21:44 STEP 1044 H=1044:00 +CPU= 61.817
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001044
12:22:23 STEP 1045 H=1045:00 +CPU= 35.535
12:23:02 STEP 1046 H=1046:00 +CPU= 33.889
12:23:47 STEP 1047 H=1047:00 +CPU= 44.003
12:24:16 STEP 1048 H=1048:00 +CPU= 28.292
12:24:43 STEP 1049 H=1049:00 +CPU= 25.455
12:25:31 STEP 1050 H=1050:00 +CPU= 46.540
12:25:58 STEP 1051 H=1051:00 +CPU= 26.118
12:26:30 STEP 1052 H=1052:00 +CPU= 30.191
12:27:16 STEP 1053 H=1053:00 +CPU= 44.559
12:27:43 STEP 1054 H=1054:00 +CPU= 26.485
12:28:17 STEP 1055 H=1055:00 +CPU= 32.612
12:29:09 STEP 1056 H=1056:00 +CPU= 49.491
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_43.zip
Uploading the intermediate file: upload_file_43.zip
12:29:55 STEP 1057 H=1057:00 +CPU= 37.330
12:30:24 STEP 1058 H=1058:00 +CPU= 28.193
12:31:12 STEP 1059 H=1059:00 +CPU= 46.544
12:31:41 STEP 1060 H=1060:00 +CPU= 27.901
12:32:07 STEP 1061 H=1061:00 +CPU= 25.393
12:32:55 STEP 1062 H=1062:00 +CPU= 46.477
12:33:23 STEP 1063 H=1063:00 +CPU= 26.898
12:33:54 STEP 1064 H=1064:00 +CPU= 29.890
12:34:42 STEP 1065 H=1065:00 +CPU= 46.105
12:35:10 STEP 1066 H=1066:00 +CPU= 26.966
12:35:40 STEP 1067 H=1067:00 +CPU= 29.583
12:36:27 STEP 1068 H=1068:00 +CPU= 45.165
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001068
12:37:02 STEP 1069 H=1069:00 +CPU= 33.533
12:37:29 STEP 1070 H=1070:00 +CPU= 26.458
12:38:11 STEP 1071 H=1071:00 +CPU= 41.503
12:38:40 STEP 1072 H=1072:00 +CPU= 28.689
12:39:05 STEP 1073 H=1073:00 +CPU= 24.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:40:20 STEP 1074 H=1074:00 +CPU= 56.085
12:40:49 STEP 1075 H=1075:00 +CPU= 27.409
12:41:29 STEP 1076 H=1076:00 +CPU= 34.988
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:42:40 STEP 1077 H=1077:00 +CPU= 53.251
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:45:10 STEP 1078 H=1078:00 +CPU= 41.021
12:45:54 STEP 1079 H=1079:00 +CPU= 38.621
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:47:23 STEP 1080 H=1080:00 +CPU= 66.475
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_44.zip
Uploading the intermediate file: upload_file_44.zip
12:47:58 STEP 1081 H=1081:00 +CPU= 32.097
12:48:41 STEP 1082 H=1082:00 +CPU= 38.031
12:49:28 STEP 1083 H=1083:00 +CPU= 44.752
12:50:01 STEP 1084 H=1084:00 +CPU= 31.489
12:50:33 STEP 1085 H=1085:00 +CPU= 30.931
12:51:18 STEP 1086 H=1086:00 +CPU= 42.936
12:51:57 STEP 1087 H=1087:00 +CPU= 36.346
12:52:31 STEP 1088 H=1088:00 +CPU= 31.455
12:53:23 STEP 1089 H=1089:00 +CPU= 49.466
12:53:53 STEP 1090 H=1090:00 +CPU= 28.377
12:54:29 STEP 1091 H=1091:00 +CPU= 32.902
12:55:29 STEP 1092 H=1092:00 +CPU= 55.984
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001092
12:55:59 STEP 1093 H=1093:00 +CPU= 28.455
12:56:33 STEP 1094 H=1094:00 +CPU= 31.870
12:57:17 STEP 1095 H=1095:00 +CPU= 42.526
12:57:52 STEP 1096 H=1096:00 +CPU= 33.592
12:58:19 STEP 1097 H=1097:00 +CPU= 26.370
12:59:09 STEP 1098 H=1098:00 +CPU= 48.185
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0574
(argv4) batchid: 957
(argv5) wuid: 12174218
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_46429dcb2608ada66e0c05d2f3f4c272 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0574_1988050100_123_957_12174218.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0574_1988050100_123_957_12174218.zip
ic_ancil_file: ic_ancil_12174218
ifsdata_file: ifsdata_12174218
climate_data_file: clim_data_12174218
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_80f59dfa961b27f33255d72c0244b74e to: /var/lib/boinc-client/slots/3/ic_ancil_12174218.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12174218.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d91c825486b70a0ca6051b68ef070a6c to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174218.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174218.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174218.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174218.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12174218.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12174218.xml
-- Model is restarting --
Adjusting last_iter, 1098, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12174218.xml
last_cpu_time: 34526
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461
The child process has been launched with process id: 24317
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24317:24317] [20230123:130021:1674475221:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
13:01:18 STEP 1080 H=1080:00 +CPU= 50.196
13:01:51 STEP 1081 H=1081:00 +CPU= 31.118
13:02:22 STEP 1082 H=1082:00 +CPU= 30.351
13:03:05 STEP 1083 H=1083:00 +CPU= 41.349
13:03:38 STEP 1084 H=1084:00 +CPU= 31.736
13:04:05 STEP 1085 H=1085:00 +CPU= 26.128
13:04:53 STEP 1086 H=1086:00 +CPU= 46.757
13:05:20 STEP 1087 H=1087:00 +CPU= 26.477
13:05:47 STEP 1088 H=1088:00 +CPU= 26.429
13:06:40 STEP 1089 H=1089:00 +CPU= 50.255
13:07:09 STEP 1090 H=1090:00 +CPU= 27.760
13:07:42 STEP 1091 H=1091:00 +CPU= 31.448
13:08:33 STEP 1092 H=1092:00 +CPU= 49.779
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001092
13:09:00 STEP 1093 H=1093:00 +CPU= 26.619
13:09:33 STEP 1094 H=1094:00 +CPU= 31.244
13:10:15 STEP 1095 H=1095:00 +CPU= 40.805
13:10:50 STEP 1096 H=1096:00 +CPU= 34.072
13:11:17 STEP 1097 H=1097:00 +CPU= 26.143
13:12:02 STEP 1098 H=1098:00 +CPU= 44.079
13:12:32 STEP 1099 H=1099:00 +CPU= 28.590
13:12:59 STEP 1100 H=1100:00 +CPU= 26.338
13:13:46 STEP 1101 H=1101:00 +CPU= 45.571
13:14:14 STEP 1102 H=1102:00 +CPU= 26.875
13:14:43 STEP 1103 H=1103:00 +CPU= 28.740
13:15:40 STEP 1104 H=1104:00 +CPU= 54.679
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_45.zip
Uploading the intermediate file: upload_file_45.zip
13:16:08 STEP 1105 H=1105:00 +CPU= 27.052
13:16:42 STEP 1106 H=1106:00 +CPU= 33.394
13:17:25 STEP 1107 H=1107:00 +CPU= 41.080
13:18:03 STEP 1108 H=1108:00 +CPU= 34.597
13:18:33 STEP 1109 H=1109:00 +CPU= 28.301
13:19:18 STEP 1110 H=1110:00 +CPU= 44.392
13:19:47 STEP 1111 H=1111:00 +CPU= 28.352
13:20:11 STEP 1112 H=1112:00 +CPU= 23.249
13:20:56 STEP 1113 H=1113:00 +CPU= 43.763
13:21:26 STEP 1114 H=1114:00 +CPU= 29.093
13:21:53 STEP 1115 H=1115:00 +CPU= 26.277
13:22:55 STEP 1116 H=1116:00 +CPU= 58.450
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001116
13:23:27 STEP 1117 H=1117:00 +CPU= 31.855
13:24:13 STEP 1118 H=1118:00 +CPU= 39.812
13:25:04 STEP 1119 H=1119:00 +CPU= 50.622
13:25:44 STEP 1120 H=1120:00 +CPU= 37.933
13:26:16 STEP 1121 H=1121:00 +CPU= 31.572
13:27:17 STEP 1122 H=1122:00 +CPU= 58.289
13:27:53 STEP 1123 H=1123:00 +CPU= 34.008
13:28:36 STEP 1124 H=1124:00 +CPU= 38.225
13:29:39 STEP 1125 H=1125:00 +CPU= 56.938
13:30:22 STEP 1126 H=1126:00 +CPU= 37.021
13:31:04 STEP 1127 H=1127:00 +CPU= 37.956
13:32:11 STEP 1128 H=1128:00 +CPU= 64.502
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_46.zip
Uploading the intermediate file: upload_file_46.zip
13:32:43 STEP 1129 H=1129:00 +CPU= 31.780
13:33:20 STEP 1130 H=1130:00 +CPU= 35.473
13:34:18 STEP 1131 H=1131:00 +CPU= 55.291
13:35:06 STEP 1132 H=1132:00 +CPU= 40.847
13:35:40 STEP 1133 H=1133:00 +CPU= 32.664
13:36:44 STEP 1134 H=1134:00 +CPU= 58.449
13:37:26 STEP 1135 H=1135:00 +CPU= 36.454
13:38:13 STEP 1136 H=1136:00 +CPU= 40.414
13:39:18 STEP 1137 H=1137:00 +CPU= 58.144
13:40:08 STEP 1138 H=1138:00 +CPU= 40.741
13:40:52 STEP 1139 H=1139:00 +CPU= 37.668
13:42:05 STEP 1140 H=1140:00 +CPU= 65.144
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001140
13:42:45 STEP 1141 H=1141:00 +CPU= 36.467
13:43:22 STEP 1142 H=1142:00 +CPU= 35.572
13:44:28 STEP 1143 H=1143:00 +CPU= 58.844
13:45:08 STEP 1144 H=1144:00 +CPU= 36.004
13:45:42 STEP 1145 H=1145:00 +CPU= 33.209
13:46:40 STEP 1146 H=1146:00 +CPU= 56.020
13:47:14 STEP 1147 H=1147:00 +CPU= 33.002
13:47:50 STEP 1148 H=1148:00 +CPU= 34.662
13:48:46 STEP 1149 H=1149:00 +CPU= 54.146
13:49:21 STEP 1150 H=1150:00 +CPU= 34.151
13:49:57 STEP 1151 H=1151:00 +CPU= 34.521
13:51:06 STEP 1152 H=1152:00 +CPU= 66.246
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_47.zip
Uploading the intermediate file: upload_file_47.zip
13:51:42 STEP 1153 H=1153:00 +CPU= 34.504
13:52:20 STEP 1154 H=1154:00 +CPU= 36.091
13:53:15 STEP 1155 H=1155:00 +CPU= 53.455
13:53:53 STEP 1156 H=1156:00 +CPU= 36.112
13:54:26 STEP 1157 H=1157:00 +CPU= 32.501
13:55:27 STEP 1158 H=1158:00 +CPU= 57.630
13:55:59 STEP 1159 H=1159:00 +CPU= 31.694
13:56:42 STEP 1160 H=1160:00 +CPU= 38.791
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:59:19 STEP 1161 H=1161:00 +CPU= 52.559
13:59:51 STEP 1162 H=1162:00 +CPU= 31.915
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0574
(argv4) batchid: 957
(argv5) wuid: 12174218
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_46429dcb2608ada66e0c05d2f3f4c272 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0574_1988050100_123_957_12174218.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0574_1988050100_123_957_12174218.zip
ic_ancil_file: ic_ancil_12174218
ifsdata_file: ifsdata_12174218
climate_data_file: clim_data_12174218
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_80f59dfa961b27f33255d72c0244b74e to: /var/lib/boinc-client/slots/3/ic_ancil_12174218.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12174218.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d91c825486b70a0ca6051b68ef070a6c to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174218.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174218.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174218.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174218.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12174218.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12174218.xml
-- Model is restarting --
Adjusting last_iter, 1163, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12174218.xml
last_cpu_time: 37822
upload_file_number: 48
last_iter: 1151
last_upload: 4147200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461
The child process has been launched with process id: 27745
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:27745:27745] [20230123:140129:1674478889:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
14:02:39 STEP 1152 H=1152:00 +CPU= 61.936
14:03:15 STEP 1153 H=1153:00 +CPU= 34.999
14:03:51 STEP 1154 H=1154:00 +CPU= 35.710
14:04:45 STEP 1155 H=1155:00 +CPU= 53.133
14:05:21 STEP 1156 H=1156:00 +CPU= 35.721
14:05:52 STEP 1157 H=1157:00 +CPU= 30.553
14:06:47 STEP 1158 H=1158:00 +CPU= 53.750
14:07:19 STEP 1159 H=1159:00 +CPU= 31.928
14:07:57 STEP 1160 H=1160:00 +CPU= 38.011
14:08:47 STEP 1161 H=1161:00 +CPU= 49.549
14:09:27 STEP 1162 H=1162:00 +CPU= 38.430
14:09:58 STEP 1163 H=1163:00 +CPU= 31.037
14:10:59 STEP 1164 H=1164:00 +CPU= 60.324
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001164
14:11:31 STEP 1165 H=1165:00 +CPU= 31.484
14:12:08 STEP 1166 H=1166:00 +CPU= 35.672
14:13:00 STEP 1167 H=1167:00 +CPU= 51.750
14:13:37 STEP 1168 H=1168:00 +CPU= 35.979
14:14:12 STEP 1169 H=1169:00 +CPU= 34.927
14:15:05 STEP 1170 H=1170:00 +CPU= 52.724
14:15:42 STEP 1171 H=1171:00 +CPU= 35.028
14:16:15 STEP 1172 H=1172:00 +CPU= 33.038
14:17:12 STEP 1173 H=1173:00 +CPU= 55.580
14:17:45 STEP 1174 H=1174:00 +CPU= 32.685
14:18:24 STEP 1175 H=1175:00 +CPU= 37.989
14:19:23 STEP 1176 H=1176:00 +CPU= 58.582
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_48.zip
Uploading the intermediate file: upload_file_48.zip
14:20:02 STEP 1177 H=1177:00 +CPU= 37.552
14:20:34 STEP 1178 H=1178:00 +CPU= 32.143
14:21:30 STEP 1179 H=1179:00 +CPU= 54.288
14:22:03 STEP 1180 H=1180:00 +CPU= 32.239
14:22:37 STEP 1181 H=1181:00 +CPU= 33.722
14:23:23 STEP 1182 H=1182:00 +CPU= 45.762
14:23:48 STEP 1183 H=1183:00 +CPU= 24.465
14:24:23 STEP 1184 H=1184:00 +CPU= 34.531
14:25:17 STEP 1185 H=1185:00 +CPU= 52.942
14:25:53 STEP 1186 H=1186:00 +CPU= 35.181
14:26:28 STEP 1187 H=1187:00 +CPU= 34.560
14:27:27 STEP 1188 H=1188:00 +CPU= 58.593
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001188
14:28:01 STEP 1189 H=1189:00 +CPU= 33.403
14:28:34 STEP 1190 H=1190:00 +CPU= 32.224
14:29:27 STEP 1191 H=1191:00 +CPU= 52.500
14:30:00 STEP 1192 H=1192:00 +CPU= 32.743
14:30:38 STEP 1193 H=1193:00 +CPU= 37.400
14:31:29 STEP 1194 H=1194:00 +CPU= 50.216
14:32:04 STEP 1195 H=1195:00 +CPU= 34.854
14:32:36 STEP 1196 H=1196:00 +CPU= 31.925
14:33:30 STEP 1197 H=1197:00 +CPU= 52.860
14:34:02 STEP 1198 H=1198:00 +CPU= 31.354
14:34:40 STEP 1199 H=1199:00 +CPU= 37.131
14:35:39 STEP 1200 H=1200:00 +CPU= 58.503
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_49.zip
Uploading the intermediate file: upload_file_49.zip
14:36:15 STEP 1201 H=1201:00 +CPU= 35.210
14:36:47 STEP 1202 H=1202:00 +CPU= 31.490
14:37:37 STEP 1203 H=1203:00 +CPU= 49.728
14:38:02 STEP 1204 H=1204:00 +CPU= 25.149
14:38:27 STEP 1205 H=1205:00 +CPU= 24.388
14:39:24 STEP 1206 H=1206:00 +CPU= 56.397
14:39:56 STEP 1207 H=1207:00 +CPU= 31.775
14:40:36 STEP 1208 H=1208:00 +CPU= 38.888
14:41:24 STEP 1209 H=1209:00 +CPU= 47.176
14:42:04 STEP 1210 H=1210:00 +CPU= 38.865
14:42:35 STEP 1211 H=1211:00 +CPU= 31.638
14:43:33 STEP 1212 H=1212:00 +CPU= 57.669
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001212
14:44:06 STEP 1213 H=1213:00 +CPU= 32.253
14:44:43 STEP 1214 H=1214:00 +CPU= 36.494
14:45:34 STEP 1215 H=1215:00 +CPU= 50.654
14:46:09 STEP 1216 H=1216:00 +CPU= 34.596
14:46:43 STEP 1217 H=1217:00 +CPU= 33.153
14:47:34 STEP 1218 H=1218:00 +CPU= 51.365
14:48:10 STEP 1219 H=1219:00 +CPU= 35.040
14:48:41 STEP 1220 H=1220:00 +CPU= 31.268
14:49:33 STEP 1221 H=1221:00 +CPU= 51.457
14:50:05 STEP 1222 H=1222:00 +CPU= 31.690
14:50:42 STEP 1223 H=1223:00 +CPU= 36.470
14:51:41 STEP 1224 H=1224:00 +CPU= 58.718
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_50.zip
Uploading the intermediate file: upload_file_50.zip
14:52:17 STEP 1225 H=1225:00 +CPU= 34.849
14:52:51 STEP 1226 H=1226:00 +CPU= 33.654
14:53:42 STEP 1227 H=1227:00 +CPU= 50.522
14:54:18 STEP 1228 H=1228:00 +CPU= 35.760
14:54:49 STEP 1229 H=1229:00 +CPU= 30.106
14:55:46 STEP 1230 H=1230:00 +CPU= 55.738
14:56:19 STEP 1231 H=1231:00 +CPU= 32.196
14:57:06 STEP 1232 H=1232:00 +CPU= 40.569
14:58:02 STEP 1233 H=1233:00 +CPU= 52.420
14:58:44 STEP 1234 H=1234:00 +CPU= 39.718
14:59:18 STEP 1235 H=1235:00 +CPU= 33.250
15:00:22 STEP 1236 H=1236:00 +CPU= 59.564
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001236
15:00:56 STEP 1237 H=1237:00 +CPU= 32.455
15:01:39 STEP 1238 H=1238:00 +CPU= 39.410
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0574
(argv4) batchid: 957
(argv5) wuid: 12174218
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_46429dcb2608ada66e0c05d2f3f4c272 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0574_1988050100_123_957_12174218.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0574_1988050100_123_957_12174218.zip
ic_ancil_file: ic_ancil_12174218
ifsdata_file: ifsdata_12174218
climate_data_file: clim_data_12174218
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_80f59dfa961b27f33255d72c0244b74e to: /var/lib/boinc-client/slots/3/ic_ancil_12174218.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12174218.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d91c825486b70a0ca6051b68ef070a6c to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174218.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174218.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174218.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174218.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12174218.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12174218.xml
-- Model is restarting --
Adjusting last_iter, 1239, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12174218.xml
last_cpu_time: 41360
upload_file_number: 51
last_iter: 1223
last_upload: 4406400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461
The child process has been launched with process id: 28471
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:28471:28471] [20230123:150250:1674482570:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
15:04:29 STEP 1224 H=1224:00 +CPU= 71.261
15:05:15 STEP 1225 H=1225:00 +CPU= 39.093
15:05:54 STEP 1226 H=1226:00 +CPU= 36.351
15:06:58 STEP 1227 H=1227:00 +CPU= 58.407
15:07:44 STEP 1228 H=1228:00 +CPU= 38.911
15:08:38 STEP 1229 H=1229:00 +CPU= 41.169
15:10:01 STEP 1230 H=1230:00 +CPU= 65.843
15:10:43 STEP 1231 H=1231:00 +CPU= 36.340
15:11:26 STEP 1232 H=1232:00 +CPU= 38.170
15:12:29 STEP 1233 H=1233:00 +CPU= 57.465
15:13:13 STEP 1234 H=1234:00 +CPU= 39.196
15:13:50 STEP 1235 H=1235:00 +CPU= 33.836
15:15:06 STEP 1236 H=1236:00 +CPU= 67.413
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001236
15:15:51 STEP 1237 H=1237:00 +CPU= 39.868
15:16:27 STEP 1238 H=1238:00 +CPU= 34.540
15:17:32 STEP 1239 H=1239:00 +CPU= 58.579
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:18:47 STEP 1240 H=1240:00 +CPU= 41.317
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:19:51 STEP 1241 H=1241:00 +CPU= 40.899
15:21:22 STEP 1242 H=1242:00 +CPU= 68.716
15:22:16 STEP 1243 H=1243:00 +CPU= 42.753
15:23:03 STEP 1244 H=1244:00 +CPU= 39.739
15:24:14 STEP 1245 H=1245:00 +CPU= 61.130
15:24:55 STEP 1246 H=1246:00 +CPU= 36.593
15:25:45 STEP 1247 H=1247:00 +CPU= 40.847
15:27:00 STEP 1248 H=1248:00 +CPU= 66.509
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174218/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0574_1988050100_123_957_12174218_1_r812428461_51.zip
Uploading the intermediate file: upload_file_51.zip
15:27:40 STEP 1249 H=1249:00 +CPU= 33.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:28:58 STEP 1250 H=1250:00 +CPU= 33.647
16:29:49 STEP 1251 H=1251:00 +CPU= 47.896
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
