climateprediction.net home page
Task 22309829

Task 22309829

Name oifs_43r3_bl_a270_2016092300_15_991_12209631_0
Workunit 12209631
Created 13 Feb 2023, 17:14:50 UTC
Sent 14 Feb 2023, 7:22:53 UTC
Report deadline 15 Apr 2023, 7:22:53 UTC
Received 15 Feb 2023, 8:28:57 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1532237
Run time 12 hours 46 min 22 sec
CPU time 10 hours 19 min 54 sec
Validate state Valid
Credit 1,813.14
Device peak FLOPS 7.50 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,062.70 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,780.31 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
  0.0000000000E+00      I       0.2551954368E-01      I
I  Relat. Hum. I      0.1939316859E+00      I       0.2901783667E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.4845361806E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   960 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000960 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+000960
 MPI-TASK:   1 -    311887313 BYTES IN      7 RECORDS TRANSFERRED IN    0.0005 SECONDS623774.6 Mbytes/s, TOTAL TIME=    0.0092( 5.4%)
 22:08:53 STEP  960 H= 240:00 +CPU= 28.281
 NSTEP =   961 STEPO   0AAA00AAA
 22:09:14 STEP  961 H= 240:15 +CPU= 20.176
 NSTEP =   962 STEPO   0AAA00AAA
 22:09:36 STEP  962 H= 240:30 +CPU= 21.121
 NSTEP =   963 STEPO   0AAA00AAA
 22:10:00 STEP  963 H= 240:45 +CPU= 23.388
 NSTEP =   964 STEPO   0AAA00AAA
 22:10:25 STEP  964 H= 241:00 +CPU= 23.995
ISTEP=     965 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     965 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   965 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 22:08:31 FULLPOS-T DYNFPOS    960     0.434    0.434    0.457     71:08     73:39 0.59062171681482E-05       2GB       0MB
 22:08:31 0AAA00AAA STEPO      960     0.400    0.400    0.406     71:09     73:39 0.59062171681482E-05       2GB       0MB
 22:08:53 0AAA00AAA STEPO      961    20.893   20.893   21.649     71:30     74:01 0.59062171681482E-05       2GB       0MB
 22:09:14 0AAA00AAA STEPO      962    20.176   20.176   20.749     71:50     74:22 0.59062171681482E-05       2GB       0MB
 22:09:36 0AAA00AAA STEPO      963    21.122   21.122   21.913     72:11     74:43 0.59062171681482E-05       2GB       0MB
 22:10:00 0AAA00AAA STEPO      964    23.388   23.388   24.181     72:34     75:08 0.59062171681482E-05       2GB       0MB
 22:10:25 0AAA00AAA STEPO      965    23.995   23.995   24.785     72:58     75:32 0.59062171681482E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "     960",
 CTIME   = "00100000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,         948, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/1/progress_file_12209631.xml

<running_values>
  <last_cpu_time>27246.330000</last_cpu_time>
  <upload_file_number>10</upload_file_number>
  <last_iter>965</last_iter>
  <last_upload>864900</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a270
(argv4) batchid: 991
(argv5) wuid: 12209631
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2c3e10ca31a972fa858131fc190a7283 to: /var/lib/boinc-client/slots/1/oifs_43r3_bl_a270_2016092300_15_991_12209631.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_bl_a270_2016092300_15_991_12209631.zip
ic_ancil_file: ic_ancil_12209631
ifsdata_file: ifsdata_12209631
climate_data_file: clim_data_12209631
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/1/ic_ancil_12209631.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12209631.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4190fbb772898eb923f986d62600877b to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12209631.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12209631.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12209631.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12209631.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12209631.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12209631.xml
-- Model is restarting --
Adjusting last_iter, 965, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12209631.xml
last_cpu_time: 27246
upload_file_number: 10
last_iter: 961
last_upload: 864900
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a270_2016092300_15_991_12209631_0_r352161327
The child process has been launched with process id: 647314
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:647314:647314] [20230214:221136:1676437896:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:12:05 STEP  960 H= 240:00 +CPU= 24.542
  22:12:28 STEP  961 H= 240:15 +CPU= 22.370
  22:12:51 STEP  962 H= 240:30 +CPU= 22.209
  22:13:13 STEP  963 H= 240:45 +CPU= 20.624
  22:13:35 STEP  964 H= 241:00 +CPU= 21.619
  22:14:00 STEP  965 H= 241:15 +CPU= 23.388
  22:14:23 STEP  966 H= 241:30 +CPU= 22.753
  22:14:44 STEP  967 H= 241:45 +CPU= 20.313
  22:15:06 STEP  968 H= 242:00 +CPU= 21.688
  22:15:29 STEP  969 H= 242:15 +CPU= 21.534
  22:15:52 STEP  970 H= 242:30 +CPU= 22.472
  22:16:17 STEP  971 H= 242:45 +CPU= 23.583
  22:16:47 STEP  972 H= 243:00 +CPU= 29.334
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000972
  22:17:11 STEP  973 H= 243:15 +CPU= 22.846
  22:17:34 STEP  974 H= 243:30 +CPU= 22.447
  22:17:56 STEP  975 H= 243:45 +CPU= 21.621
  22:18:17 STEP  976 H= 244:00 +CPU= 20.607
  22:18:38 STEP  977 H= 244:15 +CPU= 19.865
  22:19:00 STEP  978 H= 244:30 +CPU= 21.734
  22:19:23 STEP  979 H= 244:45 +CPU= 21.698
  22:19:44 STEP  980 H= 245:00 +CPU= 20.758
  22:20:05 STEP  981 H= 245:15 +CPU= 19.634
  22:20:27 STEP  982 H= 245:30 +CPU= 21.565
  22:20:51 STEP  983 H= 245:45 +CPU= 22.824
  22:21:16 STEP  984 H= 246:00 +CPU= 25.030
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000984
  22:21:37 STEP  985 H= 246:15 +CPU= 20.214
  22:21:59 STEP  986 H= 246:30 +CPU= 21.523
  22:22:22 STEP  987 H= 246:45 +CPU= 21.843
  22:22:43 STEP  988 H= 247:00 +CPU= 20.102
  22:23:03 STEP  989 H= 247:15 +CPU= 19.835
  22:23:26 STEP  990 H= 247:30 +CPU= 21.721
  22:23:48 STEP  991 H= 247:45 +CPU= 21.555
  22:24:09 STEP  992 H= 248:00 +CPU= 20.661
  22:24:30 STEP  993 H= 248:15 +CPU= 20.231
  22:24:53 STEP  994 H= 248:30 +CPU= 21.529
  22:25:17 STEP  995 H= 248:45 +CPU= 23.080
  22:25:45 STEP  996 H= 249:00 +CPU= 27.080
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000996
  22:26:05 STEP  997 H= 249:15 +CPU= 20.058
  22:26:26 STEP  998 H= 249:30 +CPU= 20.119
  22:26:49 STEP  999 H= 249:45 +CPU= 21.809
  22:27:13 STEP 1000 H= 250:00 +CPU= 23.342
  22:27:36 STEP 1001 H= 250:15 +CPU= 22.611
  22:27:58 STEP 1002 H= 250:30 +CPU= 20.984
  22:28:19 STEP 1003 H= 250:45 +CPU= 20.245
  22:28:41 STEP 1004 H= 251:00 +CPU= 20.931
  22:29:04 STEP 1005 H= 251:15 +CPU= 22.111
  22:29:26 STEP 1006 H= 251:30 +CPU= 21.326
  22:29:46 STEP 1007 H= 251:45 +CPU= 20.270
  22:30:16 STEP 1008 H= 252:00 +CPU= 28.374
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001008
  22:30:40 STEP 1009 H= 252:15 +CPU= 22.683
  22:31:04 STEP 1010 H= 252:30 +CPU= 22.724
  22:31:27 STEP 1011 H= 252:45 +CPU= 22.828
  22:31:51 STEP 1012 H= 253:00 +CPU= 23.257
  22:32:14 STEP 1013 H= 253:15 +CPU= 21.758
  22:32:34 STEP 1014 H= 253:30 +CPU= 20.045
  22:32:55 STEP 1015 H= 253:45 +CPU= 20.011
  22:33:19 STEP 1016 H= 254:00 +CPU= 23.258
  22:33:42 STEP 1017 H= 254:15 +CPU= 22.306
  22:34:07 STEP 1018 H= 254:30 +CPU= 23.870
  22:34:31 STEP 1019 H= 254:45 +CPU= 23.032
  22:35:00 STEP 1020 H= 255:00 +CPU= 28.079
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001020
  22:35:21 STEP 1021 H= 255:15 +CPU= 20.459
  22:35:42 STEP 1022 H= 255:30 +CPU= 20.500
  22:36:04 STEP 1023 H= 255:45 +CPU= 21.502
  22:36:29 STEP 1024 H= 256:00 +CPU= 23.603
  22:36:50 STEP 1025 H= 256:15 +CPU= 20.935
  22:37:11 STEP 1026 H= 256:30 +CPU= 19.933
  22:37:31 STEP 1027 H= 256:45 +CPU= 19.968
  22:37:55 STEP 1028 H= 257:00 +CPU= 23.012
  22:38:18 STEP 1029 H= 257:15 +CPU= 22.521
  22:38:42 STEP 1030 H= 257:30 +CPU= 22.762
  22:39:05 STEP 1031 H= 257:45 +CPU= 22.381
  22:39:31 STEP 1032 H= 258:00 +CPU= 25.235
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001032
  22:39:54 STEP 1033 H= 258:15 +CPU= 22.261
  22:40:18 STEP 1034 H= 258:30 +CPU= 23.171
  22:40:40 STEP 1035 H= 258:45 +CPU= 21.479
  22:41:02 STEP 1036 H= 259:00 +CPU= 20.855
  22:41:23 STEP 1037 H= 259:15 +CPU= 20.412
  22:41:47 STEP 1038 H= 259:30 +CPU= 22.519
  22:42:10 STEP 1039 H= 259:45 +CPU= 22.474
  22:42:32 STEP 1040 H= 260:00 +CPU= 21.573
  22:42:53 STEP 1041 H= 260:15 +CPU= 20.179
  22:43:15 STEP 1042 H= 260:30 +CPU= 21.236
  22:43:39 STEP 1043 H= 260:45 +CPU= 23.355
  22:44:09 STEP 1044 H= 261:00 +CPU= 28.240
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001044
  22:44:29 STEP 1045 H= 261:15 +CPU= 20.214
  22:44:50 STEP 1046 H= 261:30 +CPU= 20.282
  22:45:12 STEP 1047 H= 261:45 +CPU= 21.146
  22:45:37 STEP 1048 H= 262:00 +CPU= 23.430
  22:45:58 STEP 1049 H= 262:15 +CPU= 20.465
  22:46:19 STEP 1050 H= 262:30 +CPU= 20.481
  22:46:41 STEP 1051 H= 262:45 +CPU= 20.956
  22:47:05 STEP 1052 H= 263:00 +CPU= 23.737
  22:47:30 STEP 1053 H= 263:15 +CPU= 23.612
  22:47:51 STEP 1054 H= 263:30 +CPU= 20.668
  22:48:12 STEP 1055 H= 263:45 +CPU= 20.615
  22:48:44 STEP 1056 H= 264:00 +CPU= 30.004
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001056
  22:49:08 STEP 1057 H= 264:15 +CPU= 23.076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001056
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a270_2016092300_15_991_12209631_0_r352161327_10.zip
Uploading the intermediate file: upload_file_10.zip
  22:49:29 STEP 1058 H= 264:30 +CPU= 20.955
  22:49:51 STEP 1059 H= 264:45 +CPU= 20.602
  22:50:12 STEP 1060 H= 265:00 +CPU= 20.282
  22:50:34 STEP 1061 H= 265:15 +CPU= 21.822
  22:50:57 STEP 1062 H= 265:30 +CPU= 22.095
  22:51:18 STEP 1063 H= 265:45 +CPU= 20.311
  22:51:40 STEP 1064 H= 266:00 +CPU= 20.596
  22:52:00 STEP 1065 H= 266:15 +CPU= 20.029
  22:52:21 STEP 1066 H= 266:30 +CPU= 20.027
  22:52:42 STEP 1067 H= 266:45 +CPU= 20.243
  22:53:08 STEP 1068 H= 267:00 +CPU= 25.217
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001068
  22:53:30 STEP 1069 H= 267:15 +CPU= 20.974
  22:53:51 STEP 1070 H= 267:30 +CPU= 20.418
  22:54:12 STEP 1071 H= 267:45 +CPU= 20.537
  22:54:34 STEP 1072 H= 268:00 +CPU= 20.990
  22:54:55 STEP 1073 H= 268:15 +CPU= 20.712
  22:55:16 STEP 1074 H= 268:30 +CPU= 20.434
  22:55:37 STEP 1075 H= 268:45 +CPU= 20.629
  22:55:59 STEP 1076 H= 269:00 +CPU= 21.021
  22:56:21 STEP 1077 H= 269:15 +CPU= 20.848
  22:56:42 STEP 1078 H= 269:30 +CPU= 20.670
  22:57:03 STEP 1079 H= 269:45 +CPU= 20.108
  22:57:28 STEP 1080 H= 270:00 +CPU= 25.062
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001080
  22:57:49 STEP 1081 H= 270:15 +CPU= 20.268
  22:58:10 STEP 1082 H= 270:30 +CPU= 20.065
  22:58:30 STEP 1083 H= 270:45 +CPU= 19.604
  22:58:52 STEP 1084 H= 271:00 +CPU= 20.683
  22:59:12 STEP 1085 H= 271:15 +CPU= 19.940
  22:59:33 STEP 1086 H= 271:30 +CPU= 20.121
  22:59:54 STEP 1087 H= 271:45 +CPU= 20.011
  23:00:16 STEP 1088 H= 272:00 +CPU= 21.081
  23:00:37 STEP 1089 H= 272:15 +CPU= 20.707
  23:00:58 STEP 1090 H= 272:30 +CPU= 20.268
  23:01:19 STEP 1091 H= 272:45 +CPU= 20.357
  23:01:45 STEP 1092 H= 273:00 +CPU= 25.342
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001092
  23:02:06 STEP 1093 H= 273:15 +CPU= 20.186
  23:02:27 STEP 1094 H= 273:30 +CPU= 19.887
  23:02:47 STEP 1095 H= 273:45 +CPU= 19.824
  23:03:08 STEP 1096 H= 274:00 +CPU= 20.399
  23:03:29 STEP 1097 H= 274:15 +CPU= 20.170
  23:03:50 STEP 1098 H= 274:30 +CPU= 19.909
  23:04:11 STEP 1099 H= 274:45 +CPU= 20.552
  23:04:33 STEP 1100 H= 275:00 +CPU= 20.968
  23:04:54 STEP 1101 H= 275:15 +CPU= 20.730
  23:05:15 STEP 1102 H= 275:30 +CPU= 20.354
  23:05:36 STEP 1103 H= 275:45 +CPU= 19.993
  23:06:05 STEP 1104 H= 276:00 +CPU= 28.102
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001104
  23:06:26 STEP 1105 H= 276:15 +CPU= 20.499
  23:06:47 STEP 1106 H= 276:30 +CPU= 20.179
  23:07:08 STEP 1107 H= 276:45 +CPU= 19.979
  23:07:29 STEP 1108 H= 277:00 +CPU= 20.384
  23:07:49 STEP 1109 H= 277:15 +CPU= 20.190
  23:08:10 STEP 1110 H= 277:30 +CPU= 19.967
  23:08:32 STEP 1111 H= 277:45 +CPU= 20.807
  23:08:52 STEP 1112 H= 278:00 +CPU= 20.289
  23:09:14 STEP 1113 H= 278:15 +CPU= 20.867
  23:09:35 STEP 1114 H= 278:30 +CPU= 20.147
  23:09:57 STEP 1115 H= 278:45 +CPU= 20.868
  23:10:22 STEP 1116 H= 279:00 +CPU= 25.124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001116
  23:10:43 STEP 1117 H= 279:15 +CPU= 20.245
  23:11:04 STEP 1118 H= 279:30 +CPU= 20.030
  23:11:25 STEP 1119 H= 279:45 +CPU= 19.920
  23:11:46 STEP 1120 H= 280:00 +CPU= 21.080
  23:12:07 STEP 1121 H= 280:15 +CPU= 20.386
  23:12:28 STEP 1122 H= 280:30 +CPU= 20.359
  23:12:49 STEP 1123 H= 280:45 +CPU= 19.839
  23:13:10 STEP 1124 H= 281:00 +CPU= 20.224
  23:13:31 STEP 1125 H= 281:15 +CPU= 20.246
  23:13:52 STEP 1126 H= 281:30 +CPU= 20.201
  23:14:12 STEP 1127 H= 281:45 +CPU= 20.093
  23:14:39 STEP 1128 H= 282:00 +CPU= 25.660
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001128
  23:14:59 STEP 1129 H= 282:15 +CPU= 20.279
  23:15:20 STEP 1130 H= 282:30 +CPU= 20.257
  23:15:42 STEP 1131 H= 282:45 +CPU= 20.516
  23:16:04 STEP 1132 H= 283:00 +CPU= 21.221
  23:16:25 STEP 1133 H= 283:15 +CPU= 20.283
  23:16:46 STEP 1134 H= 283:30 +CPU= 20.379
  23:17:07 STEP 1135 H= 283:45 +CPU= 20.399
  23:17:29 STEP 1136 H= 284:00 +CPU= 21.041
  23:17:50 STEP 1137 H= 284:15 +CPU= 20.301
  23:18:12 STEP 1138 H= 284:30 +CPU= 21.211
  23:18:32 STEP 1139 H= 284:45 +CPU= 19.865
  23:18:58 STEP 1140 H= 285:00 +CPU= 24.771
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001140
  23:19:18 STEP 1141 H= 285:15 +CPU= 19.990
  23:19:39 STEP 1142 H= 285:30 +CPU= 20.057
  23:20:00 STEP 1143 H= 285:45 +CPU= 20.018
  23:20:21 STEP 1144 H= 286:00 +CPU= 20.527
  23:20:42 STEP 1145 H= 286:15 +CPU= 20.295
  23:21:02 STEP 1146 H= 286:30 +CPU= 19.966
  23:21:24 STEP 1147 H= 286:45 +CPU= 20.559
  23:21:45 STEP 1148 H= 287:00 +CPU= 20.825
  23:22:06 STEP 1149 H= 287:15 +CPU= 20.611
  23:22:28 STEP 1150 H= 287:30 +CPU= 20.430
  23:22:48 STEP 1151 H= 287:45 +CPU= 20.198
  23:23:18 STEP 1152 H= 288:00 +CPU= 28.733
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001152
  23:23:39 STEP 1153 H= 288:15 +CPU= 20.400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001152
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a270_2016092300_15_991_12209631_0_r352161327_11.zip
Uploading the intermediate file: upload_file_11.zip
  23:24:01 STEP 1154 H= 288:30 +CPU= 20.457
  23:24:22 STEP 1155 H= 288:45 +CPU= 20.662
  23:24:43 STEP 1156 H= 289:00 +CPU= 20.671
  23:25:04 STEP 1157 H= 289:15 +CPU= 20.021
  23:25:25 STEP 1158 H= 289:30 +CPU= 20.200
  23:25:46 STEP 1159 H= 289:45 +CPU= 20.731
  23:26:08 STEP 1160 H= 290:00 +CPU= 20.845
  23:26:29 STEP 1161 H= 290:15 +CPU= 20.273
  23:26:50 STEP 1162 H= 290:30 +CPU= 20.243
  23:27:11 STEP 1163 H= 290:45 +CPU= 20.399
  23:27:37 STEP 1164 H= 291:00 +CPU= 25.009
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001164
  23:27:57 STEP 1165 H= 291:15 +CPU= 20.056
  23:28:18 STEP 1166 H= 291:30 +CPU= 19.663
  23:28:39 STEP 1167 H= 291:45 +CPU= 20.780
  23:29:01 STEP 1168 H= 292:00 +CPU= 21.021
  23:29:22 STEP 1169 H= 292:15 +CPU= 20.339
  23:29:42 STEP 1170 H= 292:30 +CPU= 19.944
  23:30:04 STEP 1171 H= 292:45 +CPU= 20.484
  23:30:25 STEP 1172 H= 293:00 +CPU= 20.887
  23:30:46 STEP 1173 H= 293:15 +CPU= 20.120
  23:31:06 STEP 1174 H= 293:30 +CPU= 19.994
  23:31:27 STEP 1175 H= 293:45 +CPU= 20.108
  23:31:53 STEP 1176 H= 294:00 +CPU= 25.268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001176
  23:32:15 STEP 1177 H= 294:15 +CPU= 20.425
  23:32:35 STEP 1178 H= 294:30 +CPU= 20.229
  23:32:56 STEP 1179 H= 294:45 +CPU= 20.204
  23:33:18 STEP 1180 H= 295:00 +CPU= 20.492
  23:33:39 STEP 1181 H= 295:15 +CPU= 20.549
  23:34:00 STEP 1182 H= 295:30 +CPU= 20.351
  23:34:21 STEP 1183 H= 295:45 +CPU= 20.518
  23:34:43 STEP 1184 H= 296:00 +CPU= 20.792
  23:35:04 STEP 1185 H= 296:15 +CPU= 20.383
  23:35:25 STEP 1186 H= 296:30 +CPU= 20.399
  23:35:46 STEP 1187 H= 296:45 +CPU= 20.497
  23:36:12 STEP 1188 H= 297:00 +CPU= 25.124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001188
  23:36:33 STEP 1189 H= 297:15 +CPU= 20.040
  23:36:53 STEP 1190 H= 297:30 +CPU= 20.021
  23:37:14 STEP 1191 H= 297:45 +CPU= 20.077
  23:37:36 STEP 1192 H= 298:00 +CPU= 21.262
  23:37:57 STEP 1193 H= 298:15 +CPU= 20.440
  23:38:19 STEP 1194 H= 298:30 +CPU= 21.069
  23:38:40 STEP 1195 H= 298:45 +CPU= 20.439
  23:39:01 STEP 1196 H= 299:00 +CPU= 20.755
  23:39:23 STEP 1197 H= 299:15 +CPU= 21.355
  23:39:45 STEP 1198 H= 299:30 +CPU= 20.683
  23:40:06 STEP 1199 H= 299:45 +CPU= 20.766
  23:40:35 STEP 1200 H= 300:00 +CPU= 27.883
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001200
  23:40:57 STEP 1201 H= 300:15 +CPU= 20.608
  23:41:17 STEP 1202 H= 300:30 +CPU= 20.194
  23:41:38 STEP 1203 H= 300:45 +CPU= 19.914
  23:41:59 STEP 1204 H= 301:00 +CPU= 20.188
  23:42:20 STEP 1205 H= 301:15 +CPU= 20.479
  23:42:41 STEP 1206 H= 301:30 +CPU= 20.604
  23:43:02 STEP 1207 H= 301:45 +CPU= 20.306
  23:43:24 STEP 1208 H= 302:00 +CPU= 20.564
  23:43:45 STEP 1209 H= 302:15 +CPU= 20.190
  23:44:05 STEP 1210 H= 302:30 +CPU= 20.129
  23:44:26 STEP 1211 H= 302:45 +CPU= 20.243
  23:44:52 STEP 1212 H= 303:00 +CPU= 25.115
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001212
  23:45:14 STEP 1213 H= 303:15 +CPU= 21.010
  23:45:35 STEP 1214 H= 303:30 +CPU= 20.314
  23:45:56 STEP 1215 H= 303:45 +CPU= 20.362
  23:46:18 STEP 1216 H= 304:00 +CPU= 21.102
  23:46:39 STEP 1217 H= 304:15 +CPU= 20.986
  23:47:00 STEP 1218 H= 304:30 +CPU= 19.991
  23:47:21 STEP 1219 H= 304:45 +CPU= 20.278
  23:47:42 STEP 1220 H= 305:00 +CPU= 20.440
  23:48:03 STEP 1221 H= 305:15 +CPU= 20.219
  23:48:24 STEP 1222 H= 305:30 +CPU= 20.629
  23:48:45 STEP 1223 H= 305:45 +CPU= 20.100
  23:49:11 STEP 1224 H= 306:00 +CPU= 25.563
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001224
  23:49:32 STEP 1225 H= 306:15 +CPU= 20.095
  23:49:53 STEP 1226 H= 306:30 +CPU= 20.504
  23:50:15 STEP 1227 H= 306:45 +CPU= 20.426
  23:50:37 STEP 1228 H= 307:00 +CPU= 21.676
  23:50:59 STEP 1229 H= 307:15 +CPU= 21.071
  23:51:20 STEP 1230 H= 307:30 +CPU= 20.655
  23:51:42 STEP 1231 H= 307:45 +CPU= 21.005
  23:52:04 STEP 1232 H= 308:00 +CPU= 20.925
  23:52:24 STEP 1233 H= 308:15 +CPU= 20.102
  23:52:46 STEP 1234 H= 308:30 +CPU= 20.499
  23:53:07 STEP 1235 H= 308:45 +CPU= 20.374
  23:53:32 STEP 1236 H= 309:00 +CPU= 24.998
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001236
  23:53:54 STEP 1237 H= 309:15 +CPU= 21.089
  23:54:16 STEP 1238 H= 309:30 +CPU= 20.625
  23:54:36 STEP 1239 H= 309:45 +CPU= 19.949
  23:54:59 STEP 1240 H= 310:00 +CPU= 21.999
  23:55:20 STEP 1241 H= 310:15 +CPU= 20.170
  23:55:41 STEP 1242 H= 310:30 +CPU= 20.165
  23:56:02 STEP 1243 H= 310:45 +CPU= 20.194
  23:56:24 STEP 1244 H= 311:00 +CPU= 21.836
  23:56:46 STEP 1245 H= 311:15 +CPU= 20.716
  23:57:06 STEP 1246 H= 311:30 +CPU= 20.295
  23:57:28 STEP 1247 H= 311:45 +CPU= 20.862
  23:57:57 STEP 1248 H= 312:00 +CPU= 28.173
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001248
  23:58:18 STEP 1249 H= 312:15 +CPU= 20.050
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001248
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a270_2016092300_15_991_12209631_0_r352161327_12.zip
Uploading the intermediate file: upload_file_12.zip
  23:58:40 STEP 1250 H= 312:30 +CPU= 21.216
  23:59:01 STEP 1251 H= 312:45 +CPU= 20.604
  23:59:24 STEP 1252 H= 313:00 +CPU= 21.777
  23:59:46 STEP 1253 H= 313:15 +CPU= 20.923
  00:00:08 STEP 1254 H= 313:30 +CPU= 21.236
  00:00:29 STEP 1255 H= 313:45 +CPU= 20.816
  00:00:50 STEP 1256 H= 314:00 +CPU= 20.394
  00:01:11 STEP 1257 H= 314:15 +CPU= 20.341
  00:01:32 STEP 1258 H= 314:30 +CPU= 20.184
  00:01:52 STEP 1259 H= 314:45 +CPU= 19.884
  00:02:18 STEP 1260 H= 315:00 +CPU= 24.770
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001260
  00:02:40 STEP 1261 H= 315:15 +CPU= 20.631
  00:03:01 STEP 1262 H= 315:30 +CPU= 21.254
  00:03:22 STEP 1263 H= 315:45 +CPU= 19.809
  00:03:43 STEP 1264 H= 316:00 +CPU= 20.302
  00:04:04 STEP 1265 H= 316:15 +CPU= 19.889
  00:04:25 STEP 1266 H= 316:30 +CPU= 20.321
  00:04:45 STEP 1267 H= 316:45 +CPU= 20.043
  00:05:07 STEP 1268 H= 317:00 +CPU= 20.751
  00:05:28 STEP 1269 H= 317:15 +CPU= 20.278
  00:05:48 STEP 1270 H= 317:30 +CPU= 20.203
  00:06:09 STEP 1271 H= 317:45 +CPU= 20.303
  00:06:35 STEP 1272 H= 318:00 +CPU= 24.774
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001272
  00:06:56 STEP 1273 H= 318:15 +CPU= 20.209
  00:07:17 STEP 1274 H= 318:30 +CPU= 20.493
  00:07:38 STEP 1275 H= 318:45 +CPU= 19.689
  00:08:00 STEP 1276 H= 319:00 +CPU= 21.782
  00:08:22 STEP 1277 H= 319:15 +CPU= 21.457
  00:08:44 STEP 1278 H= 319:30 +CPU= 21.286
  00:09:06 STEP 1279 H= 319:45 +CPU= 21.268
  00:09:28 STEP 1280 H= 320:00 +CPU= 21.400
  00:09:50 STEP 1281 H= 320:15 +CPU= 20.716
  00:10:12 STEP 1282 H= 320:30 +CPU= 21.760
  00:10:33 STEP 1283 H= 320:45 +CPU= 20.452
  00:11:00 STEP 1284 H= 321:00 +CPU= 26.302
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001284
  00:11:21 STEP 1285 H= 321:15 +CPU= 20.236
  00:11:42 STEP 1286 H= 321:30 +CPU= 20.363
  00:12:03 STEP 1287 H= 321:45 +CPU= 19.904
  00:12:25 STEP 1288 H= 322:00 +CPU= 21.193
  00:12:47 STEP 1289 H= 322:15 +CPU= 21.050
  00:13:08 STEP 1290 H= 322:30 +CPU= 20.895
  00:13:30 STEP 1291 H= 322:45 +CPU= 20.649
  00:13:51 STEP 1292 H= 323:00 +CPU= 21.206
  00:14:13 STEP 1293 H= 323:15 +CPU= 20.644
  00:14:34 STEP 1294 H= 323:30 +CPU= 20.235
  00:14:54 STEP 1295 H= 323:45 +CPU= 20.048
  00:15:24 STEP 1296 H= 324:00 +CPU= 28.413
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001296
  00:15:46 STEP 1297 H= 324:15 +CPU= 21.144
  00:16:06 STEP 1298 H= 324:30 +CPU= 19.943
  00:16:28 STEP 1299 H= 324:45 +CPU= 20.360
  00:16:49 STEP 1300 H= 325:00 +CPU= 20.737
  00:17:11 STEP 1301 H= 325:15 +CPU= 20.990
  00:17:32 STEP 1302 H= 325:30 +CPU= 20.355
  00:17:53 STEP 1303 H= 325:45 +CPU= 20.595
  00:18:15 STEP 1304 H= 326:00 +CPU= 20.757
  00:18:36 STEP 1305 H= 326:15 +CPU= 21.266
  00:18:58 STEP 1306 H= 326:30 +CPU= 20.973
  00:19:20 STEP 1307 H= 326:45 +CPU= 20.891
  00:19:46 STEP 1308 H= 327:00 +CPU= 25.329
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001308
  00:20:06 STEP 1309 H= 327:15 +CPU= 20.028
  00:20:27 STEP 1310 H= 327:30 +CPU= 19.915
  00:20:48 STEP 1311 H= 327:45 +CPU= 20.721
  00:21:10 STEP 1312 H= 328:00 +CPU= 21.026
  00:21:31 STEP 1313 H= 328:15 +CPU= 20.335
  00:21:52 STEP 1314 H= 328:30 +CPU= 20.300
  00:22:13 STEP 1315 H= 328:45 +CPU= 20.332
  00:22:35 STEP 1316 H= 329:00 +CPU= 20.977
  00:22:56 STEP 1317 H= 329:15 +CPU= 20.422
  00:23:17 STEP 1318 H= 329:30 +CPU= 20.551
  00:23:38 STEP 1319 H= 329:45 +CPU= 20.491
  00:24:04 STEP 1320 H= 330:00 +CPU= 25.506
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001320
  00:24:26 STEP 1321 H= 330:15 +CPU= 20.680
  00:24:47 STEP 1322 H= 330:30 +CPU= 20.400
  00:25:09 STEP 1323 H= 330:45 +CPU= 20.718
  00:25:30 STEP 1324 H= 331:00 +CPU= 20.989
  00:25:52 STEP 1325 H= 331:15 +CPU= 21.194
  00:26:13 STEP 1326 H= 331:30 +CPU= 20.455
  00:26:35 STEP 1327 H= 331:45 +CPU= 20.871
  00:26:56 STEP 1328 H= 332:00 +CPU= 20.484
  00:27:17 STEP 1329 H= 332:15 +CPU= 20.629
  00:27:38 STEP 1330 H= 332:30 +CPU= 20.405
  00:28:00 STEP 1331 H= 332:45 +CPU= 21.071
  00:28:26 STEP 1332 H= 333:00 +CPU= 25.197
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001332
  00:28:47 STEP 1333 H= 333:15 +CPU= 20.981
  00:29:09 STEP 1334 H= 333:30 +CPU= 20.953
  00:29:29 STEP 1335 H= 333:45 +CPU= 19.645
  00:29:51 STEP 1336 H= 334:00 +CPU= 21.180
  00:30:13 STEP 1337 H= 334:15 +CPU= 20.724
  00:30:34 STEP 1338 H= 334:30 +CPU= 20.925
  00:30:55 STEP 1339 H= 334:45 +CPU= 20.636
  00:31:17 STEP 1340 H= 335:00 +CPU= 20.670
  00:31:38 STEP 1341 H= 335:15 +CPU= 20.222
  00:31:59 STEP 1342 H= 335:30 +CPU= 20.266
  00:32:20 STEP 1343 H= 335:45 +CPU= 20.748
  00:32:49 STEP 1344 H= 336:00 +CPU= 28.088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001344
  00:33:11 STEP 1345 H= 336:15 +CPU= 20.984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001344
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a270_2016092300_15_991_12209631_0_r352161327_13.zip
Uploading the intermediate file: upload_file_13.zip
  00:33:31 STEP 1346 H= 336:30 +CPU= 19.788
  00:33:53 STEP 1347 H= 336:45 +CPU= 21.063
  00:34:15 STEP 1348 H= 337:00 +CPU= 21.313
  00:34:37 STEP 1349 H= 337:15 +CPU= 20.707
  00:34:58 STEP 1350 H= 337:30 +CPU= 20.391
  00:35:19 STEP 1351 H= 337:45 +CPU= 21.058
  00:35:41 STEP 1352 H= 338:00 +CPU= 21.093
  00:36:02 STEP 1353 H= 338:15 +CPU= 20.702
  00:36:23 STEP 1354 H= 338:30 +CPU= 20.070
  00:36:44 STEP 1355 H= 338:45 +CPU= 20.213
  00:37:10 STEP 1356 H= 339:00 +CPU= 25.011
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001356
  00:37:31 STEP 1357 H= 339:15 +CPU= 20.409
  00:37:52 STEP 1358 H= 339:30 +CPU= 20.245
  00:38:13 STEP 1359 H= 339:45 +CPU= 20.625
  00:38:35 STEP 1360 H= 340:00 +CPU= 21.258
  00:38:57 STEP 1361 H= 340:15 +CPU= 21.358
  00:39:18 STEP 1362 H= 340:30 +CPU= 20.841
  00:39:40 STEP 1363 H= 340:45 +CPU= 21.301
  00:40:02 STEP 1364 H= 341:00 +CPU= 21.076
  00:40:23 STEP 1365 H= 341:15 +CPU= 20.089
  00:40:44 STEP 1366 H= 341:30 +CPU= 20.350
  00:41:05 STEP 1367 H= 341:45 +CPU= 20.503
  00:41:33 STEP 1368 H= 342:00 +CPU= 26.221
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001368
  00:41:53 STEP 1369 H= 342:15 +CPU= 19.522
  00:42:12 STEP 1370 H= 342:30 +CPU= 18.204
  00:42:30 STEP 1371 H= 342:45 +CPU= 17.743
  00:42:49 STEP 1372 H= 343:00 +CPU= 18.746
  00:43:08 STEP 1373 H= 343:15 +CPU= 17.950
  00:43:27 STEP 1374 H= 343:30 +CPU= 18.138
  00:43:45 STEP 1375 H= 343:45 +CPU= 18.231
  00:44:04 STEP 1376 H= 344:00 +CPU= 18.276
  00:44:23 STEP 1377 H= 344:15 +CPU= 17.953
  00:44:41 STEP 1378 H= 344:30 +CPU= 18.063
  00:45:00 STEP 1379 H= 344:45 +CPU= 18.078
  00:45:24 STEP 1380 H= 345:00 +CPU= 22.881
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001380
  00:45:42 STEP 1381 H= 345:15 +CPU= 17.575
  00:46:00 STEP 1382 H= 345:30 +CPU= 17.772
  00:46:19 STEP 1383 H= 345:45 +CPU= 17.947
  00:46:37 STEP 1384 H= 346:00 +CPU= 18.439
  00:46:56 STEP 1385 H= 346:15 +CPU= 17.641
  00:47:14 STEP 1386 H= 346:30 +CPU= 17.900
  00:47:32 STEP 1387 H= 346:45 +CPU= 17.838
  00:47:51 STEP 1388 H= 347:00 +CPU= 18.363
  00:48:10 STEP 1389 H= 347:15 +CPU= 18.222
  00:48:28 STEP 1390 H= 347:30 +CPU= 17.724
  00:48:47 STEP 1391 H= 347:45 +CPU= 18.128
  00:49:13 STEP 1392 H= 348:00 +CPU= 24.911
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001392
  00:49:32 STEP 1393 H= 348:15 +CPU= 18.126
  00:49:50 STEP 1394 H= 348:30 +CPU= 18.068
  00:50:09 STEP 1395 H= 348:45 +CPU= 18.096
  00:50:28 STEP 1396 H= 349:00 +CPU= 18.607
  00:50:47 STEP 1397 H= 349:15 +CPU= 17.885
  00:51:05 STEP 1398 H= 349:30 +CPU= 18.083
  00:51:24 STEP 1399 H= 349:45 +CPU= 18.246
  00:51:43 STEP 1400 H= 350:00 +CPU= 18.459
  00:52:01 STEP 1401 H= 350:15 +CPU= 17.867
  00:52:20 STEP 1402 H= 350:30 +CPU= 18.331
  00:52:39 STEP 1403 H= 350:45 +CPU= 17.964
  00:53:02 STEP 1404 H= 351:00 +CPU= 22.888
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001404
  00:53:21 STEP 1405 H= 351:15 +CPU= 18.175
  00:53:40 STEP 1406 H= 351:30 +CPU= 18.189
  00:53:58 STEP 1407 H= 351:45 +CPU= 17.822
  00:54:17 STEP 1408 H= 352:00 +CPU= 18.595
  00:54:36 STEP 1409 H= 352:15 +CPU= 18.051
  00:54:54 STEP 1410 H= 352:30 +CPU= 17.840
  00:55:13 STEP 1411 H= 352:45 +CPU= 18.163
  00:55:32 STEP 1412 H= 353:00 +CPU= 18.748
  00:55:49 STEP 1413 H= 353:15 +CPU= 16.332
  00:56:06 STEP 1414 H= 353:30 +CPU= 16.410
  00:56:23 STEP 1415 H= 353:45 +CPU= 16.560
  00:56:44 STEP 1416 H= 354:00 +CPU= 20.315
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001416
  00:56:59 STEP 1417 H= 354:15 +CPU= 15.131
  00:57:15 STEP 1418 H= 354:30 +CPU= 15.276
  00:57:30 STEP 1419 H= 354:45 +CPU= 15.130
  00:57:47 STEP 1420 H= 355:00 +CPU= 15.656
  00:58:02 STEP 1421 H= 355:15 +CPU= 15.068
  00:58:18 STEP 1422 H= 355:30 +CPU= 15.480
  00:58:34 STEP 1423 H= 355:45 +CPU= 15.197
  00:58:50 STEP 1424 H= 356:00 +CPU= 15.527
  00:59:05 STEP 1425 H= 356:15 +CPU= 15.030
  00:59:20 STEP 1426 H= 356:30 +CPU= 14.451
  00:59:34 STEP 1427 H= 356:45 +CPU= 14.406
  00:59:53 STEP 1428 H= 357:00 +CPU= 18.522
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001428
  01:00:08 STEP 1429 H= 357:15 +CPU= 14.229
  01:00:22 STEP 1430 H= 357:30 +CPU= 14.541
  01:00:37 STEP 1431 H= 357:45 +CPU= 14.483
  01:00:52 STEP 1432 H= 358:00 +CPU= 14.917
  01:01:07 STEP 1433 H= 358:15 +CPU= 14.542
  01:01:21 STEP 1434 H= 358:30 +CPU= 14.179
  01:01:36 STEP 1435 H= 358:45 +CPU= 14.362
  01:01:51 STEP 1436 H= 359:00 +CPU= 14.549
  01:02:05 STEP 1437 H= 359:15 +CPU= 14.165
  01:02:20 STEP 1438 H= 359:30 +CPU= 14.224
  01:02:34 STEP 1439 H= 359:45 +CPU= 14.276
  01:02:53 STEP 1440 H= 360:00 +CPU= 18.989
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.3191305947E-23      I       0.9459902852E-05      I
ICloud fractionI      0.0000000000E+00      I       0.5015872297E-01      I
I  Relat. Hum. I      0.1939316859E+00      I       0.3414613253E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.8999191618E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    314996113 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0053( 5.7%)
 01:02:53 STEP 1440 H= 360:00 +CPU= 18.989
 END CNT3
 NSTEP =  1440 CNT0   000000000  13.416  13.416  13.638   0.984               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN        14180/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    253    0       34      1044068    14180 ************    19.1     1.1    18.0     2.0    38.2
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.97      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  521    3965.4    3965.4     20.10      0.00
   9 SPCM     - SPECTRAL COMP.                     480     715.4     715.4      3.34      0.00
  10 SCAN2M - PHYSICS                              481    9738.6    9738.6     45.58      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 40     671.0     671.0      0.26      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 21      59.1      59.1      0.01      0.00
  17 GRIDFPOS IN CNT4                               40      22.7      22.7      0.01      0.00
  27 RESTART FILES - WRITING                        10    2683.6    2683.6      0.26      0.00
  28 RESTART FILES - READING                         1    2842.3    2842.3      0.03      0.00
  29 SU4FPOS IN CNT4                                40       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                40    4342.5    4342.5      1.69      0.00
  31 POSDDH IN STEPO                                 3       3.9       3.9      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    481    4158.2    4158.2     19.46      0.00
  39 SU0YOMB                                         1     272.6     272.6      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    481      22.3      22.3      0.10      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              480     391.8     391.8      1.83      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              480      49.3      49.3      0.23      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 21       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     1044     268.6     268.6      2.73      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      1000     151.8     151.8      1.48      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       1000      55.4      55.4      0.54      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      1044      83.5      83.5      0.85      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      37.1      37.1      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           1044      47.0      47.0      0.48      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           1000      26.9      26.9      0.26      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           1044      70.4      70.4      0.71      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           1000      37.0      37.0      0.36      0.00
 400 GSTATS                                      56453       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 52473       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    10277.4 CPU TIME    9948.2 VECTOR TIME     9948.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      126.1
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/1/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/1/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209631/ICMGGh7zg+001416
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a270_2016092300_15_991_12209631_0_r352161327_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
01:06:21 (647310): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:36 1532237 22309829 oifs_43r3_bl_a270_2016092300_15_991_12209631_0 1,295,100 37,194 0.0287


©2024 climateprediction.net