climateprediction.net home page
Task 22255852

Task 22255852

Name oifs_43r3_ps_0837_1985050100_123_954_12171481_0
Workunit 12171481
Created 21 Dec 2022, 7:03:07 UTC
Sent 8 Jan 2023, 0:44:13 UTC
Report deadline 7 Feb 2023, 0:44:13 UTC
Received 8 Jan 2023, 23:23:01 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1534455
Run time 21 hours 53 min 33 sec
CPU time 21 hours 53 min 33 sec
Validate state Invalid
Credit 14,031.36
Device peak FLOPS 4.52 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.38 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,800.69 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
CPU= 24.367
  20:42:31 STEP 2516 H=2516:00 +CPU= 25.419
  20:43:09 STEP 2517 H=2517:00 +CPU= 38.055
  20:43:35 STEP 2518 H=2518:00 +CPU= 25.411
  20:44:00 STEP 2519 H=2519:00 +CPU= 25.426
  20:44:48 STEP 2520 H=2520:00 +CPU= 47.577
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_104.zip
Uploading the intermediate file: upload_file_104.zip
  20:45:15 STEP 2521 H=2521:00 +CPU= 26.644
  20:45:41 STEP 2522 H=2522:00 +CPU= 26.562
  20:46:20 STEP 2523 H=2523:00 +CPU= 39.122
  20:46:45 STEP 2524 H=2524:00 +CPU= 24.627
  20:47:09 STEP 2525 H=2525:00 +CPU= 23.893
  20:47:47 STEP 2526 H=2526:00 +CPU= 37.953
  20:48:11 STEP 2527 H=2527:00 +CPU= 24.010
  20:48:36 STEP 2528 H=2528:00 +CPU= 24.632
  20:49:13 STEP 2529 H=2529:00 +CPU= 37.034
  20:49:37 STEP 2530 H=2530:00 +CPU= 24.635
  20:50:01 STEP 2531 H=2531:00 +CPU= 23.939
  20:50:42 STEP 2532 H=2532:00 +CPU= 41.222
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002532
  20:51:06 STEP 2533 H=2533:00 +CPU= 23.920
  20:51:31 STEP 2534 H=2534:00 +CPU= 24.898
  20:52:11 STEP 2535 H=2535:00 +CPU= 39.547
  20:52:38 STEP 2536 H=2536:00 +CPU= 27.219
  20:53:05 STEP 2537 H=2537:00 +CPU= 26.713
  20:53:45 STEP 2538 H=2538:00 +CPU= 39.694
  20:54:09 STEP 2539 H=2539:00 +CPU= 24.004
  20:54:34 STEP 2540 H=2540:00 +CPU= 24.674
  20:55:11 STEP 2541 H=2541:00 +CPU= 37.161
  20:55:36 STEP 2542 H=2542:00 +CPU= 24.633
  20:56:00 STEP 2543 H=2543:00 +CPU= 24.764
  20:56:44 STEP 2544 H=2544:00 +CPU= 43.621
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_105.zip
Uploading the intermediate file: upload_file_105.zip
  20:57:08 STEP 2545 H=2545:00 +CPU= 23.902
Uploading trickle at timestep: 9158400
  20:57:33 STEP 2546 H=2546:00 +CPU= 24.536
  20:58:10 STEP 2547 H=2547:00 +CPU= 36.987
  20:58:35 STEP 2548 H=2548:00 +CPU= 25.156
  20:59:01 STEP 2549 H=2549:00 +CPU= 25.750
  20:59:45 STEP 2550 H=2550:00 +CPU= 43.656
  21:00:12 STEP 2551 H=2551:00 +CPU= 27.813
  21:00:41 STEP 2552 H=2552:00 +CPU= 28.175
  21:01:19 STEP 2553 H=2553:00 +CPU= 38.076
  21:01:44 STEP 2554 H=2554:00 +CPU= 25.311
  21:02:09 STEP 2555 H=2555:00 +CPU= 24.513
  21:02:51 STEP 2556 H=2556:00 +CPU= 42.241
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002556
  21:03:15 STEP 2557 H=2557:00 +CPU= 24.466
  21:03:41 STEP 2558 H=2558:00 +CPU= 25.431
  21:04:19 STEP 2559 H=2559:00 +CPU= 37.856
  21:04:44 STEP 2560 H=2560:00 +CPU= 25.227
  21:05:08 STEP 2561 H=2561:00 +CPU= 24.462
  21:05:50 STEP 2562 H=2562:00 +CPU= 41.663
  21:06:18 STEP 2563 H=2563:00 +CPU= 27.905
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0837
(argv4) batchid: 954
(argv5) wuid: 12171481
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_cb62e892032113c1c9d8965e4dd12912 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0837_1985050100_123_954_12171481.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0837_1985050100_123_954_12171481.zip
ic_ancil_file: ic_ancil_12171481
ifsdata_file: ifsdata_12171481
climate_data_file: clim_data_12171481
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d847a3c58a0955de480aba5542ccae9f to: /var/lib/boinc-client/slots/9/ic_ancil_12171481.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12171481.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3113a523a1895e049a58ab3408559eb4 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12171481.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12171481.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12171481.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12171481.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12171481.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12171481.xml
-- Model is restarting --
Adjusting last_iter, 2564, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12171481.xml
last_cpu_time: 68323
upload_file_number: 106
last_iter: 2543
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499
The child process has been launched with process id: 10383
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:10383:10383] [20230108:211613:1673208973:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:16:59 STEP 2544 H=2544:00 +CPU= 43.131
  21:17:24 STEP 2545 H=2545:00 +CPU= 24.730
  21:17:52 STEP 2546 H=2546:00 +CPU= 27.708
  21:18:32 STEP 2547 H=2547:00 +CPU= 40.620
  21:18:58 STEP 2548 H=2548:00 +CPU= 25.272
  21:19:22 STEP 2549 H=2549:00 +CPU= 24.664
  21:20:02 STEP 2550 H=2550:00 +CPU= 39.232
  21:20:26 STEP 2551 H=2551:00 +CPU= 24.721
  21:20:51 STEP 2552 H=2552:00 +CPU= 24.946
  21:21:30 STEP 2553 H=2553:00 +CPU= 38.301
  21:21:55 STEP 2554 H=2554:00 +CPU= 25.222
  21:22:15 STEP 2555 H=2555:00 +CPU= 19.504
  21:22:52 STEP 2556 H=2556:00 +CPU= 37.788
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002556
  21:23:14 STEP 2557 H=2557:00 +CPU= 21.767
  21:23:42 STEP 2558 H=2558:00 +CPU= 27.800
  21:24:24 STEP 2559 H=2559:00 +CPU= 42.154
  21:24:44 STEP 2560 H=2560:00 +CPU= 19.828
  21:25:08 STEP 2561 H=2561:00 +CPU= 24.066
  21:25:44 STEP 2562 H=2562:00 +CPU= 35.968
  21:26:00 STEP 2563 H=2563:00 +CPU= 15.832
  21:26:17 STEP 2564 H=2564:00 +CPU= 17.528
  21:26:45 STEP 2565 H=2565:00 +CPU= 27.933
  21:27:08 STEP 2566 H=2566:00 +CPU= 22.956
  21:27:33 STEP 2567 H=2567:00 +CPU= 24.826
  21:28:05 STEP 2568 H=2568:00 +CPU= 32.270
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_106.zip
Uploading the intermediate file: upload_file_106.zip
  21:28:23 STEP 2569 H=2569:00 +CPU= 17.460
  21:28:40 STEP 2570 H=2570:00 +CPU= 17.391
  21:29:10 STEP 2571 H=2571:00 +CPU= 29.586
  21:29:34 STEP 2572 H=2572:00 +CPU= 23.992
  21:29:57 STEP 2573 H=2573:00 +CPU= 23.277
  21:30:36 STEP 2574 H=2574:00 +CPU= 39.467
  21:31:01 STEP 2575 H=2575:00 +CPU= 24.720
  21:31:25 STEP 2576 H=2576:00 +CPU= 24.069
  21:32:02 STEP 2577 H=2577:00 +CPU= 36.481
  21:32:26 STEP 2578 H=2578:00 +CPU= 24.339
  21:32:50 STEP 2579 H=2579:00 +CPU= 23.683
  21:33:30 STEP 2580 H=2580:00 +CPU= 40.710
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002580
  21:33:54 STEP 2581 H=2581:00 +CPU= 23.558
  21:34:18 STEP 2582 H=2582:00 +CPU= 24.261
  21:34:55 STEP 2583 H=2583:00 +CPU= 36.541
  21:35:19 STEP 2584 H=2584:00 +CPU= 24.232
  21:35:42 STEP 2585 H=2585:00 +CPU= 23.444
  21:36:23 STEP 2586 H=2586:00 +CPU= 40.213
  21:36:48 STEP 2587 H=2587:00 +CPU= 24.909
  21:37:11 STEP 2588 H=2588:00 +CPU= 23.758
  21:37:47 STEP 2589 H=2589:00 +CPU= 35.684
  21:38:10 STEP 2590 H=2590:00 +CPU= 23.268
  21:38:34 STEP 2591 H=2591:00 +CPU= 23.336
  21:39:15 STEP 2592 H=2592:00 +CPU= 41.556
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_107.zip
Uploading the intermediate file: upload_file_107.zip
  21:39:38 STEP 2593 H=2593:00 +CPU= 22.313
  21:40:00 STEP 2594 H=2594:00 +CPU= 22.834
  21:40:35 STEP 2595 H=2595:00 +CPU= 34.516
  21:40:58 STEP 2596 H=2596:00 +CPU= 22.957
  21:41:20 STEP 2597 H=2597:00 +CPU= 22.010
  21:41:56 STEP 2598 H=2598:00 +CPU= 35.558
  21:42:17 STEP 2599 H=2599:00 +CPU= 21.958
  21:42:40 STEP 2600 H=2600:00 +CPU= 22.075
  21:43:14 STEP 2601 H=2601:00 +CPU= 34.601
  21:43:37 STEP 2602 H=2602:00 +CPU= 22.718
  21:43:59 STEP 2603 H=2603:00 +CPU= 21.954
  21:44:37 STEP 2604 H=2604:00 +CPU= 38.424
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002604
  21:44:59 STEP 2605 H=2605:00 +CPU= 21.992
  21:45:22 STEP 2606 H=2606:00 +CPU= 22.556
  21:45:57 STEP 2607 H=2607:00 +CPU= 34.578
  21:46:19 STEP 2608 H=2608:00 +CPU= 22.546
  21:46:41 STEP 2609 H=2609:00 +CPU= 21.971
  21:47:11 STEP 2610 H=2610:00 +CPU= 29.864
  21:47:28 STEP 2611 H=2611:00 +CPU= 17.003
  21:47:43 STEP 2612 H=2612:00 +CPU= 15.347
  21:48:16 STEP 2613 H=2613:00 +CPU= 32.868
  21:48:39 STEP 2614 H=2614:00 +CPU= 23.255
  21:49:03 STEP 2615 H=2615:00 +CPU= 23.472
  21:49:44 STEP 2616 H=2616:00 +CPU= 41.286
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_108.zip
Uploading the intermediate file: upload_file_108.zip
  21:50:07 STEP 2617 H=2617:00 +CPU= 22.636
  21:50:31 STEP 2618 H=2618:00 +CPU= 23.565
  21:50:58 STEP 2619 H=2619:00 +CPU= 26.930
  21:51:13 STEP 2620 H=2620:00 +CPU= 15.719
  21:51:34 STEP 2621 H=2621:00 +CPU= 20.212
  21:52:09 STEP 2622 H=2622:00 +CPU= 35.553
  21:52:28 STEP 2623 H=2623:00 +CPU= 18.757
  21:52:44 STEP 2624 H=2624:00 +CPU= 15.633
  21:53:11 STEP 2625 H=2625:00 +CPU= 27.832
  21:53:27 STEP 2626 H=2626:00 +CPU= 15.810
  21:53:43 STEP 2627 H=2627:00 +CPU= 15.303
  21:54:11 STEP 2628 H=2628:00 +CPU= 28.495
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002628
  21:54:29 STEP 2629 H=2629:00 +CPU= 17.501
  21:54:52 STEP 2630 H=2630:00 +CPU= 23.053
  21:55:28 STEP 2631 H=2631:00 +CPU= 36.033
  21:55:51 STEP 2632 H=2632:00 +CPU= 23.807
  21:56:14 STEP 2633 H=2633:00 +CPU= 22.874
  21:56:47 STEP 2634 H=2634:00 +CPU= 32.809
  21:57:09 STEP 2635 H=2635:00 +CPU= 21.535
  21:57:31 STEP 2636 H=2636:00 +CPU= 22.676
  21:58:06 STEP 2637 H=2637:00 +CPU= 34.608
  21:58:29 STEP 2638 H=2638:00 +CPU= 22.817
  21:58:52 STEP 2639 H=2639:00 +CPU= 22.666
  21:59:32 STEP 2640 H=2640:00 +CPU= 40.610
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_109.zip
Uploading the intermediate file: upload_file_109.zip
  21:59:54 STEP 2641 H=2641:00 +CPU= 21.978
  22:00:17 STEP 2642 H=2642:00 +CPU= 22.686
  22:00:52 STEP 2643 H=2643:00 +CPU= 34.662
  22:01:15 STEP 2644 H=2644:00 +CPU= 22.824
  22:01:37 STEP 2645 H=2645:00 +CPU= 22.015
  22:02:12 STEP 2646 H=2646:00 +CPU= 35.500
  22:02:34 STEP 2647 H=2647:00 +CPU= 22.085
  22:02:57 STEP 2648 H=2648:00 +CPU= 22.726
  22:03:32 STEP 2649 H=2649:00 +CPU= 34.568
  22:03:54 STEP 2650 H=2650:00 +CPU= 22.871
  22:04:16 STEP 2651 H=2651:00 +CPU= 22.005
  22:04:55 STEP 2652 H=2652:00 +CPU= 38.554
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002652
  22:05:17 STEP 2653 H=2653:00 +CPU= 21.895
  22:05:40 STEP 2654 H=2654:00 +CPU= 22.732
  22:06:14 STEP 2655 H=2655:00 +CPU= 34.420
  22:06:37 STEP 2656 H=2656:00 +CPU= 22.602
  22:06:59 STEP 2657 H=2657:00 +CPU= 21.892
  22:07:34 STEP 2658 H=2658:00 +CPU= 35.189
  22:07:56 STEP 2659 H=2659:00 +CPU= 21.861
  22:08:19 STEP 2660 H=2660:00 +CPU= 22.613
  22:08:53 STEP 2661 H=2661:00 +CPU= 34.440
  22:09:16 STEP 2662 H=2662:00 +CPU= 22.576
  22:09:39 STEP 2663 H=2663:00 +CPU= 22.555
  22:10:19 STEP 2664 H=2664:00 +CPU= 40.517
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_110.zip
Uploading the intermediate file: upload_file_110.zip
  22:10:41 STEP 2665 H=2665:00 +CPU= 21.935
  22:11:04 STEP 2666 H=2666:00 +CPU= 22.830
  22:11:39 STEP 2667 H=2667:00 +CPU= 34.485
  22:12:01 STEP 2668 H=2668:00 +CPU= 22.686
  22:12:24 STEP 2669 H=2669:00 +CPU= 22.057
  22:12:59 STEP 2670 H=2670:00 +CPU= 35.406
  22:13:21 STEP 2671 H=2671:00 +CPU= 21.976
  22:13:44 STEP 2672 H=2672:00 +CPU= 22.802
  22:14:19 STEP 2673 H=2673:00 +CPU= 34.630
  22:14:41 STEP 2674 H=2674:00 +CPU= 22.791
  22:15:04 STEP 2675 H=2675:00 +CPU= 22.061
  22:15:42 STEP 2676 H=2676:00 +CPU= 38.595
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002676
  22:16:04 STEP 2677 H=2677:00 +CPU= 22.044
  22:16:27 STEP 2678 H=2678:00 +CPU= 22.791
  22:17:02 STEP 2679 H=2679:00 +CPU= 34.494
  22:17:24 STEP 2680 H=2680:00 +CPU= 22.704
  22:17:46 STEP 2681 H=2681:00 +CPU= 22.062
  22:18:22 STEP 2682 H=2682:00 +CPU= 35.329
  22:18:44 STEP 2683 H=2683:00 +CPU= 21.993
  22:19:07 STEP 2684 H=2684:00 +CPU= 22.792
  22:19:41 STEP 2685 H=2685:00 +CPU= 34.514
  22:20:04 STEP 2686 H=2686:00 +CPU= 22.699
  22:20:27 STEP 2687 H=2687:00 +CPU= 22.782
  22:21:07 STEP 2688 H=2688:00 +CPU= 40.635
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_111.zip
Uploading the intermediate file: upload_file_111.zip
  22:21:29 STEP 2689 H=2689:00 +CPU= 21.987
  22:21:52 STEP 2690 H=2690:00 +CPU= 22.803
  22:22:27 STEP 2691 H=2691:00 +CPU= 34.583
  22:22:50 STEP 2692 H=2692:00 +CPU= 22.804
  22:23:12 STEP 2693 H=2693:00 +CPU= 22.193
  22:23:47 STEP 2694 H=2694:00 +CPU= 35.461
  22:24:10 STEP 2695 H=2695:00 +CPU= 22.092
  22:24:32 STEP 2696 H=2696:00 +CPU= 22.646
  22:25:06 STEP 2697 H=2697:00 +CPU= 34.126
  22:25:29 STEP 2698 H=2698:00 +CPU= 22.464
  22:25:51 STEP 2699 H=2699:00 +CPU= 21.750
  22:26:29 STEP 2700 H=2700:00 +CPU= 38.048
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002700
  22:26:50 STEP 2701 H=2701:00 +CPU= 21.765
  22:27:13 STEP 2702 H=2702:00 +CPU= 22.486
  22:27:47 STEP 2703 H=2703:00 +CPU= 34.106
  22:28:10 STEP 2704 H=2704:00 +CPU= 22.455
  22:28:31 STEP 2705 H=2705:00 +CPU= 21.740
  22:29:06 STEP 2706 H=2706:00 +CPU= 34.954
  22:29:28 STEP 2707 H=2707:00 +CPU= 21.709
  22:29:50 STEP 2708 H=2708:00 +CPU= 22.429
  22:30:24 STEP 2709 H=2709:00 +CPU= 34.043
  22:30:47 STEP 2710 H=2710:00 +CPU= 22.372
  22:31:09 STEP 2711 H=2711:00 +CPU= 22.342
  22:31:49 STEP 2712 H=2712:00 +CPU= 40.103
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_112.zip
Uploading the intermediate file: upload_file_112.zip
  22:32:11 STEP 2713 H=2713:00 +CPU= 21.635
  22:32:33 STEP 2714 H=2714:00 +CPU= 22.276
  22:33:07 STEP 2715 H=2715:00 +CPU= 33.889
  22:33:30 STEP 2716 H=2716:00 +CPU= 22.318
  22:33:51 STEP 2717 H=2717:00 +CPU= 21.603
  22:34:26 STEP 2718 H=2718:00 +CPU= 34.780
  22:34:48 STEP 2719 H=2719:00 +CPU= 21.627
  22:35:10 STEP 2720 H=2720:00 +CPU= 22.352
  22:35:44 STEP 2721 H=2721:00 +CPU= 33.982
  22:36:06 STEP 2722 H=2722:00 +CPU= 22.371
  22:36:28 STEP 2723 H=2723:00 +CPU= 21.653
  22:37:06 STEP 2724 H=2724:00 +CPU= 37.934
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002724
  22:37:28 STEP 2725 H=2725:00 +CPU= 21.655
  22:37:50 STEP 2726 H=2726:00 +CPU= 22.374
  22:38:24 STEP 2727 H=2727:00 +CPU= 34.038
  22:38:47 STEP 2728 H=2728:00 +CPU= 22.379
  22:39:08 STEP 2729 H=2729:00 +CPU= 21.658
  22:39:43 STEP 2730 H=2730:00 +CPU= 34.872
  22:40:05 STEP 2731 H=2731:00 +CPU= 21.649
  22:40:27 STEP 2732 H=2732:00 +CPU= 22.360
  22:41:01 STEP 2733 H=2733:00 +CPU= 34.023
  22:41:24 STEP 2734 H=2734:00 +CPU= 22.368
  22:41:46 STEP 2735 H=2735:00 +CPU= 22.328
  22:42:26 STEP 2736 H=2736:00 +CPU= 40.079
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_113.zip
Uploading the intermediate file: upload_file_113.zip
  22:42:47 STEP 2737 H=2737:00 +CPU= 20.612
  22:43:01 STEP 2738 H=2738:00 +CPU= 14.769
  22:43:24 STEP 2739 H=2739:00 +CPU= 22.902
  22:43:40 STEP 2740 H=2740:00 +CPU= 15.669
  22:43:56 STEP 2741 H=2741:00 +CPU= 15.210
  22:44:21 STEP 2742 H=2742:00 +CPU= 24.053
  22:44:35 STEP 2743 H=2743:00 +CPU= 14.705
  22:44:51 STEP 2744 H=2744:00 +CPU= 15.928
  22:45:18 STEP 2745 H=2745:00 +CPU= 26.605
  22:45:33 STEP 2746 H=2746:00 +CPU= 15.413
  22:45:48 STEP 2747 H=2747:00 +CPU= 14.685
  22:46:14 STEP 2748 H=2748:00 +CPU= 26.092
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002748
  22:46:29 STEP 2749 H=2749:00 +CPU= 15.167
  22:46:45 STEP 2750 H=2750:00 +CPU= 15.943
  22:47:09 STEP 2751 H=2751:00 +CPU= 23.857
  22:47:27 STEP 2752 H=2752:00 +CPU= 17.288
  22:47:47 STEP 2753 H=2753:00 +CPU= 20.016
  22:48:21 STEP 2754 H=2754:00 +CPU= 34.636
  22:48:43 STEP 2755 H=2755:00 +CPU= 21.519
  22:48:59 STEP 2756 H=2756:00 +CPU= 16.564
  22:49:32 STEP 2757 H=2757:00 +CPU= 33.033
  22:49:55 STEP 2758 H=2758:00 +CPU= 22.338
  22:50:17 STEP 2759 H=2759:00 +CPU= 22.283
  22:50:59 STEP 2760 H=2760:00 +CPU= 40.035
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_114.zip
Uploading the intermediate file: upload_file_114.zip
  22:51:20 STEP 2761 H=2761:00 +CPU= 21.601
  22:51:43 STEP 2762 H=2762:00 +CPU= 22.417
  22:52:17 STEP 2763 H=2763:00 +CPU= 34.119
  22:52:39 STEP 2764 H=2764:00 +CPU= 22.434
  22:53:01 STEP 2765 H=2765:00 +CPU= 21.722
  22:53:36 STEP 2766 H=2766:00 +CPU= 34.985
  22:53:58 STEP 2767 H=2767:00 +CPU= 21.729
  22:54:20 STEP 2768 H=2768:00 +CPU= 22.458
  22:54:54 STEP 2769 H=2769:00 +CPU= 34.136
  22:55:17 STEP 2770 H=2770:00 +CPU= 22.452
  22:55:39 STEP 2771 H=2771:00 +CPU= 21.744
  22:56:17 STEP 2772 H=2772:00 +CPU= 38.051
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002772
  22:56:38 STEP 2773 H=2773:00 +CPU= 21.742
  22:57:01 STEP 2774 H=2774:00 +CPU= 22.491
  22:57:35 STEP 2775 H=2775:00 +CPU= 34.137
  22:57:58 STEP 2776 H=2776:00 +CPU= 22.442
  22:58:19 STEP 2777 H=2777:00 +CPU= 21.722
  22:58:54 STEP 2778 H=2778:00 +CPU= 34.990
  22:59:16 STEP 2779 H=2779:00 +CPU= 21.714
  22:59:39 STEP 2780 H=2780:00 +CPU= 22.461
  23:00:13 STEP 2781 H=2781:00 +CPU= 34.142
  23:00:35 STEP 2782 H=2782:00 +CPU= 22.417
  23:00:58 STEP 2783 H=2783:00 +CPU= 22.401
  23:01:38 STEP 2784 H=2784:00 +CPU= 40.285
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_115.zip
Uploading the intermediate file: upload_file_115.zip
  23:02:00 STEP 2785 H=2785:00 +CPU= 21.700
Uploading trickle at timestep: 10022400
  23:02:23 STEP 2786 H=2786:00 +CPU= 22.452
  23:02:57 STEP 2787 H=2787:00 +CPU= 34.202
  23:03:19 STEP 2788 H=2788:00 +CPU= 22.483
  23:03:41 STEP 2789 H=2789:00 +CPU= 21.792
  23:04:16 STEP 2790 H=2790:00 +CPU= 34.954
  23:04:38 STEP 2791 H=2791:00 +CPU= 21.724
  23:05:00 STEP 2792 H=2792:00 +CPU= 22.416
  23:05:34 STEP 2793 H=2793:00 +CPU= 34.058
  23:05:57 STEP 2794 H=2794:00 +CPU= 22.446
  23:06:17 STEP 2795 H=2795:00 +CPU= 20.321
  23:06:55 STEP 2796 H=2796:00 +CPU= 37.916
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002796
  23:07:17 STEP 2797 H=2797:00 +CPU= 21.653
  23:07:39 STEP 2798 H=2798:00 +CPU= 22.388
  23:08:13 STEP 2799 H=2799:00 +CPU= 33.985
  23:08:36 STEP 2800 H=2800:00 +CPU= 22.395
  23:08:57 STEP 2801 H=2801:00 +CPU= 21.671
  23:09:32 STEP 2802 H=2802:00 +CPU= 34.843
  23:09:54 STEP 2803 H=2803:00 +CPU= 21.643
  23:10:16 STEP 2804 H=2804:00 +CPU= 22.359
  23:10:50 STEP 2805 H=2805:00 +CPU= 34.014
  23:11:13 STEP 2806 H=2806:00 +CPU= 22.276
  23:11:35 STEP 2807 H=2807:00 +CPU= 22.262
  23:12:15 STEP 2808 H=2808:00 +CPU= 39.929
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_116.zip
Uploading the intermediate file: upload_file_116.zip
  23:12:37 STEP 2809 H=2809:00 +CPU= 21.542
  23:12:59 STEP 2810 H=2810:00 +CPU= 22.257
  23:13:33 STEP 2811 H=2811:00 +CPU= 33.912
  23:13:55 STEP 2812 H=2812:00 +CPU= 22.289
  23:14:17 STEP 2813 H=2813:00 +CPU= 21.562
  23:14:52 STEP 2814 H=2814:00 +CPU= 34.781
  23:15:13 STEP 2815 H=2815:00 +CPU= 21.595
  23:15:36 STEP 2816 H=2816:00 +CPU= 22.339
  23:16:10 STEP 2817 H=2817:00 +CPU= 33.963
  23:16:32 STEP 2818 H=2818:00 +CPU= 22.319
  23:16:54 STEP 2819 H=2819:00 +CPU= 21.611
  23:17:32 STEP 2820 H=2820:00 +CPU= 37.948
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002820
  23:17:53 STEP 2821 H=2821:00 +CPU= 21.731
  23:18:16 STEP 2822 H=2822:00 +CPU= 22.472
  23:18:50 STEP 2823 H=2823:00 +CPU= 34.100
  23:19:12 STEP 2824 H=2824:00 +CPU= 22.434
  23:19:34 STEP 2825 H=2825:00 +CPU= 21.704
  23:20:09 STEP 2826 H=2826:00 +CPU= 34.920
  23:20:31 STEP 2827 H=2827:00 +CPU= 21.691
  23:20:53 STEP 2828 H=2828:00 +CPU= 22.424
  23:21:27 STEP 2829 H=2829:00 +CPU= 34.124
  23:21:50 STEP 2830 H=2830:00 +CPU= 22.406
  23:22:12 STEP 2831 H=2831:00 +CPU= 22.375
  23:22:52 STEP 2832 H=2832:00 +CPU= 40.052
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_117.zip
Uploading the intermediate file: upload_file_117.zip
  23:23:14 STEP 2833 H=2833:00 +CPU= 21.666
  23:23:37 STEP 2834 H=2834:00 +CPU= 22.450
  23:24:11 STEP 2835 H=2835:00 +CPU= 34.052
  23:24:33 STEP 2836 H=2836:00 +CPU= 22.387
  23:24:55 STEP 2837 H=2837:00 +CPU= 21.671
  23:25:30 STEP 2838 H=2838:00 +CPU= 34.876
  23:25:51 STEP 2839 H=2839:00 +CPU= 21.699
  23:26:14 STEP 2840 H=2840:00 +CPU= 22.435
  23:26:48 STEP 2841 H=2841:00 +CPU= 34.098
  23:27:10 STEP 2842 H=2842:00 +CPU= 22.464
  23:27:32 STEP 2843 H=2843:00 +CPU= 21.706
  23:28:10 STEP 2844 H=2844:00 +CPU= 37.989
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002844
  23:28:32 STEP 2845 H=2845:00 +CPU= 21.710
  23:28:54 STEP 2846 H=2846:00 +CPU= 22.459
  23:29:28 STEP 2847 H=2847:00 +CPU= 34.099
  23:29:51 STEP 2848 H=2848:00 +CPU= 22.418
  23:30:13 STEP 2849 H=2849:00 +CPU= 21.660
  23:30:47 STEP 2850 H=2850:00 +CPU= 34.894
  23:31:09 STEP 2851 H=2851:00 +CPU= 21.683
  23:31:32 STEP 2852 H=2852:00 +CPU= 22.405
  23:32:06 STEP 2853 H=2853:00 +CPU= 34.073
  23:32:28 STEP 2854 H=2854:00 +CPU= 22.380
  23:32:50 STEP 2855 H=2855:00 +CPU= 22.361
  23:33:31 STEP 2856 H=2856:00 +CPU= 40.175
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_118.zip
Uploading the intermediate file: upload_file_118.zip
  23:33:52 STEP 2857 H=2857:00 +CPU= 21.752
  23:34:15 STEP 2858 H=2858:00 +CPU= 22.482
  23:34:49 STEP 2859 H=2859:00 +CPU= 34.227
  23:35:12 STEP 2860 H=2860:00 +CPU= 22.483
  23:35:33 STEP 2861 H=2861:00 +CPU= 21.760
  23:36:09 STEP 2862 H=2862:00 +CPU= 35.071
  23:36:30 STEP 2863 H=2863:00 +CPU= 21.794
  23:36:53 STEP 2864 H=2864:00 +CPU= 22.534
  23:37:27 STEP 2865 H=2865:00 +CPU= 34.268
  23:37:50 STEP 2866 H=2866:00 +CPU= 22.522
  23:38:12 STEP 2867 H=2867:00 +CPU= 21.797
  23:38:50 STEP 2868 H=2868:00 +CPU= 38.202
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002868
  23:39:12 STEP 2869 H=2869:00 +CPU= 21.805
  23:39:34 STEP 2870 H=2870:00 +CPU= 22.522
  23:40:08 STEP 2871 H=2871:00 +CPU= 34.274
  23:40:31 STEP 2872 H=2872:00 +CPU= 22.517
  23:40:53 STEP 2873 H=2873:00 +CPU= 21.779
  23:41:28 STEP 2874 H=2874:00 +CPU= 34.958
  23:41:49 STEP 2875 H=2875:00 +CPU= 21.674
  23:42:12 STEP 2876 H=2876:00 +CPU= 22.404
  23:42:46 STEP 2877 H=2877:00 +CPU= 34.111
  23:43:08 STEP 2878 H=2878:00 +CPU= 22.396
  23:43:31 STEP 2879 H=2879:00 +CPU= 22.371
  23:44:11 STEP 2880 H=2880:00 +CPU= 40.209
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_119.zip
Uploading the intermediate file: upload_file_119.zip
  23:44:33 STEP 2881 H=2881:00 +CPU= 21.645
  23:44:55 STEP 2882 H=2882:00 +CPU= 22.413
  23:45:30 STEP 2883 H=2883:00 +CPU= 34.121
  23:45:52 STEP 2884 H=2884:00 +CPU= 22.418
  23:46:14 STEP 2885 H=2885:00 +CPU= 21.710
  23:46:49 STEP 2886 H=2886:00 +CPU= 35.011
  23:47:10 STEP 2887 H=2887:00 +CPU= 20.809
  23:47:32 STEP 2888 H=2888:00 +CPU= 22.440
  23:48:07 STEP 2889 H=2889:00 +CPU= 34.314
  23:48:26 STEP 2890 H=2890:00 +CPU= 19.825
  23:48:48 STEP 2891 H=2891:00 +CPU= 21.773
  23:49:26 STEP 2892 H=2892:00 +CPU= 38.113
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002892
  23:49:48 STEP 2893 H=2893:00 +CPU= 21.671
  23:50:06 STEP 2894 H=2894:00 +CPU= 18.278
  23:50:31 STEP 2895 H=2895:00 +CPU= 24.683
  23:50:46 STEP 2896 H=2896:00 +CPU= 15.161
  23:51:01 STEP 2897 H=2897:00 +CPU= 15.222
  23:51:27 STEP 2898 H=2898:00 +CPU= 26.053
  23:51:44 STEP 2899 H=2899:00 +CPU= 16.214
  23:51:59 STEP 2900 H=2900:00 +CPU= 15.338
  23:52:23 STEP 2901 H=2901:00 +CPU= 23.829
  23:52:39 STEP 2902 H=2902:00 +CPU= 15.847
  23:52:55 STEP 2903 H=2903:00 +CPU= 16.418
  23:53:24 STEP 2904 H=2904:00 +CPU= 27.698
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_120.zip
Uploading the intermediate file: upload_file_120.zip
  23:53:40 STEP 2905 H=2905:00 +CPU= 16.501
  23:53:56 STEP 2906 H=2906:00 +CPU= 15.717
  23:54:22 STEP 2907 H=2907:00 +CPU= 25.969
  23:54:37 STEP 2908 H=2908:00 +CPU= 15.127
  23:54:52 STEP 2909 H=2909:00 +CPU= 14.934
  23:55:17 STEP 2910 H=2910:00 +CPU= 25.549
  23:55:33 STEP 2911 H=2911:00 +CPU= 15.161
  23:55:48 STEP 2912 H=2912:00 +CPU= 15.249
  23:56:12 STEP 2913 H=2913:00 +CPU= 23.730
  23:56:27 STEP 2914 H=2914:00 +CPU= 15.140
  23:56:42 STEP 2915 H=2915:00 +CPU= 14.958
  23:57:08 STEP 2916 H=2916:00 +CPU= 25.961
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002916
  23:57:22 STEP 2917 H=2917:00 +CPU= 14.916
  23:57:38 STEP 2918 H=2918:00 +CPU= 15.309
  23:58:01 STEP 2919 H=2919:00 +CPU= 23.452
  23:58:17 STEP 2920 H=2920:00 +CPU= 15.292
  23:58:36 STEP 2921 H=2921:00 +CPU= 19.716
  23:59:11 STEP 2922 H=2922:00 +CPU= 34.751
  23:59:33 STEP 2923 H=2923:00 +CPU= 21.573
  23:59:55 STEP 2924 H=2924:00 +CPU= 22.308
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0837
(argv4) batchid: 954
(argv5) wuid: 12171481
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_cb62e892032113c1c9d8965e4dd12912 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0837_1985050100_123_954_12171481.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0837_1985050100_123_954_12171481.zip
ic_ancil_file: ic_ancil_12171481
ifsdata_file: ifsdata_12171481
climate_data_file: clim_data_12171481
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d847a3c58a0955de480aba5542ccae9f to: /var/lib/boinc-client/slots/9/ic_ancil_12171481.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12171481.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3113a523a1895e049a58ab3408559eb4 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12171481.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12171481.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12171481.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12171481.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12171481.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12171481.xml
-- Model is restarting --
Adjusting last_iter, 2925, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12171481.xml
last_cpu_time: 78136
upload_file_number: 121
last_iter: 2903
last_upload: 10454400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499
The child process has been launched with process id: 25898
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25898:25898] [20230109:000243:1673218963:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:03:24 STEP 2904 H=2904:00 +CPU= 38.677
  00:03:46 STEP 2905 H=2905:00 +CPU= 22.055
  00:04:08 STEP 2906 H=2906:00 +CPU= 22.202
  00:04:42 STEP 2907 H=2907:00 +CPU= 33.964
  00:05:01 STEP 2908 H=2908:00 +CPU= 19.169
  00:05:19 STEP 2909 H=2909:00 +CPU= 17.378
  00:05:50 STEP 2910 H=2910:00 +CPU= 30.975
  00:06:06 STEP 2911 H=2911:00 +CPU= 16.609
  00:06:22 STEP 2912 H=2912:00 +CPU= 16.069
  00:06:53 STEP 2913 H=2913:00 +CPU= 30.657
  00:07:15 STEP 2914 H=2914:00 +CPU= 22.255
  00:07:37 STEP 2915 H=2915:00 +CPU= 21.587
  00:08:09 STEP 2916 H=2916:00 +CPU= 32.634
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002916
  00:08:32 STEP 2917 H=2917:00 +CPU= 22.182
  00:08:49 STEP 2918 H=2918:00 +CPU= 16.955
  00:09:17 STEP 2919 H=2919:00 +CPU= 28.071
  00:09:32 STEP 2920 H=2920:00 +CPU= 15.103
  00:09:48 STEP 2921 H=2921:00 +CPU= 16.442
  00:10:14 STEP 2922 H=2922:00 +CPU= 25.545
  00:10:28 STEP 2923 H=2923:00 +CPU= 14.735
  00:10:44 STEP 2924 H=2924:00 +CPU= 16.000
  00:11:17 STEP 2925 H=2925:00 +CPU= 32.066
  00:11:39 STEP 2926 H=2926:00 +CPU= 22.237
  00:12:01 STEP 2927 H=2927:00 +CPU= 22.197
  00:12:41 STEP 2928 H=2928:00 +CPU= 39.764
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171481/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0837_1985050100_123_954_12171481_0_r746502499_121.zip
Uploading the intermediate file: upload_file_121.zip
  00:13:02 STEP 2929 H=2929:00 +CPU= 21.520
  00:13:25 STEP 2930 H=2930:00 +CPU= 22.235
  00:13:58 STEP 2931 H=2931:00 +CPU= 33.843
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:13:07 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 10,022,400 74,643 0.0074
09 May 2023 16:13:02 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 9,158,400 67,740 0.0074
09 May 2023 16:13:02 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 9,158,400 67,740 0.0074
09 May 2023 16:12:57 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 8,294,400 61,344 0.0074
09 May 2023 16:12:49 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 6,739,200 50,157 0.0074
09 May 2023 16:12:49 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 6,739,200 50,157 0.0074
09 May 2023 16:12:45 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 5,788,800 43,330 0.0075
09 May 2023 16:12:45 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 5,788,800 43,330 0.0075
09 May 2023 16:12:40 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 4,924,800 37,161 0.0075
09 May 2023 16:12:35 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 3,974,400 30,545 0.0077
09 May 2023 16:12:35 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 3,974,400 30,545 0.0077
09 May 2023 16:12:28 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 2,592,000 20,625 0.0080
09 May 2023 16:12:28 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 2,592,000 20,625 0.0080
09 May 2023 16:12:24 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 1,728,000 14,308 0.0083
09 May 2023 16:12:19 1534455 22255852 oifs_43r3_ps_0837_1985050100_123_954_12171481_0 864,000 7,015 0.0081


©2024 cpdn.org