climateprediction.net home page
Task 22254243

Task 22254243

Name oifs_43r3_ps_0292_1984050100_123_953_12169936_0
Workunit 12169936
Created 21 Dec 2022, 3:00:56 UTC
Sent 8 Jan 2023, 11:18:18 UTC
Report deadline 7 Feb 2023, 11:18:18 UTC
Received 13 Jan 2023, 18:19:38 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532028
Run time 18 hours 40 min 29 sec
CPU time 18 hours 16 min 49 sec
Validate state Invalid
Credit 12,096.00
Device peak FLOPS 8.61 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,788.13 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,699.77 MB
Stderr
<core_client_version>7.20.4</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 file: /mnt/home/boinc/slots/6/ic_ancil_12169936.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d1d1154e0e28239a18cea3c174d9d587 to: /mnt/home/boinc/slots/6/ifsdata/ifsdata_12169936.zip
Unzipping the ifsdata_zip file: /mnt/home/boinc/slots/6/ifsdata/ifsdata_12169936.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/home/boinc/slots/6/159l_2/clim_data_12169936.zip
Unzipping the climate data zip file: /mnt/home/boinc/slots/6/159l_2/clim_data_12169936.zip
Checking for progress XML file: /mnt/home/boinc/slots/6/progress_file_12169936.xml
Opened progress file ok : /mnt/home/boinc/slots/6/progress_file_12169936.xml
-- Model is restarting --
Adjusting last_iter, 2461, to previous model restart step.
Creating progress file: /mnt/home/boinc/slots/6/progress_file_12169936.xml
last_cpu_time: 56578
upload_file_number: 102
last_iter: 2447
last_upload: 8812800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405
The child process has been launched with process id: 27166
Executing the command: /mnt/home/boinc/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:27166:27166] [20230111:195230:1673463150:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:53:10 STEP 2448 H=2448:00 +CPU= 34.865
  19:53:29 STEP 2449 H=2449:00 +CPU= 19.259
  19:53:48 STEP 2450 H=2450:00 +CPU= 19.173
  19:54:19 STEP 2451 H=2451:00 +CPU= 30.532
  19:54:38 STEP 2452 H=2452:00 +CPU= 19.356
  19:54:57 STEP 2453 H=2453:00 +CPU= 18.972
  19:55:29 STEP 2454 H=2454:00 +CPU= 31.349
  19:55:48 STEP 2455 H=2455:00 +CPU= 18.809
  19:56:07 STEP 2456 H=2456:00 +CPU= 19.457
  19:56:38 STEP 2457 H=2457:00 +CPU= 30.311
  19:56:58 STEP 2458 H=2458:00 +CPU= 19.946
  19:57:17 STEP 2459 H=2459:00 +CPU= 19.451
  19:57:51 STEP 2460 H=2460:00 +CPU= 33.267
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002460
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002460
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002460
  19:58:10 STEP 2461 H=2461:00 +CPU= 18.583
  19:58:30 STEP 2462 H=2462:00 +CPU= 20.106
  19:59:01 STEP 2463 H=2463:00 +CPU= 31.382
  19:59:21 STEP 2464 H=2464:00 +CPU= 19.494
  19:59:40 STEP 2465 H=2465:00 +CPU= 19.432
  20:00:14 STEP 2466 H=2466:00 +CPU= 33.169
  20:00:35 STEP 2467 H=2467:00 +CPU= 20.875
  20:00:56 STEP 2468 H=2468:00 +CPU= 21.294
  20:01:29 STEP 2469 H=2469:00 +CPU= 32.911
  20:01:50 STEP 2470 H=2470:00 +CPU= 20.734
  20:02:12 STEP 2471 H=2471:00 +CPU= 21.425
  20:02:49 STEP 2472 H=2472:00 +CPU= 37.007
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002472
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002472
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002472
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002448
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002448
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002448
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002460
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002460
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002460
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_102.zip
Uploading the intermediate file: upload_file_102.zip
  20:03:08 STEP 2473 H=2473:00 +CPU= 19.411
  20:03:29 STEP 2474 H=2474:00 +CPU= 20.408
  20:04:02 STEP 2475 H=2475:00 +CPU= 32.654
  20:04:22 STEP 2476 H=2476:00 +CPU= 19.740
  20:04:42 STEP 2477 H=2477:00 +CPU= 20.171
  20:05:14 STEP 2478 H=2478:00 +CPU= 31.738
  20:05:33 STEP 2479 H=2479:00 +CPU= 19.142
  20:05:53 STEP 2480 H=2480:00 +CPU= 20.068
  20:06:22 STEP 2481 H=2481:00 +CPU= 28.537
  20:06:41 STEP 2482 H=2482:00 +CPU= 18.118
  20:06:54 STEP 2483 H=2483:00 +CPU= 13.314
  20:07:23 STEP 2484 H=2484:00 +CPU= 28.822
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002484
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002484
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002484
  20:07:42 STEP 2485 H=2485:00 +CPU= 18.745
  20:08:03 STEP 2486 H=2486:00 +CPU= 20.193
  20:08:35 STEP 2487 H=2487:00 +CPU= 31.327
  20:08:54 STEP 2488 H=2488:00 +CPU= 17.706
  20:09:11 STEP 2489 H=2489:00 +CPU= 16.317
  20:09:34 STEP 2490 H=2490:00 +CPU= 23.371
  20:09:47 STEP 2491 H=2491:00 +CPU= 12.568
  20:10:00 STEP 2492 H=2492:00 +CPU= 12.791
  20:10:20 STEP 2493 H=2493:00 +CPU= 19.693
  20:10:32 STEP 2494 H=2494:00 +CPU= 11.201
  20:10:44 STEP 2495 H=2495:00 +CPU= 11.972
  20:11:08 STEP 2496 H=2496:00 +CPU= 24.172
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002496
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002496
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002496
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002472
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002472
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002472
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002484
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002484
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002484
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_103.zip
Uploading the intermediate file: upload_file_103.zip
  20:11:24 STEP 2497 H=2497:00 +CPU= 16.120
  20:11:45 STEP 2498 H=2498:00 +CPU= 19.756
  20:12:16 STEP 2499 H=2499:00 +CPU= 30.818
  20:12:35 STEP 2500 H=2500:00 +CPU= 18.550
  20:12:55 STEP 2501 H=2501:00 +CPU= 19.853
  20:13:28 STEP 2502 H=2502:00 +CPU= 32.578
  20:13:47 STEP 2503 H=2503:00 +CPU= 19.052
  20:14:07 STEP 2504 H=2504:00 +CPU= 19.127
  20:14:33 STEP 2505 H=2505:00 +CPU= 25.740
  20:14:48 STEP 2506 H=2506:00 +CPU= 15.256
  20:15:04 STEP 2507 H=2507:00 +CPU= 15.334
  20:15:31 STEP 2508 H=2508:00 +CPU= 27.179
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002508
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002508
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002508
  20:15:46 STEP 2509 H=2509:00 +CPU= 14.123
  20:16:03 STEP 2510 H=2510:00 +CPU= 17.153
  20:16:33 STEP 2511 H=2511:00 +CPU= 29.888
  20:16:53 STEP 2512 H=2512:00 +CPU= 19.119
  20:17:15 STEP 2513 H=2513:00 +CPU= 22.018
  20:17:49 STEP 2514 H=2514:00 +CPU= 32.929
  20:18:09 STEP 2515 H=2515:00 +CPU= 19.708
  20:18:30 STEP 2516 H=2516:00 +CPU= 21.128
  20:19:03 STEP 2517 H=2517:00 +CPU= 32.531
  20:19:22 STEP 2518 H=2518:00 +CPU= 18.523
  20:19:44 STEP 2519 H=2519:00 +CPU= 21.168
  20:20:20 STEP 2520 H=2520:00 +CPU= 35.702
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002520
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002520
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002520
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002496
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002496
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002496
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002508
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002508
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002508
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_104.zip
Uploading the intermediate file: upload_file_104.zip
  20:20:40 STEP 2521 H=2521:00 +CPU= 19.076
  20:21:01 STEP 2522 H=2522:00 +CPU= 21.218
  20:21:35 STEP 2523 H=2523:00 +CPU= 32.963
  20:21:58 STEP 2524 H=2524:00 +CPU= 22.784
  20:22:17 STEP 2525 H=2525:00 +CPU= 17.470
  20:22:40 STEP 2526 H=2526:00 +CPU= 22.987
  20:22:54 STEP 2527 H=2527:00 +CPU= 13.346
  20:23:08 STEP 2528 H=2528:00 +CPU= 14.026
  20:23:31 STEP 2529 H=2529:00 +CPU= 21.669
  20:23:47 STEP 2530 H=2530:00 +CPU= 15.887
  20:24:09 STEP 2531 H=2531:00 +CPU= 20.967
  20:24:45 STEP 2532 H=2532:00 +CPU= 35.410
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002532
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002532
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002532
  20:25:06 STEP 2533 H=2533:00 +CPU= 20.905
  20:25:26 STEP 2534 H=2534:00 +CPU= 20.586
  20:25:57 STEP 2535 H=2535:00 +CPU= 30.637
  20:26:18 STEP 2536 H=2536:00 +CPU= 20.108
  20:26:38 STEP 2537 H=2537:00 +CPU= 20.303
  20:27:11 STEP 2538 H=2538:00 +CPU= 32.454
  20:27:32 STEP 2539 H=2539:00 +CPU= 20.409
  20:27:53 STEP 2540 H=2540:00 +CPU= 21.314
  20:28:25 STEP 2541 H=2541:00 +CPU= 32.076
  20:28:46 STEP 2542 H=2542:00 +CPU= 19.942
  20:29:08 STEP 2543 H=2543:00 +CPU= 21.845
  20:29:45 STEP 2544 H=2544:00 +CPU= 36.660
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002544
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002544
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002544
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002520
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002520
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002520
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002532
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002532
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002532
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_105.zip
Uploading the intermediate file: upload_file_105.zip
  20:30:05 STEP 2545 H=2545:00 +CPU= 20.077
  20:30:26 STEP 2546 H=2546:00 +CPU= 20.781
  20:30:57 STEP 2547 H=2547:00 +CPU= 30.412
  20:31:18 STEP 2548 H=2548:00 +CPU= 20.900
  20:31:39 STEP 2549 H=2549:00 +CPU= 20.739
  20:32:11 STEP 2550 H=2550:00 +CPU= 31.918
  20:32:32 STEP 2551 H=2551:00 +CPU= 20.978
  20:32:53 STEP 2552 H=2552:00 +CPU= 21.150
  20:33:23 STEP 2553 H=2553:00 +CPU= 29.541
  20:33:44 STEP 2554 H=2554:00 +CPU= 20.560
  20:34:05 STEP 2555 H=2555:00 +CPU= 20.846
  20:34:40 STEP 2556 H=2556:00 +CPU= 34.612
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002556
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002556
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002556
  20:35:00 STEP 2557 H=2557:00 +CPU= 20.225
  20:35:22 STEP 2558 H=2558:00 +CPU= 21.092
  20:35:52 STEP 2559 H=2559:00 +CPU= 30.387
  20:36:13 STEP 2560 H=2560:00 +CPU= 20.792
  20:36:34 STEP 2561 H=2561:00 +CPU= 20.716
  20:37:06 STEP 2562 H=2562:00 +CPU= 31.908
  20:37:27 STEP 2563 H=2563:00 +CPU= 20.257
  20:37:48 STEP 2564 H=2564:00 +CPU= 21.518
  20:38:19 STEP 2565 H=2565:00 +CPU= 30.039
  20:38:39 STEP 2566 H=2566:00 +CPU= 19.672
  20:38:58 STEP 2567 H=2567:00 +CPU= 19.687
  20:39:34 STEP 2568 H=2568:00 +CPU= 35.398
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002568
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002568
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002568
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002544
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002544
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002544
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002556
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002556
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002556
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_106.zip
Uploading the intermediate file: upload_file_106.zip
  20:39:55 STEP 2569 H=2569:00 +CPU= 19.949
  20:40:15 STEP 2570 H=2570:00 +CPU= 19.988
  20:40:45 STEP 2571 H=2571:00 +CPU= 30.473
  20:41:06 STEP 2572 H=2572:00 +CPU= 20.180
  20:41:26 STEP 2573 H=2573:00 +CPU= 19.797
  20:41:58 STEP 2574 H=2574:00 +CPU= 32.023
  20:42:19 STEP 2575 H=2575:00 +CPU= 20.427
  20:42:39 STEP 2576 H=2576:00 +CPU= 20.409
  20:43:10 STEP 2577 H=2577:00 +CPU= 30.665
  20:43:30 STEP 2578 H=2578:00 +CPU= 19.900
  20:43:49 STEP 2579 H=2579:00 +CPU= 18.093
  20:44:21 STEP 2580 H=2580:00 +CPU= 31.867
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002580
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002580
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002580
  20:44:40 STEP 2581 H=2581:00 +CPU= 18.151
  20:44:58 STEP 2582 H=2582:00 +CPU= 18.184
  20:45:26 STEP 2583 H=2583:00 +CPU= 28.124
  20:45:45 STEP 2584 H=2584:00 +CPU= 18.354
  20:46:03 STEP 2585 H=2585:00 +CPU= 17.752
  20:46:31 STEP 2586 H=2586:00 +CPU= 28.577
  20:46:50 STEP 2587 H=2587:00 +CPU= 18.016
  20:47:09 STEP 2588 H=2588:00 +CPU= 19.349
  20:47:37 STEP 2589 H=2589:00 +CPU= 28.021
  20:47:58 STEP 2590 H=2590:00 +CPU= 20.137
  20:48:18 STEP 2591 H=2591:00 +CPU= 20.490
  20:48:54 STEP 2592 H=2592:00 +CPU= 35.216
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002592
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002592
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002592
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002568
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002568
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002568
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002580
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002580
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002580
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_107.zip
Uploading the intermediate file: upload_file_107.zip
  20:49:14 STEP 2593 H=2593:00 +CPU= 19.855
  20:49:34 STEP 2594 H=2594:00 +CPU= 20.074
  20:50:06 STEP 2595 H=2595:00 +CPU= 31.684
  20:50:26 STEP 2596 H=2596:00 +CPU= 19.794
  20:50:46 STEP 2597 H=2597:00 +CPU= 19.478
  20:51:18 STEP 2598 H=2598:00 +CPU= 32.061
  20:51:38 STEP 2599 H=2599:00 +CPU= 19.635
  20:51:58 STEP 2600 H=2600:00 +CPU= 19.882
  20:52:30 STEP 2601 H=2601:00 +CPU= 31.079
  20:52:50 STEP 2602 H=2602:00 +CPU= 20.335
  20:53:10 STEP 2603 H=2603:00 +CPU= 19.559
  20:53:44 STEP 2604 H=2604:00 +CPU= 33.953
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002604
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002604
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002604
  20:54:04 STEP 2605 H=2605:00 +CPU= 19.557
  20:54:24 STEP 2606 H=2606:00 +CPU= 20.044
  20:54:55 STEP 2607 H=2607:00 +CPU= 30.720
  20:55:15 STEP 2608 H=2608:00 +CPU= 20.007
  20:55:35 STEP 2609 H=2609:00 +CPU= 19.360
  20:56:07 STEP 2610 H=2610:00 +CPU= 32.032
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0292
(argv4) batchid: 953
(argv5) wuid: 12169936
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/home/boinc/slots/6
Project directory is: /mnt/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936
..mkdir for temp folder for results failed
Copying: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/home/boinc/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/home/boinc/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b42568e682a37b1d9c0844ad9144fbf7 to: /mnt/home/boinc/slots/6/oifs_43r3_ps_0292_1984050100_123_953_12169936.zip
Unzipping the namelist zip file: /mnt/home/boinc/slots/6/oifs_43r3_ps_0292_1984050100_123_953_12169936.zip
ic_ancil_file: ic_ancil_12169936
ifsdata_file: ifsdata_12169936
climate_data_file: clim_data_12169936
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_926f6e33cb192b14883e72faaafe4013 to: /mnt/home/boinc/slots/6/ic_ancil_12169936.zip
Unzipping the IC ancils zip file: /mnt/home/boinc/slots/6/ic_ancil_12169936.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d1d1154e0e28239a18cea3c174d9d587 to: /mnt/home/boinc/slots/6/ifsdata/ifsdata_12169936.zip
Unzipping the ifsdata_zip file: /mnt/home/boinc/slots/6/ifsdata/ifsdata_12169936.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/home/boinc/slots/6/159l_2/clim_data_12169936.zip
Unzipping the climate data zip file: /mnt/home/boinc/slots/6/159l_2/clim_data_12169936.zip
Checking for progress XML file: /mnt/home/boinc/slots/6/progress_file_12169936.xml
Opened progress file ok : /mnt/home/boinc/slots/6/progress_file_12169936.xml
-- Model is restarting --
Adjusting last_iter, 2611, to previous model restart step.
Creating progress file: /mnt/home/boinc/slots/6/progress_file_12169936.xml
last_cpu_time: 60352
upload_file_number: 108
last_iter: 2591
last_upload: 9331200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405
The child process has been launched with process id: 2795
Executing the command: /mnt/home/boinc/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:2795:2795] [20230112:193454:1673548494:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:35:39 STEP 2592 H=2592:00 +CPU= 34.139
  19:35:59 STEP 2593 H=2593:00 +CPU= 20.376
  19:36:19 STEP 2594 H=2594:00 +CPU= 19.111
  19:36:45 STEP 2595 H=2595:00 +CPU= 26.520
  19:37:05 STEP 2596 H=2596:00 +CPU= 19.404
  19:37:23 STEP 2597 H=2597:00 +CPU= 18.015
  19:37:53 STEP 2598 H=2598:00 +CPU= 29.993
  19:38:13 STEP 2599 H=2599:00 +CPU= 19.164
  19:38:34 STEP 2600 H=2600:00 +CPU= 19.589
  19:39:03 STEP 2601 H=2601:00 +CPU= 28.472
  19:39:24 STEP 2602 H=2602:00 +CPU= 19.544
  19:39:44 STEP 2603 H=2603:00 +CPU= 19.206
  19:40:17 STEP 2604 H=2604:00 +CPU= 31.878
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002604
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002604
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002604
  19:40:37 STEP 2605 H=2605:00 +CPU= 19.285
  19:40:57 STEP 2606 H=2606:00 +CPU= 19.421
  19:41:26 STEP 2607 H=2607:00 +CPU= 28.478
  19:41:47 STEP 2608 H=2608:00 +CPU= 20.278
  19:42:07 STEP 2609 H=2609:00 +CPU= 19.412
  19:42:38 STEP 2610 H=2610:00 +CPU= 29.312
  19:42:58 STEP 2611 H=2611:00 +CPU= 19.954
  19:43:16 STEP 2612 H=2612:00 +CPU= 15.693
  19:43:37 STEP 2613 H=2613:00 +CPU= 20.474
  19:43:50 STEP 2614 H=2614:00 +CPU= 13.293
  19:44:07 STEP 2615 H=2615:00 +CPU= 16.440
  19:44:42 STEP 2616 H=2616:00 +CPU= 31.487
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002616
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002616
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002616
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002592
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002592
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002592
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002604
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002604
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002604
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_108.zip
Uploading the intermediate file: upload_file_108.zip
  19:45:00 STEP 2617 H=2617:00 +CPU= 18.053
  19:45:19 STEP 2618 H=2618:00 +CPU= 18.434
  19:45:47 STEP 2619 H=2619:00 +CPU= 27.947
  19:46:07 STEP 2620 H=2620:00 +CPU= 19.734
  19:46:26 STEP 2621 H=2621:00 +CPU= 19.274
  19:46:58 STEP 2622 H=2622:00 +CPU= 30.637
  19:47:18 STEP 2623 H=2623:00 +CPU= 19.993
  19:47:39 STEP 2624 H=2624:00 +CPU= 20.169
  19:48:10 STEP 2625 H=2625:00 +CPU= 30.525
  19:48:31 STEP 2626 H=2626:00 +CPU= 20.723
  19:48:54 STEP 2627 H=2627:00 +CPU= 21.159
  19:49:30 STEP 2628 H=2628:00 +CPU= 34.830
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002628
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002628
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002628
  19:49:51 STEP 2629 H=2629:00 +CPU= 20.899
  19:50:13 STEP 2630 H=2630:00 +CPU= 21.249
  19:50:46 STEP 2631 H=2631:00 +CPU= 31.263
  19:51:08 STEP 2632 H=2632:00 +CPU= 21.303
  19:51:29 STEP 2633 H=2633:00 +CPU= 20.763
  19:52:02 STEP 2634 H=2634:00 +CPU= 31.860
  19:52:24 STEP 2635 H=2635:00 +CPU= 20.987
  19:52:46 STEP 2636 H=2636:00 +CPU= 21.374
  19:53:20 STEP 2637 H=2637:00 +CPU= 32.281
  19:53:37 STEP 2638 H=2638:00 +CPU= 16.409
  19:53:50 STEP 2639 H=2639:00 +CPU= 13.437
  19:54:16 STEP 2640 H=2640:00 +CPU= 23.324
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002640
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002640
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002640
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002616
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002616
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002616
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002628
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002628
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002628
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_109.zip
Uploading the intermediate file: upload_file_109.zip
  19:54:28 STEP 2641 H=2641:00 +CPU= 12.638
  19:54:45 STEP 2642 H=2642:00 +CPU= 16.800
  19:55:15 STEP 2643 H=2643:00 +CPU= 29.364
  19:55:35 STEP 2644 H=2644:00 +CPU= 19.094
  19:55:52 STEP 2645 H=2645:00 +CPU= 16.957
  19:56:12 STEP 2646 H=2646:00 +CPU= 20.085
  19:56:24 STEP 2647 H=2647:00 +CPU= 11.641
  19:56:35 STEP 2648 H=2648:00 +CPU= 11.433
  19:56:54 STEP 2649 H=2649:00 +CPU= 18.416
  19:57:08 STEP 2650 H=2650:00 +CPU= 14.408
  19:57:29 STEP 2651 H=2651:00 +CPU= 19.956
  19:58:03 STEP 2652 H=2652:00 +CPU= 33.562
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002652
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002652
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002652
  19:58:24 STEP 2653 H=2653:00 +CPU= 20.456
  19:58:44 STEP 2654 H=2654:00 +CPU= 19.471
  19:59:04 STEP 2655 H=2655:00 +CPU= 20.179
  19:59:16 STEP 2656 H=2656:00 +CPU= 12.030
  19:59:28 STEP 2657 H=2657:00 +CPU= 11.609
  19:59:54 STEP 2658 H=2658:00 +CPU= 25.730
  20:00:12 STEP 2659 H=2659:00 +CPU= 17.417
  20:00:29 STEP 2660 H=2660:00 +CPU= 17.061
  20:00:59 STEP 2661 H=2661:00 +CPU= 28.708
  20:01:19 STEP 2662 H=2662:00 +CPU= 20.310
  20:01:39 STEP 2663 H=2663:00 +CPU= 19.786
  20:02:15 STEP 2664 H=2664:00 +CPU= 35.384
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002664
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002664
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002664
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002640
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002640
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002640
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002652
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002652
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002652
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_110.zip
Uploading the intermediate file: upload_file_110.zip
  20:02:36 STEP 2665 H=2665:00 +CPU= 20.316
  20:02:56 STEP 2666 H=2666:00 +CPU= 20.265
  20:03:25 STEP 2667 H=2667:00 +CPU= 27.945
  20:03:44 STEP 2668 H=2668:00 +CPU= 18.708
  20:04:03 STEP 2669 H=2669:00 +CPU= 19.353
  20:04:34 STEP 2670 H=2670:00 +CPU= 30.410
  20:04:55 STEP 2671 H=2671:00 +CPU= 20.762
  20:05:16 STEP 2672 H=2672:00 +CPU= 20.181
  20:05:47 STEP 2673 H=2673:00 +CPU= 30.481
  20:06:08 STEP 2674 H=2674:00 +CPU= 20.646
  20:06:28 STEP 2675 H=2675:00 +CPU= 19.606
  20:06:57 STEP 2676 H=2676:00 +CPU= 29.548
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002676
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002676
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002676
  20:07:11 STEP 2677 H=2677:00 +CPU= 13.701
  20:07:26 STEP 2678 H=2678:00 +CPU= 14.645
  20:07:52 STEP 2679 H=2679:00 +CPU= 26.024
  20:08:12 STEP 2680 H=2680:00 +CPU= 19.538
  20:08:30 STEP 2681 H=2681:00 +CPU= 17.719
  20:08:59 STEP 2682 H=2682:00 +CPU= 28.079
  20:09:15 STEP 2683 H=2683:00 +CPU= 15.909
  20:09:30 STEP 2684 H=2684:00 +CPU= 14.932
  20:09:55 STEP 2685 H=2685:00 +CPU= 24.398
  20:10:13 STEP 2686 H=2686:00 +CPU= 17.842
  20:10:32 STEP 2687 H=2687:00 +CPU= 18.602
  20:11:06 STEP 2688 H=2688:00 +CPU= 33.115
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002688
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002688
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002688
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002664
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002664
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002664
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002676
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002676
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002676
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_111.zip
Uploading the intermediate file: upload_file_111.zip
  20:11:24 STEP 2689 H=2689:00 +CPU= 18.190
  20:11:43 STEP 2690 H=2690:00 +CPU= 18.752
  20:12:12 STEP 2691 H=2691:00 +CPU= 28.494
  20:12:31 STEP 2692 H=2692:00 +CPU= 19.004
  20:12:50 STEP 2693 H=2693:00 +CPU= 18.167
  20:13:18 STEP 2694 H=2694:00 +CPU= 28.484
  20:13:37 STEP 2695 H=2695:00 +CPU= 18.763
  20:13:57 STEP 2696 H=2696:00 +CPU= 19.141
  20:14:25 STEP 2697 H=2697:00 +CPU= 27.671
  20:14:44 STEP 2698 H=2698:00 +CPU= 18.601
  20:15:02 STEP 2699 H=2699:00 +CPU= 18.465
  20:15:33 STEP 2700 H=2700:00 +CPU= 30.371
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002700
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002700
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002700
  20:15:52 STEP 2701 H=2701:00 +CPU= 18.440
  20:16:11 STEP 2702 H=2702:00 +CPU= 18.633
  20:16:39 STEP 2703 H=2703:00 +CPU= 28.124
  20:16:58 STEP 2704 H=2704:00 +CPU= 18.833
  20:17:17 STEP 2705 H=2705:00 +CPU= 18.396
  20:17:46 STEP 2706 H=2706:00 +CPU= 28.495
  20:18:05 STEP 2707 H=2707:00 +CPU= 18.981
  20:18:24 STEP 2708 H=2708:00 +CPU= 18.700
  20:18:53 STEP 2709 H=2709:00 +CPU= 28.824
  20:19:12 STEP 2710 H=2710:00 +CPU= 18.774
  20:19:32 STEP 2711 H=2711:00 +CPU= 19.468
  20:20:05 STEP 2712 H=2712:00 +CPU= 33.140
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002712
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002712
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002712
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002688
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002688
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002688
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002700
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002700
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002700
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_112.zip
Uploading the intermediate file: upload_file_112.zip
  20:20:25 STEP 2713 H=2713:00 +CPU= 19.173
  20:20:45 STEP 2714 H=2714:00 +CPU= 19.123
  20:21:16 STEP 2715 H=2715:00 +CPU= 30.169
  20:21:36 STEP 2716 H=2716:00 +CPU= 20.483
  20:21:57 STEP 2717 H=2717:00 +CPU= 20.185
  20:22:28 STEP 2718 H=2718:00 +CPU= 31.048
  20:22:49 STEP 2719 H=2719:00 +CPU= 20.127
  20:23:09 STEP 2720 H=2720:00 +CPU= 20.356
  20:23:41 STEP 2721 H=2721:00 +CPU= 30.960
  20:24:02 STEP 2722 H=2722:00 +CPU= 20.865
  20:24:22 STEP 2723 H=2723:00 +CPU= 19.170
  20:24:55 STEP 2724 H=2724:00 +CPU= 33.184
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002724
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002724
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002724
  20:25:17 STEP 2725 H=2725:00 +CPU= 20.412
  20:25:39 STEP 2726 H=2726:00 +CPU= 21.260
  20:26:09 STEP 2727 H=2727:00 +CPU= 29.623
  20:26:29 STEP 2728 H=2728:00 +CPU= 19.090
  20:26:47 STEP 2729 H=2729:00 +CPU= 18.111
  20:27:17 STEP 2730 H=2730:00 +CPU= 29.782
  20:27:38 STEP 2731 H=2731:00 +CPU= 19.994
  20:27:59 STEP 2732 H=2732:00 +CPU= 20.614
  20:28:31 STEP 2733 H=2733:00 +CPU= 31.553
  20:28:53 STEP 2734 H=2734:00 +CPU= 21.044
  20:29:15 STEP 2735 H=2735:00 +CPU= 21.207
  20:29:52 STEP 2736 H=2736:00 +CPU= 36.151
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002736
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002736
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002736
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002712
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002712
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002712
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002724
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002724
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002724
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_113.zip
Uploading the intermediate file: upload_file_113.zip
  20:30:12 STEP 2737 H=2737:00 +CPU= 19.714
  20:30:32 STEP 2738 H=2738:00 +CPU= 20.401
  20:31:03 STEP 2739 H=2739:00 +CPU= 29.967
  20:31:23 STEP 2740 H=2740:00 +CPU= 19.800
  20:31:43 STEP 2741 H=2741:00 +CPU= 19.917
  20:32:14 STEP 2742 H=2742:00 +CPU= 30.322
  20:32:35 STEP 2743 H=2743:00 +CPU= 20.667
  20:32:56 STEP 2744 H=2744:00 +CPU= 20.504
  20:33:28 STEP 2745 H=2745:00 +CPU= 30.832
  20:33:49 STEP 2746 H=2746:00 +CPU= 20.774
  20:34:07 STEP 2747 H=2747:00 +CPU= 17.671
  20:34:36 STEP 2748 H=2748:00 +CPU= 28.807
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002748
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002748
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002748
  20:34:53 STEP 2749 H=2749:00 +CPU= 16.593
  20:35:14 STEP 2750 H=2750:00 +CPU= 20.108
  20:35:47 STEP 2751 H=2751:00 +CPU= 32.067
  20:36:08 STEP 2752 H=2752:00 +CPU= 20.624
  20:36:28 STEP 2753 H=2753:00 +CPU= 19.398
  20:36:58 STEP 2754 H=2754:00 +CPU= 30.096
  20:37:18 STEP 2755 H=2755:00 +CPU= 19.594
  20:37:37 STEP 2756 H=2756:00 +CPU= 19.161
  20:38:07 STEP 2757 H=2757:00 +CPU= 30.005
  20:38:26 STEP 2758 H=2758:00 +CPU= 18.636
  20:38:44 STEP 2759 H=2759:00 +CPU= 18.135
  20:39:17 STEP 2760 H=2760:00 +CPU= 32.350
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002760
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002760
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002760
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002736
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002736
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002736
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002748
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002748
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002748
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_114.zip
Uploading the intermediate file: upload_file_114.zip
  20:39:34 STEP 2761 H=2761:00 +CPU= 16.883
  20:39:52 STEP 2762 H=2762:00 +CPU= 18.160
  20:40:19 STEP 2763 H=2763:00 +CPU= 27.035
  20:40:37 STEP 2764 H=2764:00 +CPU= 18.319
  20:40:54 STEP 2765 H=2765:00 +CPU= 16.913
  20:41:21 STEP 2766 H=2766:00 +CPU= 26.674
  20:41:39 STEP 2767 H=2767:00 +CPU= 17.874
  20:41:57 STEP 2768 H=2768:00 +CPU= 17.478
  20:42:25 STEP 2769 H=2769:00 +CPU= 28.543
  20:42:46 STEP 2770 H=2770:00 +CPU= 20.686
  20:43:05 STEP 2771 H=2771:00 +CPU= 19.241
  20:43:38 STEP 2772 H=2772:00 +CPU= 32.140
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002772
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002772
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002772
  20:43:57 STEP 2773 H=2773:00 +CPU= 19.013
  20:44:16 STEP 2774 H=2774:00 +CPU= 19.621
  20:44:46 STEP 2775 H=2775:00 +CPU= 29.439
  20:45:06 STEP 2776 H=2776:00 +CPU= 19.534
  20:45:26 STEP 2777 H=2777:00 +CPU= 20.022
  20:45:55 STEP 2778 H=2778:00 +CPU= 29.033
  20:46:15 STEP 2779 H=2779:00 +CPU= 19.781
  20:46:34 STEP 2780 H=2780:00 +CPU= 19.702
  20:47:05 STEP 2781 H=2781:00 +CPU= 30.142
  20:47:25 STEP 2782 H=2782:00 +CPU= 20.210
  20:47:45 STEP 2783 H=2783:00 +CPU= 19.685
  20:48:20 STEP 2784 H=2784:00 +CPU= 35.203
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002784
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002784
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002784
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002760
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002760
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002760
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002772
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002772
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002772
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_115.zip
Uploading the intermediate file: upload_file_115.zip
  20:48:40 STEP 2785 H=2785:00 +CPU= 19.423
  20:49:00 STEP 2786 H=2786:00 +CPU= 20.538
  20:49:29 STEP 2787 H=2787:00 +CPU= 29.080
  20:49:50 STEP 2788 H=2788:00 +CPU= 20.381
  20:50:09 STEP 2789 H=2789:00 +CPU= 19.406
  20:50:39 STEP 2790 H=2790:00 +CPU= 29.527
  20:50:59 STEP 2791 H=2791:00 +CPU= 19.449
  20:51:18 STEP 2792 H=2792:00 +CPU= 19.046
  20:51:47 STEP 2793 H=2793:00 +CPU= 29.250
  20:52:07 STEP 2794 H=2794:00 +CPU= 20.078
  20:52:27 STEP 2795 H=2795:00 +CPU= 19.650
  20:53:00 STEP 2796 H=2796:00 +CPU= 32.333
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002796
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002796
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002796
  20:53:19 STEP 2797 H=2797:00 +CPU= 19.603
  20:53:39 STEP 2798 H=2798:00 +CPU= 19.476
  20:54:08 STEP 2799 H=2799:00 +CPU= 29.251
  20:54:28 STEP 2800 H=2800:00 +CPU= 19.358
  20:54:47 STEP 2801 H=2801:00 +CPU= 19.219
  20:55:17 STEP 2802 H=2802:00 +CPU= 29.967
  20:55:37 STEP 2803 H=2803:00 +CPU= 19.406
  20:55:55 STEP 2804 H=2804:00 +CPU= 18.786
  20:56:26 STEP 2805 H=2805:00 +CPU= 30.014
  20:56:45 STEP 2806 H=2806:00 +CPU= 19.156
  20:57:06 STEP 2807 H=2807:00 +CPU= 20.870
  20:57:41 STEP 2808 H=2808:00 +CPU= 35.107
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0292
(argv4) batchid: 953
(argv5) wuid: 12169936
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/home/boinc/slots/6
Project directory is: /mnt/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936
..mkdir for temp folder for results failed
Copying: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/home/boinc/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/home/boinc/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b42568e682a37b1d9c0844ad9144fbf7 to: /mnt/home/boinc/slots/6/oifs_43r3_ps_0292_1984050100_123_953_12169936.zip
Unzipping the namelist zip file: /mnt/home/boinc/slots/6/oifs_43r3_ps_0292_1984050100_123_953_12169936.zip
ic_ancil_file: ic_ancil_12169936
ifsdata_file: ifsdata_12169936
climate_data_file: clim_data_12169936
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_926f6e33cb192b14883e72faaafe4013 to: /mnt/home/boinc/slots/6/ic_ancil_12169936.zip
Unzipping the IC ancils zip file: /mnt/home/boinc/slots/6/ic_ancil_12169936.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d1d1154e0e28239a18cea3c174d9d587 to: /mnt/home/boinc/slots/6/ifsdata/ifsdata_12169936.zip
Unzipping the ifsdata_zip file: /mnt/home/boinc/slots/6/ifsdata/ifsdata_12169936.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/home/boinc/slots/6/159l_2/clim_data_12169936.zip
Unzipping the climate data zip file: /mnt/home/boinc/slots/6/159l_2/clim_data_12169936.zip
Checking for progress XML file: /mnt/home/boinc/slots/6/progress_file_12169936.xml
Opened progress file ok : /mnt/home/boinc/slots/6/progress_file_12169936.xml
-- Model is restarting --
Adjusting last_iter, 2808, to previous model restart step.
Creating progress file: /mnt/home/boinc/slots/6/progress_file_12169936.xml
last_cpu_time: 65223
upload_file_number: 116
last_iter: 2807
last_upload: 10022400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405
The child process has been launched with process id: 14192
Executing the command: /mnt/home/boinc/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:14192:14192] [20230113:174857:1673628537:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:49:39 STEP 2808 H=2808:00 +CPU= 32.657
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002808
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002808
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002808
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002784
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002784
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002784
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMGGhq0f+002796
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMSHhq0f+002796
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12169936/ICMUAhq0f+002796
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0292_1984050100_123_953_12169936_0_r1084639405_116.zip
Uploading the intermediate file: upload_file_116.zip
  17:49:58 STEP 2809 H=2809:00 +CPU= 19.260
  17:50:15 STEP 2810 H=2810:00 +CPU= 17.028
  17:50:43 STEP 2811 H=2811:00 +CPU= 27.248
  17:51:01 STEP 2812 H=2812:00 +CPU= 17.821
  17:51:19 STEP 2813 H=2813:00 +CPU= 18.542
  17:51:48 STEP 2814 H=2814:00 +CPU= 28.296
  17:52:05 STEP 2815 H=2815:00 +CPU= 17.821
  17:52:25 STEP 2816 H=2816:00 +CPU= 19.532
  17:52:55 STEP 2817 H=2817:00 +CPU= 29.596
  17:53:15 STEP 2818 H=2818:00 +CPU= 20.051
  17:53:34 STEP 2819 H=2819:00 +CPU= 19.059
  17:54:05 STEP 2820 H=2820:00 +CPU= 31.413
Moving to projects directory: /mnt/home/boinc/slots/6/ICMGGhq0f+002820
Moving to projects directory: /mnt/home/boinc/slots/6/ICMSHhq0f+002820
Moving to projects directory: /mnt/home/boinc/slots/6/ICMUAhq0f+002820
  17:54:25 STEP 2821 H=2821:00 +CPU= 19.253
  17:54:46 STEP 2822 H=2822:00 +CPU= 21.026
  17:55:15 STEP 2823 H=2823:00 +CPU= 28.655
  17:55:34 STEP 2824 H=2824:00 +CPU= 19.099
  17:55:53 STEP 2825 H=2825:00 +CPU= 18.848
  17:56:23 STEP 2826 H=2826:00 +CPU= 29.575
  17:56:42 STEP 2827 H=2827:00 +CPU= 18.994
  17:57:02 STEP 2828 H=2828:00 +CPU= 19.880
  17:57:33 STEP 2829 H=2829:00 +CPU= 29.680
  17:57:52 STEP 2830 H=2830:00 +CPU= 19.549
  17:58:13 STEP 2831 H=2831:00 +CPU= 20.261
  17:58:49 STEP 2832 H=2832:00 +CPU= 34.963
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:15:48 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 8,640,000 55,217 0.0064
09 May 2023 16:14:54 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 7,084,800 45,132 0.0064
09 May 2023 16:13:56 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 5,616,000 35,450 0.0063
09 May 2023 16:12:59 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 4,320,000 26,837 0.0062
09 May 2023 16:12:55 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 3,456,000 21,722 0.0063
09 May 2023 16:12:53 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 2,592,000 16,429 0.0063
09 May 2023 16:12:53 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 2,592,000 16,429 0.0063
09 May 2023 16:12:49 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 1,728,000 11,090 0.0064
09 May 2023 16:12:49 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 1,728,000 11,090 0.0064
09 May 2023 16:12:46 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 864,000 5,580 0.0065
09 May 2023 16:12:45 1532028 22254243 oifs_43r3_ps_0292_1984050100_123_953_12169936_0 864,000 5,580 0.0065


©2024 cpdn.org