climateprediction.net home page
Task 22287595

Task 22287595

Name oifs_43r3_ps_0645_2015050100_123_984_12201289_0
Workunit 12201289
Created 24 Dec 2022, 13:19:43 UTC
Sent 18 Jan 2023, 10:06:07 UTC
Report deadline 19 Mar 2023, 10:06:07 UTC
Received 21 Jan 2023, 18:25:03 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1525245
Run time 15 hours 38 min 47 sec
CPU time 13 hours 16 min 46 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.12 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.77 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,576.03 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_16.zip
Uploading the intermediate file: upload_file_16.zip
  11:31:48 STEP  409 H= 409:00 +CPU= 35.439
  11:32:33 STEP  410 H= 410:00 +CPU= 36.501
  11:33:39 STEP  411 H= 411:00 +CPU= 55.774
  11:34:22 STEP  412 H= 412:00 +CPU= 36.386
  11:35:04 STEP  413 H= 413:00 +CPU= 35.363
  11:36:11 STEP  414 H= 414:00 +CPU= 57.317
  11:36:54 STEP  415 H= 415:00 +CPU= 35.324
  11:37:37 STEP  416 H= 416:00 +CPU= 36.511
  11:38:43 STEP  417 H= 417:00 +CPU= 55.631
  11:39:26 STEP  418 H= 418:00 +CPU= 36.529
  11:40:07 STEP  419 H= 419:00 +CPU= 35.269
  11:41:19 STEP  420 H= 420:00 +CPU= 61.657
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000420
  11:41:59 STEP  421 H= 421:00 +CPU= 35.338
  11:42:40 STEP  422 H= 422:00 +CPU= 36.269
  11:43:43 STEP  423 H= 423:00 +CPU= 55.452
  11:44:25 STEP  424 H= 424:00 +CPU= 36.483
  11:45:04 STEP  425 H= 425:00 +CPU= 35.360
  11:46:10 STEP  426 H= 426:00 +CPU= 56.572
  11:46:51 STEP  427 H= 427:00 +CPU= 35.117
  11:47:34 STEP  428 H= 428:00 +CPU= 36.470
  11:48:38 STEP  429 H= 429:00 +CPU= 55.491
  11:49:20 STEP  430 H= 430:00 +CPU= 36.229
  11:50:03 STEP  431 H= 431:00 +CPU= 36.482
  11:51:21 STEP  432 H= 432:00 +CPU= 65.245
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_17.zip
Uploading the intermediate file: upload_file_17.zip
  11:52:00 STEP  433 H= 433:00 +CPU= 35.208
  11:52:40 STEP  434 H= 434:00 +CPU= 35.918
  11:53:41 STEP  435 H= 435:00 +CPU= 55.454
  11:54:21 STEP  436 H= 436:00 +CPU= 36.274
  11:55:03 STEP  437 H= 437:00 +CPU= 35.145
  11:56:08 STEP  438 H= 438:00 +CPU= 56.122
  11:56:48 STEP  439 H= 439:00 +CPU= 35.229
  11:57:29 STEP  440 H= 440:00 +CPU= 36.525
  11:58:30 STEP  441 H= 441:00 +CPU= 55.423
  11:59:14 STEP  442 H= 442:00 +CPU= 36.651
  11:59:54 STEP  443 H= 443:00 +CPU= 35.369
  12:01:03 STEP  444 H= 444:00 +CPU= 61.274
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000444
  12:01:43 STEP  445 H= 445:00 +CPU= 35.172
  12:02:26 STEP  446 H= 446:00 +CPU= 36.372
  12:03:30 STEP  447 H= 447:00 +CPU= 55.196
  12:04:12 STEP  448 H= 448:00 +CPU= 36.669
  12:04:52 STEP  449 H= 449:00 +CPU= 35.218
  12:05:57 STEP  450 H= 450:00 +CPU= 56.693
  12:06:36 STEP  451 H= 451:00 +CPU= 35.130
  12:07:21 STEP  452 H= 452:00 +CPU= 36.371
  12:08:27 STEP  453 H= 453:00 +CPU= 55.252
  12:09:09 STEP  454 H= 454:00 +CPU= 36.240
  12:09:51 STEP  455 H= 455:00 +CPU= 36.705
  12:11:05 STEP  456 H= 456:00 +CPU= 65.447
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_18.zip
Uploading the intermediate file: upload_file_18.zip
  12:11:45 STEP  457 H= 457:00 +CPU= 34.984
  12:12:26 STEP  458 H= 458:00 +CPU= 36.187
  12:13:29 STEP  459 H= 459:00 +CPU= 55.071
  12:14:11 STEP  460 H= 460:00 +CPU= 36.911
  12:14:53 STEP  461 H= 461:00 +CPU= 35.565
  12:16:00 STEP  462 H= 462:00 +CPU= 56.410
  12:16:42 STEP  463 H= 463:00 +CPU= 35.488
  12:17:27 STEP  464 H= 464:00 +CPU= 36.256
  12:18:37 STEP  465 H= 465:00 +CPU= 55.145
  12:19:20 STEP  466 H= 466:00 +CPU= 36.346
  12:20:01 STEP  467 H= 467:00 +CPU= 35.404
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2670MB (heap), 4416MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2670 MB (maxheap), 4416 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        PREDYNFPOS 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         STEPO 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          TRANSINVH 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           TRANSINV_MDL 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            INV_TRANS 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >TRS-LTINV_CTL   - INVERSE( 102) 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-LTINV_CTL   - INVERSE(1647) 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LTINV_MOD 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122001:1674300001:10429.510] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LE_DGEMM 
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122002:1674300002:10430.510] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:tourte:1:1:2582:2582] [20230121:122002:1674300002:10430.510] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000017111C3  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000ADF3D8  leinv_mod_mp_lein         121  leinv_mod.F90
oifs_43r3_model.e  0000000000AD9DFC  ltinv_mod_mp_ltin         296  ltinv_mod.F90
oifs_43r3_model.e  0000000000AD7AA3  ltinv_ctl_mod_mp_         100  ltinv_ctl_mod.F90
oifs_43r3_model.e  0000000000ACA56F  inv_trans_ctl_mod         274  inv_trans_ctl_mod.F90
oifs_43r3_model.e  000000000085346D  inv_trans_                600  inv_trans.F90
oifs_43r3_model.e  000000000133D94C  transinv_mdl_             286  transinv_mdl.F90
oifs_43r3_model.e  0000000001334D43  transinvh_                104  transinvh.F90
oifs_43r3_model.e  0000000000579C15  stepo_                    265  stepo.F90
oifs_43r3_model.e  0000000000803DC3  predynfpos_               147  predynfpos.F90
oifs_43r3_model.e  000000000055D9CB  cnt4_                     944  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0645
(argv4) batchid: 984
(argv5) wuid: 12201289
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e6e91692925f5de22f5ebe0bb511cdbb to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0645_2015050100_123_984_12201289.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0645_2015050100_123_984_12201289.zip
ic_ancil_file: ic_ancil_12201289
ifsdata_file: ifsdata_12201289
climate_data_file: clim_data_12201289
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_08e5ec787aa5a70a924ddc2961dca023 to: /var/lib/boinc-client/slots/10/ic_ancil_12201289.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12201289.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_953088beaaa06ad7fa9708622fbf896d to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12201289.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12201289.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12201289.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12201289.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12201289.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12201289.xml
-- Model is restarting --
Adjusting last_iter, 467, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12201289.xml
last_cpu_time: 32733
upload_file_number: 19
last_iter: 455
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309
The child process has been launched with process id: 3046
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:3046:3046] [20230121:141905:1674307145:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:20:42 STEP  456 H= 456:00 +CPU= 62.518
  14:21:23 STEP  457 H= 457:00 +CPU= 35.378
  14:22:07 STEP  458 H= 458:00 +CPU= 36.849
  14:23:12 STEP  459 H= 459:00 +CPU= 55.234
  14:23:54 STEP  460 H= 460:00 +CPU= 36.367
  14:24:34 STEP  461 H= 461:00 +CPU= 35.237
  14:25:38 STEP  462 H= 462:00 +CPU= 56.981
  14:26:19 STEP  463 H= 463:00 +CPU= 35.314
  14:27:00 STEP  464 H= 464:00 +CPU= 36.387
  14:28:04 STEP  465 H= 465:00 +CPU= 55.232
  14:28:48 STEP  466 H= 466:00 +CPU= 36.300
  14:29:29 STEP  467 H= 467:00 +CPU= 35.458
  14:30:39 STEP  468 H= 468:00 +CPU= 60.904
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000468
  14:31:17 STEP  469 H= 469:00 +CPU= 35.173
  14:31:59 STEP  470 H= 470:00 +CPU= 36.230
  14:33:01 STEP  471 H= 471:00 +CPU= 55.022
  14:33:42 STEP  472 H= 472:00 +CPU= 36.217
  14:34:22 STEP  473 H= 473:00 +CPU= 35.358
  14:35:24 STEP  474 H= 474:00 +CPU= 56.492
  14:36:03 STEP  475 H= 475:00 +CPU= 35.135
  14:36:44 STEP  476 H= 476:00 +CPU= 36.124
  14:37:48 STEP  477 H= 477:00 +CPU= 54.979
  14:38:33 STEP  478 H= 478:00 +CPU= 36.635
  14:39:17 STEP  479 H= 479:00 +CPU= 36.620
  14:40:31 STEP  480 H= 480:00 +CPU= 64.947
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_19.zip
Uploading the intermediate file: upload_file_19.zip
  14:41:13 STEP  481 H= 481:00 +CPU= 34.997
  14:41:56 STEP  482 H= 482:00 +CPU= 36.077
  14:42:58 STEP  483 H= 483:00 +CPU= 55.038
  14:43:39 STEP  484 H= 484:00 +CPU= 36.525
  14:44:16 STEP  485 H= 485:00 +CPU= 35.108
  14:45:21 STEP  486 H= 486:00 +CPU= 56.460
  14:46:02 STEP  487 H= 487:00 +CPU= 34.972
  14:46:43 STEP  488 H= 488:00 +CPU= 36.484
  14:47:46 STEP  489 H= 489:00 +CPU= 54.940
  14:48:27 STEP  490 H= 490:00 +CPU= 36.080
  14:49:06 STEP  491 H= 491:00 +CPU= 34.778
  14:50:14 STEP  492 H= 492:00 +CPU= 61.202
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000492
  14:50:53 STEP  493 H= 493:00 +CPU= 35.047
  14:51:36 STEP  494 H= 494:00 +CPU= 36.253
  14:52:37 STEP  495 H= 495:00 +CPU= 55.041
  14:53:18 STEP  496 H= 496:00 +CPU= 35.986
  14:53:59 STEP  497 H= 497:00 +CPU= 34.888
  14:55:04 STEP  498 H= 498:00 +CPU= 56.592
  14:55:44 STEP  499 H= 499:00 +CPU= 35.010
  14:56:24 STEP  500 H= 500:00 +CPU= 36.390
  14:57:27 STEP  501 H= 501:00 +CPU= 55.103
  14:58:06 STEP  502 H= 502:00 +CPU= 35.603
  14:58:47 STEP  503 H= 503:00 +CPU= 36.211
  15:00:04 STEP  504 H= 504:00 +CPU= 65.188
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000492
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_20.zip
Uploading the intermediate file: upload_file_20.zip
  15:00:43 STEP  505 H= 505:00 +CPU= 34.965
  15:01:23 STEP  506 H= 506:00 +CPU= 36.151
  15:02:28 STEP  507 H= 507:00 +CPU= 55.539
  15:03:09 STEP  508 H= 508:00 +CPU= 36.394
  15:03:50 STEP  509 H= 509:00 +CPU= 35.162
  15:04:56 STEP  510 H= 510:00 +CPU= 56.409
  15:05:38 STEP  511 H= 511:00 +CPU= 34.874
  15:06:21 STEP  512 H= 512:00 +CPU= 36.087
  15:07:26 STEP  513 H= 513:00 +CPU= 54.908
  15:08:10 STEP  514 H= 514:00 +CPU= 36.084
  15:08:50 STEP  515 H= 515:00 +CPU= 35.451
  15:10:00 STEP  516 H= 516:00 +CPU= 60.998
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000516
  15:10:43 STEP  517 H= 517:00 +CPU= 35.191
  15:11:24 STEP  518 H= 518:00 +CPU= 36.165
  15:12:27 STEP  519 H= 519:00 +CPU= 54.650
  15:13:11 STEP  520 H= 520:00 +CPU= 36.283
  15:13:54 STEP  521 H= 521:00 +CPU= 34.922
  15:15:00 STEP  522 H= 522:00 +CPU= 56.173
  15:15:40 STEP  523 H= 523:00 +CPU= 34.740
  15:16:23 STEP  524 H= 524:00 +CPU= 36.007
  15:17:31 STEP  525 H= 525:00 +CPU= 55.318
  15:18:17 STEP  526 H= 526:00 +CPU= 36.152
  15:19:00 STEP  527 H= 527:00 +CPU= 36.174
  15:20:18 STEP  528 H= 528:00 +CPU= 65.276
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_21.zip
Uploading the intermediate file: upload_file_21.zip
  15:21:01 STEP  529 H= 529:00 +CPU= 35.274
  15:21:45 STEP  530 H= 530:00 +CPU= 35.968
  15:22:48 STEP  531 H= 531:00 +CPU= 55.507
  15:23:32 STEP  532 H= 532:00 +CPU= 36.095
  15:24:15 STEP  533 H= 533:00 +CPU= 35.308
  15:25:21 STEP  534 H= 534:00 +CPU= 56.839
  15:26:03 STEP  535 H= 535:00 +CPU= 35.277
  15:26:47 STEP  536 H= 536:00 +CPU= 36.202
  15:27:51 STEP  537 H= 537:00 +CPU= 55.389
  15:28:33 STEP  538 H= 538:00 +CPU= 36.089
  15:29:13 STEP  539 H= 539:00 +CPU= 35.271
  15:30:30 STEP  540 H= 540:00 +CPU= 61.353
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000540
  15:31:12 STEP  541 H= 541:00 +CPU= 35.148
  15:31:53 STEP  542 H= 542:00 +CPU= 36.156
  15:33:00 STEP  543 H= 543:00 +CPU= 55.390
  15:33:42 STEP  544 H= 544:00 +CPU= 36.189
  15:34:23 STEP  545 H= 545:00 +CPU= 34.819
  15:35:33 STEP  546 H= 546:00 +CPU= 56.887
  15:36:12 STEP  547 H= 547:00 +CPU= 35.117
  15:36:57 STEP  548 H= 548:00 +CPU= 36.360
  15:38:01 STEP  549 H= 549:00 +CPU= 55.281
  15:38:43 STEP  550 H= 550:00 +CPU= 36.065
  15:39:26 STEP  551 H= 551:00 +CPU= 36.370
  15:40:43 STEP  552 H= 552:00 +CPU= 65.102
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_22.zip
Uploading the intermediate file: upload_file_22.zip
  15:41:25 STEP  553 H= 553:00 +CPU= 34.849
  15:42:07 STEP  554 H= 554:00 +CPU= 36.363
  15:43:11 STEP  555 H= 555:00 +CPU= 55.193
  15:43:51 STEP  556 H= 556:00 +CPU= 36.148
  15:44:32 STEP  557 H= 557:00 +CPU= 35.043
  15:45:38 STEP  558 H= 558:00 +CPU= 56.371
  15:46:18 STEP  559 H= 559:00 +CPU= 35.259
  15:47:02 STEP  560 H= 560:00 +CPU= 36.426
  15:48:09 STEP  561 H= 561:00 +CPU= 55.396
  15:48:51 STEP  562 H= 562:00 +CPU= 36.477
  15:49:34 STEP  563 H= 563:00 +CPU= 35.544
  15:50:46 STEP  564 H= 564:00 +CPU= 61.615
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000564
  15:51:28 STEP  565 H= 565:00 +CPU= 35.249
  15:52:12 STEP  566 H= 566:00 +CPU= 36.210
  15:53:16 STEP  567 H= 567:00 +CPU= 55.448
  15:53:57 STEP  568 H= 568:00 +CPU= 36.109
  15:54:38 STEP  569 H= 569:00 +CPU= 35.112
  15:55:47 STEP  570 H= 570:00 +CPU= 56.550
  15:56:28 STEP  571 H= 571:00 +CPU= 35.432
  15:57:10 STEP  572 H= 572:00 +CPU= 35.672
  15:58:14 STEP  573 H= 573:00 +CPU= 55.164
  15:58:55 STEP  574 H= 574:00 +CPU= 36.332
  15:59:35 STEP  575 H= 575:00 +CPU= 36.088
  16:00:50 STEP  576 H= 576:00 +CPU= 65.306
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_23.zip
Uploading the intermediate file: upload_file_23.zip
  16:01:28 STEP  577 H= 577:00 +CPU= 34.795
  16:02:09 STEP  578 H= 578:00 +CPU= 36.201
  16:03:11 STEP  579 H= 579:00 +CPU= 55.150
  16:03:52 STEP  580 H= 580:00 +CPU= 36.494
  16:04:31 STEP  581 H= 581:00 +CPU= 35.282
  16:05:35 STEP  582 H= 582:00 +CPU= 56.427
  16:06:15 STEP  583 H= 583:00 +CPU= 35.244
  16:06:56 STEP  584 H= 584:00 +CPU= 36.430
  16:07:57 STEP  585 H= 585:00 +CPU= 55.606
  16:08:41 STEP  586 H= 586:00 +CPU= 36.615
  16:09:20 STEP  587 H= 587:00 +CPU= 35.484
  16:10:33 STEP  588 H= 588:00 +CPU= 61.807
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000588
  16:11:15 STEP  589 H= 589:00 +CPU= 35.155
  16:11:58 STEP  590 H= 590:00 +CPU= 36.525
  16:13:05 STEP  591 H= 591:00 +CPU= 55.494
  16:13:47 STEP  592 H= 592:00 +CPU= 36.291
  16:14:31 STEP  593 H= 593:00 +CPU= 35.234
  16:15:39 STEP  594 H= 594:00 +CPU= 56.858
  16:16:20 STEP  595 H= 595:00 +CPU= 35.473
  16:17:05 STEP  596 H= 596:00 +CPU= 36.331
  16:18:13 STEP  597 H= 597:00 +CPU= 55.536
  16:18:55 STEP  598 H= 598:00 +CPU= 36.428
  16:19:40 STEP  599 H= 599:00 +CPU= 36.544
  16:20:59 STEP  600 H= 600:00 +CPU= 65.598
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_24.zip
Uploading the intermediate file: upload_file_24.zip
  16:21:39 STEP  601 H= 601:00 +CPU= 34.811
  16:22:18 STEP  602 H= 602:00 +CPU= 36.158
  16:23:20 STEP  603 H= 603:00 +CPU= 55.580
  16:24:03 STEP  604 H= 604:00 +CPU= 36.699
  16:24:43 STEP  605 H= 605:00 +CPU= 35.248
  16:25:48 STEP  606 H= 606:00 +CPU= 56.937
  16:26:28 STEP  607 H= 607:00 +CPU= 35.437
  16:27:10 STEP  608 H= 608:00 +CPU= 36.175
  16:28:14 STEP  609 H= 609:00 +CPU= 55.500
  16:28:55 STEP  610 H= 610:00 +CPU= 36.522
  16:29:35 STEP  611 H= 611:00 +CPU= 34.975
  16:30:45 STEP  612 H= 612:00 +CPU= 61.363
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000612
  16:31:24 STEP  613 H= 613:00 +CPU= 35.513
  16:32:06 STEP  614 H= 614:00 +CPU= 36.279
  16:33:11 STEP  615 H= 615:00 +CPU= 55.334
  16:33:53 STEP  616 H= 616:00 +CPU= 36.325
  16:34:33 STEP  617 H= 617:00 +CPU= 34.993
  16:35:39 STEP  618 H= 618:00 +CPU= 56.493
  16:36:21 STEP  619 H= 619:00 +CPU= 35.419
  16:37:04 STEP  620 H= 620:00 +CPU= 36.425
  16:38:08 STEP  621 H= 621:00 +CPU= 55.180
  16:38:50 STEP  622 H= 622:00 +CPU= 36.263
  16:39:34 STEP  623 H= 623:00 +CPU= 36.161
  16:40:52 STEP  624 H= 624:00 +CPU= 65.491
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_25.zip
Uploading the intermediate file: upload_file_25.zip
  16:41:32 STEP  625 H= 625:00 +CPU= 35.075
  16:42:13 STEP  626 H= 626:00 +CPU= 36.238
  16:43:18 STEP  627 H= 627:00 +CPU= 55.532
  16:43:59 STEP  628 H= 628:00 +CPU= 36.332
  16:44:39 STEP  629 H= 629:00 +CPU= 35.380
  16:45:43 STEP  630 H= 630:00 +CPU= 56.362
  16:46:23 STEP  631 H= 631:00 +CPU= 35.109
  16:47:06 STEP  632 H= 632:00 +CPU= 36.274
  16:48:05 STEP  633 H= 633:00 +CPU= 55.423
  16:48:47 STEP  634 H= 634:00 +CPU= 36.343
  16:49:28 STEP  635 H= 635:00 +CPU= 35.481
  16:50:41 STEP  636 H= 636:00 +CPU= 61.821
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000636
  16:51:23 STEP  637 H= 637:00 +CPU= 35.259
  16:52:07 STEP  638 H= 638:00 +CPU= 36.609
  16:53:12 STEP  639 H= 639:00 +CPU= 55.689
  16:53:53 STEP  640 H= 640:00 +CPU= 36.062
  16:54:35 STEP  641 H= 641:00 +CPU= 35.054
  16:55:39 STEP  642 H= 642:00 +CPU= 56.852
  16:56:21 STEP  643 H= 643:00 +CPU= 35.270
  16:57:04 STEP  644 H= 644:00 +CPU= 36.129
  16:58:10 STEP  645 H= 645:00 +CPU= 55.180
  16:58:53 STEP  646 H= 646:00 +CPU= 35.995
  16:59:36 STEP  647 H= 647:00 +CPU= 36.443
  17:00:56 STEP  648 H= 648:00 +CPU= 65.404
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_26.zip
Uploading the intermediate file: upload_file_26.zip
  17:01:36 STEP  649 H= 649:00 +CPU= 35.077
  17:02:19 STEP  650 H= 650:00 +CPU= 36.368
  17:03:23 STEP  651 H= 651:00 +CPU= 55.103
  17:04:05 STEP  652 H= 652:00 +CPU= 36.449
  17:04:47 STEP  653 H= 653:00 +CPU= 35.114
  17:05:56 STEP  654 H= 654:00 +CPU= 56.653
  17:06:37 STEP  655 H= 655:00 +CPU= 35.213
  17:07:20 STEP  656 H= 656:00 +CPU= 36.393
  17:08:25 STEP  657 H= 657:00 +CPU= 55.314
  17:09:08 STEP  658 H= 658:00 +CPU= 36.221
  17:09:48 STEP  659 H= 659:00 +CPU= 34.970
  17:11:01 STEP  660 H= 660:00 +CPU= 61.182
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000660
  17:11:41 STEP  661 H= 661:00 +CPU= 34.990
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0645
(argv4) batchid: 984
(argv5) wuid: 12201289
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e6e91692925f5de22f5ebe0bb511cdbb to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0645_2015050100_123_984_12201289.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0645_2015050100_123_984_12201289.zip
ic_ancil_file: ic_ancil_12201289
ifsdata_file: ifsdata_12201289
climate_data_file: clim_data_12201289
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_08e5ec787aa5a70a924ddc2961dca023 to: /var/lib/boinc-client/slots/10/ic_ancil_12201289.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12201289.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_953088beaaa06ad7fa9708622fbf896d to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12201289.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12201289.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12201289.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12201289.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12201289.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12201289.xml
-- Model is restarting --
Adjusting last_iter, 662, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12201289.xml
last_cpu_time: 41630
upload_file_number: 27
last_iter: 647
last_upload: 2332800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309
The child process has been launched with process id: 9882
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tourte:1:1:9882:9882] [20230121:172410:1674318250:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:25:29 STEP  648 H= 648:00 +CPU= 62.625
  17:26:09 STEP  649 H= 649:00 +CPU= 35.139
  17:26:52 STEP  650 H= 650:00 +CPU= 36.443
  17:27:58 STEP  651 H= 651:00 +CPU= 55.231
  17:28:38 STEP  652 H= 652:00 +CPU= 36.073
  17:29:22 STEP  653 H= 653:00 +CPU= 35.121
  17:30:29 STEP  654 H= 654:00 +CPU= 56.617
  17:31:12 STEP  655 H= 655:00 +CPU= 34.904
  17:31:55 STEP  656 H= 656:00 +CPU= 36.141
  17:32:58 STEP  657 H= 657:00 +CPU= 55.029
  17:33:41 STEP  658 H= 658:00 +CPU= 36.211
  17:34:23 STEP  659 H= 659:00 +CPU= 35.363
  17:35:35 STEP  660 H= 660:00 +CPU= 61.087
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000660
  17:36:16 STEP  661 H= 661:00 +CPU= 35.124
  17:36:59 STEP  662 H= 662:00 +CPU= 36.345
  17:38:03 STEP  663 H= 663:00 +CPU= 55.300
  17:38:46 STEP  664 H= 664:00 +CPU= 36.246
  17:39:27 STEP  665 H= 665:00 +CPU= 35.251
  17:40:34 STEP  666 H= 666:00 +CPU= 57.067
  17:41:15 STEP  667 H= 667:00 +CPU= 34.944
  17:41:58 STEP  668 H= 668:00 +CPU= 36.087
  17:43:01 STEP  669 H= 669:00 +CPU= 55.225
  17:43:44 STEP  670 H= 670:00 +CPU= 36.365
  17:44:25 STEP  671 H= 671:00 +CPU= 36.142
  17:45:42 STEP  672 H= 672:00 +CPU= 65.339
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_27.zip
Uploading the intermediate file: upload_file_27.zip
  17:46:25 STEP  673 H= 673:00 +CPU= 34.971
  17:47:06 STEP  674 H= 674:00 +CPU= 35.914
  17:48:10 STEP  675 H= 675:00 +CPU= 55.532
  17:48:50 STEP  676 H= 676:00 +CPU= 36.029
  17:49:30 STEP  677 H= 677:00 +CPU= 34.915
  17:50:35 STEP  678 H= 678:00 +CPU= 56.522
  17:51:17 STEP  679 H= 679:00 +CPU= 35.060
  17:51:59 STEP  680 H= 680:00 +CPU= 36.061
  17:53:06 STEP  681 H= 681:00 +CPU= 55.384
  17:53:50 STEP  682 H= 682:00 +CPU= 36.696
  17:54:30 STEP  683 H= 683:00 +CPU= 35.227
  17:55:40 STEP  684 H= 684:00 +CPU= 61.986
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000684
  17:56:20 STEP  685 H= 685:00 +CPU= 35.304
  17:57:02 STEP  686 H= 686:00 +CPU= 36.213
  17:58:08 STEP  687 H= 687:00 +CPU= 55.416
  17:58:50 STEP  688 H= 688:00 +CPU= 36.233
  17:59:32 STEP  689 H= 689:00 +CPU= 35.035
  18:00:40 STEP  690 H= 690:00 +CPU= 56.599
  18:01:21 STEP  691 H= 691:00 +CPU= 35.193
  18:02:05 STEP  692 H= 692:00 +CPU= 36.239
  18:03:12 STEP  693 H= 693:00 +CPU= 55.845
  18:03:57 STEP  694 H= 694:00 +CPU= 36.539
  18:04:42 STEP  695 H= 695:00 +CPU= 36.297
  18:06:00 STEP  696 H= 696:00 +CPU= 65.524
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_28.zip
Uploading the intermediate file: upload_file_28.zip
  18:06:41 STEP  697 H= 697:00 +CPU= 35.077
  18:07:25 STEP  698 H= 698:00 +CPU= 36.233
  18:08:32 STEP  699 H= 699:00 +CPU= 55.812
  18:09:15 STEP  700 H= 700:00 +CPU= 36.682
  18:09:56 STEP  701 H= 701:00 +CPU= 35.108
  18:11:05 STEP  702 H= 702:00 +CPU= 56.891
  18:11:48 STEP  703 H= 703:00 +CPU= 35.306
  18:12:31 STEP  704 H= 704:00 +CPU= 36.245
  18:13:36 STEP  705 H= 705:00 +CPU= 55.643
  18:14:20 STEP  706 H= 706:00 +CPU= 36.673
  18:15:02 STEP  707 H= 707:00 +CPU= 35.278
  18:16:16 STEP  708 H= 708:00 +CPU= 61.780
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000708
  18:17:00 STEP  709 H= 709:00 +CPU= 35.251
  18:17:43 STEP  710 H= 710:00 +CPU= 36.174
  18:18:48 STEP  711 H= 711:00 +CPU= 55.487
  18:19:31 STEP  712 H= 712:00 +CPU= 36.447
  18:20:12 STEP  713 H= 713:00 +CPU= 35.249
  18:21:19 STEP  714 H= 714:00 +CPU= 57.037
  18:22:01 STEP  715 H= 715:00 +CPU= 34.974
  18:22:43 STEP  716 H= 716:00 +CPU= 35.978
  18:23:47 STEP  717 H= 717:00 +CPU= 54.836
  18:24:30 STEP  718 H= 718:00 +CPU= 36.189
  18:25:13 STEP  719 H= 719:00 +CPU= 36.068
  18:26:30 STEP  720 H= 720:00 +CPU= 64.946
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_29.zip
Uploading the intermediate file: upload_file_29.zip
  18:27:11 STEP  721 H= 721:00 +CPU= 34.907
  18:27:52 STEP  722 H= 722:00 +CPU= 35.980
  18:28:57 STEP  723 H= 723:00 +CPU= 54.998
  18:29:40 STEP  724 H= 724:00 +CPU= 36.222
  18:30:22 STEP  725 H= 725:00 +CPU= 35.020
  18:31:28 STEP  726 H= 726:00 +CPU= 56.490
  18:32:09 STEP  727 H= 727:00 +CPU= 35.001
  18:32:52 STEP  728 H= 728:00 +CPU= 36.315
  18:33:57 STEP  729 H= 729:00 +CPU= 54.911
  18:34:40 STEP  730 H= 730:00 +CPU= 36.234
  18:35:23 STEP  731 H= 731:00 +CPU= 35.048
  18:36:35 STEP  732 H= 732:00 +CPU= 61.398
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000732
  18:37:17 STEP  733 H= 733:00 +CPU= 35.210
  18:37:59 STEP  734 H= 734:00 +CPU= 36.255
  18:39:07 STEP  735 H= 735:00 +CPU= 55.449
  18:39:51 STEP  736 H= 736:00 +CPU= 36.331
  18:40:34 STEP  737 H= 737:00 +CPU= 35.056
  18:41:38 STEP  738 H= 738:00 +CPU= 56.487
  18:42:19 STEP  739 H= 739:00 +CPU= 34.918
  18:43:01 STEP  740 H= 740:00 +CPU= 35.706
  18:44:05 STEP  741 H= 741:00 +CPU= 54.891
  18:44:46 STEP  742 H= 742:00 +CPU= 35.769
  18:45:27 STEP  743 H= 743:00 +CPU= 36.076
  18:46:42 STEP  744 H= 744:00 +CPU= 65.025
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_30.zip
Uploading the intermediate file: upload_file_30.zip
  18:47:20 STEP  745 H= 745:00 +CPU= 34.721
  18:48:01 STEP  746 H= 746:00 +CPU= 35.898
  18:49:02 STEP  747 H= 747:00 +CPU= 54.698
  18:49:43 STEP  748 H= 748:00 +CPU= 35.878
  18:50:22 STEP  749 H= 749:00 +CPU= 34.749
  18:51:24 STEP  750 H= 750:00 +CPU= 56.253
  18:52:03 STEP  751 H= 751:00 +CPU= 34.950
  18:52:42 STEP  752 H= 752:00 +CPU= 35.712
  18:53:42 STEP  753 H= 753:00 +CPU= 54.536
  18:54:22 STEP  754 H= 754:00 +CPU= 35.377
  18:55:01 STEP  755 H= 755:00 +CPU= 34.254
  18:56:09 STEP  756 H= 756:00 +CPU= 60.671
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000756
  18:56:49 STEP  757 H= 757:00 +CPU= 34.489
  18:57:28 STEP  758 H= 758:00 +CPU= 35.421
  18:58:29 STEP  759 H= 759:00 +CPU= 53.913
  18:59:09 STEP  760 H= 760:00 +CPU= 35.617
  18:59:49 STEP  761 H= 761:00 +CPU= 34.607
  19:00:52 STEP  762 H= 762:00 +CPU= 55.801
  19:01:32 STEP  763 H= 763:00 +CPU= 34.654
  19:02:11 STEP  764 H= 764:00 +CPU= 35.551
  19:03:12 STEP  765 H= 765:00 +CPU= 54.502
  19:03:52 STEP  766 H= 766:00 +CPU= 35.276
  19:04:32 STEP  767 H= 767:00 +CPU= 35.203
  19:05:46 STEP  768 H= 768:00 +CPU= 64.461
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201289/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0645_2015050100_123_984_12201289_0_r847907309_31.zip
Uploading the intermediate file: upload_file_31.zip
  19:06:26 STEP  769 H= 769:00 +CPU= 34.337
  19:07:08 STEP  770 H= 770:00 +CPU= 35.466
  19:08:14 STEP  771 H= 771:00 +CPU= 54.659
  19:08:56 STEP  772 H= 772:00 +CPU= 36.136
  19:09:35 STEP  773 H= 773:00 +CPU= 35.050
  19:10:42 STEP  774 H= 774:00 +CPU= 56.313
  19:11:21 STEP  775 H= 775:00 +CPU= 34.573
  19:12:02 STEP  776 H= 776:00 +CPU= 36.094
  19:13:03 STEP  777 H= 777:00 +CPU= 54.509
  19:13:45 STEP  778 H= 778:00 +CPU= 35.962
  19:14:23 STEP  779 H= 779:00 +CPU= 34.841
  19:15:34 STEP  780 H= 780:00 +CPU= 60.442
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+000780
  19:16:16 STEP  781 H= 781:00 +CPU= 34.690
  19:16:59 STEP  782 H= 782:00 +CPU= 35.896
  19:18:00 STEP  783 H= 783:00 +CPU= 55.079
  19:18:42 STEP  784 H= 784:00 +CPU= 35.753
  19:19:24 STEP  785 H= 785:00 +CPU= 34.666
  19:20:29 STEP  786 H= 786:00 +CPU= 56.568
  19:21:10 STEP  787 H= 787:00 +CPU= 34.416
  19:21:51 STEP  788 H= 788:00 +CPU= 35.737
  19:22:54 STEP  789 H= 789:00 +CPU= 54.642
  19:23:35 STEP  790 H= 790:00 +CPU= 35.491
  19:24:17 STEP  791 H= 791:00 +CPU= 35.657
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net