climateprediction.net home page
Task 22262465

Task 22262465

Name oifs_43r3_ps_0498_1991050100_123_960_12177142_1
Workunit 12177142
Created 21 Dec 2022, 23:17:34 UTC
Sent 14 Jan 2023, 9:48:56 UTC
Report deadline 13 Feb 2023, 9:48:56 UTC
Received 15 Jan 2023, 16:09:02 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1515226
Run time 21 hours 42 min 49 sec
CPU time 21 hours 20 min 11 sec
Validate state Invalid
Credit 13,305.60
Device peak FLOPS 4.58 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,206.63 MB
Peak swap size 4,977.98 MB
Peak disk usage 2,223.33 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
 zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_101.zip
Uploading the intermediate file: upload_file_101.zip
  01:44:35 STEP 2449 H=2449:00 +CPU= 22.078
  01:44:58 STEP 2450 H=2450:00 +CPU= 22.854
  01:45:32 STEP 2451 H=2451:00 +CPU= 34.536
  01:45:55 STEP 2452 H=2452:00 +CPU= 22.861
  01:46:17 STEP 2453 H=2453:00 +CPU= 22.139
  01:46:53 STEP 2454 H=2454:00 +CPU= 35.467
  01:47:15 STEP 2455 H=2455:00 +CPU= 22.125
  01:47:38 STEP 2456 H=2456:00 +CPU= 22.898
  01:48:13 STEP 2457 H=2457:00 +CPU= 34.630
  01:48:35 STEP 2458 H=2458:00 +CPU= 22.858
  01:48:58 STEP 2459 H=2459:00 +CPU= 22.132
  01:49:36 STEP 2460 H=2460:00 +CPU= 38.542
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002460
  01:49:58 STEP 2461 H=2461:00 +CPU= 22.046
  01:50:21 STEP 2462 H=2462:00 +CPU= 22.858
  01:50:56 STEP 2463 H=2463:00 +CPU= 34.608
  01:51:19 STEP 2464 H=2464:00 +CPU= 22.918
  01:51:41 STEP 2465 H=2465:00 +CPU= 22.136
  01:52:16 STEP 2466 H=2466:00 +CPU= 35.535
  01:52:38 STEP 2467 H=2467:00 +CPU= 22.103
  01:53:01 STEP 2468 H=2468:00 +CPU= 22.889
  01:53:36 STEP 2469 H=2469:00 +CPU= 34.635
  01:53:59 STEP 2470 H=2470:00 +CPU= 22.840
  01:54:22 STEP 2471 H=2471:00 +CPU= 22.784
  01:55:07 STEP 2472 H=2472:00 +CPU= 40.242
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_102.zip
Uploading the intermediate file: upload_file_102.zip
  01:55:29 STEP 2473 H=2473:00 +CPU= 21.967
  01:55:52 STEP 2474 H=2474:00 +CPU= 22.654
  01:56:26 STEP 2475 H=2475:00 +CPU= 34.449
  01:56:49 STEP 2476 H=2476:00 +CPU= 22.693
  01:57:11 STEP 2477 H=2477:00 +CPU= 21.939
  01:57:46 STEP 2478 H=2478:00 +CPU= 35.273
  01:58:08 STEP 2479 H=2479:00 +CPU= 21.977
  01:58:31 STEP 2480 H=2480:00 +CPU= 22.651
  01:59:05 STEP 2481 H=2481:00 +CPU= 34.414
  01:59:28 STEP 2482 H=2482:00 +CPU= 22.722
  01:59:50 STEP 2483 H=2483:00 +CPU= 21.976
  02:00:28 STEP 2484 H=2484:00 +CPU= 38.383
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002484
  02:00:50 STEP 2485 H=2485:00 +CPU= 21.921
  02:01:13 STEP 2486 H=2486:00 +CPU= 22.729
  02:01:48 STEP 2487 H=2487:00 +CPU= 34.406
  02:02:10 STEP 2488 H=2488:00 +CPU= 22.652
  02:02:32 STEP 2489 H=2489:00 +CPU= 21.942
  02:03:07 STEP 2490 H=2490:00 +CPU= 35.230
  02:03:29 STEP 2491 H=2491:00 +CPU= 21.981
  02:03:52 STEP 2492 H=2492:00 +CPU= 22.667
  02:04:27 STEP 2493 H=2493:00 +CPU= 34.389
  02:04:49 STEP 2494 H=2494:00 +CPU= 22.661
  02:05:12 STEP 2495 H=2495:00 +CPU= 22.614
  02:05:56 STEP 2496 H=2496:00 +CPU= 40.525
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_103.zip
Uploading the intermediate file: upload_file_103.zip
  02:06:18 STEP 2497 H=2497:00 +CPU= 21.915
  02:06:41 STEP 2498 H=2498:00 +CPU= 22.325
  02:07:14 STEP 2499 H=2499:00 +CPU= 33.629
  02:07:36 STEP 2500 H=2500:00 +CPU= 22.093
  02:07:58 STEP 2501 H=2501:00 +CPU= 21.388
  02:08:32 STEP 2502 H=2502:00 +CPU= 34.468
  02:08:54 STEP 2503 H=2503:00 +CPU= 21.403
  02:09:16 STEP 2504 H=2504:00 +CPU= 22.143
  02:09:50 STEP 2505 H=2505:00 +CPU= 33.661
  02:10:12 STEP 2506 H=2506:00 +CPU= 22.124
  02:10:33 STEP 2507 H=2507:00 +CPU= 21.408
  02:11:11 STEP 2508 H=2508:00 +CPU= 37.439
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002508
  02:11:32 STEP 2509 H=2509:00 +CPU= 21.390
  02:11:54 STEP 2510 H=2510:00 +CPU= 22.118
  02:12:28 STEP 2511 H=2511:00 +CPU= 33.636
  02:12:50 STEP 2512 H=2512:00 +CPU= 22.087
  02:13:11 STEP 2513 H=2513:00 +CPU= 21.356
  02:13:46 STEP 2514 H=2514:00 +CPU= 34.462
  02:14:07 STEP 2515 H=2515:00 +CPU= 21.356
  02:14:29 STEP 2516 H=2516:00 +CPU= 22.095
  02:15:03 STEP 2517 H=2517:00 +CPU= 33.601
  02:15:25 STEP 2518 H=2518:00 +CPU= 22.025
  02:15:47 STEP 2519 H=2519:00 +CPU= 21.999
  02:16:31 STEP 2520 H=2520:00 +CPU= 39.681
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_104.zip
Uploading the intermediate file: upload_file_104.zip
  02:16:52 STEP 2521 H=2521:00 +CPU= 21.495
  02:17:15 STEP 2522 H=2522:00 +CPU= 22.185
  02:17:49 STEP 2523 H=2523:00 +CPU= 33.797
  02:18:11 STEP 2524 H=2524:00 +CPU= 22.222
  02:18:33 STEP 2525 H=2525:00 +CPU= 21.501
  02:19:07 STEP 2526 H=2526:00 +CPU= 34.643
  02:19:29 STEP 2527 H=2527:00 +CPU= 21.505
  02:19:51 STEP 2528 H=2528:00 +CPU= 22.252
  02:20:25 STEP 2529 H=2529:00 +CPU= 33.805
  02:20:47 STEP 2530 H=2530:00 +CPU= 22.240
  02:21:09 STEP 2531 H=2531:00 +CPU= 21.499
  02:21:47 STEP 2532 H=2532:00 +CPU= 37.621
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002532
  02:22:08 STEP 2533 H=2533:00 +CPU= 21.517
  02:22:30 STEP 2534 H=2534:00 +CPU= 22.270
  02:23:04 STEP 2535 H=2535:00 +CPU= 33.817
  02:23:27 STEP 2536 H=2536:00 +CPU= 22.228
  02:23:48 STEP 2537 H=2537:00 +CPU= 21.479
  02:24:23 STEP 2538 H=2538:00 +CPU= 34.635
  02:24:45 STEP 2539 H=2539:00 +CPU= 21.629
  02:25:07 STEP 2540 H=2540:00 +CPU= 22.285
  02:25:41 STEP 2541 H=2541:00 +CPU= 33.853
  02:26:03 STEP 2542 H=2542:00 +CPU= 22.241
  02:26:25 STEP 2543 H=2543:00 +CPU= 22.197
  02:27:10 STEP 2544 H=2544:00 +CPU= 39.784
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_105.zip
Uploading the intermediate file: upload_file_105.zip
  02:27:31 STEP 2545 H=2545:00 +CPU= 21.342
  02:27:53 STEP 2546 H=2546:00 +CPU= 22.019
  02:28:27 STEP 2547 H=2547:00 +CPU= 33.534
  02:28:49 STEP 2548 H=2548:00 +CPU= 21.997
  02:29:10 STEP 2549 H=2549:00 +CPU= 21.337
  02:29:45 STEP 2550 H=2550:00 +CPU= 34.383
  02:30:06 STEP 2551 H=2551:00 +CPU= 21.335
  02:30:28 STEP 2552 H=2552:00 +CPU= 22.055
  02:31:02 STEP 2553 H=2553:00 +CPU= 33.576
  02:31:24 STEP 2554 H=2554:00 +CPU= 22.047
  02:31:45 STEP 2555 H=2555:00 +CPU= 21.337
  02:32:22 STEP 2556 H=2556:00 +CPU= 37.333
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002556
  02:32:44 STEP 2557 H=2557:00 +CPU= 21.368
  02:33:06 STEP 2558 H=2558:00 +CPU= 22.062
  02:33:40 STEP 2559 H=2559:00 +CPU= 33.588
  02:34:02 STEP 2560 H=2560:00 +CPU= 22.053
  02:34:23 STEP 2561 H=2561:00 +CPU= 21.323
  02:34:57 STEP 2562 H=2562:00 +CPU= 34.331
  02:35:19 STEP 2563 H=2563:00 +CPU= 21.351
  02:35:42 STEP 2564 H=2564:00 +CPU= 23.671
  02:36:16 STEP 2565 H=2565:00 +CPU= 33.467
  02:36:39 STEP 2566 H=2566:00 +CPU= 22.807
  02:37:01 STEP 2567 H=2567:00 +CPU= 22.486
  02:37:45 STEP 2568 H=2568:00 +CPU= 39.866
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_106.zip
Uploading the intermediate file: upload_file_106.zip
  02:38:07 STEP 2569 H=2569:00 +CPU= 21.509
  02:38:29 STEP 2570 H=2570:00 +CPU= 22.202
  02:39:03 STEP 2571 H=2571:00 +CPU= 33.738
  02:39:25 STEP 2572 H=2572:00 +CPU= 22.192
  02:39:46 STEP 2573 H=2573:00 +CPU= 21.474
  02:40:21 STEP 2574 H=2574:00 +CPU= 34.587
  02:40:42 STEP 2575 H=2575:00 +CPU= 21.474
  02:41:05 STEP 2576 H=2576:00 +CPU= 22.187
  02:41:38 STEP 2577 H=2577:00 +CPU= 33.723
  02:42:01 STEP 2578 H=2578:00 +CPU= 22.236
  02:42:22 STEP 2579 H=2579:00 +CPU= 21.489
  02:43:00 STEP 2580 H=2580:00 +CPU= 37.514
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002580
  02:43:21 STEP 2581 H=2581:00 +CPU= 21.487
  02:43:44 STEP 2582 H=2582:00 +CPU= 22.183
  02:44:17 STEP 2583 H=2583:00 +CPU= 33.677
  02:44:40 STEP 2584 H=2584:00 +CPU= 22.206
  02:45:01 STEP 2585 H=2585:00 +CPU= 21.468
  02:45:36 STEP 2586 H=2586:00 +CPU= 34.510
  02:45:57 STEP 2587 H=2587:00 +CPU= 21.427
  02:46:19 STEP 2588 H=2588:00 +CPU= 22.136
  02:46:53 STEP 2589 H=2589:00 +CPU= 33.643
  02:47:15 STEP 2590 H=2590:00 +CPU= 22.127
  02:47:37 STEP 2591 H=2591:00 +CPU= 22.058
  02:48:21 STEP 2592 H=2592:00 +CPU= 39.537
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_107.zip
Uploading the intermediate file: upload_file_107.zip
  02:48:42 STEP 2593 H=2593:00 +CPU= 21.394
  02:49:05 STEP 2594 H=2594:00 +CPU= 22.396
  02:49:39 STEP 2595 H=2595:00 +CPU= 34.320
  02:50:02 STEP 2596 H=2596:00 +CPU= 22.728
  02:50:24 STEP 2597 H=2597:00 +CPU= 22.010
  02:51:00 STEP 2598 H=2598:00 +CPU= 35.246
  02:51:22 STEP 2599 H=2599:00 +CPU= 22.019
  02:51:45 STEP 2600 H=2600:00 +CPU= 22.716
  02:52:19 STEP 2601 H=2601:00 +CPU= 34.476
  02:52:42 STEP 2602 H=2602:00 +CPU= 22.837
  02:53:04 STEP 2603 H=2603:00 +CPU= 22.112
  02:53:43 STEP 2604 H=2604:00 +CPU= 38.546
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002604
  02:54:05 STEP 2605 H=2605:00 +CPU= 22.131
  02:54:28 STEP 2606 H=2606:00 +CPU= 22.878
  02:55:02 STEP 2607 H=2607:00 +CPU= 34.630
  02:55:25 STEP 2608 H=2608:00 +CPU= 22.896
  02:55:47 STEP 2609 H=2609:00 +CPU= 22.113
  02:56:23 STEP 2610 H=2610:00 +CPU= 35.447
  02:56:45 STEP 2611 H=2611:00 +CPU= 22.112
  02:57:08 STEP 2612 H=2612:00 +CPU= 22.822
  02:57:42 STEP 2613 H=2613:00 +CPU= 34.530
  02:58:05 STEP 2614 H=2614:00 +CPU= 22.791
  02:58:28 STEP 2615 H=2615:00 +CPU= 22.735
  02:59:14 STEP 2616 H=2616:00 +CPU= 40.790
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_108.zip
Uploading the intermediate file: upload_file_108.zip
  02:59:36 STEP 2617 H=2617:00 +CPU= 22.048
  02:59:59 STEP 2618 H=2618:00 +CPU= 22.849
  03:00:33 STEP 2619 H=2619:00 +CPU= 34.615
  03:00:56 STEP 2620 H=2620:00 +CPU= 22.837
  03:01:19 STEP 2621 H=2621:00 +CPU= 22.118
  03:01:54 STEP 2622 H=2622:00 +CPU= 35.496
  03:02:16 STEP 2623 H=2623:00 +CPU= 22.111
  03:02:39 STEP 2624 H=2624:00 +CPU= 22.899
  03:03:14 STEP 2625 H=2625:00 +CPU= 34.663
  03:03:37 STEP 2626 H=2626:00 +CPU= 22.873
  03:03:59 STEP 2627 H=2627:00 +CPU= 22.120
  03:04:38 STEP 2628 H=2628:00 +CPU= 38.619
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002628
  03:05:00 STEP 2629 H=2629:00 +CPU= 22.109
  03:05:23 STEP 2630 H=2630:00 +CPU= 22.804
  03:05:58 STEP 2631 H=2631:00 +CPU= 34.622
  03:06:21 STEP 2632 H=2632:00 +CPU= 22.867
  03:06:43 STEP 2633 H=2633:00 +CPU= 22.095
  03:07:18 STEP 2634 H=2634:00 +CPU= 35.446
  03:07:40 STEP 2635 H=2635:00 +CPU= 22.110
  03:08:03 STEP 2636 H=2636:00 +CPU= 22.801
  03:08:38 STEP 2637 H=2637:00 +CPU= 34.647
  03:09:01 STEP 2638 H=2638:00 +CPU= 22.799
  03:09:24 STEP 2639 H=2639:00 +CPU= 22.781
  03:10:08 STEP 2640 H=2640:00 +CPU= 40.499
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_109.zip
Uploading the intermediate file: upload_file_109.zip
  03:10:30 STEP 2641 H=2641:00 +CPU= 21.872
Uploading trickle at timestep: 9504000
  03:10:53 STEP 2642 H=2642:00 +CPU= 22.638
  03:11:28 STEP 2643 H=2643:00 +CPU= 34.394
  03:11:50 STEP 2644 H=2644:00 +CPU= 22.658
  03:12:12 STEP 2645 H=2645:00 +CPU= 21.926
  03:12:48 STEP 2646 H=2646:00 +CPU= 35.223
  03:13:10 STEP 2647 H=2647:00 +CPU= 21.971
  03:13:33 STEP 2648 H=2648:00 +CPU= 22.690
  03:14:07 STEP 2649 H=2649:00 +CPU= 34.371
  03:14:30 STEP 2650 H=2650:00 +CPU= 22.727
  03:14:52 STEP 2651 H=2651:00 +CPU= 21.992
  03:15:30 STEP 2652 H=2652:00 +CPU= 38.422
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002652
  03:15:52 STEP 2653 H=2653:00 +CPU= 21.917
  03:16:15 STEP 2654 H=2654:00 +CPU= 22.738
  03:16:50 STEP 2655 H=2655:00 +CPU= 34.432
  03:17:12 STEP 2656 H=2656:00 +CPU= 22.638
  03:17:34 STEP 2657 H=2657:00 +CPU= 21.928
  03:18:10 STEP 2658 H=2658:00 +CPU= 35.245
  03:18:32 STEP 2659 H=2659:00 +CPU= 21.915
  03:18:54 STEP 2660 H=2660:00 +CPU= 22.698
  03:19:29 STEP 2661 H=2661:00 +CPU= 34.363
  03:19:52 STEP 2662 H=2662:00 +CPU= 22.653
  03:20:14 STEP 2663 H=2663:00 +CPU= 22.631
  03:20:59 STEP 2664 H=2664:00 +CPU= 40.583
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_110.zip
Uploading the intermediate file: upload_file_110.zip
  03:21:22 STEP 2665 H=2665:00 +CPU= 22.026
  03:21:44 STEP 2666 H=2666:00 +CPU= 22.850
  03:22:19 STEP 2667 H=2667:00 +CPU= 34.649
  03:22:42 STEP 2668 H=2668:00 +CPU= 22.833
  03:23:04 STEP 2669 H=2669:00 +CPU= 22.080
  03:23:39 STEP 2670 H=2670:00 +CPU= 35.469
  03:24:02 STEP 2671 H=2671:00 +CPU= 22.076
  03:24:24 STEP 2672 H=2672:00 +CPU= 22.860
  03:24:59 STEP 2673 H=2673:00 +CPU= 34.608
  03:25:22 STEP 2674 H=2674:00 +CPU= 22.851
  03:25:44 STEP 2675 H=2675:00 +CPU= 22.172
  03:26:23 STEP 2676 H=2676:00 +CPU= 38.626
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002676
  03:26:45 STEP 2677 H=2677:00 +CPU= 22.114
  03:27:08 STEP 2678 H=2678:00 +CPU= 22.847
  03:27:43 STEP 2679 H=2679:00 +CPU= 34.699
  03:28:05 STEP 2680 H=2680:00 +CPU= 22.897
  03:28:28 STEP 2681 H=2681:00 +CPU= 22.154
  03:29:03 STEP 2682 H=2682:00 +CPU= 35.438
  03:29:25 STEP 2683 H=2683:00 +CPU= 22.112
  03:29:48 STEP 2684 H=2684:00 +CPU= 22.832
  03:30:23 STEP 2685 H=2685:00 +CPU= 34.622
  03:30:46 STEP 2686 H=2686:00 +CPU= 22.811
  03:31:08 STEP 2687 H=2687:00 +CPU= 22.831
  03:31:54 STEP 2688 H=2688:00 +CPU= 40.914
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_111.zip
Uploading the intermediate file: upload_file_111.zip
  03:32:16 STEP 2689 H=2689:00 +CPU= 21.665
  03:32:38 STEP 2690 H=2690:00 +CPU= 22.422
  03:33:13 STEP 2691 H=2691:00 +CPU= 34.441
  03:33:35 STEP 2692 H=2692:00 +CPU= 22.699
  03:33:57 STEP 2693 H=2693:00 +CPU= 21.926
  03:34:33 STEP 2694 H=2694:00 +CPU= 35.330
  03:34:55 STEP 2695 H=2695:00 +CPU= 22.001
  03:35:17 STEP 2696 H=2696:00 +CPU= 22.722
  03:35:52 STEP 2697 H=2697:00 +CPU= 34.494
  03:36:15 STEP 2698 H=2698:00 +CPU= 22.766
  03:36:37 STEP 2699 H=2699:00 +CPU= 21.996
  03:37:15 STEP 2700 H=2700:00 +CPU= 38.432
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002700
  03:37:37 STEP 2701 H=2701:00 +CPU= 21.994
  03:38:00 STEP 2702 H=2702:00 +CPU= 22.774
  03:38:35 STEP 2703 H=2703:00 +CPU= 34.428
  03:38:57 STEP 2704 H=2704:00 +CPU= 22.762
  03:39:19 STEP 2705 H=2705:00 +CPU= 21.956
  03:39:55 STEP 2706 H=2706:00 +CPU= 35.253
  03:40:17 STEP 2707 H=2707:00 +CPU= 22.004
  03:40:39 STEP 2708 H=2708:00 +CPU= 22.708
  03:41:14 STEP 2709 H=2709:00 +CPU= 34.441
  03:41:36 STEP 2710 H=2710:00 +CPU= 22.653
  03:41:59 STEP 2711 H=2711:00 +CPU= 22.600
  03:42:44 STEP 2712 H=2712:00 +CPU= 40.635
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_112.zip
Uploading the intermediate file: upload_file_112.zip
  03:43:05 STEP 2713 H=2713:00 +CPU= 21.436
  03:43:27 STEP 2714 H=2714:00 +CPU= 22.124
  03:44:01 STEP 2715 H=2715:00 +CPU= 33.702
  03:44:23 STEP 2716 H=2716:00 +CPU= 22.130
  03:44:45 STEP 2717 H=2717:00 +CPU= 21.424
  03:45:19 STEP 2718 H=2718:00 +CPU= 34.545
  03:45:41 STEP 2719 H=2719:00 +CPU= 21.441
  03:46:03 STEP 2720 H=2720:00 +CPU= 22.168
  03:46:37 STEP 2721 H=2721:00 +CPU= 33.711
  03:46:59 STEP 2722 H=2722:00 +CPU= 22.159
  03:47:20 STEP 2723 H=2723:00 +CPU= 21.464
  03:47:58 STEP 2724 H=2724:00 +CPU= 37.495
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002724
  03:48:19 STEP 2725 H=2725:00 +CPU= 21.452
  03:48:42 STEP 2726 H=2726:00 +CPU= 22.160
  03:49:15 STEP 2727 H=2727:00 +CPU= 33.682
  03:49:37 STEP 2728 H=2728:00 +CPU= 22.146
  03:49:59 STEP 2729 H=2729:00 +CPU= 21.420
  03:50:33 STEP 2730 H=2730:00 +CPU= 34.503
  03:50:55 STEP 2731 H=2731:00 +CPU= 21.427
  03:51:17 STEP 2732 H=2732:00 +CPU= 22.118
  03:51:51 STEP 2733 H=2733:00 +CPU= 33.648
  03:52:13 STEP 2734 H=2734:00 +CPU= 22.094
  03:52:35 STEP 2735 H=2735:00 +CPU= 22.055
  03:53:19 STEP 2736 H=2736:00 +CPU= 39.800
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_113.zip
Uploading the intermediate file: upload_file_113.zip
  03:53:41 STEP 2737 H=2737:00 +CPU= 21.514
  03:54:03 STEP 2738 H=2738:00 +CPU= 22.225
  03:54:37 STEP 2739 H=2739:00 +CPU= 33.792
  03:54:59 STEP 2740 H=2740:00 +CPU= 22.221
  03:55:21 STEP 2741 H=2741:00 +CPU= 21.523
  03:55:55 STEP 2742 H=2742:00 +CPU= 34.685
  03:56:17 STEP 2743 H=2743:00 +CPU= 21.538
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0498
(argv4) batchid: 960
(argv5) wuid: 12177142
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8866c4f3be9800b3e99c52171738e129 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0498_1991050100_123_960_12177142.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0498_1991050100_123_960_12177142.zip
ic_ancil_file: ic_ancil_12177142
ifsdata_file: ifsdata_12177142
climate_data_file: clim_data_12177142
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8d32566c8244c3ab6d6c331a922f27b4 to: /var/lib/boinc-client/slots/1/ic_ancil_12177142.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12177142.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5966736dce4deb7f34e553e64f3527af to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12177142.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12177142.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12177142.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12177142.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12177142.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12177142.xml
-- Model is restarting --
Adjusting last_iter, 2744, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12177142.xml
last_cpu_time: 72823
upload_file_number: 114
last_iter: 2735
last_upload: 9849600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734
The child process has been launched with process id: 19282
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19282:19282] [20230115:045723:1673787443:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:58:10 STEP 2736 H=2736:00 +CPU= 39.167
  04:58:32 STEP 2737 H=2737:00 +CPU= 22.102
  04:58:55 STEP 2738 H=2738:00 +CPU= 22.588
  04:59:29 STEP 2739 H=2739:00 +CPU= 34.477
  04:59:54 STEP 2740 H=2740:00 +CPU= 22.052
  05:00:31 STEP 2741 H=2741:00 +CPU= 21.813
  05:01:10 STEP 2742 H=2742:00 +CPU= 34.614
  05:01:33 STEP 2743 H=2743:00 +CPU= 21.193
  05:01:57 STEP 2744 H=2744:00 +CPU= 22.165
  05:02:33 STEP 2745 H=2745:00 +CPU= 33.650
  05:02:55 STEP 2746 H=2746:00 +CPU= 22.115
  05:03:16 STEP 2747 H=2747:00 +CPU= 21.495
  05:03:57 STEP 2748 H=2748:00 +CPU= 37.686
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002748
  05:04:18 STEP 2749 H=2749:00 +CPU= 21.535
  05:04:41 STEP 2750 H=2750:00 +CPU= 22.237
  05:05:14 STEP 2751 H=2751:00 +CPU= 33.878
  05:05:37 STEP 2752 H=2752:00 +CPU= 22.218
  05:05:58 STEP 2753 H=2753:00 +CPU= 21.484
  05:06:33 STEP 2754 H=2754:00 +CPU= 34.617
  05:06:54 STEP 2755 H=2755:00 +CPU= 21.463
  05:07:16 STEP 2756 H=2756:00 +CPU= 22.155
  05:07:50 STEP 2757 H=2757:00 +CPU= 33.731
  05:08:12 STEP 2758 H=2758:00 +CPU= 22.172
  05:08:35 STEP 2759 H=2759:00 +CPU= 22.118
  05:09:26 STEP 2760 H=2760:00 +CPU= 39.802
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_114.zip
Uploading the intermediate file: upload_file_114.zip
  05:09:48 STEP 2761 H=2761:00 +CPU= 21.425
  05:10:10 STEP 2762 H=2762:00 +CPU= 22.132
  05:10:43 STEP 2763 H=2763:00 +CPU= 33.722
  05:11:06 STEP 2764 H=2764:00 +CPU= 22.145
  05:11:27 STEP 2765 H=2765:00 +CPU= 21.485
  05:12:02 STEP 2766 H=2766:00 +CPU= 34.676
  05:12:24 STEP 2767 H=2767:00 +CPU= 21.523
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           25  MAX ETADOT VERTICAL VEL.=   1.982414180027482E-007
  LON  =    113.750000000000       degrees
  LAT  =   -38.6913661222016       degrees
  05:12:46 STEP 2768 H=2768:00 +CPU= 22.212
  05:13:20 STEP 2769 H=2769:00 +CPU= 33.834
  05:13:42 STEP 2770 H=2770:00 +CPU= 22.261
  05:14:03 STEP 2771 H=2771:00 +CPU= 21.508
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           27  MAX ETADOT VERTICAL VEL.=   2.814515357665185E-007
  LON  =    111.250000000000       degrees
  LAT  =   -40.9343383242788       degrees
  05:14:41 STEP 2772 H=2772:00 +CPU= 37.611
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002772
  05:15:03 STEP 2773 H=2773:00 +CPU= 21.477
  05:15:25 STEP 2774 H=2774:00 +CPU= 22.220
  05:15:59 STEP 2775 H=2775:00 +CPU= 33.807
  05:16:21 STEP 2776 H=2776:00 +CPU= 22.170
  05:16:42 STEP 2777 H=2777:00 +CPU= 21.453
  05:17:17 STEP 2778 H=2778:00 +CPU= 34.608
  05:17:39 STEP 2779 H=2779:00 +CPU= 21.462
  05:18:01 STEP 2780 H=2780:00 +CPU= 22.174
  05:18:35 STEP 2781 H=2781:00 +CPU= 33.774
  05:18:57 STEP 2782 H=2782:00 +CPU= 22.151
  05:19:19 STEP 2783 H=2783:00 +CPU= 22.119
  05:20:07 STEP 2784 H=2784:00 +CPU= 39.592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_115.zip
Uploading the intermediate file: upload_file_115.zip
  05:20:28 STEP 2785 H=2785:00 +CPU= 21.435
  05:20:51 STEP 2786 H=2786:00 +CPU= 22.140
  05:21:24 STEP 2787 H=2787:00 +CPU= 33.769
  05:21:47 STEP 2788 H=2788:00 +CPU= 22.165
  05:22:08 STEP 2789 H=2789:00 +CPU= 21.459
  05:22:43 STEP 2790 H=2790:00 +CPU= 34.612
  05:23:04 STEP 2791 H=2791:00 +CPU= 21.477
  05:23:26 STEP 2792 H=2792:00 +CPU= 22.197
  05:24:00 STEP 2793 H=2793:00 +CPU= 33.812
  05:24:22 STEP 2794 H=2794:00 +CPU= 22.226
  05:24:44 STEP 2795 H=2795:00 +CPU= 21.493
  05:25:21 STEP 2796 H=2796:00 +CPU= 37.631
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002796
  05:25:43 STEP 2797 H=2797:00 +CPU= 21.501
  05:26:05 STEP 2798 H=2798:00 +CPU= 22.220
  05:26:39 STEP 2799 H=2799:00 +CPU= 33.797
  05:27:01 STEP 2800 H=2800:00 +CPU= 22.186
  05:27:23 STEP 2801 H=2801:00 +CPU= 21.439
  05:27:57 STEP 2802 H=2802:00 +CPU= 34.566
  05:28:19 STEP 2803 H=2803:00 +CPU= 21.466
  05:28:41 STEP 2804 H=2804:00 +CPU= 22.206
  05:29:15 STEP 2805 H=2805:00 +CPU= 33.774
  05:29:37 STEP 2806 H=2806:00 +CPU= 22.145
  05:29:59 STEP 2807 H=2807:00 +CPU= 22.125
  05:30:44 STEP 2808 H=2808:00 +CPU= 39.589
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_116.zip
Uploading the intermediate file: upload_file_116.zip
  05:31:06 STEP 2809 H=2809:00 +CPU= 21.373
  05:31:28 STEP 2810 H=2810:00 +CPU= 22.137
  05:32:02 STEP 2811 H=2811:00 +CPU= 33.746
  05:32:24 STEP 2812 H=2812:00 +CPU= 22.156
  05:32:45 STEP 2813 H=2813:00 +CPU= 21.438
  05:33:20 STEP 2814 H=2814:00 +CPU= 34.589
  05:33:41 STEP 2815 H=2815:00 +CPU= 21.455
  05:34:04 STEP 2816 H=2816:00 +CPU= 22.190
  05:34:37 STEP 2817 H=2817:00 +CPU= 33.777
  05:35:00 STEP 2818 H=2818:00 +CPU= 22.212
  05:35:21 STEP 2819 H=2819:00 +CPU= 21.487
  05:35:59 STEP 2820 H=2820:00 +CPU= 37.576
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002820
  05:36:20 STEP 2821 H=2821:00 +CPU= 21.472
  05:36:42 STEP 2822 H=2822:00 +CPU= 22.186
  05:37:16 STEP 2823 H=2823:00 +CPU= 33.740
  05:37:38 STEP 2824 H=2824:00 +CPU= 22.164
  05:38:00 STEP 2825 H=2825:00 +CPU= 21.457
  05:38:35 STEP 2826 H=2826:00 +CPU= 34.576
  05:38:56 STEP 2827 H=2827:00 +CPU= 21.456
  05:39:18 STEP 2828 H=2828:00 +CPU= 22.171
  05:39:52 STEP 2829 H=2829:00 +CPU= 33.714
  05:40:14 STEP 2830 H=2830:00 +CPU= 22.138
  05:40:36 STEP 2831 H=2831:00 +CPU= 22.074
  05:41:22 STEP 2832 H=2832:00 +CPU= 39.657
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_117.zip
Uploading the intermediate file: upload_file_117.zip
  05:41:43 STEP 2833 H=2833:00 +CPU= 21.430
  05:42:05 STEP 2834 H=2834:00 +CPU= 22.109
  05:42:39 STEP 2835 H=2835:00 +CPU= 33.728
  05:43:01 STEP 2836 H=2836:00 +CPU= 22.124
  05:43:23 STEP 2837 H=2837:00 +CPU= 21.443
  05:43:57 STEP 2838 H=2838:00 +CPU= 34.525
  05:44:19 STEP 2839 H=2839:00 +CPU= 21.443
  05:44:41 STEP 2840 H=2840:00 +CPU= 22.197
  05:45:15 STEP 2841 H=2841:00 +CPU= 33.799
  05:45:37 STEP 2842 H=2842:00 +CPU= 22.170
  05:45:59 STEP 2843 H=2843:00 +CPU= 21.455
  05:46:36 STEP 2844 H=2844:00 +CPU= 37.539
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002844
  05:46:58 STEP 2845 H=2845:00 +CPU= 21.454
  05:47:20 STEP 2846 H=2846:00 +CPU= 22.170
  05:47:54 STEP 2847 H=2847:00 +CPU= 33.739
  05:48:16 STEP 2848 H=2848:00 +CPU= 22.131
  05:48:37 STEP 2849 H=2849:00 +CPU= 21.439
  05:49:12 STEP 2850 H=2850:00 +CPU= 34.535
  05:49:33 STEP 2851 H=2851:00 +CPU= 21.420
  05:49:56 STEP 2852 H=2852:00 +CPU= 22.141
  05:50:29 STEP 2853 H=2853:00 +CPU= 33.722
  05:50:51 STEP 2854 H=2854:00 +CPU= 22.080
  05:51:14 STEP 2855 H=2855:00 +CPU= 21.918
  05:52:00 STEP 2856 H=2856:00 +CPU= 39.361
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_118.zip
Uploading the intermediate file: upload_file_118.zip
  05:52:21 STEP 2857 H=2857:00 +CPU= 21.325
  05:52:43 STEP 2858 H=2858:00 +CPU= 21.900
  05:53:17 STEP 2859 H=2859:00 +CPU= 33.571
  05:53:39 STEP 2860 H=2860:00 +CPU= 22.059
  05:54:00 STEP 2861 H=2861:00 +CPU= 21.339
  05:54:34 STEP 2862 H=2862:00 +CPU= 34.234
  05:54:56 STEP 2863 H=2863:00 +CPU= 21.372
  05:55:18 STEP 2864 H=2864:00 +CPU= 22.075
  05:55:51 STEP 2865 H=2865:00 +CPU= 33.616
  05:56:13 STEP 2866 H=2866:00 +CPU= 22.098
  05:56:35 STEP 2867 H=2867:00 +CPU= 21.178
  05:57:12 STEP 2868 H=2868:00 +CPU= 37.418
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002868
  05:57:33 STEP 2869 H=2869:00 +CPU= 21.371
  05:57:56 STEP 2870 H=2870:00 +CPU= 22.107
  05:58:29 STEP 2871 H=2871:00 +CPU= 33.497
  05:58:51 STEP 2872 H=2872:00 +CPU= 22.064
  05:59:12 STEP 2873 H=2873:00 +CPU= 21.343
  05:59:47 STEP 2874 H=2874:00 +CPU= 34.428
  06:00:08 STEP 2875 H=2875:00 +CPU= 21.342
  06:00:30 STEP 2876 H=2876:00 +CPU= 21.925
  06:01:04 STEP 2877 H=2877:00 +CPU= 33.554
  06:01:26 STEP 2878 H=2878:00 +CPU= 22.028
  06:01:48 STEP 2879 H=2879:00 +CPU= 21.996
  06:02:33 STEP 2880 H=2880:00 +CPU= 39.618
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734_119.zip
Uploading the intermediate file: upload_file_119.zip
  06:02:55 STEP 2881 H=2881:00 +CPU= 21.282
  06:03:17 STEP 2882 H=2882:00 +CPU= 21.987
  06:03:50 STEP 2883 H=2883:00 +CPU= 33.492
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0498
(argv4) batchid: 960
(argv5) wuid: 12177142
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177142
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8866c4f3be9800b3e99c52171738e129 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0498_1991050100_123_960_12177142.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0498_1991050100_123_960_12177142.zip
ic_ancil_file: ic_ancil_12177142
ifsdata_file: ifsdata_12177142
climate_data_file: clim_data_12177142
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8d32566c8244c3ab6d6c331a922f27b4 to: /var/lib/boinc-client/slots/1/ic_ancil_12177142.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12177142.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5966736dce4deb7f34e553e64f3527af to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12177142.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12177142.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12177142.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12177142.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12177142.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12177142.xml
-- Model is restarting --
Adjusting last_iter, 2884, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12177142.xml
last_cpu_time: 76742
upload_file_number: 120
last_iter: 2879
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0498_1991050100_123_960_12177142_1_r1928908734
The child process has been launched with process id: 19502
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:19502:19502] [20230115:070435:1673795075:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:05:31 STEP 2880 H=2880:00 +CPU= 40.305
  07:05:56 STEP 2881 H=2881:00 +CPU= 22.566
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:20:37 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 9,504,000 70,063 0.0074
09 May 2023 16:20:30 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 8,640,000 63,678 0.0074
09 May 2023 16:20:23 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 7,776,000 57,341 0.0074
09 May 2023 16:20:16 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 6,912,000 51,011 0.0074
09 May 2023 16:20:09 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 6,048,000 44,696 0.0074
09 May 2023 16:20:03 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 5,184,000 38,397 0.0074
09 May 2023 16:19:55 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 4,320,000 32,130 0.0074
09 May 2023 16:19:48 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 3,456,000 25,834 0.0075
09 May 2023 16:19:42 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 2,592,000 19,563 0.0075
09 May 2023 16:19:37 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 1,728,000 13,293 0.0077
09 May 2023 16:19:29 1515226 22262465 oifs_43r3_ps_0498_1991050100_123_960_12177142_1 864,000 6,895 0.0080


©2024 cpdn.org