climateprediction.net home page
Task 22280701

Task 22280701

Name oifs_43r3_ps_0178_2009050100_123_978_12194822_0
Workunit 12194822
Created 23 Dec 2022, 20:22:53 UTC
Sent 23 Dec 2022, 21:09:59 UTC
Report deadline 22 Jan 2023, 21:09:59 UTC
Received 11 Jan 2023, 20:07:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537350
Run time 22 hours 32 min 47 sec
CPU time 22 hours 29 min 17 sec
Validate state Invalid
Credit 13,305.60
Device peak FLOPS 5.57 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,416.74 MB
Peak swap size 4,978.08 MB
Peak disk usage 2,003.71 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_100.zip
Uploading the intermediate file: upload_file_100.zip
  15:54:34 STEP 2425 H=2425:00 +CPU= 24.711
  15:55:00 STEP 2426 H=2426:00 +CPU= 26.101
  15:55:39 STEP 2427 H=2427:00 +CPU= 39.219
  15:56:02 STEP 2428 H=2428:00 +CPU= 22.450
  15:56:27 STEP 2429 H=2429:00 +CPU= 25.525
  15:57:07 STEP 2430 H=2430:00 +CPU= 39.781
  15:57:33 STEP 2431 H=2431:00 +CPU= 25.874
  15:58:00 STEP 2432 H=2432:00 +CPU= 26.592
  15:58:38 STEP 2433 H=2433:00 +CPU= 38.784
  15:59:04 STEP 2434 H=2434:00 +CPU= 25.757
  15:59:30 STEP 2435 H=2435:00 +CPU= 25.734
  16:00:08 STEP 2436 H=2436:00 +CPU= 38.314
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002436
  16:00:33 STEP 2437 H=2437:00 +CPU= 24.674
  16:00:57 STEP 2438 H=2438:00 +CPU= 24.267
  16:01:33 STEP 2439 H=2439:00 +CPU= 36.333
  16:02:00 STEP 2440 H=2440:00 +CPU= 26.403
  16:02:25 STEP 2441 H=2441:00 +CPU= 25.404
  16:03:05 STEP 2442 H=2442:00 +CPU= 39.729
  16:03:30 STEP 2443 H=2443:00 +CPU= 25.164
  16:03:57 STEP 2444 H=2444:00 +CPU= 26.553
  16:04:35 STEP 2445 H=2445:00 +CPU= 38.598
  16:05:00 STEP 2446 H=2446:00 +CPU= 24.778
  16:05:25 STEP 2447 H=2447:00 +CPU= 24.321
  16:06:10 STEP 2448 H=2448:00 +CPU= 45.744
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_101.zip
Uploading the intermediate file: upload_file_101.zip
  16:06:30 STEP 2449 H=2449:00 +CPU= 19.795
  16:06:56 STEP 2450 H=2450:00 +CPU= 26.016
  16:07:36 STEP 2451 H=2451:00 +CPU= 39.271
  16:08:01 STEP 2452 H=2452:00 +CPU= 25.382
  16:08:25 STEP 2453 H=2453:00 +CPU= 23.783
  16:09:01 STEP 2454 H=2454:00 +CPU= 36.522
  16:09:27 STEP 2455 H=2455:00 +CPU= 25.317
  16:09:52 STEP 2456 H=2456:00 +CPU= 25.554
  16:10:31 STEP 2457 H=2457:00 +CPU= 39.326
  16:10:57 STEP 2458 H=2458:00 +CPU= 25.753
  16:11:22 STEP 2459 H=2459:00 +CPU= 24.690
  16:12:04 STEP 2460 H=2460:00 +CPU= 42.284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002460
  16:12:30 STEP 2461 H=2461:00 +CPU= 25.389
  16:12:55 STEP 2462 H=2462:00 +CPU= 24.861
  16:13:32 STEP 2463 H=2463:00 +CPU= 37.555
  16:13:58 STEP 2464 H=2464:00 +CPU= 26.256
  16:14:24 STEP 2465 H=2465:00 +CPU= 25.581
  16:15:02 STEP 2466 H=2466:00 +CPU= 37.819
  16:15:27 STEP 2467 H=2467:00 +CPU= 25.068
  16:15:53 STEP 2468 H=2468:00 +CPU= 25.813
  16:16:32 STEP 2469 H=2469:00 +CPU= 39.445
  16:16:58 STEP 2470 H=2470:00 +CPU= 25.388
  16:17:24 STEP 2471 H=2471:00 +CPU= 26.352
  16:18:10 STEP 2472 H=2472:00 +CPU= 46.206
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_102.zip
Uploading the intermediate file: upload_file_102.zip
  16:18:36 STEP 2473 H=2473:00 +CPU= 25.818
  16:19:02 STEP 2474 H=2474:00 +CPU= 26.129
  16:19:41 STEP 2475 H=2475:00 +CPU= 38.249
  16:20:07 STEP 2476 H=2476:00 +CPU= 26.688
  16:20:32 STEP 2477 H=2477:00 +CPU= 24.590
  16:21:11 STEP 2478 H=2478:00 +CPU= 39.503
  16:21:36 STEP 2479 H=2479:00 +CPU= 24.589
  16:22:02 STEP 2480 H=2480:00 +CPU= 25.648
  16:22:41 STEP 2481 H=2481:00 +CPU= 39.048
  16:23:06 STEP 2482 H=2482:00 +CPU= 25.453
  16:23:32 STEP 2483 H=2483:00 +CPU= 26.113
  16:24:15 STEP 2484 H=2484:00 +CPU= 42.412
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002484
  16:24:40 STEP 2485 H=2485:00 +CPU= 25.580
  16:25:05 STEP 2486 H=2486:00 +CPU= 25.071
  16:25:45 STEP 2487 H=2487:00 +CPU= 39.374
  16:26:10 STEP 2488 H=2488:00 +CPU= 25.388
  16:26:36 STEP 2489 H=2489:00 +CPU= 25.804
  16:27:15 STEP 2490 H=2490:00 +CPU= 39.430
  16:27:37 STEP 2491 H=2491:00 +CPU= 21.341
  16:28:02 STEP 2492 H=2492:00 +CPU= 25.579
  16:28:41 STEP 2493 H=2493:00 +CPU= 38.288
  16:29:02 STEP 2494 H=2494:00 +CPU= 21.438
  16:29:24 STEP 2495 H=2495:00 +CPU= 21.983
  16:30:10 STEP 2496 H=2496:00 +CPU= 45.780
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_103.zip
Uploading the intermediate file: upload_file_103.zip
  16:30:34 STEP 2497 H=2497:00 +CPU= 24.228
  16:30:58 STEP 2498 H=2498:00 +CPU= 23.753
  16:31:35 STEP 2499 H=2499:00 +CPU= 36.846
  16:32:01 STEP 2500 H=2500:00 +CPU= 26.317
  16:32:26 STEP 2501 H=2501:00 +CPU= 24.732
  16:33:02 STEP 2502 H=2502:00 +CPU= 35.653
  16:33:24 STEP 2503 H=2503:00 +CPU= 22.393
  16:33:50 STEP 2504 H=2504:00 +CPU= 25.582
  16:34:28 STEP 2505 H=2505:00 +CPU= 38.719
  16:34:54 STEP 2506 H=2506:00 +CPU= 25.768
  16:35:19 STEP 2507 H=2507:00 +CPU= 24.327
  16:36:01 STEP 2508 H=2508:00 +CPU= 42.311
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002508
  16:36:25 STEP 2509 H=2509:00 +CPU= 23.893
  16:36:48 STEP 2510 H=2510:00 +CPU= 22.821
  16:37:27 STEP 2511 H=2511:00 +CPU= 39.593
  16:37:52 STEP 2512 H=2512:00 +CPU= 24.763
  16:38:16 STEP 2513 H=2513:00 +CPU= 24.062
  16:38:54 STEP 2514 H=2514:00 +CPU= 38.282
  16:39:20 STEP 2515 H=2515:00 +CPU= 25.889
  16:39:46 STEP 2516 H=2516:00 +CPU= 25.855
  16:40:25 STEP 2517 H=2517:00 +CPU= 39.089
  16:40:51 STEP 2518 H=2518:00 +CPU= 25.806
  16:41:18 STEP 2519 H=2519:00 +CPU= 26.354
  16:42:03 STEP 2520 H=2520:00 +CPU= 45.775
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_104.zip
Uploading the intermediate file: upload_file_104.zip
  16:42:28 STEP 2521 H=2521:00 +CPU= 24.703
  16:42:54 STEP 2522 H=2522:00 +CPU= 25.514
  16:43:31 STEP 2523 H=2523:00 +CPU= 37.307
  16:43:56 STEP 2524 H=2524:00 +CPU= 24.842
  16:44:19 STEP 2525 H=2525:00 +CPU= 23.382
  16:44:57 STEP 2526 H=2526:00 +CPU= 37.497
  16:45:21 STEP 2527 H=2527:00 +CPU= 24.546
  16:45:47 STEP 2528 H=2528:00 +CPU= 25.460
  16:46:24 STEP 2529 H=2529:00 +CPU= 37.388
  16:46:50 STEP 2530 H=2530:00 +CPU= 25.347
  16:47:14 STEP 2531 H=2531:00 +CPU= 24.465
  16:47:57 STEP 2532 H=2532:00 +CPU= 42.798
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002532
  16:48:22 STEP 2533 H=2533:00 +CPU= 25.412
  16:48:48 STEP 2534 H=2534:00 +CPU= 25.876
  16:49:26 STEP 2535 H=2535:00 +CPU= 38.161
  16:49:45 STEP 2536 H=2536:00 +CPU= 18.673
  16:50:10 STEP 2537 H=2537:00 +CPU= 24.963
  16:50:50 STEP 2538 H=2538:00 +CPU= 39.970
  16:51:16 STEP 2539 H=2539:00 +CPU= 25.666
  16:51:42 STEP 2540 H=2540:00 +CPU= 25.855
  16:52:20 STEP 2541 H=2541:00 +CPU= 38.069
  16:52:46 STEP 2542 H=2542:00 +CPU= 26.692
  16:53:12 STEP 2543 H=2543:00 +CPU= 26.115
  16:53:58 STEP 2544 H=2544:00 +CPU= 45.851
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_105.zip
Uploading the intermediate file: upload_file_105.zip
  16:54:21 STEP 2545 H=2545:00 +CPU= 22.333
  16:54:47 STEP 2546 H=2546:00 +CPU= 26.244
  16:55:26 STEP 2547 H=2547:00 +CPU= 39.458
  16:55:54 STEP 2548 H=2548:00 +CPU= 27.388
  16:56:19 STEP 2549 H=2549:00 +CPU= 25.484
  16:56:59 STEP 2550 H=2550:00 +CPU= 39.682
  16:57:23 STEP 2551 H=2551:00 +CPU= 24.261
  16:57:48 STEP 2552 H=2552:00 +CPU= 24.338
  16:58:24 STEP 2553 H=2553:00 +CPU= 36.572
  16:58:49 STEP 2554 H=2554:00 +CPU= 24.760
  16:59:14 STEP 2555 H=2555:00 +CPU= 24.987
  16:59:57 STEP 2556 H=2556:00 +CPU= 42.597
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002556
  17:00:23 STEP 2557 H=2557:00 +CPU= 25.859
  17:00:49 STEP 2558 H=2558:00 +CPU= 26.044
  17:01:27 STEP 2559 H=2559:00 +CPU= 37.963
  17:01:53 STEP 2560 H=2560:00 +CPU= 26.311
  17:02:15 STEP 2561 H=2561:00 +CPU= 22.325
  17:02:52 STEP 2562 H=2562:00 +CPU= 36.088
  17:03:17 STEP 2563 H=2563:00 +CPU= 25.696
  17:03:42 STEP 2564 H=2564:00 +CPU= 24.385
  17:04:20 STEP 2565 H=2565:00 +CPU= 38.532
  17:04:46 STEP 2566 H=2566:00 +CPU= 25.957
  17:05:13 STEP 2567 H=2567:00 +CPU= 26.094
  17:05:58 STEP 2568 H=2568:00 +CPU= 45.315
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_106.zip
Uploading the intermediate file: upload_file_106.zip
  17:06:21 STEP 2569 H=2569:00 +CPU= 22.591
  17:06:45 STEP 2570 H=2570:00 +CPU= 24.830
  17:07:24 STEP 2571 H=2571:00 +CPU= 38.671
  17:07:52 STEP 2572 H=2572:00 +CPU= 27.354
  17:08:18 STEP 2573 H=2573:00 +CPU= 26.061
  17:08:57 STEP 2574 H=2574:00 +CPU= 39.828
  17:09:21 STEP 2575 H=2575:00 +CPU= 23.674
  17:09:48 STEP 2576 H=2576:00 +CPU= 26.657
  17:10:27 STEP 2577 H=2577:00 +CPU= 39.321
  17:10:53 STEP 2578 H=2578:00 +CPU= 26.070
  17:11:16 STEP 2579 H=2579:00 +CPU= 22.969
  17:11:58 STEP 2580 H=2580:00 +CPU= 41.228
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002580
  17:12:24 STEP 2581 H=2581:00 +CPU= 26.030
  17:12:50 STEP 2582 H=2582:00 +CPU= 25.978
  17:13:27 STEP 2583 H=2583:00 +CPU= 37.821
  17:13:51 STEP 2584 H=2584:00 +CPU= 23.073
  17:14:13 STEP 2585 H=2585:00 +CPU= 22.766
  17:14:51 STEP 2586 H=2586:00 +CPU= 37.450
  17:15:14 STEP 2587 H=2587:00 +CPU= 23.267
  17:15:40 STEP 2588 H=2588:00 +CPU= 25.453
  17:16:19 STEP 2589 H=2589:00 +CPU= 39.018
  17:16:45 STEP 2590 H=2590:00 +CPU= 25.867
  17:17:11 STEP 2591 H=2591:00 +CPU= 26.074
  17:17:55 STEP 2592 H=2592:00 +CPU= 44.763
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_107.zip
Uploading the intermediate file: upload_file_107.zip
  17:18:20 STEP 2593 H=2593:00 +CPU= 24.718
  17:18:43 STEP 2594 H=2594:00 +CPU= 22.458
  17:19:21 STEP 2595 H=2595:00 +CPU= 38.876
  17:19:48 STEP 2596 H=2596:00 +CPU= 26.388
  17:20:14 STEP 2597 H=2597:00 +CPU= 25.827
  17:20:53 STEP 2598 H=2598:00 +CPU= 39.311
  17:21:19 STEP 2599 H=2599:00 +CPU= 25.695
  17:21:43 STEP 2600 H=2600:00 +CPU= 23.760
  17:22:17 STEP 2601 H=2601:00 +CPU= 34.640
  17:22:42 STEP 2602 H=2602:00 +CPU= 24.670
  17:23:06 STEP 2603 H=2603:00 +CPU= 23.547
  17:23:47 STEP 2604 H=2604:00 +CPU= 41.875
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002604
  17:24:12 STEP 2605 H=2605:00 +CPU= 24.746
  17:24:36 STEP 2606 H=2606:00 +CPU= 23.713
  17:25:14 STEP 2607 H=2607:00 +CPU= 37.694
  17:25:38 STEP 2608 H=2608:00 +CPU= 24.338
  17:26:02 STEP 2609 H=2609:00 +CPU= 24.040
  17:26:41 STEP 2610 H=2610:00 +CPU= 38.469
  17:27:05 STEP 2611 H=2611:00 +CPU= 24.680
  17:27:29 STEP 2612 H=2612:00 +CPU= 24.041
  17:28:07 STEP 2613 H=2613:00 +CPU= 37.796
  17:28:31 STEP 2614 H=2614:00 +CPU= 23.892
  17:28:55 STEP 2615 H=2615:00 +CPU= 24.125
  17:29:38 STEP 2616 H=2616:00 +CPU= 42.113
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_108.zip
Uploading the intermediate file: upload_file_108.zip
  17:30:01 STEP 2617 H=2617:00 +CPU= 23.340
  17:30:25 STEP 2618 H=2618:00 +CPU= 23.673
  17:31:04 STEP 2619 H=2619:00 +CPU= 39.048
  17:31:29 STEP 2620 H=2620:00 +CPU= 25.338
  17:31:53 STEP 2621 H=2621:00 +CPU= 23.922
  17:32:30 STEP 2622 H=2622:00 +CPU= 36.885
  17:32:55 STEP 2623 H=2623:00 +CPU= 24.692
  17:33:21 STEP 2624 H=2624:00 +CPU= 26.556
  17:33:57 STEP 2625 H=2625:00 +CPU= 35.419
  17:34:21 STEP 2626 H=2626:00 +CPU= 24.566
  17:34:45 STEP 2627 H=2627:00 +CPU= 23.290
  17:35:27 STEP 2628 H=2628:00 +CPU= 42.455
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002628
  17:35:48 STEP 2629 H=2629:00 +CPU= 20.398
  17:36:12 STEP 2630 H=2630:00 +CPU= 24.391
  17:36:51 STEP 2631 H=2631:00 +CPU= 39.295
  17:37:18 STEP 2632 H=2632:00 +CPU= 26.128
  17:37:42 STEP 2633 H=2633:00 +CPU= 24.119
  17:38:20 STEP 2634 H=2634:00 +CPU= 38.731
  17:38:45 STEP 2635 H=2635:00 +CPU= 24.061
  17:39:10 STEP 2636 H=2636:00 +CPU= 25.125
  17:39:46 STEP 2637 H=2637:00 +CPU= 35.892
  17:40:11 STEP 2638 H=2638:00 +CPU= 25.534
  17:40:36 STEP 2639 H=2639:00 +CPU= 25.290
  17:41:20 STEP 2640 H=2640:00 +CPU= 43.529
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_109.zip
Uploading the intermediate file: upload_file_109.zip
  17:41:44 STEP 2641 H=2641:00 +CPU= 24.384
Uploading trickle at timestep: 9504000
  17:42:08 STEP 2642 H=2642:00 +CPU= 23.273
  17:42:47 STEP 2643 H=2643:00 +CPU= 39.347
  17:43:12 STEP 2644 H=2644:00 +CPU= 25.226
  17:43:37 STEP 2645 H=2645:00 +CPU= 25.063
  17:44:19 STEP 2646 H=2646:00 +CPU= 41.310
  17:44:44 STEP 2647 H=2647:00 +CPU= 25.078
  17:45:07 STEP 2648 H=2648:00 +CPU= 23.349
  17:45:47 STEP 2649 H=2649:00 +CPU= 39.744
  17:46:12 STEP 2650 H=2650:00 +CPU= 24.554
  17:46:36 STEP 2651 H=2651:00 +CPU= 24.784
  17:47:20 STEP 2652 H=2652:00 +CPU= 43.726
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002652
  17:47:45 STEP 2653 H=2653:00 +CPU= 24.858
  17:48:11 STEP 2654 H=2654:00 +CPU= 26.341
  17:48:51 STEP 2655 H=2655:00 +CPU= 39.819
  17:49:17 STEP 2656 H=2656:00 +CPU= 25.702
  17:49:41 STEP 2657 H=2657:00 +CPU= 23.849
  17:50:19 STEP 2658 H=2658:00 +CPU= 38.050
  17:50:44 STEP 2659 H=2659:00 +CPU= 25.338
  17:51:09 STEP 2660 H=2660:00 +CPU= 25.320
  17:51:49 STEP 2661 H=2661:00 +CPU= 39.116
  17:52:15 STEP 2662 H=2662:00 +CPU= 26.103
  17:52:41 STEP 2663 H=2663:00 +CPU= 26.082
  17:53:26 STEP 2664 H=2664:00 +CPU= 45.645
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_110.zip
Uploading the intermediate file: upload_file_110.zip
  17:53:52 STEP 2665 H=2665:00 +CPU= 25.836
  17:54:18 STEP 2666 H=2666:00 +CPU= 26.055
  17:54:55 STEP 2667 H=2667:00 +CPU= 36.920
  17:55:20 STEP 2668 H=2668:00 +CPU= 24.779
  17:55:44 STEP 2669 H=2669:00 +CPU= 23.516
  17:56:19 STEP 2670 H=2670:00 +CPU= 35.638
  17:56:44 STEP 2671 H=2671:00 +CPU= 24.348
  17:57:06 STEP 2672 H=2672:00 +CPU= 22.684
  17:57:46 STEP 2673 H=2673:00 +CPU= 39.425
  17:58:11 STEP 2674 H=2674:00 +CPU= 25.597
  17:58:36 STEP 2675 H=2675:00 +CPU= 24.584
  17:59:19 STEP 2676 H=2676:00 +CPU= 42.462
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002676
  17:59:41 STEP 2677 H=2677:00 +CPU= 22.315
  18:00:05 STEP 2678 H=2678:00 +CPU= 24.455
  18:00:43 STEP 2679 H=2679:00 +CPU= 37.587
  18:01:05 STEP 2680 H=2680:00 +CPU= 22.260
  18:01:30 STEP 2681 H=2681:00 +CPU= 24.697
  18:02:08 STEP 2682 H=2682:00 +CPU= 38.182
  18:02:33 STEP 2683 H=2683:00 +CPU= 25.113
  18:02:59 STEP 2684 H=2684:00 +CPU= 25.725
  18:03:38 STEP 2685 H=2685:00 +CPU= 39.207
  18:04:05 STEP 2686 H=2686:00 +CPU= 26.402
  18:04:29 STEP 2687 H=2687:00 +CPU= 24.708
  18:05:16 STEP 2688 H=2688:00 +CPU= 46.358
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_111.zip
Uploading the intermediate file: upload_file_111.zip
  18:05:41 STEP 2689 H=2689:00 +CPU= 25.015
  18:06:04 STEP 2690 H=2690:00 +CPU= 23.189
  18:06:40 STEP 2691 H=2691:00 +CPU= 36.243
  18:07:06 STEP 2692 H=2692:00 +CPU= 25.570
  18:07:29 STEP 2693 H=2693:00 +CPU= 23.446
  18:08:09 STEP 2694 H=2694:00 +CPU= 39.681
  18:08:33 STEP 2695 H=2695:00 +CPU= 23.931
  18:08:59 STEP 2696 H=2696:00 +CPU= 25.839
  18:09:38 STEP 2697 H=2697:00 +CPU= 39.703
  18:10:04 STEP 2698 H=2698:00 +CPU= 25.724
  18:10:29 STEP 2699 H=2699:00 +CPU= 24.395
  18:11:08 STEP 2700 H=2700:00 +CPU= 39.402
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002700
  18:11:31 STEP 2701 H=2701:00 +CPU= 22.651
  18:11:57 STEP 2702 H=2702:00 +CPU= 26.518
  18:12:36 STEP 2703 H=2703:00 +CPU= 39.100
  18:13:01 STEP 2704 H=2704:00 +CPU= 24.723
  18:13:27 STEP 2705 H=2705:00 +CPU= 25.465
  18:14:06 STEP 2706 H=2706:00 +CPU= 39.830
  18:14:31 STEP 2707 H=2707:00 +CPU= 24.826
  18:14:57 STEP 2708 H=2708:00 +CPU= 25.422
  18:15:36 STEP 2709 H=2709:00 +CPU= 39.528
  18:16:01 STEP 2710 H=2710:00 +CPU= 24.536
  18:16:22 STEP 2711 H=2711:00 +CPU= 20.847
  18:17:00 STEP 2712 H=2712:00 +CPU= 38.224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_112.zip
Uploading the intermediate file: upload_file_112.zip
  18:17:25 STEP 2713 H=2713:00 +CPU= 24.590
  18:17:51 STEP 2714 H=2714:00 +CPU= 26.438
  18:18:30 STEP 2715 H=2715:00 +CPU= 38.700
  18:18:54 STEP 2716 H=2716:00 +CPU= 23.762
  18:19:16 STEP 2717 H=2717:00 +CPU= 22.618
  18:19:54 STEP 2718 H=2718:00 +CPU= 38.006
  18:20:18 STEP 2719 H=2719:00 +CPU= 23.129
  18:20:43 STEP 2720 H=2720:00 +CPU= 25.458
  18:21:21 STEP 2721 H=2721:00 +CPU= 38.288
  18:21:45 STEP 2722 H=2722:00 +CPU= 24.139
  18:22:09 STEP 2723 H=2723:00 +CPU= 23.021
  18:22:50 STEP 2724 H=2724:00 +CPU= 41.060
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002724
  18:23:08 STEP 2725 H=2725:00 +CPU= 18.218
  18:23:30 STEP 2726 H=2726:00 +CPU= 22.614
  18:24:08 STEP 2727 H=2727:00 +CPU= 37.939
  18:24:34 STEP 2728 H=2728:00 +CPU= 26.013
  18:24:58 STEP 2729 H=2729:00 +CPU= 23.922
  18:25:38 STEP 2730 H=2730:00 +CPU= 39.332
  18:26:03 STEP 2731 H=2731:00 +CPU= 24.909
  18:26:28 STEP 2732 H=2732:00 +CPU= 25.112
  18:27:06 STEP 2733 H=2733:00 +CPU= 37.839
  18:27:32 STEP 2734 H=2734:00 +CPU= 26.378
  18:27:58 STEP 2735 H=2735:00 +CPU= 26.164
  18:28:44 STEP 2736 H=2736:00 +CPU= 46.024
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_113.zip
Uploading the intermediate file: upload_file_113.zip
  18:29:07 STEP 2737 H=2737:00 +CPU= 22.927
  18:29:27 STEP 2738 H=2738:00 +CPU= 19.453
  18:30:04 STEP 2739 H=2739:00 +CPU= 36.933
  18:30:29 STEP 2740 H=2740:00 +CPU= 25.307
  18:30:53 STEP 2741 H=2741:00 +CPU= 24.402
  18:31:34 STEP 2742 H=2742:00 +CPU= 40.495
  18:31:55 STEP 2743 H=2743:00 +CPU= 21.660
  18:32:19 STEP 2744 H=2744:00 +CPU= 23.632
  18:32:57 STEP 2745 H=2745:00 +CPU= 38.062
  18:33:23 STEP 2746 H=2746:00 +CPU= 25.379
  18:33:46 STEP 2747 H=2747:00 +CPU= 23.664
  18:34:19 STEP 2748 H=2748:00 +CPU= 33.012
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002748
  18:34:38 STEP 2749 H=2749:00 +CPU= 18.594
  18:35:02 STEP 2750 H=2750:00 +CPU= 24.067
  18:35:36 STEP 2751 H=2751:00 +CPU= 34.391
  18:36:00 STEP 2752 H=2752:00 +CPU= 23.662
  18:36:24 STEP 2753 H=2753:00 +CPU= 23.653
  18:36:56 STEP 2754 H=2754:00 +CPU= 32.324
  18:37:16 STEP 2755 H=2755:00 +CPU= 20.444
  18:37:40 STEP 2756 H=2756:00 +CPU= 23.268
  18:38:18 STEP 2757 H=2757:00 +CPU= 37.600
  18:38:43 STEP 2758 H=2758:00 +CPU= 25.339
  18:39:09 STEP 2759 H=2759:00 +CPU= 25.977
  18:39:55 STEP 2760 H=2760:00 +CPU= 45.689
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_114.zip
Uploading the intermediate file: upload_file_114.zip
  18:40:20 STEP 2761 H=2761:00 +CPU= 25.789
  18:40:46 STEP 2762 H=2762:00 +CPU= 25.336
  18:41:24 STEP 2763 H=2763:00 +CPU= 38.815
  18:41:52 STEP 2764 H=2764:00 +CPU= 27.086
  18:42:17 STEP 2765 H=2765:00 +CPU= 25.344
  18:42:56 STEP 2766 H=2766:00 +CPU= 39.320
  18:43:22 STEP 2767 H=2767:00 +CPU= 25.441
  18:43:48 STEP 2768 H=2768:00 +CPU= 25.960
  18:44:19 STEP 2769 H=2769:00 +CPU= 31.506
  18:44:45 STEP 2770 H=2770:00 +CPU= 25.898
  18:45:11 STEP 2771 H=2771:00 +CPU= 25.397
  18:45:54 STEP 2772 H=2772:00 +CPU= 42.946
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002772
  18:46:19 STEP 2773 H=2773:00 +CPU= 25.218
  18:46:43 STEP 2774 H=2774:00 +CPU= 24.501
  18:47:22 STEP 2775 H=2775:00 +CPU= 38.233
  18:47:47 STEP 2776 H=2776:00 +CPU= 25.825
  18:48:12 STEP 2777 H=2777:00 +CPU= 24.215
  18:48:50 STEP 2778 H=2778:00 +CPU= 37.922
  18:49:13 STEP 2779 H=2779:00 +CPU= 23.758
  18:49:39 STEP 2780 H=2780:00 +CPU= 25.223
  18:50:14 STEP 2781 H=2781:00 +CPU= 35.042
  18:50:40 STEP 2782 H=2782:00 +CPU= 26.367
  18:51:06 STEP 2783 H=2783:00 +CPU= 25.636
  18:51:52 STEP 2784 H=2784:00 +CPU= 46.325
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_115.zip
Uploading the intermediate file: upload_file_115.zip
  18:52:18 STEP 2785 H=2785:00 +CPU= 25.314
  18:52:42 STEP 2786 H=2786:00 +CPU= 24.111
  18:53:16 STEP 2787 H=2787:00 +CPU= 34.166
  18:53:39 STEP 2788 H=2788:00 +CPU= 22.894
  18:54:04 STEP 2789 H=2789:00 +CPU= 24.710
  18:54:43 STEP 2790 H=2790:00 +CPU= 39.106
  18:55:07 STEP 2791 H=2791:00 +CPU= 24.345
  18:55:30 STEP 2792 H=2792:00 +CPU= 22.990
  18:56:09 STEP 2793 H=2793:00 +CPU= 38.679
  18:56:35 STEP 2794 H=2794:00 +CPU= 25.801
  18:56:59 STEP 2795 H=2795:00 +CPU= 24.115
  18:57:42 STEP 2796 H=2796:00 +CPU= 43.167
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002796
  18:58:08 STEP 2797 H=2797:00 +CPU= 25.700
  18:58:34 STEP 2798 H=2798:00 +CPU= 26.090
  18:59:12 STEP 2799 H=2799:00 +CPU= 38.472
  18:59:39 STEP 2800 H=2800:00 +CPU= 26.593
  19:00:04 STEP 2801 H=2801:00 +CPU= 24.860
  19:00:43 STEP 2802 H=2802:00 +CPU= 39.754
  19:01:09 STEP 2803 H=2803:00 +CPU= 26.000
  19:01:36 STEP 2804 H=2804:00 +CPU= 26.325
  19:02:15 STEP 2805 H=2805:00 +CPU= 39.173
  19:02:41 STEP 2806 H=2806:00 +CPU= 26.306
  19:03:08 STEP 2807 H=2807:00 +CPU= 26.478
  19:03:54 STEP 2808 H=2808:00 +CPU= 46.636
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_116.zip
Uploading the intermediate file: upload_file_116.zip
  19:04:21 STEP 2809 H=2809:00 +CPU= 26.521
  19:04:47 STEP 2810 H=2810:00 +CPU= 26.289
  19:05:20 STEP 2811 H=2811:00 +CPU= 33.122
  19:05:46 STEP 2812 H=2812:00 +CPU= 25.172
  19:06:09 STEP 2813 H=2813:00 +CPU= 22.869
  19:06:46 STEP 2814 H=2814:00 +CPU= 37.088
  19:07:10 STEP 2815 H=2815:00 +CPU= 23.865
  19:07:35 STEP 2816 H=2816:00 +CPU= 25.133
  19:08:11 STEP 2817 H=2817:00 +CPU= 35.745
  19:08:35 STEP 2818 H=2818:00 +CPU= 24.521
  19:08:59 STEP 2819 H=2819:00 +CPU= 23.500
  19:09:41 STEP 2820 H=2820:00 +CPU= 42.214
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002820
  19:10:05 STEP 2821 H=2821:00 +CPU= 24.314
  19:10:32 STEP 2822 H=2822:00 +CPU= 26.715
  19:11:09 STEP 2823 H=2823:00 +CPU= 37.565
  19:11:35 STEP 2824 H=2824:00 +CPU= 25.842
  19:12:00 STEP 2825 H=2825:00 +CPU= 24.879
  19:12:36 STEP 2826 H=2826:00 +CPU= 35.907
  19:12:59 STEP 2827 H=2827:00 +CPU= 22.426
  19:13:25 STEP 2828 H=2828:00 +CPU= 26.012
  19:14:04 STEP 2829 H=2829:00 +CPU= 39.010
  19:14:31 STEP 2830 H=2830:00 +CPU= 26.632
  19:14:55 STEP 2831 H=2831:00 +CPU= 24.736
  19:15:38 STEP 2832 H=2832:00 +CPU= 42.194
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_117.zip
Uploading the intermediate file: upload_file_117.zip
  19:16:02 STEP 2833 H=2833:00 +CPU= 24.904
  19:16:29 STEP 2834 H=2834:00 +CPU= 26.608
  19:17:08 STEP 2835 H=2835:00 +CPU= 39.099
  19:17:34 STEP 2836 H=2836:00 +CPU= 26.007
  19:17:59 STEP 2837 H=2837:00 +CPU= 25.129
  19:18:39 STEP 2838 H=2838:00 +CPU= 39.618
  19:19:01 STEP 2839 H=2839:00 +CPU= 22.415
  19:19:26 STEP 2840 H=2840:00 +CPU= 24.897
  19:20:05 STEP 2841 H=2841:00 +CPU= 38.348
  19:20:29 STEP 2842 H=2842:00 +CPU= 23.949
  19:20:54 STEP 2843 H=2843:00 +CPU= 25.142
  19:21:38 STEP 2844 H=2844:00 +CPU= 43.744
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002844
  19:22:02 STEP 2845 H=2845:00 +CPU= 24.403
  19:22:30 STEP 2846 H=2846:00 +CPU= 27.356
  19:23:09 STEP 2847 H=2847:00 +CPU= 39.294
  19:23:34 STEP 2848 H=2848:00 +CPU= 25.382
  19:24:00 STEP 2849 H=2849:00 +CPU= 25.574
  19:24:40 STEP 2850 H=2850:00 +CPU= 40.174
  19:25:06 STEP 2851 H=2851:00 +CPU= 26.073
  19:25:33 STEP 2852 H=2852:00 +CPU= 26.607
  19:26:10 STEP 2853 H=2853:00 +CPU= 37.350
  19:26:35 STEP 2854 H=2854:00 +CPU= 24.882
  19:27:02 STEP 2855 H=2855:00 +CPU= 26.953
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4031MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4031 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-RADLSW       (1210) 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               RADIATION_SCHEME 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                radiation_interface:radiation 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192716:1671910036:80222.602] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 radiation_mcica_sw:solver_mcica_sw 
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192717:1671910037:80223.603] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:3861:3861] [20221224:192717:1671910037:80223.603] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000196ABFB  radiation_mcica_s         286  radiation_mcica_sw.F90
oifs_43r3_model.e  0000000001862DC8  radiation_interfa         419  radiation_interface.F90
oifs_43r3_model.e  00000000012D30E5  radiation_scheme_         484  radiation_scheme.F90
oifs_43r3_model.e  000000000129A415  radintg_                 1455  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0178
(argv4) batchid: 978
(argv5) wuid: 12194822
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e2cf6c84aead8e566b0951b661824095 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0178_2009050100_123_978_12194822.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0178_2009050100_123_978_12194822.zip
ic_ancil_file: ic_ancil_12194822
ifsdata_file: ifsdata_12194822
climate_data_file: clim_data_12194822
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9d8bb003769847c97c157162de8ccf64 to: /var/lib/boinc-client/slots/5/ic_ancil_12194822.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12194822.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8add824e89a655bbd50de118bfa497d8 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12194822.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12194822.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12194822.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12194822.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12194822.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12194822.xml
-- Model is restarting --
Adjusting last_iter, 2856, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12194822.xml
last_cpu_time: 80169
upload_file_number: 118
last_iter: 2855
last_upload: 10195200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909
The child process has been launched with process id: 1152
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:185252:1673463172:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:54:43 STEP 2856 H=2856:00 +CPU= 47.680
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194822/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_2009050100_123_978_12194822_0_r1850859909_118.zip
Uploading the intermediate file: upload_file_118.zip
  18:55:10 STEP 2857 H=2857:00 +CPU= 26.929
  18:55:36 STEP 2858 H=2858:00 +CPU= 26.624
  18:56:18 STEP 2859 H=2859:00 +CPU= 41.156
  18:56:43 STEP 2860 H=2860:00 +CPU= 25.582
  18:57:10 STEP 2861 H=2861:00 +CPU= 27.071
  18:57:55 STEP 2862 H=2862:00 +CPU= 43.976
  18:58:21 STEP 2863 H=2863:00 +CPU= 26.447
  18:58:49 STEP 2864 H=2864:00 +CPU= 28.151
  18:59:31 STEP 2865 H=2865:00 +CPU= 42.111
  18:59:59 STEP 2866 H=2866:00 +CPU= 27.832
  19:00:25 STEP 2867 H=2867:00 +CPU= 26.014
  19:01:12 STEP 2868 H=2868:00 +CPU= 46.450
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002868
  19:01:39 STEP 2869 H=2869:00 +CPU= 26.981
  19:02:07 STEP 2870 H=2870:00 +CPU= 27.739
  19:02:49 STEP 2871 H=2871:00 +CPU= 42.673
  19:03:17 STEP 2872 H=2872:00 +CPU= 27.857
  19:03:43 STEP 2873 H=2873:00 +CPU= 26.172
  19:04:26 STEP 2874 H=2874:00 +CPU= 42.449
  19:04:52 STEP 2875 H=2875:00 +CPU= 25.805
  19:05:18 STEP 2876 H=2876:00 +CPU= 26.007
  19:05:58 STEP 2877 H=2877:00 +CPU= 40.254
  19:06:26 STEP 2878 H=2878:00 +CPU= 28.379
  19:06:56 STEP 2879 H=2879:00 +CPU= 29.335
ECCODES ERROR   :  IO ERROR: No space left on device (No space left on device)
 GRIB_WRITE_BYTES_INT        50001                  415264  FAILED         -11
 GRIB_API ERROR MSG: Input output problem
MPL_ABORT: CALLED FROM PROCESSOR      1 THRD     1
 MPL_ABORT: THRD           1   GRIB_WRITE_BYTES_INT FAILED
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2598 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        DYNFPOS 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         STEPO_FPOS 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          IOFPOS 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           WRPLFP 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            WROUTSPGB 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             IOSTREAM_MIX:IO_PUT 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              IOSTREAM_MIX:SPEC_OUT 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               IOSTREAM_MIX:WRITE_RECORD 
[EC_DRHOOK:c1:1:1:1152:1152] [20230111:190703:1673464023:851.337] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                GRIB_API:IGRIB_WRITE_BYTES_INT 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/5/oifs_43r3_model.exe' : sigcontextptr=0x7ffea98351a0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 22) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/5/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x8c3229 0xa04cea 0x9cf706 0x9ca9b2 0x9c07bc 0x5e3597 0x79a194 0x79823d 0x721c72 0x716b23 0x55d9d8 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:52:56 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 9,504,000 73,830 0.0078
09 May 2023 15:52:43 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 8,640,000 66,679 0.0077
09 May 2023 15:52:31 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 7,776,000 59,509 0.0077
09 May 2023 15:52:19 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 6,912,000 52,337 0.0076
09 May 2023 15:52:08 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 6,048,000 45,157 0.0075
09 May 2023 15:51:56 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 5,184,000 37,977 0.0073
09 May 2023 15:51:46 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 4,320,000 30,746 0.0071
09 May 2023 15:51:35 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 3,456,000 23,506 0.0068
09 May 2023 15:51:24 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 2,592,000 16,396 0.0063
09 May 2023 15:51:14 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 1,728,000 9,873 0.0057
09 May 2023 15:51:08 1537350 22280701 oifs_43r3_ps_0178_2009050100_123_978_12194822_0 864,000 4,603 0.0053


©2024 cpdn.org