Name | hadcm3n_4h8q_2020_40_008406722_0 |
Workunit | 8557578 |
Created | 20 Aug 2013, 7:28:24 UTC |
Sent | 20 Aug 2013, 7:36:44 UTC |
Report deadline | 19 Nov 2013, 15:03:55 UTC |
Received | 4 Oct 2013, 10:48:45 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 22 (0x00000016) Unknown error code |
Computer ID | 1223320 |
Run time | 14 days 10 hours 34 min 25 sec |
CPU time | 11 days 12 hours 38 min 25 sec |
Validate state | Invalid |
Credit | 7,153.92 |
Device peak FLOPS | 2.86 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu |
Stderr | <core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:16:33 (4065): No heartbeat from core client for 30 sec - exiting 22:16:36 (4065): No heartbeat from core client for 30 sec - exiting 22:16:37 (4065): No heartbeat from core client for 30 sec - exiting 22:16:42 (4065): No heartbeat from core client for 30 sec - exiting 22:16:43 (4065): No heartbeat from core client for 30 sec - exiting 22:16:44 (4065): No heartbeat from core client for 30 sec - exiting 22:16:45 (4065): No heartbeat from core client for 30 sec - exiting 22:16:46 (4065): No heartbeat from core client for 30 sec - exiting 22:16:47 (4065): No heartbeat from core client for 30 sec - exiting 22:16:48 (4065): No heartbeat from core client for 30 sec - exiting 22:16:49 (4065): No heartbeat from core client for 30 sec - exiting 22:16:50 (4065): No heartbeat from core client for 30 sec - exiting 22:16:51 (4065): No heartbeat from core client for 30 sec - exiting 22:16:52 (4065): No heartbeat from core client for 30 sec - exiting 22:16:53 (4065): No heartbeat from core client for 30 sec - exiting 22:16:54 (4065): No heartbeat from core client for 30 sec - exiting 22:16:55 (4065): No heartbeat from core client for 30 sec - exiting 22:16:56 (4065): No heartbeat from core client for 30 sec - exiting 22:16:57 (4065): No heartbeat from core client for 30 sec - exiting 22:16:58 (4065): No heartbeat from core client for 30 sec - exiting 22:16:59 (4065): No heartbeat from core client for 30 sec - exiting 22:17:00 (4065): No heartbeat from core client for 30 sec - exiting 22:17:01 (4065): No heartbeat from core client for 30 sec - exiting 22:17:02 (4065): No heartbeat from core client for 30 sec - exiting 22:17:03 (4065): No heartbeat from core client for 30 sec - exiting 22:17:04 (4065): No heartbeat from core client for 30 sec - exiting 22:17:05 (4065): No heartbeat from core client for 30 sec - exiting 22:17:06 (4065): No heartbeat from core client for 30 sec - exiting 22:17:07 (4065): No heartbeat from core client for 30 sec - exiting 22:17:08 (4065): No heartbeat from core client for 30 sec - exiting 22:17:09 (4065): No heartbeat from core client for 30 sec - exiting 22:17:10 (4065): No heartbeat from core client for 30 sec - exiting 22:17:11 (4065): No heartbeat from core client for 30 sec - exiting 22:17:12 (4065): No heartbeat from core client for 30 sec - exiting 22:17:13 (4065): No heartbeat from core client for 30 sec - exiting 22:17:14 (4065): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:26:29 (3808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:26:30 (3808): No heartbeat from core client for 30 sec - exiting 22:26:31 (3808): No heartbeat from core client for 30 sec - exiting 22:26:32 (3808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:30:05 (3988): No heartbeat from core client for 30 sec - exiting 23:30:09 (3988): No heartbeat from core client for 30 sec - exiting 23:30:10 (3988): No heartbeat from core client for 30 sec - exiting 23:30:11 (3988): No heartbeat from core client for 30 sec - exiting 23:30:12 (3988): No heartbeat from core client for 30 sec - exiting 23:30:13 (3988): No heartbeat from core client for 30 sec - exiting 23:30:14 (3988): No heartbeat from core client for 30 sec - exiting 23:30:15 (3988): No heartbeat from core client for 30 sec - exiting 23:30:16 (3988): No heartbeat from core client for 30 sec - exiting 23:30:17 (3988): No heartbeat from core client for 30 sec - exiting 23:30:18 (3988): No heartbeat from core client for 30 sec - exiting 23:30:19 (3988): No heartbeat from core client for 30 sec - exiting 23:30:20 (3988): No heartbeat from core client for 30 sec - exiting 23:30:21 (3988): No heartbeat from core client for 30 sec - exiting 23:30:22 (3988): No heartbeat from core client for 30 sec - exiting 23:30:23 (3988): No heartbeat from core client for 30 sec - exiting 23:30:24 (3988): No heartbeat from core client for 30 sec - exiting 23:30:25 (3988): No heartbeat from core client for 30 sec - exiting 23:30:26 (3988): No heartbeat from core client for 30 sec - exiting 23:30:27 (3988): No heartbeat from core client for 30 sec - exiting 23:30:28 (3988): No heartbeat from core client for 30 sec - exiting 23:30:29 (3988): No heartbeat from core client for 30 sec - exiting 23:30:30 (3988): No heartbeat from core client for 30 sec - exiting 23:30:31 (3988): No heartbeat from core client for 30 sec - exiting 23:30:34 (3988): No heartbeat from core client for 30 sec - exiting 23:30:35 (3988): No heartbeat from core client for 30 sec - exiting 23:30:36 (3988): No heartbeat from core client for 30 sec - exiting 23:30:37 (3988): No heartbeat from core client for 30 sec - exiting 23:30:38 (3988): No heartbeat from core client for 30 sec - exiting 23:30:39 (3988): No heartbeat from core client for 30 sec - exiting 23:30:40 (3988): No heartbeat from core client for 30 sec - exiting 23:30:41 (3988): No heartbeat from core client for 30 sec - exiting 23:30:42 (3988): No heartbeat from core client for 30 sec - exiting 23:30:43 (3988): No heartbeat from core client for 30 sec - exiting 23:30:44 (3988): No heartbeat from core client for 30 sec - exiting 23:30:45 (3988): No heartbeat from core client for 30 sec - exiting 23:30:46 (3988): No heartbeat from core client for 30 sec - exiting 23:30:47 (3988): No heartbeat from core client for 30 sec - exiting 23:30:48 (3988): No heartbeat from core client for 30 sec - exiting 23:30:49 (3988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:30:50 (3988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:30:15 (3910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:30:17 (3910): No heartbeat from core client for 30 sec - exiting 23:30:18 (3910): No heartbeat from core client for 30 sec - exiting 23:30:19 (3910): No heartbeat from core client for 30 sec - exiting 23:30:20 (3910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:24:30 (3944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:15:15 (3912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:39:40 (3733): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:23:11 (3814): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:55:03 (3959): No heartbeat from core client for 30 sec - exiting 08:55:06 (3959): No heartbeat from core client for 30 sec - exiting 08:55:07 (3959): No heartbeat from core client for 30 sec - exiting 08:55:08 (3959): No heartbeat from core client for 30 sec - exiting 08:55:09 (3959): No heartbeat from core client for 30 sec - exiting 08:55:10 (3959): No heartbeat from core client for 30 sec - exiting 08:55:11 (3959): No heartbeat from core client for 30 sec - exiting 08:55:12 (3959): No heartbeat from core client for 30 sec - exiting 08:55:13 (3959): No heartbeat from core client for 30 sec - exiting 08:55:14 (3959): No heartbeat from core client for 30 sec - exiting 08:55:15 (3959): No heartbeat from core client for 30 sec - exiting 08:55:16 (3959): No heartbeat from core client for 30 sec - exiting 08:55:17 (3959): No heartbeat from core client for 30 sec - exiting 08:55:18 (3959): No heartbeat from core client for 30 sec - exiting 08:55:19 (3959): No heartbeat from core client for 30 sec - exiting 08:55:20 (3959): No heartbeat from core client for 30 sec - exiting 08:55:21 (3959): No heartbeat from core client for 30 sec - exiting 08:55:22 (3959): No heartbeat from core client for 30 sec - exiting 08:55:23 (3959): No heartbeat from core client for 30 sec - exiting 08:55:24 (3959): No heartbeat from core client for 30 sec - exiting 08:55:25 (3959): No heartbeat from core client for 30 sec - exiting 08:55:26 (3959): No heartbeat from core client for 30 sec - exiting 08:55:27 (3959): No heartbeat from core client for 30 sec - exiting 08:55:28 (3959): No heartbeat from core client for 30 sec - exiting 08:55:29 (3959): No heartbeat from core client for 30 sec - exiting 08:55:30 (3959): No heartbeat from core client for 30 sec - exiting 08:55:31 (3959): No heartbeat from core client for 30 sec - exiting 08:55:32 (3959): No heartbeat from core client for 30 sec - exiting 08:55:33 (3959): No heartbeat from core client for 30 sec - exiting 08:55:34 (3959): No heartbeat from core client for 30 sec - exiting 08:55:35 (3959): No heartbeat from core client for 30 sec - exiting 08:55:36 (3959): No heartbeat from core client for 30 sec - exiting 08:55:37 (3959): No heartbeat from core client for 30 sec - exiting 08:55:38 (3959): No heartbeat from core client for 30 sec - exiting 08:55:39 (3959): No heartbeat from core client for 30 sec - exiting 08:55:40 (3959): No heartbeat from core client for 30 sec - exiting 08:55:41 (3959): No heartbeat from core client for 30 sec - exiting 08:55:42 (3959): No heartbeat from core client for 30 sec - exiting 08:55:43 (3959): No heartbeat from core client for 30 sec - exiting 08:55:44 (3959): No heartbeat from core client for 30 sec - exiting 08:55:45 (3959): No heartbeat from core client for 30 sec - exiting 08:55:46 (3959): No heartbeat from core client for 30 sec - exiting 08:55:47 (3959): No heartbeat from core client for 30 sec - exiting 08:55:48 (3959): No heartbeat from core client for 30 sec - exiting 08:55:49 (3959): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:14:32 (4096): No heartbeat from core client for 30 sec - exiting 22:14:33 (4096): No heartbeat from core client for 30 sec - exiting 22:14:34 (4096): No heartbeat from core client for 30 sec - exiting 22:14:35 (4096): No heartbeat from core client for 30 sec - exiting 22:14:36 (4096): No heartbeat from core client for 30 sec - exiting 22:14:37 (4096): No heartbeat from core client for 30 sec - exiting 22:14:38 (4096): No heartbeat from core client for 30 sec - exiting 22:14:39 (4096): No heartbeat from core client for 30 sec - exiting 22:14:40 (4096): No heartbeat from core client for 30 sec - exiting 22:14:41 (4096): No heartbeat from core client for 30 sec - exiting 22:14:42 (4096): No heartbeat from core client for 30 sec - exiting 22:14:43 (4096): No heartbeat from core client for 30 sec - exiting 22:14:44 (4096): No heartbeat from core client for 30 sec - exiting 22:14:45 (4096): No heartbeat from core client for 30 sec - exiting 22:14:46 (4096): No heartbeat from core client for 30 sec - exiting 22:14:47 (4096): No heartbeat from core client for 30 sec - exiting 22:14:48 (4096): No heartbeat from core client for 30 sec - exiting 22:14:49 (4096): No heartbeat from core client for 30 sec - exiting 22:14:50 (4096): No heartbeat from core client for 30 sec - exiting 22:14:51 (4096): No heartbeat from core client for 30 sec - exiting 22:14:52 (4096): No heartbeat from core client for 30 sec - exiting 22:14:54 (4096): No heartbeat from core client for 30 sec - exiting 22:14:55 (4096): No heartbeat from core client for 30 sec - exiting 22:14:56 (4096): No heartbeat from core client for 30 sec - exiting 22:14:57 (4096): No heartbeat from core client for 30 sec - exiting 22:14:58 (4096): No heartbeat from core client for 30 sec - exiting 22:14:59 (4096): No heartbeat from core client for 30 sec - exiting 22:15:00 (4096): No heartbeat from core client for 30 sec - exiting 22:15:01 (4096): No heartbeat from core client for 30 sec - exiting 22:15:02 (4096): No heartbeat from core client for 30 sec - exiting 22:15:03 (4096): No heartbeat from core client for 30 sec - exiting 22:15:04 (4096): No heartbeat from core client for 30 sec - exiting 22:15:05 (4096): No heartbeat from core client for 30 sec - exiting 22:15:06 (4096): No heartbeat from core client for 30 sec - exiting 22:15:07 (4096): No heartbeat from core client for 30 sec - exiting 22:15:08 (4096): No heartbeat from core client for 30 sec - exiting 22:15:09 (4096): No heartbeat from core client for 30 sec - exiting 22:15:10 (4096): No heartbeat from core client for 30 sec - exiting 22:15:11 (4096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:30:23 (3848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:11 (5571): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:25:41 (12441): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:11:34 (7984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:51 (13161): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:37:55 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:10:58 (3907): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7726400] [0xf7726425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75121df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7515825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74fd4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8761, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76f9400] [0xf76f9425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf74e51df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf74e8825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74d04d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8761, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77d4400] [0xf77d4425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75c01df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75c3825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75ab4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8761, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf774f400] [0xf774f425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf753b1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf753e825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75264d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8761, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77d0400] [0xf77d0425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75bc1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75bf825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75a74d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8761, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7771400] [0xf7771425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf755d1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7560825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75484d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8761, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
04 Oct 2013 00:22:03 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 596,160 | 981,486 | 1.6463 |
03 Oct 2013 10:47:11 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 570,240 | 939,144 | 1.6469 |
02 Oct 2013 18:44:13 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 544,320 | 895,867 | 1.6458 |
02 Oct 2013 05:17:27 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 518,400 | 852,718 | 1.6449 |
01 Oct 2013 12:52:52 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 492,480 | 809,973 | 1.6447 |
30 Sep 2013 20:04:44 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 466,560 | 767,222 | 1.6444 |
30 Sep 2013 04:28:16 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 440,640 | 724,433 | 1.6440 |
29 Sep 2013 14:36:16 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 414,720 | 681,925 | 1.6443 |
28 Sep 2013 21:45:22 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 388,800 | 638,843 | 1.6431 |
28 Sep 2013 04:02:37 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 362,880 | 596,691 | 1.6443 |
27 Sep 2013 06:11:18 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 336,960 | 554,429 | 1.6454 |
26 Sep 2013 14:50:27 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 311,040 | 511,262 | 1.6437 |
25 Sep 2013 16:45:06 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 285,120 | 467,718 | 1.6404 |
25 Sep 2013 09:24:11 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 259,200 | 424,889 | 1.6392 |
24 Sep 2013 03:16:30 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 233,280 | 381,862 | 1.6369 |
23 Sep 2013 10:03:45 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 207,360 | 339,536 | 1.6374 |
23 Sep 2013 10:03:45 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 181,440 | 296,959 | 1.6367 |
21 Sep 2013 17:14:43 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 155,520 | 255,713 | 1.6442 |
20 Sep 2013 22:37:27 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 129,600 | 213,168 | 1.6448 |
20 Sep 2013 05:28:04 | 1223320 | 15928260 | hadcm3n_4h8q_2020_40_008406722_0 | 103,680 | 171,805 | 1.6571 |
©2024 cpdn.org