Name | hadcm3n_3dq0_1980_40_008366680_2 |
Workunit | 8517539 |
Created | 14 Aug 2013, 16:19:09 UTC |
Sent | 14 Aug 2013, 19:12:27 UTC |
Report deadline | 14 Nov 2013, 2:39:38 UTC |
Received | 13 Sep 2013, 1:02:20 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 22 (0x00000016) Unknown error code |
Computer ID | 1231328 |
Run time | 16 days 6 hours 25 min 44 sec |
CPU time | 9 days 18 hours 34 min 30 sec |
Validate state | Invalid |
Credit | 5,287.68 |
Device peak FLOPS | 3.01 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin |
Stderr | <core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 03:36:30 (70541): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:36:31 (70541): No heartbeat from core client for 30 sec - exiting 03:36:32 (70541): No heartbeat from core client for 30 sec - exiting 03:36:33 (70541): No heartbeat from core client for 30 sec - exiting 03:36:34 (70541): No heartbeat from core client for 30 sec - exiting 03:36:35 (70541): No heartbeat from core client for 30 sec - exiting 03:36:36 (70541): No heartbeat from core client for 30 sec - exiting 03:36:37 (70541): No heartbeat from core client for 30 sec - exiting 03:36:38 (70541): No heartbeat from core client for 30 sec - exiting 03:36:39 (70541): No heartbeat from core client for 30 sec - exiting 03:36:40 (70541): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:50:06 (489): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:50:07 (489): No heartbeat from core client for 30 sec - exiting 07:50:08 (489): No heartbeat from core client for 30 sec - exiting 07:50:09 (489): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:43:55 (20440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:43:57 (20440): No heartbeat from core client for 30 sec - exiting 03:43:58 (20440): No heartbeat from core client for 30 sec - exiting 03:43:59 (20440): No heartbeat from core client for 30 sec - exiting 03:44:00 (20440): No heartbeat from core client for 30 sec - exiting 03:44:01 (20440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:17:23 (41129): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:18:25 (43284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:18:27 (43284): No heartbeat from core client for 30 sec - exiting 03:19:12 (44503): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:53:21 (61063): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:53:23 (61063): No heartbeat from core client for 30 sec - exiting 07:53:24 (61063): No heartbeat from core client for 30 sec - exiting 07:53:25 (61063): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 00:50:24 (70958): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:49:58 (1645): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:24:05 (2865): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:19:30 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:36:21 (11634): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:38:24 (13115): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:38:26 (13115): No heartbeat from core client for 30 sec - exiting 05:38:27 (13115): No heartbeat from core client for 30 sec - exiting 05:38:28 (13115): No heartbeat from core client for 30 sec - exiting 05:38:29 (13115): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:05:31 (31470): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:05:33 (31470): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:48:28 (71677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:10:07 (74032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:36:02 (81748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:56:19 (83255): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:56:21 (83255): No heartbeat from core client for 30 sec - exiting 02:56:22 (83255): No heartbeat from core client for 30 sec - exiting 02:56:23 (83255): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:28:55 (91578): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:28:58 (91578): No heartbeat from core client for 30 sec - exiting 19:28:59 (91578): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:37:04 (1554): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:27:11 (3438): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:42:18 (3786): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:42:20 (3786): No heartbeat from core client for 30 sec - exiting 00:42:21 (3786): No heartbeat from core client for 30 sec - exiting 00:42:22 (3786): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:20:53 (31169): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:33:18 (41921): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Sep 11 21:33:31 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x95089c5a __kill + 10 1 libSystem.B.dylib 0x9511c5a5 raise + 26 2 libSystem.B.dylib 0x951326e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x95028c0e mach_wait_until + 10 1 libSystem.B.dylib 0x950b0429 nanosleep + 345 2 libSystem.B.dylib 0x950b02ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void*) + 78 5 libSystem.B.dylib 0x95056259 _pthread_start + 345 6 libSystem.B.dylib 0x950560de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib 0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib 0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Sep 11 21:33:35 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x95089c5a __kill + 10 1 libSystem.B.dylib 0x9511c5a5 raise + 26 2 libSystem.B.dylib 0x951326e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x95028c0e mach_wait_until + 10 1 libSystem.B.dylib 0x950b0429 nanosleep + 345 2 libSystem.B.dylib 0x950b02ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void*) + 78 5 libSystem.B.dylib 0x95056259 _pthread_start + 345 6 libSystem.B.dylib 0x950560de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib 0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib 0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Sep 11 21:33:38 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x95089c5a __kill + 10 1 libSystem.B.dylib 0x9511c5a5 raise + 26 2 libSystem.B.dylib 0x951326e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x95028c0e mach_wait_until + 10 1 libSystem.B.dylib 0x950b0429 nanosleep + 345 2 libSystem.B.dylib 0x950b02ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void*) + 78 5 libSystem.B.dylib 0x95056259 _pthread_start + 345 6 libSystem.B.dylib 0x950560de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib 0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib 0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Sep 11 21:33:41 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x95089c5a __kill + 10 1 libSystem.B.dylib 0x9511c5a5 raise + 26 2 libSystem.B.dylib 0x951326e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x95028c0e mach_wait_until + 10 1 libSystem.B.dylib 0x950b0429 nanosleep + 345 2 libSystem.B.dylib 0x950b02ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void*) + 78 5 libSystem.B.dylib 0x95056259 _pthread_start + 345 6 libSystem.B.dylib 0x950560de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib 0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib 0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Sep 11 21:33:44 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x95089c5a __kill + 10 1 libSystem.B.dylib 0x9511c5a5 raise + 26 2 libSystem.B.dylib 0x951326e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x95028c0e mach_wait_until + 10 1 libSystem.B.dylib 0x950b0429 nanosleep + 345 2 libSystem.B.dylib 0x950b02ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void*) + 78 5 libSystem.B.dylib 0x95056259 _pthread_start + 345 6 libSystem.B.dylib 0x950560de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib 0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib 0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Sep 11 21:33:46 2013 Thread 0 Crashed: CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Thu Sep 12 18:00:02 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x95089c5a __kill + 10 1 libSystem.B.dylib 0x9511c5a5 raise + 26 2 libSystem.B.dylib 0x951326e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x95028c0e mach_wait_until + 10 1 libSystem.B.dylib 0x950b0429 nanosleep + 345 2 libSystem.B.dylib 0x950b02ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void*) + 78 5 libSystem.B.dylib 0x95056259 _pthread_start + 345 6 libSystem.B.dylib 0x950560de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib 0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib 0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=51866, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
11 Sep 2013 01:34:12 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 440,640 | 808,642 | 1.8352 |
09 Sep 2013 06:52:37 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 414,720 | 760,641 | 1.8341 |
07 Sep 2013 12:04:54 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 388,800 | 712,245 | 1.8319 |
06 Sep 2013 03:25:06 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 362,880 | 664,031 | 1.8299 |
04 Sep 2013 08:56:46 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 336,960 | 616,020 | 1.8282 |
02 Sep 2013 14:21:35 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 311,040 | 567,862 | 1.8257 |
01 Sep 2013 05:23:00 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 285,120 | 519,324 | 1.8214 |
30 Aug 2013 12:42:04 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 259,200 | 473,653 | 1.8274 |
29 Aug 2013 04:53:14 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 233,280 | 425,498 | 1.8240 |
27 Aug 2013 08:35:39 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 207,360 | 376,594 | 1.8161 |
25 Aug 2013 14:59:22 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 181,440 | 328,805 | 1.8122 |
24 Aug 2013 06:02:29 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 155,520 | 280,749 | 1.8052 |
22 Aug 2013 15:26:36 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 129,600 | 233,840 | 1.8043 |
21 Aug 2013 06:31:14 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 103,680 | 189,983 | 1.8324 |
19 Aug 2013 14:40:53 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 77,760 | 145,870 | 1.8759 |
18 Aug 2013 05:24:00 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 51,840 | 97,175 | 1.8745 |
16 Aug 2013 10:39:57 | 1231328 | 15918562 | hadcm3n_3dq0_1980_40_008366680_2 | 25,920 | 48,650 | 1.8769 |
©2024 cpdn.org