climateprediction.net home page
Task 15870264

Task 15870264

Name hadcm3n_u548_2020_40_008339194_2
Workunit 8490055
Created 28 Jun 2013, 2:43:14 UTC
Sent 28 Jun 2013, 3:01:12 UTC
Report deadline 27 Sep 2013, 10:28:23 UTC
Received 11 Jul 2013, 4:14:47 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status 203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID 1259513
Run time 4 days 22 hours 23 min 10 sec
CPU time 4 days 13 hours 1 min 6 sec
Validate state Invalid
Credit 3,732.48
Device peak FLOPS 3.06 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
windows_intelx86
Stderr 
<core_client_version>7.0.28</core_client_version>
<![CDATA[
<message>
aborted by user
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 07/03/13 08:47:43
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_u548_2020_40_008339194;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 70202, Write: 0, Other 155445

- I/O Transfers Counters -
Read: 0, Write: 5298, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5568, QuotaPeakNonPagedPoolUsage: 5632

- Virtual Memory Usage -
VirtualSize: 1051308032, PeakVirtualSize: 1051308032

- Pagefile Usage -
PagefileUsage: 231612416, PeakPagefileUsage: 231612416

- Working Set Size -
WorkingSetSize: 226652160, PeakWorkingSetSize: 226652160, PageFaultCount: 58931

*** Dump of thread ID 2844 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 84375000.000000, User Time: 53356720128.000000, Wait Time: 540778.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=00000007 ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f52baa0 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40b8afab 40b1926e 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001224ef 1132fb24 00122451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 2664 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 540772.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 07/03/13 09:22:39
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_u548_2020_40_008339194;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 10745, Write: 0, Other 28077

- I/O Transfers Counters -
Read: 0, Write: 750, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5568, QuotaPeakNonPagedPoolUsage: 5632

- Virtual Memory Usage -
VirtualSize: 1051308032, PeakVirtualSize: 1051308032

- Pagefile Usage -
PagefileUsage: 231612416, PeakPagefileUsage: 231612416

- Working Set Size -
WorkingSetSize: 226639872, PeakWorkingSetSize: 226639872, PageFaultCount: 55701

*** Dump of thread ID 2492 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 11718750.000000, User Time: 7022343680.000000, Wait Time: 674878.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=00000007 ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f52baa0 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40afa486 40ac0f28 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001224ef 1132fb24 00122451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 2348 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 674873.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 07/03/13 09:23:58
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_u548_2020_40_008339194;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 2603, Write: 0, Other 10454

- I/O Transfers Counters -
Read: 0, Write: 474, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5568, QuotaPeakNonPagedPoolUsage: 5632

- Virtual Memory Usage -
VirtualSize: 1051308032, PeakVirtualSize: 1051308032

- Pagefile Usage -
PagefileUsage: 231587840, PeakPagefileUsage: 231587840

- Working Set Size -
WorkingSetSize: 226586624, PeakWorkingSetSize: 226586624, PageFaultCount: 55225

*** Dump of thread ID 648 (state: Ready): ***

- Information -
Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 2656250.000000, User Time: 772812480.000000, Wait Time: 679960.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=00000007 ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f52baa0 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40afc8c2 40acdf29 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001224ef 1132fb24 00122451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 2220 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 679960.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 07/03/13 09:25:13
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_u548_2020_40_008339194;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 2603, Write: 0, Other 10438

- I/O Transfers Counters -
Read: 0, Write: 288, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5568, QuotaPeakNonPagedPoolUsage: 5632

- Virtual Memory Usage -
VirtualSize: 1051308032, PeakVirtualSize: 1051308032

- Pagefile Usage -
PagefileUsage: 231587840, PeakPagefileUsage: 231587840

- Working Set Size -
WorkingSetSize: 226611200, PeakWorkingSetSize: 226611200, PageFaultCount: 55231

*** Dump of thread ID 1648 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 2812500.000000, User Time: 713750016.000000, Wait Time: 684762.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=0000000f ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f531540 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40afb5c5 40ac75b9 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001224ef 1132fb24 00122451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 2912 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 684761.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 07/03/13 09:42:07
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_u548_2020_40_008339194;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 14415, Write: 0, Other 35691

- I/O Transfers Counters -
Read: 0, Write: 1130, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5568, QuotaPeakNonPagedPoolUsage: 5632

- Virtual Memory Usage -
VirtualSize: 1051308032, PeakVirtualSize: 1051308032

- Pagefile Usage -
PagefileUsage: 231591936, PeakPagefileUsage: 231604224

- Working Set Size -
WorkingSetSize: 226627584, PeakWorkingSetSize: 226627584, PageFaultCount: 55902

*** Dump of thread ID 2940 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 15312500.000000, User Time: 9984531456.000000, Wait Time: 749615.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=00000007 ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f52baa0 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40b54c66 40ac0482 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001224ef 1132fb24 00122451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 2656 (state: Ready): ***

- Information -
Status: Base Priority: Unknown, Priority: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 749597.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 07/03/13 10:41:41
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_u548_2020_40_008339194;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 47554, Write: 0, Other 107007

- I/O Transfers Counters -
Read: 0, Write: 3504, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5552, QuotaPeakNonPagedPoolUsage: 5632

- Virtual Memory Usage -
VirtualSize: 1051308032, PeakVirtualSize: 1051308032

- Pagefile Usage -
PagefileUsage: 231620608, PeakPagefileUsage: 231620608

- Working Set Size -
WorkingSetSize: 226680832, PeakWorkingSetSize: 226680832, PageFaultCount: 57740

*** Dump of thread ID 3372 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 52187500.000000, User Time: 35307032576.000000, Wait Time: 978380.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=00000007 ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f52baa0 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40ba1e1e 40b798de 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001224ef 1132fb24 00122451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 3360 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 978377.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 07/03/13 10:42:47
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_u548_2020_40_008339194;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 2287, Write: 0, Other 10329

- I/O Transfers Counters -
Read: 0, Write: 470, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5552, QuotaPeakNonPagedPoolUsage: 5632

- Virtual Memory Usage -
VirtualSize: 1051308032, PeakVirtualSize: 1051308032

- Pagefile Usage -
PagefileUsage: 231587840, PeakPagefileUsage: 231587840

- Working Set Size -
WorkingSetSize: 226566144, PeakWorkingSetSize: 226566144, PageFaultCount: 55220

*** Dump of thread ID 3744 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 3593750.000000, User Time: 634062528.000000, Wait Time: 982612.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=00000007 ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f52baa0 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40b9bbbd 40b7570d 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001224ef 1132fb24 00122451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 3508 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 982607.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2628, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2628, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2576, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1588, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2452, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2452, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=700, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Abort request from BOINC...
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Jul 2013 10:51:20 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 311,040 367,737 1.1823
09 Jul 2013 18:54:43 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 285,120 335,730 1.1775
09 Jul 2013 10:08:00 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 259,200 303,800 1.1721
09 Jul 2013 01:09:13 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 233,280 276,244 1.1842
08 Jul 2013 12:48:23 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 207,360 243,128 1.1725
07 Jul 2013 19:27:34 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 181,440 210,094 1.1579
07 Jul 2013 10:16:23 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 155,520 177,588 1.1419
06 Jul 2013 18:52:15 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 129,600 144,864 1.1178
06 Jul 2013 11:06:54 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 103,680 111,844 1.0787
06 Jul 2013 05:08:09 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 77,760 83,399 1.0725
06 Jul 2013 04:36:24 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 51,840 53,799 1.0378
04 Jul 2013 14:25:11 1259513 15870264 hadcm3n_u548_2020_40_008339194_2 25,920 21,310 0.8221


©2024 cpdn.org