Task 18282700

Name	hadam3p_anz_f2c5_2013_1_009729166_0
Workunit	9801011
Created	8 Apr 2015, 19:49:12 UTC
Sent	9 Apr 2015, 23:56:10 UTC
Report deadline	22 Mar 2016, 5:16:10 UTC
Received	19 Apr 2015, 15:17:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1333395
Run time	5 days 12 hours 4 min 58 sec
CPU time	5 days 0 hours 14 min 59 sec
Validate state	Invalid
Credit	4,484.28
Device peak FLOPS	3.83 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.4.42</core_client_version> <![CDATA[ <stderr_txt> 10:00:43 (31140): start_timer_thread(): CreateThread() failed, errno 0 10:00:44 (25356): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:45:19 (15304): start_timer_thread(): CreateThread() failed, errno 0 16:45:20 (14520): start_timer_thread(): CreateThread() failed, errno 0 16:55:05 (12648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:31 (15528): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 17:10:12 (15792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:31 (11144): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 11:20:34 (11324): No heartbeat from core client for 30 sec - exiting 11:20:35 (11324): No heartbeat from core client for 30 sec - exiting 11:20:36 (11324): No heartbeat from core client for 30 sec - exiting 11:20:37 (11324): No heartbeat from core client for 30 sec - exiting 11:20:38 (11324): No heartbeat from core client for 30 sec - exiting 11:20:39 (11324): No heartbeat from core client for 30 sec - exiting 11:20:40 (11324): No heartbeat from core client for 30 sec - exiting 11:20:41 (11324): No heartbeat from core client for 30 sec - exiting 11:20:42 (11324): No heartbeat from core client for 30 sec - exiting 11:20:43 (11324): No heartbeat from core client for 30 sec - exiting 11:20:44 (11324): No heartbeat from core client for 30 sec - exiting 11:20:45 (11324): No heartbeat from core client for 30 sec - exiting 11:20:46 (11324): No heartbeat from core client for 30 sec - exiting 11:20:47 (11324): No heartbeat from core client for 30 sec - exiting 11:20:48 (11324): No heartbeat from core client for 30 sec - exiting 11:20:49 (11324): No heartbeat from core client for 30 sec - exiting 11:20:50 (11324): No heartbeat from core client for 30 sec - exiting 11:20:51 (11324): No heartbeat from core client for 30 sec - exiting 11:20:52 (11324): No heartbeat from core client for 30 sec - exiting 11:20:53 (11324): No heartbeat from core client for 30 sec - exiting 11:20:54 (11324): No heartbeat from core client for 30 sec - exiting 11:20:55 (11324): No heartbeat from core client for 30 sec - exiting 11:20:56 (11324): No heartbeat from core client for 30 sec - exiting 11:20:57 (11324): No heartbeat from core client for 30 sec - exiting 11:20:58 (11324): No heartbeat from core client for 30 sec - exiting 11:20:59 (11324): No heartbeat from core client for 30 sec - exiting 11:21:00 (11324): No heartbeat from core client for 30 sec - exiting 11:21:01 (11324): No heartbeat from core client for 30 sec - exiting 11:21:02 (14356): start_timer_thread(): CreateThread() failed, errno 0 11:21:02 (11324): No heartbeat from core client for 30 sec - exiting 11:21:03 (15064): start_timer_thread(): CreateThread() failed, errno 0 11:21:03 (11324): No heartbeat from core client for 30 sec - exiting 11:21:04 (11324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:21:18 (12320): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 11:23:45 (15084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:22 (15792): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:27:43 (14768): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 11:29:15 (8516): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:15:36 (14420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:15:54 (11972): start_timer_thread(): CreateThread() failed, errno 0 12:15:56 (11744): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 15:40:35 (11828): start_timer_thread(): CreateThread() failed, errno 0 15:40:36 (9092): start_timer_thread(): CreateThread() failed, errno 0 18:20:24 (4656): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:40:34 (8568): start_timer_thread(): CreateThread() failed, errno 0 12:40:36 (12452): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8568, selfPID=12452, iMonCtr=1 No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8568, selfPID=8568, iMonCtr=1 10:49:40 (13768): start_timer_thread(): CreateThread() failed, errno 0 10:49:41 (14352): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:27:06 (24368): start_timer_thread(): CreateThread() failed, errno 0 20:27:07 (9080): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:21:57 (15332): start_timer_thread(): CreateThread() failed, errno 0 22:21:59 (20956): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:14:35 (27972): start_timer_thread(): CreateThread() failed, errno 0 21:14:37 (18368): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 02:32:07 (29388): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16620, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 10:58:36 (10020): start_timer_thread(): CreateThread() failed, errno 0 10:58:38 (12644): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:39:13 (9964): start_timer_thread(): CreateThread() failed, errno 0 21:39:15 (15336): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 00:27:32 (5496): start_timer_thread(): CreateThread() failed, errno 0 00:27:33 (13772): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:55:38 (23028): start_timer_thread(): CreateThread() failed, errno 0 10:55:39 (26340): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:46:12 (18568): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:29:10 (13228): start_timer_thread(): CreateThread() failed, errno 0 08:29:11 (13408): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:07:49 (6064): start_timer_thread(): CreateThread() failed, errno 0 16:07:51 (11176): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6064, selfPID=6064, iMonCtr=1 No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6064, selfPID=11176, iMonCtr=1 13:40:06 (11056): start_timer_thread(): CreateThread() failed, errno 0 13:40:08 (9348): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:36:17 (3592): start_timer_thread(): CreateThread() failed, errno 0 14:36:18 (13464): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 03:22:50 (4176): start_timer_thread(): CreateThread() failed, errno 0 03:22:51 (15232): start_timer_thread(): CreateThread() failed, errno 0 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4176, selfPID=4176, iMonCtr=1 No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4176, selfPID=15232, iMonCtr=1 10:36:29 (12576): start_timer_thread(): CreateThread() failed, errno 0 10:36:29 (14600): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 11:36:41 (18344): start_timer_thread(): CreateThread() failed, errno 0 11:36:43 (17676): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 04:18:12 (8740): start_timer_thread(): CreateThread() failed, errno 0 04:18:13 (13716): start_timer_thread(): CreateThread() failed, errno 0 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8740, selfPID=8740, iMonCtr=1 No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8740, selfPID=13716, iMonCtr=1 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 10:27:51 (13856): start_timer_thread(): CreateThread() failed, errno 0 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 10:27:52 (10468): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13856, selfPID=13856, iMonCtr=1 No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13856, selfPID=10468, iMonCtr=1 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 00:15:57 (10936): start_timer_thread(): CreateThread() failed, errno 0 Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=10936, iMonCtr=1 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 00:16:07 (724): start_timer_thread(): CreateThread() failed, errno 0 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=724, selfPID=11864, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_f2c5_2013_1_009729166_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_f2c5_2013_1_009729166_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_f2c5_2013_1_009729166_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Apr 2015 00:01:29	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	103,979	424,615	4.0837
16 Apr 2015 21:07:39	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	92,459	376,935	4.0768
16 Apr 2015 06:35:52	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	80,939	328,868	4.0632
15 Apr 2015 04:15:50	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	69,419	281,026	4.0483
14 Apr 2015 07:33:02	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	57,899	233,558	4.0339
13 Apr 2015 17:05:07	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	46,379	187,051	4.0331
13 Apr 2015 03:48:07	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	34,859	140,346	4.0261
12 Apr 2015 14:33:26	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	23,339	93,565	4.0090
11 Apr 2015 13:59:46	1333395	18282700	hadam3p_anz_f2c5_2013_1_009729166_0	11,819	46,387	3.9248