BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Sunset Ride 
Sunset Ride 

General Notice: BOB is going to retire...please see details here. All BOB forums have been converted to "read only" as of September 1st.
General Notice: No events within the next 45 days.

Benchmark YOUR hardware and compare the results V1.1
1 members found this topic helpful
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
 
Search this topic... | Search DI: Performance and Tuning... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Author Message
dlinstedt
Forum Member
Forum Member



Joined: 24 Jun 2007

Posts: 6



PostPosted: Fri Sep 21, 2007 10:34 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

wdaehn, please contact me - I'd love to chat with you over e-mail, or phone... wondering how we can work together going forward.

Thanks,
Dan Linstedt
Back to top
buckshot
Senior Member
Senior Member



Joined: 09 Aug 2007

Posts: 42
Location: Australia



PostPosted: Thu Jan 17, 2008 2:51 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Source DB : Oracle 9i running on Windows 2003 standard VirtualServer 3Gig RAM
Network Source to DI = 1GBitEth
DI Server : Windows 2003 Standard SP1, 4 * Intel Xeon CPU 3.6Ghz 3.5Gig RAM
Network DI to Target = 1GbitEth
Target DB Oracle 9i running on Windows 2003 Enterprise SP2 server 4 * Zeon 5160 @ 3Ghz 12Gig RAM



(11.7) 01-17-08 17:03:36 (8056:0400) DATAFLOW: Process to execute data flow <DF_Benchmark_Setup> is started.
(11.7) 01-17-08 17:04:30 (1244:6432) DATAFLOW: Process to execute data flow <DF_Benchmark_read> is completed.
(11.7) 01-17-08 17:04:30 (6596:5544) PRINTFN: Time______________________________ 52 secounds
(11.7) 01-17-08 17:09:18 (7724:1312) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader> is completed.
(11.7) 01-17-08 17:09:18 (6596:5544) PRINTFN: Time______________________________ 287 secounds
(11.7) 01-17-08 17:14:28 (7864:6392) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load> is completed.
(11.7) 01-17-08 17:14:28 (6596:5544) PRINTFN: Time______________________________ 309 secounds
(11.7) 01-17-08 17:14:52 (6784:2036) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread> is completed.
(11.7) 01-17-08 17:14:52 (6596:5544) PRINTFN: Time______________________________ 24 secounds
(11.7) 01-17-08 17:16:53 (6236:5068) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1> is completed.
(11.7) 01-17-08 17:16:53 (6596:5544) PRINTFN: Time______________________________ 120 secounds
(11.7) 01-17-08 17:18:13 (6396:2712) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10> is completed.
(11.7) 01-17-08 17:18:13 (6596:5544) PRINTFN: Time______________________________ 79 secounds
(11.7) 01-17-08 17:18:13 (6596:5544) JOB: Job <Job_Benchmark> is completed successfully.


is this any good ?

Buck
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Fri Jan 25, 2008 6:20 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Your read over the network is just 52 secs? I'd say that is physically impossible. Where is your source database? I understand it is hosted on a VMWare but where is that VMWare running? Same server as DI jobserver is, isn't it?

The regular loads are good, the API bulkloader is about the same speed. That means your target database is running in archive log mode and the table is not set to "unrecoverable". Execute the statement "alter table target NOLOGGING" and run the API test again. You should see great improvement.

The lookup_DOP1 I don't like. Is it possible the DF is set to "pageable cache"? Change that DF property to "in-memory" and when executing the job disable the options to collect and use collected statistics.

I have updated the result page: http://diamond.businessobjects.com/node/5074

_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Wed Feb 06, 2008 3:04 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

@buckshot: Any update?
_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
buckshot
Senior Member
Senior Member



Joined: 09 Aug 2007

Posts: 42
Location: Australia



PostPosted: Wed Feb 13, 2008 12:42 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Sorry Werner I forgot to look back here !

As stated in last post, Source, Target and Jobserver are all on different servers.
Source is a virtual server, target is a 'real' server and jobserver is a 'real' server.

Even faster over the network this time! I double checked with one of our network guys and the network is 1GBitEth
maybe we have our packet sizes or something set optimally for large data transfer?

(11.7) 02-13-08 15:25:14 (6876:7480) DATAFLOW: Process to execute data flow <DF_Benchmark_read> is completed.
(11.7) 02-13-08 15:25:14 (5396:5964) PRINTFN: Time______________________________ 44 secounds
(11.7) 02-13-08 15:28:42 (8112:3196) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader> is completed.
(11.7) 02-13-08 15:28:42 (5396:5964) PRINTFN: Time______________________________ 207 secounds
(11.7) 02-13-08 15:33:24 (6596:6252) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load> is completed.
(11.7) 02-13-08 15:33:24 (5396:5964) PRINTFN: Time______________________________ 281 secounds
(11.7) 02-13-08 15:33:47 (0780:2268) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread> is completed.
(11.7) 02-13-08 15:33:47 (5396:5964) PRINTFN: Time______________________________ 22 secounds
(11.7) 02-13-08 15:34:47 (7416:6432) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1> is completed.
(11.7) 02-13-08 15:34:47 (5396:5964) PRINTFN: Time______________________________ 59 secounds
(11.7) 02-13-08 15:35:59 (0780:8012) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10> is completed.
(11.7) 02-13-08 15:35:59 (5396:5964) PRINTFN: Time______________________________ 72 secounds
(11.7) 02-13-08 15:35:59 (5396:5964) JOB: Job <Job_Benchmark> is completed successfully.


Thanks for the tips!

BuckShot.
Back to top
gpn01
Principal Member
Principal Member



Joined: 30 Nov 2004

Posts: 224
Location: London, UK


flag
PostPosted: Tue Apr 15, 2008 5:00 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Name of the Test: GPN01_VMWare

Source: SQL2005
Network source to DI: 1GBit Ethernet
DI server: Windows 2003 Enterprise SP2 32bit /1 CPU @ 2.33GHz/7.56GB/1Disk (VMWare)

DI: 11.5.3.9
Network DI to Target: shared with above
Target server: SQL2005
Target: SQL2005

RESULTS:
(11.5) 04-15-08 09:59:47 (3116:7552) JOB: Job <Job_Benchmark_MS> is started.
(11.5) 04-15-08 09:59:47 (1956:7012) DATAFLOW: Process to execute data flow <DF_Benchmark_Setup_MS> is started.
(11.5) 04-15-08 09:59:47 (1956:7012) DATAFLOW: Data flow <DF_Benchmark_Setup_MS> is started.
(11.5) 04-15-08 09:59:48 (1956:7012) DATAFLOW: Data flow <DF_Benchmark_Setup_MS> is completed successfully.
(11.5) 04-15-08 09:59:48 (1956:7012) DATAFLOW: Process to execute data flow <DF_Benchmark_Setup_MS> is completed.
(11.5) 04-15-08 09:59:49 (6716:7252) DATAFLOW: Process to execute data flow <DF_Benchmark_read_MS> is started.
(11.5) 04-15-08 09:59:49 (6716:7252) DATAFLOW: Data flow <DF_Benchmark_read_MS> is started.
(11.5) 04-15-08 10:07:54 (6716:7252) DATAFLOW: Data flow <DF_Benchmark_read_MS> is completed successfully.
(11.5) 04-15-08 10:07:54 (6716:7252) DATAFLOW: Process to execute data flow <DF_Benchmark_read_MS> is completed.
(11.5) 04-15-08 10:07:54 (3116:7552) PRINTFN: Time______________________________ 485 secounds
(11.5) 04-15-08 10:07:54 (2072:3172) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader_MS> is started.
(11.5) 04-15-08 10:07:55 (2072:3172) DATAFLOW: Data flow <DF_Benchmark_API_bulkloader_MS> is started.
(11.5) 04-15-08 10:29:53 (2072:3172) DATAFLOW: Data flow <DF_Benchmark_API_bulkloader_MS> is completed successfully.
(11.5) 04-15-08 10:29:53 (2072:3172) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader_MS> is completed.
(11.5) 04-15-08 10:29:53 (3116:7552) PRINTFN: Time______________________________ 1318 secounds
(11.5) 04-15-08 10:29:53 (2920:5416) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load_MS> is started.
(11.5) 04-15-08 10:29:54 (2920:5416) DATAFLOW: Data flow <DF_Benchmark_regular_load_MS> is started.
(11.5) 04-15-08 10:56:06 (2920:5416) DATAFLOW: Data flow <DF_Benchmark_regular_load_MS> is completed successfully.
(11.5) 04-15-08 10:56:06 (2920:5416) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load_MS> is completed.
(11.5) 04-15-08 10:56:06 (3116:7552) PRINTFN: Time______________________________ 1573 secounds
(11.5) 04-15-08 10:56:06 (5008:3104) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread_MS> is started.
(11.5) 04-15-08 10:56:07 (5008:3104) DATAFLOW: Data flow <DF_Benchmark_single_thread_MS> is started.
(11.5) 04-15-08 10:56:27 (5008:3104) DATAFLOW: Data flow <DF_Benchmark_single_thread_MS> is completed successfully.
(11.5) 04-15-08 10:56:27 (5008:3104) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread_MS> is completed.
(11.5) 04-15-08 10:56:28 (3116:7552) PRINTFN: Time______________________________ 21 secounds
(11.5) 04-15-08 10:56:28 (7680:7084) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1_MS> is started.
(11.5) 04-15-08 10:56:29 (7680:7084) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP1_MS> is started.
(11.5) 04-15-08 10:57:13 (7680:7084) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP1_MS> is completed successfully.
(11.5) 04-15-08 10:57:13 (7680:7084) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1_MS> is completed.
(11.5) 04-15-08 10:57:13 (3116:7552) PRINTFN: Time______________________________ 45 secounds
(11.5) 04-15-08 10:57:13 (4172:4660) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10_MS> is started.
(11.5) 04-15-08 10:57:14 (4172:4660) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP10_MS> is started.
(11.5) 04-15-08 10:58:00 (4172:4660) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP10_MS> is completed successfully.
(11.5) 04-15-08 10:58:00 (4172:4660) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10_MS> is completed.
(11.5) 04-15-08 10:58:00 (3116:7552) PRINTFN: Time______________________________ 46 secounds
(11.5) 04-15-08 10:58:00 (3116:7552) JOB: Job <Job_Benchmark_MS> is completed successfully.
Back to top
gpn01
Principal Member
Principal Member



Joined: 30 Nov 2004

Posts: 224
Location: London, UK


flag
PostPosted: Wed Apr 16, 2008 3:34 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Name of the Test: GPN01_Prod

Source: SQL2005
Network source to DI: 1GBit Ethernet
DI server: Windows 2003 Standard SP2 32bit /2 CPU @ 3.4GHz/3.5GB/1Disk

DI: 11.5.3.9
Network DI to Target: shared with above
Target server: SQL2005
Target: SQL2005

(Source & Target are same database)


(11.5) 04-16-08 08:40:52 (35604:37536) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 08:40:52 (35604:37536) DATAFLOW: Process to execute data flow <DF_Benchmark_read_MS> is started.
(11.5) 04-16-08 08:40:52 (35604:37536) DATAFLOW: Data flow <DF_Benchmark_read_MS> is started.
(11.5) 04-16-08 08:44:43 (35604:37536) DATAFLOW: Data flow <DF_Benchmark_read_MS> is completed successfully.
(11.5) 04-16-08 08:44:43 (35604:37536) DATAFLOW: Process to execute data flow <DF_Benchmark_read_MS> is completed.
(11.5) 04-16-08 08:44:43 (43424:45588) PRINTFN: Time______________________________ 231 secounds
(11.5) 04-16-08 08:44:43 (42372:30352) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 08:44:43 (42372:30352) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader_MS> is started.
(11.5) 04-16-08 08:44:43 (42372:30352) DATAFLOW: Data flow <DF_Benchmark_API_bulkloader_MS> is started.
(11.5) 04-16-08 09:07:42 (42372:30352) DATAFLOW: Data flow <DF_Benchmark_API_bulkloader_MS> is completed successfully.
(11.5) 04-16-08 09:07:42 (42372:30352) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader_MS> is completed.
(11.5) 04-16-08 09:07:42 (43424:45588) PRINTFN: Time______________________________ 1379 secounds
(11.5) 04-16-08 09:07:42 (45140:39264) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 09:07:42 (45140:39264) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load_MS> is started.
(11.5) 04-16-08 09:07:43 (45140:39264) DATAFLOW: Data flow <DF_Benchmark_regular_load_MS> is started.
(11.5) 04-16-08 09:26:41 (45140:39264) DATAFLOW: Data flow <DF_Benchmark_regular_load_MS> is completed successfully.
(11.5) 04-16-08 09:26:42 (45140:39264) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load_MS> is completed.
(11.5) 04-16-08 09:26:42 (43424:45588) PRINTFN: Time______________________________ 1139 secounds
(11.5) 04-16-08 09:26:43 (45808:40036) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 09:26:43 (45808:40036) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread_MS> is started.
(11.5) 04-16-08 09:26:43 (45808:40036) DATAFLOW: Data flow <DF_Benchmark_single_thread_MS> is started.
(11.5) 04-16-08 09:27:12 (45808:40036) DATAFLOW: Data flow <DF_Benchmark_single_thread_MS> is completed successfully.
(11.5) 04-16-08 09:27:12 (45808:40036) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread_MS> is completed.
(11.5) 04-16-08 09:27:12 (43424:45588) PRINTFN: Time______________________________ 29 secounds
(11.5) 04-16-08 09:27:12 (43172:32192) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 09:27:12 (43172:32192) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1_MS> is started.
(11.5) 04-16-08 09:27:13 (43172:32192) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP1_MS> is started.
(11.5) 04-16-08 09:28:26 (43172:32192) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP1_MS> is completed successfully.
(11.5) 04-16-08 09:28:26 (43172:32192) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1_MS> is completed.
(11.5) 04-16-08 09:28:27 (43424:45588) PRINTFN: Time______________________________ 74 secounds
(11.5) 04-16-08 09:28:27 (46736:24524) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 09:28:27 (46736:24524) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10_MS> is started.
(11.5) 04-16-08 09:28:27 (46736:24524) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP10_MS> is started.
(11.5) 04-16-08 09:29:11 (46736:24524) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP10_MS> is completed successfully.
(11.5) 04-16-08 09:29:11 (46736:24524) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10_MS> is completed.
(11.5) 04-16-08 09:29:12 (43424:45588) PRINTFN: Time______________________________ 45 secounds
(11.5) 04-16-08 09:29:12 (43424:45588) JOB: Job <Job_Benchmark_MS> is completed successfully.
Back to top
gpn01
Principal Member
Principal Member



Joined: 30 Nov 2004

Posts: 224
Location: London, UK


flag
PostPosted: Wed Apr 16, 2008 4:59 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Name of the Test: GPN01_UAT

Source: SQL2005
Network source to DI: 1GBit Ethernet
DI server: Windows 2003 Standard SP2 32bit /2 CPU @ 3.4GHz/2GB/1Disk

DI: 11.5.3.9
Network DI to Target: shared with above
Target server: SQL2005
Target: SQL2005

(Source & Target are same database)

(11.5) 04-16-08 09:51:22 (6728:4748) JOB: Job <Job_Benchmark_MS> is started.
(11.5) 04-16-08 09:51:22 (7252:7196) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 09:51:22 (7252:7196) DATAFLOW: Process to execute data flow <DF_Benchmark_Setup_MS> is started.
(11.5) 04-16-08 09:51:22 (7252:7196) DATAFLOW: Data flow <DF_Benchmark_Setup_MS> is started.
(11.5) 04-16-08 09:51:23 (7252:7196) DATAFLOW: Data flow <DF_Benchmark_Setup_MS> is completed successfully.
(11.5) 04-16-08 09:51:23 (7252:7196) DATAFLOW: Process to execute data flow <DF_Benchmark_Setup_MS> is completed.
(11.5) 04-16-08 09:51:23 (1800:4400) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 09:51:23 (1800:4400) DATAFLOW: Process to execute data flow <DF_Benchmark_read_MS> is started.
(11.5) 04-16-08 09:51:24 (1800:4400) DATAFLOW: Data flow <DF_Benchmark_read_MS> is started.
(11.5) 04-16-08 09:55:33 (1800:4400) DATAFLOW: Data flow <DF_Benchmark_read_MS> is completed successfully.
(11.5) 04-16-08 09:55:34 (1800:4400) DATAFLOW: Process to execute data flow <DF_Benchmark_read_MS> is completed.
(11.5) 04-16-08 09:55:34 (6728:4748) PRINTFN: Time______________________________ 250 secounds
(11.5) 04-16-08 09:55:34 (2880:7708) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 09:55:34 (2880:7708) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader_MS> is started.
(11.5) 04-16-08 09:55:35 (2880:7708) DATAFLOW: Data flow <DF_Benchmark_API_bulkloader_MS> is started.
(11.5) 04-16-08 10:01:45 (2880:7708) DATAFLOW: Data flow <DF_Benchmark_API_bulkloader_MS> is completed successfully.
(11.5) 04-16-08 10:01:45 (2880:7708) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader_MS> is completed.
(11.5) 04-16-08 10:01:45 (6728:4748) PRINTFN: Time______________________________ 370 secounds
(11.5) 04-16-08 10:01:45 (5672:5500) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 10:01:45 (5672:5500) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load_MS> is started.
(11.5) 04-16-08 10:01:46 (5672:5500) DATAFLOW: Data flow <DF_Benchmark_regular_load_MS> is started.
(11.5) 04-16-08 10:21:12 (5672:5500) DATAFLOW: Data flow <DF_Benchmark_regular_load_MS> is completed successfully.
(11.5) 04-16-08 10:21:12 (5672:5500) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load_MS> is completed.
(11.5) 04-16-08 10:21:13 (6728:4748) PRINTFN: Time______________________________ 1168 secounds
(11.5) 04-16-08 10:21:14 (7464:4580) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 10:21:14 (7464:4580) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread_MS> is started.
(11.5) 04-16-08 10:21:14 (7464:4580) DATAFLOW: Data flow <DF_Benchmark_single_thread_MS> is started.
(11.5) 04-16-08 10:21:46 (7464:4580) DATAFLOW: Data flow <DF_Benchmark_single_thread_MS> is completed successfully.
(11.5) 04-16-08 10:21:46 (7464:4580) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread_MS> is completed.
(11.5) 04-16-08 10:21:46 (6728:4748) PRINTFN: Time______________________________ 32 secounds
(11.5) 04-16-08 10:21:46 (7772:3504) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 10:21:46 (7772:3504) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1_MS> is started.
(11.5) 04-16-08 10:21:53 (7772:3504) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP1_MS> is started.
(11.5) 04-16-08 10:22:58 (7772:3504) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP1_MS> is completed successfully.
(11.5) 04-16-08 10:22:58 (7772:3504) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1_MS> is completed.
(11.5) 04-16-08 10:22:59 (6728:4748) PRINTFN: Time______________________________ 72 secounds
(11.5) 04-16-08 10:22:59 (7068:5592) JOB: Current system configuration : <SP_DEV_LONDON>.
(11.5) 04-16-08 10:22:59 (7068:5592) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10_MS> is started.
(11.5) 04-16-08 10:23:00 (7068:5592) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP10_MS> is started.
(11.5) 04-16-08 10:23:42 (7068:5592) DATAFLOW: Data flow <DF_Benchmark_lookup_DOP10_MS> is completed successfully.
(11.5) 04-16-08 10:23:42 (7068:5592) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10_MS> is completed.
(11.5) 04-16-08 10:23:42 (6728:4748) PRINTFN: Time______________________________ 43 secounds
(11.5) 04-16-08 10:23:42 (6728:4748) JOB: Job <Job_Benchmark_MS> is completed successfully.
Back to top
dnewton
Forum Addict
Forum Addict



Joined: 30 Jan 2004
ASUG Icon
Posts: 3446
Location: Massachusetts, USA


flag
PostPosted: Thu Apr 17, 2008 3:11 pm 
Post subject: Re: Benchmark YOUR hardware and compare the results

dlinstedt wrote:

Now, the requirements for the machines:
1. 300-400 mbyte per second on Disk I/O throughput (minimum)
2. Raid 0+1 (NOT RAID 5, NOT RAID 10)
3. Minimum 2 x 2GBit fibre optic direct connect to Disk Arrays
4. Disk arrays are DEDICATED STORAGE (dedicated to RDBMS Servers)
...
7. RDBMS Hardware: usually a LOT larger than this, but for SQLServer, it simply increases to 4 dual core 3.0 GHZ cpu's, 64 bit OS, 64 Bit SQLServer2005, and 12 to 16 GB RAM, and 10,000 RPM internal SCSI drives ALL ON RAID 0+1


Just to clarify something. From everything I've read,

RAID 0+1 is also occasionally called "RAID 01" (not to be confused with RAID 1)
RAID 1+0 is also frequently called "RAID 10", and is the same thing as RAID 10.

The choice is really between RAID 0+1 and RAID 1+0 (RAID 10)

From everything I've read, there is no performance difference, and, RAID 1+0 has better redundancy characteristics. (edit: AAIIIEE! That's why this is so confusing! RAID 1+0 is better than RAID 0+1 for redundancy/uptime.)

Do you have different performance stats?


Last edited by dnewton on Thu Apr 24, 2008 7:52 am, edited 1 time in total
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Fri Apr 18, 2008 6:19 pm 
Post subject: Re: Benchmark YOUR hardware and compare the results

Any idea why the API bulkload in prod takes longer than the regular load? In dev all is in order.
_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
fayssal
Forum Member
Forum Member



Joined: 25 Apr 2008

Posts: 25


flag
PostPosted: Wed Jul 02, 2008 3:39 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Sorry i have forgot to put the time icon_redface.gif

DI JobServer; the source and the target are on the same server:

Source DB: Oracle 10g

Source Server: Solaris10, 4G of Memory, 2Proc 1,6Ghz

DI Server: Same as above (on same machine as source db)

Network, source to DI: n/a

DI Version: 11.7.2

Network, DI to Target: n/a

Target db: Oracle 10g (on same DB as source)

Results

<DF_Benchmark_read_MS>: 300 seconds
<DF_Benchmark_API_bulkloader_MS>: 310 seconds
<DF_Benchmark_regular_load_MS>: 11814 seconds
<DF_Benchmark_single_thread_MS>: 187 seconds
<DF_Benchmark_lookup_DOP1_MS>: 135 seconds
<DF_Benchmark_lookup_DOP10_MS>: 103 seconds

What do you think?
are this results normal??
icon_rolleyes.gif

_________________
Fayssal
BOCP-BOE
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Wed Jul 02, 2008 4:29 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Would have expected better numbers. Especially the fact that
<DF_Benchmark_single_thread_MS>: 187 seconds
takes longer than
<DF_Benchmark_lookup_DOP1_MS>: 135 seconds
worries me. BTW, you have picked the SQL Server version of the dataflows. They are not that much different and given your numbers you seem to have done all correct but it worries me a bit.

_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
eepjr24
Forum Enthusiast
Forum Enthusiast



Joined: 16 Sep 2005
ASUG Icon
Posts: 1815
Location: Orlando, FL


flag
PostPosted: Wed Jul 02, 2008 1:12 pm 
Post subject: Re: Benchmark YOUR hardware and compare the results

Name of the Test: Development Server
Source: Oracle 10g (10.2)
Network source to DI: Gigabit
DI server:
Red Hat Enterprise Linux AS release 3 (Taroon Update 9)
2.4.21-57.ELsmp
4 Intel(R) Xeon(TM) CPU @ 3.40GHz / 1024 MB
DI: 11.7.3
Network DI to Target: Gigabit
Target server: Same box, different DB
Target: Oracle 10g (10.2)

<DF_Benchmark_read>: 76 seconds
<DF_Benchmark_API_bulkloader>: 717 seconds
<DF_Benchmark_regular_load>: 1897 seconds
<DF_Benchmark_single_thread>: 104 seconds
<DF_Benchmark_lookup_DOP1>: 145 seconds
<DF_Benchmark_lookup_DOP10>: 107 seconds

Remarks:
This seemed to hit the DB alot harder than the DI box. Load average topped at about 3 on the DB, about 1.1 on the job server. Other than that I will let others draw their own conclusions. I should be able to run it in Prod one day soon.

- Ernie

_________________
Please Search before you post.
___________________________
Ernie Phelps
Senior Manager
Protiviti
http://www.protiviti.com
Back to top
gpn01
Principal Member
Principal Member



Joined: 30 Nov 2004

Posts: 224
Location: London, UK


flag
PostPosted: Thu Jul 03, 2008 2:16 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Hi, any chance of the statistics summary page being updated ? Maybe a neat little graph comparing results too ?

Last edited by gpn01 on Fri Jul 04, 2008 2:17 am, edited 1 time in total
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Fri Jul 04, 2008 1:29 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Will do that. For a graph there are too many variables, aren't there? If you have an idea for the diagram, let me know.
_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Page 3 of 10 All times are GMT - 5 Hours
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.1042 seconds using 17 queries. (SQL 0.0021 Parse 0.0678 Other 0.0344)
CCBot/2.0 (https://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo