BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Sunset Ride 
Sunset Ride 

General Notice: BOB is going to retire...please see details here. All BOB forums have been converted to "read only" as of September 1st.
General Notice: No events within the next 45 days.

Benchmark YOUR hardware and compare the results V1.1
1 members found this topic helpful
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
 
Search this topic... | Search DI: Performance and Tuning... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Author Message
gpn01
Principal Member
Principal Member



Joined: 30 Nov 2004

Posts: 224
Location: London, UK


flag
PostPosted: Tue Mar 31, 2009 2:21 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

wdaehn wrote:
please use IE to download/view the page. I have raised that problem within SAP.


I'm using IE7 when I get the error.
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Tue Mar 31, 2009 6:53 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Just tried, it is working for me. IE7 as well.
Anyway, I have talked to SDN support again.

_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Sat Apr 04, 2009 1:07 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

mad.gif

It took a while but we finally found the root cause - me.

The link pointed to an internal server, not the public server. So it was working for me when I was in the internal network but not for you. Didn't know that.

I just tried different options and finally was able to download it with everything from everywhere.

_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
gpn01
Principal Member
Principal Member



Joined: 30 Nov 2004

Posts: 224
Location: London, UK


flag
PostPosted: Mon Apr 06, 2009 2:01 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

I thought it was something wrong with our firewall cryin.gif

Now working ok!
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Mon Apr 06, 2009 2:26 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

I said I am sorry Image link


icon_mrgreen.gif

Looking forward to see your numbers....

_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
gpn01
Principal Member
Principal Member



Joined: 30 Nov 2004

Posts: 224
Location: London, UK


flag
PostPosted: Mon Apr 06, 2009 3:49 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Name of the Test: DI Dev
Source: SQL 2005
Network source to DI: 1Gb Ethernet
DI server: Windows 2003SP2 Std 64-bit /4 CPU @ 2.66GHz/16Gb/1Disk
DI: 11.5.3.9
Network DI to Target: shared with above
Target server: shared with above
Target: SQL 2005 (shared with above)

i.e. source, target and Jobserver all on same box.

(11.5) 04-06-09 08:40:02 (8536:8840) DATAFLOW: Process to execute data flow <DF_Benchmark_read_MS> is completed.
(11.5) 04-06-09 08:40:02 (0152:7864) PRINTFN: Time______________________________ 69 secounds
(11.5) 04-06-09 08:42:28 (5304:2412) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader_MS> is completed.
(11.5) 04-06-09 08:42:29 (0152:7864) PRINTFN: Time______________________________ 146 secounds
(11.5) 04-06-09 08:49:35 (8004:7336) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load_MS> is completed.
(11.5) 04-06-09 08:49:35 (0152:7864) PRINTFN: Time______________________________ 426 secounds
(11.5) 04-06-09 08:49:51 (1884:0864) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread_MS> is completed.
(11.5) 04-06-09 08:49:51 (0152:7864) PRINTFN: Time______________________________ 15 secounds
(11.5) 04-06-09 08:50:29 (3712:7228) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1_MS> is completed.
(11.5) 04-06-09 08:50:29 (0152:7864) PRINTFN: Time______________________________ 38 secounds
(11.5) 04-06-09 08:50:47 (8216:9460) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10_MS> is completed.
(11.5) 04-06-09 08:50:48 (0152:7864) PRINTFN: Time______________________________ 18 secounds
Back to top
gpn01
Principal Member
Principal Member



Joined: 30 Nov 2004

Posts: 224
Location: London, UK


flag
PostPosted: Mon Apr 06, 2009 3:55 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Name of the Test: DI Dev VM
Source: SQL 2005
Network source to DI: 1Gb Ethernet
DI server: Windows 2003SP2 32-bit /2 CPU @ 3GHz/3.75Gb/1Disk (VMWare)
DI: 11.5.3.9
Network DI to Target: s1Gb Ethernet
Target server: unknown
Target: SQL 2005 (shared with source)

(11.5) 04-06-09 09:19:29 (3660:2740) DATAFLOW: Process to execute data flow <DF_Benchmark_read_MS> is completed.
(11.5) 04-06-09 09:19:29 (4036:1756) PRINTFN: Time______________________________ 217 secounds
(11.5) 04-06-09 09:25:02 (2088:2392) DATAFLOW: Process to execute data flow <DF_Benchmark_API_bulkloader_MS> is completed.
(11.5) 04-06-09 09:25:02 (4036:1756) PRINTFN: Time______________________________ 332 secounds
(11.5) 04-06-09 09:32:02 (2940:1692) DATAFLOW: Process to execute data flow <DF_Benchmark_regular_load_MS> is completed.
(11.5) 04-06-09 09:32:02 (4036:1756) PRINTFN: Time______________________________ 420 secounds
(11.5) 04-06-09 09:32:20 (2596:4076) DATAFLOW: Process to execute data flow <DF_Benchmark_single_thread_MS> is completed.
(11.5) 04-06-09 09:32:20 (4036:1756) PRINTFN: Time______________________________ 18 secounds
(11.5) 04-06-09 09:33:02 (2336:1556) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP1_MS> is completed.
(11.5) 04-06-09 09:33:02 (4036:1756) PRINTFN: Time______________________________ 41 secounds
(11.5) 04-06-09 09:33:40 (2268:1732) DATAFLOW: Process to execute data flow <DF_Benchmark_lookup_DOP10_MS> is completed.
(11.5) 04-06-09 09:33:41 (4036:1756) PRINTFN: Time______________________________ 38 secounds
Back to top
dnewton
Forum Addict
Forum Addict



Joined: 30 Jan 2004
ASUG Icon
Posts: 3446
Location: Massachusetts, USA


flag
PostPosted: Mon Apr 06, 2009 6:52 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

gpn01 wrote:
Name of the Test: DI Dev


These numbers are are some of the fastest ones yet. Can you describe your hardware configuration in more detail, particularly the disk subsystem?

Another random question, is your job engine in unicode (UTF8) mode, or something else?

thanks
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Mon Apr 06, 2009 2:25 pm 
Post subject: Re: Benchmark YOUR hardware and compare the results

29 customer numbers we have now
_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
data_guy
Forum Associate
Forum Associate



Joined: 19 Aug 2006
ASUG Icon
Posts: 723
Location: Somewhere EveryWhere!


flag
PostPosted: Tue Jun 09, 2009 10:16 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Hi,

New project, so its time for new benchmark!

Source and target DB: Oracle on Unix, it is 64bit (10.2.0.3) in Indianapolis, IN
DI Job server: Virtual on Windows Server 2003 R2 32 bit, 4 Intel Xeon 3.4Ghz, 4GB, 1Gpbs Switch operational in Madison, WI
DI Repo: Oracle on Unix, it is 64bit (10.2.0.3) in Madison, WI


Process to execute data flow <DF_Benchmark_read> is completed.
Time______________________________ 278 secounds

Process to execute data flow <DF_Benchmark_API_bulkloader> is completed.
Time______________________________ 527 secounds

Process to execute data flow <DF_Benchmark_regular_load> is completed.
Time______________________________ 856 secounds

Process to execute data flow <DF_Benchmark_single_thread> is completed.
Time______________________________ 39 secounds

Process to execute data flow <DF_Benchmark_lookup_DOP1> is completed.
Time______________________________ 135 secounds

Process to execute data flow <DF_Benchmark_lookup_DOP10> is completed.
Time______________________________ 210 secounds


Any advice, on this figures!


Thanks,

_________________
@Data is Delicious

++++++++++++++++
Where you see risk,
I see opportunity!
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Wed Jun 10, 2009 4:04 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Yes, install the jobserver on the target database machine.... yesnod.gif
_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
megablst
Senior Member
Senior Member



Joined: 03 Mar 2006

Posts: 46
Location: Reading, UK


flag
PostPosted: Thu Jun 11, 2009 11:04 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Name of the Test: megablst_dev_db_and_dev_physical_ETL_svr
Source: SQL Server 2000 SP3
Network source to DI: 1GBit Ethernet
DI server: Physical - Windows 2003-Ent Ed. - SP2, Quad Core Xeon @ 3.4Ghz / 11.5 Gb (PAE)
DI: 11.7.0.1
Network DI to Target: 1GBit
Target DB server: Windows 2003 box w local disk in RAID 5 config
Target: SQL Server 2000 SP3

Network Read Benchmark - Time: 267
API Bulk Load Benchmark - Time: 234
API Bulk Load Partitioned Target DOP0 Benchmark - Time: 115
API Bulk Load Partitioned Target DOP10 Benchmark - Time: 108
Regular Load Single Loader Benchmark - Time: 919
Regular Load Multiple Loader Benchmark - Time: 527
Single Thread Benchmark - Time: 44
DOP1 Lookup Benchmark - Time: 108
DOP10 Lookup Benchmark - Time: 29
______________________________________________________
Name of the Test: megablst_dev_db_and_systest_virtual_ETL_svr
Note: same db servers as in previous figures - only the ETL server is changed
Source: SQL Server 2000 SP3
Network source to DI: 1GBit Ethernet
DI server: Virtual - Windows 2003-Std Ed. - SP1, Quad Core Opteron 875 @ 2.2GHz / 3Gb
DI: 11.7.0.1
Network DI to Target: 1GBit
Target DB server: Windows 2003 box w local disk in RAID 5 config
Target: SQL Server 2000 SP3

Network Read Benchmark - Time: 236
API Bulk Load Benchmark - Time: 187
API Bulk Load Partitioned Target DOP0 Benchmark - Time: 285
API Bulk Load Partitioned Target DOP10 Benchmark - Time: 195
Regular Load Single Loader Benchmark - Time: 795
Regular Load Multiple Loader Benchmark - Time: 527
Single Thread Benchmark - Time: 36
DOP1 Lookup Benchmark - Time: 70
DOP10 Lookup Benchmark - Time: 51

______________________________________________________
Name of the Test: megablst_dev_db_and_UAT_physical_ETL_svr
Note: same db servers as in previous figures - only the ETL server is changed
Source: SQL Server 2000 SP3
Network source to DI: 1GBit Ethernet
DI server: Physical - Windows 2003 SP2, Quad Core Xeon 5150 @ 2.66Ghz / 3.5Gb
DI: 11.7.0.1
Network DI to Target: 1GBit
Target DB server: Windows 2003 box w local disk in RAID 5 config
Target: SQL Server 2000 SP3

Network Read Benchmark - Time: 138
API Bulk Load Benchmark - Time: 123
API Bulk Load Partitioned Target DOP0 Benchmark - Time: 112
API Bulk Load Partitioned Target DOP10 Benchmark - Time: 109
Regular Load Single Loader Benchmark - Time: 758
Regular Load Multiple Loader Benchmark - Time: 508
Single Thread Benchmark - Time: 24
DOP1 Lookup Benchmark - Time: 38
DOP10 Lookup Benchmark - Time: 16

______________________________________________________
Conclusions we found:
1. Virtual ETL servers perform fine relative to physical ETL servers except when partitioning / degress of parallelism start to increase and then performance significantly degrades relative to a physical ETL server.

Indeed our virtual server out performed our physical dev ETL server for none parallel / partitioned ETL.

2. Our ETL servers in terms of processing were never maxed out more than 75% CPU across all cores under any of the tests. Bottleneck always seems to be database which admittedly aren't well configured in terms of disks.

3. We found issues with DI core dumping randomly and only *very* occasionally with high degrees of parallelism / partitioning on a virtual server. On a physical server no problems were seen with the same amount of parallelism / partitioning.

Hope that helps someone
megablst
Back to top
dnewton
Forum Addict
Forum Addict



Joined: 30 Jan 2004
ASUG Icon
Posts: 3446
Location: Massachusetts, USA


flag
PostPosted: Fri Jun 26, 2009 12:04 pm 
Post subject: Re: Benchmark YOUR hardware and compare the results

dnewton wrote:
Test with new SAN and DI 11.7.3.5 - Source, Target, and DI on same server
Source: MS SQL 2005
Single Server: Win2003 r2 SP1 x64, (2) 2.8Ghz AMD CPUs (dualcore), 32GB RAM, RAID 10 on many SAN spindles
DI Server: Same as above
Network, source to DI: n/a
DI Version: 11.7.3.5
Network, DI to Target: n/a
Target db: MS SQL 2005 (different db, different SAN spindles)
Target server: Same as above

Results

<DF_Benchmark_read_MS>: 112 seconds
<DF_Benchmark_API_bulkloader_MS>: 298 seconds
<DF_Benchmark_regular_load_MS>: 260 seconds
<DF_Benchmark_single_thread_MS>: 22 seconds
<DF_Benchmark_lookup_DOP1_MS>: 79 seconds
<DF_Benchmark_lookup_DOP10_MS>: 328 seconds


So I repeated my October 2008 test, but on a slightly newer Windows server, and more importantly, this time the underlying disk subsystem is a NetApp 3160 SAN, RAID 4.

Results

<DF_Benchmark_read_MS>: 107 seconds
<DF_Benchmark_API_bulkloader_MS>: 238 seconds
<DF_Benchmark_regular_load_MS>: 382 seconds
<DF_Benchmark_single_thread_MS>: 19 seconds
<DF_Benchmark_lookup_DOP1_MS>: 71 seconds
<DF_Benchmark_lookup_DOP10_MS>: 305 seconds

And once again, even on this brand-new machine and new environment ... on that last test (DOP10), when I change the DF from Pageable to In Memory, the time dropped down from 305 to 20 seconds.

It's interesting that the bulk load is 25% faster but the regular load is 45% slower...
Back to top
titmarsh
Senior Member
Senior Member



Joined: 30 Aug 2006

Posts: 33
Location: Brisbane


flag
PostPosted: Tue Jul 14, 2009 2:21 am 
Post subject: Re: Benchmark YOUR hardware and compare the results

Hi All,

First run of benchmark results are below:

Name of the Test: VMWare image
Source: Oracle 10g
Network source to DI: 1Gb Ethernet
DI server: Windows 2003R2SP2 32bit / 1 CPU (Intel E5450) @ 3.00GHz/3.75GB/1Disk (VMWare)
DI: Data Services XI3.1
Network DI to Target: 1Gb Ethernet
Target server: unknown
Target: Oracle 10g

Note: Source, Target, and DS Job Server are all on different machines.

Results:
Process to execute data flow <DF_Benchmark_read> is completed.
Time______________________________ 40 secounds

Process to execute data flow <DF_Benchmark_API_bulkloader> is completed.
Time______________________________ 302 secounds

Process to execute data flow <DF_Benchmark_regular_load> is completed.
Time______________________________ 551 secounds

Process to execute data flow <DF_Benchmark_single_thread> is completed.
Time______________________________ 50 secounds

Process to execute data flow <DF_Benchmark_lookup_DOP1> is completed.
Time______________________________ 36 secounds

Process to execute data flow <DF_Benchmark_lookup_DOP10> is completed.
Time______________________________ 39 secounds


Cheers,
Aaron
Back to top
data_guy
Forum Associate
Forum Associate



Joined: 19 Aug 2006
ASUG Icon
Posts: 723
Location: Somewhere EveryWhere!


flag
PostPosted: Fri Sep 25, 2009 12:58 pm 
Post subject: Re: Benchmark YOUR hardware and compare the results

Hi all,

We just got our new much improved machine for Dev, now everything is on LAN icon_smile.gif

DI 11.7.3..2 on Win Server 2003 64 bit with 32 GB RAM, 4 Intel Dual Core CPU running at 3.4Ghz.

Source and Target DB on same DB server, Oracle 10g on Unix, it is 64bit (10.2.0.4)

Network Switch: 1Gpbs Switch operational

Disabled "Collect Statistics for Optimization" Option using IN MEMORY CACHE :

552 832 DATAFLOW 9/25/2009 1:00:49 PM Process to execute data flow <DF_Benchmark_read> is completed.
1600 1372 PRINTFN 9/25/2009 1:00:49 PM Time______________________________ 38 secounds

4256 3812 DATAFLOW 9/25/2009 1:06:49 PM Process to execute data flow <DF_Benchmark_API_bulkloader> is completed.
1600 1372 PRINTFN 9/25/2009 1:06:49 PM Time______________________________ 359 secounds

668 3876 DATAFLOW 9/25/2009 1:19:24 PM Process to execute data flow <DF_Benchmark_regular_load> is completed.
1600 1372 PRINTFN 9/25/2009 1:19:24 PM Time______________________________ 755 secounds

3600 4568 DATAFLOW 9/25/2009 1:20:05 PM Process to execute data flow <DF_Benchmark_single_thread> is completed.
1600 1372 PRINTFN 9/25/2009 1:20:06 PM Time______________________________ 41 secounds

4504 5084 DATAFLOW 9/25/2009 1:21:01 PM Process to execute data flow <DF_Benchmark_lookup_DOP1> is completed.
1600 1372 PRINTFN 9/25/2009 1:21:01 PM Time______________________________ 55 secounds

4152 408 DATAFLOW 9/25/2009 1:24:52 PM Process to execute data flow <DF_Benchmark_lookup_DOP10> is completed.
1600 1372 PRINTFN 9/25/2009 1:24:53 PM Time______________________________ 231 secounds



Enabled "Collect Statistics for Optimization" Option and using PAGEABLE CACHE :

4656 4532 DATAFLOW 9/25/2009 1:26:51 PM Process to execute data flow <DF_Benchmark_read> is completed.
4684 1604 PRINTFN 9/25/2009 1:26:51 PM Time______________________________ 38 secounds

480 4932 DATAFLOW 9/25/2009 1:36:43 PM Process to execute data flow <DF_Benchmark_API_bulkloader> is completed.
4684 1604 PRINTFN 9/25/2009 1:36:43 PM Time______________________________ 591 secounds

4976 988 DATAFLOW 9/25/2009 1:49:01 PM Process to execute data flow <DF_Benchmark_regular_load> is completed.
4684 1604 PRINTFN 9/25/2009 1:49:02 PM Time______________________________ 738 secounds

4708 4484 DATAFLOW 9/25/2009 1:49:50 PM Process to execute data flow <DF_Benchmark_single_thread> is completed.
4684 1604 PRINTFN 9/25/2009 1:49:51 PM Time______________________________ 49 secounds

584 3944 DATAFLOW 9/25/2009 1:50:47 PM Process to execute data flow <DF_Benchmark_lookup_DOP1> is completed.
4684 1604 PRINTFN 9/25/2009 1:50:47 PM Time______________________________ 56 secounds

4696 4300 DATAFLOW 9/25/2009 1:54:32 PM Process to execute data flow <DF_Benchmark_lookup_DOP10> is completed.
4684 1604 PRINTFN 9/25/2009 1:54:33 PM Time______________________________ 225 secounds


There is a significant improvement on numbers for LAN vs WAN sure!


Thanks,



icon_cool.gif

_________________
@Data is Delicious

++++++++++++++++
Where you see risk,
I see opportunity!
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Page 6 of 10 All times are GMT - 5 Hours
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.1009 seconds using 17 queries. (SQL 0.0019 Parse 0.0680 Other 0.0310)
CCBot/2.0 (https://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo