BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Follow BOB on Twitter! 
Follow BOB on Twitter! (Opens a new window)  

General Notice: BOB is going to retire...please see details here.
General Notice: No events within the next 45 days.

Caching source tables


 
Search this topic... | Search DI: Performance and Tuning... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Author Message
msarrafi
Forum Member
Forum Member



Joined: 01 Mar 2006

Posts: 7
Location: Harrisburg, PA


flag
PostPosted: Tue Apr 04, 2006 2:36 pm 
Post subject: Caching source tables

If the performance option for a source table is set to 'Cache' and is used multiple times in a data flow, is a full table scan performed each time the table is used as input to a query transform? Would a single instance of a table set with cache enabled be better if used as input to multiple transforms? (Even for very large tables with millions of rows).

Which would be better performing, using multiple instances of the same table in a single data flow or connect a single instance of a table to multiple query transforms? I'm assuming that each instance of the same table will require memory and resources.
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Tue Apr 04, 2006 3:19 pm 
Post subject: Re: Caching source tables

multiple instances: joins, group bys etc can be pushed down & you are using database cache

one instance: Table is read just once and you are using DI engine caching (very likely).


For example, if I join T1 with T2 and T1 with T3 I would use two T1 instances so the join is pushed down. If a pushdown is not possible anyway, this approach does not really make sense.

My advice: Try both as there is no common for this.

_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
dnewton
Forum Addict
Forum Addict



Joined: 30 Jan 2004
ASUG Icon
Posts: 3458
Location: Massachusetts, USA


flag
PostPosted: Sat Apr 08, 2006 7:48 am 
Post subject: Re: Caching source tables

We use multiple instances. When you use one instance, DI can't optimize which columns to read out of the database, so it reads all of them. And it caches everything locally, which can cause memory issues and performance issues.

DI's whole performance supposition is that the database is set up and optimized for things like Group By or Where condition filtering. This means that you get high ETL performance if you have a strong database. Conversely, if your database infrastructure is weak, you will get slow performance.
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Page 1 of 1 All times are GMT - 5 Hours
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Get community updates via Twitter:

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.0264 seconds using 17 queries. (SQL 0.0025 Parse 0.0008 Other 0.0231)
CCBot/2.0 (https://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo