BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Want to sponsor BOB? 
Want to sponsor BOB? (Opens a new window)  

General Notice: BOB is going to retire...please see details here.
General Notice: No events within the next 45 days.

How DI process Data?


 
Search this topic... | Search DI: Performance and Tuning... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Author Message
RajPatel
Senior Member
Senior Member



Joined: 19 May 2005

Posts: 53



PostPosted: Fri Jun 10, 2005 8:38 am 
Post subject: How DI process Data?

Hi

I am wondering how DI works with data?
All other ETL tools transform their data completely by each transform,

Ex:
Sort >> Join >> Expression >> Join >> output..

ETL tool will
1> Sort entire data set and make temp binary file in temp directory..
2> Joiner will take sorted data file and join with another source.
3> Expression with take complete data set and go aurethmatic
4> Ans so…on


But in DI, I can see from job monitor that all processes data in all transform..

What DI works on row- by row? Or Data set Or Partial way ?


Thanks
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10593
Location: Germany


flag
PostPosted: Fri Jun 10, 2005 3:20 pm 
Post subject: Re: How DI process Data?

DI has a parallel pipelined engine with a buffer of a couple of thousend rows between each transforms. Obviously, some tasks cannot be done before all data is ready, e.g. a query that does the sorting inside the engine (and cannot be pushed down to the source database) has to collect all the data before it knows what the first row is. Hence, these blocking operations should be avoided as much as possible.

In addition this engine can partition data to even execute the same transformation step in parallel.

_________________
Werner Daehn
ex Product Manager Data Services at SAP SE

www.rtdi.io
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Page 1 of 1 All times are GMT - 5 Hours
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Get community updates via Twitter:

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.0298 seconds using 17 queries. (SQL 0.0023 Parse 0.0008 Other 0.0266)
CCBot/2.0 (https://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo