BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Want to sponsor BOB? 
Want to sponsor BOB? (Opens a new window)  

General Notice: BOB is going to retire...please see details here.
General Notice: No events within the next 45 days.

How DI process Data?

Search this topic... | Search DI: Performance and Tuning... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Author Message
Senior Member
Senior Member

Joined: 19 May 2005

Posts: 53

PostPosted: Fri Jun 10, 2005 8:38 am 
Post subject: How DI process Data?


I am wondering how DI works with data?
All other ETL tools transform their data completely by each transform,

Sort >> Join >> Expression >> Join >> output..

ETL tool will
1> Sort entire data set and make temp binary file in temp directory..
2> Joiner will take sorted data file and join with another source.
3> Expression with take complete data set and go aurethmatic
4> Ans so…on

But in DI, I can see from job monitor that all processes data in all transform..

What DI works on row- by row? Or Data set Or Partial way ?

Back to top
Werner Daehn
Forum Devotee
Forum Devotee

Joined: 17 Dec 2004

Posts: 10593
Location: Germany

PostPosted: Fri Jun 10, 2005 3:20 pm 
Post subject: Re: How DI process Data?

DI has a parallel pipelined engine with a buffer of a couple of thousend rows between each transforms. Obviously, some tasks cannot be done before all data is ready, e.g. a query that does the sorting inside the engine (and cannot be pushed down to the source database) has to collect all the data before it knows what the first row is. Hence, these blocking operations should be avoided as much as possible.

In addition this engine can partition data to even execute the same transformation step in parallel.

Werner Daehn
ex Product Manager Data Services at SAP SE
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Page 1 of 1 All times are GMT - 5 Hours
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Get community updates via Twitter:

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.0438 seconds using 17 queries. (SQL 0.0041 Parse 0.0014 Other 0.0383)
CCBot/2.0 (
Hosted by | Terms of Service
phpBB Customizations by the
Shameless plug for Moments of Light Logo