BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Follow BOB on Twitter! 
Follow BOB on Twitter! (Opens a new window)  

General Notice: BOB is going to retire...please see details here.
General Notice: No events within the next 45 days.

Need to Extract few SAP tables with data from 35Mill -200 Mi


 
Search this topic... | Search DI: SAP-specific ERP/BW... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: SAP-specific ERP/BW  Previous TopicPrint TopicNext Topic
Author Message
zuluking
Senior Member
Senior Member



Joined: 31 May 2012

Posts: 51



PostPosted: Tue Nov 15, 2016 5:33 pm 
Post subject: Need to Extract few SAP tables with data from 35Mill -200 Mi

Hi we are extracting data from SAP using ABAP data
flows and loading into SQL server on cloud.
Most of the tables have been loaded but anything more than 30 million rows we are facing issues, like unknown exception and jobs failing.
What kind of optimization is recommended in this case ?
We have dropped the PK's , tried to run it in background by tick marking the option in the data store.....anything else we need to try ?

Any help appreciated.
Back to top
eganjp
Forum Addict
Forum Addict



Joined: 12 Sep 2007
ASUG Icon
Posts: 4280
Location: Denver Area


flag
PostPosted: Thu Dec 15, 2016 1:54 am 
Post subject: Re: Need to Extract few SAP tables with data from 35Mill -20

What is your data transfer method? FTP or RFC?
_________________
Jim Egan
ProKarma-an SAP Consulting Partner
Speaker at RMOUG Training Days 2013
Speaker at Sapphire Now 2013
Speaker at Business Objects User Conference 2013
-My ETL is faster than your database! - Advanced ETL performance tuning for Data Services
-Kill ETL bugs before they kill you!
My ETL blog

Latest Blog Posts
• 2014-04-17 Can Your ETL Make Your Reporting Faster?
• 2014-04-04 ProKarma Acquires Mantis Technology Group
• 2014-04-02 How to Create Charts in Webi (Part 2 of 2)
Back to top
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1111
Location: Wellington, NZ


flag
PostPosted: Mon Dec 19, 2016 12:49 am 
Post subject: Re: Need to Extract few SAP tables with data from 35Mill -20

I've had no problems extracting 20-60 (and more) million records from ECC 6.0 on HANA using background-enabled RFC extracts.

SAP always seems to recommend using ABAP data flows with RFC Streaming, which certainly does work as well but we've had no issues extracting large volumes using the simple RFC Streaming interface as well, nor did we notice any real performance differences. Which really isn't a huge surprise as both methods use the same output mechanism, that being streaming records out in buckets of 5000 records.

Are you extracting from an on premise based SAP ERP environment into a cloud based SQL Server instance? Or is your SAP ERP environment also in the cloud?

If your ERP is on premise, I would staging the data locally first, preferably in a database otherwise in text files if you have to - and then load into the cloud based SQL instance. At least to investigate where you are running into problems.

Also ensure that your RFC connections are background enabled as they will time out as foreground jobs which such volumes of data.
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: SAP-specific ERP/BW  Previous TopicPrint TopicNext Topic
Page 1 of 1 All times are GMT - 5 Hours
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Get community updates via Twitter:

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.0479 seconds using 18 queries. (SQL 0.0234 Parse 0.0008 Other 0.0237)
CCBot/2.0 (https://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo