BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Streamline and extend SAP BusinessObjects software 
Streamline and extend SAP BusinessObjects software (Opens a new window)  

General Notice: No events within the next 45 days.

error - The pipe has been ended

Goto page 1, 2  Next
 
Search this topic... | Search DI: Performance and Tuning... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Author Message
rachel.l.su
Senior Member
Senior Member



Joined: 30 Nov 2004

Posts: 48
Location: Melbourne



PostPosted: Wed Jan 24, 2007 7:19 pm 
Post subject: error - The pipe has been ended

We have a job which normally runs for about an hour. It recently reported the following error.

(11.5) 01-19-07 10:51:40 (E) (4904:3440) RUN-050604: |Session JOB_ABC|Workflow WF_ABC|Dataflow DF_ABC_3
Cannot write pipe, due to error <The pipe has been ended. >.


It's on DI 11.5 with SQL 2000 repository and backend DW databases (on separate servers, Win2000 OS). The DI job takes a big xml file from shared network folder then process the data, put it in tables, and then reads from tables and spits smaller xml files into SQL DB.

The error message was reported in the data flow which pulls data from tables and convert and pushes it to xml tables.

At this stage we are not sure where this error came from - from DI or SQL or NT? Can anyone shed some light? Much appreciated!
Back to top
Nicolas Hadj-Blaha
Principal Member
Principal Member



Joined: 10 Mar 2005
ASUG Icon
Posts: 480
Location: Hamilton, New Zealand


flag
PostPosted: Wed Jan 24, 2007 7:25 pm 
Post subject: Re: error - The pipe has been ended

I would recommend you check with the SQL DBAs as it looks likely to be some database settings
see other topic

_________________
AgData New Zealand
Back to top
rachel.l.su
Senior Member
Senior Member



Joined: 30 Nov 2004

Posts: 48
Location: Melbourne



PostPosted: Wed Jan 24, 2007 7:31 pm 
Post subject: Re: error - The pipe has been ended

I just checked with our Prod DBA and asked him to check corresponding SQL logs. He confirmed that this is an application issue (DI?) and no SQL logs are found for corresponding time.

I found there was other posts on this forum mentioning this intermittent error on DI. However no good answers were supplied. icon_sad.gif

Can anyone help pls?

http://www.forumtopics.com/busobj/viewtopic.php?t=73605&highlight=cannot+write+pipe
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10590
Location: Germany


flag
PostPosted: Thu Jan 25, 2007 2:07 am 
Post subject: Re: error - The pipe has been ended

rachel.l.su wrote:
[...]The DI job takes a big xml file from shared network folder then process the data[...]


Very likely, that's the problem. Consuming too much memory. Execute it and watch the al_engine processes. I guess one gets close to 2GB before it fails.


You should look into the XML Pipeline transform to shred one XML file into multiple chunks of smaller pieces. This way, we need memory for the smaller elements only.

_________________
Werner Daehn
Product Manager Data Services
SAP AG
“The opinions expressed in this post are my own views and not those of SAP.”

PLEASE SUPPORT THIS: Run one ATL job and tell us the benchmark result. Details to be found here.
Back to top
rachel.l.su
Senior Member
Senior Member



Joined: 30 Nov 2004

Posts: 48
Location: Melbourne



PostPosted: Thu Jan 25, 2007 2:16 am 
Post subject: Re: error - The pipe has been ended

Thanks so much for your prompt reply!

The first xml read bit does consume a fair bit of memory - up to 1.4GB. However, the failed step does not consume that much memory. Having said that, this occurred in production so it's possible that there are other processes consuming memory thus affected this one.

Would this be the only possible cause of this error? So that we should concentrate on this? icon_rolleyes.gif
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10590
Location: Germany


flag
PostPosted: Thu Jan 25, 2007 3:24 am 
Post subject: Re: error - The pipe has been ended

You can always uncover a new bug. I would say, the probability that it is the XML processing consuming so much memory is 97%.
_________________
Werner Daehn
Product Manager Data Services
SAP AG
“The opinions expressed in this post are my own views and not those of SAP.”

PLEASE SUPPORT THIS: Run one ATL job and tell us the benchmark result. Details to be found here.
Back to top
dastocks
Principal Member
Principal Member



Joined: 11 Dec 2006

Posts: 163



PostPosted: Thu Jan 25, 2007 6:40 am 
Post subject: Re: error - The pipe has been ended

We had this on long (i.e. continuously looping) jobs. There is a hot fix available in 11.5.1.18 that resolved this issue (for us).
Back to top
dnewton
Forum Addict
Forum Addict



Joined: 30 Jan 2004
ASUG Icon
Posts: 3456
Location: Massachusetts, USA


flag
PostPosted: Thu Jan 25, 2007 4:32 pm 
Post subject: Re: error - The pipe has been ended

Also, try copying the XML file local to the job server, rather than reading it across the network.
Back to top
rachel.l.su
Senior Member
Senior Member



Joined: 30 Nov 2004

Posts: 48
Location: Melbourne



PostPosted: Thu Jan 25, 2007 7:51 pm 
Post subject: Re: error - The pipe has been ended

thanks so much for all of your advice! will try them out soon.
Back to top
rachel.l.su
Senior Member
Senior Member



Joined: 30 Nov 2004

Posts: 48
Location: Melbourne



PostPosted: Tue Jan 30, 2007 5:54 pm 
Post subject: Re: error - The pipe has been ended

Hi Werner,

Yes the XML pipeline worked! icon_surprised.gif It reduces memory usage to 250MB for a 41MB input XML file (relatively simple XSD but with approx 200,000 repeating nodes)! It does take 20~30% longer to process this step, but still a good outcome as this is not the most time-consuming part of the job so overall not causing major delays but did resolve our memory issue. Thanks again!

One more question I want to ask - how does the xml pipeline work? In this case we're reading xml file from a folder, so I guess the xml-pipeline "reading" is been pushed down to the OS? (reading from the manual)? We got our own java adapter...can we still get the benefit of xml-pipeline transform on top of our adapter? Or do we need to add some codes to our adapter to support xml-pipeline? icon_rolleyes.gif Looking forward to your feedback.


hi dastocks, we haven't asked for the hotfix yet. Thanks for the information though!

hi dnewton, due to current implementation, we can't do much about changing the file location. Since the number of files we're getting is only 10, the network time should be minimal...and it's all within the same intranet so should be ok. Thanks for the advice!
Back to top
dnewton
Forum Addict
Forum Addict



Joined: 30 Jan 2004
ASUG Icon
Posts: 3456
Location: Massachusetts, USA


flag
PostPosted: Tue Jan 30, 2007 6:57 pm 
Post subject: Re: error - The pipe has been ended

rachel.l.su wrote:

hi dnewton, due to current implementation, we can't do much about changing the file location. Since the number of files we're getting is only 10, the network time should be minimal...and it's all within the same intranet so should be ok. Thanks for the advice!


What we generally do is first have a Script step in the job, and that step runs a batch file which copies the file from the network to a local directory on the job server.
Back to top
DI_NooB
Forum Member
Forum Member



Joined: 01 Dec 2006

Posts: 20



PostPosted: Thu Feb 01, 2007 1:54 pm 
Post subject: Re: error - The pipe has been ended

rachel.l.su wrote:
Hi Werner,

...We got our own java adapter...can we still get the benefit of xml-pipeline transform on top of our adapter? Or do we need to add some codes to our adapter to support xml-pipeline? icon_rolleyes.gif Looking forward to your feedback.

...


What does your adapter do? We are having some issues with creating an adapter. There aren't many sources of information about them. We need our adapter to call a Java API and use the data structure returned from the API as a source in DI. Is that anything like you are doing? It seems the only thing I can find is that the adapters want to work with files or tables as the source.
Back to top
rachel.l.su
Senior Member
Senior Member



Joined: 30 Nov 2004

Posts: 48
Location: Melbourne



PostPosted: Thu Feb 01, 2007 5:59 pm 
Post subject: Re: error - The pipe has been ended

Yes the adapter reads from DB and return XML data structure to DI. So I can confirm that can be done. However I'm not previliged to sharing the details - firstly I didn't write it, and secondly it's the company's IP. Please ask others.
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10590
Location: Germany


flag
PostPosted: Sun Feb 04, 2007 3:21 am 
Post subject: Re: error - The pipe has been ended

The XML-Pipeline can only used together with XML files. Normally, we read XML files and parse them, with the XML pipeline you configure a SAX parser and we read the output of the sax parser instead of the XML file. So like in your case, instead of reading one XML node with 20'000 childs, the SAX parser will shred that into 20'000 XML "files" - the output of the pre-parsing.


It seems you implemented a function based adapter: https://boc.sdn.sap.com/node/5946
Something that acts and feels like a stored procedure. You call it with input parameters, then the procedure dows something and once it is done, will return a result.

I haven't implemented the "send email" part in above sample yet, but you can always go to the \DI\adapters\sdk\sample\testadapter\..." directory and strip down the sample code there to the function based version. In case you need help, let me know.

_________________
Werner Daehn
Product Manager Data Services
SAP AG
“The opinions expressed in this post are my own views and not those of SAP.”

PLEASE SUPPORT THIS: Run one ATL job and tell us the benchmark result. Details to be found here.


Last edited by Werner Daehn on Wed Aug 06, 2008 3:02 am, edited 1 time in total
Back to top
rachel.l.su
Senior Member
Senior Member



Joined: 30 Nov 2004

Posts: 48
Location: Melbourne



PostPosted: Wed Feb 07, 2007 9:35 pm 
Post subject: Re: error - The pipe has been ended

Thanks so much for the explanation Werner! Also thanks to dnewton for the copying advice. Will have that in mind in the next design. icon_biggrin.gif
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: Performance and Tuning  Previous TopicPrint TopicNext Topic
Page 1 of 2 All times are GMT - 5 Hours
Goto page 1, 2  Next
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Get community updates via Twitter:

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.0507 seconds using 17 queries. (SQL 0.0034 Parse 0.0364 Other 0.0109)
CCBot/2.0 (http://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo