BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Want to sponsor BOB? 
Want to sponsor BOB? (Opens a new window)  

General Notice: No events within the next 45 days.

Pipe error - how to map back to specific transform?

Goto page 1, 2  Next
 
Search this topic... | Search DI: Designer and Job Design... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: Designer and Job Design  Previous TopicPrint TopicNext Topic
Author Message
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1110
Location: Wellington, NZ


flag
PostPosted: Wed May 26, 2010 7:18 pm 
Post subject: Pipe error - how to map back to specific transform?

A "Pipe ended" error happened while executing one of our data migration jobs. It's a bit random as I ran the same job, with the same volume of data (a few million records) several times before without any problems. Some parts of the job are quite memory intensive, there are plenty of lookups and custom functions to crunch the data.

(Our Job Server resides on a virtual server which shares resources with other virtual servers so it could be related to that.)

In the error log, the following string of errors were listed:

Code:

Pipe Listener for DF_TR_DO33_WO_Transform_1_3 Named pipe error occurred: <The pipe has been ended.>

SubDataflow DF_TR_DO33_WO_Transform_1_1|Pipe Listener for IPCTarget2_Query-Function5_common Named pipe error occurred: <The pipe has been ended.>
                                                     
SubDataflow DF_TR_DO33_WO_Transform_1_2|Pipe Writer for IPCSource1_Query-Function4_common Named pipe error occurred: <The handle is invalid.>
                                                   
Data flow <DF_TR_DO33_WO_Transform_1_3> with pid <4440> failed to stop. Diagnostic information <1-4-021-1001-Error from Job>.>


When looking in my trace log, they relate to these items:
Code:

DF_TR_DO33_WO_Transform_1_3/IPCTarget2_Query-Function5_common
DF_TR_DO33_WO_Transform_1_3/IPCSource1_Query-Function4_common


Is there any way to relate these back to specific function calls or transforms within the DataFlow?
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10590
Location: Germany


flag
PostPosted: Thu May 27, 2010 1:05 am 
Post subject: Re: Pipe error - how to map back to specific transform?

Pipe ended means that your dataflow was split into subdataflows that are connected via a inter-process-communication writer and reader (IPCSource, IPCTarget) to stream data from one sub-dataflow to the other. One of the two ended and hence the pipe between them is not valid anymore.

Very likely because too much memory was tried to be allocated?

_________________
Werner Daehn
Product Manager Data Services
SAP AG
“The opinions expressed in this post are my own views and not those of SAP.”

PLEASE SUPPORT THIS: Run one ATL job and tell us the benchmark result. Details to be found here.
Back to top
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1110
Location: Wellington, NZ


flag
PostPosted: Thu May 27, 2010 1:26 am 
Post subject: Re: Pipe error - how to map back to specific transform?

wdaehn wrote:
Pipe ended means that your dataflow was split into subdataflows that are connected via a inter-process-communication writer and reader (IPCSource, IPCTarget) to stream data from one sub-dataflow to the other. One of the two ended and hence the pipe between them is not valid anymore.

Very likely because too much memory was tried to be allocated?


Yes, that's what I thought too.. but how can I easily find out which function is consuming that much memory? It's arather large dataflow.

Hence my question if I can somehow trace "Function4" or "Function5" back to specific lookup_ext or custom function calls in the dataflow icon_smile.gif
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10590
Location: Germany


flag
PostPosted: Thu May 27, 2010 3:16 am 
Post subject: Re: Pipe error - how to map back to specific transform?

That's my point. The error is raised in the IPC Reader/Loader. But the root cause is somewhere else. There is no easy way to find out, unfortunately.

What would happen if you change the dataflow to use pageable cache? What is the biggest dataset you are caching with lookups? Do all cachings make sense or would for some be a row-by-row better anyhow? (And there is an index to support the row-by-row select)

_________________
Werner Daehn
Product Manager Data Services
SAP AG
“The opinions expressed in this post are my own views and not those of SAP.”

PLEASE SUPPORT THIS: Run one ATL job and tell us the benchmark result. Details to be found here.
Back to top
dnewton
Forum Addict
Forum Addict



Joined: 30 Jan 2004
ASUG Icon
Posts: 3458
Location: Massachusetts, USA


flag
PostPosted: Thu May 27, 2010 6:52 am 
Post subject: Re: Pipe error - how to map back to specific transform?

I would throw in some Data Transfer steps to see at which point the problem is occurring.
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10590
Location: Germany


flag
PostPosted: Thu May 27, 2010 9:57 am 
Post subject: Re: Pipe error - how to map back to specific transform?

Mister Pragmatic icon_wink.gif


seeya.gif

_________________
Werner Daehn
Product Manager Data Services
SAP AG
“The opinions expressed in this post are my own views and not those of SAP.”

PLEASE SUPPORT THIS: Run one ATL job and tell us the benchmark result. Details to be found here.
Back to top
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1110
Location: Wellington, NZ


flag
PostPosted: Thu May 27, 2010 3:12 pm 
Post subject: Re: Pipe error - how to map back to specific transform?

wdaehn wrote:
What would happen if you change the dataflow to use pageable cache? What is the biggest dataset you are caching with lookups? Do all cachings make sense or would for some be a row-by-row better anyhow? (And there is an index to support the row-by-row select)


It is set to pagable cache, the biggest data set is about 3 million records but they are quite lean so it should fit in cache. I'm not using a TC transform - it's a straight forward data pump with transformations.

But I'll chuck in a data transform and see where it goes wrong but it hasn't failed anymore since. According to the IT team, the host server was also being heavily utilized at the time of failure and since all resources are set to be shared, it may have had an effect as well?
Back to top
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1110
Location: Wellington, NZ


flag
PostPosted: Wed Jul 14, 2010 5:39 pm 
Post subject: Re: Pipe error - how to map back to specific transform?

The Data Transfers did not help. The problem is that I can run the job throughout the day just fine, but when I schedule it to run at night, it keeps coming up with this named pipe error.

I'm open to suggestions icon_rolleyes.gif
Back to top
Werner Daehn
Forum Devotee
Forum Devotee



Joined: 17 Dec 2004

speaker.gif*6
Posts: 10590
Location: Germany


flag
PostPosted: Thu Jul 15, 2010 2:04 am 
Post subject: Re: Pipe error - how to map back to specific transform?

Optimize the dataflow. Can you post the ATL for the DF or a screenshot plus descriptions of what it is doing?
_________________
Werner Daehn
Product Manager Data Services
SAP AG
“The opinions expressed in this post are my own views and not those of SAP.”

PLEASE SUPPORT THIS: Run one ATL job and tell us the benchmark result. Details to be found here.
Back to top
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1110
Location: Wellington, NZ


flag
PostPosted: Mon Jul 19, 2010 4:32 pm 
Post subject: Re: Pipe error - how to map back to specific transform?

The problem is the ESX Server, not the data flow. The records volume is consistent, the transformations are consistent - but something at night is putting a large load on the entire ESX Server. It looks like it just runs out of resources.

(Did I already mention that I do not like nor believe in virtual servers for large ETL loads?)
Back to top
eganjp
Forum Addict
Forum Addict



Joined: 12 Sep 2007
ASUG Icon
Posts: 4278
Location: Denver Area


flag
PostPosted: Thu Jul 22, 2010 8:20 pm 
Post subject: Re: Pipe error - how to map back to specific transform?

Unfortunately I have run into the same problem. Run the job at a scheduled off-hours time and it works some times and fails other times. Run the job during the day and it never fails. This was in a dataflow processing 60 million rows.

There is a timeout setting in the configuration (it escapes me at the time though) that may help. It was recommended to me here on the forum. But it didn't work for me.

What did work was to write the intermediate data (in the branch of the dataflow that was failing) to a staging table and then work on that data again in a subsequent dataflow. After I did that I had no more problems. It's ugly and I'm not happy with it, but I don't have to babysit it anymore.

_________________
Jim Egan
ProKarma-an SAP Consulting Partner
Speaker at RMOUG Training Days 2013
Speaker at Sapphire Now 2013
Speaker at Business Objects User Conference 2013
-My ETL is faster than your database! - Advanced ETL performance tuning for Data Services
-Kill ETL bugs before they kill you!
My ETL blog

Latest Blog Posts
• 2014-04-17 Can Your ETL Make Your Reporting Faster?
• 2014-04-04 ProKarma Acquires Mantis Technology Group
• 2014-04-02 How to Create Charts in Webi (Part 2 of 2)
Back to top
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1110
Location: Wellington, NZ


flag
PostPosted: Sun Jul 25, 2010 6:07 pm 
Post subject: Re: Pipe error - how to map back to specific transform?

I've waived the white flag and have broken down the data flow into 3 seperate data flows - each populating a staging table. So far, the job has been running stable but it hasn't been a week yet so it may be too early to tell.

However, if this approach works then I really wonder why two Data Transforms wouldn't solve the problem? I have broken up the data flow at the same spot as where I had inserted two Data Transforms previously. Both were set to use a table. In essence, this should be exactly the same as breaking up the data flow into 3 data flows?
Back to top
eganjp
Forum Addict
Forum Addict



Joined: 12 Sep 2007
ASUG Icon
Posts: 4278
Location: Denver Area


flag
PostPosted: Sun Jul 25, 2010 6:19 pm 
Post subject: Re: Pipe error - how to map back to specific transform?

My understanding is by creating a dead-end on one branch - by writing to a table - it closes out that branch of the flow and no communication between the branches is necessary. If you use a Data Transform then the branches must maintain communications.
_________________
Jim Egan
ProKarma-an SAP Consulting Partner
Speaker at RMOUG Training Days 2013
Speaker at Sapphire Now 2013
Speaker at Business Objects User Conference 2013
-My ETL is faster than your database! - Advanced ETL performance tuning for Data Services
-Kill ETL bugs before they kill you!
My ETL blog

Latest Blog Posts
• 2014-04-17 Can Your ETL Make Your Reporting Faster?
• 2014-04-04 ProKarma Acquires Mantis Technology Group
• 2014-04-02 How to Create Charts in Webi (Part 2 of 2)
Back to top
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1110
Location: Wellington, NZ


flag
PostPosted: Sun Jul 25, 2010 7:42 pm 
Post subject: Re: Pipe error - how to map back to specific transform?

True but this problem started before I inserted any Data Transforms - half way through the data flow, it just loses the plot somehow. So I wonder what on the job server is causing it to lose communication with all the named pipes? Is there something broken in the threading process?
Back to top
dnewton
Forum Addict
Forum Addict



Joined: 30 Jan 2004
ASUG Icon
Posts: 3458
Location: Massachusetts, USA


flag
PostPosted: Mon Jul 26, 2010 5:03 pm 
Post subject: Re: Pipe error - how to map back to specific transform?

Do you have "run as a separate process" checked off on any of the steps in your dataflow? If so, try turning that off.

We get broken pipe errors (Win 2003, now Win 2008) under 12.2, quite often. Never in the same dataflow or at the same time of day.
cuss.gif
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: Designer and Job Design  Previous TopicPrint TopicNext Topic
Page 1 of 2 All times are GMT - 5 Hours
Goto page 1, 2  Next
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Get community updates via Twitter:

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.0356 seconds using 17 queries. (SQL 0.0032 Parse 0.0010 Other 0.0315)
CCBot/2.0 (https://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo