BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Follow BOB on Twitter! 
Follow BOB on Twitter! (Opens a new window)  

General Notice: BOB is going to retire...please see details here.
General Notice: No events within the next 45 days.

Add third-party Python modules?


 
Search this topic... | Search DI: Installation, Administration, and Connectivities... | Search Box
Register or Login to Post    Forum Index -> Data Integrator -> DI: Installation, Administration, and Connectivities  Previous TopicPrint TopicNext Topic
Author Message
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1111
Location: Wellington, NZ


flag
PostPosted: Sun Sep 01, 2013 7:30 pm 
Post subject: Add third-party Python modules?

Is it possible to add/install/import third party Python modules - to extend the functionality of the User Defined Transform (UDT)?

Currently UDT only supports the out-of-the-box Python 2.6 functionality and I find myself re-inventing the wheel over again, despite third-party modules being available that would take care of this.

I suspect that all the code in the UDT gets parsed by the Python compiler externally to Data Services - so once the modules have been installed, it should be possible to import them as part of the UDT?

Has anyone had any success with this?
Back to top
jlynn73
Forum Associate
Forum Associate



Joined: 27 Oct 2009

Posts: 573
Location: DesMoines Iowa


flag
PostPosted: Thu Sep 05, 2013 8:00 am 
Post subject: Re: Add third-party Python modules?

I actually use a lib folder on our SAN that (most) all my python resides.

That path is added to the python search path so when you do an import it also searches that folder.

I dont have direct access to the python install folder on the job server, so this helps me out a lot. I still need our admin to install new modules, but I can put my code where I can easily access it.
Back to top
ErikR
Forum Enthusiast
Forum Enthusiast



Joined: 10 Jan 2007

Posts: 1111
Location: Wellington, NZ


flag
PostPosted: Thu Sep 05, 2013 6:51 pm 
Post subject: Re: Add third-party Python modules?

Thanks. I managed to get this working my ensuring that the Python directory and underlying Script/Lib directories are all added to the path on the server. That way, I can use any external library by just referencing it in my Python code within the UDT. (e.g. import oauth2 etc.)

By adding the paths, I can now also use PIP to add new modules etc.
Back to top
bmvrus
Principal Member
Principal Member



Joined: 08 May 2011

Posts: 227
Location: Moscow



PostPosted: Tue Jan 28, 2014 5:23 pm 
Post subject: Re: Add third-party Python modules?

Please vote

Add support Python 3.3.x in DS


https://ideas.sap.com/ct/ct_a_view_idea.bix?c=34503CB9-F213-4EF9-8603-E500CB16D712&idea_id=63BBBB2B-3E0B-4444-93E6-F23C7BE977C0
Back to top
jlynn73
Forum Associate
Forum Associate



Joined: 27 Oct 2009

Posts: 573
Location: DesMoines Iowa


flag
PostPosted: Thu Jan 30, 2014 11:51 am 
Post subject: Re: Add third-party Python modules?

If they went to python 3+ I would migrate all my python code out of DS.
Back to top
eganjp
Forum Addict
Forum Addict



Joined: 12 Sep 2007
ASUG Icon
Posts: 4280
Location: Denver Area


flag
PostPosted: Thu Jan 30, 2014 1:09 pm 
Post subject: Re: Add third-party Python modules?

jlynn73 wrote:
If they went to python 3+ I would migrate all my python code out of DS.


I don't use Python so I'm curious why you said this.

_________________
Jim Egan
ProKarma-an SAP Consulting Partner
Speaker at RMOUG Training Days 2013
Speaker at Sapphire Now 2013
Speaker at Business Objects User Conference 2013
-My ETL is faster than your database! - Advanced ETL performance tuning for Data Services
-Kill ETL bugs before they kill you!
My ETL blog

Latest Blog Posts
• 2014-04-17 Can Your ETL Make Your Reporting Faster?
• 2014-04-04 ProKarma Acquires Mantis Technology Group
• 2014-04-02 How to Create Charts in Webi (Part 2 of 2)
Back to top
Johannes Vink
Forum Enthusiast
Forum Enthusiast



Joined: 20 Mar 2012

Posts: 1138


flag
PostPosted: Fri Jan 31, 2014 2:58 am 
Post subject: Re: Add third-party Python modules?

As far as I know between the 2.x and 3.x branch there is not much backwards compatability. So 3.x is not definitely better based on the version number.

But jlynn73 can explain this better, I think he is the expert on Python here. I only copy pasted some things together until it worked... Sometimes you have requirements that are not possible in BODS, or not in a performing way...

Interesting enough I am considering moving the python code out of BODS. Because it comes with the DQ license and that is a heavy license. So writing a flat file to the disk, call python via the command line will also work. Does it give overhead? Probably, but then you could jump to any version of Python you want and implement more fancy stuff as parallel processing.

Setting the DOP of the DF with a DQ Python component also works, but I have the feeling it is less efficient.

_________________
Consultant at Teradata

The opinions expressed in this post are my own views and not those of Teradata. Apply any advise in this post at your own discretion.
Back to top
jlynn73
Forum Associate
Forum Associate



Joined: 27 Oct 2009

Posts: 573
Location: DesMoines Iowa


flag
PostPosted: Sun Feb 02, 2014 2:52 pm 
Post subject: Re: Add third-party Python modules?

There is no fear of migrating that which is not inside DataServices. yesnod.gif

I spent lot of time converting python scripts into DS scripting language, or out converted it. I believe I shaved 60% off our merge purge match processing times by taking the python out. Using python isnt such a big deal until you start pushing millions of records through it, combined with several other queries... and it becomes pretty inefficient.

If you are planning on using it, and are using a large dictionary/complex data structures... a neat trick is to pickle the data structure to a file. Then you can work against the data structure outside of DS, finish the code and port it back in to DS.

love the cPickle module.
Back to top
zordon
Forum Member
Forum Member



Joined: 19 Sep 2019

Posts: 1



PostPosted: Mon Oct 07, 2019 1:38 am 
Post subject: Re: Add third-party Python modules?

I noticed that pip was not installed on the job server but luckily found two alternatives:
1. if you have admin right to the server - extract the 3rd party module to a folder and then run python setup.py install for each module you need installed
2. without admin rights but with write access to the Python's install folder on the server, first install the 3rd party module on your local machine and then copy and paste the relevant package from your Python27\Lib\site-packages to the server's Python's \Lib\site-packages folder

Re: Python 3, I read it will form part of DS 4.2 SP13 when that becomes available.
Back to top
jlynn73
Forum Associate
Forum Associate



Joined: 27 Oct 2009

Posts: 573
Location: DesMoines Iowa


flag
PostPosted: Wed Oct 09, 2019 1:38 pm 
Post subject: Re: Add third-party Python modules?

install PIP first and then use PIP for the rest of your modules.

I'm not looking forward to migrating 75,000 lines of 2.7.9 Python code.
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> Data Integrator -> DI: Installation, Administration, and Connectivities  Previous TopicPrint TopicNext Topic
Page 1 of 1 All times are GMT - 5 Hours
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Get community updates via Twitter:

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.0391 seconds using 17 queries. (SQL 0.0127 Parse 0.0009 Other 0.0255)
CCBot/2.0 (https://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo