Home:ALL Converter>Python imports in Hadoop

Python imports in Hadoop

Ask Time:2015-04-16T04:40:57         Author:Hangman4358

Json Formatter

Ok, I am writing a java based Hadoop MR task. Part of the task is calling an outside python script as a new Process, passing it information and reading it back the resut. I have done this a few times before without problems when not working with hadoop.

I can also call a single python script as a new process in hadoop when it does not import anything, or only things that are on the nodes' python install.

the current python script calls an import on another script which is usually just sitting in the same directory and that works fine when not running on hadoop.

In hadoop I have added both files to the distributed cache so I do not understand why the script could not import the other one.

Author:Hangman4358,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/29660487/python-imports-in-hadoop
yy