I have a Python UDF which is connecting to HBase using Happybase. If I run the code from Python 2.7 it works perfectly.
However when I call the Python UDF from Pig 0.15.0 I am getting the following error:
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1121: Python Error. Traceback (most recent call last): import happybase ImportError: No module named happybase
In my Pig script I am registering my Python script (pigtest.py) like this:
REGISTER 'pigtest.py' using jython as myfuncs;
I tried to set the Happybase path in my Python script as follows but that didn't make a difference:
import sys
sys.path.append('/usr/local/lib/python2.7/dist-packages/happybase')
import happybase
I also tried adding "/usr/local/lib/python2.7/dist-packages/happybase" to the JYTHON_PATH in the .bashrc file (I'm on Ubuntu) but same error comes up.
It seems to me like I need to set the Happybase path somewhere, but I can't figure out where.
I was able to solve this with the help of the Jython user mailing list. You need to specify the parent directory of the Happybase folder, not the path to the Happybase dir itself like I was doing, by doing one of the following:
Append the location to the sys.path in the Python script:
import sys
sys.path.append('/usr/local/lib/python2.7/dist-packages')
import happybase
OR
Add the location as an environment variable to JYTHONPATH (not JYTHON_PATH!):
export JYTHONPATH = $JYTHONPATH:/usr/local/lib/python2.7/dist-packages
Related
I followed the instructions and successfully installed pypostal python package (package to help parse addresses) https://github.com/openvenues/pypostal.
I'm trying to set this up so the python script can be callable through an apache server on an ubuntu box. It works fine when executing the script from Terminal. However, it doesn't work when I make a call to apache to execute the script and I get the following error in the apache logs. I believe it might be some pathing issue but I haven't had much luck to resolve it. Any ideas will be appreciated.
Error:
File "/var/www/html/cgi-bin/get_parsedAddress.py", line 5, in
from postal.parser import parse_address
ModuleNotFoundError: No module named 'postal'
python script contents:
import sys
from postal.parser import parse_address
addressList = parse_address(sys.argv[1])
print(addressList)
Structure of my working directory:

The import command in the train.py file:
from src.pl_data_modules import ConsecDataModule
It works when I type them in the python command line interface but leads to a ModuleNotFoundError when I directly run train.py.
I know that this is relevant to the python import mechanism but how to solve this issue?
I'm having problem when I trying to import a custom module I have which is very simple but I'm always getting:
Traceback (most recent call last):
File "demo_module1.py", line 12, in <module>
import mymodule
ModuleNotFoundError: No module named 'mymodule'
I have tried to set environment variables:
set PYTHONHOME=C:\Software\python-3.7.4
set PYTHONPATH=%PYTHONPATH%;C:\pyproys\test
Everything is located here: 'C:\pyproys\test'
The only way it works is if I add it directly in the code "But I don't want to do it in every single script I have so don't want to maintain it in that way".
import sys
sys.path.append('C:\pyproys\\test')
print(sys.path)
Here is the script I'm trying to run:
demo_module1.py
import mymodule
mymodule.greeting("Jonathan")
'mymodule.py' is in the same folder as 'demo_module1.py'
I'm expecting the code to run fine by just executing:
python demo_module1.py
Can someone please point me out what I'm doing wrong?
Try to find the directory /lib/site-packages in your Python folder in C drive and paste your module in that folder and then restart the system. Hope it'll solve your issue.
Whenever I try to import external packages of python in a pig udf, it shows the following error
Python Error. Traceback (most recent call last):
File "pythonudf.py", line 5, in
from bs4 import BeautifulSoup
ImportError: No module named bs4
I've tried including the library path
import sys
sys.path.append('/usr/local/lib/python3.5/dist-packages')
And set
export JYTHONPATH=$JYTHONPATH:/usr/local/lib/python3.5/dist-packages
But it is still showing the same error.
What else can I do?
The script isn't running in local or mapreduce mode.
PS: Other functions which do not import external packages are running perfectly.
EDIT:
The packages in the python code are installed.
Use -embedded option when executing pig with python udf importing packages.Reference
pig -embedded jython pythonudf.py
When I execute a Python script I receive:
Traceback (most recent call last):
File "/Users/.../Documents/development/python/migrate_upper.py", line 3, in <module>
import mysql.connector
ImportError: No module named connector
I am executing like that:
$ python migrate_upper.py
It worked 1 month ago, I haven't worked with Python since then. I spent 2 hours trying to understand what is wrong but I got lost with PYTHONPATH, pip and other hints.
However when I send the script to Python shell:
$ python < migrate_upper.py
everything works. I think that this is not the proper way to execute python scripts. How can I get the script working without the Python shell?
I resolved it with the help of Ethan's comment.
I had a folder mysql beneath ../development/python. I can't remember why I put it there. I guess python tried to import that folder instead of ../site-packages/mysql.