I have a python project which i built into a single executable on Linux using pyinstaller. The python also uses python's sh module.
While running the single executable, it seems the following commands do not work:
getattr(sh, 'mount')
getattr(sh, 'ls')
Although when i check the os.environ['PATH'] that is set correctly and the following work:
sh.which('mount') -> /bin/mount
I understand that sh module behaves differently than other python packages as it dynamically import commands. etc.
Is that causing a problem with pyinstaller?
To reproduce your issue more exactly:
cat > test.py << EOF
import sh
ls = getattr(sh, 'ls')
print(ls('-d', '/tmp'))
EOF
Normally, this works fine:
$ python test.py
/tmp
But with PyInstaller, AttributeError is raised:
$ ./pyinstaller test.py
$ ./dist/test/test
Traceback (most recent call last):
File "<string>", line 3, in <module>
AttributeError: 'module' object has no attribute 'ls'
You are correct, PyInstaller has a custom module loader, which does not interplay well with the sh module. Briefly.
This can be resolved by reproducing the same "Magic" that sh.Environment.__getitem__ method performs.
We may modify test.py as follows:
import sh
ls = sh.Command._create('ls')
print(ls('-d', '/tmp'))
and this works fine.
$ ./pyinstaller test.py
$ ./dist/test/test
/tmp
sh.py overloads sys.modules[__name__] with an instance of sh.SelfWrapper. This is where it conflicts with PyInstaller, whose custom Module importer does not evaluate sys.modules as standard python.
Related
I have the following scenario:
A Python3 Package (meross_iot) installed through pip and located in
~/.local/lib/python3.7/site-packages
A Python script (meross_electricity.py) that imports from this package: from meross_iot.controller.mixins.electricity import ElectricityMixin
A shell script (launcher.sh) that is meant to be a wrapper so that the .py script is run at startup:
#!/bin/sh
# launcher.sh
# navigate to home directory, then to this directory, then execute python script, then back home
cd /
cd home/pi/Documents
sudo python meross_electricity.py
cd /
If I simply execute the .py file everything works as expected, imports done, etc. If I try to run the .sh script I get the following error:
pi#home:~/Documents $ ./launcher.sh
Traceback (most recent call last):
File "meross_electricity.py", line 4, in <module>
from meross_iot.controller.mixins.electricity import ElectricityMixin
ModuleNotFoundError: No module named 'meross_iot'
Can someone please help me solve this issue?
Thanks!
Installed package with sudo worked .
Answer based on comment by #KlausD
This question already has answers here:
What is the purpose of the -m switch?
(6 answers)
Closed 2 years ago.
I'm seeing different package handling behavior in python3 when I'm running something like python -m moduala.test and python moduala/testa.py. In the -m case imports for moduala.modualb.pkgx in testa.py work and in the other case they don't work.
python --version
# Python 3.6.9
mkdir -p testpyproj/pkga/pkgb
cd testpyproj/
touch pkga/__init__.py
touch pkga/pkgb/__init__.py
echo 'print("# python!")' >pkga/pkgb/modx.py
echo 'from pkga.pkgb import modx' > pkga/test.py
python -m pkga.test
# python!
python pkga/test.py
# Traceback (most recent call last):
# File "pkga/test.py", line 1, in <module>
# from pkga.pkgb import modx
# ImportError: No module named pkga.pkgb
From the man page I would expect these to be the same:
-m module-name
Searches sys.path for the named module and
runs the corresponding .py file as a script.
What is the difference between python -m and python [file] for loading modules and handling packages?
Why does the python [file] not find pkga.pkgb?
When python runs a script from the command line, like:
$ python pkga/test.py
it knows that it is running a relative script. And so, as a result, it changes the value of sys.path[0] to be the directory of the script.
When python opens a module from the command line, it has to have a module search path (aka sys.path) in order to find the module. So it sets sys.path[0] to "" (empty string) to indicate "the current directory". Then it searches through all the places indicated by sys.path trying to find the module name you have given.
The result of this is that in one case, your sys.path[0] is going to be "" while in the other case it will be /path/to/current/directory/pkga. And then it cannot resolve pkga.pkgb when the search path starts from that location.
You can confirm this by adding:
import sys
print(sys.path)
to the start of pkga/test.py, before any other imports. You will see the print output just before the module import exception.
This is my python script:
!/usr/bin/python
import sys
import magic
m=magic.from_file('<file with absolute path goes here>')
print(m)
On running this from the command line:
$ python script.py
Microsoft Word 2007+
results in the output of the TYPE of the document that file is. Using python-magic.
Now running the same script from scala using below code:
import sys.process._
def compWithLibmagic(){
val result = "python /script.py" !
}
throws the below error:
Traceback (most recent call last):
File "script.py", line 3, in <module>
import magic
ImportError: No module named magic
PS: I have both python 2.7 and python 3.6 installed on my machine and running the script using any of them from the command line runs just fine so I guess both of them are bundled with the MAGIC packages correctly.
Would highly appreciate any kind of help.
I will try to answer my own question in the best possible way.
While trying to execute the script.py from the command line using python compiler python2.7 was being used and when I tried to invoke the same script.py from the scala compiler using
import sys.process._
def compWithLibmagic(){
val result = "python /script.py" !
}
it was using python3 which was getting unnoticed. It took me a couple of days to actually figure this thing out and finally I removed all the python2.7 and python3 libraries and installed python 2.7 again and it worked like a charm.
It works from both python3 command and Scala code for me.
This should work.
import sys.process._
def compWithLibmagic(){
val result = "python3 script.py".!!
}
I'm trying to call a python script from a bash script. I get import errors only if I try to run the .py from the bash script. If I run with python myscript.py everything is fine. This is my bash script:
while true; do
python script.py
echo "Restarting...";
sleep 3;
done
The error I get:
Traceback (most recent call last):
File "script.py", line 39, in <module>
from pokemongo_bot import logger
File "/Users/Paolo/Downloads/folder/t/__init__.py", line 4, in <module>
import googlemaps
ImportError: No module named googlemaps
There is more to this story that isn't in your question.
Your PYTHONPATH variable is getting confused somewhere along the way.
Insert a couple quick test lines:
in bash:
echo $PYTHONPATH
in your python:
import os
print os.environ["PYTHONPATH"]
At some point, the path to googlemaps got lost.
Your problem is in the script itself, your bash code is OK!. If you don't have problem running python scrip.py from bash directly, you should test if you use the same interpreter for both calls. You can check the shebang line in the python script, it is the first line in the file for example #!/usr/bin/env python or #!/usr/bin/python and compare it to the output of which python command if the output is different try to change or add the shebang line in to the file. If you call directly file in bash ./some_script.py bash reads the first line and if it is shebang line he wil execute the specific command for the file. My point is that if you use two diferent interpreters for calling file directly with python script.py and indirectly ./script.py one of them may not have the proper python modules.
Howto code:
$ which python
/usr/local/bin/python
So the second line is the path for your interpreter to build a shebang from it write in the first line of your script file this.
#!/usr/local/bin/python
I am am using runit to manage a process on Ubuntu 12.04. I get the below error in the logs when I run:
sv up test/
I assume it is a python path issue.
ImportError: No module named htcommon.ht_redis
Traceback (most recent call last):
File "/home/ubuntu/workspace/htFrontEnd/htanalytics/ht_rpc_server.py", line 17, in <module>
from htpData import HTPItemBase, HTPUserBase
File "/home/ubuntu/workspace/htFrontEnd/htanalytics/htpData.py", line 9, in <module>
from htcommon.ht_redis import HTRedisConnection
ImportError: No module named htcommon.ht_redis]
I have also set the path in /etc/environment and also set in .bashrc.
Below is my runit script.
#!/bin/sh
exec 2>&1
exec export PYTHONPATH=$PYTHONPATH:/home/ubuntu/workspace/htFrontEnd/heythat
exec export PYTHONPATH=$PYTHONPATH:/home/ubuntu/workspace/htFrontEnd/heythat/htanalytics
exec /usr/bin/python /home/ubuntu/workspace/htFrontEnd/htanalytics/ht_rpc_server.py >> /tmp/ht_rpc_server.log 2>&1
root#aws-rpc-server-east-staging-20130203070552:/etc/sv#
When I run the process from the command line I get no issues and it works.
/usr/bin/python /home/ubuntu/workspace/htFrontEnd/heythat/htanalytics/ht_rpc_server.py
Why will runit not work? Why can it not find the path?
Look like a few potential problems with this script. I don't think an 'export' can be 'exec'd. At least, it fails in my version of bourne. The exec command replaces the current process (your script) with the called command and doesn't return unless the called command fails. So, it doesn't make sense to 'exec' the exports anyway. Also they can be collapsed into one line. So, you're script should look more like this:
#!/bin/sh
export PYTHONPATH=$PYTHONPATH:/home/ubuntu/workspace/htFrontEnd/heythat:/home/ubuntu/workspace/htFrontEnd/heythat/htanalytics
exec /usr/bin/python /home/ubuntu/workspace/htFrontEnd/htanalytics/ht_rpc_server.py >> /tmp/ht_rpc_server.log 2>&1