Develop Reference

How does python use paths when a script is called using eval() from a php script and when the same function is called from a bash script?

I have an executable python script which archives data from mysql server using the pymysql library. The script works well from the command line.
I call this script from a php script using escapeshellcmd function and I've gotten it to work.
I also have created a bash script that I intend to use from crontab to archive the information as well. I can make this script work as well, by making changes outlined below.
Somehow I have gotten into python versions and path problems.
if I include
#!/home/tim/anaconda3/bin/python
as the first line of the python script it works when called by the php script (using www-data as the user, I believe). It doesn't work from the bash script or the command line, giving the following error:
File "./signal_archive.py", line 22, in <module>
import pymysql
ModuleNotFoundError: No module named 'pymysql'
However, if the first line of the python script is as follows:
#!/usr/bin/python3
the script works from the bash script and the command line but not from the php script. It gives the following error:
File "/home/tim/python/commodities_related/signal_archive.py", line 23, in <module>
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
Both packages are installed on my system. Thinking pointing the script to the path would help, I added the following to the python script but no luck so far.
sys.path.append('/usr/lib/python3/dist-packages:')
sys.path.append('/usr/local/lib/python3.5/dist-packages:')
There is obviously something I'm missing; I think it is that php script is called by www-user and I don't know the default path. The bash file is called by my user with the path specified in the .bashrc file. However, I may need to point the apache or php (www-user) to use a specific installation of python.
EDIT-
To be more clear, a php script (phpfile1.php) calls the python script. When I call phpfile1.php from another php script (phpfile2.php) running on apache2 I everything works using the
#!/home/tim/anaconda3/bin/python
When I call the same file (phpfile1.php) from a different php script (phpfile3.php) from a bash script it fails.
Additionally, if I run the file in place using the following
./signal_archive.py
I get the error but if I run it using the following command it works:
python signal_archive.py
Any ideas if this is right or how to do it? Thanks.

I fixed this in 2 steps:
It turns out that I needed to add the path to anaconda3 to my .bash_profile file.
export PATH="/home/tim/anaconda3/bin:$PATH"
When anaconda3 is installed it modifies the .bashrc file with the previous code snippet. However .bash_profile made the difference in this case.
I also modified the top of the python file to use the anaconda path for execution, as well as add the path for the specific python packages.
#!/home/tim/anaconda3/bin/python
import sys
sys.path.insert(1, '/usr/local/lib/python3.5/dist-packages')

Run Python script with Automator // why does it only work if I include export PATH=/usr/local/bin:$PATH and what does it mean?

I was trying to run a Python script via Mac's Automator and the command is very straight forward:
"cd /Users/myname/Desktop/project && python3 myprojectapp.py".
However, every time I tried to run it, Automator raised an error such as ModuleNotFoundError. This was however, impossible since I had all libraries (e.g. Pandas) installed and running the command in the Terminal as written above worked flawlessly.
Now, I've read somewhere for a similar problem to just include:
"export PATH=/usr/local/bin:$PATH" before the command and it worked. Now, before I go on with my life, I would like to understand what exactly this extra line does and how it affects Automator to the point of making the script work.
Thank you in advance!

That command basically modifies the environment variable PATH and puts the directory /usr/local/bin before everything that is currently in PATH. However, that command is temporary, and the environment variable PATH is restored when the session closes.
What could be happening is the python you're running in terminal and the python Automator is running are different./usr/local/bin probably contains the same python version as you are using in terminal. Take a look at ~/.bash_profile to see if something similar to export PATH=/usr/local/bin:$PATH is in there.
Another way to check is to type which python in both and see if it points to the same python. You probably have yet another python somewhere in the list of directories in your PATH variable.
It's common to use virtual python environments to keep track of which python is running and to experiment with python without messing with system python. Examples of these include: Anaconda and virtualenv.

Module Not Found in python script

I'm a python beginner, and I'm following this tutorial for a webscraper https://hackernoon.com/building-a-web-scraper-from-start-to-finish-bb6b95388184
I'm on Windows 10, have setup a venv, activated and installed 2 modules using pip, and moved my script into the Scripts folder (from my understanding, this is the equivalent of the /bin/ folder on linux installations). The modules are bs4 and requests. I see both of these in the /Lib/ folder of my venv. I am using the Atom editor from atom.io, and the Scripts Package to run my script.
My script errors with a "module not found" error. Relevant snippet below:
scraper.py
from bs4 import BeautifulSoup
import requests
I get the error on both imports, indicating I've setup my project/imports incorrectly. I have no shebang line in my script, and suspect this is the problem.
My project structure looks like:
\ScraperProject
|-\ScrEnv
|-\Include
|-\Lib
|-\site-packages
|-\bs4
|-\requests
|-\Scripts
|-scraper.py
|-pyvenv.cfg
What is the proper way for me to import these modules into my script in a Windows environment?

with what python version are you working in the venv? Maybe try to uninstall those update pip and reinstall those again, making sure that you install them with pyhton3

My issue is related to running it from the Atom editor, something I hadn't considered before. It's calling the python executable from my PATH variable, and not the one in my venv. When running the script from the cmd window, and calling it with the python from my project folder/venv, it runs as expected. On to figuring out how to configure Atom to use the venv executables. Thank you very much for the interest and help!

Virtualenv within single executable

I currently have an executable file that is running Python code inside a zipfile following this: https://blogs.gnome.org/jamesh/2012/05/21/python-zip-files/
The nice thing about this is that I release a single file containing the app. The problems arise in the dependencies. I have attempted to install files using pip in custom locations and when I embed them in the zip I always have import issues or issues that end up depending on host packages.
I then started looking into virtual environments as a way to ensure package dependencies. However, it seems that the typical workflow on the target machine is to source the activation script and run the code within the virtualenv. What I would like to do is have a single file containing a Python script and all its dependencies and for the user to just execute the file. Is this possible given that the Python interpreter is actually packaged with the virtualenv? Is it possible to invoke the Python interpreter from within the zip file? What is the recommended approach for this from a Python point of view?

You can create a bash script that creates the virtual env and runs the python scripts aswell.
!#/bin/bash
virtualenv .venv
.venv/bin/pip install <python packages>
.venv/bin/python script

Set PYTHONPATH through shell script

I am using a third party python module in my code and i need to install that module using easy_install in my local/current directory.
I have to write a shell script which will export PYTHONPATH then do easy_install on that path ,then after executing this script i run my python script which will import that module.
However,I can not persist the PYTHONPATH set from the script after the script is done executing and hence import fails.
My script looks like this:
export PYTHONPATH="$HOME/final/dnsserver/Local/lib/python2.7/site-packages"
easy_install --prefix=$HOME/final/dnsserver/Local pygeoip
How can I make sure that the path will be setand exported even after script is done.
P.S. I do not have root access on this machine