Develop Reference

How to run bash commands from Python preferably with the os library?

I am trying to run the following python script named test.py. It contains multiple bash commands which I would like to execute in a Linux terminal (unix). This is the content of the file:
import os
os.system('echo install virtualenv')
os.system('sudo pip install virtualenv')
os.system('echo create virtual environment')
os.system('virtualenv my_virtualenvironment')
os.system('echo activate virtual environment')
os.system('source my_virtualenvironment/bin/activate')
I am running the Python script using the following in the terminal:
python3 test.py
The problem that I have is that the commands do not run the same way as they would on a Linux terminal. The output is the following error when trying to execute the last line of the Python script:
sh: 1: source: not found
The last command source my_virtualenvironment/bin/activate normally runs fine if I execute it directly in the terminal (without my Python script). Now, what does sh: 1: mean and why does it not work with my code? I would expect to get something starting with bash: .
Also I have found this solution, but I would like not to use lists for executing commands and maybe even to stick with the os library (if there is a simpler solution without os, I am also open for that):
https://stackoverflow.com/a/62355400/11535508

source is a bash built-in command, not an executable.
Use the full path to the python interpreter in your commands instead of venv activation, e.g. os.system('<venv>/bin/python ...').
The second option is to write your commands into a separate bash script and call it from python:
os.system('bash script.sh')

Setting up crontab for my Django Application

I had an issue with setting up crontab for my Django application for a week and I have almost figured out the same. (Issue linked with Unable to make a function call using Django-cron)
My crontab -e syntax is
* * * * * /Users/ashwin/dashboard/proj_application/exec.sh >> /Users/ashwin/dashboard/proj_application/data.log 2>&1
And in my exec.sh, I have
#!/bin/bash
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
python -c 'import proj_application.cron as cron; cron.test()'
And in cron.py,
from django.core.mail import send_mail
from smtplib import SMTP
from email.mime.text import MIMEText
import datetime
def test():
message = "<p>This is test mail scheduled to send every minute</p>"
my_email = MIMEText(message, "html")
my_email["From"] = "xxx#domain.com"
my_email["To"] = "yyy#domain.com"
my_email["Subject"] = "Title"
sender = "person1#domain.com"
receivers = ["person2#domain.com"]
with SMTP("localhost") as smtp:
smtp.login(sender, "yyy#1234")
smtp.sendmail(sender, receivers, my_email.as_string())
Actual Problem:
The crontab is now able to call the exec.sh file and I am able to print $CWD in echo, when the call comes to cron.py, It is unable to recognize django.core.mail and throws the following error.
from django.core.mail import send_mail
ImportError: No module named django.core.mail
I think, I need to set up virtual environment or python variable path somewhere, but since I am new to crontab, I am not sure how to do that.
Any assistance is appreciated. Thanks in advance.

The way I interpret this is that you are normally running your script in a virutal environment, and it is failing now that you added it to a cron job. If this is incorrect, just skip to the last paragraph.
To run a cron job through a virutal environment, you need to use the virtual environment's python as the python you want. E.g to run the cron.py file:
* * * * * /path/to/venv/bin/python3 /path/to/cron.py >> /Users/ashwin/dashboard/proj_application/data.log 2>&1
This is the way I would recommend doing it, as it doesn't seem like that shell script is entirely necessary (anything in there can easily be done at the top of the python script), but if it is you can do something similar where you call the virtual environment's python instead of the default python. like this:
/path/to/venv/bin/python3 -c 'import proj_application.cron as cron; cron.test()'
However, if this doesn't work, this may not be an issue with the cron job, and instead with the django setup of the mail client (e.g. making sure it is included in the installed apps, etc.)
Since it is a little unclear where your issue is (mail client setup, cron setup, venv configuration, etc), if you are sure it is an issue with running your scripts you usually run in a virtual environment, than these steps should help, otherwise I would make sure your mail client is configured correctly. Make sure your scripts are correct by running them in the console (not through a cron job), and then you can come back and update the post with more detailed info about where the problem you have lies.

yes you are correct you may need to use virtual environment(although its optional but best practice).
To create virtual environment(expects python to be installed prior)
python -m venv venv
To activate (path may differ based on OS(mine is windows))
source venv/Scripts/activate
To install dependency in virtual environment
pip install Django
Now you all set to use virtual env's Django.
which python should map to venv path.
Now you have two ways to run python script
#1 use direct python path
absolute/path/of/venv/bin/python3 -c 'import proj_application.cron as cron; cron.test()'
#2 activate virtual environment and use the bash script as same.
#!/bin/bash
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
source venv/Scripts/python # this will be in windows
python -c 'import proj_application.cron as cron; cron.test()'

Why are .pth files loaded twice in a Python virtual environment?

In the user site .pth files are processed once at interpreter startup:
$ echo 'import sys; sys.stdout.write("hello world\n")' > ~/.local/lib/python3.8/site-packages/hello.pth
$ python3.8 -c ""
hello world
And it's the same behaviour in the system site, e.g. /usr/local/lib/python3.8/site-packages/.
But in venv they are processed twice:
$ rm ~/.local/lib/python3.8/site-packages/hello.pth
$ /usr/local/bin/python3.8 -m venv .venv
$ source .venv/bin/activate
(.venv) $ echo 'import sys; sys.stdout.write("hello world\n")' > .venv/lib/python3.8/site-packages/hello.pth
(.venv) $ python -c ""
hello world
hello world
Why are path configuration files processed twice in a virtual environment?

Looks like it all happens in the site module (not so surprising). In particular in the site.main() function.
The loading of the .pth files happens either in site.addsitepackages() or in site.addusersitepackages(), depending in which folder the file is placed. Well more precisely both these function call site.addpackage(), where it actually happens.
In your first example, outside a virtual environment, the file is placed in the directory for user site packages. So the console output happens when site.main() calls site.addusersitepackages().
In the second example, within a virtual environment, the file is placed in the virtual environment's own site packages directory. So the console output happens when site.main() calls site.addsitepackages() directly and also via site.venv() a couple of lines earlier that also calls site.addsitepackages() if it detects that the interpreter is running inside a virtual environment, i.e. if it finds a pyvenv.cfg file.
So in short: inside a virtual environment site.addsitepackages() runs twice.
As to what the intention is for this behavior, there is a note:
# Doing this here ensures venv takes precedence over user-site
addsitepackages(known_paths, [sys.prefix])
Which, from what I can tell, matters in case the virtual environment has been configured to allow system site packages.
Maybe it could have been solved differently so that the path configuration files are not loaded twice.

Using subprocess in anaconda environment

I use Python 3.6.6 :: Anaconda, Inc.
I try to use subprocess to call other python script.
subprocess.run("python -V", shell=True)
I tried this code but outcome is
Python 2.7.12
My local python get caught
I also tried
subprocess.run("bash -c 'source /home/hclee/anaconda3/envs/calamari/lib/python3.6/venv/scripts/common/activate
/home/hclee/anaconda3/envs/calamari && python -V && source deactivate'", shell=True)
but I got the same result

Run source activate root in linux, or activate root in Windows to activate the environment before running your code.
If this does not help you, try for a quick and dirty fix e.g.:
subprocess.run('bash -c "source activate root; python -V"', shell=True)
The reason you need to call bash is that python's run will not call the bash environment, but another which is a bit more constrained and does not contain source, so here we need to call bash... But as mentioned, if you need this command, either you are doing something special, or something is wrong with your environment...
deactivate is not needed, it does nothing cause the shell it was run on will be destroyed...
Note: For newest conda versions, the provided code will work, but there are also these options that work similarly:
conda deactivate:
subprocess.run('bash -c "conda deactivate; python -V"', shell=True)
conda activate root or base:
subprocess.run('bash -c "conda activate root; python -V"', shell=True)
subprocess.run('bash -c "conda activate base; python -V"', shell=True)

I don't think sourcing a conda env in every subprocess call of your code is a good idea.
Instead you can find the bin dir of your current sourced env and grab the full path to binaries from there. Then pass these to subprocess when you want to call them.
import os
import sys
import subprocess
# what conda env am I in (e.g., where is my Python process from)?
ENVBIN = sys.exec_prefix
# what binaries am I looking for that are installed in this env?
BIN1 = os.path.join(ENVBIN, "bin", "ipython")
BIN2 = os.path.join(ENVBIN, "bin", "python")
BIN3 = os.path.join(ENVBIN, "bin", "aws")
# let's make sure they exist, no typos.
for bin in (BIN1, BIN2, BIN3):
assert os.path.exists(bin), "missing binary {} in env {}".format(bin, ENVBIN)
# then use their full paths when making subprocess calls
for bin in (BIN1, BIN2, BIN3):
cmd = ["which", bin]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
print(proc.communicate()[0].decode())
The printed results show that subprocess is using the binaries from the conda environment instead of from the base (default) env.
.../miniconda3/envs/aws/bin/ipython
.../miniconda3/envs/aws/bin/python
.../miniconda3/envs/aws/bin/aws

If your console/notebook is already using the correct environment, you can call subprocess with sys.executable to use current environment:
import sys
import subprocess
subprocess.run(f"{sys.executable} -V", shell=True)

cannot compile cpp files under the python pyc script file

I am trying to compile & execute my hw cpp file under the python script file which we are given by lecturer. the how-to-manual.pdf he sent us it says use:
c:\>python ./submit.pyc problemID -u username -p password -b //submit.pyc is already given to us
and here is the manifest.txt we are given:
[main]
problem = gc
build =
g++ main.cpp -o solver
run =
./solver %f
my cpp file works normally like this:
./solver input_file
However, I am trying (I have to) to do this under the windows OS. I have Python 2.7.x installed and python.exe is in the Command PATH. I can't run it under the linux ssh sytem because there is 2.4.x python installed and I can't touch it (school's system).
Anyway, when I execute the line above, it returns me:
Command execution failed:
g++ solver.cpp -o solver
I think I told everything I can. So, any idea that what I have to do else? except asking to lecturer:)

For the above to work it needs to be able to find g++ so you need to add the directory that it resides in to the PATH environment variable. This can be done from within your python script or on the command line with:
path=Where\g++\lives;%path%
This will only apply within the current DOS session.
Or you can add it permanenty through system settings->advanced settings->environmental variables
You could also look at using a python virtual environments on the schools linux system.