Develop Reference

How to run a Python script from Apache on Raspberry Pi?

So, on a Raspberry Pi I'm using a camera app with a web interface, I wanted to add LED lighting by adding a neopixel. I have successfully done this and can now turn it on and off running two python scripts.
Explanation and question:
I have a python script in /usr/local/bin that is executable.
It is owned by 'root root'.
I have a shell script in /var/www/html/macros that is executable and has to run the python script in /usr/local/bin.
The shell script is owned by 'www-data'
When I manually run the python file, it executes the script.
When I manually run the shell script, it executes the python script.
When I run the shell script by clicking on a button on my webpage, it seems to execute the shell script correctly, however, it looks like it doesn't execute the python script.
What can I do to fix this?
I'm not that experienced with permissions, but I wanted to emphasize on the fact that this is a closed system that does not contain any sensitive information. So safety/best practice is not a concern. I just want to make this work.

I'm not an expert in this area, but I believe to access /usr/local/bin/ you need root privileges which explains why you're having success but not Apache.
Rather than give Apache root permissions, it's best to simply remove the requirement from the individual file you want to execute. This can be accomplished by
$ cd /usr/local/bin
$ sudo chmod 777 your_script.py

Now, after 11 hours and a group of people thinking along we found a solution to the problem.
The problem turned out to be that the Web interface can only execute as 'www-data', and the NeoPixel library that the python script depends on needs to be executed as sudo/root.
These two factors make it so that there will never be a direct way of getting the scripts to work together.
However, the idea emerged to use some sort of pipe.
A brilliant user suggested to me to use sshpass. This would allow to pass data to ssh and have it essentially be executed as a root user.
The data from the web interface would be relayed to the sshpass and this would successfully run the needed scripts with the needed privileges.
Special thanks to Minty Trebor and Falcounet from the RRF for LPC/STM Discord!

Python script can't find file when using Windows Scheduler

I have a batch file to execute Python and run a Python script that works fine when run manually. But when run using Windows Task Scheduler, the Python Script gives me a traceback because it can't find a .TXT file that's in the same file as the Python script.
There are many flavors of the "batch file works manually but not with Task Scheduler" problem. I think my problem may be related to the fact that my Python script is in a virtual environment. It may also be related to Windows user accounts and permissions. Past posts about Task Scheduler have been resolved with changing settings related to user accounts but so far that hasn't worked for me.
More details on Windows Task Scheduler settings:
Here are the settings I've chosen for the task:
My task gets closest to working when I use the "run only when user is logged on option. If I choose "run whether user is logged on or not" the task opens a Command Prompt but doesn't appear to execute anything.
More details about batch file
My batch file looks like this:
"C:\Users\MyPathtovirtualenvironment\virtualenv\Scripts\python.exe" "C:\Users\fried\Desktop\calendaralert\court_calendar_alert1.3_automatic.py"
pause
When run manually, it does what it's supposed to: it executes Python from within the virtual environment, then loads the Python script, which runs correctly except when it's run from Windows Task Scheduler.
More details on Python script
The Python script works fine when run by itself in the command prompt and as a batch file, so I don't think there's anything directly wrong with it. Here's the part of the script that causes the traceback.
with open('replacements.txt') as f:
replacements = dict(x.rstrip().split("!") for x in f)
The replacements.txt file is in the same folder as my Python program. I also tried putting copies of the file within the Virtual Environment folder and the /scripts subfolder without success.

Just came across this problem, while running a python script on a windows ec2 instance. The problem got solved by adding this at the top:
import os
path = "your-directory-path"
os.chdir(path)

Run my Python script at login on linux

I have a Python script, and I want it to be autostarted at every login. It's in a linux system. I followed a guide that explains that is enough to create a .desktop file in ~/.config/autostart/*.desktop and write:
[Desktop Entry]
Name=MyApp
Type=Application
Exec=python3 ~/.myapp/myapp
Terminal=false
I tried several times to reboot but the program doesn't execute, even if it seems to be active in the list of application of startup in my lxde environment.

Often things like the ~ (tilde) aren't evaluated when placed in a config file. Try using the complete path instead (/home/user/… instead of ~/…) and see if this works. If this works, you can try to use $HOME instead ($HOME/…) to make this more portable and abstract.

If you want to run your script on terminal login, place it to /etc/profile.d/
For KDE (at least, KDE 5) you could add applications to autorun in System Settings > Startup and Shutdown > Autostart (either *.desktop files or scripts), it adds links to ~/.config/autostart.

You can achieve this by adding this line python /home/user/program.py in your .bashrc file. It will be invoked each time when you login to your system.

Python Script not running in cron

I am trying to run a Python script from cron. I am using crontab to run the command as a user instead of root. My Python script has the shebang at the top #! /usr/bin/env python and I did chmod +x it to make the script executable.
The script does work when running it from a shell but not when using crontab. I did check the /var/log/cron file and saw that the script runs but that absolutely nothing from stdout or stderr prints anywhere.
Finally, I made a small script that prints the date and a string and called that every minute that worked but this other script does not. I am not sure why I am getting these variable results...
Here is my entry in crontab
SHELL=/bin/bash
#min #hour day-of-month month day-of-week command
#-------------------------------------------------------------------------
*/5 * * * * /path/to/script/script.py
Here is the source code of my script that will not run from crontab but will run from the shell it is in when called like so ./script.py. The script is executable after I use the chmod +x command on it, by the way...:
#! /usr/bin/env python
#create a file and write to it
import time
def create_and_write():
with open("py-write-out.out", "a+") as w:
w.write("writing to file. hello from python. :D\n")
w.write("the time is: " + str(time.asctime()) + "\n")
w.write("--------------------------------\n")
def main():
create_and_write()
if __name__ == "__main__":
main()
EDIT
I figured out what was going wrong. I needed to put the absolute file paths in the script otherwise it would write to some directory I had not planned for.

Ok, I guess this thread is still going to help googlers.
I use a workaround to run python scripts with cron jobs. In fact, python scripts need to be handled with delicate care with cron job.
So I let a bash script take care of all of it.I create a bash script which has the command to run the python script. Then I schedule a cron job to run bash script. You can even use a redirector to log the output of bash command executing the python script.For example
#reboot /home/user/auto_delete.sh
auto_delete.sh may contain following lines:-
#!/bin/sh
echo $(date) >> bash_cron_log.txt
/usr/bin/python /home/user/auto_delete.py >> bash_cron_log.txt
So I don' need to worry about Cron Jobs crashing on python scripts.

You can resolve this yourself by following these steps:
Modify the cron as /path/to/script/script.py > /tmp/log 2> &1
Now, let the cron run
Now read the file /tmp/log
You will find out the reason of the issue you are facing, so that you can fix it.
In my experience, the issue is mostly with the environment.
In cron, the env variables are not set. So you may have to explicitly set the env for your script in cron.

I would like to emphasise one more thing. I was trying to run a python script in cron using the same trick (using shell script) and this time it didn't run. It was #reboot cron. So I used redirection in crontab as I mentioned in one of the above comments. And understood the problem.
I was using lot many file handlers in python script and cron runs the script from user's home directory. In that case python could not find the files used for file handlers and will crash.
What I used for troubleshooting was that I created a crontab as below. It would run the start.sh and throw all the stderror or stdoutput to file status.txt and I got error in status.txt saying that the file I used for file handler was not found. That was right because python script was executed by cron from user's home directory and then the script starts searching for files in home directory only.
#reboot /var/www/html/start.sh > /cronStatus/status.txt 2>&1
This will write everything happening during cron execution to status.txt file. You can see the error over there. I will again advice running python scripts using bash scripts for cronjobs. SOLUTION:- Either you use full path for all the files being used in script (which wasn't feasible for me, since I don't want script to be location dependent). Or you execute script from correct directory
So I created my cron as below-
#reboot cd /var/www/html/ && /var/www/html/start.sh
This cron will first change directory to correct location and then start the script. Now I don't have to worry about hardcoding full path for all files in script. Yeah, it may sound being lazy though ;)
And my start.sh looks like-
#!/bin/sh
/usr/bin/python /var/www/html/script.py
Hope it helps
Regards,
Kriss

I had a similar issue but with a little bit different scenario: I had a bash script (run.sh) and a Python script (hello.py). When I typed
sh run.sh
from the script directory, it worked; if I added the sh run.sh command in my crontab, it did not work.
To troubleshoot the issue I added the following line in my run.sh script:
printenv > env.txt
In this way you are able to see the environment variable used when you (or the crontab) run the script. By analyzing the differences between the env.txt generated from the manual run and the crontab run I noticed that the PWD variable was different. In my case I was able to resolve by adding the following line at the top of my .sh script:
cd /PYTHON_SCRIPT_ABSOLUTE_PATH/
Hope this could help you!

Another reason may be that the way we judged executed or not executed is wrong.

Problems running python script by windows task scheduler that does pscp

Not sure if anyone has run into this, but I'll take suggestions for troubleshooting and/or alternative methods.
I have a Windows 2008 server on which I am running several scheduled tasks. One of those tasks is a python script that uses pscp to log into a linux box, checks for new files and if there is anything new, copies them down to a local directory on the C: drive. I've put some logging into the script at key points as well and I'm using logging.basicConfig(level=DEBUG).
I built the command using a variable, command = 'pscp -pw xxxx name#ip:/ c:\local_dir' and then I use subprocess.call(command) to execute the command.
Now here's the weird part. If I run the script manually from the command line, it works fine. New files are downloaded and processed. However, if the Task Scheduler runs the script, no new files are downloaded. The script is running under the same user, but yet yields different results.
According to the log files created by the script and on the linux box, the script successfully logs into the linux box. However, no files are downloaded despite there being new files. Again, when I run it via the command line, files are downloaded.
Any ideas? suggestions, alternative methods?
Thanks.

You can use the windows Task Scheduler, but make sure the "optional" field "Start In" is filled in.
In the Task Scheduler app, add an action that specifies your python file to run "doSomeWork" and fill in the Start in (optional) input with the directory that contains the file.. So for example if you have a python file in:
C:\pythonProject\doSomeWork.py
You would enter:
Program/Script: doSomeWork.py
Start in (optional): C:\pythonProject

I had the same issue when trying to open an MS Access database on a Linux VM. Running the script at the Windows 7 command prompt worked but running it in Task Scheduler didn't. With Task Scheduler it would find the database and verify it existed but wouldn't return the tables within it.
The solution was to have Task Scheduler run cmd as the Program/Script with the arguments /c python C:\path\to\script.py (under Add arguments (optional)).
I can't tell you why this works but it solved my problem.

I'm having a similar issue. In testing I found that any type of call with subprocess stops the python script when run in task scheduler but works fine when run on the command line.
import subprocess
print('Start')
test = subprocess.check_output(["dir"], shell=True)
print('First call finished')
When run on command line this outputs:
Start
First call finished
When run from task scheduler the output is:
Start
In order to get the output from task scheduler I run the python script from a batch file as follows:
python test.py >> log.txt
I run the script through the batch file both on command line and through task scheduler.

Brad's answer is right. Subprocess needs the shell context to work and the task manager can launch python without that. Another way to do it is to make a batch file that is launched by the task scheduler that calls python c:\path\to\script.py etc. The only difference to this is that if you run into a script that has a call to os.getcwd() you will always get the root where the script is but you get something else when you make the call to cmd from task scheduler.

Last edit - start
After experiments... If you put there full path to python program it works without highest privileges (as admin). Meaning task settings like this:
program: "C:\Program Files\Python37\python.exe"
arguments: "D:\folder\folder\python script.py"
I have no idea why, but it works even if script uses subprocess and multiple threads.
Last edit - end
What I did is I changed task settings: checked Run with highest privileges. And task started to work perfectly while running python [script path].
But keep in mind, that title contains "Administrator: " at the begining... always...
P.S. Thanks guys for pointing out that subprocess is a problem. It made me think of task settings.
I had similar problem when one script is running from Windows Task Scheduler, and another one doesn't.
Running cmd with python [script path] didn't work for me on Windows 8.1 Embedded x64. Not sure why. Probably because of necessity to have spaces in path and issue with quotes.
Hope my answer helps someone. ;)

Create a batch file add your python script in your batch file and then schedule that batch file .it will work .
Example : suppose your python script is in folder c:\abhishek\script\merun.py
first you have to go to directory by cd command .so your batch file would be like :
cd c:\abhishek\script
python merun.py
it work for me .

Just leaving this for posterity: A similar issue I faced was resolved by using the UNC (\10.x.xx.xx\Folder\xxx)path everywhere in my .bat and .py scripts instead of the letter assigned to the drive (\K:\Folder\xxx).

I had this issue before. I was able to run the task manually in Windows Task Scheduler, but not automatically. I remembered that there was a change in the time made by another user, maybe this change made the task scheduler to error out. I am not sure. Therefore, I created another task with a different name, for the same script, and the script worked automatically. Try to create a test task running the same script. Hopefully that works!

For Anaconda installation of python in windows below solution worked for me
create a batch file with below
"C:\Users\username\Anaconda3\condabin\activate" && python "script.py" &&
deactivate
Setup task to run this batch file