I have a python script that starts a program and automates through it, continuously processing new data and saves to a pre-set directory.
What is the recommended way to run the Python script forever, logging errors when they occur, and restarting when it crashes?
so far I've came across the os.execv and have this to start:
import sys
import os
def pyexcept(t, v, tb):
import traceback
## restarts the script
os.execv( sys.executable, '')
but I often get stuck trying to figure out the next step, could someone explain the next steps i could take, ty!
When the python script crashes, the program is not running anymore, therefore the script cannot execute more lines of code.
You have 2 options:
Make sure your python script doesn't crash, which is very much recommended. You can do this by handling the exceptions thrown by your program.
Option 1
I assume you are new to python, so here is an example of a python script that handles an exception calls the same function again.
from time import sleep
def run_forever():
try:
# Create infinite loop to simulate whatever is running
# in your program
while True:
print("Hello!")
sleep(10)
# Simulate an exception which would crash your program
# if you don't handle it!
raise Exception("Error simulated!")
except Exception:
print("Something crashed your program. Let's restart it")
run_forever() # Careful.. recursive behavior
# Recommended to do this instead
handle_exception()
def handle_exception():
# code here
pass
run_forever()
If you want to restart the python script you would need another python script (assuming you want to do this with python) that checks if the process is still alive and if not then run it again with python.
Option 2
This is the script that starts another python script called 'test.py' via the command python test.py.
Make sure you have the right file path, if you put the scripts in the same folder, you usually don't need the full path and only the script name.
Notably, make sure that command 'python' is recognized by your system, it could in some cases by 'python3'
script_starter.py
from subprocess import run
from time import sleep
# Path and name to the script you are trying to start
file_path = "test.py"
restart_timer = 2
def start_script():
try:
# Make sure 'python' command is available
run("python "+file_path, check=True)
except:
# Script crashed, lets restart it!
handle_crash()
def handle_crash():
sleep(restart_timer) # Restarts the script after 2 seconds
start_script()
start_script()
In case you are interested in the code I used for the test file: 'test.py', I post it here.
test.py
from time import sleep
while True:
sleep(1)
print("Hello")
raise Exception("Hello")
It should be self-evident that if the Python script crashes, it cannot restart itself as it is no longer resident in memory. The answer, then, must come from something outside of Python operating at a lower level of abstraction than an application.
Linux often solves this with systemd. You could create a unit file describing the run of your application and start that service. systemd will then monitor the running process and restart it if it dies.
Windows usually solves this with Windows Services, which are executables with a special win32 API wrapper around them to allow the system to make calls into them to check their status, stop them, or restart them. NSSM (Non-Sucking Service Manager) is a helpful utility to create a service from any arbitrary command.
You have to use some external service to watch your python application. In windows you can run your python application as a windows service using Windows Service Wrapper.
Windows service wrapper (WinSW) is originally developed for run Jenkins client and server as a Windows service in Windows machine. But unlike other service wrappers, WinSW allow you to run any other applications like python.
You can provide all the configuration details as a XML file.
Few configurations you may interested
executable --> you can define which executable you wish to run (python in your case)
aruments --> you can provide arguments for the executable (ex - you can provide your application)
onFailure --> You can use onfailure configuration to specify what to do if your applications fails.
logpath --> you can specify the location for your logs.
and many more. Please read this file to know about all the configurations.
You should place your configuration XML file in the same directory where the WinSW executable located. Also there are few must implemented configurations which need to manage windows service (id, name, description, executable).
You can download the latest version from this link.
Related
So I was browsing repl.it and saw that someone made it possible to run firefox in the repl window. There was a file called Makefile and it had this code in it. I'm wondering what it means and where they are getting Firefox from.
.PHONY: run
run:
install-pkg python firefox
python3 launch.py
Then there is a python file called launch.py
def delete():
time.sleep(10)
os.remove("nohup.out")
print ("Deleted nohup.out.")
thread = threading.Thread(target=delete)
thread.start()
os.system("firefox")
I'm genuinely curious where firefox is coming from and if I can substitute for another app like discord.
Aswell as what is makefile
Here is a link to the repl where you can hten view the code. https://replit.com/#Jackerin0/Firefox-fixed-originally-made-by-polygott?v=1
Makefile is a utility for writing and executing series of command-line instructions for things like compiling code, testing code, formatting code, running code, downloading/uploading data, cleaning your directory etc ... Basically, it helps automate your dev workflows into simple commands (make run, make test, make clean).
Here's what it does in this case:
.PHONY: run # explicitly declare the a routine "run" exists, which can be called with `make run`
run: # run routine
install-pkg python firefox # first, install python and firefox. This will make both firefox and python available as executable binaries that can be launched from the command line
python3 launch.py # this runs the python script
So when you type make run in terminal, it will run the install-pkg and python3 commands.
Then in the python file:
def delete():
time.sleep(10) # sleep for 10 seconds
os.remove("nohup.out") # remove the file named nohup.out
print ("Deleted nohup.out.")
thread = threading.Thread(target=delete) # create a thread to run the delete function
thread.start() # start the thread
os.system("firefox") # run the executable for firefox (you can replace this with any command from the command line)
The nohup file is created when running a background task. Not sure why it's being created in this context (maybe because of something specific to firefox or repl), or why you need to delete it.
So I've researched and tested multiple methods to get this to work, but both our IT guy and myself are unsure how to proceed. Ultimately I need to run three python scripts on a server which look at a set of Excel files and scans them for errors. If there are errors, it will email the appropriate people to fix them. The scripts themselves work fine when run through the CMD prompt, either individually or in a batch file. These scripts work perfectly when run manually from the command prompt.
However, when I try to schedule one of the scripts or the batch file, they do not run in their entirety. I have a print statement early on in the scripts and that runs fine in the Python.exe, but then the script abruptly quits. This makes me think that Python runs fine, but the contents of the script aren't running correctly through the scheduler.
To simplify troubleshooting I created two test scripts, test.py and test2.py, and placed them in a simple directory of the server, C:\Store. All this first script does is print something to the console, and write something to a file and save it to the root directory, after which it will wait for user input. The second script is simpler and just prints the architecture of Python and it's build path. Both scripts work fine when run from the command prompt manually.
**test.py** Does NOT Work as scheduled task
import io
print('JUST WORRRRRKKK!')
with open("//kaicmapp/Store/foobar.txt", "w") as file:
file.write("Hello!")
file.close()
raw_input("press any button")
**test2.py** Works as scheduled task
import sys
import platform
print("Python EXE : " + sys.executable)
print("Architecture : " + platform.architecture()[0])
raw_input("\n\nPress ENTER to quit")
What I am using for these test scripts:
Python 2.7 32 bit
Windows Server 2012
Windows User Profile with Admin Rights
Python 2.7 exe added as environment variable. Renamed to python2.exe in installation directory, as I also have Python 3 installed on the serverand want to be able to run them exclusively
Things I've tried:
Running Task Scheduler as Admin
Run with/without highest privileges
Run only when current user is logged in
Run regardless of current user logged in, password entered correctly
Run the scripts in a batch file
The solution listed here, results in same outcome in both scripts
Things I've noticed:
Using python2 "C:\Store\test2.py" in the scheduler will run the script successfully, with the prompt waiting for user input. Run result from the scheduler says The operation completed successfully. (0x0)
Using the same task as above, but running test.py instead, will not result any output from the Python shell, it simply pops up and disappears. Run result from scheduler says Incorrect function. (0x80070001)
Any ideas on how to proceed? I am not very familiar with server administrative tasks, so I feel pretty lost here.
This is a Microsoft scheduler issue.
https://social.technet.microsoft.com/Forums/windows/en-US/c8ad90e1-8435-4e13-9af9-098e72a1926d/error-code-incorrect-function-0x80070001-when-running-a-scheduled-task?forum=winserver8gen
Please check if this solution works for you.
I have a Python script that should open my Linux terminal, browser, file manager and text editor on system startup. I decided crontab is a suitable way to automatically run the script. Unfortunately, it doesn't went well, nothing happened when I reboot my laptop. So, I captured the output of the script to a file in order to get some clues. It seems my script is only partially executed. I use Debian 8 (Jessie), and here's my Python script:
#!/usr/bin/env python3
import subprocess
import webbrowser
def action():
subprocess.call('gnome-terminal')
subprocess.call('subl')
subprocess.call(('xdg-open', '/home/fin/Documents/Learning'))
webbrowser.open('https://reddit.com/r/python')
if __name__ == '__main__':
action()
here's the entry in my crontab file:
#reboot python3 /home/fin/Labs/my-cheatcodes/src/dsktp_startup_script/dsktp_startup_script.py > capture_report.txt
Here's the content of capture_report.txt file (I trim several lines, since its too long, it only prints my folder structures. seems like it came from 'xdg-open' line on Python script):
Directory list of /home/fin/Documents/Learning/
Type Format Sort
[Tree ] [Standard] [By Name] [Update]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
/
... the rest of my dir stuctures goes here
I have no other clue what's possible going wrong here. I really appreciate your advice guys. thanks.
No, cron is not suitable for this. The cron daemon has no connection to your user's desktop session, which will not be running at system startup, anyway.
My recommendation would be to hook into your desktop environment's login scripts, which are responsible for starting various desktop services for you when you log in, anyway, and easily extended with your own scripts.
I'd do as tripleee suggested, but your job might be failing because it requires an X session, since you're trying to open a browser. You should put export DISPLAY=:0; after the schedule in your cronjob, as in
#reboot export DISPLAY=:0; python3 /home/fin/Labs/my-cheatcodes/src/dsktp_startup_script/dsktp_startup_script.py > capture_report.txt
If this doesn't work, you could try replacing :0 with the output of echo $DISPLAY in a graphical terminal.
I know how to set-up run configurations to pass parameters to a specific python script. There are several entry points, I don't want a run configuration for each one do I? What I want to do instead is launch a python script from a command line shell script and be able to attach the PyCharm debugger to the python script that is executed and have it stop at break points. I've tried to use a pre-launch condition of a utility python script that will sleep for 10 seconds so I can attempt to "attach to process" of the python script. That didn't work. I tried to import pdb and settrace to see if that would stop it for attaching to the process, but that looks to be command line debugging specific only. Any clues would be appreciated.
Thanks!
You can attach the debugger to a python process launched from terminal:
Use Menu Tools --> Attach to process then select python process to debug.
If you want to debug a file installed in site-packages you may need to open the file from its original location.
You can to pause the program manually from debugger and inspect the suspended Thread to find your source file.
I'm trying to make my python script run upon startup but I get the error message windowserror access denied, but I should be able to make programs start upon boot because teamviewer ( a third-party program I downloaded ) runs every time I restart my computer so I know that I should be able to make my program run at startup (I might be doing something different though, so if you could shed some light on what teamviewer is doing differently to get its script to run at startup that would be helpful).
Here is my script
import _winreg, webbrowser
key = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER,'Software\Microsoft\Windows\CurrentVersion\Run')
_winreg.SetValueEx(key,'pytest',0,_winreg.REG_BINARY,'C:\Users\"USERNAME"\Desktop\test.py')
key.Close()
webbrowser.open('www.youtube.com')
Any input is appreciated.
import webbrowser
webbrowser.open('www.youtube.com')
Get rid of all of that _winreg stuff. Instead, you (assuming double-clicking on a py file opens the console) should simply place it in your startup folder (C:\Users\yourusername\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup on Windows 7, and C:\Documents and Settings\yourusername\Start Menu\Programs\Startup in XP). This works because Windows tries to open all files in the startup folder, and if Python opens PYs by default, Windows will open the Python console. Try restarting, that should work.
Baboon:
I am a little late posting, but you seem to have left off the sam at the end of your code here.
When you open a key you need to add the user rights, if you do not _winreg defaults to "READ":
Here is a snippet from the python site
http://docs.python.org/2/library/_winreg.html#access-rights
sam is an integer that specifies an access mask that describes the desired security access for the key. Default is KEY_READ. See Access Rights for other allowed values.
Here is the code corrected:
import _winreg, webbrowser
key = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER,'Software\Microsoft\Windows\CurrentVersion\Run',_winreg.KEY_SET_VALUE)
_winreg.SetValueEx(key,'pytest',0,_winreg.REG_BINARY,'C:\Users\"USERNAME"\Desktop\test.py')
key.Close()
webbrowser.open('www.youtube.com')