I have a python script which is working fine so far. However, my program does not exit properly. I can debug until and I'm returning to the end, but the programm keeps running.
main.main() does a lot of stuff: it downloads (http, ftp, sftp, ...) some csv files from a data provider, converts the data into a standardized file format and loads everyting into the database.
This works fine. However, the program does not exit. How can I find out, where the programm is "waiting"?
There exist more than one provider - the script terminates correctly for all providers except for one (sftp download, I'm using paramiko)
if __name__ == "__main__":
main.log = main.log2both
filestoconvert = []
#filestoconvert = glob.glob(r'C:\Data\Feed\ProviderName\download\*.csv')
main.main(['ProviderName'], ['download', 'convert', 'load'], filestoconvert)
I'm happy for any thoughts and ideas!
If your program does not terminate it most likely means you have a thread still working.
To list all the running threads you can use :
threading.enumerate()
This function lists all Thread that are currently running (see documentation)
If this is not enough you might need a bit of script along with the function (see documentation):
sys._current_frames()
So to print stacktrace of all alive threads you would do something like :
import sys, traceback, threading
thread_names = {t.ident: t.name for t in threading.enumerate()}
for thread_id, frame in sys._current_frames().iteritems():
print("Thread %s:" % thread_names.get(thread_id, thread_id))
traceback.print_stack(frame)
print()
Good luck !
You can involve the python debugger for a script.py with
python -m pdb script.py
You find the pdb commands at http://docs.python.org/library/pdb.html#debugger-commands
You'd better use GDB, which allows to pinpoint hung processes, like jstack in Java
This question is 10 years old, but I post my solution for someone with a similar issue with a non-finishing Python script like mine.
In my case, the debugging process didn't help. All debugging outputs showed only one thread. But the suggestion by #JC Plessis that some work should be going on helped me find the cause.
I was using Selenium with the chrome driver, and I was finishing the selenium process after closing the only tab that was open with
driver.close()
But later, I changed the code to use a headless browser, and the Selenium driver wasn't closed after driver.close(), and the python script was stuck indefinitely. It results that the right way to shutdown the Selenium driver was actually.
driver.quit()
That solved the problem, and the script was finally finishing again.
You can use sys.settrace to pinpoint which function blocks. Then you can use pdb to step through it.
Related
I have a linux server used for scientific calculations.
The server provides bsub < [filename] for submitting [filename] to queues, the calculation would return files like "A.chk" after 2 hours;
And also bjobs for checking status of all jobs, including jobID, run status(RUN, PEND, EXIT), etc.
Now I want to meet the following demand:
Submitting A.sh to the server;
Waitting for the calculation terminates, assume that it generates B.chk;
Do another calculation with B.chk and an existing script C.py
Do all these stuff automatically in one .py script.
I've managed to get the latest jobID and run status in a python script.
And I've tried the following solution:
import os
import time
os.system("bsub < %s" %(A.sh))
job_ID = get_jobID()
while(get_runstatus(jobID) != "RUN" and get_runstatus(jobID) != "PEND"):
time.sleep(30)
if "B.chk" in os.listdir():
os.system("python C.py")
But the while cycle in the server occupies too much public resources, so I'm not willing to do that. Then I searched for solutions like process.join() in SubProcess, but the job squence on the server cannot be treated as subprocess in python. So I'm here asking you for better solutions. Thank you.
While this doesnt answer your question fully, there is a Python API for LSF on github
https://github.com/IBMSpectrumComputing/lsf-python-api
Earlier this year, I made a python 3.7.0 script that modifies my Steam profile picture. It worked great, provided I copied/pasted the sessionid, steamLogin, and steamLoginSecure cookies from my browser.
Lately, I wanted to automate obtaining these cookies, so I added a simple function using ValvePython to log in and return the cookies:
def GetCookies():
client = SteamClient()
client.cli_login()
webCookies = client.get_web_session_cookies()
if webCookies == None:
raise Exception("Unable to get web session. Try again later.")
return {"steamLogin": webCookies.get("steamLogin"),
"steamLoginSecure": webCookies.get("steamLoginSecure"),
"sessionid": str(client.session_id),
"steamid": str(client.steam_id)}
But now there's a problem. My script works great, but now the script restarts when it hits one of the two urlopen commands later in the script. I didn't have this issue before adding this function, and it only fails on urlopen commands that come after the function is called, and the specific line it fails on varies.
In IDLE, it says "RESTART: Shell". There're no errors that I can see or catch. On cmd, it just closes without writing anything before close. I tried running the script in an online debugger, and it works without fail.
What could be the reason for the script just restarting without an error? Is there some process in GetCookies() that is too much to handle or something? How can I go about figuring out why the script is restarting?
Any help would be appreciated.
In my code, I save a screenshot prior to submitting a form via element.submit() and afterwards. I also have logging surrounding the call to webdriver.save_screenshot(). What I am seeing is that sometimes saving a screenshot blocks the process until I kill the chromedriver and/or Chrome processes.
This screenshot locking up Chrome happens on both Mac OS X and Ubuntu 16.04 Server (AWS EC2). I have only been able to get this to repro on headless Chrome on the Mac.
Any thoughts or suggestions on why this would occur? My guess at this point is that it has to do with element.submit() because I have yet to see this happen except (immediately) after that call.
This isn't an ideal solution but in order to keep from blocking indefinitely I've tapped into Python's threading module to take a screenshot like so:
import threading
t = threading.Thread(target=webdriver.get_screenshot_as_file, args=[fn])
t.start()
t.join(10)
assert not t.isAlive(), "Screenshot failed"
If an exception occurs then I quit or terminate the browser process which results in the thread dying off as well.
I have done many research on the forum but I didn't find a clear answer to my question.
I have created a python code which extracts data from a website and put it in a SQL database (with SQLITE).
Now, I would like to make it run 24 hours a day on a server so that it collects data at each moment.
Although I know how to code, I don't have a broad knowledge of IT in general.
Would you know by chance how to solve my problem ?
Edit: I was more thinking about putting my scripts on a cloud server to make it run instead of running it directly from my pc.
Thank you very much.
Lcs
I can think of two options:
1) You use a simple "while true" and "wait" in python, so the script is running constantly.
import time
while True:
# Code
time.sleep(5) # goes to sleep for 5 seconds
You can also do this in shell script (infinite loop+sleep) and then run the program when you want.
2) You can use cronjobs to run the pytho script. You basically have the script the will scrap the website and it will be launched every X seconds/minutes/hours. You can read more about how to use cronjobs in Google
Depends on what system you are using (Unix or windows) you can use either:
1. Windows Task Scheduler for Windows
2. Crontab for Unix
I need to be able to take a screenshot (of a vnc session, if putting this in the title and tags wasn't clear enough) within a python script under OSX. The remote system is already running a vnc server which I am using for other purposes, and will eventually cover the full range of common desktop operating systems, so I would prefer to keep using vnc as opposed to some other solution.
I do not have a vnc window open on my test server, as it runs headless. I have tried using vncdotool, but I'd prefer not to have to shell out, and trying to mimic the control flow causes problems because Twisted does not allow you to restart the reactor, but if you leave it running it blocks the main thread, and there seem to be problems trying to run the reactor in a separate Thread or Process...
Does anyone have any ideas?
Building upon what tangentStorm suggested, using selenium to take a screenshot. Try doing this. Open up src/Selenium2Library/keywords/_screenshot.py and look at lines 24-30.
background leaking when the page layout is somehow broken.
"""
path, link = self._get_screenshot_paths(filename)
self._current_browser().save_screenshot(path)
# Image is shown on its own row and thus prev row is closed on purpose
self._html('</td></tr><tr><td colspan="3"><a href="%s">'
Delete the line self._current_browser().save_screenshot(path) and add directly in its place
if hasattr(self._current_browser(), 'get_screenshot_as_file'):
self._current_browser().get_screenshot_as_file(path)
else:
self._current_browser().save_screenshot(path)
So in all it should look like:
background leaking when the page layout is somehow broken.
"""
path, link = self._get_screenshot_paths(filename)
if hasattr(self._current_browser(), 'get_screenshot_as_file'):
self._current_browser().get_screenshot_as_file(path)
else:
self._current_browser().save_screenshot(path)
# Image is shown on its own row and thus prev row is closed on purpose
self._html('</td></tr><tr><td colspan="3"><a href="%s">'
Then try using selenium to take the screenshot.
Reference: Fix
After reading your comments, it seems what you actually want to do is take screenshots of remote web browsers running your flash game.
... And you're using selenium to test those remote web browsers.
... Why don't you just have selenium take the screenshots for you?
http://selenium.googlecode.com/svn/trunk/docs/api/java/org/openqa/selenium/TakesScreenshot.html
I don't know of any library that does this in python for OSX.
However, there are at least three other ways to get the screenshot:
Use the java.awt.Robot class from jython. (Except twisted probably won't run on jython.)
Port Apple's ScreenSnapshot example to Cython and compile it into a python module. (Of course you can do the same thing in C, but Cython makes it much more fun.)
If you can move your server to win32, or just run win32 on your mac via parallels, then you can use the python imaging library's ImageGrab module.
However, I think shelling out to the OS is still the easiest answer. Instead of trying to get it all to run in a single process, just have two processes running: your main twisted process, and some other server that uses threads or whatever.
Then just pass messages back and forth when you want to take a screenshot. You can do this with a simple socket connection (just write another handler to in your twisted server, and have the screenshot server connect as a client)...
If it were me, I'd probably use an AMQP server like RabbitMQ to handle the message-passing, but that may be overkill for what you're doing.
Depending on your code, you might be able to use deferToThread to run the call to screencapture and return the filepath or a pil.Image instance (or whatever you need).
Using the example at http://twistedmatrix.com/documents/current/core/howto/gendefer.html#auto5 it might look something like...
from subprocess import call
import tempfile
from twisted.internet import reactor, threads
import Image ## pip install pil
## Blocking code that takes the screenshot and saves to file
def take_screenshot():
tmp_file_path = tempfile.mktemp(suffix='.png')
# os.system('screencapture %s' % tmp_file_path)
retcode = call(['screencapture', tmp_file_path])
if retcode < 0:
img = Image.open(tmp_file_path)
return img
else:
return None
## Callback fired by the deferToThread
def do_something_with_screenshot(img):
print img.filename, img.format, img.size, img.mode
reactor.stop() ## just here for this example
def run():
# get our Deferred which will be called with the largeFibonnaciNumber result
d = threads.deferToThread(take_screenshot)
# add our callback to print it out
d.addCallback(do_something_with_screenshot)
if __name__ == '__main__':
run()
reactor.run()
Perhaps you can convince the robotframework or Selenium to send a CaptureScreen Sensetalk command to Eggplant Drive.
The Taking a Screenshot post in the TestPlant forums mentions this command.