Develop Reference

Python, Use threading and schedule to keep running a function constantly

I am making a bot that auto-posts to Instagram using instabot, now the thing is that if I exceed a number of request the bot terminate the script after retrying for some minutes.
The solution I came up with is to schedule the script to run every hour or so, and to ensure that the script will keep running constantly I used threading to restart the posting function when the thread is dead.
The function responsible for posting, in this code if the bot instance from instabot retried sending requests for some minutes and failed, it terminates the whole script.
def main():
create_db()
try:
os.mkdir("images")
print("[INFO] Images Directory Created")
except:
print("[INFO] Images Directory Found")
# GET A SUBMISSION FROM HOT
submissions = list(reddit.subreddit('memes').hot(limit=100))
for sub in submissions:
print("*"*68)
url = sub.url
print(f'[INFO] URL : {url}')
if "jpg" in url or "png" in url:
if not sub.stickied:
print("[INFO] Valid Post")
if check_if_exist(sub.id) is None:
id_ = sub.id
name = sub.title
link = sub.url
status = "FALSE"
print(f"""
[INFO] ID = {id_}
[INFO] NAME = {name}
[INFO] LINK = {link}
[INFO] STATUS = {status}
""")
# SAVE THE SUBMISSION TO THE DATABASE
insert_db(id_, name, link, status)
post_instagram(id_)
print(f"[INFO] Picture Uploaded, Next Upload is Scheduled in 60 min")
break
time.sleep(5 * 60)
The scheduling function:
def func_start():
schedule.every(1).hour.do(main)
while True:
schedule.run_pending()
time.sleep(10 * 60)
And last piece of code:
if __name__ == '__main__':
t = threading.Thread(target=func_start)
while True:
if not t.is_alive():
t.start()
else:
pass
So basically I want to keep running the main function every hour or so, but I am not having any successes.

Looks to me like schedule and threading are overkill for your use case as your script only performs one single task, so you do not need concurrency and can run the whole thing in the main thread. You primarily just need to catch exceptions from the main function. I would go with something like this:
if __name__ == '__main__':
while True:
try:
main()
except Exception as e:
# will handle exceptions from `main` so they do not
# terminate the script
# note that it would be better to only catch the exact
# exception you know you want to ignore (rather than
# the very broad `Exception`), and let other ones
# terminate the script
print("Exception:", e)
finally:
# will sleep 10 minutes regardless whether the last
# `main` run succeeded or not, then continue running
# the infinite loop
time.sleep(10 * 60)
...unless you actually want each main run to start precisely in 60-minute intervals, in which case you'll probably need either threading or schedule. Because, if running main takes, say, 3 to 5 minutes, then simply sleeping 60 minutes after each execution means you'll be launching the function every 63 to 65 minutes.

How to timeout Paramiko sftp.put() with signal module or other in Python?

I would like timeout the function sftp.put(), I have tried with signal Module but the script doesn't die if the upload time is over 10s.
I use that to transfer files by ssh (paramiko).
[...]
def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't upload the fileeeeeeeeeeee!!!!")
[...]
raspi = paramiko.SSHClient()
raspi.set_missing_host_key_policy(paramiko.AutoAddPolicy())
raspi.connect(ip , username= "", password= "" , timeout=10)
sftp = raspi.open_sftp()
[...]
signal.signal(signal.SIGALRM, handler)
signal.alarm(10)
sftp.put(source, destination , callback=None, confirm=True)
signal.alarm(0)
raspi.close()
[...]
Update 1:
I want to abort the transfer if the server stops responding for a while. Actually, my python script check (in loop) any files in a folder, and send it to this remote server. But in the problem here I want to leave this function in the case of the server become inaccessible suddenly during a transfer (ip server changing, no internet anymore,...). But when I simulate a disconnection, the script stays stuck at this function sftp.put anyway...)
Update 2:
When the server goes offline during a transfer, put() seems to be blocked forever. This happens with this line too:
sftp.get_channel().settimeout(xx)
How to do when we lose the Channel?
Update 3 & script goal
Ubuntu 18.04
and paramiko version 2.6.0
Hello,
To follow your remarks and questions, I have to give more details about my very Ugly script, sorry about that :)
Actually, I don’t want to have to kill a thread manually and open a new one. For my application I want that the script run totally in autonomous, and if something wrong during the process, it can still go on. For that I use the Python exception handling. Everything does what I want except when the remote server going off during a transfer: The script stays blocked in the put() function, I think inside a loop.
Below, the script contains in total 3 functions to timeout this thanks to your help, but apparently nothing can leave this damned sftp.put()! Do you have some new idea ?
Import […]
[...]
def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't upload the fileeeeeeeeeeee!!!!")
def check_time(size, file_size):
global start_time
if (time.time() - start_time) > 10:
raise Exception
i = 0
while i == 0:
try:
time.sleep(1) # CPU break
print ("go!")
#collect ip server
fichierIplist = open("/home/robert/Documents/iplist.txt", "r")
file_lines = fichierIplist.readlines()
fichierIplist.close()
last_line = file_lines [len (file_lines)-1]
lastKnowip = last_line
data = glob.glob("/home/robert/Documents/data/*")
items = len(data)
if items != 0:
time.sleep(60) #anyway
print("some Files!:)")
raspi = paramiko.SSHClient()
raspi.set_missing_host_key_policy(paramiko.AutoAddPolicy())
raspi.connect(lastKnowip, username= "", password= "" , timeout=10)
for source in data: #Upload file by file
filename = os.path.basename(source) #
destination = '/home/pi/Documents/pest/'+ filename #p
sftp = raspi.open_sftp()
signal.signal(signal.SIGALRM, handler)
signal.alarm(10)
sftp.get_channel().settimeout(10)
start_time = time.time()
sftp.put(source, destination, callback=check_time)
sftp.close()
signal.alarm(0)
raspi.close()
else:
print("noFile!")
except:
pass

If you want to timeout, when the server stops responding:
set the timeout argument of SSHClient.connect (your doing that already),
and set sftp.get_channel().settimeout as already suggested by #EOhm
If you want to timeout even when the server is responding, but slowly, implement the callback argument to abort the transfer after certain time:
start_time = time.time()
def check_time(size, file_size):
global start_time
if (time.time() - start_time) > ...:
raise Exception
sftp.put(source, destination, callback=check_time)
This won't cancel the transfer immediately. To optimize transfer performance, Paramiko queues the write requests to the server. Once you attempt to cancel the transfer, Paramiko has to wait for the responses to those requests in SFTPFile.close() to clear the queue. You might solve that by using SFTPClient.putfo() and avoiding calling the SFTPFile.close() when the transfer is cancelled. But you won't be able to use the connection afterwards. Of course, you can also not use the optimization, then you can cancel the transfer without delays. But that kind of defies the point of all this, doesn't it?
Alternatively, you can run the transfer in a separate thread and kill the thread if it takes too long. Ugly but sure solution.

Use sftp.get_channel().settimeout(s) for that instead.

After trying a lot of things and with your help and advice, I have found a reliable solution for what I wanted. I execute sftp.put in a separate Thread and my script do what I want.
Many thanks for your help
Now if the server shuts down during a transfer, after 60 sec, my script goes on using:
[...]
import threading
[...]
th = threading.Thread(target=sftp.put, args=(source,destination))
th.start()
h.join(60)
[...]

How can i start a web server and update my function in one script

I start a server and use some data from my function. But I want this function to update data and display new one on my server. However when I start a web server it only takes the first compiled data from function.
I use "schedule" - imported library, that can schedule my task to compile my function in time i choose. Also bottle web framework to start server and make routing.
def read_file():
f=open("345.txt", "r")
hi.contents = f.read()
print(hi.contents)
def server_start():
#route('/as', method = 'GET')
def display_status():
try:
return hi.contents
except Exception:
logging.exception("")
return "Service unavailable. Check logs"
run(host='0.0.0.0', port=8033)
print("sadq")
schedule.every(3).seconds.do(read_file)
server_start()
while True:
schedule.run_pending()
time.sleep(1)
I expect to get updated results on my web server. Would be very glad if you help me or give some good advices. Thak you all.

First I would run bottle with an async process, specifically gevent.
import gevent
from gevent import monkey, signal
monkey.patch_all()
from bottle import app
import scheduler
app = Bottle()
#route('/as', method = 'GET')
def display_status():
try:
return scheduler.contents
except Exception:
logging.exception("")
return "Service unavailable. Check logs"
print("sadq")
server = WSGIServer(("0.0.0.0", int(8083)), app)
def shutdown():
print('Shutting down ...')
server.stop(timeout=60)
exit(signal.SIGTERM)
gevent.signal(signal.SIGTERM, shutdown)
gevent.signal(signal.SIGINT, shutdown) #CTRL C
server.serve_forever()
Then I would launch your scheduler as such in a separate file scheduler.py:
from gevent import spawn, sleep
import schedule
contents = ''
def read_file():
global contents
f=open("345.txt", "r")
contents = f.read()
print(contents)
def start_thread():
while 1:
schedule.run_pending()
sleep(1)
schedule.every(3).seconds.do(read_file)
spawn(start_thread)

How to solve websocket ping timeout?

While running the following piece of code (in theory it should send a value every minute)
from __future__ import print_function
from twisted.internet.ssl import CertificateOptions
options = CertificateOptions()
from os import environ
from twisted.internet.defer import inlineCallbacks
from twisted.internet import reactor
from autobahn.twisted.wamp import ApplicationSession, ApplicationRunner
from autobahn import wamp
from datetime import datetime, timedelta
import xlwings as wb
import time
import xlwings as wb
class Component(ApplicationSession):
"""
An application component that publishes an event every second.
"""
#inlineCallbacks
def onJoin(self, details):
print("session attached")
while True:
try:
wb.Book(r'C:\Users\Administrator\Desktop\Datasets\test_feed.xlsx')
e = wb.Range('A2').value
b = wb.Range('C2').value
c = wb.Range('E2').value
except Exception:
print("----Waiting for RTD server response----")
time.sleep(1)
try:
epoch = datetime(now.year, now.month, now.day)
result = epoch + timedelta(days=c)
result = result.replace(microsecond=0, second=0)
if result > now:
now = result
print("Stock", e, "Time", now, "Price", b)
self.publish(u'com.myapp.ma', b)
except Exception:
print("-----Waiting1 for RTD server response----")
time.sleep(1)
def onDisconnect(self):
print("disconnected")
reactor.stop()
if __name__ == '__main__':
runner = ApplicationRunner(
environ.get("AUTOBAHN_DEMO_ROUTER", u"ws://127.0.0.1:8080/ws"),
u"crossbardemo")
runner.run(Component)
The following error is returned
2017-12-28T18:43:52+0100 [Router 1604] dropping connection to peer tcp4:127.0.0.1:61531 with abort=True: WebSocket ping timeout (peer did not respond with pong in time)
2017-12-28T18:43:52+0100 [Router 1604] session "8526139172223346" left realm "crossbardemo"
What I've tried to solve this problem:
I)
from twisted.internet.ssl import CertificateOptions
options = CertificateOptions()
if __name__ == '__main__':
runner = ApplicationRunner(
environ.get("AUTOBAHN_DEMO_ROUTER", u"ws://127.0.0.1:8080/ws"),
u"crossbardemo", ssl=options)
runner.run(Component)
II)
if __name__ == '__main__':
runner = ApplicationRunner(
environ.get("AUTOBAHN_DEMO_ROUTER", u"ws://127.0.0.1:8080/ws"),
u"crossbardemo",
)
runner.run(Component, auto_reconnect=True)
III)
Regedit
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.0\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
IV)
install certifi module (pip install certifi) set SSL_CERT_FILE, like
export SSL_CERT_FILE="$(python -m certifi)"
With still the same error. I am running on Windows 10, with crossbar demo router, autobahn and twisted.
Link to router configuration:
https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/wamp/pubsub/basic/.crossbar
Also, the following example code is working properly:
counter = 100
while True:
print("publish: com.myapp.ma", counter)
self.publish(u'com.myapp.ma', counter)
counter += 100
yield sleep(30)

For Twisted to process further I/O events, you have to give control back to the reactor. Twisted implements a cooperative multitasking system. Various tasks run in the reactor thread. This is accomplished by each task only spending a brief time in control. Code like:
while True:
...
sleep(1)
prevents any other tasks from gaining control to execute and also prevents the reactor from gaining control to service I/O events.
Since this code is within a function decorated with inlineCallbacks, there is a very small change that will make it at least not completely incompatible with Twisted's mode of operation.
Instead of time.sleep(1), try this expression:
yield deferLater(reactor, 1, lambda: None)
And import deferLater from twisted.internet.task. This will perform a "sleep" which gives control back to the reactor and lets other tasks execute during the sleep. This should allow Autobahn to send the necessary ping/pong messages as well as allow it to process your publish call.

Python: How to kill or sleep a process if a certain condition met?

I have built a scraper that keeps running in infinite loop. It is also using a pool of proxies on 10+ AWS machines. I want to make it like if a single IP hits X times then it should not be hit again, so far so good as I am using MySQL for the purpose. The issue is, what if ALL proxies finished quota, how can I make it sleep or delay and get a new life in new day. Right now I did following but eventually it terminates entire program:
def parse():
if random_proxy == '': # ALL Proxies finished quota for the day
raise Exception('Proxy Not Found')
r = session.get(u, headers=headers, proxies={'https': proxy_url})
if __name__ == '__main__':
idx = 0
exception_string = ''
POOL_COUNT = 50
try:
if conn is not None:
links = get_links(conn, POOL_COUNT)
if conn.open:
conn.close()
with Pool(POOL_COUNT) as p:
result = p.map(parse, links)
except Exception as ex:
print('Main program exception')

A very simple way to deal with your issue:
SLEEP_INTERVAL = 300 # 5 minutes
def parse():
random_proxy = get_random_proxy()
while not random_proxy:
# ALL Proxies finished quota for the day
time.sleep(SLEEP_INTERVAL)
random_proxy = get_random_proxy()
r = session.get(u, headers=headers, proxies={'https': proxy_url})
This will block your workers until a new proxy will be available.