Develop Reference

how to stop a while loop in python-telegram from client side

How can I stop a while loop inside a function which is passed as a parameter for a telegram bot CommandHandler, using a command or text from the client side?
I have this bot:
from telegram import *
from datetime import datetime
import time
from telegram.ext import *
import ast #ignore this. i'll use it later
RUN_FUNC = True
def func_starter(update: Update, context):
update.effective_chat.send_message('Function innitiated.')
counter = 0
RUN_FUNC = True
while RUN_FUNC:
print(f'function running for {counter} seconds.')
counter += 1
time.sleep(1)
def func_stopper(update: Update, context):
update.effective_chat.send_message('function stopped')
RUN_FUNC = False
def main():
updater = Updater('*********:***********************************', use_context=True)
dp = updater.dispatcher
dp.add_handler(CommandHandler("stop", func_stopper))
dp.add_handler(CommandHandler("start", func_starter))
updater.start_polling(0)
updater.idle()
main()
So the bot gets the /start command from the client and it starts the func_starter function which has conditional while loop in it. But because the program never makes it past the while loop, any other command / text from the client, never gets registered by the python code therefore the loop goes on forever. I want to be able to stop the func_starter function, using a command from the client side.
I even made a global variable but obviously to no avail lol. It is logical I think that the program never makes it past the while loop, so is there any way to listen for new commands while in the loop?

I see two options:
pass run_async=True to CommandHandler("start", …), which will make func_starter run in its own thread.
Instead of using a while loop inside callbacks, use JobQueue to schedule the next steps

Calling requests_html or pyppeteer in Python multithreading ErrorError: signal only works in main thread

I probably learned that the error was due to the use of coroutine io in pyppeteer and requests_html, which conflicted with multithreading, but I can't find a way to fix this.I don't speak English very much, I use google translation.
import asyncio
from pyppeteer import launch
from requests_html import HTMLSession
# Simulation using requests_html
def test1():
session = HTMLSession()
_r = session.get('http://bbs.tianya.cn/post-free-6085404-1.shtml' )
_r.html.render()
html = _r.html.html
print(html)
# main
async def main():
browser = await launch()
page = await browser.newPage()
await page.goto('http://example.com')
await page.screenshot({'path': 'example.png'})
await browser.close()
##The pyppeteer method is called at work
def aJob(arg):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(main())
## Multi-threaded task generation
def multiThread():
from multiprocessing.dummy import Pool as ThreadPool
cpus = 1 # 线程池大小
pool = ThreadPool(cpus)
_lstParam = range(0, 3)
pool.map(aJob, _lstParam)
pool.close()
pool.join()
if __name__ == "__main__":
loop = asyncio.new_event_loop()
multiThread()
I want to call pyppeteer or requests_html to simulate browsing the webpage in multithreading, but I always get the error "ValueError: signal only works in main thread" or "RuntimeError: There is no current event loop in thread 'Thread-1'." I have tried many methods, but I have been unable to run successfully.Please give me help, thank you!

Pyppeteer use signal to close the browser process but signal only works in main thread. If you don't really need this feature just set handleSIGINT,handleSIGTERM,handleSIGHUP to False on pyppeteer.launch method

Python asyncio (aiohttp, aiofiles)

I seem to be having a difficult time understanding pythons asyncio. I have not written any code, as all the examples I see are for one-off runs. Create a few coroutine's, add them to an event loop, then run the loop, they run the tasks switching between them, done. Which does not seem all that helpful for me.
I want to use asyncio to not interrupt the operation in my application (using pyqt5). I want to create some functions that when called run in the asyncio event loop, then when they are done they do a callback.
What I imagine is. Create a separate thread for asyncio, create the loop and run it forever. Create some functions getFile(url, fp), get(url), readFile(file), etc. Then in the UI, I have a text box with a submit button, user enters url, clicks submit, it downloads the file.
But, every example I see, I cannot see how to add a coroutine to a running loop. And I do not see how I could do what I want without adding to a running loop.
#!/bin/python3
import asyncio
import aiohttp
import threading
loop = asyncio.get_event_loop()
def async_in_thread(loop):
asyncio.set_event_loop(loop)
loop.run_forever()
async def _get(url, callback):
print("get: " + url)
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
result = await response.text()
callback(result)
return
def get(url, callback):
asyncio.ensure_future(_get(url, callback))
thread = threading.Thread(target=async_in_thread, args=(loop, ))
thread.start()
def stop():
loop.close()
def callme(data):
print(data)
stop()
get("http://google.com", callme)
thread.join()
This is what I imagine, but it does not work.

To add a coroutine to a loop running in a different thread, use asyncio.run_coroutine_threadsafe:
def get(url, callback):
asyncio.run_coroutine_threadsafe(_get(url, callback))
In general, when you are interacting with the event loop from outside the thread that runs it, you must run everything through either run_coroutine_threadsafe (for coroutines) or loop.call_soon_threadsafe (for functions). For example, to stop the loop, use loop.call_soon_threadsafe(loop.stop). Also note that loop.close() must not be invoked inside a loop callback, so you should place that call in async_in_thread, right after the call to run_forever(), at which point the loop has definitely stopped running.
Another thing with asyncio is that passing explicit when_done callbacks isn't idiomatic because asyncio exposes the concept of futures (akin to JavaScript promises), which allow attaching callbacks to a not-yet-available result. For example, one could write _get like this:
async def _get(url):
print("get: " + url)
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.text()
It doesn't need a callback argument because any interested party can convert it to a task using loop.create_task and use add_done_callback to be notified when the task is complete. For example:
def _get_with_callback(url, callback):
loop = asyncio.get_event_loop()
task = loop.create_task(_get(url))
task.add_done_callback(lambda _fut: callback(task.result()))
In your case you're not dealing with the task directly because your code aims to communicate with the event loop from another thread. However, run_coroutine_threadsafe returns a very useful value - a full-fledged concurrent.futures.Future which you can use to register done callbacks. Instead of accepting a callback argument, you can expose the future object to the caller:
def get(url):
return asyncio.run_coroutine_threadsafe(_get(url), loop)
Now the caller can choose a callback-based approach:
future = get(url)
# call me when done
future.add_done_callback(some_callback)
# ... proceed with other work ...
or, when appropriate, they can even wait for the result:
# give me the response, I'll wait for it
result = get(url).result()
The latter is by definition blocking, but since the event loop is safely running in a different thread, it is not affected by the blocking call.

Install QualMash to smooth integration between Qt and asyncio.
Example from the project's README gives an inspiration for how it looks like:
import sys
import asyncio
import time
from PyQt5.QtWidgets import QApplication, QProgressBar
from quamash import QEventLoop, QThreadExecutor
app = QApplication(sys.argv)
loop = QEventLoop(app)
asyncio.set_event_loop(loop) # NEW must set the event loop
progress = QProgressBar()
progress.setRange(0, 99)
progress.show()
async def master():
await first_50()
with QThreadExecutor(1) as exec:
await loop.run_in_executor(exec, last_50)
async def first_50():
for i in range(50):
progress.setValue(i)
await asyncio.sleep(.1)
def last_50():
for i in range(50,100):
loop.call_soon_threadsafe(progress.setValue, i)
time.sleep(.1)
with loop: ## context manager calls .close() when loop completes, and releases all resources
loop.run_until_complete(master())

Python Asyncio run_forever() and Tasks

I adapted this code for using Google Cloud PubSub in Async Python: https://github.com/cloudfind/google-pubsub-asyncio
import asyncio
import datetime
import functools
import os
from google.cloud import pubsub
from google.gax.errors import RetryError
from grpc import StatusCode
async def message_producer():
""" Publish messages which consist of the current datetime """
while True:
await asyncio.sleep(0.1)
async def proc_message(message):
await asyncio.sleep(0.1)
print(message)
message.ack()
def main():
""" Main program """
loop = asyncio.get_event_loop()
topic = "projects/{project_id}/topics/{topic}".format(
project_id=PROJECT, topic=TOPIC)
subscription_name = "projects/{project_id}/subscriptions/{subscription}".format(
project_id=PROJECT, subscription=SUBSCRIPTION)
subscription = make_subscription(
topic, subscription_name)
def create_proc_message_task(message):
""" Callback handler for the subscription; schedule a task on the event loop """
print("Task created!")
task = loop.create_task(proc_message(message))
subscription.open(create_proc_message_task)
# Produce some messages to consume
loop.create_task(message_producer())
print("Subscribed, let's do this!")
loop.run_forever()
def make_subscription(topic, subscription_name):
""" Make a publisher and subscriber client, and create the necessary resources """
subscriber = pubsub.SubscriberClient()
try:
subscriber.create_subscription(subscription_name, topic)
except:
pass
subscription = subscriber.subscribe(subscription_name)
return subscription
if __name__ == "__main__":
main()
I basically removed the publishing code and only use the subscription code.
However, initially I did not include the loop.create_task(message_producer()) line. I figured that tasks were created as they were supposed to however they never actually run themselves. Only if I add said line the code properly executes and all created Tasks run. What causes this behaviour?

PubSub is calling the create_proc_message_task callback from a different thread. Since create_task is not thread-safe, it must only be called from the thread that runs the event loop (typically the main thread). To correct the issue, replace loop.create_task(proc_message(message)) with asyncio.run_coroutine_threadsafe(proc_message(message), loop) and message_producer will no longer be needed.
As for why message_producer appeared to fix the code, consider that run_coroutine_threadsafe does two additional things compared to create_task:
It operates in a thread-safe fashion, so the event loop data structures are not corrupted when this is done concurrently.
It ensures that the event loop wakes up at the soonest possible opportunity, so that it can process the new task.
In your case create_task added the task to the loop's runnable queue (without any locking), but failed to ensure the wakeup, because that is not needed when running in the event loop thread. The message_producer then served to force the loop to wake up in regular intervals, which is when it also checks and executes the runnable tasks.

Asyncio process blocking

So i have a basic discord bot which accepts input
import discord
import asyncio
import threading
loop = asyncio.new_event_loop()
bot = discord.Client()
def run_asyncio_loop(loop):
asyncio.set_event_loop(loop)
loop.run_forever()
Hangman.set_bot(bot)
#bot.event
async def on_message(message):
bot.loop.create_task(Hangman.main(message))
asyncioLoop = threading.Thread(target = run_asyncio_loop, args = (loop,))
asyncioLoop.start()
bot.run(BotConstants.TOKEN)
In this example it calls the hangman game which does not block anything as i have tested this using asyncio.sleep(n) but when i go to do a something in hangman it blocks it.
class Hangman():
async def main(message):
await Hangman.make_guess(message)
async def update_score(message):
sheetLoaded = Spreadsheet.load_ws(...)
userExists = Spreadsheet.user_exists(...)
if (not userExists):
Spreadsheet.add_user(...)
Spreadsheet.add_score(...)
await Hangman.bot.send_message(message.channel, msg)
elif (not sheetLoaded):
await Hangman.bot.send_message(message.channel, msg)
async def make_guess(message):
# perform guess
if (matched):
await Hangman.bot.send_message(message.channel, msg)
Hangman.GAMES.pop(message.server.id)
await Hangman.update_score(message)
When Hangman.update_score() is called it blocks it. so it won't process any commands until the score has been updated which means for about 5 or so seconds (not long but with lots of users spamming it it's an issue) the bot does not accept any other messages
What am i missing to be able to make the process run in the background while still accept new inputs?

Asyncio is still single-threaded. The only way for the event loop to run is for no other coroutine to be actively executing. Using yield from/await suspends the coroutine temporarily, giving the event loop a chance to work. So unless you call another coroutine using yield (from) or await or return, the process is blocked. You can add await asyncio.sleep(0) in between steps of Hangman.update_score to divide the process blocking in multiple parts, but that will only ensure less "hanging" time, not actually speed up your thread.
To make the process actually run in the background, you could try something along the lines of:
from concurrent.futures import ProcessPoolExecutor
executor = ProcessPoolExecutor(2)
asyncio.ensure_future(loop.run_in_executor(executor, Hangman.update_score(message)))