I'm writing a code for a simple chat client in python. I have the GUI, a php server to store strings and other data. I want to make my code capable of updating the chat (conversation Text field) each 1 second.
I post a bit of pseudo-code:
Initialize Gui
Setup Users
UserX write messageX
messageX sent to server
At this point I need something that checks each second if userX(that could be user1 or user2) has new messages to display.
If I put something like:
while True:
time.sleep(1)
checkAndDisplayNewMessages()
the GUI doesn't appear! Because at the end of the code I got a mainloop()
To resume, I want my code to give the possibility to the user to send and receive messages asynchronously! With a part of code for sending messages if the user type in any message and the other part to constantly check for new messages while the program runs.
You did not mention which GUI toolkit you are using; from mainloop() I guess it's Tk.
The answer to this question explains how to set up a recurring event. Multithreading is not required.
You need to detach the way you fetch for new messages from the main thread of your applications. That can be easily done with threads in Python, it'd look something like this:
import threading
def fetch_messages(ui):
while not ui.ready():
#this loop syncs this process with the UI.
#we don't want to start showing messages
#until the UI is not ready
time.sleep(1)
while True:
time.sleep(1)
checkAndDisplayNewMessages()
def mainlogic():
thread_messages = threading.Thread(target=fetch_messages,args=(some_ui,))
thread_messages.start()
some_ui.show() # here you can go ahead with your UI stuff
# while messages are fetched. This method should
# set the UI to ready.
This implementation will run in parallel the process to seek for more messages and also will launch the UI. It is important that the UI is sync with the process to seek for messages otherwise you'd end up with funny exceptions. This is achieved by the first loop in the fetch_messages function.
Related
Hi currently I am using python telepot to:
Listen to incoming message
Based on the message it will do a long operation to generate some image.
Problem: The message is being process 1 by 1.
So if 2 users send in the message at the same time, it will process and send the image 1 by 1.
Have started reading Multiprocessing, but it is not getting what I want. What I have achieve now.
Switched to aiogram, listen to message
Pass the message to Process(target=imageprocessing.incoming, args=(userMsg,))
This kind of work, but every time a message comes in it will start a new process, it is kind of slow because it has to initialise some libraries etc before it process it.
Thus it is possible to:
Start X amount of "sub-process" in the main script initially.
Main script will listen for incoming message, queue it and pass it to one of the "sub-process"
So that X amount of message simultaneously based on how many "sub-process" I have defined initially.
Have been self-learning python as a side hobby.
Not sure if I have used the term correctly.
Try googling but can't find a solution for now, got a feeling I might be looking up the wrong keyword.
Anyone has any idea what should I look for?
Thank you very much.
I've got a simple pyramid app up and running, most of the views are a fairly thin wrapper around an sqlite database, with forms thrown in to edit/add some information.
A couple of times a month a new chunk of data will need to be added to this system (by csv import). The data is saved in an SQL table (the whole process right till commit takes about 4 seconds).
Every time a new chunk of data is uploaded, this triggers a recalculation of other tables in the database. The recalculation process takes a fairly long time (about 21-50 seconds for a month's worth of data).
Currently I just let the browser/client sit there waiting for the process to finish, but I do foresee the calculation process taking more and more time as the system gets more usage. From a UI perspective, this obviously looks like a hung process.
What can I do to indicate to the user that:-
That the long wait is normal/expected?
How MUCH longer they should have to wait (progress bar etc.)?
Note: I'm not asking about long-polling or websockets here, as this isn't really an interactive application and based on my basic knowledge websockets/async are overkill for my purposes.
I guess a follow-on question at this point, am I doing the wrong thing running processes in my view functions? Hardly seem to see that being done in examples/tutorials around the web. Am I supposed to be using celery or similar in this situation?
You're right, doing long calculations in a view function is generally frowned upon - I mean, if it's a typical website with random visitors who are able to hung a webserver thread for a minute then it's a recipe for a DoS vulnerability. But in some situations (internal website, few users, only admin has access to the "upload csv" form) you may get away with it. In fact, I used to have maintenance scripts which ran for hours :)
The trick here is to avoid browser timeouts - at the moment your client sends the data to the server and just sits there waiting for any reply, without any idea whether their request is being processed or not. Generally, at about 60 seconds the browser (or proxy, or frontend webserver) may become impatient and close the connection. Your server process will then get an error trying writing anything to the already closed connection and crash/raise an error.
To prevent this from happening the server needs to write something to the connection periodically, so the client sees that the server is alive and won't close the connection.
"Normal" Pyramid templates are buffered - i.e. the output is not sent to the client until the whole template to generated. Because of that you need to directly use response.app_iter / response.body_file and output some data there periodically.
As an example, you can duplicate the Todo List Application in One File example from Pyramid Cookbook and replace the new_view function with the following code (which itself has been borrowed from this question):
#view_config(route_name='new', request_method='GET', renderer='new.mako')
def new_view(request):
return {}
#view_config(route_name='new', request_method='POST')
def iter_test(request):
import time
if request.POST.get('name'):
request.db.execute(
'insert into tasks (name, closed) values (?, ?)',
[request.POST['name'], 0])
request.db.commit()
def test_iter():
i = 0
while True:
i += 1
if i == 5:
yield str('<p>Done! Click here to see the results</p>')
raise StopIteration
yield str('<p>working %s...</p>' % i)
print time.time()
time.sleep(1)
return Response(app_iter=test_iter())
(of cource, this solution is not too fancy UI-wise, but you said you didn't want to mess with websockets and celery)
So is the long running process triggered by browser action? I.e., the user is uploading the CSV that gets processed and then the view is doing the processing right there? For short-ish running browser processes I've used a loading indicator via jQuery or javascript, basically popping a modal animated spinner or something while a process runs, then when it completes hiding the spinner.
But if you're getting into longer and longer processes I think you should really look at some sort of background processing that will offload it from the UI. It doesn't have to be a message based worker, but even something like the end user uploads the file and a "to be processed" entry gets set in a database. Then you could have a pyramid script scheduled periodically in the background polling the status table and running anything it finds. You can move your file processing that is in the view to a separate method, and that can be called from the command line script. Then when the processing is finished it can update the status table indicating it is finished and that feedback could be presented back to the user somewhere, and not blocking their UI the whole time.
I'm building a telegram bot and for the start I used the structure from an example of the api wrapper. In the py script there is an infinite loop which is polling the telegram api to get new messages for the bot. And processes each new message one by one.
while True:
for update in bot.getUpdates(offset=LAST_UPDATE_ID, timeout=10):
chat_id = update.message.chat.id
update_id = update.update_id
if update.message.text:
#do things with the message \ start other functions and so on
What I foresee already now, is that with some messages\requests - i'll have a longer processing time and other messages, if the even came at the same time - will wait. For the user it will look like a delay in answering. Which boils down to a simple dependency: more user chatting = more delay.
I was thinking this: Can I have this main script bot.py run and check for new messages and each time a message arrived - this script will kickstart another script answer.py to do the processing of the message and reply.
And to start as many as needed those answer.py scripts in parallel.
I can also use bot.py to log all incoming things into DB with reference data about the user who is sending a message and then have another process processing all newly logged data and marking it as answered - but also then it should process each new entry parallel to each other.
I'm not a guru in python and is asking for some ideas and guidance on how to approach this? Thank you!
What you need are threads, or some frameworks that can handle many requests asynchronously, e.g. Twisted, Tornado, or asyncio in Python 3.4.
Here is an implementation using threads:
import threading
def handle(message):
##### do your response here
offset = None
while True:
for update in bot.getUpdates(offset=offset, timeout=10):
if update.message.text:
t = threading.Thread(target=handle, args=(update.message,))
t.start()
offset = update.update_id + 1
##### log the message if you want
This way, the call to handle() would not block, and the loop can go on handling the next message.
For more complicated situations, for example if you have to maintain states across messages from the same chat_id, I recommend taking a look at telepot, and this answer:
Handle multiple questions for Telegram bot in python
In short, telepot spawns threads for you, freeing you from worrying about the low-level details and letting you focus on the problem at hand.
I need to execute a command on a simple button press event in my Django project (for which I'm using "subprocess.Popen()" in my views.py ).
After I execute this script it may take anywhere from 2 minutes to 5 minutes to complete. So while the script executes I need to disable the html button but I want the users to continue using other web pages while the script finishes in the background. Now the real problem is that I want to enable the html button back, when the process finishes!
I'm stuck at this from many days. Any help or suggestion is really really appreciated.
I think you have to use some "realtime" libraries for django. I personally know django-realtime (simple one) and swampdragon (less simple, but more functional). With both of this libraries you can create web-socket connection and send messages to clients from server that way. It may be command for enabling html button or javascript alert or whatever you want.
In your case I advice you first option, because you can send message to client directly from any view. And swampdragon needs model to track changes as far I know.
Like valentjedi suggested, you should be using swampdragon for real time with django.
You should take the first tutorial here: http://swampdragon.net/tutorial/part-1-here-be-dragons-and-thats-a-good-thing/
Then read this as it holds knowledge required to accomplish what you want:
http://swampdragon.net/tutorial/building-a-real-time-server-monitor-app-with-swampdragon-and-django/
However there is a difference between your situation and the example given above, in your situation:
Use Celery or any other task queue, since the action you wait for takes long time to finish, you will need to pass it to the background. (You can also make these tasks occur one after another if you don't want to freeze your system with enormous memory usage).
Move the part of code that runs the script to your celery task, in this case, Popen should be called in your Celery task and not in your view (router in swampdragon).
You then create a channel with the user's unique identifier, and add relevant swampdragon javascript code in your html file for the button to subscribe to that user's channel (also consider disabling the feature on your view (router) since front-end code can be tempered with.
The channel's role will be to pull the celery task state, you
then disable or enable the button according to the state of
the task.
overview:
Create celery task for your script.
Create a user unique channel that pulls the task state.
Disable or enable the button on the front-end according to the state of the taks, consider displaying failure message in case the script fails so that the user restart again.
Hope this helps!
I would like to create a Twilio app in python that runs in the background on my mac osx desktop and performs an action whenever a new text message arrives. I'm only interested in processing the most recently received text message.
My current solution is a polling mechanism where I have an infinite loop and in the loop I call TwilioRestClient(...).messages.list() every single time and process the first message in the returned array. I sleep for 1 second in each iteration in order to avoid taxing the processor too much.
The current implementation usually works but sometimes lags at the TwilioRestClient(...).messages.list() call for multiple seconds. I'm assuming this is because I'm using the trial version of Twilio. However, any advice on how to fix this would be greatly appreciated. The other main issue is that the implementation is extremely wasteful. The app should really be sleeping for the vast majority of the time it is running and there should be brief spurts of activity when new messages arrive.
Here's the current implementation:
def run(self):
last_message = self.get_most_recent_message_body().lower()
self.is_running = True
while self.is_running:
message = self.get_most_recent_message_body().lower()
if (message != last_message):
last_message = message
self.commander.execute(message)
sleep(1);
def get_most_recent_message_body(self):
messages = TwilioRestClient(self.TWILIO_ACCOUNT_SID, self.TWILIO_AUTH_TOKEN).messages.list()
most_recent_message = messages[0]
return most_recent_message.body
I want the app to be as responsive as possible. The optimal solution would be some kind of Observer pattern or some kind of signal use where Twilio alerts my app to "wake up" whenever a new message arrives. Is there a way to implement this?
Thanks!
Twilio developer evangelist here.
You can absolutely do that without polling. Let me explain.
When you buy a number with Twilio you can set two URLs that point to an application that you build, one for voice and one for messaging. Then, when Twilio receives a call or a text message it will make an HTTP request (a webhook) to the url you set with the details of the call or the message. Your application can respond with some XML (called TwiML) to tell Twilio what to do with the message or call.
I'm guessing you're writing a python application as you tagged this question with Python. Here's a quick guide to getting started receiving SMS messages with Python.