Develop Reference

Python websockets: how to create dynamically restartable connections

I've been using websocket.WebSocketApp().run_forever to subscribe to a websocket to obtain a stream of data. However, now I need to dynamically restart the connection at certain times of the day to change the channel parameters.
I am thinking of basically running two concurrent tasks:
The original task that connects to the websocket
A task that runs forever and checks the time of day. If a certain time is reached, it will restart task 1. with new parameters.
There's quite a lot of technologies out there but after trying a few options, there seems to always be some caveat where some library doesn't play well with being asynchronous. If anyone has tackled a problem like this before, any pointers would be greatly appreciated.

Initiate a parallel process from within a python script?

I'm building a telegram bot and for the start I used the structure from an example of the api wrapper. In the py script there is an infinite loop which is polling the telegram api to get new messages for the bot. And processes each new message one by one.
while True:
for update in bot.getUpdates(offset=LAST_UPDATE_ID, timeout=10):
chat_id = update.message.chat.id
update_id = update.update_id
if update.message.text:
#do things with the message \ start other functions and so on
What I foresee already now, is that with some messages\requests - i'll have a longer processing time and other messages, if the even came at the same time - will wait. For the user it will look like a delay in answering. Which boils down to a simple dependency: more user chatting = more delay.
I was thinking this: Can I have this main script bot.py run and check for new messages and each time a message arrived - this script will kickstart another script answer.py to do the processing of the message and reply.
And to start as many as needed those answer.py scripts in parallel.
I can also use bot.py to log all incoming things into DB with reference data about the user who is sending a message and then have another process processing all newly logged data and marking it as answered - but also then it should process each new entry parallel to each other.
I'm not a guru in python and is asking for some ideas and guidance on how to approach this? Thank you!

What you need are threads, or some frameworks that can handle many requests asynchronously, e.g. Twisted, Tornado, or asyncio in Python 3.4.
Here is an implementation using threads:
import threading
def handle(message):
##### do your response here
offset = None
while True:
for update in bot.getUpdates(offset=offset, timeout=10):
if update.message.text:
t = threading.Thread(target=handle, args=(update.message,))
t.start()
offset = update.update_id + 1
##### log the message if you want
This way, the call to handle() would not block, and the loop can go on handling the next message.
For more complicated situations, for example if you have to maintain states across messages from the same chat_id, I recommend taking a look at telepot, and this answer:
Handle multiple questions for Telegram bot in python
In short, telepot spawns threads for you, freeing you from worrying about the low-level details and letting you focus on the problem at hand.

How to avoid polling in Twilio?

I would like to create a Twilio app in python that runs in the background on my mac osx desktop and performs an action whenever a new text message arrives. I'm only interested in processing the most recently received text message.
My current solution is a polling mechanism where I have an infinite loop and in the loop I call TwilioRestClient(...).messages.list() every single time and process the first message in the returned array. I sleep for 1 second in each iteration in order to avoid taxing the processor too much.
The current implementation usually works but sometimes lags at the TwilioRestClient(...).messages.list() call for multiple seconds. I'm assuming this is because I'm using the trial version of Twilio. However, any advice on how to fix this would be greatly appreciated. The other main issue is that the implementation is extremely wasteful. The app should really be sleeping for the vast majority of the time it is running and there should be brief spurts of activity when new messages arrive.
Here's the current implementation:
def run(self):
last_message = self.get_most_recent_message_body().lower()
self.is_running = True
while self.is_running:
message = self.get_most_recent_message_body().lower()
if (message != last_message):
last_message = message
self.commander.execute(message)
sleep(1);
def get_most_recent_message_body(self):
messages = TwilioRestClient(self.TWILIO_ACCOUNT_SID, self.TWILIO_AUTH_TOKEN).messages.list()
most_recent_message = messages[0]
return most_recent_message.body
I want the app to be as responsive as possible. The optimal solution would be some kind of Observer pattern or some kind of signal use where Twilio alerts my app to "wake up" whenever a new message arrives. Is there a way to implement this?
Thanks!

Twilio developer evangelist here.
You can absolutely do that without polling. Let me explain.
When you buy a number with Twilio you can set two URLs that point to an application that you build, one for voice and one for messaging. Then, when Twilio receives a call or a text message it will make an HTTP request (a webhook) to the url you set with the details of the call or the message. Your application can respond with some XML (called TwiML) to tell Twilio what to do with the message or call.
I'm guessing you're writing a python application as you tagged this question with Python. Here's a quick guide to getting started receiving SMS messages with Python.

Query Python3 script to get stats about the script

I have a script that continually runs and accepts data (For those that are familiar and if it helps, it is connected to EMDR - https://eve-market-data-relay.readthedocs.org).
Inside the script I have debugging built in so that I can see how much data is currently in the queue for the threads to process, however this is built to be used with just printing to the console. What I would like to do is be able to either run the same script with an additional option or a totally different script that would return the current queue count without having to enable debug.
Is there a way to do this could someone please point me in the direction of the documentation/libaries that I need to research?

There are many ways to solve this; two that come to mind:
You can write the queue count to a k/v store (like memcache or redis) and then have another script read that for you and do whatever other actions required.
You can create a specific logger for your informational output (like the queue length) and set it to log somewhere else other than the console. For example, you could use it to send you an email or log to an external service, etc. See the logging cookbook for examples.

How to prebuffer an incoming network stream with gstreamer?

I'm using gstreamer to stream audio over the network. My goal is seemingly simple: Prebuffer the incoming stream up to a certain time/byte threshold and then start playing it.
I might be overlooking a really simple feature of gstreamer, but so far, I haven't been able to find a way to do that.
My (simplified) pipeline looks like this: udpsrc -> alsasink. So far all my attempts at achieving my goal have been using a queue element in between:
Add a queue element in between.
Use the min-threshold-time property. This actually works but the problem is, that it makes all the incoming data spend the specified minimum amount of time in the queue, rather than just the beginning, which is not what I want.
To work around the previous problem, I tried to have the queue notify my code when data enters the audio sink for the first time, thinking that this is the time to unset the min-time property that I set earlier, and thus, achieving the "prebuffering" behavior.
Here's is a rough equivalent of the code I tried:
def remove_thresh(pad, info, queue):
pad.remove_data_probe(probe_id)
queue.set_property("min-threshold-time", 0)
queue.set_property("min-threshold-time", delay)
queue.set_property("max-size-time", delay * 2)
probe_id = audiosink.get_pad("sink").add_data_probe(remove_thresh, queue)
This doesn't work for two reasons:
My callback gets called way earlier than the delay variable I provided.
After it gets called, all of the data that was stored in the queue is lost. the playback starts as if the queue weren't there at all.
I think I have a fundamental misunderstanding of how this thing works. Does anyone know what I'm doing wrong, or alternatively, can provide a (possibly) better way to do this?
I'm using python here, but any solution in any language is welcome.
Thanks.

Buffering has already been implemented in GStreamer. Some elements, like the queue, are capable of building this buffer and post bus messages regarding the buffer level (the state of the queue).
An application wanting to have more network resilience, then, should listen to these messages and pause playback if the buffer level is not high enough (usually, whenever it is below 100%).
So, all you have to do is set the pipeline to the PAUSED state while the queue is buffering. In your case, you only want to buffer once, so use any logic for this (maybe set flag variables to pause the pipeline only the first time).
Set the "max-size-bytes" property of the queue to the value you want.
Either listen to the "overrun" signal to notify you when the buffer becomes full or use gst_message_parse_buffering () to find the buffering level.
Once your buffer is full, set the pipeline to PLAYING state and then ignore all further buffering messages.
Finally, for a complete streaming example, you can refer to this tutorial: https://gstreamer.freedesktop.org/documentation/tutorials/basic/streaming.html
The code is in C, but the walkthroughs should help you with you want.

I was having the exact same problems as you with a different pipeline (appsrc), and after spending days trying to find an elegant solution (and ending up with code remarkably similar to what you posted)... all I did was switch the flag is-live to False and the buffering worked automagically. (no need for min-threshold-time or anything else)
Hope this helps.