Develop Reference

Twisted - Deferred, if its not asynchronous, then whats the point?

I've been tasked with learning Twisted.
I am also somewhat new to Python in general, but have used other modern programming languages.
In reading over Twisted documentation, I keep running into examples that are
Not complete executable examples
Run in one thread
Coming from other languages, when I use some asynchronous mechanism, there is usually another thread of execution while I carry out some manner of work, then I am notified when that work is completed, and I react to its results.
I do see that it has some built in asynchronous mechanisms, but none of them provide the user with a means to create custom CPU bound asynchronous tasks akin to 'Tasks' in C# or 'work' with boost::asio in C++ that would run in parallel to the main thread.
I see that Twisted provides a means to asynchronously wait on IO and do things in on the same thread while waiting, if we are waiting on:
network reads and writes
keyboard input
It also shows me how to:
Do some manner of integration with GUI tool kits to make use of their event loop, but doesn't go into detail.
Schedule tasks using reactor on a timer, but doesn't do that task in parallel to anything else
It talks about async/await, but that is for python 3 only, and I am using python 2.7
I figured the some manner of thread pooling must be built into the reactor, but then when I read about the reactor, it says that everything runs on the main thread in reactor.run().
So, I am left confused.
What is the point of deferreds, creating a callback chain and reacting to the results, if we aren't running anything in parallel?
If we are running asynchronous code, how are we making our own custom asynchronous functions? (see keyword async in C#)
In other languages, I might create an async task to count from 1 to 10, while on the main thread, I might count from 'a' to 'z' at the same time. When the the task is complete I would get notified via a callback on a thread from a threadpool. I'd have the option to sync up, if I wanted to, by calling some 'wait' method. While the definition of "asynchronous" only involves the posting of the task, the getting of the result, and the callback when its done....I've never seen it used without doing things in parallel.

I'll address your questions (and statements that seem confusing) one-by-one:
"Examples that are not complete"
Restating what I posted in the comments: see my two previous answers for complete examples ( https://stackoverflow.com/a/30399317/3334178 & https://stackoverflow.com/a/23274411/3334178 ) and go through Krondo's Twisted Introduction
You said you are discounting these because "The examples are the network code in twisted, which has the asynchronisity built in and hidden.". I disagree with that assertion and will explain this in the next section.
"Examples are not asynchronous"
When your talking about "asynchronous programming" in the vain of pythons twisted/tornado/asyncio (or Node.JS or C select/poll/kpoll) your talking about model/pattern of programming that allows the programmer shape their code so that parts of it can run while other parts are blocked (in almost all cases the blocking is caused by a part of the program having to wait for IO).
These libraries/languages will certainly have ways they can do threading and/or multiprocessing, but those are layers grafted on top of the async design - and if that's genuinely what you need (I.E. you have an exclusively CPU bound need) the async systems are going to be a bad choice.
Let's use your "hidden away" comment to get into this a bit more
"Network examples are asych, but the asynchronousity is built in and hidden away"
The fundamental element of the async design is that you write your code so it should never block for IO - You've been calling out network but really we are talking about network/disk/keyboard/mouse/sound/serial - anything that (for whatever reason) can run slower than the CPU (and that the OS has a file-descriptor for).
Also, there isn't anything really "hidden away" about how it functions - async programming always uses non-blocking (status checking / call-back) calls for any IO channel it can operate on. If you dig enough in the twisted codebase all the async logic is in plain sight (Krondo's tutorial is really good for giving examples of this)
Let me use the keyboard as an example.
In sync code, you would use an input or a read - and the program would pause waiting for that line (or key) to be typed.
In async code (at least in featureful implementations like twisted) you will fetch the file-descriptor for "input" and register it with call-back function, to be called when the file-descriptor changes, to the OS-level async engine (select, poll, kpoll, etc...)
The act of doing that registration - which takes almost no time LETS YOU run other logic while the keyboard logic waits for the keyboard event to happen (see the stdio.StandardIO(keyboardobj,sys.stdin.fileno()) line from near the end of my example code in https://stackoverflow.com/a/30399317/3334178).
"[This] leads me to believe there is some other means to use deferreds with asynchronous"
deferreds aren't magic. They are just clever lists of function callback. There are numerous clever ways they can be chained together, but in the end, they are just a tool to help you take advantage of the logic above
"It also talks about async/await, that is for python 3 only, and I am using python 2.7"
async and await are just the python 3 way of doing what was done in python2 with #defer.inlineCallbacks and yield. These systems are shortcuts that rewire code so that to the reader the code looks and acts like sync code, but when its run the code is morphed into a "register a callback and move-on" flow
"when I read about the reactor, it says that everything runs on the main thread in reactor.run()"
Yes, because (as above) async is about not-waiting-for-IO - its not about threading or multi-processing
Your last few questions "point of deferreds" and "how do you make asynchronous" feel like I answered them above - but if not, let me know in the comments, and I'll spell them out.
Also your comment requesting "an example where we count from 1 to 10 in some deferred function while we count from a to z in the main thread?" doesn't make sense when talking about async (both because you talk about a "thread" - which is a different construct, and because those are both (likely) CPU tasks), but I will give you a different example that counts while watching for keyboard input (which is something that definitely DOES make sense when talking about async:
#!/usr/bin/env python
#
# Frankenstein-esk amalgam of example code
# Key of which comes from the Twisted "Chat" example
# (such as: http://twistedmatrix.com/documents/12.0.0/core/examples/chatserver.py)
import sys # so I can get at stdin
import os # for isatty
import termios, tty # access to posix IO settings
from twisted.internet import reactor
from twisted.internet import stdio # the stdio equiv of listenXXX
from twisted.protocols import basic # for lineReceiver for keyboard
from twisted.internet import task
class counter(object):
runs = 0
def runEverySecond():
counter.runs += 1
print "async counting demo: " + str(counter.runs)
# to set keyboard into cbreak mode - so keys don't require a CR before causing an event
class Cbreaktty(object):
org_termio = None
my_termio = None
def __init__(self, ttyfd):
if(os.isatty(ttyfd)):
self.org_termio = (ttyfd, termios.tcgetattr(ttyfd))
tty.setcbreak(ttyfd)
print ' Set cbreak mode'
self.my_termio = (ttyfd, termios.tcgetattr(ttyfd))
else:
raise IOError #Not something I can set cbreak on!
def retToOrgState(self):
(tty, org) = self.org_termio
print ' Restoring terminal settings'
termios.tcsetattr(tty, termios.TCSANOW, org)
class KeyEater(basic.LineReceiver):
def __init__(self):
self.setRawMode() # Switch from line mode to "however much I got" mode
def rawDataReceived(self, data):
key = str(data).lower()[0]
if key == 'q':
reactor.stop()
else:
print "--------------"
print "Press:"
print " q - to cleanly shutdown"
print "---------------"
# Custom tailored example for SO:56013998
#
# This code is a mishmash of styles and techniques. Both to provide different examples of how
# something can be done and because I'm lazy. Its been built and tested on OSX and linux,
# it should be portable (other then perhaps termal cbreak mode). If you want to ask
# questions about this code contact me directly via mail to mike at partialmesh.com
#
#
# Once running press any key in the window where the script was run and it will give
# instructions.
def main():
try:
termstate = Cbreaktty(sys.stdin.fileno())
except IOError:
sys.stderr.write("Error: " + sys.argv[0] + " only for use on interactive ttys\n")
sys.exit(1)
keyboardobj = KeyEater()
l = task.LoopingCall(runEverySecond)
l.start(1.0) # call every second
stdio.StandardIO(keyboardobj,sys.stdin.fileno())
reactor.run()
termstate.retToOrgState()
if __name__ == '__main__':
main()
(I know technically I didn't use a deferred - but I ran out of time - and this case is a bit too simple to really need it (I don't have a chain of callback anywhere, which is what deferreds are for))

How to handle blocking calls / while 1-s, inside a tkinter's mainloop? ZMQ / Tkinter

I am using ZeroMQ, which is a messaging library (presumably async I/O), if you don't know what it is you can think of it as similar to socket library in python, the sockets used for messaging are usually run within an infinite while loop with a small sleep for keep everything cool.
I have the code written in a separate file and I have a GUI based on the working of that code separate, I want to integrate the two codes.
But the issue I come across is that I can not possibly put a while True, or a blocking socket.recv() inside tkinter's .mainloop().
I want to receive on on a socket, which is blocking - BUT I can manage that part of the issue, zmq sockets can either be polled on (check periodically to see if we have any pending messages to process) or equivalently you can use zmq.DONTWAIT which does the same thing.
The issue remaining however is that I need a while True, so that the socket is constantly polled, say every millisecond to see if we have messages.
How do I put a while True inside the tkinter .mainloop() that allows me to check the state of that socket constantly?
I would visualize something like this :
while True:
update_gui() # contains the mainloop and all GUI code
check_socket() # listener socket for incoming traffic
if work:
# # do a work, while GUI will hang for a bit.
I have checked the internet, and came across solution on SO, which says that you can use the After property of widgets but I am not sure how that works. If someone could help me out I would be super grateful !
Code for reference :
zmq.DONTWAIT throws an exception if you do not have any pending messages which makes us move forward in the loop.
while 1:
if socket_listen and int(share_state):
try:
msg = socket_listen.recv_string(zmq.DONTWAIT)
except:
pass
time.sleep(0.01)
I would like that I could put this inside the .mainloop() and along with the GUI this also gets checked every iteration.
Additional info : Polling here equates to :
check if we have messages on socket1
if not then proceed normally
else do work.

How do I put a while True inside the tkinter .mainloop() that allows me to check the state of that socket constantly?
Do not design such part using an explicit while True-loop, better use the tkinter-native tooling: asking .after() to re-submit the call not later than a certain amount of time ( let for other things to happen concurrently, yet having a reasonable amount of certainty, your requested call will still be activated no later than "after" specified amount of milliseconds ).
I love Tkinter architecture of co-existing event processing
So if one keeps the Finite-State-Automata ( a game, or a GUI front-end ) clean crafted on the Tkinter-grounds, one can enjoy delivering ZeroMQ-messages data being coordinated "behind" the scene, right by Tkinter-native tools, so no imperative-code will be needed whatsoever. Just let the messages get translated into tkinter-monitored-variables, if you need to have indeed smart-working GUI integration.
aScheduledTaskID = aGuiRelatedOBJECT.after( msecs_to_wait,
aFunc_to_call = None,
*args
)
# -> <_ID_#_>
# ... guarantees a given wait-time + just a one, soloist-call
# after a delay of at least delay_ms milliseconds.
# There is no upper limit to how long it will actually take, but
# your callback-FUN will be called NO SOONER than you requested,
# and it will be called only once.
# aFunc_to_call() may "renew" with .after()
#
# .after_cancel( aScheduledTaskID ) # <- <id> CANCELLED from SCHEDULER
#
# .after_idle() ~ SCHEDULE A TASK TO BE CALLED UPON .mainloop() TURNED-IDLE
#
# aScheduledTaskOnIdleID = aGuiOBJECT.after_idle( aFunc_to_call = None,
# *args
# )
# -> <_ID_#_>
That's cool on using the ready-to-reuse tkinter native-infrastructure scheduler tools in a smart way, isn't it?
Epilogue:
( Blocking calls? Better never use blocking calls at all. Have anyone ever said blocking calls here? :o) )
a while True, or a blocking socket.recv() inside tkinter's .mainloop().
well, one can put such a loop into a component aligned with tkinter native-infrastructure scheduler, yet this idea is actually an antipattern and can turn things into wreck havoc ( not only for tkinter, in general in any event-loop handler it is a bit risky to expect any "competitive" event-handler loop to somehow tolerate or behave in a peacefull the co-existence of adjacent intentions - problems will appear ( be it from a straight blocking or due to one being a just too much dominant in scheduling resources or other sorts of a war on time and resources ) ).

how to use lldb trace-thread and python SBThread eBroadcastBitSelectedFrameChanged event

I want to trace thread by log all the symbol it call, so I found tow method
1、the lldb settings list:
'target.process.thread' variables:
trace-thread -- If true, this thread will single-step and log execution.
it means the lldb will log execution, but I can't find where is the log
2、lldb python SBThread has a event eBroadcastBitSelectedFrameChanged, I think it will callback when thread frame change, but why SBThread has no broadcaster?

1) This setting was put in mostly to help diagnose problems with lldb's stepping algorithms. Since it causes all execution to go by instruction single step, it's going to make your program execute very slowly, so it hasn't been used for anything other than that purpose (and I haven't used it for that purpose in a good while, so it might have bit-rotted.) The output is supposed to go to the debugger's stdout.
2) eBroadcastBitSelectedFrameChanged is only sent when the user changes the selected frame with command line commands. It's meant to allow a GUI like Xcode that also allows command line interaction to keep the GUI sync'ed with user commands in the console. There isn't a GetBroadcaster for threads, because threads come and go and you generally want to listen to ALL the threads, not just a particular one. To do that, call SBThread.GetBroadcasterClassName and then sign your listener up for events by class name (StartListeningForEventClass).
If you have a need to listen to a particular thread, file an enhancement request to the bug tracker at http://lldb.llvm.org.

Python - Executing Functions During a raw_input() Prompt [duplicate]

I'm trying to write a simple Python IRC client. So far I can read data, and I can send data back to the client if it automated. I'm getting the data in a while True, which means that I cannot enter text while at the same time reading data. How can I enter text in the console, that only gets sent when I press enter, while at the same time running an infinite loop?
Basic code structure:
while True:
read data
#here is where I want to write data only if it contains '/r' in it

Another way to do it involves threads.
import threading
# define a thread which takes input
class InputThread(threading.Thread):
def __init__(self):
super(InputThread, self).__init__()
self.daemon = True
self.last_user_input = None
def run(self):
while True:
self.last_user_input = input('input something: ')
# do something based on the user input here
# alternatively, let main do something with
# self.last_user_input
# main
it = InputThread()
it.start()
while True:
# do something
# do something with it.last_user_input if you feel like it

What you need is an event loop of some kind.
In Python you have a few options to do that, pick one you like:
Twisted https://twistedmatrix.com/trac/
Asyncio https://docs.python.org/3/library/asyncio.html
gevent http://www.gevent.org/
and so on, there are hundreds of frameworks for this, you could also use any of the GUI frameworks like tkinter or PyQt to get a main event loop.
As comments have said above, you can use threads and a few queues to handle this, or an event based loop, or coroutines or a bunch of other architectures. Depending on your target platforms one or the other might be best. For example on windows the console API is totally different to unix ptys. Especially if you later need stuff like colour output and so on, you might want to ask more specific questions.

You can use a async library (see answer of schlenk) or use https://docs.python.org/2/library/select.html
This module provides access to the select() and poll() functions
available in most operating systems, epoll() available on Linux 2.5+
and kqueue() available on most BSD. Note that on Windows, it only
works for sockets; on other operating systems, it also works for other
file types (in particular, on Unix, it works on pipes). It cannot be
used on regular files to determine whether a file has grown since it
was last read.

Execute Multi-threading process

I'm writing a code for a simple chat client in python. I have the GUI, a php server to store strings and other data. I want to make my code capable of updating the chat (conversation Text field) each 1 second.
I post a bit of pseudo-code:
Initialize Gui
Setup Users
UserX write messageX
messageX sent to server
At this point I need something that checks each second if userX(that could be user1 or user2) has new messages to display.
If I put something like:
while True:
time.sleep(1)
checkAndDisplayNewMessages()
the GUI doesn't appear! Because at the end of the code I got a mainloop()
To resume, I want my code to give the possibility to the user to send and receive messages asynchronously! With a part of code for sending messages if the user type in any message and the other part to constantly check for new messages while the program runs.

You did not mention which GUI toolkit you are using; from mainloop() I guess it's Tk.
The answer to this question explains how to set up a recurring event. Multithreading is not required.

You need to detach the way you fetch for new messages from the main thread of your applications. That can be easily done with threads in Python, it'd look something like this:
import threading
def fetch_messages(ui):
while not ui.ready():
#this loop syncs this process with the UI.
#we don't want to start showing messages
#until the UI is not ready
time.sleep(1)
while True:
time.sleep(1)
checkAndDisplayNewMessages()
def mainlogic():
thread_messages = threading.Thread(target=fetch_messages,args=(some_ui,))
thread_messages.start()
some_ui.show() # here you can go ahead with your UI stuff
# while messages are fetched. This method should
# set the UI to ready.
This implementation will run in parallel the process to seek for more messages and also will launch the UI. It is important that the UI is sync with the process to seek for messages otherwise you'd end up with funny exceptions. This is achieved by the first loop in the fetch_messages function.