I want to confirm the following questions:
There is no native coroutine decorator in Python 2. It would be provided by something like [1].
Prior to Python 3.4, all other Python 3 releases require pip install asyncio in order to use import asyncio.coroutine [2].
trollius is the reference port implementation of aysncio and tulips for Python 2 (and the community thinks that's the one to use)?
Thank you.
I have used Python generators as coroutines, using only built-in methods. I have not used coroutines in any other environment, so my approach may be misinformed.
Here's some starter code I wrote that uses generators to send and receive data in a coroutine capacity, without even using Python 3's yield from syntax:
def sleep(timer, action=None):
''' wait for time to elapse past a certain timer '''
yield # first yield cannot accept a message
now = then = yield
while now - then < timer:
now = yield
if action:
action()
else:
yield "timer finished"
def buttonwait():
''' yields button presses '''
yield
yield
while True:
c = screen.getch()
if c:
yield c
yield
next, the wait function which manages the coroutines, sending the current time and listening for data
def wait(processes):
start = time.time()
for process in processes:
process.next()
process.send(start)
while True:
now = time.time()
for process in processes:
value = process.send(now)
if value:
return value
last, an implementation of those:
def main():
processes = []
process.append(timer(5)
processes.append(buttonwait())
wait(processes)
I used this on a Raspberry Pi with a 2x16 LCD screen to:
respond to button presses
turn off the backlight after a timeout
scroll long text
move motors
send serial commands
It's a bit complicated to get started, knowing where to put the yields and whatnot, but seems reasonably functional once it is.
Judging from your question, I think you've conflated two things: co-routines and async I/O-run loops. They do not depend upon each other.
Because in Python one can send values into a generator, you can create code implementing the co-routine pattern. All that is needed is generator.send(). Your link to David Beazley's code is a good example of how to create a co-routine. It is just a pattern. The new to Python v3.3 yield from allows this pattern to be used even more flexibly. David has an excellent talk on all of the things you can do with co-routines in a synchronous computing world.
Python's asyncio module depends upon the yield from construct. They have created an async co-routine to package this capability and associate it with a run loop. The goal, I believe, is to allow folks to easily build run loop oriented applications. There is still a role for the Twisteds and Tornados of the world.
(I myself am wary of projects like Trollius. They are kind of "Dancing Bear" project. They miss the point. Asyncio is about bringing a straightforward run loop implementation to Python 3 as a standard service. Python 2 already has two or more excellent async I/O libraries. Albeit these libraries are complex. IOW, if you are starting with Python 3 and your needs are straightforward, then use asyncio otherwise use Tornado. [Twisted isn't ported yet.] If you are starting with Python 2, then Twisted or Tornado is probably where you should start. Trollius? A code incompatible version with Python 3? An excellent "Dancing Bear.")
In my book, asyncio is an excellent reason to move your code to Python 3. We live in an asynchronous world and run loops are too important a feature to be project specific.
Anon,
Andrew
Related
I have a library that does calls to smart contracts in the ethereum chain to read data
So for simplicity, my code is like this:
import library
items = [
"address1",
"address2",
"address3",
]
for item in items:
data = library.get_smartcontractinfo(item)
print(data)
if __name__ == '__main__':
main()
I am new to concurrency and this is a topic I need to explore further, as there are many options to do concurrency but seems asyncio is the one most people go for
The library I a musing is not built with asyncio or any sort of concurrency in mind. This means that each time I call the library.get_smartcontractinfo() function then I need to wait until it completes the query so it can do the next iteration, which is blocking the speed.
Lets say that I cannot modify the library, althought maybe I will in the future, but I wanto get something done asap with the existing code
What would be the easiest way to do simultaneous queries so I can get the info as fast as I can in an efficient way?
What about being rate limited? And would it be possible to group these calls into one without rewriting the library code?
Thank you.
Assuming that library.get_smartcontractinfo() does a lot of network I/O, you could use a ThreadPoolExecutor from concurrent.futures to run more of them in parallel.
The documentation has a good example.
Assuming the function library.get_smartcontractinfo() is a I/O bound, you have multiple options to go with asyncio. If you want to use pure asyncio you can go with something like
async def main():
loop = asyncio.get_running_loop()
all_runs = [loop.run_in_executor(None, library.get_smartcontractinfo, item) for item in items]
results = await asyncio.gather(*all_runs)
Bascially running the sync function in a thread. To run those concurrently, you first create all coroutines without awaiting them, and finally pass those into gather.
If you want to use some additional library, I can recommend using anyio or asyncer which basically is a nice wrapper around anyio. With `asyncer?, you basically can change the one line where you transfer a sync function into an async one to
from asyncer import asyncify
...
all_runs = [asyncify(library.get_smartcontractinfo)(item) for item in items]
the rest stays the same.
I've been tasked with learning Twisted.
I am also somewhat new to Python in general, but have used other modern programming languages.
In reading over Twisted documentation, I keep running into examples that are
Not complete executable examples
Run in one thread
Coming from other languages, when I use some asynchronous mechanism, there is usually another thread of execution while I carry out some manner of work, then I am notified when that work is completed, and I react to its results.
I do see that it has some built in asynchronous mechanisms, but none of them provide the user with a means to create custom CPU bound asynchronous tasks akin to 'Tasks' in C# or 'work' with boost::asio in C++ that would run in parallel to the main thread.
I see that Twisted provides a means to asynchronously wait on IO and do things in on the same thread while waiting, if we are waiting on:
network reads and writes
keyboard input
It also shows me how to:
Do some manner of integration with GUI tool kits to make use of their event loop, but doesn't go into detail.
Schedule tasks using reactor on a timer, but doesn't do that task in parallel to anything else
It talks about async/await, but that is for python 3 only, and I am using python 2.7
I figured the some manner of thread pooling must be built into the reactor, but then when I read about the reactor, it says that everything runs on the main thread in reactor.run().
So, I am left confused.
What is the point of deferreds, creating a callback chain and reacting to the results, if we aren't running anything in parallel?
If we are running asynchronous code, how are we making our own custom asynchronous functions? (see keyword async in C#)
In other languages, I might create an async task to count from 1 to 10, while on the main thread, I might count from 'a' to 'z' at the same time. When the the task is complete I would get notified via a callback on a thread from a threadpool. I'd have the option to sync up, if I wanted to, by calling some 'wait' method. While the definition of "asynchronous" only involves the posting of the task, the getting of the result, and the callback when its done....I've never seen it used without doing things in parallel.
I'll address your questions (and statements that seem confusing) one-by-one:
"Examples that are not complete"
Restating what I posted in the comments: see my two previous answers for complete examples ( https://stackoverflow.com/a/30399317/3334178 & https://stackoverflow.com/a/23274411/3334178 ) and go through Krondo's Twisted Introduction
You said you are discounting these because "The examples are the network code in twisted, which has the asynchronisity built in and hidden.". I disagree with that assertion and will explain this in the next section.
"Examples are not asynchronous"
When your talking about "asynchronous programming" in the vain of pythons twisted/tornado/asyncio (or Node.JS or C select/poll/kpoll) your talking about model/pattern of programming that allows the programmer shape their code so that parts of it can run while other parts are blocked (in almost all cases the blocking is caused by a part of the program having to wait for IO).
These libraries/languages will certainly have ways they can do threading and/or multiprocessing, but those are layers grafted on top of the async design - and if that's genuinely what you need (I.E. you have an exclusively CPU bound need) the async systems are going to be a bad choice.
Let's use your "hidden away" comment to get into this a bit more
"Network examples are asych, but the asynchronousity is built in and hidden away"
The fundamental element of the async design is that you write your code so it should never block for IO - You've been calling out network but really we are talking about network/disk/keyboard/mouse/sound/serial - anything that (for whatever reason) can run slower than the CPU (and that the OS has a file-descriptor for).
Also, there isn't anything really "hidden away" about how it functions - async programming always uses non-blocking (status checking / call-back) calls for any IO channel it can operate on. If you dig enough in the twisted codebase all the async logic is in plain sight (Krondo's tutorial is really good for giving examples of this)
Let me use the keyboard as an example.
In sync code, you would use an input or a read - and the program would pause waiting for that line (or key) to be typed.
In async code (at least in featureful implementations like twisted) you will fetch the file-descriptor for "input" and register it with call-back function, to be called when the file-descriptor changes, to the OS-level async engine (select, poll, kpoll, etc...)
The act of doing that registration - which takes almost no time LETS YOU run other logic while the keyboard logic waits for the keyboard event to happen (see the stdio.StandardIO(keyboardobj,sys.stdin.fileno()) line from near the end of my example code in https://stackoverflow.com/a/30399317/3334178).
"[This] leads me to believe there is some other means to use deferreds with asynchronous"
deferreds aren't magic. They are just clever lists of function callback. There are numerous clever ways they can be chained together, but in the end, they are just a tool to help you take advantage of the logic above
"It also talks about async/await, that is for python 3 only, and I am using python 2.7"
async and await are just the python 3 way of doing what was done in python2 with #defer.inlineCallbacks and yield. These systems are shortcuts that rewire code so that to the reader the code looks and acts like sync code, but when its run the code is morphed into a "register a callback and move-on" flow
"when I read about the reactor, it says that everything runs on the main thread in reactor.run()"
Yes, because (as above) async is about not-waiting-for-IO - its not about threading or multi-processing
Your last few questions "point of deferreds" and "how do you make asynchronous" feel like I answered them above - but if not, let me know in the comments, and I'll spell them out.
Also your comment requesting "an example where we count from 1 to 10 in some deferred function while we count from a to z in the main thread?" doesn't make sense when talking about async (both because you talk about a "thread" - which is a different construct, and because those are both (likely) CPU tasks), but I will give you a different example that counts while watching for keyboard input (which is something that definitely DOES make sense when talking about async:
#!/usr/bin/env python
#
# Frankenstein-esk amalgam of example code
# Key of which comes from the Twisted "Chat" example
# (such as: http://twistedmatrix.com/documents/12.0.0/core/examples/chatserver.py)
import sys # so I can get at stdin
import os # for isatty
import termios, tty # access to posix IO settings
from twisted.internet import reactor
from twisted.internet import stdio # the stdio equiv of listenXXX
from twisted.protocols import basic # for lineReceiver for keyboard
from twisted.internet import task
class counter(object):
runs = 0
def runEverySecond():
counter.runs += 1
print "async counting demo: " + str(counter.runs)
# to set keyboard into cbreak mode - so keys don't require a CR before causing an event
class Cbreaktty(object):
org_termio = None
my_termio = None
def __init__(self, ttyfd):
if(os.isatty(ttyfd)):
self.org_termio = (ttyfd, termios.tcgetattr(ttyfd))
tty.setcbreak(ttyfd)
print ' Set cbreak mode'
self.my_termio = (ttyfd, termios.tcgetattr(ttyfd))
else:
raise IOError #Not something I can set cbreak on!
def retToOrgState(self):
(tty, org) = self.org_termio
print ' Restoring terminal settings'
termios.tcsetattr(tty, termios.TCSANOW, org)
class KeyEater(basic.LineReceiver):
def __init__(self):
self.setRawMode() # Switch from line mode to "however much I got" mode
def rawDataReceived(self, data):
key = str(data).lower()[0]
if key == 'q':
reactor.stop()
else:
print "--------------"
print "Press:"
print " q - to cleanly shutdown"
print "---------------"
# Custom tailored example for SO:56013998
#
# This code is a mishmash of styles and techniques. Both to provide different examples of how
# something can be done and because I'm lazy. Its been built and tested on OSX and linux,
# it should be portable (other then perhaps termal cbreak mode). If you want to ask
# questions about this code contact me directly via mail to mike at partialmesh.com
#
#
# Once running press any key in the window where the script was run and it will give
# instructions.
def main():
try:
termstate = Cbreaktty(sys.stdin.fileno())
except IOError:
sys.stderr.write("Error: " + sys.argv[0] + " only for use on interactive ttys\n")
sys.exit(1)
keyboardobj = KeyEater()
l = task.LoopingCall(runEverySecond)
l.start(1.0) # call every second
stdio.StandardIO(keyboardobj,sys.stdin.fileno())
reactor.run()
termstate.retToOrgState()
if __name__ == '__main__':
main()
(I know technically I didn't use a deferred - but I ran out of time - and this case is a bit too simple to really need it (I don't have a chain of callback anywhere, which is what deferreds are for))
I am downloading pictures from the internet, and as it turns out, I need to download lots of pictures. I am using a version of the following code fragment (actually looping through the links I intend to download and downloading the pictures :
import urllib
urllib.urlretrieve(link, filename)
I am downloading roughly 1000 pictures every 15 minutes, which is awfully slow based on the number of pictures I need to download.
For efficiency, I set a timeout every 5 seconds (still many downloads last much longer):
import socket
socket.setdefaulttimeout(5)
Besides running a job on a computer cluster to parallelize downloads, is there a way to make the picture download faster / more efficient?
my code above was very naive as I did not take advantage of multi-threading. It obviously takes for url requests to be responded but there is no reason why the computer cannot make further requests while the proxy server responds.
Doing the following adjustments, you can improve efficiency by 10x - and there are further ways for improving efficiency, with packages such as scrapy.
To add multi-threading, do something like the following, using the multiprocessing package:
1) encapsulate the url retrieving in a function:
import import urllib.request
def geturl(link,i):
try:
urllib.request.urlretrieve(link, str(i)+".jpg")
except:
pass
2) then create a collection with all urls as well as names you want for the downloaded pictures:
urls = [url1,url2,url3,urln]
names = [i for i in range(0,len(urls))]
3)Import the Pool class from the multiprocessing package and create an object using such class (obviously you would include all imports in the first line of your code in a real program):
from multiprocessing.dummy import Pool as ThreadPool
pool = ThreadPool(100)
then use the pool.starmap() method and pass the function and the arguments of the function.
results = pool.starmap(geturl, zip(links, d))
note: pool.starmap() works only in Python 3
When a program enters I/O wait, the execution is paused so that the kernel can perform the low-level operations associated with the I/O request (this is called a context switch) and is not resumed until the I/O operation is completed.
Context switching is quite a heavy operation. It requires us to save the state of our program (losing any sort of caching we had at the CPU level) and give up the use of the CPU. Later, when we are allowed to run again, we must spend time reinitializing our program on the motherboard and getting ready to resume (of course, all this happens behind the scenes).
With concurrency, on the other hand, we typically have a thing called an “event loop” running that manages what gets to run in our program, and when. In essence, an event loop is simply a list of functions that need to be run. The function at the top of the list gets run, then the next, etc.
The following shows a simple example of an event loop:
from Queue import Queue
from functools import partial
eventloop = None
class EventLoop(Queue):
def start(self):
while True:
function = self.get()
function()
def do_hello():
global eventloop
print "Hello"
eventloop.put(do_world)
def do_world():
global eventloop
print "world"
eventloop.put(do_hello)
if __name__ == "__main__":
eventloop = EventLoop()
eventloop.put(do_hello)
eventloop.start()
If the above seems like something you may use, and you'd also like to see how gevent, tornado, and AsyncIO, can help with your issue, then head out to your (University) library, check out High Performance Python by Micha Gorelick and Ian Ozsvald, and read pp. 181-202.
Note: above code and text are from the book mentioned.
I'm trying to write a simple Python IRC client. So far I can read data, and I can send data back to the client if it automated. I'm getting the data in a while True, which means that I cannot enter text while at the same time reading data. How can I enter text in the console, that only gets sent when I press enter, while at the same time running an infinite loop?
Basic code structure:
while True:
read data
#here is where I want to write data only if it contains '/r' in it
Another way to do it involves threads.
import threading
# define a thread which takes input
class InputThread(threading.Thread):
def __init__(self):
super(InputThread, self).__init__()
self.daemon = True
self.last_user_input = None
def run(self):
while True:
self.last_user_input = input('input something: ')
# do something based on the user input here
# alternatively, let main do something with
# self.last_user_input
# main
it = InputThread()
it.start()
while True:
# do something
# do something with it.last_user_input if you feel like it
What you need is an event loop of some kind.
In Python you have a few options to do that, pick one you like:
Twisted https://twistedmatrix.com/trac/
Asyncio https://docs.python.org/3/library/asyncio.html
gevent http://www.gevent.org/
and so on, there are hundreds of frameworks for this, you could also use any of the GUI frameworks like tkinter or PyQt to get a main event loop.
As comments have said above, you can use threads and a few queues to handle this, or an event based loop, or coroutines or a bunch of other architectures. Depending on your target platforms one or the other might be best. For example on windows the console API is totally different to unix ptys. Especially if you later need stuff like colour output and so on, you might want to ask more specific questions.
You can use a async library (see answer of schlenk) or use https://docs.python.org/2/library/select.html
This module provides access to the select() and poll() functions
available in most operating systems, epoll() available on Linux 2.5+
and kqueue() available on most BSD. Note that on Windows, it only
works for sockets; on other operating systems, it also works for other
file types (in particular, on Unix, it works on pipes). It cannot be
used on regular files to determine whether a file has grown since it
was last read.
I admit to being very lazy: I need to do this fairly quickly and cannot get my head round the Python3 asyncio module. (Funnily, I found the boost one fairly intuitive.)
I need to readline a file object (a pipe) that will block from time to time. During this, I want to be able fire off another activity at set intervals (say every 30 minutes), regardless of the availability of anything to read from the file.
Can anyone help me with a skeleton to do this using python3 asyncio? (I cannot install a third-party module such as twisted.)
asyncio (as well as other asynchronous libraries like twisted and tornado) doesn't support non-blocking IO for files -- only sockets and pipes are processed asynchronously.
The main reason is: Unix systems have no good way to process files. Say, on Linux any file read is blocking operation.
See also https://groups.google.com/forum/#!topic/python-tulip/MvpkQeetWZA
UPD.
For schedule periodic activity I guess to use asyncio.Task:
#asyncio.coroutine
def periodic(reader, delay):
data = yield from reader.read_exactly(100) # read 100 bytes
yield from asyncio.sleep(delay)
task = asyncio.Task(reader, 30*60)
Snippet assumes reader is asyncio.StreamReader instance.