I am trying to convert a Python HTTP client featuring requests to aiohttp. The logic is to send a GET call to a REST endpoint which streams data occasionally and print the lines it returns.
I have a code using requests with stream=True option and iter_lines, it works pretty fine:
import json
import requests
def main():
with requests.get('https://my-streaming-url.com', stream=True) as r:
if r.encoding is None:
r.encoding = 'utf-8'
for line in r.iter_lines(decode_unicode=True):
if line:
# Print each line emitted by the streaming api
print(json.loads(line))
if __name__ == '__main__':
main()
Now, I want to convert this logic to aiohttp streaming api and tried:
import asyncio
import aiohttp
import json
loop = asyncio.get_event_loop()
async def main():
r = aiohttp.request('get', 'https://my-streaming-url.com')
async for line in r.content:
print(json.loads(line))
if __name__ == '__main__':
loop.run_until_complete(connect_and_listen())
loop.close()
I get an error like:
... in connect_and_listen
async for line in r.content:
AttributeError: '_SessionRequestContextManager' object has no attribute 'content'
sys:1: RuntimeWarning: coroutine 'ClientSession._request' was never awaited
Unclosed client session
client_session: aiohttp.client.ClientSession object at 0x7fac6ec24310
I tried a few ways like removing loop.close() from main, removing async from the for loop, but none helped.
What am I missing here? How can I print a streaming api lines with aiohttp?
P.S: My Python version is 3.7.5
As throughout the documentation usage of ClientSession class is encouraged, I had this code also encapsulated a session like follows and it worked:
async def main():
async with aiohttp.ClientSession(raise_for_status=True) as session:
async with session.get(cashcog_stream_url) as r:
async for line in r.content:
Another point is loop.close() is apparently does not affect the way app works and can be removed.
Your missing the await keyword.
aiohttp.request is an async context manager. you should use it with an async with statement
async def main():
async with aiohttp.request('get', 'http://localhost:5000') as r:
async for line in r.content:
print(json.loads(line))
Related
I'm working with asyncio and aiohttp to call an API many times. While can print the responses, I want to collate the responses into a combined structure - a list or pandas dataframe etc.
In my example code I'm connecting to 2 urls and printing a chunk of the response. How can I collate the responses and access them all?
import asyncio, aiohttp
async def get_url(session, url, timeout=300):
async with session.get(url, timeout=timeout) as response:
http = await response.text()
print(str(http[:80])+'\n')
return http # becomes a list item when gathered
async def async_payload_wrapper(async_loop):
# test with 2 urls as PoC
urls = ['https://google.com','https://yahoo.com']
async with aiohttp.ClientSession(loop=async_loop) as session:
urls_to_check = [get_url(session, url) for url in urls]
await asyncio.gather(*urls_to_check)
if __name__ == '__main__':
event_loop = asyncio.get_event_loop()
event_loop.run_until_complete(async_payload_wrapper(event_loop))
I've tried printing to a file, and that works but it's slow and I need to read it again for further processing. I've tried appending to a global variable without success. E.g. using a variable inside get_url that is defined outside it generates an error, eg:
NameError: name 'my_list' is not defined or
UnboundLocalError: local variable 'my_list' referenced before assignment
Thanks #python_user that's exactly what I was missing and the returned type is indeed a simple list. I think I'd tried to pick up the responses inside the await part which doesn't work.
My updated PoC code below.
Adapting this for the API, JSON and pandas should now be easy : )
import asyncio, aiohttp
async def get_url(session, url, timeout=300):
async with session.get(url, timeout=timeout) as response:
http = await response.text()
return http[:80] # becomes a list element
async def async_payload_wrapper(async_loop):
# test with 2 urls as PoC
urls = ['https://google.com','https://yahoo.com']
async with aiohttp.ClientSession(loop=async_loop) as session:
urls_to_check = [get_url(session, url) for url in urls]
responses = await asyncio.gather(*urls_to_check)
print(type(responses))
print(responses)
if __name__ == '__main__':
event_loop = asyncio.get_event_loop()
event_loop.run_until_complete(async_payload_wrapper(event_loop))
I'm trying to mock a websockets data stream and I'm getting this error: 'async_generator' object is not an iterator
This is my generator code:
from time import sleep
mock_sf_record = '{"payload": ...}'
async def generateMessages():
sleep(5)
yield mock_sf_record
and the code that calls this code:
async def subscribe(subscription):
global RECEIVED_MESSAGES_CACHE
...
while True:
messageStream = await(next(generateMessages())) if ENV == 'dev' else await websocket.recv()
What can I do? What am I doing wrong? I'm basically using the generateMessages() generator to create a stream of messages, but this isn't working...
The code that is calling subscribe:
for subscription in SUBSCRIPTION_TYPES:
loop.create_task(subscribe(subscription))
loop.run_forever()
More importantly, if I change the code to use a synchronous generator, this only generates messages for a single subscription and I never seem to generate messsages for any other subscription... it seems to block on a single thread. Why is this?
messageStream = (next(generateMessages())) if ENV == 'dev' else await websocket.recv()
and
# generator that generates mock SF data
from asyncio import sleep
mock_sf_record = '{"payload": ...}'
def generateMessages():
sleep(5)
yield mock_sf_record
Why does the synchronous generator cause problems?
The right way:
async def subscribe(subscription):
global RECEIVED_MESSAGES_CACHE
...
gen = generateMessages() # init async generator
messageStream = (await gen.__anext__()) if ENV == 'dev' else (await websocket.recv())
https://www.python.org/dev/peps/pep-0525/#support-for-asynchronous-iteration-protocol
I am setting up a lambda function which makes asynchronous requests using asyncio and aiohttp. Even thought the code works fine while running locally, once I upload it to lambda it returns:
"errorMessage": "A Future or coroutine is required"
I've looked at already opened issues such as Boto3: A Future or coroutine is required and TypeError: A Future or coroutine is required but couldn't make it to work.
I am not sure why is returning the error message while I have a defined coroutine
base_url = "https://www.reed.co.uk/api/1.0/jobs/"
url_list = [
"38012438",
"38012437",
"38012436"]
def lambda_handler(event, context):
client = boto3.client('s3',
aws_access_key_id="aws_access_key_id",
aws_secret_access_key="aws_secret_access_key")
async def fetch(session, url):
auth = aiohttp.BasicAuth(login='login', password='')
async with aiohttp.ClientSession(auth=auth) as session:
async with session.get(url) as response:
return await response.json()
async def fetch_all(urls, loop):
async with aiohttp.ClientSession(loop=loop) as session:
results = await asyncio.gather(*[fetch(session, base_url + url) for url in url_list], return_exceptions=True)
return results
loop = asyncio.get_event_loop()
htmls = loop.run_until_complete(fetch_all(urls, loop))
#I believe this is where the coroutine is
with open("/tmp/JOBS.json", "w") as f:
json.dump(htmls, f)
I just want the combined content of my requests to be uploaded to a json file.
I apologise for my limited coding skills as I am new to python, lambda and etc.
Check your requirements.txt file. I got the same error when I added asyncio to the requirements.txt. It seems like AWS uses a special version of python. Asyncio is a part of python. And if you install it separately then it doesn't work in AWS Lambda.
I need to make asynchronous requests using the Requests library. In Python 3.7 if I try from requests import async I get SyntaxError: invalid syntax.
async has become a reserved with in Python 3.7. How to I get around this situation?
Lukasa who is with the requests lib said:
At the current time there are no plans to support async and await. This is not because they aren't a good idea: they are. It's because to use them requires quite substantial code changes.
Right now requests is a purely synchronous library that, at the bottom of its stack, uses httplib to send and receive data. We cannot move to an async model unless we replace httplib. The best we could do is provide a shorthand to run a request in a thread, but asyncio already has just such a shorthand, so I don't believe it would be valuable.
Right now I am quietly looking at whether we can rewrite requests to work just as well in a synchronous environment as in an async one. However, the reality is that doing so will be a lot of work, involving rewriting a lot of our stack, and may not happen for many years, if ever.
But don't worry aiohttp is very similar to requests.
Here's an example.
import aiohttp
import asyncio
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
async def main():
async with aiohttp.ClientSession() as session:
html = await fetch(session, 'http://python.org')
print(html)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
You can use asyncio to make asynchronous requests. Here is an example:
import asyncio
import requests
async def main():
loop = asyncio.get_event_loop()
futures = [
loop.run_in_executor(
None,
requests.get,
'http://example.org/'
)
for i in range(20)
]
for response in await asyncio.gather(*futures):
pass
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
i have a project on python 2.7. i need to make async post call to connect aws. i have an code for async in python 3.5 by using asyncio.but my code needs to work on python2.7 2.7.please guide me how to resolve this issue.
import asyncio
import json
from aiohttp import ClientSession
HEADERS = {'Content-type':'application/json'}
async def hello(url):
data = {"mac": 'mm','minor':3,'distance':1,'timestamp':4444,'uuid':'aa','rssi':1,'tx':34}
async with ClientSession() as session:
async with session.post(url,json=data) as response:
response = await response.read()
print(response)
loop = asyncio.get_event_loop()
url = "http://192.168.101.74:9090/api/postreader"
while True:
loop.run_until_complete(hello(url))
Try using gevent instead of asyncio?
http://www.gevent.org/
https://pypi.org/project/gevent/#downloads