Develop Reference

How to collect the responses from aiohttp sessions

I'm working with asyncio and aiohttp to call an API many times. While can print the responses, I want to collate the responses into a combined structure - a list or pandas dataframe etc.
In my example code I'm connecting to 2 urls and printing a chunk of the response. How can I collate the responses and access them all?
import asyncio, aiohttp
async def get_url(session, url, timeout=300):
async with session.get(url, timeout=timeout) as response:
http = await response.text()
print(str(http[:80])+'\n')
return http # becomes a list item when gathered
async def async_payload_wrapper(async_loop):
# test with 2 urls as PoC
urls = ['https://google.com','https://yahoo.com']
async with aiohttp.ClientSession(loop=async_loop) as session:
urls_to_check = [get_url(session, url) for url in urls]
await asyncio.gather(*urls_to_check)
if __name__ == '__main__':
event_loop = asyncio.get_event_loop()
event_loop.run_until_complete(async_payload_wrapper(event_loop))
I've tried printing to a file, and that works but it's slow and I need to read it again for further processing. I've tried appending to a global variable without success. E.g. using a variable inside get_url that is defined outside it generates an error, eg:
NameError: name 'my_list' is not defined or
UnboundLocalError: local variable 'my_list' referenced before assignment

Thanks #python_user that's exactly what I was missing and the returned type is indeed a simple list. I think I'd tried to pick up the responses inside the await part which doesn't work.
My updated PoC code below.
Adapting this for the API, JSON and pandas should now be easy : )
import asyncio, aiohttp
async def get_url(session, url, timeout=300):
async with session.get(url, timeout=timeout) as response:
http = await response.text()
return http[:80] # becomes a list element
async def async_payload_wrapper(async_loop):
# test with 2 urls as PoC
urls = ['https://google.com','https://yahoo.com']
async with aiohttp.ClientSession(loop=async_loop) as session:
urls_to_check = [get_url(session, url) for url in urls]
responses = await asyncio.gather(*urls_to_check)
print(type(responses))
print(responses)
if __name__ == '__main__':
event_loop = asyncio.get_event_loop()
event_loop.run_until_complete(async_payload_wrapper(event_loop))

How to connect to User Data Stream Binance?

I need to listen to User Data Stream, whenever there's an Order Event - order execution, cancelation, and so on - I'd like to be able to listen to those events and create notifications.
So I got my "listenKey" and I'm not sure if it was done the right way but I executed this code and it gave me something like listenKey.
Code to get listenKey:
def get_listen_key_by_REST(binance_api_key):
url = 'https://api.binance.com/api/v1/userDataStream'
response = requests.post(url, headers={'X-MBX-APIKEY': binance_api_key})
json = response.json()
return json['listenKey']
print(get_listen_key_by_REST(API_KEY))
And the code to listen to User Data Stream - which doesn't work, I get no json response.
socket = f"wss://fstream-auth.binance.com/ws/btcusdt#markPrice?listenKey=<listenKeyhere>"
def on_message(ws, message):
json_message = json.loads(message)
print(json_message)
def on_close(ws):
print(f"Connection Closed")
# restart()
def on_error(ws, error):
print(f"Error")
print(error)
ws = websocket.WebSocketApp(socket, on_message=on_message, on_close=on_close, on_error=on_error)
I have read the docs to no avail. I'd appreciate it if someone could point me in the right direction.

You can create a basic async user socket connection from the docs here along with other useful info for the Binance API. Here is a simple example:
import asyncio
from binance import AsyncClient, BinanceSocketManager
async def main():
client = await AsyncClient.create(api_key, api_secret, tld='us')
bm = BinanceSocketManager(client)
# start any sockets here, i.e a trade socket
ts = bm.user_socket()
# then start receiving messages
async with ts as tscm:
while True:
res = await tscm.recv()
print(res)
await client.close_connection()
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

I just figured this out myself and I was able to get mine to work so I'll try my best to guide you. I believe you're just missing this line of code after you create your WebSocket object:
ws.run_forever()
Some other reasons it might not be working; If you want to detect orders on your futures account then you need to use the futures endpoint. I think the one your using is for spot trading (Not sure).
url = 'https://fapi.binance.com'
and just in case it's not clear to you. You must replace:
<listenkeyhere>
in the socket url with your listen key, angle brackets, and all.

trying to reduce the time or make requests at the same time

Is there anyway I can make all this requests at the same time? I'm trying to reduce the time...
def pokemons():
for i in range(1, 800):
url = f"https://pokeapi.co/api/v2/pokemon/{i}"
requisicao = requests.get(url)
try:
lista = requisicao.json()
except ValueError:
print("ERRO TIPO")

What you want is AIOHTTP Asynchronous HTTP Client/Server.
import asyncio
import aiohttp
import ssl
async def fetch(session, pokemon_num):
url = f"https://pokeapi.co/api/v2/pokemon/{pokemon_num}"
async with session.get(url, ssl=ssl.SSLContext()) as response:
return await response.json()
async def fetch_all(loop):
async with aiohttp.ClientSession(loop=loop) as session:
results = await asyncio.gather(*[fetch(session, pokemon_n) for pokemon_n in range(800)], return_exceptions=True)
return results
if __name__ == '__main__':
loop = asyncio.get_event_loop()
result = loop.run_until_complete(fetch_all(loop))
print(result)
I ran this code and got all the requests in a total of 19.5 seconds. I hope its good for your case.
The snippet above comes from another answer from YuriiKramarenko, if it suits you, you can give him a thumbs up. I adjusted it for your specific parameters.
Good luck catching them all!

Using two PIP packages together that use event loops

I'm trying to combine a Twitch API package (twitchio) with a webserver (sanic) with the intent of serving chat commands to a game running locally to the python script. I don't have to use Sanic or twitchio but those are the best results I've found for my project.
I think I understand why what I have so far isn't working but I'm at a total loss of how to resolve the problem. Most of the answers I've found so far deal with scripts you've written to use asyncio and not packages that make use of event loops.
I can only get the webserver OR the chat bot to function. I can't get them to both run concurrently.
This is my first attempt at using Python so any guidance is greatly appreciated.
#!/usr/bin/python
import os
from twitchio.ext import commands
from sanic import Sanic
from sanic.response import json
app = Sanic(name='localserver')
bot = commands.Bot(
token=os.environ['TMI_TOKEN'],
client_id=os.environ['CLIENT_ID'],
nick=os.environ['BOT_NICK'],
prefix=os.environ['BOT_PREFIX'],
initial_channels=[os.environ['CHANNEL']]
)
#app.route("/")
async def query(request):
return json(comcache)
#bot.event
async def event_ready():
'Called once when the bot goes online.'
print(f"{os.environ['BOT_NICK']} is online!")
ws = bot._ws # this is only needed to send messages within event_ready
await ws.send_privmsg(os.environ['CHANNEL'], f"/me has landed!")
#bot.event
async def event_message(ctx):
'Runs every time a message is sent in chat.'
# make sure the bot ignores itself and the streamer
if ctx.author.name.lower() == os.environ['BOT_NICK'].lower():
return
await bot.handle_commands(ctx)
# await ctx.channel.send(ctx.content)
if 'hello' in ctx.content.lower():
await ctx.channel.send(f"Hi, #{ctx.author.name}!")
#bot.command(name='test')
async def test(ctx):
await ctx.send('test passed!')
if __name__ == "__main__":
app.run(host="127.0.0.1", port=8080)
bot.run()

Connection constantly closing with Python websockets

I've tried setting up a WebSocket server, and an API to access it via browser
I'm a week new into Python, and I hope this somewhat makes sense.
I have a Server class
**Server.py**
# WS server that sends messages at random intervals
import asyncio
import datetime
import random
import websockets
async def time(websocket, path):
while True:
now = datetime.datetime.utcnow().isoformat() + "Z"
await websocket.send(now)
await asyncio.sleep(random.random() * 3)
async def echo(websocket, path):
async for message in websocket:
await websocket.send(message)
async def router(websocket, path):
if path == "/get":
await time(websocket)
elif path == "/post":
await echo(websocket)
asyncio.get_event_loop().run_until_complete(
websockets.serve(router, "127.0.0.1", 8081, ping_interval=None))
asyncio.get_event_loop().run_forever()
and an API class
**API.py**
from quart import Quart
import json
import websockets
app = Quart(__name__)
#app.route('/get',methods=['GET'])
async def foo1():
uri = "ws://localhost:8081"
async with websockets.connect(uri) as websocket:
try:
receivedMessage = await websocket.recv()
print(f"< {receivedMessage}")
except:
print('Reconnecting')
websocket = await websockets.connect(uri)
return ' '
#app.route('/post', methods=['POST'])
async def foo2():
uri = "ws://localhost:8081"
async with websockets.connect(uri) as websocket:
await websocket.send("Hello world!")
try:
await websocket.recv()
except:
print('Reconnecting')
websocket = await websockets.connect(uri)
if __name__ == '__main__':
app.run()
My aim was to be able to access different functions from the Server, based on the API's Path which was being accessed (seen it here: https://github.com/aaugustin/websockets/issues/547)
The problem I'm getting is that "Reconnecting" appears constantly, and nothing actually happens when I make use of /get or /post (i.e. it seems to be closing after .recv() occurs - I know this because I had seen ConnectionClosedOk - no reason 1000 earlier.
I really don't understand what I'm doing right now lol, it is very confusing. I did get things to work when Server only had 1 function (constantly spitting out timestamps), and the API only had a GET function (to display the current time of the request).. things now don't really work after I implemented the router function.
Any help would be greatly, greatly appreciated. My Python knowledge is incredibly limited, so as verbose an answer as possible would really help me out.
I'm assuming that the POST method is going to result in "Hello world!" appearing on the console for the Server, but I'm probably wrong about that :( I just seen it in some tutorial documentation.
EDIT: Also, I'm not able to access localhost:(port)/post; it says Specified method is invalid for this resource; why is that?