I'm writing an app using Tornado. I need to make a lot of HTTP requests, but Tornado's HTTP client sucks a bit (doesn't have Keep-Alive support and is slow), so I'm trying to use Pulsar HttpClient:
import tornado.web
import tornado.gen
import tornado.httpserver
from tornado.platform.asyncio import AsyncIOMainLoop
from tornado.platform import asyncio as tornasync
import asyncio
from pulsar.apps import http as pulsar_http
class MyHandler(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
http_client = self.application.http_client
future = tornasync.to_tornado_future(asyncio.async(http_client.request('GET', 'http://httpbin.org', timeout=.25)))
try:
result = yield future
except TimeoutError as e:
print('Timeout!')
print(result.get_content())
self.write('OK')
self.finish()
if __name__ == '__main__':
AsyncIOMainLoop().install()
app = tornado.web.Application([(r'/', MyHandler)], debug=False)
server = tornado.httpserver.HTTPServer(app)
server.bind(8888)
server.start(1)
app.http_client = pulsar_http.HttpClient(loop=asyncio.get_event_loop())
asyncio.get_event_loop().run_forever()
Bun when a timeout occurs, I get an exception:
Traceback (most recent call last):
File "/home/vitaliy/.virtualenvs/tornado/local/lib/python3.4/site-packages/tornado/web.py", line 1415, in _execute
result = yield result
File "/home/vitaliy/.virtualenvs/tornado/local/lib/python3.4/site-packages/tornado/gen.py", line 870, in run
value = future.result()
File "/home/vitaliy/.virtualenvs/tornado/local/lib/python3.4/site-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
File "/home/vitaliy/.virtualenvs/tornado/local/lib/python3.4/site-packages/tornado/gen.py", line 876, in run
yielded = self.gen.throw(*exc_info)
File "bpp.py", line 19, in get
result = yield future
File "/home/vitaliy/.virtualenvs/tornado/local/lib/python3.4/site-packages/tornado/gen.py", line 870, in run
value = future.result()
File "/home/vitaliy/.virtualenvs/tornado/local/lib/python3.4/site-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
File "/usr/lib/python3.4/asyncio/tasks.py", line 300, in _step
result = coro.send(value)
File "/usr/lib/python3.4/asyncio/tasks.py", line 436, in wait_for
raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
Can I catch this exception somehow?
Just import that error into your code:
from concurrent.futures import TimeoutError
Otherwise you can't catch it
Related
I am on Linux/Python3.8.5
python3 -m pip list | grep gcsfs
gcsfs 2021.4.0
I looked at the docs, specifically chapter 5 - Async: https://buildmedia.readthedocs.org/media/pdf/gcsfs/stable/gcsfs.pdf
I also found an example from here: https://github.com/dask/gcsfs/issues/285, which is shown below:
import asyncio
import gcsfs
async def main():
loop = asyncio.get_event_loop()
fs = gcsfs.GCSFileSystem(project="my_project", asynchronous=True, loop=loop)
await fs.set_session()
async with await fs.open("my_bucket/my_blob") as fp:
b = await fp.read()
print(len(b))
asyncio.get_event_loop().run_until_complete(main())
The error is:
Traceback (most recent call last):
File "test_gcsfs_async.py", line 13, in <module>
asyncio.get_event_loop().run_until_complete(main())
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "test_gcsfs_async.py", line 7, in main
await fs.set_session()
AttributeError: 'GCSFileSystem' object has no attribute 'set_session'
If I simply remove the call to set_session(), then I get this error:
Traceback (most recent call last):
File "test_gcsfs_async.py", line 13, in <module>
asyncio.get_event_loop().run_until_complete(main())
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "test_gcsfs_async.py", line 9, in main
async with await fs.open("my_bucket/my_blob") as fp:
File "/usr/local/lib/python3.8/dist-packages/fsspec/spec.py", line 942, in open
f = self._open(
File "/usr/local/lib/python3.8/dist-packages/gcsfs/core.py", line 1247, in _open
return GCSFile(
File "/usr/local/lib/python3.8/dist-packages/gcsfs/core.py", line 1378, in __init__
super().__init__(
File "/usr/local/lib/python3.8/dist-packages/fsspec/spec.py", line 1270, in __init__
self.details = fs.info(path)
File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 72, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 40, in sync
raise RuntimeError("Loop is not running")
RuntimeError: Loop is not running
Unfortunately, the file-like interface of GCSFile is not async-compatible. From the issue you linked:
The file-like interface itself is not asynchronous, since there is state (the current buffers and file positions).
This may be implemented in the future, but would require a certain amount of work.
Note that the coroutine method you were after is called _set_session (with the leading underscore). This should be better documented - feel free to raise an issue or submit a PR.
I am writing my API using aiohttp, asycpg and asyncpgsa.
I create my application:
async def create_app():
app = web.Application(client_max_size=1024 ** 2 * 70)
Then I execute these lines:
async def on_start(app):
app['db'] = asyncpgsa.create_pool(dsn="postgresql://127.0.0.1:5432/backend")
async def on_shutdown(app):
app['db'].close()
app.on_startup.append(on_start)
app.on_cleanup.append(on_shutdown)
In general, in the example where I got it from, it is written like this:
app['db'] = await asyncpgsa.create_pool(dsn="postgresql://127.0.0.1:5432/backend")
But if I write like this, then an error is thrown "ConnectionRefusedError: [Errno 10061] Connect call failed ('127.0.0.1', 5432)"
But that's okay. Now, when the user visits the URL I need, this function should be triggered:
async def post(request):
async with request.app["db"].acquire() as conn:
query = select([datab.post])
result = await conn.fetch(query)
The datab file says this:
from sqlalchemy import Table, Text, VARCHAR, Integer, MetaData, Column
meta = MetaData()
post = Table(
"post", meta,
Column("id", Integer, primary_key=True),
Column("title", VARCHAR, nullable=True),
Column("body", Text))
But when I go to the URL I want, the site gives me "500 Internal Server Error Server got itself in trouble "
And Pycharm: Error handling request asyncpg.exceptions._base.InterfaceError: pool is not initialized
Very little has been written on the Internet about asyncpg (sa), so I will be immensely grateful if you can help me fix the problem.
I'll add my own code.
main.py
from aiohttp import web
from demo import create_app
import argparse
import sqlalchemy
import asyncpgsa
async def post_handler(request):
body1 = await request.json()
print(body1)
return web.json_response(data=body1, status=201)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--host", help="IPv4/IPv6 address API server would listen on", default="0.0.0.0")
parser.add_argument('--port', help='TCP port API server would listen on', default=8080, type=int)
args = parser.parse_args()
app = create_app()
web.run_app(app, host=args.host, port=args.port)
if __name__ == '__main__':
main()
app.py
from aiohttp import web
import asyncpgsa
from .routes import setup_routes
async def on_start(app):
app['db'] = asyncpgsa.create_pool(dsn="postgresql://127.0.0.1:5432/backendyandex")
async def on_shutdown(app):
app['db'].close()
async def create_app():
app = web.Application(client_max_size=1024 ** 2 * 70)
setup_routes(app)
app.on_startup.append(on_start)
app.on_cleanup.append(on_shutdown)
return app
If you write
app['db'] = await asyncpgsa.create_pool(dsn="postgresql://127.0.0.1:5432/backend")
Then an error occurs
unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<_run_app() done, defined at C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\web.py:287> exception=ConnectionRefusedError(10061, "Connect call failed ('127.0.0.1', 5432)")>
Traceback (most recent call last):
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\web.py", line 508, in run_app
loop.run_until_complete(main_task)
File "C:\Users\lisgl\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 587, in run_until_complete
return future.result()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\web.py", line 319, in _run_app
await runner.setup()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\web_runner.py", line 275, in setup
self._server = await self._make_server()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\web_runner.py", line 375, in _make_server
await self._app.startup()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\web_app.py", line 416, in startup
await self.on_startup.send(self)
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\signals.py", line 34, in send
await receiver(*args, **kwargs) # type: ignore
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\demo\app.py", line 11, in on_start
app['db'] = await asyncpgsa.create_pool(dsn="postgresql://127.0.0.1:5432/backendyandex")
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\pool.py", line 407, in _async__init__
await self._initialize()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\pool.py", line 435, in _initialize
await first_ch.connect()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\pool.py", line 127, in connect
self._con = await self._pool._get_new_connection()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\pool.py", line 482, in _get_new_connection
**self._connect_kwargs)
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\connection.py", line 1997, in connect
max_cacheable_statement_size=max_cacheable_statement_size,
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\connect_utils.py", line 677, in _connect
raise last_error
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\connect_utils.py", line 668, in _connect
record_class=record_class,
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\connect_utils.py", line 634, in _connect_addr
tr, pr = await compat.wait_for(connector, timeout=timeout)
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\compat.py", line 103, in wait_for
return await asyncio.wait_for(fut, timeout)
File "C:\Users\lisgl\AppData\Local\Programs\Python\Python37\lib\asyncio\tasks.py", line 442, in wait_for
return fut.result()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\connect_utils.py", line 547, in _create_ssl_connection
host, port)
File "C:\Users\lisgl\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 962, in create_connection
raise exceptions[0]
File "C:\Users\lisgl\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 949, in create_connection
await self.sock_connect(sock, address)
File "C:\Users\lisgl\AppData\Local\Programs\Python\Python37\lib\asyncio\selector_events.py", line 473, in sock_connect
return await fut
File "C:\Users\lisgl\AppData\Local\Programs\Python\Python37\lib\asyncio\selector_events.py", line 503, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 10061] Connect call failed ('127.0.0.1', 5432)
site.py
from aiohttp import web
from sqlalchemy import select
from . import datab
async def post(request):
async with request.app["db"].acquire() as conn:
query = select([datab.post])
result = await conn.fetch(query)
return web.Response(body=str(result))
And here is the error that crashes if you go to the URL I need:
Error handling request
Traceback (most recent call last):
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\web_protocol.py", line 422, in _handle_request
resp = await self._request_handler(request)
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\aiohttp\web_app.py", line 499, in _handle
resp = await handler(request)
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\demo\site.py", line 13, in post
async with request.app["db"].acquire() as conn:
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\pool.py", line 785, in __aenter__
self.connection = await self.pool._acquire(self.timeout)
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\pool.py", line 622, in _acquire
self._check_init()
File "C:\Users\lisgl\Desktop\PycharmProjects\BackendYandex\venv\lib\site-packages\asyncpg\pool.py", line 745, in _check_init
raise exceptions.InterfaceError('pool is not initialized')
asyncpg.exceptions._base.InterfaceError: pool is not initialized
Your dsn is incorrect, so you can't connect to your database.
postgres://user:pass#host:port/database?option=value is the correct format. You forgot the user and the password.
Also asyncpgsa.create_pool() should be awaited, if not. You don't get the error, because you only assign a coroutine to the variable app['db']. So the connection pool is also not created.
Your second error (from site.py) is caused by not initizalising the connection pool.
More about that you can find in the documentation of asyncpg here (because asyncpgsa's connection pool is based on asyncpg's connection pool).
I have a Flask application, getting this error while trying to integrate flask with faust.
app.py
import mode.loop.eventlet
import logging
import logging.config
import json
from flask import Flask
from elasticapm.contrib.flask import ElasticAPM
def create_app():
app = Flask(__name__)
configure_apm(app)
configure_logging()
register_blueprints(app)
register_commands(app)
return app
main.py
from flask import jsonify
from litmus.app import create_app
from intercepter import Intercepter
app = create_app()
app.wsgi_app = Intercepter(app.wsgi_app , app)
#app.route('/status')
def status():
return jsonify({'status': 'online'}), 200
another controller
#api_blue_print.route('/v1/analyse', methods=['POST'])
def analyse():
analyse_with_historic_data.send(value=[somedata])
return jsonify({'message': 'Enqueued'}), 201
analyse_with_historic_data.py
#app.agent(analysis_topic)
async def analyse_with_historic_data(self, stream):
async for op in stream:
entity_log = EntityLog.where('id', op.entity_log_id).first()
Error Trace:
Traceback (most recent call last):
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
timer()
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
cb(*args, **kw)
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
waiter.switch()
greenlet.error: cannot switch to a different thread
Traceback (most recent call last):
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
timer()
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
cb(*args, **kw)
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
waiter.switch()
greenlet.error: cannot switch to a different thread
Traceback (most recent call last):
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/queue.py", line 118, in switch
self.greenlet.switch(value)
greenlet.error: cannot switch to a different thread
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/threading.py", line 551, in wait
signaled = self._cond.wait(timeout)
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/threading.py", line 299, in wait
gotit = waiter.acquire(True, timeout)
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/semaphore.py", line 107, in acquire
hubs.get_hub().switch()
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
return self.greenlet.switch()
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 350, in run
self.wait(sleep_time)
File "/Users/sahilpaudel/.pyenv/versions/3.6.5/lib/python3.6/site-packages/eventlet/hubs/kqueue.py", line 96, in wait
time.sleep(seconds)
I have trying to fix this issue by monkey.patch_all but that too it didn't work out giving another stacktrace that lock cannot be released something.
Something similar happened to me when I tried to debug a flask application using Pycharm.
What I finally did to eventually solve my issue was to enable gevent compatibility in Pycharm:
File -> settings -> Build,Execution,Deployment -> Python debugger -> Gevent compatible
It's a weird error since when I try/catch it, it prints nothings.
I'm using sanic server to asyncio.gather a bunch of images concurrently, more than 3 thousand images.
I haven't got this error when dealing with a smaller sample size.
Simplified example :
from sanic import Sanic
from sanic import response
from aiohttp import ClientSession
from asyncio import gather
app = Sanic()
#app.listener('before_server_start')
async def init(app, loop):
app.session = ClientSession(loop=loop)
#app.route('/test')
async def test(request):
data_tasks = []
#The error only happened when a large amount of images were used
for imageURL in request.json['images']:
data_tasks.append(getRaw(imageURL))
await gather(*data_tasks)
return response.text('done')
async def getRaw(url):
async with app.session.get(url) as resp:
return await resp.read()
What could this error be? If it is some kind of limitation of my host/internet, how can I avoid it?
I'm using a basic droplet from DigitalOcean with 1vCPU and 1GB RAM if that helps
Full stack error :
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/sanic/app.py", line 750, in handle_request
response = await response
File "server-sanic.py", line 53, in xlsx
await gather(*data_tasks)
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "server-sanic.py", line 102, in add_data_to_sheet
await add_img_to_sheet(sheet, rowIndex, colIndex, val)
File "server-sanic.py", line 114, in add_img_to_sheet
image_data = BytesIO(await getRaw(imgUrl))
File "server-sanic.py", line 138, in getRaw
async with app.session.get(url) as resp:
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 690, in __aenter__
self._resp = yield from self._coro
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 277, in _request
yield from resp.start(conn, read_until_eof)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client_reqrep.py", line 637, in start
self._continue = None
File "/usr/local/lib/python3.5/dist-packages/aiohttp/helpers.py", line 732, in __exit__
raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError
There is no benefit to launching a million requests at once. Limit it to 10 or whatever works and wait for those before continuing the loop.
for imageURL in request.json['images']:
data_tasks.append(getRaw(imageURL))
if len(data_tasks) > 10:
await gather(*data_tasks)
data_tasks = []
await gather(*data_tasks)
I'm trying to create a web app that communicates with Telegram. And trying to use Sanic web framework with Telepot. Both are asyncio based. Now I'm getting a very weird error.
This is my code:
import datetime
import telepot.aio
from sanic import Sanic
app = Sanic(__name__, load_env=False)
app.config.LOGO = ''
#app.listener('before_server_start')
async def server_init(app, loop):
app.bot = telepot.aio.Bot('anything', loop=loop)
# here we fall
await app.bot.sendMessage(
"#test",
"Wao! {}".format(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),)
)
if __name__ == "__main__":
app.run(
debug=True
)
The error that I'm getting is:
[2018-01-18 22:41:43 +0200] [10996] [ERROR] Experienced exception while trying to serve
Traceback (most recent call last):
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/sanic/app.py", line 646, in run
serve(**server_settings)
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/sanic/server.py", line 588, in serve
trigger_events(before_start, loop)
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/sanic/server.py", line 496, in trigger_events
loop.run_until_complete(result)
File "uvloop/loop.pyx", line 1364, in uvloop.loop.Loop.run_until_complete
File "/home/mk/Dev/project/sanic-telepot.py", line 14, in server_init
"Wao! {}".format(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),)
File "/usr/lib/python3.6/asyncio/coroutines.py", line 109, in __next__
return self.gen.send(None)
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/telepot/aio/__init__.py", line 100, in sendMessage
return await self._api_request('sendMessage', _rectify(p))
File "/usr/lib/python3.6/asyncio/coroutines.py", line 109, in __next__
return self.gen.send(None)
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/telepot/aio/__init__.py", line 78, in _api_request
return await api.request((self._token, method, params, files), **kwargs)
File "/usr/lib/python3.6/asyncio/coroutines.py", line 109, in __next__
return self.gen.send(None)
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/telepot/aio/api.py", line 139, in request
async with fn(*args, **kwargs) as r:
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/aiohttp/client.py", line 690, in __aenter__
self._resp = yield from self._coro
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/aiohttp/client.py", line 221, in _request
with timer:
File "/home/mk/Dev/project/venv/lib/python3.6/site-packages/aiohttp/helpers.py", line 712, in __enter__
raise RuntimeError('Timeout context manager should be used '
RuntimeError: Timeout context manager should be used inside a task
Telepot inside uses aiohttp as dependency and for the HTTP calls. And the very similar code is working if I make very similar functionality with just aiohttp.web.
I'm not sure to what project this problem is related. Also, all other dependencies like redis, database connections that I connected the same approach are working perfectly.
Any suggestions how to fix it?