Develop Reference

python motor asyncio can't start new thread

I use asyncio to write to mongo, using motor library.
When I have few bulk_writes, it works with no problem.
However, when I have many write in the same time, I get an exception RuntimeError: can't start new thread.
File "/usr/local/lib/python3.7/site-packages/motor/metaprogramming.py", line 77, in method **unwrapped_kwargs)
File "/usr/local/lib/python3.7/site-packages/motor/frameworks/asyncio/__init__.py", line 74 in run_on_executor
_EXECUTOR, functools.partial(fn, *args, **kwargs))
File "uvloop/loop.pyx", line 2702, in uvloop.loop.Loop.run_in_exector
File "/usr/local/lib/python3.7/concurrent/features/thread.py", line 160, in submit
self._adjust_thread_count()
File "/usr/local/lib/python3.7/concurrent/features/thread.py", line 181, in _adjust_thread_count
t.start()
File "usr/local/lib/python3.7/threading.py", line 847, in start
_start_new_thread(self._bootsrap, ())
RuntimeError: can't start new thread
I tried to change maxPoolSize, but it didn't work.
Important facts:
In my local computer, it works with no errors. However, in Openshift I have this problems.
In Openshift I run my code via gunicorn via gunicorn app:app --worker-class uvicorn.workers.UvicornWorker
In Openshift, when I have only one worker, it works. But with 2+ worker I have this problem.
I don't open many connection of AsyncIOMontorClient, I have only two at a time.
With pymongo with almost the same code, I have no error, but there is no asyncio support in pymongo.
Without the part of mongo, my code works with no problems.

Solved.
There is a limit of 1024 threads per openshift pod.

msgpack-python failing to deserialize -> getting ExtraData error

I'm trying to make MQTTtoROS Bridge work, and i keep getting this error:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2627, in _thread_main
self.loop_forever(retry_first_connection=True)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1407, in loop_forever
rc = self.loop(timeout, max_packets)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 923, in loop
rc = self.loop_read(max_packets)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1206, in loop_read
rc = self._packet_read()
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1799, in _packet_read
rc = self._packet_handle()
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2275, in _packet_handle
return self._handle_publish()
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2461, in _handle_publish
self._handle_on_message(message)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2615, in _handle_on_message
t[1](self, self._userdata, message)
File "/home/animu/catkin_ws/src/mqtt_bridge-master/src/mqtt_bridge/bridge.py", line 114, in _callback_mqtt
ros_msg = self._create_ros_message(mqtt_msg)
File "/home/animu/catkin_ws/src/mqtt_bridge-master/src/mqtt_bridge/bridge.py", line 124, in _create_ros_message
msg_dict = self._deserialize(mqtt_msg.payload)
File "msgpack/_unpacker.pyx", line 143, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:2143)
ExtraData: unpack(b) received extra data.
I can't find anything on it in the internet, as this bridge is i guess not commonly used. The only similar problems were in Salt and Kafka, but the solution is nowhere to be found. All python libraries are up to date, i double checked. The bridge sends messages from RoS to MQTT without any problems, both STR and BOOL types. Any message sent from MQTT ends up as this error with no reception from ROS.

Its a bit late but I'll give some advice for future readers.
First, make sure you have installed all requirements for the bridge to function. Check them by reading requirement.txt
Second, Edit mqtt_bridge configuration file to match topics from ROS and from your MQTT server. Also IP address/port of MQTT server.
Thats it.

Since Animu is a student, I assume that this was an assignment from his training or internship company. An answer is probably no use anymore, but since I also had this problem, I hereby offer a solution for future readers:
In the bridge repository there is a file called "demo_params.yaml". Or if you have already named it differently, then the .yaml file that contains your "settings".
This file includes the following:
mqtt:
client:
protocol: 4 # MQTTv311
connection:
host: localhost
postage: 1883
keepalive: 60
private_path: device / 001
serializer: msgpack: dumps
deserializer: msgpack: loads
bridge:
# ping pong
- factory: mqtt_bridge.bridge: RosToMqttBridge
msg_type: std_msgs.msg: Bool
topic_from: / ping
topic_to: ping
- factory: mqtt_bridge.bridge: MqttToRosBridge
msg_type: std_msgs.msg: Bool
topic_from: ping
topic_to: / pong
As you can see, it says msgpack is used to serialize and deserialize your messages that are sent back and forth. This mainly works for ROS to MQTT. The other way around, this does not work, as no correct actions are performed in the Python code. You have two solutions for this.
Continue to work with msgpack and make sure that the MQTT messages you publish are already encoded as msgpack would serialize them itself. (binary). This is a tricky solution, since you just want to keep MQTT messages human readable if you publish them manually. If you have written another program that publishes the MQTT messages, feel free to serialize the message first with msgpack and then publish. Then the bridge also works.
The other option is to have JSON serialization, instead of msgpack serialization. This is the default option of the bridge, but you can also specify this in your .yaml file. You do this by editing this:
serializer: msgpack: dumps
deserializer: msgpack: loads
to this:
serializer: json: dumps
deserializer: json: loads
Now you can publish mqtt messages both manually and with the help of software.
You do this as follows:
mosquitto_pub -t 'echo' -m '{"data": "test"}

python boto3 connection error with no apparent cause

I'm hitting an error with code that connects to AWS using boto3. The error just started yesterday afternoon, and between the last time I didn't get the error and the first time I got the error I don't see anything that changed.
The error is:
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL:
In .aws/config I have:
$ cat ~/.aws/config
[default]
region=us-east-1
Here's what I know:
Using the same AWS credentials and config on another machine, I don't see the error.
Using different AWS credentials and config on the same machine, I do see the error.
I'm the only one in our group that has this issue for any credentials on any machine.
I don't think I changed anything that would affect this between the last time this worked and the first time it didn't. It seems like I'd have had to change some AWS specific configuration on my side or some low level libraries, and I didn't make any such change. I was talking with a colleague for 30-45 minutes and when I returned and picked up where I left off the issue first appeared.
Any thoughts or ideas on troubleshooting this?
Full exception dump follows.
$ python
Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.client('ec2').describe_regions()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/botocore/client.py", line 200, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Library/Python/2.7/site-packages/botocore/client.py", line 244, in _make_api_call
operation_model, request_dict)
File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 173, in make_request
return self._send_request(request_dict, operation_model)
File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 203, in _send_request
success_response, exception):
File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 267, in _needs_retry
caught_exception=caught_exception)
File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 226, in emit
return self._emit(event_name, kwargs)
File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 209, in _emit
response = handler(**kwargs)
File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 250, in __call__
caught_exception)
File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 273, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 313, in __call__
caught_exception)
File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 222, in __call__
return self._check_caught_exception(attempt_number, caught_exception)
File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 355, in _check_caught_exception
raise caught_exception
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://ec2.us-east-1.amazonaws.com/"

Issue resolved. It turns out that a couple of seemingly unrelated actions independent of anything boto related resulted in HTTP_PROXY and HTTPS_PROXY environment variables being improperly set, which was then breaking the botocore calls under both boto3 and the aws cli. Removing both environment variables resolved the problem.
I'll leave this up as I found it very difficult to find anything pointing to this as a possible cause of this error. Might save someone else some of the hair pulling I went through.

I just had a similar issue. All of a sudden, no connection possible anymore to my s3 through boto3 on django while I had still the possibility to do the actions on my Heroku environment.
Appeared I recently installed the amazon CLI where my configuration was different and the CLI overrules the environment variables... Damn. took me 3 hours to find.
through aws configure I now set
AWS Access Key ID [****************MPIA]: "your true key here without quotes"
AWS Secret Access Key [****************7DWm]: "your true secret access key here without quotes"
Default region name [eu-west-1]: "your true region here without quotes"
Default output format [None]: [here i did just an enter in order not to change this]
Just posting this for the sake of anyone having this issue.

I came across same error when my connection went down - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://ec2.us-east-2.amazonaws.com/"
When connection is restored then it worked without any issue. Probable reason for this error could be
Connection error
Region is not available to cater your request as every request hits endpoint on AWS (more detail can be found on https://docs.aws.amazon.com/general/latest/gr/rande.html#billing-pricing )
It seems like Boto3 has matured enough to throw exception for more proper reason of failure to precisely know what is going.
Also if you have any issue related to your config then most of them are encapsulated with ClientError exception.

Catching bottle server errors

I am trying to get my bottle server so that when one person in a game logs out, everyone can immediately see it. As I am using long polling, there is a request open with all the users.
The bit I am having trouble with is catching the exception that is thrown when the user leaves the page from the long polling that can no longer connect to the page. The error message is here.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 425, in run_application
self.process_result()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 416, in process_result
self.write(data)
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 373, in write
self.socket.sendall(msg)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 509, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 483, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
<WSGIServer fileno=3 address=0.0.0.0:8080>: Failed to handle request:
request = GET /refreshlobby/1 HTTP/1.1 from ('127.0.0.1', 53331)
application = <bottle.Bottle object at 0x7f9c05672750>
127.0.0.1 - - [2013-07-07 10:59:30] "GET /refreshlobby/1 HTTP/1.1" 200 160 6.038377
The function to handle that page is this.
#route('/refreshlobby/<id>')
def refreshlobby(id):
while True:
yield lobby.refresh()
gevent.sleep(1)
I tried catching the exception within the function, and in a decorator which I put to wrap #route, neither of which worked. I tried making an #error(500) decorator, but that didn't trigger, either. It seems that this is to do with the internals of bottle.
Edit: I know now that I need to be catching socket.error, but I don't know whereabouts in my code

The WSGI runner
Look closely at the traceback: this in not happening in your function, but in the WSGI runner.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
The way the WSGI runner works, in your case, is:
Receives a request
Gets a partial response from your code
Sends it to the client (this is where the exception is raised)
Repeats steps 2-3
You can't catch this exception
This error is not raised in your code.
It happens when you try to send a response to a client that closed the connection.
You'll therefore not be able to catch this error from within your code.
Alternate solutions
Unfortunately, it's not possible to tell from within the generator (your code) when it stops being consumed.
It's also not a good idea to rely on your generator being garbage collected.
You have a couple other solutions.
"Last seen"
Another way to know when an user disconnects would probably be to record a "last seen", after your yield statement.
You'll be able to identify clients that disconnected if their last seen is far in the past.
Other runner
Another, non-WSGI runner, will be more appropriate for a realtime application. You could give tornado a try.

Random timeout error with Pika and gevent

I've been trying to make use of RabbitMQ from within my gevent program by using the Pika library (monkey patched by gevent), gevent likes randomly throwing a timeout error.
What should I do? Is there another library I could use?
WARNING:root:Document not found, retrying primary.
Traceback (most recent call last):
...
File "/usr/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 32, in __init__
BaseConnection.__init__(self, parameters, None, reconnection_strategy)
File "/usr/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 50, in __init__
reconnection_strategy)
File "/usr/lib/python2.7/dist-packages/pika/connection.py", line 170, in __init__
self._connect()
File "/usr/lib/python2.7/dist-packages/pika/connection.py", line 228, in _connect
self.parameters.port or spec.PORT)
File "/usr/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 44, in _adapter_connect
self._handle_read()
File "/usr/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 151, in _handle_read
data = self.socket.recv(self._suggested_buffer_size)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 427, in recv
wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 169, in wait_read
switch_result = get_hub().switch()
File "/usr/lib/python2.7/dist-packages/gevent/hub.py", line 164, in switch
return greenlet.switch(self)
timeout: timed out

Pika is not ideally suited to use with gevent because pika implements its own asynchronous connection to RabbitMQ based on non-blocking sockets. This just does not fit well with gevent's implementation of the same.
You may want to consider using py-amqplib or kombu

I'm also having timeout problems with using Pika in a Django/Gunicorn application. I played with raising connection_attempts or increasing the timeout but RabbitMQ always closed the connection with a handshake error. The latter seems to indicate that Pika never transmitted any data on the socket.
The cause for the timeouts could be this libevent bug - at least in my environment the script attached to the bug is able to reproduce the issue.
You could try upgrading to gevent>=1.0 (at the time of writing not released yet):
wget http://gevent.googlecode.com/files/gevent-1.0b4.tar.gz
pip install gevent-1.0b4.tar.gz