Develop Reference

Use multiprocess for boto3 s3 upload_fileobj causes SSLError

In AWS Lambda with runtimes python3.9 and boto3-1.20.32, I run the following code,
s3_client = boto3.client(service_name="s3")
s3_bucket = "bucket"
s3_other_bucket = "other_bucket"
def multiprocess_s3upload(tar_index: dict):
def _upload(filename, bytes_range):
src_key = ...
# get single raw file in tar with bytes range
s3_obj = s3_client.get_object(
Bucket=s3_bucket,
Key=src_key,
Range=f"bytes={bytes_range}"
)
# upload raw file
# error occur !!!!!
s3_client.upload_fileobj(
s3_obj["Body"],
s3_other_bucket,
filename
)
def _wait(procs):
for p in procs:
p.join()
processes = []
proc_limit = 256 # limit concurrent processes to avoid "open too much files" error
for filename, bytes_range in tar_index.items():
# filename = "hello.txt"
# bytes_range = "1024-2048"
proc = Process(
target=_upload,
args=(filename, bytes_range)
)
proc.start()
processes.append(proc)
if len(processes) == proc_limit:
_wait(processes)
processes = []
_wait(processes)
This program is extract partial raw files in a tar file in a s3 bucket, then upload each raw file to another s3 bucket. There may be thousands of raw files in a tar file, so I use multiprocess to speed up s3 upload operation.
And, I got the exception in a subprocess about SSLError for processing the same tar file randomly. I tried different tar file and got the same result. Only the last one subprocess threw the exception, the remaining worked fine.
Process Process-2:
Traceback (most recent call last):
File "/var/runtime/urllib3/response.py", line 441, in _error_catcher
yield
File "/var/runtime/urllib3/response.py", line 522, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/var/lang/lib/python3.9/http/client.py", line 463, in read
n = self.readinto(b)
File "/var/lang/lib/python3.9/http/client.py", line 507, in readinto
n = self.fp.readinto(b)
File "/var/lang/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/var/lang/lib/python3.9/ssl.py", line 1242, in recv_into
return self.read(nbytes, buffer)
File "/var/lang/lib/python3.9/ssl.py", line 1100, in read
return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2633)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lang/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self._target(*self._args, **self._kwargs)
File "/var/task/main.py", line 144, in _upload
s3_client.upload_fileobj(
File "/var/runtime/boto3/s3/inject.py", line 540, in upload_fileobj
return future.result()
File "/var/runtime/s3transfer/futures.py", line 103, in result
return self._coordinator.result()
File "/var/runtime/s3transfer/futures.py", line 266, in result
raise self._exception
File "/var/runtime/s3transfer/tasks.py", line 269, in _main
self._submit(transfer_future=transfer_future, **kwargs)
File "/var/runtime/s3transfer/upload.py", line 588, in _submit
if not upload_input_manager.requires_multipart_upload(
File "/var/runtime/s3transfer/upload.py", line 404, in requires_multipart_upload
self._initial_data = self._read(fileobj, threshold, False)
File "/var/runtime/s3transfer/upload.py", line 463, in _read
return fileobj.read(amount)
File "/var/runtime/botocore/response.py", line 82, in read
chunk = self._raw_stream.read(amt)
File "/var/runtime/urllib3/response.py", line 544, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/var/runtime/urllib3/response.py", line 452, in _error_catcher
raise SSLError(e)
urllib3.exceptions.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2633)
According to this 10-years-ago similar question Multi-threaded S3 download doesn't terminate, the root cause might be boto3 s3 upload use a non-thread-safe library for sending http request. But, the solution doesn't work for me.
I found a boto3 issue about my question. This the problem has disappeared without any change on the author part.
Actually, the problem has recently disappeared on its own, without any (!) change on my part. As I thought, the problem was created and fixed by Amazon. I'm only afraid what if it will be a thing again...
Does anyone know how to fix this?

According to boto3 documentation about multiprocessing (doc),
Resource instances are not thread safe and should not be shared across threads or processes. These special classes contain additional meta data that cannot be shared. It's recommended to create a new Resource for each thread or process:
My modified code,
def multiprocess_s3upload(tar_index: dict):
def _upload(filename, bytes_range):
src_key = ...
# get single raw file in tar with bytes range
s3_client = boto3.client(service_name="s3") # <<<< one clien per thread
s3_obj = s3_client.get_object(
Bucket=s3_bucket,
Key=src_key,
Range=f"bytes={bytes_range}"
)
# upload raw file
s3_client.upload_fileobj(
s3_obj["Body"],
s3_other_bucket,
filename
)
def _wait(procs):
...
...
It seems that no SSLError exception occurs.

Timeout issue with python google cloud storage libraries when using Python Multithreading

The problem that I'm trying to solve is to unzip a bunch of zip files sitting in the GCS bucket. As part of processing these files, I update the file status accordingly in a MongoDB using API endpoints. I have a method that takes a dictionary as an argument and do the following.
Update the status of the file to PROCESSING
Download the file from the GCS bucket.
Unzip the file.
Upload the unzipped file back to the GCS bucket.
Update the file status as FINISHED
As I have multiple files sitting in the bucket to unzip tried using python multithreading to parallelize the operation to process each zip file in a separate thread, but after a minute or so getting the requests.exceptions.Timeout error thrown by google libraries.
When I try processing the files without using multithreading then it works fine without any error. Below are the code and stack trace for reference. Any help is much appreciated.
def process_files_multithreading(self, file_info):
print("process id: " +str(os.getpid()))
try:
print("Updating the file status to PROCESSING")
update_file_status_url = utils.getUpdateFileStatusUrl(config["host"],
file_info["fileId"])
utils.updateFileStatus(update_file_status_url,"PROCESSING")
client = gcs_utils._getGCSConnection()
blob_path = file_info["fileLocation"]
bucket = client.get_bucket(file_info["bucketName"])
blob = bucket.blob(blob_path)
zipbytes = io.BytesIO(blob.download_as_string())
if is_zipfile(zipbytes):
with ZipFile(zipbytes, 'r') as myzip:
print(myzip.namelist())
for contentfilename in myzip.namelist():
contentfile = myzip.read(contentfilename)
blob = bucket.blob(blob_path + "/" + contentfilename)
blob.upload_from_string(contentfile)
print("proccessed file {} successfully".format(blob_path))
print("updating the file status to FINISHED")
update_file_status_url = utils.getUpdateFileStatusUrl(config["host"], file_info["fileId"])
utils.updateFileStatus(update_file_status_url,"FINISHED")
except(Exception):
print("Exception in processing the file")
print("updating the file status to FAILED")
update_file_status_url = utils.getUpdateFileStatusUrl(config["host"], file_info["fileId"])
utils.updateFileStatus(update_file_status_url,"FAILED")
print(traceback.format_exc())
'''
calling the above function using ThreadPoolExecutor
'''
# files is the list of dictionaries
with ThreadPoolExecutor(len(files)) as pool:
pool.map(self.process_files_multithreading, files)
Exception:
Traceback (most recent call last):
blob.upload_from_string(contentfile)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1361, in upload_from_string
predefined_acl=predefined_acl,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1261, in upload_from_file
client, file_obj, content_type, size, num_retries, predefined_acl
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1171, in _do_upload
client, stream, content_type, size, num_retries, predefined_acl
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1118, in _do_resumable_upload
response = upload.transmit_next_chunk(transport)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/resumable_media/requests/upload.py", line 425, in transmit_next_chunk
retry_strategy=self._retry_strategy,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136, in http_request
return _helpers.wait_and_retry(func, RequestsMixin._get_status_code, retry_strategy)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150, in wait_and_retry
response = func()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/auth/transport/requests.py", line 287, in request
**kwargs
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/auth/transport/requests.py", line 110, in __exit__
raise self._timeout_error_type()
requests.exceptions.Timeout

Why am I getting "RuntimeError: This event loop is already running"

Im working on a slack bot using the new slack 2.0 python library. I am new to python decorators and I suspect that is part of my problem.
Here is my code...
#!/opt/rh/rh-python36/root/usr/bin/python
import os
import slack
# instantiate Slack client
slack_token = os.environ['SLACK_BOT_TOKEN']
rtmclient = slack.RTMClient(token=slack_token)
webclient = slack.WebClient(token=slack_token)
# get the id of my user
bot_id = webclient.auth_test()['user_id']
print('Bot ID: {0}'.format(bot_id))
def get_user_info(user_id):
user_info = webclient.users_info(user=user_id)['ok']
return user_info
#slack.RTMClient.run_on(event='message')
def parse_message(**payload):
data = payload['data']
user_id = data['user']
print(get_user_info(user_id))
rtmclient.start()
It outputs the Bot ID(using the webclient) when started but then crashes with RuntimeError: This event loop is already running when I make another call to webclient.
[root#slackbot-01 bin]# scl enable rh-python36 /root/slackbot/bin/slackbot.py
Bot ID: UBT547D31
Traceback (most recent call last):
File "/root/slackbot/bin/slackbot.py", line 24, in <module>
rtmclient.start()
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 197, in start
return self._event_loop.run_until_complete(future)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 467, in run_until_complete
return future.result()
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 339, in _connect_and_read
await self._read_messages()
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 390, in _read_messages
await self._dispatch_event(event, data=payload)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 440, in _dispatch_event
self._execute_in_thread(callback, data)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 465, in _execute_in_thread
future.result()
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/slackbot/bin/slackbot.py", line 22, in parse_message
print(get_user_info(user_id))
File "/root/slackbot/bin/slackbot.py", line 15, in get_user_info
user_info = webclient.users_info(user=user_id)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/web/client.py", line 1368, in users_info
return self.api_call("users.info", http_verb="GET", params=kwargs)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/web/base_client.py", line 154, in api_call
return self._event_loop.run_until_complete(future)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 454, in run_until_complete
self.run_forever()
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 408, in run_forever
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
The really confusing part to me is that if I comment out the line that makes the first call to webclient.auth_test(), I have no issues at all. My call to webclient.users_info() works every time rtmclient sends me data.
#!/opt/rh/rh-python36/root/usr/bin/python
import os
import slack
# instantiate Slack client
slack_token = os.environ['SLACK_BOT_TOKEN']
rtmclient = slack.RTMClient(token=slack_token)
webclient = slack.WebClient(token=slack_token)
# get the id of my user
#bot_id = webclient.auth_test()['user_id']
#print('Bot ID: {0}'.format(bot_id))
def get_user_info(user_id):
user_info = webclient.users_info(user=user_id)['ok']
return user_info
#slack.RTMClient.run_on(event='message')
def parse_message(**payload):
data = payload['data']
user_id = data['user']
print(get_user_info(user_id))
rtmclient.start()
[root#slackbot-01 bin]# scl enable rh-python36 /root/slackbot/bin/slackbot.py
True
True
^C[root#slackbot-01 bin]#
I need to get the bot id so that I can make sure it doesnt answer it's own messages. I don't why my code doesnt work after I get the bot id outside of the parse message function with a decorator.
What am I doing wrong here?

The python event loop is a tricky thing to program libraries around and there are some issues with the way the event queue is managed in the 2.0 version of SlackClient. It looks like some improvements were made with 2.1 but it appears to be a work in progress, and I still encounter this. I'd expect there will be future updates to make it more robust.
In the meantime, the following code at the top of your file (use pip to install) usually resolves it for me:
import nest_asyncio
nest_asyncio.apply()
Keep in mind this will alter the way the rest of your application is handling the event queue, if that's a factor.

If you're using RTM, the RTMClient creates a WebClient for you. The handle for it should be getting passed to you in the payload when you handle an event. You can check your ID by looking for the 'open' event which is always dispatched after RTM successfully connects and doing the lookup inside your 'open' event handler.

AXL Python getLine using pattern instead of uuid does not work (CUCM 11.5)

I'm struggling with getting for example line info via getLine using pattern instead of uuid. According to getLine AXL schema for CUCM 11.5 it should be possible to choose either uuid or pattern as lookup criteria. The python (2.7) module that I'm using is suds-jurko 0.6.
Base code is as follows:
from suds.client import Client
import ssl
wsdl = 'file:///C:/Users/xyz/Documents/axlplugin/schema/current/AXLAPI.wsdl'
location = 'https://10.10.20.1/axl/'
username = '***'
password = '***'
ssl._create_default_https_context = ssl._create_unverified_context
client = Client(wsdl, location=location, username=username, password=password)
First of all, the proof that the pattern I want to use as a lookup string for getLine actually exists:
>>> line2 = client.service.listLine({'pattern': '1018'}, returnedTags={'description': ''})
>>> line2
(reply){
return =
(return){
line[] =
(LLine){
_uuid = "{1EC56035-6B5D-283A-4DF0-EFEFA01FCEFF}"
description = None
},
}
}
Then, I try getLine using uuid which works well:
>>> line5 = client.service.getLine('1EC56035-6B5D-283A-4DF0-EFEFA01FCEFF')
>>> line5
(reply){
return =
(return){
line =
(RLine){
_uuid = "{1EC56035-6B5D-283A-4DF0-EFEFA01FCEFF}"
pattern = "1018"
description = None
usage = "Device"
routePartitionName = ""
aarNeighborhoodName = ""
----Rest omitted for brevity---
Now, if I try to use pattern, not uuid, I have following error:
line6 = client.service.getLine({'pattern': '1018'}, {'routePartitionName': ''})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\suds\client.py", line 521, in __call__
return client.invoke(args, kwargs)
File "C:\Python27\lib\site-packages\suds\client.py", line 576, in invoke
soapenv = binding.get_message(self.method, args, kwargs)
File "C:\Python27\lib\site-packages\suds\bindings\binding.py", line 109, in get_message
content = self.bodycontent(method, args, kwargs)
File "C:\Python27\lib\site-packages\suds\bindings\document.py", line 95, in bodycontent
add_param, self.options().extraArgumentErrors)
File "C:\Python27\lib\site-packages\suds\argparser.py", line 83, in parse_args
return arg_parser(args, kwargs, extra_parameter_errors)
File "C:\Python27\lib\site-packages\suds\argparser.py", line 108, in __call__
self.__process_parameters()
File "C:\Python27\lib\site-packages\suds\argparser.py", line 299, in __process_parameters
self.__process_parameter(*pdef)
File "C:\Python27\lib\site-packages\suds\argparser.py", line 294, in __process_parameter
self.__in_choice_context(), value)
File "C:\Python27\lib\site-packages\suds\bindings\document.py", line 86, in add_param
p = self.mkparam(method, pdef, value)
File "C:\Python27\lib\site-packages\suds\bindings\document.py", line 130, in mkparam
return Binding.mkparam(self, method, pdef, object)
File "C:\Python27\lib\site-packages\suds\bindings\binding.py", line 225, in mkparam
return marshaller.process(content)
File "C:\Python27\lib\site-packages\suds\mx\core.py", line 59, in process
self.append(document, content)
File "C:\Python27\lib\site-packages\suds\mx\core.py", line 72, in append
self.appender.append(parent, content)
File "C:\Python27\lib\site-packages\suds\mx\appender.py", line 88, in append
appender.append(parent, content)
File "C:\Python27\lib\site-packages\suds\mx\appender.py", line 229, in append
Appender.append(self, child, cont)
File "C:\Python27\lib\site-packages\suds\mx\appender.py", line 168, in append
self.marshaller.append(parent, content)
File "C:\Python27\lib\site-packages\suds\mx\core.py", line 71, in append
if self.start(content):
File "C:\Python27\lib\site-packages\suds\mx\literal.py", line 86, in start
raise TypeNotFound(content.tag)
suds.TypeNotFound: Type not found: 'pattern'
Trying different syntax also gives error:
line6 = client.service.getLine('1018')
No handlers could be found for logger "suds.client"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\suds\client.py", line 521, in __call__
return client.invoke(args, kwargs)
File "C:\Python27\lib\site-packages\suds\client.py", line 581, in invoke
result = self.send(soapenv)
File "C:\Python27\lib\site-packages\suds\client.py", line 619, in send
description=tostr(e), original_soapenv=original_soapenv)
File "C:\Python27\lib\site-packages\suds\client.py", line 670, in process_reply
raise WebFault(fault, replyroot)
suds.WebFault: Server raised fault: 'Item not valid: The specified Line was not found'
Am I using incorrect syntax or it's actually not possible to use pattern however schema file states differently?
Additionally, I'm wondering how to extract only uuid value from for example listLine method, is it possible via python axl?

In Suds, if memory serves me correctly, you can use keyword arguments for get and update requests.
For example:
resp = client.service.getLine(routePartitionName='PT-ONCLUSTER', pattern='\+49301234567')
if that doesn't work, try:
resp = client.service.getLine({'routePartitionName': 'PT-ONCLUSTER', 'pattern': '\+49301234567'})
Or better yet, check out python-zeep instead. It's much faster and better maintained than suds. Some example code with Zeep:
# -*- coding: utf-8 -*-
from zeep import Client
from zeep.cache import SqliteCache
from zeep.transports import Transport
from zeep.exceptions import Fault
from zeep.plugins import HistoryPlugin
from requests import Session
from requests.auth import HTTPBasicAuth
from urllib3 import disable_warnings
from urllib3.exceptions import InsecureRequestWarning
from lxml import etree
disable_warnings(InsecureRequestWarning)
username = 'admin'
password = 'password'
# If you're not disabling SSL verification, host should be the FQDN of the server rather than IP
host = '10.1.1.1'
wsdl = 'file://C:/path/to/wsdl/AXLAPI.wsdl'
location = 'https://{host}:8443/axl/'.format(host=host)
binding = "{http://www.cisco.com/AXLAPIService/}AXLAPIBinding"
# Create a custom session to disable Certificate verification.
# In production you shouldn't do this,
# but for testing it saves having to have the certificate in the trusted store.
session = Session()
session.verify = False
session.auth = HTTPBasicAuth(username, password)
transport = Transport(cache=SqliteCache(), session=session, timeout=20)
history = HistoryPlugin()
client = Client(wsdl=wsdl, transport=transport, plugins=[history])
service = client.create_service(binding, location)
def show_history():
for item in [history.last_sent, history.last_received]:
print(etree.tostring(item["envelope"], encoding="unicode", pretty_print=True))
try:
resp = service.getLine(pattern='1018', routePartitionName='')
except Fault:
show_history()

Python cherrypy and bottle with ssl - server fails to save a file bigger than 102199 bytes

The next problem happens only when SSL is enabled!
I am running a server with bottle version 0.12.6 and cherrypy version 3.2.2 using https.
The client code sends a file to the server and the server saves it.
when i send a file with size below 102199 bytes, it is received and saved successfully. However, When i send a file with size bigger or equal to 102199, I get the exception:
The Server Code:
from bottle import request, response,static_file, run,server_names
from OpenSSL import crypto,SSL
from bottle import Bottle, run, request, server_names, ServerAdapter
app = Bottle()
app.mount('/test' , app)
class MySSLCherryPy(ServerAdapter):
def run(self, handler):
from cherrypy import wsgiserver
server = wsgiserver.CherryPyWSGIServer((self.host, self.port), handler)
server.ssl_certificate = "./cert"
server.ssl_private_key = "./key"
try:
server.start()
finally:
server.stop()
#app.post('/upload')
def received_file():
file = request.files.file
# file.save("./newfile")
file_path="./newfile"
with open(file_path, 'w') as open_file:
open_file.write(file.read())
if __name__=='__main__':
server_names['mysslcherrypy'] = MySSLCherryPy
run(app, host='0.0.0.0', port=4430, server='mysslcherrypy')
exit(0)
Why does the server fail to get file more than a given limit? is there a limit that i need to change?
(I tried to set the constant MEMFILE_MAX at the function received_file but it didn't help)
The problem vanish if the server is http and not https!
The exception in plain text (in case you cannot view the image):
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/bottle.py", line 861, in _handle
return route.call(**args)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1727, in wrapper
rv = callback(*a, **ka)
File "testser", line 28, in received_file
file = request.files.file
File "/usr/lib/python2.6/site-packages/bottle.py", line 165, in get
if key not in storage: storage[key] = self.getter(obj)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1106, in files
for name, item in self.POST.allitems():
File "/usr/lib/python2.6/site-packages/bottle.py", line 165, in get
if key not in storage: storage[key] = self.getter(obj)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1222, in POST
args = dict(fp=self.body, environ=safe_env, keep_blank_values=True)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1193, in body
self._body.seek(0)
File "/usr/lib/python2.6/site-packages/bottle.py", line 165, in get
if key not in storage: storage[key] = self.getter(obj)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1162, in _body
for part in body_iter(read_func, self.MEMFILE_MAX):
File "/usr/lib/python2.6/site-packages/bottle.py", line 1125, in _iter_body
part = read(min(maxread, bufsize))
File "/usr/lib/python2.6/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 329, in read
data = self.rfile.read(size)
File "/usr/lib/python2.6/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1052, in read
assert n <= left, "recv(%d) returned %d bytes" % (left, n)
AssertionError: recv(47) returned 48 bytes
Solution
In the file bottle.py I changed the value of .MEMFILE_MAX to be 10000000 and by this i solved the problem. The best way to do this is from your server code by adding the next line:
bottle.BaseRequest.MEMFILE_MAX=30000000000