Develop Reference

An error occurred when downloading files from web using Python

I am new to the Python.I was trying to download a file from a website.This is the URL of the file I need http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
I run my code many times.It alwys works.But sometimes an error will occurr.
Here is my code
import requests
url = 'http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz'
r = requests.get(url, allow_redirects=True,timeout=5)
I learned this from https://www.tutorialspoint.com/downloading-files-from-web-using-python.
The Traceback is
Traceback (most recent call last):
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1371, in getresponse
response.begin()
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 319, in begin
version, status, reason = self._read_status()
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 280, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\socket.py", line 704, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\packages\six.py", line 770, in reraise
raise value
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 336, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='yann.lecun.com', port=80): Read timed out. (read timeout=5)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\jojo\source\repos\PythonApplication5\PythonApplication5\PythonApplication5.py", line 9, in <module>
r = requests.get(url, allow_redirects=True,timeout=5)
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\jojo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='yann.lecun.com', port=80): Read timed out. (read timeout=5)
I think it is a network connection error,but I am not sure.If so,how could I avoid it?
SORRY FOR THIS SILLY QUESTION!!!

ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL:

So I'm trying to figure out Sagemaker and can get an endpoint for a tensorflow model up and running. Strangely enough, when I send small images to it, up to 220x220 (40kb), it runs just fine! However I get a connection closed error when I try anything larger, say 225x225.
I've tried from different networks and computers and get the same error. Also, I don't think I'm anywhere near the 5MB limit for sagemaker requests so I must admit I'm stumped on this one. Anyone happen to know what might be the cause of this? Thanks for your help.
I'm calling my model via boto as follows:
config = Config(connect_timeout=999999, read_timeout=9999999)
client = boto3.client("runtime.sagemaker", config=config)
data = {"instances": [{"x":x, "x2":x2}]}
response = client.invoke_endpoint(EndpointName='dcscn',
Body=json.dumps(data),
ContentType="application/json")
response_body = response['Body']
The error I get is:
Traceback (most recent call last):
File "/Users/bhhj/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/bhhj/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Users/bhhj/anaconda/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 125, in _send_request
method, url, body, headers, *args, **kwargs)
File "/Users/bhhj/anaconda/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Users/bhhj/anaconda/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 152, in _send_output
self.send(msg)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 236, in send
return super(AWSConnection, self).send(str)
File "/Users/bhhj/anaconda/lib/python3.6/http/client.py", line 986, in send
self.sock.sendall(data)
File "/Users/bhhj/anaconda/lib/python3.6/ssl.py", line 975, in sendall
v = self.send(byte_view[count:])
File "/Users/bhhj/anaconda/lib/python3.6/ssl.py", line 944, in send
return self._sslobj.write(data)
File "/Users/bhhj/anaconda/lib/python3.6/ssl.py", line 642, in write
return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 258, in send
decode_content=False,
File "/Users/bhhj/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/bhhj/anaconda/lib/python3.6/site-packages/urllib3/util/retry.py", line 343, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/bhhj/anaconda/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/Users/bhhj/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/bhhj/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Users/bhhj/anaconda/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 125, in _send_request
method, url, body, headers, *args, **kwargs)
File "/Users/bhhj/anaconda/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Users/bhhj/anaconda/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 152, in _send_output
self.send(msg)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 236, in send
return super(AWSConnection, self).send(str)
File "/Users/bhhj/anaconda/lib/python3.6/http/client.py", line 986, in send
self.sock.sendall(data)
File "/Users/bhhj/anaconda/lib/python3.6/ssl.py", line 975, in sendall
v = self.send(byte_view[count:])
File "/Users/bhhj/anaconda/lib/python3.6/ssl.py", line 944, in send
return self._sslobj.write(data)
File "/Users/bhhj/anaconda/lib/python3.6/ssl.py", line 642, in write
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "ping_endpoint.py", line 100, in <module>
ContentType="application/json")
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/client.py", line 648, in _make_api_call
operation_model, request_dict, request_context)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/client.py", line 667, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/endpoint.py", line 231, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/endpoint.py", line 244, in _send
return self.http_session.send(request)
File "/Users/bhhj/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 289, in send
endpoint_url=request.url
botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL:

It can be a network error.
If you are connected to a VPN , try after disconnecting it
try with better internet connection
my case, closing the VPN worked

In my case it was a firewall issue. How to troubleshoot: first, to do the same thing with aws cli and --debug flag. Already, with debug, you may see something. Then note the url and then try to ping the machine, in my case I had:
ping api.sagemaker.us-west-2.amazonaws.com
33 packets transmitted, 0 received, 100% packet loss, time 800ms.
Whitelisting the url solved the issue

How do you solve a timeout error in a python dropbox upload script?

I'm currently working on a project to do a timelapse with a raspberry pi3 (which works well, capturing the photos ain't a problem).
However, I used another script to upload those photos to a dropbox account and it work all well, until I checked it out today, seeing a timeout error.
Does anyone know how this could solved?
This is the error:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 453, in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1426, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1149, in _raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
conn.connect()
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connection.py", line 344, in connect
ssl_context=context)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 456, in wrap_socket
raise timeout('select timed out')
socket.timeout: select timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='content.dropboxapi.com', port=443): Read timed out. (read timeout=30)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/Desktop/time/picture/drop.py", line 16, in <module>
response = db.files_upload(f.read(), dname)
File "/home/pi/.local/lib/python3.5/site-packages/dropbox/base.py", line 2293, in files_upload
f,
File "/home/pi/.local/lib/python3.5/site-packages/dropbox/dropbox.py", line 274, in request
timeout=timeout)
File "/home/pi/.local/lib/python3.5/site-packages/dropbox/dropbox.py", line 365, in request_json_string_with_retry
timeout=timeout)
File "/home/pi/.local/lib/python3.5/site-packages/dropbox/dropbox.py", line 449, in request_json_string
timeout=timeout,
File "/home/pi/.local/lib/python3.5/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/pi/.local/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/pi/.local/lib/python3.5/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/pi/.local/lib/python3.5/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='content.dropboxapi.com', port=443): Read timed out. (read timeout=30)
>>>
This is how the code to upload on dropbox looks:
import dropbox
from time import sleep
db = dropbox.Dropbox('apikeyplaceholder')
for i in range(4500): ##4500
name = 'image{0:04d}.jpg'
fname = '/home/pi/Desktop/time/picture/'+name.format(i)
dname = '/'+name.format(i)
f = open(fname, 'rb')
response = db.files_upload(f.read(), dname)
print ('uploaded:', response)
sleep(585)
f.close()
NEW ERROR:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 317, in _send_until_done
return self.connection.send(data)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1256, in send
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1151, in _raise_ssl_error
raise WantWriteError()
OpenSSL.SSL.WantWriteError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.5/http/client.py", line 1107, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 936, in _send_output
self.send(message_body)
File "/usr/lib/python3.5/http/client.py", line 908, in send
self.sock.sendall(data)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 328, in sendall
sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 320, in _send_until_done
raise timeout()
socket.timeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.5/http/client.py", line 1107, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 936, in _send_output
self.send(message_body)
File "/usr/lib/python3.5/http/client.py", line 908, in send
self.sock.sendall(data)
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 328, in sendall
sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
File "/home/pi/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 320, in _send_until_done
raise timeout()
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout())
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/Desktop/time/bild/drop.py", line 20, in <module>
response = upload(f, dname)
File "/usr/local/lib/python3.5/dist-packages/backoff/_sync.py", line 94, in retry
ret = target(*args, **kwargs)
File "/home/pi/Desktop/time/bild/drop.py", line 11, in upload
return db.files_upload(f.read(), dname)
File "/home/pi/.local/lib/python3.5/site-packages/dropbox/base.py", line 2293, in files_upload
f,
File "/home/pi/.local/lib/python3.5/site-packages/dropbox/dropbox.py", line 274, in request
timeout=timeout)
File "/home/pi/.local/lib/python3.5/site-packages/dropbox/dropbox.py", line 365, in request_json_string_with_retry
timeout=timeout)
File "/home/pi/.local/lib/python3.5/site-packages/dropbox/dropbox.py", line 449, in request_json_string
timeout=timeout,
File "/home/pi/.local/lib/python3.5/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/pi/.local/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/pi/.local/lib/python3.5/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/pi/.local/lib/python3.5/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout())

Before you implement code to handle the timeout you first need to think about how you want to handle it. A common strategy is the following:
Try re-uploading a certain number of times, potentially coupled with exponential backoff (e.g. using this library)
If that fails either skip the current image or abort the process completely.
Depending on your setup, you might want to put the uploading code into a separate thread so that delays during uploads don't interfere with capturing the photos.
A simple example using the backoff library I've linked to above (untested):
from time import sleep
import backoff
import dropbox
import requests
db = dropbox.Dropbox('apikeyplaceholder')
#backoff.on_exception(backoff.expo, requests.exceptions.ReadTimeout)
def upload(f, name):
return db.files_upload(f.read(), dname)
NAME_PATTERN = '/image{0:04d}.jpg'
for i in range(4500):
name = NAME_PATTERN.format(i)
fname = '/home/pi/Desktop/time/picture' + name
with open(fname, 'rb') as f:
response = upload(f, name)
print ('uploaded:', response)
sleep(585)

Elasticsearch().ping() displays errors from urllib3 if the server is not up

I have this expectation of Elasticsearch.ping: it should return True if Elasticsearch can be contacted, and False if it cannot. However, if connection cannot be established, ping will display a bunch of errors before finally returning False. Is there a way to prevent it? I really don't need all that, I just want my False.
$ python3
Python 3.6.5 (default, Jun 17 2018, 12:13:06)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import elasticsearch
>>> elasticsearch.Elasticsearch().ping()
HEAD http://localhost:9200/ [status:N/A request:0.002s]
Traceback (most recent call last):
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
raise err
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 172, in perform_request
response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 343, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 196, in connect
conn = self._new_conn()
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x109d19550>: Failed to establish a new connection: [Errno 61] Connection refused
HEAD http://localhost:9200/ [status:N/A request:0.002s]
Traceback (most recent call last):
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
raise err
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 172, in perform_request
response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 343, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 196, in connect
conn = self._new_conn()
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x109d195c0>: Failed to establish a new connection: [Errno 61] Connection refused
HEAD http://localhost:9200/ [status:N/A request:0.002s]
Traceback (most recent call last):
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
raise err
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 172, in perform_request
response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 343, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 196, in connect
conn = self._new_conn()
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x109d196a0>: Failed to establish a new connection: [Errno 61] Connection refused
HEAD http://localhost:9200/ [status:N/A request:0.002s]
Traceback (most recent call last):
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
raise err
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 172, in perform_request
response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 343, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 196, in connect
conn = self._new_conn()
File "/Users/amadan/work/aistqa/venv/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x109d19780>: Failed to establish a new connection: [Errno 61] Connection refused
False
>>>

Seems like an error that some people encountered :
https://github.com/elastic/elasticsearch-py/issues/666
to sum up the above discussion try and suppress the logger in elastic search to only log errors , like so:
import logging
logging.basicConfig(level=logging.ERROR)
and see if it helps you .
EDIT: the OP fixed the problem i a more concrete way to elasticsearch with :
import logging;
logging.getLogger('elasticsearch').level = logging.ERROR
both ways works.

For me the provided solution didn't work, since the starktrace is generated by urllib3. So for me this worked out:
import logging
logging.getLogger("urllib3").setLevel(logging.ERROR)

Troubleshooting Python API Error Message (Highrise API)

I ran this script (https://github.com/mikehking/highrise-analysis/blob/master/highrise-analysis.py) to calculate usage by user for the Highrise CRM system (www.highrisehq.com). Recently, the script has been returning this error; but I'm not sure what it is telling me -- can anyone help with where I should be looking to fix this?
Thanks
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 557, in urlopen
body=body, headers=headers)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.4/http/client.py", line 1065, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.4/http/client.py", line 1103, in _send_request
self.endheaders(body)
File "/usr/lib/python3.4/http/client.py", line 1061, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 906, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 841, in send
self.connect()
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 133, in connect
ssl_version=resolved_ssl_version)
File "/usr/lib/python3/dist-packages/urllib3/util.py", line 619, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 364, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 578, in __init__
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 805, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 330, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 590, in urlopen
raise SSLError(e)
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:598)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mikehking/workspace/highrise-analysis/highrise-analysis.py", line 188, in <module>
Create_Notes_Backup(PROD_API_KEY, PROD_API_USR, 'highrise-production-notes.bak', 'highrise-production-users.bak', 'highrise-production-people.bak', 'highrise-production-cases.bak', trailing_days = 365) # Production Environment
File "/home/mikehking/workspace/highrise-analysis/highrise-analysis.py", line 71, in Create_Notes_Backup
tmp_notes = high.get_person_notes(person.highrise_id)
File "/usr/local/lib/python3.4/dist-packages/highton/highton.py", line 436, in get_person_notes
return self._get_notes(subject_id, 'people')
File "/usr/local/lib/python3.4/dist-packages/highton/highton.py", line 433, in _get_notes
highrise_type, subject_id)), Note)
File "/usr/local/lib/python3.4/dist-packages/highton/highton.py", line 115, in _get_data
content = self._get_request(endpoint, params).content
File "/usr/local/lib/python3.4/dist-packages/highton/highton.py", line 44, in _get_request
params=params,
File "/usr/lib/python3/dist-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 455, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 558, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:598)

The last traceback suggests that the script is trying to Create_Notes_Backup, but encounters an error when communicating with the Highrise API.
The error message:
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)
is consistent with a server that accepts your connection but then closes it immediately or prematurely. This might be a firewall issue.
If you have OpenSSL installed on this machine, you can try something like this to check the connection:
openssl s_client -crlf -connect USER.highrisehq.com:443
where USER is your Highrise user. You should normally get a long TLS handshake trace full of technical details, ending with --- and then it waits for your input.