I'm a begginer in Python and have been struggling with downloading a file using the package requests. My code works for an url with http, but not for https. The code is the following:
import requests
url = 'https://firms.modaps.eosdis.nasa.gov/active_fire/shapes/zips/Central_America_24h.zip'
print "downloading with requests"
r = requests.get(url)
with open("Central_America_24h.zip", "wb") as code:
code.write(r.content)
I get the following errors:
Warning (from warnings module):
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\packages\urllib3\util\ssl_.py", line 90
InsecurePlatformWarning
InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
Traceback (most recent call last):
File "C:\Users\lmixco\Downloads\descarga.py", line 8, in <module>
r = requests.get(url)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\api.py", line 68, in get
return request('get', url, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 594, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 196, in resolve_redirects
**adapter_kwargs
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol
I have been looking for a solution, but have not found anything. Any help will be appreciated. Thanks in advance.
you need to install a package , use this
pip install requests[security]
for debain / ubuntu , install this before installing above package
apt-get install python-dev libffi-dev libssl-dev
Related
I'm a new user of python. I don't know why but requests always throws an InvalidURL exception:
>>> import requests
>>> r = requests.get('https://www.google.es/')
The output:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/requests/models.py", line 380, in prepare_url
scheme, auth, host, port, path, query, fragment = parse_url(url)
File "/usr/lib/python3/dist-packages/urllib3/util/url.py", line 392, in parse_url
return six.raise_from(LocationParseError(source_url), None)
File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://www.google.es/
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/dist-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 516, in request
prep = self.prepare_request(req)
File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 459, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/local/lib/python3.7/dist-packages/requests/models.py", line 314, in prepare
self.prepare_url(url, params)
File "/usr/local/lib/python3.7/dist-packages/requests/models.py", line 382, in prepare_url
raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: https://www.google.es/
This error is independent of the url I give. How do I handle this?
The version of Python is 3.7.7 and 2.23.0 for requests.
Best regards.
You faced Error due to the New version of urllib3 (some users tends to face this issue).
The error is not due to requests but issue is rather in urllib3 (new ver) that gets installed when installing requests 2.21.0+.
To avoid this either try updating urllib3:
python -m pip install --upgrade urllib3
or install the requests v2.21.0:
pip uninstall requests # to remove current version
pip install requests==2.21.0
Just downgrade it to v2.21.0 version
See if you have a hidden character in your URL.
I've wasted tons of time and that was the problem..
this can happen when you copy-paste the URL from somewhere.
it's happen sometimes when the URL is not the valid one. I have this error and after hours found that I have a small spaces between each / on the URL...
so I suggest to write the URL again on requests.get for no get this mistake..
When searching or installing packages with pip on MacOS (10.11.6), I am getting an SSL error similar to the one reproduced below.
The problem occurs for both python2 and python3. I acquired the versions (2.7.14 and 3.6.5) via www.python.org as Frameworks. The version of pip was 9.0.1. The problem is new (April 2018) and seems related to an update of OSX.
How to recover from this problem?
$ pip2 search numpy
Exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/search.py", line 45, in run
pypi_hits = self.search(query, options)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/search.py", line 62, in search
hits = pypi.search({'name': query, 'summary': query}, 'or')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1243, in __call__
return self.__send(self.__name, args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1602, in __request
verbose=self.__verbose
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 775, in request
headers=headers, stream=True)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 522, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 386, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 47, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)
There are two ways to recover from this problem.
Solution 1. As can be read here, Python.org sites stopped the support for older TLS versions, which breaks pip < 9.0.3. A new release of pip was provided, but because one cannot simply update via the broken pip, one has to bootstrap the installation. This worked for me:
# For python2
curl https://bootstrap.pypa.io/get-pip.py | python2
# For python3 (haven't tested this myself)
curl https://bootstrap.pypa.io/get-pip.py | python3
Credits for these commands go back to this SO post.
Solution 2. For Python 3.6 frameworks, a script Install Certificates.command is deployed and installed under /Applications/Python 3.6/. Executing it resolved the problem, but only for pip3.
You just need to reinstall Python
brew reinstall python
When I tried to get the web page with Python Requests module for the first time on Elementary OS, I faced with SSLError. There is simple solution:
Python 3, trying:
import requests
page = requests.get('https://api.github.com/events')
Getting SSLError:
/usr/local/bin/python3 /home/led/PycharmProjects/urllib_p/urllib_p.py
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 549, in urlopen
conn = self._get_conn(timeout=pool_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 251, in _get_conn
return conn or self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 764, in _new_conn
raise SSLError("Can't connect to HTTPS URL because the SSL "
requests.packages.urllib3.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/led/PycharmProjects/urllib_p/urllib_p.py", line 5, in <module>
page = requests.get('https://api.github.com/events')
File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available.
verify=False make no sense (same Error)
Go to Python_directory/Modules (Python-3.6.4/Modules in my case).
Open Setup file with text editor and search for SSL. You will find this:
# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
#SSL=/usr/local/ssl
#_ssl _ssl.c \
# -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
# -L$(SSL)/lib -lssl -lcrypto
Uncomment last four lines.
In terminal: go to Python_directory, then:
sudo ./configure
sudo make
sudo make install
I fixed this with those commands:
sudo cp -f /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/local/lib/
sudo cp -f /usr/lib/x86_64-linux-gnu/libssl.so.1.1 /usr/local/lib/
sudo ln -sfn /usr/lib/x86_64-linux-gnu/libcrypto.a /usr/local/lib/
sudo ln -sfn /usr/lib/x86_64-linux-gnu/libssl.a /usr/local/lib/
sudo ln -sfn /usr/lib/x86_64-linux-gnu/libssl.so.1.1 /usr/local/lib/
Python 2.7.10
I'm making a very basic API call to my gitlab repo.
I can successfully curl the endpoint and I can succesfully call it with Postman
However it fails when trying to use Python's request module.
Curl Works:
curl https://gitlab.example.com/api/v4/projects
Outputs the expected JSON response from the endpoint (no auth needed)
Postman I also get the expected result:
However when I submit via requests:
[Cert is self signed, hence the verify=False]
r = requests.get("https://gitlab.example.com/api/v4/projects", verify=False)
Error thrown:
Traceback (most recent call last):
File "./repo_analyzer.py", line 36, in <module>
r = requests.get("https://xxxxxxxxxxxx/api/v4/projects", verify=False)
File "/Library/Python/2.7/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))
I'm a bit at a loss on why it's not working with python, but works otherwise.
After searching a ton of blogs and posts about this, it seemed that I needed to update requests[security] package through pip.
I was getting errors running both pip install -U requests[security] and sudo pip install -U requests[security] so I started from scratch:
Downloaded Python 2.7 and reinstalled from the official site
pip install requests
pip install -U requests
pip install -U requests[security]
After that, the calls were successful.
I am trying to use pip from virtual enviroment on Linux Ubuntu 14.04 x64.
While using pip from shell as:
pip search bla
it works fine.
But using from virtualenv it shows error:
(proj_testing)uxu#box:~$ pip search bla
/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Exception:
Traceback (most recent call last):
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main
status = self.run(options, args)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/commands/search.py", line 43, in run
pypi_hits = self.search(query, options)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/commands/search.py", line 60, in search
hits = pypi.search({'name': query, 'summary': query}, 'or')
File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/download.py", line 785, in request
headers=headers, stream=True)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 508, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/download.py", line 373, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
It doesn't make sense, anyone got idea?
Try execute pip search bla like superuser
For ubuntu the command is:
sudo pip search bla
From https://urllib3.readthedocs.org/en/latest/security.html:
Certain Python platforms (specifically, versions of Python earlier than 2.7.9) have restrictions in their ssl module that limit the configuration that urllib3 can apply. In particular, this can cause HTTPS requests that would succeed on more featureful platforms to fail, and can cause certain security features to be unavailable. If you encounter this warning, it is strongly recommended you upgrade to a newer Python version, or that you use pyOpenSSL as described in the OpenSSL / PyOpenSSL section.
You probably have a less recent version of python in your virtual environment. Perhaps this is why this error only occurs in the virtual environment.