I am having trouble using python's request package to submit GET request to
Puppet 3.7's REST API. I have looked at this documentation here:
http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification
But I am still having trouble. Here is my script:
[root#ppt-001 RESTClients]# cat add-group.py
#!/usr/bin/env python
import requests
# curl https://ppt-001.example.com:4433/classifier-api/v1/groups \
# -H "Content-Type: application/json" \
# --cert /etc/puppetlabs/puppet/ssl/certs/ppt-001.example.com.pem \
# --key /etc/puppetlabs/puppet/ssl/private_keys/ppt-001.example.com.pem \
# --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem | python -m json.tool
url='https://ppt-001.example.com:4433/classifier-api/v1/groups'
headers = {"Content-Type": "application/json"}
data={}
cacert='/etc/puppetlabs/puppet/ssl/certs/ca.pem'
key='/etc/puppetlabs/puppet/ssl/private_keys/ppt-001.example.com.pem'
cert='/etc/puppetlabs/puppet/ssl/certs/ppt-001.example.com.pem'
result = requests.get(url,
data=data, #no data needed for this request
headers=headers, #dict {"Content-Type":"application/json"}
cert=(cacert,key), #key/cert pair
verify=cert
)
print result.json()
I am using this version of python:
[root#ppt-001 RESTClients]# python -V
Python 2.7.5
Here is what happens when I execute my script:
[root#ppt-001 RESTClients]# ./add-group.py
/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Traceback (most recent call last):
File "./add-group.py", line 21, in <module>
verify=cert
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
[root#ppt-001 RESTClients]#
I suspect that requests does not like the self-signed cert that Puppet uses, but if I issue this curl command ...
curl https://ppt-001.example.com:4433/classifier-api/v1/groups \
-H "Content-Type: application/json" \
--cert /etc/puppetlabs/puppet/ssl/certs/ppt-001.example.com.pem \
--key /etc/puppetlabs/puppet/ssl/private_keys/ppt-001.example.com.pem \
--cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem | python -m json.tool
... everything works fine.
UPDATE:
I have install requests[security]:
[root#ppt-001 RESTClients]# pip install requests[security]
Requirement already satisfied (use --upgrade to upgrade): requests[security] in /usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg
Installing extra requirements: 'security'
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/lib64/python2.7/site-packages (from requests[security])
Downloading/unpacking ndg-httpsclient (from requests[security])
Downloading ndg_httpsclient-0.4.0.tar.gz
Running setup.py egg_info for package ndg-httpsclient
Downloading/unpacking pyasn1 (from requests[security])
Downloading pyasn1-0.1.7.tar.gz (68kB): 68kB downloaded
Running setup.py egg_info for package pyasn1
Installing collected packages: ndg-httpsclient, pyasn1
Running setup.py install for ndg-httpsclient
Skipping installation of /usr/lib/python2.7/site-packages/ndg/__init__.py (namespace package)
Installing /usr/lib/python2.7/site-packages/ndg_httpsclient-0.4.0-py2.7-nspkg.pth
Installing ndg_httpclient script to /usr/bin
Running setup.py install for pyasn1
Successfully installed ndg-httpsclient pyasn1
Cleaning up...
But now I get this putput when I run my script:
[root#ppt-001 RESTClients]# ./add-group.py
Traceback (most recent call last):
File "./add-group.py", line 25, in <module>
verify=cert
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
self._validate_conn(conn)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
conn.connect()
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/packages/urllib3/connection.py", line 238, in connect
ssl_version=resolved_ssl_version)
File "/usr/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py", line 260, in ssl_wrap_socket
ctx.use_privatekey_file(keyfile)
OpenSSL.SSL.Error: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
cert=(cacert,key), #key/cert pair
verify=cert
....
OpenSSL.SSL.Error: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
I think you need to use (cert,key) as cert and use cacert instead for verification:
cert=(cert,key), #key/cert pair
verify=cacert
Related
When searching or installing packages with pip on MacOS (10.11.6), I am getting an SSL error similar to the one reproduced below.
The problem occurs for both python2 and python3. I acquired the versions (2.7.14 and 3.6.5) via www.python.org as Frameworks. The version of pip was 9.0.1. The problem is new (April 2018) and seems related to an update of OSX.
How to recover from this problem?
$ pip2 search numpy
Exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/search.py", line 45, in run
pypi_hits = self.search(query, options)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/search.py", line 62, in search
hits = pypi.search({'name': query, 'summary': query}, 'or')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1243, in __call__
return self.__send(self.__name, args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1602, in __request
verbose=self.__verbose
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 775, in request
headers=headers, stream=True)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 522, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 386, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 47, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)
There are two ways to recover from this problem.
Solution 1. As can be read here, Python.org sites stopped the support for older TLS versions, which breaks pip < 9.0.3. A new release of pip was provided, but because one cannot simply update via the broken pip, one has to bootstrap the installation. This worked for me:
# For python2
curl https://bootstrap.pypa.io/get-pip.py | python2
# For python3 (haven't tested this myself)
curl https://bootstrap.pypa.io/get-pip.py | python3
Credits for these commands go back to this SO post.
Solution 2. For Python 3.6 frameworks, a script Install Certificates.command is deployed and installed under /Applications/Python 3.6/. Executing it resolved the problem, but only for pip3.
You just need to reinstall Python
brew reinstall python
Python 2.7.10
I'm making a very basic API call to my gitlab repo.
I can successfully curl the endpoint and I can succesfully call it with Postman
However it fails when trying to use Python's request module.
Curl Works:
curl https://gitlab.example.com/api/v4/projects
Outputs the expected JSON response from the endpoint (no auth needed)
Postman I also get the expected result:
However when I submit via requests:
[Cert is self signed, hence the verify=False]
r = requests.get("https://gitlab.example.com/api/v4/projects", verify=False)
Error thrown:
Traceback (most recent call last):
File "./repo_analyzer.py", line 36, in <module>
r = requests.get("https://xxxxxxxxxxxx/api/v4/projects", verify=False)
File "/Library/Python/2.7/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))
I'm a bit at a loss on why it's not working with python, but works otherwise.
After searching a ton of blogs and posts about this, it seemed that I needed to update requests[security] package through pip.
I was getting errors running both pip install -U requests[security] and sudo pip install -U requests[security] so I started from scratch:
Downloaded Python 2.7 and reinstalled from the official site
pip install requests
pip install -U requests
pip install -U requests[security]
After that, the calls were successful.
On Ubuntu 14.04, I install python, pip, setuptools, virtualenv (as root).
As a regular user, I run
$ pip list
apt-xapian-index (0.45)
chardet (2.0.1)
pip (8.0.2)
python-apt (0.9.3.5ubuntu2)
python-debian (0.1.21-nmu2ubuntu2)
requests (2.2.1)
setuptools (19.7)
six (1.5.2)
ssh-import-id (3.21)
urllib3 (1.7.1)
virtualenv (14.0.5)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
And then I can create a virtualenv
$ virtualenv fooenv
And everything is right with the world! The problem comes about that if I switch to root user and try to create a virtual environment
$ sudo su -
$ virtualenv fooenv
New python executable in /root/fooenv/bin/python
Installing setuptools, pip, wheel...
Complete output from command /root/fooenv/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel:
Collecting setuptools
/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/commands/install.py", line 299, in run
requirement_set.prepare_files(finder)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/req/req_set.py", line 359, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/req/req_set.py", line 576, in _prepare_file
session=self.session, hashes=hashes)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/download.py", line 809, in unpack_url
hashes=hashes
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/download.py", line 648, in unpack_http_url
hashes)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/download.py", line 841, in _download_http_url
stream=True,
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/download.py", line 377, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
SSLError: [Errno 185090050] _ssl.c:344: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
File "/usr/local/bin/virtualenv", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 703, in main
symlink=options.symlink)
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 904, in create_environment
download=download,
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 861, in install_wheel
call_subprocess(cmd, show_stdout=False, extra_env=env)
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 781, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /root/fooenv/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel failed with error code 2
So what gives? Why can my regular user create a virtualenv, but not my root user?
I'm a begginer in Python and have been struggling with downloading a file using the package requests. My code works for an url with http, but not for https. The code is the following:
import requests
url = 'https://firms.modaps.eosdis.nasa.gov/active_fire/shapes/zips/Central_America_24h.zip'
print "downloading with requests"
r = requests.get(url)
with open("Central_America_24h.zip", "wb") as code:
code.write(r.content)
I get the following errors:
Warning (from warnings module):
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\packages\urllib3\util\ssl_.py", line 90
InsecurePlatformWarning
InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
Traceback (most recent call last):
File "C:\Users\lmixco\Downloads\descarga.py", line 8, in <module>
r = requests.get(url)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\api.py", line 68, in get
return request('get', url, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 594, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 196, in resolve_redirects
**adapter_kwargs
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol
I have been looking for a solution, but have not found anything. Any help will be appreciated. Thanks in advance.
you need to install a package , use this
pip install requests[security]
for debain / ubuntu , install this before installing above package
apt-get install python-dev libffi-dev libssl-dev
Trying to install pip on a new python installation. I am stuck with proxy errors. Looks like a bug in get-pip or urllib3??
Question is do I have to go through the pain of setting up CNTLM as described here or is there a shortcut?
get-pip.py documentation says use --proxy="[user:passwd#]proxy.server:port" option to specify proxy and relevant authentication. But seems like pip passes on the whole thing as it is to urllib3 which interprets "myusr" as the url scheme, because of the ':' I guess (?).
C:\ProgFiles\Python27>get-pip.py --proxy myusr:mypswd#111.222.333.444:80
Downloading/unpacking pip
Cleaning up...
Exception:
Traceback (most recent call last):
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\basecommand.py", line 122, in main
status = self.run(options, args)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\commands\install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\req.py", line 1177, in prepare_files
url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\index.py", line 194, in find_requirement
page = self._get_page(main_index_url, req)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\index.py", line 568, in _get_page
session=self.session,
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\index.py", line 670, in get_page
resp = session.get(url, headers={"Accept": "text/html"})
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\sessions.py", line 468, in get
return self.request('GET', url, **kwargs)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\download.py", line 237, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\adapters.py", line 305, in send
conn = self.get_connection(request.url, proxies)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\adapters.py", line 215, in get_connection
block=self._pool_block)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\packages\urllib3\poolmanager.py", line 258, in proxy_fro
m_url
return ProxyManager(proxy_url=url, **kw)
File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\packages\urllib3\poolmanager.py", line 214, in __init__
'Not supported proxy scheme %s' % self.proxy.scheme
AssertionError: Not supported proxy scheme myusr
Storing debug log for failure in C:\Users\myusr\pip\pip.log
C:\ProgFiles\Python27>
When I run the command without the usrname and password it works fine, but proxy rejects the request saying it needs authentication ("407 authenticationrequired").
C:\ProgFiles\Python27>get-pip.py --proxy 111.222.333.444:80
Downloading/unpacking pip
Cannot fetch index base URL https://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement pip
Cleaning up...
No distributions at all found for pip
Storing debug log for failure in C:\Users\sg0219898\pip\pip.log
C:\ProgFiles\Python27>cat C:\Users\sg0219898\pip\pip.log
------------------------------------------------------------
C:\ProgFiles\Python27\get-pip.py run on 09/29/14 16:23:26
Downloading/unpacking pip
Getting page https://pypi.python.org/simple/pip/
Could not fetch URL https://pypi.python.org/simple/pip/: connection error: ('Cannot connect to proxy.', error('Tunnel connection failed: 407 authenticationrequired',))
Will skip URL https://pypi.python.org/simple/pip/ when looking for download links for pip
Getting page https://pypi.python.org/simple/
Could not fetch URL https://pypi.python.org/simple/: connection error: ('Cannot connect to proxy.', error('Tunnel connection failed: 407 authenticationrequired',))
Will skip URL https://pypi.python.org/simple/ when looking for download links for pip
Cannot fetch index base URL https://pypi.python.org/simple/
URLs to search for versions for pip:
* https://pypi.python.org/simple/pip/
Getting page https://pypi.python.org/simple/pip/
Could not fetch URL https://pypi.python.org/simple/pip/: connection error: ('Cannot connect to proxy.', error('Tunnel connection failed: 407 authenticationrequired',))
Will skip URL https://pypi.python.org/simple/pip/ when looking for download links for pip
Could not find any downloads that satisfy the requirement pip
Cleaning up...
Removing temporary dir c:\users\sg0219~1\appdata\local\temp\pip_build_SG0219898...
No distributions at all found for pip
Exception information:
Traceback (most recent call last):
File "c:\users\sg0219~1\appdata\local\temp\tmp36ynxd\pip.zip\pip\basecommand.py", line 122, in main
status = self.run(options, args)
File "c:\users\sg0219~1\appdata\local\temp\tmp36ynxd\pip.zip\pip\commands\install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "c:\users\sg0219~1\appdata\local\temp\tmp36ynxd\pip.zip\pip\req.py", line 1177, in prepare_files
url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
File "c:\users\sg0219~1\appdata\local\temp\tmp36ynxd\pip.zip\pip\index.py", line 277, in find_requirement
raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for pip
C:\ProgFiles\Python27>
I had a brief look at urllib3\poolmanager.py and it doesn't seem to have anything to do with username/passwords.
This is complaining about the scheme for the URL (which urlparse is understanding to be myusr), to work around that you should instead be doing:
get-pip.py --proxy http://myusr:mypswd#111.222.333.444:80
This is because the script requires the environment variables http_proxy or https_proxy contains the scheme in the URL. Set the environment variables
export http_proxy="http://<hostname>:<port>"
export https_proxy="https://<hostname>:<port>"
before run the "python get-pip.py"
import requests
proxy = {
'http' : 'http://138.197.222.35:80',
'https' : 'http://1138.197.222.35:8080'
}
r = requests.get('http://httpbin.org/ip', proxies=proxy)
print (r)
Append 'http://' and 'https://' with your ips.
I had this issue when working with linux, windows worked fine with ip alone.
Use pip install xxx --proxy=https://xx.xx.xx.xx:xxxx
After Python 3.6, when using the proxy, the parameter value of proxies in requests.get(url=url, headers=headers, proxies=...) changed.
Before 3.6 includes, proxies={ 'https': '127.0.0.1:8080'} or proxies={'http': '127.0.0.1:8080'} is fine, but this type of dictionary is not suitable for Python 3.7 and above.
In Python3.7 and above, you must add http:// or https:// in front of ip:port, that is, proxies={'http':'http://127.0.0.1:8080'} or proxies={'https':'https://127.0.0.1:8080'}
For the problem you have mentioned, it depends on how the proxy server authentication has been set up.
For example, my intranet uses Windows AD and probably the proxy server is using Windows Integrated auth. Thus when I do pip install --proxy http://<server-ip>:<port> <module-name>, it works fine. Note that I did not have to type in username & password ,likely due to integrated auth.
So, you'll need to find out the authentication being used by your proxy server.You can use Fiddler(or any other network analyzer) tool to check the WWW-Authenticate headers in the 407 response, to check what auth mechs the server supports.
This can be a case when you use ";" in your passwords. This cannot be parsed properly that results in ProxySchemeUnknown error raised.
The below things worked for me :
Step 1) Set HTTPS_PROXY and HTTP_PROXY:
export HTTPS_PROXY="http://proxy-address:8080"
export HTTP_PROXY="http://proxy-address:8080"
step 2) : Run pip with below configuration
./pip.exe install pyspark --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host pypi.org --proxy "proxy-address:8080"
On Windows you can use the SET command instead of export:
SET HTTP_PROXY="http://proxy-address:8080"
SET HTTPS_PROXY="http://proxy-address:8080"
If you are not behind a proxy, you need to unset like this:
SET HTTP_PROXY=
SET HTTPS_PROXY=
it works to install boto3 on win7.
pip install boto3 --proxy=https://user:pwd#x.x.x.x:8080
Collecting boto3
Downloading boto3-1.20.33-py3-none-any.whl (131 kB)
Collecting botocore<1.24.0,>=1.23.33
Downloading botocore-1.23.33-py3-none-any.whl (8.5 MB)
Collecting s3transfer<0.6.0,>=0.5.0
Downloading s3transfer-0.5.0-py3-none-any.whl (79 kB)
Collecting jmespath<1.0.0,>=0.7.1
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
Downloading urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, python-dateutil, jmespath, urllib3, botocore, s3transfer, boto3
Successfully installed boto3-1.20.33 botocore-1.23.33 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.0 six-1.16.0 urllib3-1.26.8