I want to set proxy in requests and get content of url source:
privoxy = 'localhost:8118'
proxy_dict = {'http':privoxy, 'https':privoxy, 'ftp':privoxy, 'ssl':privoxy}
content = session.get(url=url, proxies=proxy_dict).content
The following error:
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 467, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 305, in send
conn = self.get_connection(request.url, proxies)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 206, in get_connection
except_on_missing_scheme(proxy)
File "/usr/lib/python2.7/dist-packages/requests/utils.py", line 636, in except_on_missing_scheme
raise MissingSchema('Proxy URLs must have explicit schemes.')
requests.exceptions.MissingSchema: Proxy URLs must have explicit schemes.
Set the scheme to http, in your case:
privoxy = 'http://localhost:8118'
(It can be others of course, like 'https', 'ftp', etc.)
More from Requests docs for Proxies:
Note that proxy URLs must include the scheme.
Related
I've been trying to authenticate from a server using PyDrive. I'm trying to use a proxy but I keep getting a 403 Forbidden error. I'm not sure if my code for using the proxy is correct, or if this is even possible.
The error:
Traceback (most recent call last):
File "/Users/user/Desktop/Python/files/test_post.py", line 67, in <module>
file1.Upload(param={'supportsAllDrives': True, "http": gauth.http})
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pydrive/files.py", line 285, in Upload
self._FilesInsert(param=param)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pydrive/auth.py", line 75, in _decorated
return decoratee(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pydrive/files.py", line 368, in _FilesInsert
metadata = self.auth.service.files().insert(**param).execute(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
return wrapped(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 901, in execute
_, body = self.next_chunk(http=http, num_retries=num_retries)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
return wrapped(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 1006, in next_chunk
resp, content = _retry_request(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/googleapiclient/http.py", line 190, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/oauth2client/transport.py", line 173, in new_request
resp, content = request(orig_request_method, uri, method, body,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/oauth2client/transport.py", line 280, in request
return http_callable(uri, method=method, body=body, headers=headers,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httplib2/__init__.py", line 1701, in request
(response, content) = self._request(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httplib2/__init__.py", line 1421, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httplib2/__init__.py", line 1343, in _conn_request
conn.connect()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httplib2/__init__.py", line 1133, in connect
sock.connect((self.host, self.port))
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httplib2/socks.py", line 512, in connect
self.__negotiatehttp(destpair[0], destpair[1])
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httplib2/socks.py", line 465, in __negotiatehttp
raise HTTPError((statuscode, statusline[2]))
httplib2.socks.HTTPError: (403, b'Forbidden')
I should note: the script seems to work fine up until it gets to the file upload command with PyDrive. I've tried passing http into the params and tried it without it. They both don't work.
Here is the code for setting up the proxy:
proxy_info = httplib2.ProxyInfo(proxy_type=httplib2.socks.PROXY_TYPE_HTTP_NO_TUNNEL,
proxy_host='myproxyhost',
proxy_port=8080)
print("Proxy info variable set")
gauth.http = httplib2.Http(proxy_info=proxy_info)
print("gauth.http is set")
# Try to load saved client credentials
gauth.LoadCredentialsFile("mycreds.txt")
if gauth.credentials is None:
# Authenticate if they're not there
print("Gauth credentials is none conditional")
gauth.GetFlow()
gauth.flow.params.update({'access_type': 'offline'})
gauth.flow.params.update({'approval_prompt': 'force'})
gauth.CommandLineAuth()
elif gauth.access_token_expired:
print("Gauth access token expired conditional.")
# Refresh them if expired
gauth.Refresh()
else:
print("Gauth authroized conditional")
# Initialize the saved creds
gauth.CommandLineAuth()
print("Commandlineauth allowed!")
gauth.Authorize()```
headers = {'Accept': 'application/json',}
response = requests.get(self.url, headers=headers, verify=self.ssl)
Getting error when passing False as input from command line. It works fine when verify=False is set in the code without passing it as input.
Error:
response = requests.get(self.api, headers=headers, verify=self.ssl)
File "/usr/lib/python3/dist-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 407, in send
self.cert_verify(conn, request.url, verify, cert)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 226, in cert_verify
"invalid path: {0}".format(cert_loc))
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: False
parser.add_argument('--ssl', default=True, action="store_false")
this question is duplicate. but i am still asking because i am not finding any solution from given answers. so guys don't make it duplicate, instead help me to solve this problem.
i am running my server as https using bottle(paste). when trying to access from java and postman , it's verifying correctly and response coming.
versions used :
python 2.7.13
requests.version
'2.11.1'
import cryptography
cryptography.version
'2.1.3'
import OpenSSL
OpenSSL.version
'17.3.0'
but unfortunately when trying to access from python modules - it's throwing issue. ?
requests.post(url, verify=_certfile, proxies=proxies) #tried #failed to verify certificate
urllib2.urlopen(url, data=data, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
cafile=None, capath=None, cadefault=False, context=context) #tried #failed to verify certificate
import httplib2
http = httplib2.Http(disable_ssl_certificate_validation=True)
headers, content = http.request(url, "POST", body=data, headers=headers)#tried #failed to verify certificate
httplib.HTTPSConnection(host='10.201.41.50', port=8018, key_file=key_file, cert_file=cert_file, strict=None, context=context) #tried #failed to verify certificate
and my server is running using paste server using bottle framework.
ssl_cxt = SSL.Context(SSL.SSLv23_METHOD)
...
...
ssl_cxt.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT, verify_cb)
paste.httpserver.serve(app, host=IP, port=PORT, server_version=" ", ssl_context=ssl_cxt)
issues i am facing (different with each lib)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "Continuum\Anaconda2\lib\site-packages\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: unknown error (_ssl.c:2947)
============
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "Continuum\Anaconda2\lib\site-packages\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:661)
File "Continuum\Anaconda2\lib\httplib.py", line 1038, in endheaders
self._send_output(message_body)
File "Continuum\Anaconda2\lib\httplib.py", line 882, in _send_output
self.send(msg)
File "Continuum\Anaconda2\lib\httplib.py", line 844, in send
self.connect()
File "Continuum\Anaconda2\lib\httplib.py", line 1262, in connect
self.sock = self._context.wrap_socket(self.sock,
AttributeError: 'Context' object has no attribute 'wrap_socket'
I tried to solve using urllib2, socket and ssl. i am able to hit the server. i tried with .pkcs12 but seems urllib2 expects only .pem files. so will advice try to create client certificate with .pem extension specifically when you try to access from python.
import ssl
import socket
import urllib2
import json
url = "https://localhost:port/hello/"
_certfile_pem = "ssl_client.pem"
key_file = "ssl_key.pem"
cert_file = "ssl_cert.pem"
headers = {"Content-Type":"application/json"}
data = {}
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.verify_mode = ssl.CERT_REQUIRED
context.load_cert_chain(certfile=cert_file, keyfile=key_file)
context.load_verify_locations(_certfile_pem)
context.wrap_socket(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) #
socket overwrite required.
try:
req = urllib2.Request(url,data=json.dumps(data))
response = urllib2.urlopen(req, context=context)
content = response.read()
print content
except urllib2.HTTPError as e:
content = e.read()
print "HTTPError : ", e
I am trying to make a GET request over TOR which is listening on 127.0.0.1:9050
I've installed request socks: pip install -U requests[socks]
import requests
tor_proxy = {'http': 'socks5://127.0.0.1:9050', 'https': 'socks5://127.0.0.1:9050'}
r = requests.get("http://www.google.com", proxies=tor_proxy, timeout=20)
Here is the error from when I run the code
Traceback (most recent call last):
r = requests.get("http://www.google.com", proxies=tor_proxy, timeout=20)
File "C:\Python27\lib\site-packages\requests\api.py", line 71, in get
return request('get', url, params=params, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 585, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 403, in send
timeout=timeout
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 578, in urlopen
chunked=chunked)
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 362, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Python27\lib\httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 1097, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 1053, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 897, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 859, in send
self.connect()
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connection.py", line 167, in connect
conn = self._new_conn()
File "C:\Python27\lib\site-packages\requests\packages\urllib3\contrib\socks.py", line 81, in _new_conn
**extra_kw
File "C:\Python27\lib\site-packages\socks.py", line 195, in create_connection
sock.connect((remote_host, remote_port))
File "C:\Python27\lib\site-packages\socks.py", line 747, in connect
negotiate(self, dest_addr, dest_port)
File "C:\Python27\lib\site-packages\socks.py", line 419, in _negotiate_SOCKS5
CONNECT, dest_addr)
File "C:\Python27\lib\site-packages\socks.py", line 482, in _SOCKS5_request
resolved = self._write_SOCKS5_address(dst, writer)
File "C:\Python27\lib\site-packages\socks.py", line 517, in _write_SOCKS5_address
addr_bytes = socket.inet_pton(family, host)
AttributeError: 'module' object has no attribute 'inet_pton'
Jun 11 13:13:55.000 [notice] Tried for 120 seconds to get a connection to [scrubbed]:0. Giving up. (waiting for socks info)
Thanks
The problem is with the library itself, in particular PySocks socks.py, inet_pton is only available on unix:
socket.inet_pton(address_family, ip_string)
Convert an IP address from its family-specific string format to a packed, binary format. inet_pton() is useful when a library or network protocol calls for an object of type struct in_addr (similar to inet_aton()) or struct in6_addr.
Supported values for address_family are currently AF_INET and AF_INET6. If the IP address string ip_string is invalid, socket.error will be raised. Note that exactly what is valid depends on both the value of address_family and the underlying implementation of inet_pton().
Availability: Unix (maybe not all platforms).
One workaround is to install win_inet_pton and import it in your script, the method will automatically be added to the socket lib.
I am trying to send a request with the header: ":hello". However, the leading colon causes the script to not function properly, and emit this traceback:
Traceback (most recent call last):
(first few lines removed for my privacy)
File "C:\Python27\lib\site-packages\requests\api.py", line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 370, in send
timeout=timeout
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 353, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Python27\lib\httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 1096, in _send_request
self.putheader(hdr, value)
File "C:\Python27\lib\httplib.py", line 1030, in putheader
raise ValueError('Invalid header name %r' % (header,))
ValueError: Invalid header name ':hello'
Is there a workaround for this? My script is:
import requests
headers = {'user-agent': 'alsotesting', ':hello': 'test'}
requests.post("my server", headers=headers)
As your error says, :header is not a valid HTTP header name (you cannot start headers with ":" - see documentation). You should change
headers = {'user-agent': 'alsotesting', ':hello': 'test'}
to
headers = {'user-agent': 'alsotesting', 'hello': 'test'}
Edit: HTTP/2 uses pseudo-headers fields, which start with a colon (see documentation). Also, as explained here, you may see some headers starting with a colon in Chrome's Developer Tools, which can happen when Chrome is talking to a web server using SPDY - and also HTTP/2 (which is based on SPDY/2), which correspond to pseudo-headers. As stated in documentation, pseudo-header fields are not HTTP header fields.
In conclusion, header fields starting with a colon are not allowed with standard HTTP protocol, so that is why you are getting the Invalid header name error