I am getting an url with:
r = requests.get("http://myserver.com")
As I can see in the 'access.log' of "myserver.com", the client's system proxy is used. But I want to disable using proxies at all with requests.
The only way I'm currently aware of for disabling proxies entirely is the following:
Create a session
Set session.trust_env to False
Create your request using that session
import requests
session = requests.Session()
session.trust_env = False
response = session.get('http://www.stackoverflow.com')
This is based on this comment by Lukasa and the (limited) documentation for requests.Session.trust_env.
Note: Setting trust_env to False also ignores the following:
Authentication information from .netrc (code)
CA bundles defined in REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE (code)
If however you only want to disable proxies for a particular domain (like localhost), you can use the NO_PROXY environment variable:
import os
import requests
os.environ['NO_PROXY'] = 'stackoverflow.com'
response = requests.get('http://www.stackoverflow.com')
You can choose proxies for each request. From the docs:
import requests
proxies = {
"http": "http://10.10.1.10:3128",
"https": "http://10.10.1.10:1080",
}
requests.get("http://example.org", proxies=proxies)
So to disable the proxy, just set each one to the empty string:
import requests
proxies = {
"http": "",
"https": "",
}
requests.get("http://example.org", proxies=proxies)
Update: Switched from None to "", see comments.
The way to stop requests/urllib from proxying any requests is to set the the no_proxy (or NO_PROXY) environment variable to * e.g. in bash:
export no_proxy='*'
Or from Python:
import os
os.environ['no_proxy'] = '*'
To understand why this works is because the urllib.request.getproxies function first checks for any proxies set in the environment variables (e.g. http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, etc) or if none are set then it will check for system configured proxies using platform specific calls (e.g. On MacOS it will check using the system scutil/configd interfaces, and on Windows it will check the Registry). As mentioned in the comments if any proxy variables are set you can reset them as #udani suggested, or unset them like this from Python:
del os.environ['HTTP_PROXY']
Then when urllib attempts to use any proxies the proxyHandler function it will check for the presence and setting of the no_proxy environment variable - which can either be set to specific hostnames as mentioned above or it can be set the special * value whereby all hosts bypass the proxy.
With Python3, jtpereyda's solution didn't work, but the following did:
proxies = {
"http": "",
"https": "",
}
requests library respects environment variables.
http://docs.python-requests.org/en/latest/user/advanced/#proxies
So try deleting environment variables HTTP_PROXY and HTTPS_PROXY.
import os
for k in list(os.environ.keys()):
if k.lower().endswith('_proxy'):
del os.environ[k]
I implemented #jtpereyda's solution in our production codebase which worked fine on normal successful HTTP requests (200 OK), but this code ended up not working when receiving an HTTP redirect (301 Moved Permamently). Instead use:
requests.get("https://pypi.org/pypi/pillow/9.0.0/json", proxies={"http": "", "https": ""})
For comparison, this line causes a requests.exception.SSLError when behind a proxy (pypi.org tries to redirect us to Pillow with an uppercase P):
requests.get("https://pypi.org/pypi/pillow/9.0.0/json", proxies={"http": None, "https": None})
r = requests.post('https://localhost:44336/api/',data='',verify=False)
I faced the same issue when connecting with localhost to access my .net backend from a Python script with the request module.
I set verify to False, which cancels the default SSL verification.
P.s - above code will throw a warning that can be neglected by below one
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
r=requests.post('https://localhost:44336/api/',data='',verify=False)
For those for which no_proxy="*" doesnt work, try 0.0.0.0/32, that worked for me.
Related
Requests is not honoring the proxies flag.
There is something I am missing about making a request over a proxy with python requests library.
If I enable the OS system proxy, then it works, but if I make the request with just requests module proxies setting, the remote machine will not see the proxy set in requests, but will see my real ip, it is as if not proxy was set.
The bellow example will show this effect, at the time of this post the bellow proxy is alive but any working proxy should replicate the effect.
import requests
proxy ={
'http:': 'https://143.208.200.26:7878',
'https:': 'http://143.208.200.26:7878'
}
data = requests.get(url='http://ip-api.com/json', proxies=proxy).json()
print('Ip: %s\nCity: %s\nCountry: %s' % (data['query'], data['city'], data['country']))
I also tried changing the proxy_dict format:
proxy ={
'http:': '143.208.200.26:7878',
'https:': '143.208.200.26:7878'
}
But still it has not effect.
I am using:
-Windows 10
-python 3.9.6
-urllib 1.25.8
Many thanks in advance for any response to help sort this out.
Ok is working yea !!! .
The credits for solving this goes to (Olvin Rogh) Thanks Olvin for your help and pointing out my problem. I was adding colon ":" inside the keys
This code is working now.
PROXY = {'https': 'https://143.208.200.26:7878',
'http': 'http://143.208.200.26:7878'}
with requests.Session() as session:
session.proxies = PROXY
r = session.get('http://ip-api.com/json')
print(json.dumps(r.json(), indent=2))
appreciate your help here, thanks in advance.
My Problem:
I am using Python's Requests module for get/post requests to a Django REST API behind a work proxy. I am unable to get past the proxy and I encounter an error. I have summarised this below:
Using the following code (what I've tried):
s = requests.Session()
s.headers = {
"User-Agent": [someGenericUserAgent]
}
s.trust_env = False
proxies = {
'http': 'http://[domain]\[userName]:[password]#[proxy]:8080',
'https': 'https://[domain]\[userName]:[password]#[proxy]:8080'
}
os.environ['NO_PROXY'] = [APIaddress]
os.environ['no_proxy'] = [APIaddress]
r = s.post(url=[APIaddress], proxies=proxies)
With this I get an error:
... OSError('Tunnel connection failed: 407 Proxy Authentication Required')))
Additional Context:
This is on a windows 10 machine.
Work uses a "automatic proxy setup" script (.pac), looking at the script there are a number of proxies that will be automatically assigned depending on the IP address of the machine. All of these proxies I have tried under [proxy] above, with the same error.
The above works when I am not running through the work network, and I don't use the additional proxy settings (removing proxies=proxies). i.e on my home network.
I have no issues with a get request via my browser via the proxy to the Django REST API view.
Things I am uncertain about:
I don't know if I am using the right [proxy]. Is there a way to verify this? I have tried using [findMyProxy].com sites, using the ip addresses it still doesn't work.
I don't know if I am using [domain]\[userName] correctly. is a \ correct? my work does use a domain.
I'm certain it is not a requests issue, as trying to do pip install --proxy http://[domain]\[userName]:[password]#[proxy]:8080 someModule bares the same 407 error.
Any help appreciated.
How I came to the solution:
I used curl to establish a <200 response>, after a lot of trial and error, success was:
$ curl -U DOMAIN\USER:PW -v -x http://LOCATION_OF_PAC_FILE --proxy-ntlm www.google.com
Where -U is the domain, user name and password.
-V is verbose, made it easier for debugging.
-x is the proxy, in my case the location to the .pac file. Curl automatically determines the proxy IP from the PAC. Requests does not do this by default (that I know of).
I used curl to determine that my proxy was using ntlm.
www.google.com as an external site to test the proxy auth.
NOTE: only 1 off \ between domain and username.
Trying to make request use ntlm, I found was impossible by default and instead used requests-ntlm2.
The PAC file through request-ntlm2 did not work so I used pypac to autodiscover the PAC file then determined the proxy based on the URL.
The working code is as follows:
from pypac import PACSession
from requests_ntlm2 import (
HttpNtlmAuth,
HttpNtlmAdapter,
NtlmCompatibility
)
username = 'DOMAIN\\USERNAME'
password = 'PW'
# Don't need the following thanks to pypacs
# proxy_ip = 'PROXY_IP'
# proxy_port = "PORT"
# proxies = {
# 'http': 'http://{}:{}'.format(proxy_ip, proxy_port),
# 'https': 'http://{}:{}'.format(proxy_ip, proxy_port)
# }
ntlm_compatibility = NtlmCompatibility.NTLMv2_DEFAULT
# session = requests.Session() <- replaced with PACSession()
session = PACSession()
session.mount(
'https://',
HttpNtlmAdapter(
username,
password,
ntlm_compatibility=ntlm_compatibility
)
)
session.mount(
'http://',
HttpNtlmAdapter(
username,
password,
ntlm_compatibility=ntlm_compatibility
)
)
session.auth = HttpNtlmAuth(
username,
password,
ntlm_compatibility=ntlm_compatibility
)
# Don't need the following thanks to pypacs
# session.proxies = proxies
response = session.get('http://www.google.com')
I am using request library to connect to an api and get data. I have the proxy to connect and it works with AuthenticatedProxy
This is how my code looks.
import requests
s = requests.Session()
url = "https://testapi"
urlkey = “testkey”
proxies = {'https': 'http://<UserName>:<Password>#proxy_url:8080'}
resp = s.get(url, params={'key':urlkey }, proxies = proxies)
content = resp.content
print content
Username and Password is exposed here and I want to avoid that. How can I acheived that ? Can i ask request to use defaultCredentials or 'account that is running the python script' credentials ?
In .net, following config works:
<system.net>
<defaultProxy enabled="true" useDefaultCredentials="true">
<proxy proxyaddress="https://testapi:8080" bypassonlocal="True"/>
</defaultProxy>
</system.net>
Thanks all.
Similar question here.
From the request docs:
You can also configure proxies by setting the environment variables
HTTP_PROXY and HTTPS_PROXY.
So you could avoid revealing the credentials in the script by setting the environment variable instead:
$ export HTTPS_PROXY="http://<UserName>:<Password>#proxy_url:8080"
And then in the script you would use the proxy by just calling:
resp = s.get(url, params={'key':urlkey })
I'm looking to change the proxy that my python script runs from. Would this be possible by say changing the HTTP_PROXY at the beginning and end of my python script like the following?
export HTTP_PROXY="http://user:pass#10.10.1.10:3128/"
I'm doing this because there is a bug in phantomjs/selenium that stops me from being able to put in proxy authentication.
alternatively, is there another way I can set the proxy that my python script runs through at the beginning and end of the script?
That should be possible, using the requests package:
import requests
http_proxy = "http://10.10.1.10:3128/"
https_proxy = "https://user:pass#10.10.1.10:3128/"
ftp_proxy = "http://10.10.1.10:3128/"
proxyDict = { "http" : http_proxy,
"https" : https_proxy,
"ftp" : ftp_proxy}
Then to request info through the proxy:
page = requests.get('https://www.google.com.au/', proxies=proxyDict)
How can I make an HTTP request via a SOCKs proxy (simply using ssh -D as the proxy)? I've tried using requests with SOCK proxies but it doesn't appear to work (I saw this pull request). For example:
proxies = { "http": "socks5://localhost:9999/" }
r = requests.post( endpoint, data=request, proxies=proxies )
It'd be convenient to keep using the requests library, but I can also switch to urllib2 if that is known to work.
Since SOCKS support has been added to requests 2.10.0, it is remarkably simple, and very close to what you have
Install requests[socks]:
$ pip install requests[socks]
Set up your proxies variable, and make use of it:
>>> import requests
>>> proxies = {
"http":"socks5://localhost:9999",
"https":"socks5://localhost:9999"
}
>>> requests.get(
"https://api.ipify.org?format=json",
proxies=proxies
).json()
{u'ip': u'123.xxx.xxx.xxx'}
A few things to note are to not use a / on the end of the proxies URL, and that you can also use socks4:// as the scheme too if the SOCKS server doesn't support SOCKS5.
SOCKS support for requests is still pending. If you want, you can view my Github repository here to see my branch of the Socksipy library. This is the branch that is currently being integrated into requests; it will be some time before requests fully supports it, though.
https://github.com/Anorov/PySocks/
It should work okay with urllib2. Import sockshandler in your file, and follow the example inside of it. You'll want to create an opener like this:
opener = urllib2.build_opener(SocksiPyHandler(socks.PROXY_TYPE_SOCKS5, "localhost", 9050))
Then you can use opener.open(url) and it should tunnel through the proxy.