Develop Reference

Configure proxy settings for requests encapsulated by cs in Python

For example in the requests library you can set proxy configuration explicitely.
With exoscale/cs (a Python client for Apache Cloud Stack) it does not work for me.
Environment variables are set correctly to locally meaningful values accepted locally in other contexts:
'http_proxy': 'x.x.x.x:nnn'
Error:
requests.exceptions.ProxyError: HTTPSConnectionPool(host='yyyy', port=443):
Max retries exceeded with url: /portal/client
Observations so far:
Host to connect is yyyy:443
Proxy is xxxx:nnn (xxxxx is a valid DNS name)
Requests' error is a ProxyError but as message might be read, tries to connect to target hosts directly?
Wait, it seems like cs incorporates requests!
How to tell the encapsulated requests to to use proxy?

As it seems, requesrts does parse OS environment variables.
% export http_proxy="http://hello.test"
% python -c "import urllib.request; print(urllib.request.getproxies())"
{'http': 'http://hello.test'}

Python soap client - connection having issue

I am using Python soap API client Zeep and here is the code that I have written:
from zeep import Client
def myapi(request):
client = Client("https://siteURL.asmx?wsdl")
key = client.service.LogOnUser('myusername', 'mypassord')
print(key)
it is giving me an error as: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
While I am trying below command the URL works well and shows all the services it has
python -mzeep https://siteURL.asmx?wsdl
Please help to understand what is the reason above code is not working.
PS: I couldn't share site URL which I am trying to connect to.
Additional Info: The site/page is accessible only through intranet and I am testing locally from intranet itself.
Traceback error:
Exception Type: ConnectionError at /music/mypersonalapi/
Exception Value: HTTPSConnectionPool(host='URL I have hidden', port=81):
Max retries exceeded with url: /ABC/XYZ/Logon.asmx
(Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0546E770>:
Failed to establish a new connection:
[WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))
Please note: I have removed URL and Host information from my traceback due to confidentiality

What this does:
python -mzeep https://site/url.asmx?wsdl
is:
c = Client("https://site/url.asmx?wsdl")
c.wsdl.dump()
both alternatives are using port 443 since that is the default https port.
From your traceback we see
Exception Value: HTTPSConnectionPool(host='URL I have hidden', port=81):
which would have been similar to
python -mzeep https://site:81/url.asmx?wsdl
I.e. the command line and your code are not connecting to the same address (also note that port values less than 1024 requires system level permissions to use -- in case you're writing/controlling the service too).
The last line does say "..failed because the connected party did not properly respond after a period of time..", but that is not the underlying reason. In line 3 you can read
Max retries exceeded with url: /ABC/XYZ/Logon.asmx
in other words, you've tried (and failed) to log on too many times and the server is probably doubling the time it uses to respond every time you try (a well known mitigation strategy for "things" that fail to login multiple times -- i.e. look like an attack). The extended delay is most likely causing the error message you see at the bottom.
You'll need to wait a while, or reset your account for the service, and if the service is yours then perhaps turn off this feature during development?

Maybe this can help. I had the same connexion problem (Max retries exceeded...). I solved it by increasing the transport timeout.
client = Client(wsdl=wsdl, transport=Transport(session=session, timeout=120))

Connect to .onion websites on tor using python?

Here is the code that i have till now
import socks
import socket
import requests
import json
socks.setdefaultproxy(proxy_type=socks.PROXY_TYPE_SOCKS5, addr="127.0.0.1", port=9050)
socket.socket = socks.socksocket
data = json.loads(requests.get("http://freegeoip.net/json/").text)
and it works fine. The problem is when i use a .onion url it shows error
Failed to establish a new connection: [Errno -2] Name or service not known
After researching a little i found that although the http request is made over tor the resolution still occours over clearnet. What is the proper way so i can also have the domain resolved over tor network to connect to .onion urls ?

Try to avoid the monkey patching if possible. If you're using modern version of requests, then you should have this functionality already.
import requests
import json
proxies = {
'http': 'socks5h://127.0.0.1:9050',
'https': 'socks5h://127.0.0.1:9050'
}
data = requests.get("http://altaddresswcxlld.onion",proxies=proxies).text
print(data)
It's important to specify the proxies using the socks5h:// scheme so that DNS resolution is handled over SOCKS so Tor can resolve the .onion address properly.

There is a more simple solution for this, but therefore you will need Kali Linux. If you have this OS, you can install tor service and kalitorify, start tor service with: sudo service tor start and start kalitorify with sudo kalitorify -t. Now your trafic will be send through tor and you can access .onion sites just as they would be normal sites.

Using Proxies to Handle HTTPS Connection Requests

I'm trying to run code on a site that only accepts HTTPS connections, and am having trouble incorporating it with proxies.
I run code such as this to instantiate the proxy:
os.environ['https_proxy'] = 'http://' + proxy
And when I try to complete requests using said previously implemented proxy (I'm going through the site's API), I always get this error:
HTTPSConnectionPool(host=[ . . . ], port=443): Max retries exceeded with url: [. . .] (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fab996ef790>: Failed to establish a new connection: [Errno -2] Name or service not known',)))
The question I have is to of course how to alleviate the error, though more primarily, when a HTTPS connection is forced, what are the ways to work around it so you're not completely stopped from utilizing or maneuvering around the site (with proxies)?

The proxy server's host name can not be resolved to an IP address.
Either there is a problem with the proxy's host name, or there is a problem with the DNS server. If you are sure that the host is correct, try using its IP address, e.g.
proxy = '192.168.1.1:1234'
os.environ['https_proxy'] = 'http://' + proxy
If that works then the proxy is OK, but the name resolution is failing for some reason. Try using curl and see if that works, e.g.
https_proxy='https://localhost:1234' curl -v https://httpbin.org

How to NOT use a proxy with Python Mechanize

I am currently using Python + Mechanize for retrieving pages from a local server. As you can see the code uses "localhost" as a proxy. The proxy is an instance of the Fiddler2 debug proxy. This works exactly as expected. This indicates that my machine can reach the test_box.
import time
import mechanize
url = r'http://test_box.test_domain.com:8000/helloWorldTest.html'
browser = mechanize.Browser();
browser.set_proxies({"http": "127.0.0.1:8888"})
browser.add_password(url, "test", "test1234")
start_timer = time.time()
resp = browser.open(url)
resp.read()
latency = time.time() - start_timer
However when I remove the browser.set_proxies statement it stops to work. I get an error <"urlopen error [Errno 10061] No connection could be made because the target machine actively refused it>". The point is that I can access the test_box from my machine with any browser. This also indicates that test_box can be reached from my machine.
My suspicion is that this has something to do with Mechanize trying to guess the proper proxy settings. That is: my Browsers are configured to go to a web proxy for any domain but test_domain.com. So I suspect that mechanize tries to use the web proxy while it should actually not use the proxy.
How can I tell mechanize to NOT guess any proxy settings and instead force it to try to connect directly to the test_box?

Argh, found it out myself. The docstring says:
"To avoid all use of proxies, pass an empty proxies dict."
This fixed the issue.