Develop Reference

How do I import a python library while behind a proxy?

I am trying to import a python library using:
import cenpy as cp
but I get an error message:
ConnectionError: HTTPSConnectionPool(host='api.census.gov', port=443): Max retries exceeded with url: /data.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000013167B552B0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
I have had this issue before while calling a website. It has to do with the proxy settings. I resolved those other issues using code like this:
import requests
s = requests.Session()
s.proxies = {
"https":"https://user:pass#server:port",
"http":"http://user:pass#server:port"
}
and then:
s.get('http://web.address')
Is there anyway to implement the request session so that I am able to import the library?
Using Python 3.9.12

So I did some more digging and found out the library does place a call to the API during import. There seems to be a workaround for this but it is not implemented their code yet. I tried a few more things and I wanted to share what worked for me. You have to make sure that the code below runs prior to importing the library making the call. This code should allow for all other call/get requests to run through the proxy without having to use a requests session.
The snippets below will set the proxy environment variables
import os
os.environ['http_proxy'] = 'http://<user>:<pass>#<proxy>:<port>'
os.environ['https_proxy'] = 'http://<user>:<pass>#<proxy>:<port>'
Or to be more thorough:
import os
proxy = 'http://<user>:<pass>#<proxy>:<port>'
os.environ['http_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy
Remember that this should be at the very top of your script, or at least prior to any connection requests. Also, make sure you are using the correct IP address for the proxy, as that tripped me up as well.
Credit goes here and here.

Using Proxies to Handle HTTPS Connection Requests

I'm trying to run code on a site that only accepts HTTPS connections, and am having trouble incorporating it with proxies.
I run code such as this to instantiate the proxy:
os.environ['https_proxy'] = 'http://' + proxy
And when I try to complete requests using said previously implemented proxy (I'm going through the site's API), I always get this error:
HTTPSConnectionPool(host=[ . . . ], port=443): Max retries exceeded with url: [. . .] (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fab996ef790>: Failed to establish a new connection: [Errno -2] Name or service not known',)))
The question I have is to of course how to alleviate the error, though more primarily, when a HTTPS connection is forced, what are the ways to work around it so you're not completely stopped from utilizing or maneuvering around the site (with proxies)?

The proxy server's host name can not be resolved to an IP address.
Either there is a problem with the proxy's host name, or there is a problem with the DNS server. If you are sure that the host is correct, try using its IP address, e.g.
proxy = '192.168.1.1:1234'
os.environ['https_proxy'] = 'http://' + proxy
If that works then the proxy is OK, but the name resolution is failing for some reason. Try using curl and see if that works, e.g.
https_proxy='https://localhost:1234' curl -v https://httpbin.org

requests.exceptions.SSLError: hostname 'boxfwd.com' doesn't match either of 'nycmsk.com', 'www.nycmsk.com'

Very simple example on python:
import requests
c = requests.get(u'https://boxfwd.com').content
print c
And on my local computer all works fine.
But on server I see this error:
requests.exceptions.SSLError: hostname 'boxfwd.com' doesn't match either of 'nycmsk.com', 'www.nycmsk.com'
Why I see this error on server ?
In browser I see certificate to *.boxfwd.com

It seems that in your server another domain (nycmsk.com) is also hosted and requests picks up that certificate.
Look here for a potential solution: https://2.python-requests.org/en/master/community/faq/#what-are-hostname-doesn-t-match-errors
Also probably duplicate with: using requests with TLS doesn't give SNI support

Same type of error I was getting while integrating my HAProxy with Datadog for monitoring.
Stacktrace:
haproxy
-------
- instance #0 [ERROR]: '(\'Connection aborted.\', BadStatusLine("\'\'",))'
- Collected 0 metrics, 0 events & 0 service checks
The reason was my EC2 box url was accessible with https that causes ssl was enable.After adding 'disable_ssl_validation: true' in my haproxy.yaml in data-agent/conf.d/haproxy.yaml it worked

Logging into a website which uses Microsoft ForeFront "Thread Management Gateway"

I want to use python to log into a website which uses Microsoft Forefront, and retrieve the content of an internal webpage for processing.
I am not new to python but I have not used any URL libraries.
I checked the following posts:
How can I log into a website using python?
How can I login to a website with Python?
How to use Python to login to a webpage and retrieve cookies for later usage?
Logging in to websites with python
I have also tried a couple of modules such as requests. Still I am unable to understand how this should be done, Is it enough to enter username/password? Or should I somehow use the cookies to authenticate? Any sample code would really be appreciated.
This is the code I have so far:
import requests
NAME = 'XXX'
PASSWORD = 'XXX'
URL = 'https://intra.xxx.se/CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=3'
def main():
# Start a session so we can have persistant cookies
session = requests.session()
# This is the form data that the page sends when logging in
login_data = {
'username': NAME,
'password': PASSWORD,
'SubmitCreds': 'login',
}
# Authenticate
r = session.post(URL, data=login_data)
# Try accessing a page that requires you to be logged in
r = session.get('https://intra.xxx.se/?t=1-2')
print r
main()
but the above code results in the following exception, on the session.post-line:
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='intra.xxx.se', port=443): Max retries exceeded with url: /CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=3 (Caused by <class 'socket.error'>: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)
UPDATE:
I noticed that I was providing wrong username/password.
Once that was updated I get a HTTP-200 response with the above code, but when I try to access any internal site I get a HTTP 401 response. Why Is this happening? What is wrong with the above code? Should I be using the cookies somehow?

TMG can be notoriously fussy about what types of connections it blocks. The next step is to find out why TMG is blocking your connection attempts.
If you have access to the TMG server, log in to it, start the TMG management user-interface (I can't remember what it is called) and have a look at the logs for failed requests coming from your IP address. Hopefully it should tell you why the connection was denied.
It seems you are attempting to connect to it over an intranet. One way I've seen it block connections is if it receives them from an address it considers to be on its 'internal' network. (TMG has two network interfaces as it is intended to be used between two networks: an internal network, whose resources it protects from threats, and an external network, where threats may come from.) If it receives on its external network interface a request that appears to have come from the internal network, it assumes the IP address has been spoofed and blocks the connection. However, I can't be sure that this is the case as I don't know what this TMG server's internal network is set up as nor whether your machine's IP address is on this internal network.

Python urllib2 giving "network unreachable error" if the URL is https

I am trying to fetch some urls using urllib2 library.
a = urllib2.urlopen("http://www.google.com")
ret = a.read()
Code above is working fine, and giving expected result. But when I make the url https, it gives "network unreachable" error
a = urllib2.urlopen("https://www.google.com")
urllib2.URLError: <urlopen error [Errno 101] Network is unreachable>
Is there any problem with ssl? My python version is Python2.6.5. I am also behind an academic proxy server. I have the settings in bash file. Anyway, since http is opening proxy shouldn't be the problem here.

Normally the issue in cases like this is the proxy you are behind having an out of date or untrusted SSL certificate. urllib is fussier than most browsers when it comes to SSL and this is why you might be getting this error.

The http url didn't give error because http_proxy variable was set already. By setting https_proxy the above error disappears.
export http_proxy = "http://{proxy-address}"
Set samething for https_proxy
export https_proxy = "http://{proxy-address}"