Edit: I don't know if this is a thing, but often this happens when i've been testing a lot and I have a lot of python scripts running. Usually none active at the same time but I'll have ran a few of them in a short space of time. I don't know if that's why teh error starts happening.
Ok here's my dilemma.
I've got a NordVPN account and I want to randomly loop through the ip's for the requests i'm making to google. This works fine .. sometimes. For some reason after a few request pings my program starts slowing down and then I start getting 'Max Connection Exceeded' errors and as far as I can tell the ip's are fine.
I started thinking that google was blocking the ip but then I request to other website and the same error continues.. I started thinking that maybe the proxy server only allows a certain amount of requests in a certain time frame so I tried some free proxies but I get the same errors again..
Either way if I leave it alone for a while it seems to work fine and then a few dozen pings later and the errors start again.
The only think I can think of and not sure how I can test or even if it's possible to test remotely (work router) is that maybe some open connections remain open and affects my code and then when the backlog of open connections are thrown out I can resume as normal. I started looking for if I needed to "close" my connection after my get request but it doesn't seem necessary (although I wasn't even sure what I was looking for).
This is one version of my code, I've tried without sessions, I've tried a different way of writing the sessions. All seem to work in the same way:
import requests, random
username = 'xxx'
password = 'yyy'
with open('proxies.txt', 'r') as p:
proxy_lost = p.read().splitlines()
with open('data.txt', 'r') as d:
data = d.read().splitlines()
for i in data:
result = None
while result is None:
try:
proxy = proxy_list[random.randint(0,len(proxy_list)-1)] + '.nordvpn.com'
prox = 'socks5://{}:{}#{}:{}'.format(username, password, proxy, 1080) #I've also tried socks5h
with requests.Session() as r:
r.proxies['http': prox]
r.proxies['https': prox]
result = r.get('http://icanhazip.com')
print(result.text.strip())
If anyone has any idea whatsoever any help is appreciated. I've reached a point where i'm struggling to think of new ideas to try.
This is an example of one of the errors I get during this whole process:
Traceback (most recent call last): File
"C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py",
line 809, in connect
negotiate(self, dest_addr, dest_port) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py",
line 444, in _negotiate_SOCKS5
self, CONNECT, dest_addr) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py",
line 503, in _SOCKS5_request
raise SOCKS5AuthError("SOCKS5 authentication failed") socks.SOCKS5AuthError: SOCKS5 authentication failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\contrib\socks.py",
line 88, in _new_conn
**extra_kw File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py",
line 209, in create_connection
raise err File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py",
line 199, in create_connection
sock.connect((remote_host, remote_port)) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py",
line 47, in wrapper
return function(*args, **kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py",
line 814, in connect
raise GeneralProxyError("Socket error", error) socks.GeneralProxyError: Socket error: SOCKS5 authentication failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py",
line 600, in urlopen
chunked=chunked) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py",
line 354, in _make_request
conn.request(method, url, **httplib_request_kw) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py",
line 1244, in request
self._send_request(method, url, body, headers, encode_chunked) File
"C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py",
line 1290, in _send_request
self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py",
line 1239, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked) File
"C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py",
line 1026, in _send_output
self.send(msg) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py",
line 966, in send
self.connect() File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py",
line 181, in connect
conn = self._new_conn() File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\contrib\socks.py",
line 110, in _new_conn
"Failed to establish a new connection: %s" % error urllib3.exceptions.NewConnectionError:
:
Failed to establish a new connection: SOCKS5 authentication failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py",
line 449, in send
timeout=timeout File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py",
line 638, in urlopen
_stacktrace=sys.exc_info()[2]) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\retry.py",
line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError:
SOCKSHTTPConnectionPool(host='icanhazip.com', port=80): Max retries
exceeded with url: / (Caused by
NewConnectionError(': Failed to establish a new connection: SOCKS5
authentication failed'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users*\Documents_Scripts\Find
Proxies\FindProxies.py", line 23, in
result = requests.get('http://icanhazip.com', proxies=proxies) File
"C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py",
line 75, in get
return request('get', url, params=params, **kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py",
line 60, in request
return session.request(method=method, url=url, **kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py",
line 533, in request
resp = self.send(prep, **send_kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py",
line 646, in send
r = adapter.send(request, **kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py",
line 516, in send
raise ConnectionError(e, request=request) requests.exceptions.ConnectionError:
SOCKSHTTPConnectionPool(host='icanhazip.com', port=80): Max retries
exceeded with url: / (Caused by
NewConnectionError(': Failed to establish a new connection: SOCKS5
authentication failed'))
Are you also closing the sessions again after use?
Try with:
r = requests.session(config={'keep_alive': False})
Inspired by:
python-requests-close-http-connection
Related
I am trying to download a zip file that is stored here:
http://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N45W074.SRTMGL1.hgt.zip
If you paste this into the browser and hit enter, it will download the .zip folder.
If you inspect the browser while this is happening, you will see that there is an internal redirect going on:
And eventually the zip gets downloaded.
I am trying to automate this downloading using the python requests library by doing the following:
import requests
requests.get(url,
allow_redirects=True,
headers={'User-Agent':'Chrome/107.0.0.0'})
I've tried tons of combinations, using the full header string from the HTML inspection, forcing verify=True, with and without redirects, adding a HTTPBasicAuth user/pass that says is required although the file seems to download fine without any credentials.
Honestly no clue what I'm missing, this is not my expertise. I keep getting this error:
>>> requests.get(url,
... allow_redirects=True,
... headers={'User-Agent':'Chrome/107.0.0.0'})
Traceback (most recent call last):
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
raise err
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine
actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 1037, in _send_output
self.send(msg)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 975, in send
self.connect()
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connection.py", line 205, in connect
conn = self._new_conn()
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x0000016B76A6FE50>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\adapters.py", line 489, in send
resp = conn.urlopen(
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='e4ftl01.cr.usgs.gov', port=80): Max retries exceeded with url: /MEASURES/SRTMGL1.003/2000.02.11/N45W074.SRTMGL1.hgt.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000016B76A6FE50>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='e4ftl01.cr.usgs.gov', port=80): Max retries exceeded with url: /MEASURES/SRTMGL1.003/2000.02.11/N45W074.SRTMGL1.hgt.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000016B76A6FE50>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
Can someone help me arrive at the code that will result in a successful request? I know how to write the request into a zip afterwards..
Change http to Https: This should work
import requests
# download zip file from url
url = "https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N45W074.SRTMGL1.hgt.zip"
r = requests.get(url)
with open("N45W074.SRTMGL1.hgt.zip", "wb") as f:
f.write(r.content)
Thanks to #cnemri for the tip.
It was a combination of changing http to https and also including this specific cookie header in the request header:
headers = {'Cookie': '_gid=GA1.2.48775707.1669266346; _hjSessionUser_606685=eyJpZCI6IjQ1MzEzM2QzLTI3MWEtNWM0YS04M2YzLWRmMmMzNDk4NjY1ZSIsImNyZWF0ZWQiOjE2NjkyNjYzNDU2MjUsImV4aXN0aW5nIjp0cnVlfQ==; ERS_production_2=b7dfade669180a6d6d250b030ffc3cf2UZZa8%2F471MfcV%2FaNuFtu6Pli%2BpP8jKOIwR4JvQjm%2B6DLPjl679vVf5SCDk7C5TLQsj7qckIev6lmtGb6Mes5RKDHUs%2BBp3EAjKW2%2BMCUpV%2Fnx0z1pdaCQQ%3D%3D; EROS_SSO_production_secure=eyJjcmVhdGVkIjoxNjY5MjY5MjEzLCJ1cGRhdGVkIjoiMjAyMi0xMS0yMyAyMzo1MzoyOCIsImF1dGhUeXBlIjoiRVJTIiwiYXV0aFNlcnZpY2UiOiJFUk9TIiwidmVyc2lvbiI6MS4xLCJzdGF0ZSI6ImVjMDY3YmMzYmRhMzBiZTUyNTkxYTNiZTYwMTMwZWNmNjAwMWU1Y2JlMGMxZmNkYTU4Y2Y4OTY0YjRlNTJkOTEiLCJpZCI6InY5U1Q4YVl3M1hRKCsyIiwic2VjcmV0IjoiPl8lMCZPYiY9MUVkclRLZTt4fHVZLVcyU1VvTiA1SE17KiQqaG12OSxvPl5%2BdyJ9; _ga_0YWDZEJ295=GS1.1.1669269458.1.0.1669269460.0.0.0; _ga=GA1.2.1055277358.1669266346; _ga_71JPYV1CCS=GS1.1.1669306300.2.1.1669306402.0.0.0; DATA=Y3_gg9uD6LmunsfDHneR9wAAARQ'}
then, this worked:
requests.get(url, headers=headers)
I'm not sure if this is the "solution" or just a workaround I've discovered. Still appreciate any input. I think it may just be credentials that are stored?
When sending a request with authentication, I get a requests.exceptions.SSLError error which you can See below.
proxies = { 'https' : "http://user:pass#ip:port/" }
url = "https://httpbin.org/ip"
numberResponse = requests.get(url,proxies=proxies).text
print(numberResponse)
The requests.exceptions.SSLError
Traceback (most recent call last):
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connection.py", line 359, in connect
conn = self._connect_tls_proxy(hostname, conn)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connection.py", line 496, in _connect_tls_proxy
return ssl_wrap_socket(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\ssl_.py", line 428, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\ssl_.py", line 472, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1125)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\retry.py", line 573, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /ip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1125)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:/Users/K_Yuk/OneDrive/Desktop/Gmail generator/test.py", line 15, in <module>
numberResponse = requests.get(url,proxies=proxies).text
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /ip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1125)')))
So then I tired verify=False as one of the requests.get() parameters but then get a requests.exceptions.ProxyError error which you can see below :
proxies = { 'https' : "http://user:pass#10.10.1.10:3128/"}
url = "https://httpbin.org/ip"
numberResponse = requests.get(url,proxies=proxies,verify=False).text
print(numberResponse)
The requests.exceptions.ProxyError
Traceback (most recent call last):
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connection.py", line 359, in connect
conn = self._connect_tls_proxy(hostname, conn)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connection.py", line 496, in _connect_tls_proxy
return ssl_wrap_socket(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\ssl_.py", line 428, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\ssl_.py", line 472, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\retry.py", line 573, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /ip (Caused by ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:/Users/K_Yuk/OneDrive/Desktop/Gmail generator/test.py", line 15, in <module>
numberResponse = requests.get(url,proxies=proxies,verify=False).text
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\K_Yuk\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /ip (Caused by ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory')))
I tired to look every for the answer but nothing seems to work. I can't send a request with a proxy with
authentication. Any ideas?
The problem is very likely not the authentication. Unfortunately, you don't provide details of the proxy configuration and the URL you use for the proxy. The only thing you provide is:
proxies = { 'https' : eampleIpWithAuth }
Based on the reference to _connect_tls_proxy in the stacktrace the eampleIpWithAuth is very likely something like https://..., i.e. you try to access the proxy itself over HTTPS. Note that accessing a proxy over HTTPS is different from using a HTTP proxy for HTTPS. When accessing a HTTPS URL over a HTTPS proxy one essentially does double encryption to the proxy:
client --- [HTTPS wrapped inside HTTPS] --- proxy --- [HTTPS] --- server
Whereas with a HTTPS URL over a "normal" HTTP proxy there is only single encryption, i.e. it looks (simplified) like this:
client --- [HTTPS wrapped inside HTTP] --- proxy --- [HTTPS] --- server
Very likely the proxy you want to use is a plain HTTP proxy, and not a HTTPS proxy. This is actually the most common case.
The error happens since the proxy is not able to speak TLS but gets accessed by TLS. The fix is to use http://proxy and not https://proxy as the proxy address. Note that the latter worked in older versions of Python since proxy over HTTPS was not supported and a value of https:// for the protocol was treated the same as http://.
I had a similar problem with my script in python 3.9 version.
When I try to get some data from web server, I had exception like Your:
urllib3.exceptions.MaxRetryError
Simple add this line to configure Your requests
import requests
import urllib3
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL:#SECLEVEL=1'
I hope, this help You to solve Your issue
I have tried to find the right answer for my situation, but it seems it may not exists yet.
In my current training course (very beginner level) I have an exercise to get some weather json info from a url (Open Weather in this case). Currently, this is the only way I have been taught to do this:
url = "http://api.openweathermap.org/data/2.5/weather?q="+city+"&appid="+api_key
request = requests.get(url)
json = request.json()
print(json)
I have my "city" and "api_key" parameters set as instructed, and have been able to successfully run my code and get the json info when connected to my personal home network. However, when I connect to my work network, physically at the office or via VPN, I get this magical wall:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 169, in _new_conn
conn = connection.create_connection( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\connection.py",
line 96, in create_connection
raise err File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\connection.py",
line 86, in create_connection
sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 699, in urlopen
httplib_response = self._make_request( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 394, in _make_request
conn.request(method, url, **httplib_request_kw) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers) File
"C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 1257,
in request
self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line
1303, in _send_request
self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 1252,
in endheaders
self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line
1012, in _send_output
self.send(msg) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 952, in
send
self.connect() File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 200, in connect
conn = self._new_conn() File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 181, in _new_conn
raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000255A4B11370>:
Failed to establish a new connection: [WinError 10060] A connection
attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\adapters.py",
line 439, in send
resp = conn.urlopen( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 755, in urlopen
retries = retries.increment( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\retry.py",
line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError:
HTTPConnectionPool(host='api.openweathermap.org', port=80): Max
retries exceeded with url:
/data/2.5/weather?q=*REDACTED*&appid=*REDACTED* (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x00000255A4B11370>: Failed to establish a new connection: [WinError
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"c:\Users\*REDACTED*\Training\Code\Python\weather.py", line 7, in
<module>
request = requests.get(url) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\api.py",
line 75, in get
return request('get', url, params=params, **kwargs)
return session.request(method=method, url=url, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\sessions.py",
line 542, in request
resp = self.send(prep, **send_kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\sessions.py",
line 655, in send
r = adapter.send(request, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\adapters.py",
line 516, in send
raise ConnectionError(e, request=request) requests.exceptions.ConnectionError:
HTTPConnectionPool(host='api.openweathermap.org', port=80): Max
retries exceeded with url:
/data/2.5/weather?q=*REDACTED*&appid=*REDACTED* (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x00000255A4B11370>: Failed to establish a new connection: [WinError
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond')) PS
C:\Users\*REDACTED*\Training\Code\Python&
C:/Users/*REDACTED*/Python/Python39/python.exe
"c:/Users/*REDACTED*/Training/Code/Python/weather.py" Traceback (most
recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 169, in _new_conn
conn = connection.create_connection( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\connection.py",
line 96, in create_connection
raise err File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\connection.py",
line 86, in create_connection
sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 699, in urlopen
httplib_response = self._make_request( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 394, in _make_request
conn.request(method, url, **httplib_request_kw) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers) File
"C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 1257,
in request
self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line
1303, in _send_request
self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 1252,
in endheaders
self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line
1012, in _send_output
self.send(msg) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 952, in
send
self.connect() File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 200, in connect
conn = self._new_conn() File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 181, in _new_conn
raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000229035A1370>:
Failed to establish a new connection: [WinError 10060] A connection
attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\adapters.py",
line 439, in send
resp = conn.urlopen( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 755, in urlopen
retries = retries.increment( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\retry.py",
line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError:
HTTPConnectionPool(host='api.openweathermap.org', port=80): Max
retries exceeded with url:
/data/2.5/weather?q=*REDACTED*&appid=*REDACTED* (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x00000229035A1370>: Failed to establish a new connection: [WinError
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"c:\Users\*REDACTED*\Training\Code\Python\weather.py", line 7, in
<module>
request = requests.get(url) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\api.py",
line 75, in get
return request('get', url, params=params, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\api.py",
line 61, in request
return session.request(method=method, url=url, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\sessions.py",
line 542, in request
resp = self.send(prep, **send_kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\sessions.py",
line 655, in send
r = adapter.send(request, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\adapters.py",
line 516, in send
raise ConnectionError(e, request=request) requests.exceptions.ConnectionError:
HTTPConnectionPool(host='api.openweathermap.org', port=80): Max
retries exceeded with url:
/data/2.5/weather?q=*REDACTED*&appid=*REDACTED* (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x00000229035A1370>: Failed to establish a new connection: [WinError
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond'))
Quick ref of facts/actions taken:
-Tried setting the proxy globally to all Python instances via the set proxy... command.
-Not using a venv.
-Tried to add the --proxy... argument at the end of the command before executing via the the Visual Studio Code terminal.
-On Windows 10 Enterprise.
I am unable to move to the next module of the course because I need this code to work so that I know how to fix the error in the next set of codes I will have to do. I also apologize if this is a lot of words to get through (I am a noob for a reason, I guess), but I really appreciate anyone who could guide me.
For Requests proxy set in own manner Proxies in Requests
Thanks to the help of a colleague, I was able to better interpret the link that Сергей Зайков referenced. It was a proxy issue after all.
After making my adjustment in the code, I was able to modify it by adding the lines below and updating the request.get
http_proxy = "http://*REDACTED*:8080"
proxyDict = {"http" : http_proxy}
request = requests.get(url, proxies=proxyDict)
I am now able to run my code as intended.
For a while, i have been trying to use cloud9 to give me an extra hand on repetitive tasks on my job. However, the majority of those tasks have web scraping involved, and I am having a hard time using the urllib library.
It does not work even for simple codes. It keeps running forever, and I can't find the reasons for that. I will appreciate some tips...
from urllib.request import urlopen
import json
import ssl
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = ('Enter - ')
html = urlopen(url, context=ctx).read()
str_data = open(html).read()
json_data = json.loads(str_data)
for entry in json_data:
name = entry[0];
title = entry[1];
print((name, title))
Without providing much debug information, one can only assume that you're probably timing out, since you're not providing the urlopen function with a timeout.
To avoid this, put a reasonable timeout to your urlopen function so that it doesn't run indefinitely.
Do yourself a favor and replace that with Requests. Your example simplifies to:
import requests
url = "..."
json_data = requests.get(url).json()
for entry in json_data:
name = entry[0]
title = entry[1]
print(name, title)
That also removes a whole awful lot of potential for mistakes from your code.
New code
import requests
import json
url = "-"
json_data = requests.get(url).json()
info = json.loads(json_data)
print('User count:', len(info))
print(info)
Debug Mode:
[IKP3db-g] 15:43:52,971733 - INFO - IKP3db 1.4.1 - Inouk Python Debugger for CPython 3.6+
[IKP3db-g] 15:43:52,972714 - INFO - IKP3db listening on 127.0.0.1:15471
[IKP3db-g] 15:43:53,003016 - INFO - Connected with 127.0.0.1:50936
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connection.py", line 334, in connect
conn = self._new_conn()
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fa2bb9729e8>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ot.cloud.mapsfinancial.com', port=443): Max retries exceeded with url: /pegasus/api/consulta/caixa/saldos/carteira/DVG1%20FIA/2018-01-08 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa2bb9729e8>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/ikp3db.py", line 2047, in main
File "/usr/local/lib/python3.6/dist-packages/ikp3db.py", line 1526, in _runscript
exec(statement, globals, locals)
File "<string>", line 1, in <module>
File "/home/ubuntu/environment/Testing.py", line 5, in <module>
json_data = requests.get(url).json()
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='ot.cloud.mapsfinancial.com', port=443): Max retries exceeded with url: xxxxxxxx (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa2bb9729e8>: Failed to establish a new connection: [Errno 110] Connection timed out',))[IKP3db-g] 15:46:04,668520 - INFO - Uncaught exception. Entering post mortem debugging
I'm trying to use python-gitlab to list all my gitlab projects. But when I use the following code to list part of my projects in gitlab, I got list index out of range error.
# file name: poc.py
import gitlab
gl = gitlab.Gitlab(
'http://my-gitlab.com:10080',
private_token='<my_private_token>'
)
projects = gl.projects.list(all=True)
for project in projects:
print(project.attributes['name'])
My gitlab server has more than 300 projects.
The following is the complete error message:
Traceback (most recent call last):
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/gitlab/__init__.py", line 783, in next
item = self._data[self._current]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x107784be0>: Failed to establish a new connection: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/MyUserName/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='my-gitlab.com', port=80): Max retries exceeded with url: /api/v4/projects?membership=false&order_by=created_at&owned=false&page=2&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x107784be0>: Failed to establish a new connection: [Errno 61] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "poc.py", line 15, in <module>
projects = gl.projects.list(as_list=False, all=True)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/gitlab/exceptions.py", line 255, in wrapped_f
return f(*args, **kwargs)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/gitlab/mixins.py", line 133, in list
obj = self.gitlab.http_list(path, **data)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/gitlab/__init__.py", line 597, in http_list
return list(GitlabList(self, url, query_data, **kwargs))
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/gitlab/__init__.py", line 779, in __next__
return self.next()
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/gitlab/__init__.py", line 788, in next
self._query(self._next_url)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/gitlab/__init__.py", line 716, in _query
**kwargs)
File "/Users/MyUserName/.pyenv/versions/3.6.5/lib/python3.6/site-packages/gitlab/__init__.py", line 498, in http_request
result = self.session.send(prepped, timeout=timeout, **settings)
File "/Users/MyUserName/.local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/Users/MyUserName/.local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='my-gitlab.com', port=80): Max retries exceeded with url: /api/v4/projects?membership=false&order_by=created_at&owned=false&page=2&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x107784be0>: Failed to establish a new connection: [Errno 61] Connection refused',))
The error message requests.exceptions.ConnectionError: HTTPConnectionPool(host='my-gitlab.com', port=80): Max retries exceeded says that it uses port 80 instead of 10080 to communicate with gitlab server. This is because there was an IndexError triggered at line 825 of gitlab/init.py. So python-gitlab tried to use port 80 to see if this port works.
I've checked that the length of self._data is 20. And I don't know why self._current was added to 20. self._data[20] triggers IndexError because the index's maximum value is 19.
For now, I use the following workaround to list all the projects:
import gitlab
gl = gitlab.Gitlab(
'http://my-gitlab.com:10080',
private_token='<my_private_token>'
)
current_page = 1
projects = dest_gitlab.projects.list(as_list=False)
total_pages = projects.total_pages
while (current_page <= total_pages):
projects = gl.projects.list(as_list=False, per_page=100, page=current_page)
for project in projects:
print(project.attributes['name'])
current_page += 1
But this is inconvenient to list all projects.
The document here doesn't mention anything about this error.
Anyone know how to list all projects using python-gitlab?
Is this a bug of python-gitlab?
Is len(self._data) == 20 an expected behavior when calling gl.projects.list(all=True)?
The reason for python-gitlab using port 80 to communicate with gitlab server is because this gitlab bug.
For now, I can only use the workaround or wait for gitlab to fix this bug.
import gitlab
gl = gitlab.Gitlab(
'http://my-gitlab.com:10080',
private_token='<my_private_token>'
)
current_page = 1
projects = dest_gitlab.projects.list(as_list=False)
total_pages = projects.total_pages
while (current_page <= total_pages):
projects = gl.projects.list(as_list=False, per_page=100, page=current_page)
for project in projects:
print(project.attributes['name'])
current_page += 1