I have created an .exe file with the help of pyinstaller module, the problem resides when I perform a request to an endpoint via the .exe using https proxies which throws me an error:
requests.exceptions.ProxyError: HTTPSConnectionPool(host='www.lustmexico.com', port=443): Max retries exceeded with url: /products.json (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1059: The handshake operation timed out')))
But instead, when I execute the request via my main.py file (e.g. the main entry point of the program, using python files, not .exe converted yet) no error happens
Here's how my proxies are configured:
ip = "IP OF MY PROXY"
proxies = {
"https": 'https://' + ip,
"http": 'http://' + ip
}
return proxy
And the way I perform the request is:
r = requests.get(self.link + "/products.json", proxies=proxies, headers=headers, timeout=timeout)
At first instance I guessed was the timeout, but its to high now and I have tested and is not, for sure, the main error cause
After doing my long research I found that there was an error on my https proxies or SSL installed in my machine but not really sure about that, yet I don't understand the problem, please help
Related
I try to use or change IP address and do web scraping on public data accessible by all but I can't find a solution. I am trying to do a rotating ip address. I'm on Windows 10 and the Anaconda IDE.
For example, i execute this code below:
import requests
domain = "https://www.undernews.fr"
#define your proxies
#the socks5h method allows the socks server to translate the #hostname. So make sure that you add 'socks5h'.
proxies = {
'http': 'socks5h://127.0.0.1:9050',
'https': 'socks5h://127.0.0.1:9050'
}
a = requests.get(domain.strip(), proxies=proxies).text
print(a)
And my kernel return:
File "C:\Users\FirstName\anaconda3\lib\site-packages\requests\adapters.py", line 519, in send
raise ConnectionError(e, request=request)
ConnectionError: SOCKSHTTPSConnectionPool(host='www.undernews.fr', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x000001CF0D4EA3A0>: Failed to establish a new connection: [WinError 10061] Aucune connexion n’a pu être établie car l’ordinateur cible l’a expressément refusée'))
It looks like those proxies have been blocked:
Translated part of your error:
"No connection could be established because the target computer expressly refused it"
Maybe try looking into using proxyscrape to get a list of "free" proxies. Or at least try with a different proxy IP
I tried to use proxy with requests library
import requests
proxies = {'https': 'http://xxx.xxx.xxx.xx:yyyy',
'http': 'http://xx.xxx.xxx.xxx:yyyy'}
r = requests.get('https://www.instagram.com', proxies=proxies)
print(r.status_code)
and faced this problem:
requests.exceptions.ProxyError: HTTPSConnectionPool(host='www.wikipedia.org', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000013CB6D8D610>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond')))
I researched many different sites and solutions to this problem but nothing helped.
Then I started asking questions: "How does a proxy work", "How to choose a proxy?".
For my project, I need several (maybe even several dozen different proxies), so buying was not my option. (I used public proxies, correct me,
if it is possible to buy one proxy or vpn account, so that it is not one permanent proxy adress, but many different ones)
Also, in the process of searching for an answer, I was faced with a strange (in my opinion) reaction of the program to changing the source of the Internet on the computer. From router, public wi-fi and mobile internet got different error results. How is this possible?
You should try this
Your code
proxies = {'https': 'http://xxx.xxx.xxx.xx:yyyy',
'http': 'http://xx.xxx.xxx.xxx:yyyy'}
New (remove http and https preffix in proxies dict)
proxies = {'https': 'xxx.xxx.xxx.xx:yyyy',
'http': 'xx.xxx.xxx.xxx:yyyy'}
I also had a similar error, generally that error occurs for HTTPS validation, you can try adding the parameter Verify = False
r = requests.get('https://www.instagram.com', proxies=proxies, Verify=False)
I have this host: http://retsau.torontomls.net:7890/and I want to access http://retsau.torontomls.net:7890/rets-treb3pv/server/login, how can I accomplish this using Python Requests? All my attempts till now have failed.
I also followed the solution here - Python Requests - Use navigate site by servers IP and came up with this -
response = requests.get(http://206.152.41.279/rets-treb3pv/server/login, headers={'Host': retsau.torontomls.net})
but that resulted in this error:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='206.152.41.279', port=80): Max retries exceeded with url: /rets-treb3pv/server/login (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10a4f6d84>: Failed to establish a new connection: [Errno 60] Operation timed out',))
Funny thing is everything seems to work perfectly fine on Postman, I am able to access all sorts of URLs on that server, from logging in to searching for something.
You left out the port number (7890) from the URL to your get call:
response = requests.get('http://206.152.41.279:7890/rets-treb3pv/server/login', headers={'Host': 'retsau.torontomls.net'})
# ^^^^ Add this
Also, unless you actually have a specific reason for accessing the site by IP address, it would make more sense to put the FQDN in the URL rather than the Host header:
response = requests.get('http://retsau.torontomls.net:7890/rets-treb3pv/server/login')
I try to use https proxy in python like this:
proxiesDict ={
'http': 'http://' + proxy_line,
'https': 'https://' + proxy_line
}
response = requests.get('https://api.ipify.org/?format=json', proxies=proxiesDict, allow_redirects=False)
proxy_line is a proxy read from file in the format of ip:port. I checked this https proxy in browser and it works. But in python this code hangs for a few seconds and then i get exception:
HTTPSConnectionPool(host='api.ipify.org', port=443): Max retries exceeded with url: /?format=json (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0425E450>: Failed to establish a new connection: [WinError 10060]
I tried to use socks5 proxy, and it works on socks5 proxies with a PySocks installed. But for https i get this exception, can someone help me
When specifying a proxy list for requests, the key is the protocol, and the value is the domain/ip. You don't need to specify http:// or https:// again, for the actual value.
So, your proxiesDict will be:
proxiesDict = {
'http': proxy_line,
'https': proxy_line
}
You can also configure proxies by setting the enviroment variables:
$ export HTTP_PROXY="http://proxyIP:PORT"
$ export HTTPS_PROXY="http://proxyIP:PORT"
Then, you only need to execute your python script without proxy request.
Also, you can configure your proxy with http://user:password#host
For more information see this documentation: http://docs.python-requests.org/en/master/user/advanced/
Try using pycurl this function may help:
import pycurl
def pycurl_downloader(url, proxy_url, proxy_usr):
"""
Download files with pycurl
the proxy configuration:
proxy_url = 'http://10.0.0.0:3128'
proxy_usr = 'user:password'
"""
c = pycurl.Curl()
c.setopt(pycurl.FOLLOWLOCATION, 1)
c.setopt(pycurl.MAXREDIRS, 5)
c.setopt(pycurl.CONNECTTIMEOUT, 30)
c.setopt(pycurl.AUTOREFERER, 1)
if proxy_url: c.setopt(pycurl.PROXY, proxy_url)
if proxy_usr: c.setopt(pycurl.PROXYUSERPWD, proxy_usr)
content = StringIO()
c.setopt(pycurl.URL, url)
c.setopt(c.WRITEFUNCTION, content.write)
try:
c.perform()
c.close()
except pycurl.error, error:
errno, errstr = error
print 'An error occurred: ', errstr
return content.getvalue()
I'm trying to run code on a site that only accepts HTTPS connections, and am having trouble incorporating it with proxies.
I run code such as this to instantiate the proxy:
os.environ['https_proxy'] = 'http://' + proxy
And when I try to complete requests using said previously implemented proxy (I'm going through the site's API), I always get this error:
HTTPSConnectionPool(host=[ . . . ], port=443): Max retries exceeded with url: [. . .] (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fab996ef790>: Failed to establish a new connection: [Errno -2] Name or service not known',)))
The question I have is to of course how to alleviate the error, though more primarily, when a HTTPS connection is forced, what are the ways to work around it so you're not completely stopped from utilizing or maneuvering around the site (with proxies)?
The proxy server's host name can not be resolved to an IP address.
Either there is a problem with the proxy's host name, or there is a problem with the DNS server. If you are sure that the host is correct, try using its IP address, e.g.
proxy = '192.168.1.1:1234'
os.environ['https_proxy'] = 'http://' + proxy
If that works then the proxy is OK, but the name resolution is failing for some reason. Try using curl and see if that works, e.g.
https_proxy='https://localhost:1234' curl -v https://httpbin.org