Develop Reference

Timeout Error when using proxy with httplib2 in Python

i have a simple code that uses proxy to submit a "GET" request to google .
import httplib2
http = httplib2.Http(proxy_info = httplib2.ProxyInfo(proxy_type=3,proxy_host=myProxy, proxy_port=myPort))
resp, content = http.request("http://google.com", "GET")
print(resp)
print(content)
For some reason i get a Timeout error :
resp, content = http.request("http://google.com", "GET")
File "C:\Python35\lib\site-packages\httplib2\__init__.py", line 1322, in requ
st
(response, content) = self._request(conn, authority, uri, request_uri, meth
d, body, headers, redirections, cachekey)
File "C:\Python35\lib\site-packages\httplib2\__init__.py", line 1072, in _req
est
(response, content) = self._conn_request(conn, request_uri, method, body, h
aders)
File "C:\Python35\lib\site-packages\httplib2\__init__.py", line 995, in _conn
request
conn.connect()
File "C:\Python35\lib\http\client.py", line 849, in connect
(self.host,self.port), self.timeout, self.source_address)
File "C:\Python35\lib\socket.py", line 711, in create_connection
raise err
File "C:\Python35\lib\socket.py", line 702, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] A connection attempt failed because the connecte
party did not properly respond after a period of time, or established connecti
n failed because connected host has failed to respond
I'm using a valid proxy, but this module doesn't works, Does anybody knows why this happened?

Win Error 10060 means that no connection can be made and that either the host or the connection(proxy) is at fault. Since google is virtually never down we can narrow down the problem to the proxy configuration.
Since you seemingly configured the proxy correctly (assuming you pass the right types and arugments ) you might want to check what SOCKS protocol the server additionally supports.
Either your're not passing valid values as a proxy configuration or it is the server's fault.
You could also try to diagnose with wireshark to see if any packets even make it to the server.

Repeated POST request is causing error "socket.error: (99, 'Cannot assign requested address')"

I have a web-service deployed in my box. I want to check the result of this service with various input. Here is the code I am using:
import sys
import httplib
import urllib
apUrl = "someUrl:somePort"
fileName = sys.argv[1]
conn = httplib.HTTPConnection(apUrl)
titlesFile = open(fileName, 'r')
try:
for title in titlesFile:
title = title.strip()
params = urllib.urlencode({'search': 'abcd', 'text': title})
conn.request("POST", "/somePath/", params)
response = conn.getresponse()
data = response.read().strip()
print data+"\t"+title
conn.close()
finally:
titlesFile.close()
This code is giving an error after same number of lines printed (28233). Error message:
Traceback (most recent call last):
File "testService.py", line 19, in ?
conn.request("POST", "/somePath/", params)
File "/usr/lib/python2.4/httplib.py", line 810, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.4/httplib.py", line 833, in _send_request
self.endheaders()
File "/usr/lib/python2.4/httplib.py", line 804, in endheaders
self._send_output()
File "/usr/lib/python2.4/httplib.py", line 685, in _send_output
self.send(msg)
File "/usr/lib/python2.4/httplib.py", line 652, in send
self.connect()
File "/usr/lib/python2.4/httplib.py", line 636, in connect
raise socket.error, msg
socket.error: (99, 'Cannot assign requested address')
I am using Python 2.4.3. I am doing conn.close() also. But why is this error being given?

This is not a python problem.
In linux kernel 2.4 the ephemeral port range is from 32768 through 61000. So number of available ports = 61000-32768+1 = 28233. From what i understood, because the web-service in question is quite fast (<5ms actually) thus all the ports get used up. The program has to wait for about a minute or two for the ports to close.
What I did was to count the number of conn.close(). When the number was 28000 wait for 90sec and reset the counter.

BIGYaN identified the problem correctly and you can verify that by calling "netstat -tn" right after the exception occurs. You will see very many connections with state "TIME_WAIT".
The alternative to waiting for port numbers to become available again is to simply use one connection for all requests. You are not required to call conn.close() after each call of conn.request(). You can simply leave the connection open until you are done with your requests.

I too faced similar issue while executing multiple POST statements using python's request library in Spark. To make it worse, I used multiprocessing over each executor to post to a server. So thousands of connections created in seconds that took few seconds each to change the state from TIME_WAIT and release the ports for the next set of connections.
Out of all the available solutions available over the internet that speak of disabling keep-alive, using with request.Session() et al, I found this answer to be working which makes use of 'Connection' : 'close' configuration as header parameter. You may need to put the header content in a separte line outside the post command though.
headers = {
'Connection': 'close'
}
with requests.Session() as session:
response = session.post('https://xx.xxx.xxx.x/xxxxxx/x', headers=headers, files=files, verify=False)
results = response.json()
print results
This is my answer to the similar issue using the above solution.

Python YQL package Error

Hi I am new to Python...
I am trying to use YQL using Python.
I installed httplib2-0.7.0, oauth2 and then installed yql package
For this sample code :
import yql
y = yql.Public()
query = 'select * from flickr.photos.search where text="panda" limit 3';
result = y.execute(query)
print result
I got the following error message.
Please help!!
Traceback (most recent call last):
File "test.py", line 4, in
result = y.execute(query)
File "C:\Python27\lib\site-packages\yql-0.7-py2.7.egg\yql__init__.py", line 306, in execute
resp, content = self.http.request(url, http_method)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1436, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey
)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1188, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1123, in _conn_request
conn.connect()
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 890, in connect
self.disable_ssl_certificate_validation, self.ca_certs)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 76, in _ssl_wrap_socket
cert_reqs=cert_reqs, ca_certs=ca_certs)
File "C:\Python27\lib\ssl.py", line 344, in wrap_socket
ciphers=ciphers)
File "C:\Python27\lib\ssl.py", line 119, in init
ciphers)
ssl.SSLError: [Errno 185090050] _ssl.c:336: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
When I tried to use the *twitter python wrapper, I ended up getting the same SSL error.*
Please tell me what to do

I can see two likely issues:
SSL Certificate
I'm not familiar with Python or that library you're using, but the error sounds like it can't verify the SSL certificate. (Possibly because there is no suitable local SSL certificate bundle for authentication.) You may be able to configure it to skip the SSL certificate verification.
YQL Flickr Query
The YQL query is not correct and gives an error in the YQL console:
select * from flickr.photos.search where text="panda" limit 3
Actually, given the age of this question, it may have worked in June 2011. Now the Flickr tables require an API key as well, so the working query would look like:
select * from flickr.photos.search where text="panda" and api_key="insert-your-key-here" limit 3

python httplib Name or service not known

I'm trying to use httplib to send credit card information to authorize.net. When i try to post the request, I get the following traceback:
File "./lib/cgi_app.py", line 139, in run res = method()
File "/var/www/html/index.py", line 113, in ProcessRegistration conn.request("POST", "/gateway/transact.dll", mystring, headers)
File "/usr/local/lib/python2.7/httplib.py", line 946, in request self._send_request(method, url, body, headers)
File "/usr/local/lib/python2.7/httplib.py", line 987, in _send_request self.endheaders(body)
File "/usr/local/lib/python2.7/httplib.py", line 940, in endheaders self._send_output(message_body)
File "/usr/local/lib/python2.7/httplib.py", line 803, in _send_output self.send(msg)
File "/usr/local/lib/python2.7/httplib.py", line 755, in send self.connect()
File "/usr/local/lib/python2.7/httplib.py", line 1152, in connect self.timeout, self.source_address)
File "/usr/local/lib/python2.7/socket.py", line 567, in create_connection raise error, msg
gaierror: [Errno -2] Name or service not known
I build my request like so:
mystring = urllib.urlencode(cardHash)
headers = {"Content-Type": "text/xml", "Content-Length": str(len(mystring))}
conn = httplib.HTTPSConnection("secure.authorize.net:443", source_address=("myurl.com", 443))
conn.request("POST", "/gateway/transact.dll", mystring, headers)
to add another layer to this, it was working on our development server which has httplib 2.6 and without the source_address parameter in httplib.HTTPSConnection.
Any help is greatly appreciated.
===========================================================
EDIT:
I can run it from command line. Apparently this is some sort of permissions issue. Any ideas what permissions I would need to grant to which users to make this happen? Possibly Apache can't open the port?

As an (obvious) heads up, this same error can also be triggered by including the protocol in the host parameter. For example this code:
conn = httplib.HTTPConnection("http://secure.authorize.net", 80, ....)
will also cause the "gaierror: [Errno -2] Name or service not known" error, even if all your networking setup is correct.

gaierror: [Errno -2] Name or service not known
This error often indicates a failure of your DNS resolver. Does ping secure.authorize.net return successful replies from the same server that receives the gaierror? Does the hostname have a typo in it?

The problem ultimately came down to the fact that selinux was stopping apache from getting that port. Disabling selinux fixed the problems. I had an issue later where i didn't have /var/www/.python-eggs/, so MySQLdb was hosing on import. But after a mkdir, it was fixed.

pass the port separately from the host:
conn = httplib.HTTPSConnection("secure.authorize.net", 443, ....)

urllib.urlopen isn't working. Is there a workaround?

I'm getting a getaddress error and after doing some sleuthing, it looks like it might be my corporate intranet not allowing the connection (I'm assuming due to security, although it is strange that IE works but won't allow Python to open a url). Is there a safe way to get around this?
Here's the exact error:
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
b = urllib.urlopen('http://www.google.com')
File "C:\Python26\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python26\lib\urllib.py", line 203, in open
return getattr(self, name)(url)
File "C:\Python26\lib\urllib.py", line 342, in open_http
h.endheaders()
File "C:\Python26\lib\httplib.py", line 868, in endheaders
self._send_output()
File "C:\Python26\lib\httplib.py", line 740, in _send_output
self.send(msg)
File "C:\Python26\lib\httplib.py", line 699, in send
self.connect()
File "C:\Python26\lib\httplib.py", line 683, in connect
self.timeout)
File "C:\Python26\lib\socket.py", line 498, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno 11001] getaddrinfo failed
More info: I also get this error with urllib2.urlopen

You probably need to fill in proxy information.
import urllib2
proxy_handler = urllib2.ProxyHandler({'http': 'http://yourcorporateproxy:12345/'})
proxy_auth_handler = urllib2.HTTPBasicAuthHandler()
proxy_auth_handler.add_password('realm', 'host', 'username', 'password')
opener = urllib2.build_opener(proxy_handler, proxy_auth_handler)
opener.open('http://www.stackoverflow.com')

Check you are using the correct proxy.
You can get the proxy information by using urllib.getproxies (note: getproxies does not work with dynamic proxy configuration, like when using PAC).
Update As per information about empty proxy list, I would suggest using an urlopener, with the proxy name and information.
Some good information about how use proxies urlopeners:
Urllib manual
Michael Foord's introduction to urllib

Possibly this is a DNS issue, try urlopen with the IP address of the web server you're accessing, i.e.
import urllib
URL="http://66.102.11.99" # www.google.com
f = urllib.urlopen(URL)
f.read()
If this succeeds, then it's probably a DNS issue rather than a proxy issue (but you should also check your proxy setup).

Looks like a DNS problem.
Since you are using Windows, you can try run this command
nslookup www.google.com
To check if the web address can be resolved successfully.
If not, it is a network setting issue
If OK, then we have to look at possible alternative causes

I was facing the same issue.
In my system the proxy configuration is through a .PAC file.
So i opended that file, took out the default proxy url, for me it was http://168.219.61.250:8080/
Following test code worked for me :
import urllib2
proxy_support = urllib2.ProxyHandler({'http': 'http://168.219.61.250:8080/'})
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://python.org/')
html = response.read()
print html
You might need to add some more code, if your proxy requires authentication
Hope this helps!!