Overview: I am calling an API with python2.7 using the urllib2 and json libraries. When I run the script I receive the following error if the results count is higher than 20:
error: [Errno 10054] An existing connection was forcibly closed by the
remote host
Script:
import json
import urllib2
import webbrowser
url = "http://poi.????.com:xxxx/k/search?name==smith&limit==30"
response = urllib2.urlopen(url)
webbrowser.open(url)
data = json.load(response)
count = "count = "+str(data['count'])
print count
for i in data['results']:
name = i['name']
print name
Expected Outcome: I run the script, API responds with a '200 OK' status and my webbrowser.open(url) line opens the response in my browser. My IDLE shell prints the output.
Issue: When I use the limit parameter in the API and set it to <20 it responds as expected in my IDLE shell. However if I set limit > 20 or don't set a limit, I still get a 200 OK response, the browser populates with the results but my IDLE shell produces the 'connection was forcibly closed' error.
Software: I've tried this on windows 7/8 and Ubuntu 14.04. At home via VPN and at work across a wired network.
Traceback Call in IDLE:
Traceback (most recent call last):
File "C:\Users\xxx\userQueryService.py", line 9, in <module>
data = json.load(response)
File "C:\Python27\lib\json\__init__.py", line 286, in load
return loads(fp.read(),
File "C:\Python27\lib\socket.py", line 351, in read
data = self._sock.recv(rbufsize)
File "C:\Python27\lib\httplib.py", line 573, in read
s = self.fp.read(amt)
File "C:\Python27\lib\socket.py", line 380, in read
data = self._sock.recv(left)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
Traceback call in Ubuntu python shell:
>>> data = json.load(response)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 1 column 43303 (char 43302)
I managed to solve my issue by using the requests library instead of urllib2 and json. I've included my code for those that have a similar issue (note I also included a exception handler):
import webbrowser
import requests
url = "http://poi.????.com:xxxx/k/search?name==smith&limit==30"
r = requests.get(url)
try:
data = r.json()
webbrowser.open(url)
count = "count = "+str(data['count'])
print count
for i in data['results']:
name = i['name']
print name
except IOError as (errno, strerror):
print "I/O error({0}): {1}".format(errno, strerror)
Related
I have spent the last few months running neo4j on my local machine. I have been using py2neo to orchestrate the db with python. I have just switched to running the database on an aws ec2 instance and I am recieving errors whenever I try to send requests to the database.
I am using the following simple code to test the database, this code works perfectly on my local machine.
from py2neo import *
neo4j_url = 'http://localhost:7474/'
user = 'neo4j'
pwd = 'neo4j'
graph = Graph(neo4j_url, auth=(user,pwd))
print(graph)
graph.run("Create (n:Person{name: 'Andy person'})")
This code errors with
Graph('http://localhost:7474')
Traceback (most recent call last):
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 443, in from_json
content = json_loads(data, object_hook=JSONHydrant.json_to_packstream)
File "/usr/lib64/python3.8/json/__init__.py", line 370, in loads
return cls(**kw).decode(s)
File "/usr/lib64/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test_connection.py", line 9, in <module>
graph.run("Create (n:Person{name: 'Andy person'})")
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/database.py", line 405, in run
return self.auto().run(cypher, parameters, **kwparameters)
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/database.py", line 989, in run
result = self._connector.auto_run(cypher, parameters,
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/__init__.py", line 1340, in auto_run
return cx.auto_run(cypher, parameters, graph_name=graph_name, readonly=readonly)
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 177, in auto_run
rs = HTTPResponse.from_json(r.status, r.data.decode("utf-8"))
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 445, in from_json
raise_from(ProtocolError("Cannot decode response content as JSON"), error)
File "<string>", line 3, in raise_from
py2neo.errors.ProtocolError: Cannot decode response content as JSON
Does anyone have idea why this error is occuring on my ec2 instance but not on my local machine?
I am using the interface of a website to get data, and I have run multiple programs at the same time. I wrote exception capture in the program. I still get a response 502 error and the program is interrupted, and several programs will be interrupted at the same time. What is the reason?
def search(name):
global n
path = 'https://dev.***.com/api/company/queryByName?name=' + str(name)
s = requests.session()
s.keep_alive = False # 关闭多余连接
try:
r = s.get(path,timeout=3)
print(n,r)
except (ReadTimeout,HTTPError,ConnectionError) as e:
print(e)
return search(name)
else:
n=n+1
result = json.loads(r.text)
Traceback (most recent call last):
File "D:/PyCharm Community Edition/project/company/30.py", line 72, in <module>
data1['social_credit_code'], data1['industry'], data1['reg_place'] = zip(*data1['companyName'].apply(search))
File "C:\Users\13750\AppData\Roaming\Python\Python36\site-packages\pandas\core\series.py", line 3848, in apply
739 <Response [502]>
mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas\_libs\lib.pyx", line 2329, in pandas._libs.lib.map_infer
File "D:/PyCharm Community Edition/project/company/30.py", line 49, in search
result = json.loads(r.text)
File "C:\Users\13750\.conda\envs\py36\lib\json\__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Users\13750\.conda\envs\py36\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\13750\.conda\envs\py36\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The requests API will only raise an exception if you are not able to communicate with a server. In this case you did reach a server, but the server then responded by telling you 502 Bad Gateway. This error usually means you communicated with some proxy server which was unable to forward your message to the final destination.
Regardless, that response will be captured by the requests API and returned as a Response object. After you receive a response you always need to make sure that the return code is what you expect (commonly 200). requests has a convenient way to do so:
r = s.get(path,timeout=3)
if r.ok:
# do your work
In this case you didnt check if the response code was okay, and because the response code indicated an error, you didn't receive any JSON data like you thought you did. Which is why the code followed through to the else statement and gave you a JSONDecodeError.
As the traceback clearly shows, a JSONDecoderError is being raised and your code is not catching it.
You should probably not attempt to decode the content of a 502 response. If you want such responses to raise an exception use raise_for_status
try:
r = s.get(path,timeout=3)
r.raise_for_status()
print(n,r)
except (ReadTimeout,HTTPError,ConnectionError) as e:
...
I have been working on a chatbot interface to icinga2, and have not found a persistent way to survive the restart/reload of the icinga2 server. After a week of moving try/except blocks, using requests sessions, et al, it's time to reach out to the community.
Here is the current iteration of the request function:
def i2api_request(url, headers={}, data={}, stream=False, *, auth=api_auth, ca=api_ca):
''' Do not call this function directly; it's a helper for the i2* command functions '''
# Adapted from http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/icinga2-api
# Section 11.10.3.1
try:
r = requests.post(url,
headers=headers,
auth=auth,
data=json.dumps(data),
verify=ca,
stream=stream
)
except (requests.exceptions.ChunkedEncodingError,requests.packages.urllib3.exceptions.ProtocolError, http.client.IncompleteRead,ValueError) as drop:
return("No connection to Icinga API")
if r.status_code == 200:
for line in r.iter_lines():
try:
if stream == True:
yield(json.loads(line.decode('utf-8')))
else:
return(json.loads(line.decode('utf-8')))
except:
debug("Could not produce JSON from "+line)
continue
else:
#r.raise_for_status()
debug('Received a bad response from Icinga API: '+str(r.status_code))
print('Icinga2 API connection lost.')
(The debug function just flags and prints the indicated error to the console.)
This code works fine handling events from the API and sending them to the chatbot, but if the icinga server is reloaded, as would be needed after adding a new server definition in /etc/icinga2..., the listener crashes.
Here is the error response I get when the server is restarted:
Exception in thread Thread-11:
Traceback (most recent call last):
File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 447, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 228, in _error_catcher
yield
File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 498, in read_chunked
self._update_chunk_length()
File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 451, in _update_chunk_length
raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/errbot/err3/lib/python3.4/site-packages/requests/models.py", line 664, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 349, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 526, in read_chunked
self._original_response.close()
File "/usr/lib64/python3.4/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 246, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/home/errbot/plugins/icinga2bot.py", line 186, in report_events
for line in queue:
File "/home/errbot/plugins/icinga2bot.py", line 158, in i2events
for line in queue:
File "/home/errbot/plugins/icinga2bot.py", line 98, in i2api_request
for line in r.iter_lines():
File "/home/errbot/err3/lib/python3.4/site-packages/requests/models.py", line 706, in iter_lines
for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
File "/home/errbot/err3/lib/python3.4/site-packages/requests/models.py", line 667, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
With Icinga2.4, this crash happened every time the server was restarted. I thought the problem had gone away after we upgraded to 2.5, but it now appears to have turned into a heisenbug.
I wound up getting advice on IRC to reorder the try/except blocks and make sure they were in the right places. Here's the working result.
def i2api_request(url, headers={}, data={}, stream=False, *, auth=api_auth, ca=api_ca):
''' Do not call this function directly; it's a helper for the i2* command functions '''
# Adapted from http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/icinga2-api
# Section 11.10.3.1
debug(url)
debug(headers)
debug(data)
try:
r = requests.post(url,
headers=headers,
auth=auth,
data=json.dumps(data),
verify=ca,
stream=stream
)
debug("Connecting to Icinga server")
debug(r)
if r.status_code == 200:
try:
for line in r.iter_lines():
debug('in i2api_request: '+str(line))
try:
if stream == True:
yield(json.loads(line.decode('utf-8')))
else:
return(json.loads(line.decode('utf-8')))
except:
debug("Could not produce JSON from "+line)
return("Could not produce JSON from "+line)
except (requests.exceptions.ChunkedEncodingError,ConnectionRefusedError):
return("Connection to Icinga lost.")
else:
debug('Received a bad response from Icinga API: '+str(r.status_code))
print('Icinga2 API connection lost.')
except (requests.exceptions.ConnectionError,
requests.packages.urllib3.exceptions.NewConnectionError) as drop:
debug("No connection to Icinga API. Error received: "+str(drop))
sleep(5)
return("No connection to Icinga API.")
What i am trying to do is read a line(an ip address), open the website with that address, and then repeat with all the addresses in the file. instead, i get an error. I am new to python, so maybe its a simple mistake. Thanks in advance !!!
CODE:
>>> f = open("proxy.txt","r"); #file containing list of ip addresses
>>> address = (f.readline()).strip(); # to remove \n at end of line
>>>
>>> while line:
proxy = urllib2.ProxyHandler({'http': address })
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')
address = (f.readline()).strip();
ERROR:
Traceback (most recent call last):
File "<pyshell#15>", line 5, in <module>
urllib2.urlopen('http://www.google.com')
File "D:\Programming\Python\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "D:\Programming\Python\lib\urllib2.py", line 394, in open
response = self._open(req, data)
File "D:\Programming\Python\lib\urllib2.py", line 412, in _open
'_open', req)
File "D:\Programming\Python\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "D:\Programming\Python\lib\urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "D:\Programming\Python\lib\urllib2.py", line 1174, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
It means that the proxy is unavailable.
Here's a proxy checker that checks several proxies simultaneously:
#!/usr/bin/env python
import fileinput # accept proxies from files or stdin
try:
from gevent.pool import Pool # $ pip install gevent
import gevent.monkey; gevent.monkey.patch_all() # patch stdlib
except ImportError: # fallback on using threads
from multiprocessing.dummy import Pool
try:
from urllib2 import ProxyHandler, build_opener
except ImportError: # Python 3
from urllib.request import ProxyHandler, build_opener
def is_proxy_alive(proxy, timeout=5):
opener = build_opener(ProxyHandler({'http': proxy})) # test redir. and such
try: # send request, read response headers, close connection
opener.open("http://example.com", timeout=timeout).close()
except EnvironmentError:
return None
else:
return proxy
candidate_proxies = (line.strip() for line in fileinput.input())
pool = Pool(20) # use 20 concurrent connections
for proxy in pool.imap_unordered(is_proxy_alive, candidate_proxies):
if proxy is not None:
print(proxy)
Usage:
$ python alive-proxies.py proxy.txt
$ echo user:password#ip:port | python alive-proxies.py
What is the appropriate way for handling exceptions from libraries imported by other libraries in Python?
For example, I have a library called "pycontrol" that I import into my main program. "pycontrol" imports the "suds" library. The "suds" library, in turn, imports the "urllib2" library. I've noticed that when the "suds" library has trouble connecting to remote resources it is accessing through "urllib2," these exceptions trickle up to my main program.
My best guess at this point is to import urllib2 and suds into my global name space and catch typical exceptions that get thrown by them and aren't handled in "pycontrol".
Is there some other best practice as to how one might approach this?
A basic idea of what the snippet of code looks like (without importing suds or urllib2 into global name space):
import pycontrol.pycontrol as pc
print "Connecting to iControl API on LTM %s..." % ltm
try:
b = pc.BIGIP(hostname=ltm, username=user, password=pw,
wsdls=wsdl_list, fromurl=True,
debug=soap_debug)
except (<whattocatch>), detail:
print "Error: could not connect to iControl API on LTM %s... aborting!" % ltm
print "Details: %s" % detail
exitcode = 1
else:
print "Connection successfully established."
Here's a sample traceback:
Connecting to iControl API on LTM s0-bigip1-lb2.lab.zynga.com...
Traceback (most recent call last):
File "./register.py", line 507, in <module>
main()
File "./register.py", line 415, in main
b = build_bigip_object(ltm, user, pw, WSDLS, soap_debug = False)
File "./register.py", line 85, in build_bigip_object
debug=soap_debug)
File "build/bdist.macosx-10.6-universal/egg/pycontrol/pycontrol.py", line 81, in __init__
File "build/bdist.macosx-10.6-universal/egg/pycontrol/pycontrol.py", line 103, in _get_clients
File "build/bdist.macosx-10.6-universal/egg/pycontrol/pycontrol.py", line 149, in _get_suds_client
File "/Library/Python/2.6/site-packages/suds/client.py", line 111, in __init__
self.wsdl = reader.open(url)
File "/Library/Python/2.6/site-packages/suds/reader.py", line 136, in open
d = self.fn(url, self.options)
File "/Library/Python/2.6/site-packages/suds/wsdl.py", line 136, in __init__
d = reader.open(url)
File "/Library/Python/2.6/site-packages/suds/reader.py", line 73, in open
d = self.download(url)
File "/Library/Python/2.6/site-packages/suds/reader.py", line 88, in download
fp = self.options.transport.open(Request(url))
File "/Library/Python/2.6/site-packages/suds/transport/https.py", line 60, in open
return HttpTransport.open(self, request)
File "/Library/Python/2.6/site-packages/suds/transport/http.py", line 62, in open
return self.u2open(u2request)
File "/Library/Python/2.6/site-packages/suds/transport/http.py", line 118, in u2open
return url.open(u2request, timeout=tm)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 383, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 401, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 361, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 1138, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 1105, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
I think you answered you question yourself. Import urllib2 and catch the exception in your module.
from urllib2 import URLError
try:
# something
except URLError, e:
# Do something in case of error.
Why do you need to catch specific exceptions at all? After all, any exception (not only URLError) raised from b = pc.BIGIP(...) means you cannot go on.
I suggest:
import traceback
try:
b = pc.BIGIP(...)
except:
traceback.print_exc()
exitcode = 1
else:
do_something_with_connection(b)
Another idea: Why bother with catching the exception at all? The Python interpreter will dump a stack trace to stderr and exit the program when it encounters an unhandled exception:
b = bc.BIGIP(...)
do_something_with_connection(b)
Or if you need to write to an error log:
import logging
import sys
def main():
b = bc.BIGIP(...)
do_something_with_connection(b)
if __name__ == "__main__":
try:
main()
except:
logging.exception("An unexpected error occured")
sys.exit(1)