I have a program that uses urllib to periodically fetch a url, and I see intermittent
errors like :
I/O error(socket error): [Errno 111] Connection refused.
It works 90% of the time, but the othe r10% it fails. If retry the fetch immediately after it fails, it succeeds. I'm unable to figure out why this is so. I tried to see if any ports are available, and they are. Any debugging ideas?
For additional info, the stack trace is:
File "/usr/lib/python2.6/urllib.py", line 203, in open
return getattr(self, name)(url)
File "/usr/lib/python2.6/urllib.py", line 342, in open_http
h.endheaders()
File "/usr/lib/python2.6/httplib.py", line 868, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 740, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 699, in send
self.connect()
File "/usr/lib/python2.6/httplib.py", line 683, in connect
self.timeout)
File "/usr/lib/python2.6/socket.py", line 512, in create_connection
raise error, msg
Edit - A google search isn't very helpful, what I got out of it is that the server
I'm fetching from sometimes refuses connections, how can I verify its not a bug in my code
and this is indeed the case?
Use a packet sniffer like Wireshark to look at what happens. You need to see a SYN-flagged packet outgoing, a SYN+ACK-flagged incoming and then a ACK-flagged outgoing. After that, the port is considered open on the local side.
If you only see the first packet and the error message comes after several seconds of waiting, the other side is not answering at all (like in: unplugged cable, overloaded server, misguided packet was discarded) and your local network stack aborts the connection attempt. If you see RST packets, the host actually denies the connection. If you see "ICMP Port unreachable" or host unreachable packets, a firewall or the target host inform you of the port actually being closed.
Of course you cannot expect the service to be available at all times (consider all the points of failure in between you and the data), so you should try again later.
Getting an ECONNREFUSED errno means that your kernel was refused a connection at the other end, so if it's a bug, it's either in your kernel or in the other end.
What you can do is to trap the error in a very specific way and try again in a little while, since this seems to work:
# This is Python > 2.5 code
import errno, time
for attempt in range(MAXIMUM_NUMBER_OF_ATTEMPTS):
try:
# your urllib call here
except EnvironmentError as exc: # replace " as " with ", " for Python<2.6
if exc.errno == errno.ECONNREFUSED:
time.sleep(A_COUPLE_OF_SECONDS)
else:
raise # re-raise otherwise
else: # we tried, and we had no failure, so
break
else: # we never broke out of the for loop
raise RuntimeError("maximum number of unsuccessful attempts reached")
Replace the two all-caps constants with your favourite numbers.
I previously had this problem with my EC2 instance (I was serving couchdb to serve resources -- am considering Amazon's S3 for the future).
One thing to check (assuming Ec2) is that the couchdb port is added to your open ports within your security policy.
I specifically encountered
"[Errno 111] Connection refused"
over EC2 when the instance was stopped and started. The problem seems to be a pidfile race. The solution for me was killing couchdb (entirely and properly) via:
pkill -f couchdb
and then restarting with:
/etc/init.d/couchdb restart
I'm not exactly sure what's causing this. You can try looking in your socket.py (mine is a different version, so line numbers from the trace don't match, and I'm afraid some other details might not match as well).
Anyway, it seems like a good practice to put your url fetching code in a try: ... except: ... block, and handle this with a short pause and a retry. The URL you're trying to fetch may be down, or too loaded, and that's stuff you'll only be able to handle in with a retry anyway.
Its seems that server is not running properly so ensure that with terminal by
telnet ip port
example
telnet localhost 8069
It will return connected to localhost so it indicates that there is no problem with the connection
Else it will return Connection refused it indicates that there is problem with the connection
Related
Im having some problem when joining Electron and Python using web sockets (mostly as a learning experience):
Build a desktop app using Electron Ok
Build a Python program that monitors some things Ok
Connect Python with Electron using socket.io Not working
First I tested socket.io in python with the 'latency' example in https://github.com/miguelgrinberg/python-socketio/tree/master/examples/server/aiohttp. It worked great when I took a look at http://localhost:8080/ in Chrome.
The next step was to ask Electron to load the same url in the desktop app with mainWindow.loadURL('http://localhost:8080') (I used the template from the official electron-quick-start example). I just had to add two lines into the html as shown in here: Electron: jQuery is not defined for jquery to load correctly in Electron, but it again worked nicely.
Now I tried to create an index.html with the same content as latency.html and load it with mainWindow.loadFile('gui/index.html'). I changed the line var socket = io.connect(); for var socket = io.connect('ws://localhost:8080'); (I did also tried ws://127.0.0.1:8080 and ws://192.168.<x>.<x>:8080). The html file loads correctly but the socket begins to connect and disconnect repeatedly with a Bad Request error (I added two console.log lines):
(console.log) connected
(error) POST http://localhost:8080/socket.io/?EIO=3&transport=polling&t=MosNuF3&sid=c62ce5a6090c4b72bf3f7c6916da6ce7 400 (Bad Request) **polling-xhr.js:264**
(console.log) disconnected
(warning) websocket.js:235 WebSocket connection to 'ws://localhost:8080/socket.io/?EIO=3&transport=websocket&sid=c62ce5a6090c4b72bf3f7c6916da6ce7' failed: WebSocket is closed before the connection is established.
(error) POST http://localhost:8080/socket.io/?EIO=3&transport=polling&t=MosNuFS&sid=c62ce5a6090c4b72bf3f7c6916da6ce7 400 (Bad Request) **polling-xhr.js:264**
(console.log) connected
etc...
(console.log) disconnected
etc...
On the server side there is also an error when I stop the Electron app:
Unhandled exception
Traceback (most recent call last):
File "D:\Programming\Tools\Anaconda3\lib\site-packages\aiohttp\web_protocol.py", line 447, in start
await resp.prepare(request)
File "D:\Programming\Tools\Anaconda3\lib\site-packages\aiohttp\web_response.py", line 353, in prepare
return await self._start(request)
File "D:\Programming\Tools\Anaconda3\lib\site-packages\aiohttp\web_response.py", line 667, in _start
return await super()._start(request)
File "D:\Programming\Tools\Anaconda3\lib\site-packages\aiohttp\web_response.py", line 410, in _start
await writer.write_headers(status_line, headers)
File "D:\Programming\Tools\Anaconda3\lib\site-packages\aiohttp\http_writer.py", line 112, in write_headers
self._write(buf)
File "D:\Programming\Tools\Anaconda3\lib\site-packages\aiohttp\http_writer.py", line 67, in _write
raise ConnectionResetError('Cannot write to closing transport')
ConnectionResetError: Cannot write to closing transport
What does this Bad Request error mean in this context ? How would the connection work correctly ?
Thanks.
In case someone hits the same problem. The answer is pretty simple, from the documentation for socket.io here:
For security reasons, this server enforces a same-origin policy by
default. In practical terms, this means the following:
If an incoming HTTP or WebSocket request includes the Origin header, this header must match the scheme and host of the connection
URL. In case of a mismatch, a 400 status code response is returned and
the connection is rejected.
No restrictions are imposed on incoming requests that do not include the Origin header.
If necessary, the cors_allowed_origins option can be used to allow
other origins. This argument can be set to a string to set a single
allowed origin, or to a list to allow multiple origins. A special
value of '*' can be used to instruct the server to allow all origins,
but this should be done with care, as this could make the server
vulnerable to Cross-Site Request Forgery (CSRF) attacks.
When using electron to load the gui, the server is not the same as the python server that's doing the monitoring, thus the Bad Request (not same origin). The solution is to just modify as follows the server properties adding cors_allowed_origins='*':
socketio.AsyncServer(async_mode='aiohttp', cors_allowed_origins='*')
I'm working on Python 2.7 code to read a value from HTML page by using urllib2 library. I want to timeout the urllib2.urlopen function after 5 seconds in case of no Internet and jump to remaining code.
It works as expected when computer is connected to working internet connection. And for testing if I set timeout=0.1 it timed out suddenly without opening url, as expected. But when there is no Internet, timeout not works either I set timeout to 0.1, 5, or any other value. It simply does not timed out.
This is my Code:
import urllib2
url = "https://alfahd.witorbit.net/fingerprint.php?n"
try:
response = urllib2.urlopen(url , timeout=5).read()
print response
except Exception as e:
print e
Result when connected to Internet with timeout value 5:
180
Result when connected to Internet with timeout value 0.1 :
<urlopen error timed out>
Seems timeout is working.
Result when NOT connected to Internet and with any timeout value (it timed out after about 40 seconds every time I open url despite of any value I set for timeout=:
<urlopen error [Errno -3] Temporary failure in name resolution>
How can I timeout urllib2.urlopen when there is no Internet connectivity? Am I missing some thing? Please guide me to solve this issue. Thanks!
Because name resolution happens before the request is made, it's not subject to the timeout. You can prevent this error in name resolution by providing the IP for the host in your /etc/hosts file. For example, if the host is subdomain.example.com and the IP is 10.10.10.10 you would add the following line in the /etc/hosts file
10.10.10.10 subdomain.example.com
Alternatively, you may be able to simply use the IP address directly, however, some webservers require you use the hostname, in which case you'll need to modify the hosts file to use the name offline.
I'm trying to connect to control port (9051) of tor from a remote machine using stem python library.
dum.py
from stem import Signal
from stem.control import Controller
def set_new_ip():
"""Change IP using TOR"""
with Controller.from_port(address = '10.130.8.169', port=9051) as controller:
controller.authenticate(password='password')
controller.signal(Signal.NEWNYM)
set_new_ip()
I'm getting the following error
Traceback (most recent call last):
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/socket.py", line 398, in _make_socket
control_socket.connect((self._control_addr, self._control_port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "dum.py", line 28, in <module>
set_new_ip();
File "dum.py", line 7, in set_new_ip
with Controller.from_port(address = '10.130.4.162', port=9051) as controller:
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/control.py", line 998, in from_port
control_port = stem.socket.ControlPort(address, port)
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/socket.py", line 372, in __init__
self.connect()
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/socket.py", line 243, in connect
self._socket = self._make_socket()
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/socket.py", line 401, in _make_socket
raise stem.SocketError(exc)
stem.SocketError: [Errno 111] Connection refused
Then I went through /etc/tor/torrc config file.
It says
The port on which Tor will listen for local connections from Tor
controller applications, as documented in control-spec.txt.
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:E5364A963AF943CB607CFDAE3A49767F2F8031328D220CDDD1AE30A471
SocksListenAddress 0.0.0.0:9050
CookieAuthentication 1
My question is ,
How do I connect to control port of Tor from a remote host?
Is there is any work around or config parameter that I need to set?
a possible duplicate of Stem is giving the "Unable to connect to port 9051" error which has no answers
Tested with Tor 0.3.3.7.
ControlListenAddress config is OBSOLETE and Tor will ignore it and log the following message
[warn] Skipping obsolete configuration option 'ControlListenAddress'
You can still set ControlPort to 0.0.0.0:9051 in your torrc file. Though, Tor is not very happy about it (and rightly so) and will warn you
You have a ControlPort set to accept connections from a non-local
address. This means that programs not running on your computer can
reconfigure your Tor. That's pretty bad, since the controller protocol
isn't encrypted! Maybe you should just listen on 127.0.0.1 and use a
tool like stunnel or ssh to encrypt remote connections to your control
port.
Also, you have to set either CookieAuthentication or HashedControlPassword otherwise ControlPort will be closed
You have a ControlPort set to accept unauthenticated connections from
a non-local address. This means that programs not running on your
computer can reconfigure your Tor, without even having to guess a
password. That's so bad that I'm closing your ControlPort for you. If
you need to control your Tor remotely, try enabling authentication and
using a tool like stunnel or ssh to encrypt remote access.
All the risks mentioned in #drew010's answer still stand.
You'd need to set ControlListenAddress in addition to the ControlPort. You could set that to to 0.0.0.0 (binds to all addresses) or a specific IP your server listens on.
If you choose to do this it would be extremely advisable to configure your firewall to only allow control connections from specific IP's and block them from all others.
Also note, the control port traffic will not be encrypted, so it'd also be advisable to use cookie authentication so your password isn't sent over the net.
You could also run a hidden service to expose the control port over Tor and then connect to the hidden service using Stem and Tor.
But the general answer is ControlListenAddress needs to be set to bind to an IP other than 127.0.0.1 (localhost).
I'm playing a bit with Twisted and created a simple 'server'.
I'd like to let the server listening on multiple ports (1025-65535) instead of a single port.
How can i do this ?
My code:
from twisted.internet.protocol import Protocol,ServerFactory
from twisted.internet import reactor
class QuickDisconnectProtocol(Protocol):
def connectionMade(self):
print "Connection from : ", self.transport.getPeer()
self.transport.loseConnection() # terminate connection
f = ServerFactory()
f.protocol = QuickDisconnectProtocol
reactor.listenTCP(6666,f)
reactor.run()
Already tried this:
for i in range (0, 64510):
reactor.listenTCP(1025+i,f)
reactor.run()
But received an error:
Traceback (most recent call last):
File "Server.py", line 14, in <module>
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 436, in listenTCP
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 641, in startListening
twisted.internet.error.CannotListenError: Couldn't listen on any:2044: [Errno 24] Too many open files.
Each listening port requires a file descriptor ("open file"), and each file descriptor takes up one element of your maximum file descriptors quota.
This stack overflow question has an answer explaining how to raise this limit on Linux, and this blog post has resources as to how to do it on OS X.
That said, the other respondents who have told you that this is not a particularly sane thing to do are right. In particular, your operating system may stop working if you actually go all the way up to 65535, this overrules the entire ephemeral port range, which means you may not be able to make TCP client connections from this machine any more. So it would be good to explain in your question why you are trying to do this.
The usual solution is to have one listening port ( chosen by the server! ). If you want each client on its own port, then the server chooses the port, starts listening on it, and replies to the client with the port it will use for further requests.
It is not a real good use of port resources! If the server needs to keep state information for each client then it should issue a unique ID to each client when the client first connects and the client should use this ID for every request to the server.
However, with a little care, you can often design the system so that the server does not need to keep separate state information for each client.
Currently I am using an app built in python. When I run it in personal computer, it works without problems.
However, when I move it into a production server. It keeps showing me the error attached as below:.
I've done some research and I got the reason that the end user browser stops the connection while the server is still busy sending data.
I wonder why did it happen and what is the root cause that prevents it from running properly in production server, while it works on my personal computer. Any advice is appreciated
Exception happened during processing of request from ('127.0.0.1', 34226)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 284, in
_handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 641, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 694, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
Your server process has received a SIGPIPE writing to a socket. This usually happens when you write to a socket fully closed on the other (client) side. This might be happening when a client program doesn't wait till all the data from the server is received and simply closes a socket (using close function).
In a C program you would normally try setting to ignore SIGPIPE signal or setting a dummy signal handler for it. In this case a simple error will be returned when writing to a closed socket. In your case a python seems to throw an exception that can be handled as a premature disconnect of the client.
The broken pipe error usually occurs if your request is blocked or takes too long and after request-side timeout, it'll close the connection and then, when the respond-side (server) tries to write to the socket, it will throw a pipe broken error.
It depends on how you tested it, and possibly on differences in the TCP stack implementation of the personal computer and the server.
For example, if your sendall always completes immediately (or very quickly) on the personal computer, the connection may simply never have broken during sending. This is very likely if your browser is running on the same machine (since there is no real network latency).
In general, you just need to handle the case where a client disconnects before you're finished, by handling the exception.
Remember that TCP communications are asynchronous, but this is much more obvious on physically remote connections than on local ones, so conditions like this can be hard to reproduce on a local workstation. Specifically, loopback connections on a single machine are often almost synchronous.
This might be because you are using two method for inserting data into database and this cause the site to slow down.
def add_subscriber(request, email=None):
if request.method == 'POST':
email = request.POST['email_field']
e = Subscriber.objects.create(email=email).save() <====
return HttpResponseRedirect('/')
else:
return HttpResponseRedirect('/')
In above function, the error is where arrow is pointing. The correct implementation is below:
def add_subscriber(request, email=None):
if request.method == 'POST':
email = request.POST['email_field']
e = Subscriber.objects.create(email=email)
return HttpResponseRedirect('/')
else:
return HttpResponseRedirect('/')
If it's a python a web application or service such as Flask or FastAPI, this error might occur if the production server is configured to timeout a request that takes too long. There are relevant parameters in Gunicorn and Uvicorn such as GRACEFUL_TIMEOUT and TIMEOUT that need to be configured according to the needs of your application. You may also want to check your reverse proxy or gateway timeout thresholds.
Try this code at the top of your program:
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
It should fix the issue.