I'm playing a bit with Twisted and created a simple 'server'.
I'd like to let the server listening on multiple ports (1025-65535) instead of a single port.
How can i do this ?
My code:
from twisted.internet.protocol import Protocol,ServerFactory
from twisted.internet import reactor
class QuickDisconnectProtocol(Protocol):
def connectionMade(self):
print "Connection from : ", self.transport.getPeer()
self.transport.loseConnection() # terminate connection
f = ServerFactory()
f.protocol = QuickDisconnectProtocol
reactor.listenTCP(6666,f)
reactor.run()
Already tried this:
for i in range (0, 64510):
reactor.listenTCP(1025+i,f)
reactor.run()
But received an error:
Traceback (most recent call last):
File "Server.py", line 14, in <module>
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 436, in listenTCP
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 641, in startListening
twisted.internet.error.CannotListenError: Couldn't listen on any:2044: [Errno 24] Too many open files.
Each listening port requires a file descriptor ("open file"), and each file descriptor takes up one element of your maximum file descriptors quota.
This stack overflow question has an answer explaining how to raise this limit on Linux, and this blog post has resources as to how to do it on OS X.
That said, the other respondents who have told you that this is not a particularly sane thing to do are right. In particular, your operating system may stop working if you actually go all the way up to 65535, this overrules the entire ephemeral port range, which means you may not be able to make TCP client connections from this machine any more. So it would be good to explain in your question why you are trying to do this.
The usual solution is to have one listening port ( chosen by the server! ). If you want each client on its own port, then the server chooses the port, starts listening on it, and replies to the client with the port it will use for further requests.
It is not a real good use of port resources! If the server needs to keep state information for each client then it should issue a unique ID to each client when the client first connects and the client should use this ID for every request to the server.
However, with a little care, you can often design the system so that the server does not need to keep separate state information for each client.
Related
I'm trying to connect to control port (9051) of tor from a remote machine using stem python library.
dum.py
from stem import Signal
from stem.control import Controller
def set_new_ip():
"""Change IP using TOR"""
with Controller.from_port(address = '10.130.8.169', port=9051) as controller:
controller.authenticate(password='password')
controller.signal(Signal.NEWNYM)
set_new_ip()
I'm getting the following error
Traceback (most recent call last):
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/socket.py", line 398, in _make_socket
control_socket.connect((self._control_addr, self._control_port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "dum.py", line 28, in <module>
set_new_ip();
File "dum.py", line 7, in set_new_ip
with Controller.from_port(address = '10.130.4.162', port=9051) as controller:
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/control.py", line 998, in from_port
control_port = stem.socket.ControlPort(address, port)
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/socket.py", line 372, in __init__
self.connect()
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/socket.py", line 243, in connect
self._socket = self._make_socket()
File "/home/jkl/anaconda3/lib/python3.5/site-packages/stem/socket.py", line 401, in _make_socket
raise stem.SocketError(exc)
stem.SocketError: [Errno 111] Connection refused
Then I went through /etc/tor/torrc config file.
It says
The port on which Tor will listen for local connections from Tor
controller applications, as documented in control-spec.txt.
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:E5364A963AF943CB607CFDAE3A49767F2F8031328D220CDDD1AE30A471
SocksListenAddress 0.0.0.0:9050
CookieAuthentication 1
My question is ,
How do I connect to control port of Tor from a remote host?
Is there is any work around or config parameter that I need to set?
a possible duplicate of Stem is giving the "Unable to connect to port 9051" error which has no answers
Tested with Tor 0.3.3.7.
ControlListenAddress config is OBSOLETE and Tor will ignore it and log the following message
[warn] Skipping obsolete configuration option 'ControlListenAddress'
You can still set ControlPort to 0.0.0.0:9051 in your torrc file. Though, Tor is not very happy about it (and rightly so) and will warn you
You have a ControlPort set to accept connections from a non-local
address. This means that programs not running on your computer can
reconfigure your Tor. That's pretty bad, since the controller protocol
isn't encrypted! Maybe you should just listen on 127.0.0.1 and use a
tool like stunnel or ssh to encrypt remote connections to your control
port.
Also, you have to set either CookieAuthentication or HashedControlPassword otherwise ControlPort will be closed
You have a ControlPort set to accept unauthenticated connections from
a non-local address. This means that programs not running on your
computer can reconfigure your Tor, without even having to guess a
password. That's so bad that I'm closing your ControlPort for you. If
you need to control your Tor remotely, try enabling authentication and
using a tool like stunnel or ssh to encrypt remote access.
All the risks mentioned in #drew010's answer still stand.
You'd need to set ControlListenAddress in addition to the ControlPort. You could set that to to 0.0.0.0 (binds to all addresses) or a specific IP your server listens on.
If you choose to do this it would be extremely advisable to configure your firewall to only allow control connections from specific IP's and block them from all others.
Also note, the control port traffic will not be encrypted, so it'd also be advisable to use cookie authentication so your password isn't sent over the net.
You could also run a hidden service to expose the control port over Tor and then connect to the hidden service using Stem and Tor.
But the general answer is ControlListenAddress needs to be set to bind to an IP other than 127.0.0.1 (localhost).
Im writing HTTP Proxy Server that will open socket with the browser and the request from the browser will go to my HTTP Proxy Server and my server will open socket with the server that the browser ask for and send him the request.
It will go like this:
Browser --request--> HTTP Proxy Server --request--> Web Server
Browser <--response-- HTTP Proxy Server <--response-- Web Server
Now I need those sockets will be clearly that I can use scapy to see the layers of each packet and manipulate it like I want to. (for security reasons e.g Block Phishing or something like that)
In this code I write simple socket with the browser just for testing and learning about browser behavior with HTTP Requests.
from scapy.all import *
import socket
socket_with_browser = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
print "Start"
socket_with_browser.bind(('127.0.0.1',8080))
socket_with_browser.listen(1)
conn , addr = socket_with_browser.accept()
stream_sock_browser = StreamSocket(conn)
r = stream_sock_browser.recv(4096)
r[TCP].show()
socket_with_browser.close()
I get the following Error:
Traceback (most recent call last):
File "<string>", line 254, in run_nodebug
File "C:\Python26\ProxyServer\module1.py", line 22, in <module>
r[TCP].show()
File "C:\Python26\Lib\site-packages\scapy\packet.py", line 817, in __getitem__
raise IndexError("Layer [%s] not found" % lname)
IndexError: Layer [TCP] not found
There is a way that I can get packet from socket and use it (get packet layers or something) with scapy? Maybe I declare the socket badly? By the way Im using Windows 7 and python 2.6
Six years late but your question might appear to others with similar issues.
The problem with your approach is that not every packet received has a TCP layer, so you should use the haslayer() method.
if r.haslayer(TCP): # yes, without quotation marks
r.show()
When I try to connect to the Gmail server, python throws an error:
>>> from smtplib import SMTP
>>> m = SMTP('smtp.gmail.com', 587)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\smtplib.py", line 249, in __init__
(code, msg) = self.connect(host, port)
File "C:\Python27\lib\smtplib.py", line 309, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "C:\Python27\lib\smtplib.py", line 284, in _get_socket
return socket.create_connection((port, host), timeout)
File "C:\Python27\lib\socket.py", line 571, in create_connection
raise err
socket.error: [Errno 10060]
The rest of the output is in a diferent language but it basically says that the host (gmail) didn't respond.
I can see my email on a browser here at my work, probably there's a network configuration that doesn't allow me to automate the email delivering.
Is there a work around to let python act as a regular browser?
First, you can only access gmail's SMTP servers as a client with some form of authentication; the recommended way is with oauth. See this page and this one for details. So, your code won't work, even when you get past this problem.
However, that doesn't explain why it's rejecting your connection before you even get a chance to log in.
The most likely possibility is that gmail's routers are maintaining a dynamic whitelist of IPs. When you use a properly-logged-in connection of some other kind, you get added to the whitelist for N seconds, meaning you're allowed to connect to port 587; otherwise, you're rejected. This would be similar to the traditional SMTP-after-IMAP auth scheme, but not restricted to IMAP, and handled at the router instead of the SMTP service (presumably to lower the cost or make DoS attacks more difficult).
There's a good way to test this: Configure Outlook, Mail.app, or some other mail client that knows how to connect to gmail, and uses SMTP to send mail via gmail. Run your script a few seconds after fetching mail in the mail client. If it works, that's the problem. And in that case, the fix is to do the same kind of connection and login (IMAP? web service?) that the mail client does.
Or, of course, you can use the sample code Google provides at the above links instead of working it out from first principles.
(Of course gmail also has to accept server-to-server SMTP connections, but they could easily have a different auth scheme for that. I'm assuming you're trying to do a client-to-server connection, rather than server-to-server.)
The other possibility is that you're on some kind of blacklist—e.g., gmail thinks your IP belongs to a spammer. This could also be dynamic—maybe anyone who makes a connection to port 587 but doesn't oauth properly gets blocked for the next N seconds. At any rate, this is also easy to test: Configure Outlook, Mail.app, etc. If this is the problem, they won't be able to send mail either.
There's a third possibility, that no one is allowed to connect to port 587, and Google wants you to use port 565 or 25 instead.
For easier debugging, you might want to consider writing an even simpler script that just creates a socket and connects, instead of using smtplib:
import socket
s = socket.socket()
s.connect(('smtp.gmail.com', 587))
Or, even more simply, just netcat from the command line:
nc smtp.gmail.com 587
To answer your side question:
Is there a work around to let python act as a regular browser?
That's not the issue. A regular browser doesn't make SMTP connections; it makes web service connections using Javascript code downloaded from gmail.com.
Of course Python can also make web services connections.
And it can act as much or as little like a "regular browser" (e.g., User-Agent, Referer, etc. headers) as you desire, but that probably isn't relevant—either the gmail web service API is public and has clear, published rules for how to authenticate yourself (in which case you just do what the rules say), or it's private and you shouldn't be trying to fool whatever protection they're using unless you want to get into an arms race.
At any rate, in this case, we know it's public, so we don't have to guess.
Currently I am using an app built in python. When I run it in personal computer, it works without problems.
However, when I move it into a production server. It keeps showing me the error attached as below:.
I've done some research and I got the reason that the end user browser stops the connection while the server is still busy sending data.
I wonder why did it happen and what is the root cause that prevents it from running properly in production server, while it works on my personal computer. Any advice is appreciated
Exception happened during processing of request from ('127.0.0.1', 34226)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 284, in
_handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 641, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 694, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
Your server process has received a SIGPIPE writing to a socket. This usually happens when you write to a socket fully closed on the other (client) side. This might be happening when a client program doesn't wait till all the data from the server is received and simply closes a socket (using close function).
In a C program you would normally try setting to ignore SIGPIPE signal or setting a dummy signal handler for it. In this case a simple error will be returned when writing to a closed socket. In your case a python seems to throw an exception that can be handled as a premature disconnect of the client.
The broken pipe error usually occurs if your request is blocked or takes too long and after request-side timeout, it'll close the connection and then, when the respond-side (server) tries to write to the socket, it will throw a pipe broken error.
It depends on how you tested it, and possibly on differences in the TCP stack implementation of the personal computer and the server.
For example, if your sendall always completes immediately (or very quickly) on the personal computer, the connection may simply never have broken during sending. This is very likely if your browser is running on the same machine (since there is no real network latency).
In general, you just need to handle the case where a client disconnects before you're finished, by handling the exception.
Remember that TCP communications are asynchronous, but this is much more obvious on physically remote connections than on local ones, so conditions like this can be hard to reproduce on a local workstation. Specifically, loopback connections on a single machine are often almost synchronous.
This might be because you are using two method for inserting data into database and this cause the site to slow down.
def add_subscriber(request, email=None):
if request.method == 'POST':
email = request.POST['email_field']
e = Subscriber.objects.create(email=email).save() <====
return HttpResponseRedirect('/')
else:
return HttpResponseRedirect('/')
In above function, the error is where arrow is pointing. The correct implementation is below:
def add_subscriber(request, email=None):
if request.method == 'POST':
email = request.POST['email_field']
e = Subscriber.objects.create(email=email)
return HttpResponseRedirect('/')
else:
return HttpResponseRedirect('/')
If it's a python a web application or service such as Flask or FastAPI, this error might occur if the production server is configured to timeout a request that takes too long. There are relevant parameters in Gunicorn and Uvicorn such as GRACEFUL_TIMEOUT and TIMEOUT that need to be configured according to the needs of your application. You may also want to check your reverse proxy or gateway timeout thresholds.
Try this code at the top of your program:
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
It should fix the issue.
I have a program that uses urllib to periodically fetch a url, and I see intermittent
errors like :
I/O error(socket error): [Errno 111] Connection refused.
It works 90% of the time, but the othe r10% it fails. If retry the fetch immediately after it fails, it succeeds. I'm unable to figure out why this is so. I tried to see if any ports are available, and they are. Any debugging ideas?
For additional info, the stack trace is:
File "/usr/lib/python2.6/urllib.py", line 203, in open
return getattr(self, name)(url)
File "/usr/lib/python2.6/urllib.py", line 342, in open_http
h.endheaders()
File "/usr/lib/python2.6/httplib.py", line 868, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 740, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 699, in send
self.connect()
File "/usr/lib/python2.6/httplib.py", line 683, in connect
self.timeout)
File "/usr/lib/python2.6/socket.py", line 512, in create_connection
raise error, msg
Edit - A google search isn't very helpful, what I got out of it is that the server
I'm fetching from sometimes refuses connections, how can I verify its not a bug in my code
and this is indeed the case?
Use a packet sniffer like Wireshark to look at what happens. You need to see a SYN-flagged packet outgoing, a SYN+ACK-flagged incoming and then a ACK-flagged outgoing. After that, the port is considered open on the local side.
If you only see the first packet and the error message comes after several seconds of waiting, the other side is not answering at all (like in: unplugged cable, overloaded server, misguided packet was discarded) and your local network stack aborts the connection attempt. If you see RST packets, the host actually denies the connection. If you see "ICMP Port unreachable" or host unreachable packets, a firewall or the target host inform you of the port actually being closed.
Of course you cannot expect the service to be available at all times (consider all the points of failure in between you and the data), so you should try again later.
Getting an ECONNREFUSED errno means that your kernel was refused a connection at the other end, so if it's a bug, it's either in your kernel or in the other end.
What you can do is to trap the error in a very specific way and try again in a little while, since this seems to work:
# This is Python > 2.5 code
import errno, time
for attempt in range(MAXIMUM_NUMBER_OF_ATTEMPTS):
try:
# your urllib call here
except EnvironmentError as exc: # replace " as " with ", " for Python<2.6
if exc.errno == errno.ECONNREFUSED:
time.sleep(A_COUPLE_OF_SECONDS)
else:
raise # re-raise otherwise
else: # we tried, and we had no failure, so
break
else: # we never broke out of the for loop
raise RuntimeError("maximum number of unsuccessful attempts reached")
Replace the two all-caps constants with your favourite numbers.
I previously had this problem with my EC2 instance (I was serving couchdb to serve resources -- am considering Amazon's S3 for the future).
One thing to check (assuming Ec2) is that the couchdb port is added to your open ports within your security policy.
I specifically encountered
"[Errno 111] Connection refused"
over EC2 when the instance was stopped and started. The problem seems to be a pidfile race. The solution for me was killing couchdb (entirely and properly) via:
pkill -f couchdb
and then restarting with:
/etc/init.d/couchdb restart
I'm not exactly sure what's causing this. You can try looking in your socket.py (mine is a different version, so line numbers from the trace don't match, and I'm afraid some other details might not match as well).
Anyway, it seems like a good practice to put your url fetching code in a try: ... except: ... block, and handle this with a short pause and a retry. The URL you're trying to fetch may be down, or too loaded, and that's stuff you'll only be able to handle in with a retry anyway.
Its seems that server is not running properly so ensure that with terminal by
telnet ip port
example
telnet localhost 8069
It will return connected to localhost so it indicates that there is no problem with the connection
Else it will return Connection refused it indicates that there is problem with the connection