I'm working on a slightly larger project of my own and I need to make a localhost proxy in python.
The way I wrote mine is that there's a TCP server (using socket and SOCK_STREAM) on port 8080 on the localhost. It accepts a request from the local host, using slicing, string.find(), and gethostbyname() finds that target IP, so it opens up another TCP socket, sends the request and recv's a reply. After that, it relays the reply back to the localhost proxy which in turn throws it back at the browser.
This is the code with ample debugging messages and a debug file to collect the requests of the browser and the replies received back (also note this is just a prototype, hence the limited for loop instead of a while 1 loop):
import socket
local = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
f = open('test.txt', 'a')
local.bind(('localhost', 8080))
local.listen(5)
for i in xrange(20):
print '=====%d=====\n' % i
out = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
data, addr = local.accept()
print 'Connection accepted'
buffer = data.recv(4096)
print 'data recieved'
f.write('=============================================================\n')
f.write(buffer)
end = buffer.find('\n')
print buffer
#print buffer[:end]
host = buffer[:end].split()[1]
end = host[7:].find('/')
print host[7:(end+7)]
host_ip = socket.gethostbyname(host[7:(end+7)])
#print 'remote host: ' + host + ' IP: ' + host_ip
print 'sending buffer to remote host'
out.connect((host_ip, 80))
out.sendall(buffer)
print 'recieving data from remote host'
reply = out.recv(4096)
out.close()
print 'data recieved from remote host'
f.write('+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n')
f.write(reply)
f.write('\n\n\n')
print 'sending data back to local host'
data.sendall(reply)
print 'data sent'
local.close()
out.close()
f.close()
Now my problem is that it seems to work fine for the first few requests, it gets the html and a few images but at some point it always stops at the "data received" point and quits, because it gets no data ie. the buffer is empty. The browser still shows it's loading elements of the page, but when it stops and I look at the text log file, I see that the buffer was empty, meaning that the browser didn't submit anything to the proxy?
I am guessing that the issue lies somewhere in how a browser submits requests and my script not reacting properly to this behavior.
I know I could use the Twist framework, however I want to learn to write this kinda stuff myself. I've been reading about SocketServer and I might use that, but I have no clue if it'll solve the issue because frankly, I don't really understand what's causing the issue here. Is my script too slow for the browser? Do servers send more than one answer and my receiving socket should listen for more packets? Is my buffer size (4096) too small?
I'd really appreciate a nudge in the right direction.
Thanks!
Well, I managed to answer my question. What I suspected previously was partly true - the browser was waiting for something and that something was replies.
I fired up wire shark, did some experiments and I noticed that my proxy makes a lot of ugly TCP RST appear in wireshark. I also noticed that in a normal connection, a lot of the server replies are split up into a few different packets.
Basically, my program wasn't getting all the answers back from the server because the out.recv was getting just one part of the reply. The obvious answer was to make a loop and listen for all the replies. I found the perfect solution at http://www.binarytides.com/receive-full-data-with-the-recv-socket-function-in-python/ .
I quickly recoded my program a bit and it works like a charm. Now I can move on forward with my whole project.
I hope this might help anyone else in the future with a similar issue.
Related
I'm trying to retrieve data from a PLC (AutomationDirect P2000). I have set up the PLC as the server with their software program (I can also connect to it with their software via Ethernet and use Wireshark to see it is in fact sending UDP packets to my machine at roughly every 200ms). I am trying to set up a very simple Python script to retrieve said data, without bothering to encode it or do anything with it, but my program hangs at the socket.recv(). Whenever I try to run it "Got here" will be printed, but "Now here" will not. From what I've read the fact that it hangs means there's no data to be received, but from my (limited) understanding of what I see on Wireshark this is not the case. I am pretty new to all of this and would appreciate any help.
I have tried using socket.recvfrom(), which produces the same result. I've also tried using socket.bind() instead of socket.connect() but I get a "The requested address is not valid in its context" exception. Additionally, I've tried playing around with various IPs and ports. For example, I've tried using IP = '' instead of the actual IP, and I've tried the source/destination information from Wireshark as what I try to bind or connect to, but nothing thus far has worked.
import socket
IP = '192.168.3.1'
PORT = 9999
BUFFER_SIZE = 4096
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((IP, PORT))
while True:
print("Got here")
data = s.recv(BUFFER_SIZE)
print("Now here")
print(f"Received {data}")
I am expecting to get a print out of the packet in byte format, but instead the program is hanging. If I try socket.bind() instead of socket.connect() I get an error message reading "...line 8, in
s.bind((IP, PORT))
OSError: [WinError 10049] The requested address is not valid in its context"
you can't use bind like this, because the ip address does not belong to your PC.
when you connect to the server, it (the server) doesn't send anything, but you try to get data from the server, so the socket awaits until it gets data, and only then it will continue the execution (this is called a blocking function, since it blocks the execution until it finishes).
The issue was with how I set up the PLC as the server. The UDP data I was seeing on port 9999 wasn't the communications I was thinking it was, and was only the inherent communication between the PLC and the network via its proprietary program. For anyone curious, I am using a P2000 PLC from AutomationDirect and initially I set it up as an EtherNet/IP Adapter following one of their videos, but I had to use the Custom Protocol over Ethernet functionality provided in the "Communications" section.
I'm new to the world of networking and wanted to clarify some of my thoughts regarding a problem I am facing at the moment. I saw this post which makes me think what I'm doing may be impossible, but I thought it would be worth a shot to ask on here and see what more qualified people think about it.
I am a TA for an intro computer science course, and I am writing a final project for students to complete at the end of the semester. Essentially, the project would be to fill in the holes in the implementation of a messaging client. I have set it up so each client would run two threads (one to listen for incoming messages, and one to wait for input to send messages to the other client). I have gotten this to work successfully on localhost with communication between two different port numbers, and am trying to find a way to have this work over the network so the two clients do not necessarily have to be on the same machine.
After struggling through a few methods, I came up with this solution: I would host a server on Heroku that would keep track of the clients' IPs and port numbers, and use a rest API so that one client could easily get the IP and port of the other client they are trying to communicate with. I have tested this, and the API seems to work. Thus, a client can create a socket endpoint and send it to this server to be entered into its database, and when the communication is terminated, it is removed from the database (this JSON would store a username as the primary key and internally manage an IP and port number) as the connection is now closed.
So, what I have is each client with an IP and port number knowing the IP and port number it is trying to communicate with. My last struggle is to actually form the connection. I understand there is a distinction between localhost (127.0.0.1) and the public IP for an internet endpoint. Upon searching, I found a way to find the public IP for the current user to share with the database, but I cannot bind to it. Whenever I try to, I get sockets error code 13: permission denied. I would imagine that if I tried connecting to the public IP of the other machine, I would get a similar error (but I cannot test the client until I can get a server running!).
I read online that some router work would be needed to actually form this connection between two machines. I guess I'm struggling to understand the practicality of socket programming if such a simple operation (connecting two socket endpoints on two different computers) requires so much tweaking. Is there something I am missing?
For reference, here is a general outline of my code thus far. The server:
# Server thread
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((LOCAL_IP, AVAILABLE_PORT))
s.listen(1)
# In my code, there is a quitting mechanism which closes s as well.
while True:
client_socket, addr = s.accept()
data = client_socket.recv(1024)
print "Received: " + data
client_socket.close()
...and the client:
# Client thread
# It is an infinite loop so I am always waiting for another potential message to send
while True:
x = raw_input()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((OTHER_MACHINE_LOCAL_IP, OTHER_PORT))
sock.sendall(x)
sock.close()
Right now, I cannot make progress with these permission denied errors. Is there any way I could get this to work? Understand that, being that this would be for around 250ish students to use who are all intro CS students, I would want to avoid having to instruct them to do anything with their routers.
If there is another method to do this which would make this easier that I am missing, I would also love to hear any suggestions :) Thanks in advance!
I'm currently working with python's socket library for the first time and i'm not very experienced with computer networking.
I'm able to connect to the server and the tcp handshake has happened as viewed by wireshark. After establishing a connection to the server(I have no control over the server), the connection stays open for a while, but after a small amount of time, the server sends a "FIN, ACK" and the connection is terminated. I'm trying to understand how I can keep this connection alive while the client is capable of reaching the server.
Looking at a tcp connection, it seems a packet can be sent every so often. Maybe a sort of keep alive message. I had thought using socket.send('hello') every 5 seconds in another thread would keep the connection with the server open, but I still get the "FIN, ACK" after some time.
In the documentation I found a setsockopt() but using this made no noticeable difference. I've tried client.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) both before and after the connection is made. I don't completely understand how this method is supposed to work, so maybe I used it incorrectly. There isn't much mention of this. I read somewhere about it being broken on windows. I don't know the truth in that.
What am I missing? The documentation for sockets doesn't seem to have anything about this unless I may have missed something.
import socket
import time
import threading
SERVER_IP = 'THE SERVER'
SERVER_PORT = SERVER_PORT
SOURCE_IP = socket.gethostname()
SOURCE_PORT = 57004
KEEP_ALIVE_INTERVAL = 5
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def keep_alive(interval):
data = 'hello'
while True:
client.send(data)
time.sleep(interval)
client.connect((SERVER_IP, SERVER_PORT))
t = threading.Thread(target=keep_alive, args = (KEEP_ALIVE_INTERVAL,))
t.start()
while True:
data = client.recv(1024)
if not data:
break
print data
client.close()
For enabling keep alive there is a duplicate question at How to change tcp keepalive timer using python script?
Keep in mind some servers and intermediate proxies forcibly close long lived connections regardless of keep alives being used or not, in which case you will see a FIN,ACK after X amount of time no matter what.
I have a program (.exe) that gets certain info from a server, and I wanted to be able to get that info from the command line too. I started netcat and listened on the port the program uses to communicate with its target, because I wanted to know what "requests" to make from a script, and I received this over netcat, plain text:
net.tcp://[my ip address]:41012/Lapis.Btouch/ServerInfo
I tried sending exactly that to the target (replacing my IP for its IP) using socket.send() but it wouldn't return anything. What does that mean and how can I get the data?
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('[server ip address]',41012))
while True:
s.send('net.tcp://[server ip address]:41012/Lapis.Btouch/ServerInfo')
response = s.recv(1024)
print response
s.close()
That's the code I'm using to send the request to the target server. It won't return anything, so I think I'm not making the request in the right way. Thanks for any help.
Try capture your network packet on port 41012, via tcpdump or wireshark or ...
Then check:
does your code send the request?
does the response returned to you?
If the answer for question (1) is False, problem is side of you and your machine. so go and solve it (review your code, check your firewall and ...)
If the answer of question (1) is True, but server doesn't send any response (this case often not happened) review your request format (check you URL or test it via browser or ...). also based on the server you connect to, maybe needed to your machine's IP recognized for server (I mean some server only give response to requests, that come from known IP addresses, in this case you need introduce yourself to server (e.g. add your IP to trusted list on server), before sending request)
Last case, if answer of both questions are correct (I guess it's your problem), you most correct your code (response = s.recv(1024)) why do you used 1024 for your response length?. Use correct and accurate parameters.
In python exist several methods for receiving response via socket (from server), search stackoverflow and you can find useful tips and commands. commands like use non-blocking ways, ascync ways and ...
So I'm messing around with a very simple echo server/client kind of deal. I'm eventually going to be making a game server, but for now I'm just trying to setup the basic parts.
Here's the server: http://pastebin.com/qtfrMGur
Here's the client: http://pastebin.com/3VK3VxPX
So, my problem is that when the client disconnects, I get socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host. I was under the impression that select.select() returned false(or an empty list, whatever) when the readable socket list had no pending information. This seems to be the cause of the error, but then why is select.select() returning true(on line 26) if the client is no longer there? The actual error occurs when I try to retrieve some info in line 27. I'm really confused by this, assuming the client-side of the socket is closed, shouldn't it just return [], and then go to the else on line 31 and destroy that client?
I don't really know what else to post, this seems like a very elementary problem, usually I use a more abstract library like twisted that makes it easy, but for this project I must use traditional BSD sockets. If you guys have any other info you need just ask.
Thanks.
EDIT: So I put a print statement in there to print the value of select.select(...)[0], like so:
for cl in clients:
ready = select.select([cl], [], [], .1)[0]
print ready
if ready:
data = cl.recv(size)
if data:
print data
#cl.send(data)
else:
print "Client Quit\n"
clients.remove(cl)
cl.shutdown(socket.SHUT_RDWR)
cl.close()
print len(clients)
As you can see I do a proper shutdown now as well. The problem is, that right as the client quits, select.select(...) returns the client socket as a readable source, so cl.recv gets called on it, which then throws an error because cl is closed on the other end. At least this is my suspicion.
So, could somebody please explain what I may be doing wrong here? Why does the client send a last dying message, is it some kind of functionality that I should be taking advantage of?
Thanks
EDIT2: So guys I fired up my debian machine and the code works flawlessly, it's only in windows that it's doing this. Is this some sort of bug? I was under the impression that the socket library abstracted all the stuff beneath the hood to make the interface identical between OS.
You should call the shutdown method of the socket, before closing it on the client side, like this:
s.shutdown(socket.SHUT_RDWR)
s.close()
That way it won't allow further sending / receiving, check out the Python docs about it, it's nicely explained there.
UPDATE: fixed client code
import socket
host = 'localhost'
port = 50000
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host,port))
msg = ''
while msg !='quit':
msg = raw_input()
s.send(msg)
#data = s.recv(size)
s.shutdown(socket.SHUT_RDWR)
s.close()