i am relatively new to python, so please be considerate...
i'm implementing a server and a client via raw_sockets.
i have the necessary privileges.
now, the server i defined so:
host = socket.gethostbyname(socket.gethostname())
address = (host, 22224)
sockSer = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
sockSer.bind(address)
sockSer.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)
packet, addr = sockSer .recvfrom(4096) # wait for packet from client
Q1) why can't i simply type: hosts = 'localhost'.
if i do so, it doesn't allow me to write the line: sockSer.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON). and then the server doesn't receive my client's messages.
only when doing gethostbyname(socket.gethostname()) i get 192.168.1.101
and then it works.
in a different class:
the client socket:
host = socket.gethostbyname(socket.gethostname())
address = (host, 22224)
sockCli = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
Q2) do i also need to type: sockCli.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)
or maybe sockCli.connect(address)? seems that it works without the connect command.
for the client socket?
now, the problems arise when i do the following:
1) send a packet from client to server:
header=...
payload='a'
sockCli.sendto(header + payload, address)
2) receive packet in server and send something back to client:
while(true):
data, addr = sockSer.recvfrom(4096)
header2=...
payload2='b'
sockSer.sendto(header2 + payload2, addr)
now, my important question is:
Q3) the server sent only 1 packet to client, with payload 'b'.
what happens is, my client actually receives 2 packets in the while loop:
first packet is what the client itself sent to server, and the other packet is what the client got from the server.
hence my output is 'ab' instead of simply 'b'
why is this happening???
NOTE: i didn't type the entire code, but i think my syntax,parsing,header composition etc.. are correct.
is there an obvious problem in my code?
if necessary i'll upload the entire code.
thanks
I got this too.
my solution is add a judge in the receive code,such as if I send Ping package so I only want ECHO　Reply( type 0 code 0), I write
if type != 0:
continue
and you also can write as
if addr == my_ip:
continue
It seems not has any smooth solution
Q1: I was able to bind to localhost and call IOCTL with both parameters just fine. Assuming your client is also running on the same system, ensure the client is sending to "localhost", otherwise your server will never receive the packets. If your client is on another system, obviously your server will never receive the packets.
Q2: You do not need IOCTL for sending the packet. Just send it via sendto().
Q3: The reason you're seeing two replies is, the kernel is also processing the echo request, in addition to your own user-space code.
Although you can use ICMP for arbitrary message passing, as someone else pointed out this isn't its intended design. You may find that your data portion is truncated out in message replies. For example, when sending echo requests, your reply likely will contain everything you sent; however, a reply that is type 3 code 3 may not include your data, but only the first 8 bytes of the ICMP header.
Related
I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)
recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...
Suppose we have an echo server and a file transfer app.
There is a sender (Client) to send files and a receiver (Server) to receive files. The echo server will echo whatever received from the Client and the Server.
However, Client and Server cannot communicate directly, i.e., all packets have to go through the echo server. For example, the Client sends a UDP packet to the Echo server, and the Echo server echo that packet to Server and Server send Acknowledgment to the Echo server, and the Echo server echo that ack packet to the Client.
The objective is to implement a reliable UDP for file transfer. And we have only one UDP socket.
This figure demonstrates what the setup is
Client, Server and Echo Server
I have tried to use multi-thread and select.select and both do not work perfectly
The issue with multi-thread is that since Client and Server cannot communicate internally, and we have only one socket, it is difficult to choose who should send or receive now.
The issue with select.select is that the return list always has writeable non-empty, which makes the Client continues to send a bunch of packets before the readable is ready.
Here is the implementation for both Client and Server inside one file (say transceiver.py) what I do not use select.select (instead using send bool variable) but it seems to work fine. But I do believe this is bad practice, so I wonder what can I do to improve my design.
def create_transceiver(ip, port):
address = (ip, port)
udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp.settimeout(1)
inout = [udp]
client = Client(udp, address)
server = Server(udp, address)
client_to_server = True
send = True
while True:
# infds, outfds, errfds = select.select(inout, inout, [])
if not send: # len(infds) != 0
if client_to_server:
server.start_recv()
client_to_server = False
else:
client.start_recv()
client_to_server = True
send = True
elif send: # len(outfds) != 0
if client_to_server:
if client.has_ack_all():
print(server.write_content())
break
client.start_send()
client_to_server = True
else:
server.start_send()
client_to_server = False
send = False
Here is the implementation of Echo Server:
import socket
ip = "10.10.1.100"
port = 8888
address = (ip, port)
udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_socket.bind(address)
while True:
data, address = udp_socket.recvfrom(2048)
udp_socket.sendto(data, address)
And we have only one UDP socket.
As far as I understand this objective, you have a server process somewhere (you call it echo server) that is listening on a specific port. Also there is a client that wants to send data to some sort of server.
You provided code shows an implementation of what is called a state machine that (in your case) switches between receiving and sending.
However, Client and Server cannot communicate directly
The scenario you are describing makes your echo server a classic server that handles different types of clients. In your case this would be your "client" and your "server". I would like to call these just client-A and client-B. Most tutorials on the internet would call them Alice and Bob, I guess.
The objective is to implement a reliable UDP for file transfer.
So you want to transfer files between different clients using UDP as the base protocol.
UDP is not very well suited for this purpose. It does not garantee delivery of each packet transmitted. It is possible that packets arrive in different order than they were sent.
Usually you would use TCP for this kind of transmission. UDP is usually used for live streaming data like audio/video calls and stuff like that.
For more information on the differences between UDP and TCP you might check out the wikipedia pages for each:
https://en.wikipedia.org/wiki/User_Datagram_Protocol
https://en.wikipedia.org/wiki/Transmission_Control_Protocol
It is possible to use UDP for your transfers but you would have to implement all the safeties provided by TCP yourself.
I assume, your client and your server are actually different programs. Otherwise there would be a way they could communicate directly.
If that is the case, this tutorial might give you a starting point:
https://pythonprogramming.net/server-chatroom-sockets-tutorial-python-3/
I have a Python program that uses a sockets to send and receive UDP multicast messages on multicast IP address 224.0.1.1 and UDP port 20001.
On the receive the side, I create a single receive socket, and call socket.setsockopt once using socket option IP_ADD_MEMBERSHIP to join the IP multicast group.
However, Wireshark reports that the single call to setsockopt causes two separate join (IGMPv2 membership report) messages to be sent out:
One join message using Ethernet source address 01:00:52:00:01:01, which is the Ethernet multicast address corresponding to the IP multicast group.
One join message using Ethernet source address a8:66:7f:3a:2b:1a, which is the Ethernet unicast address corresponding to the physical "en0" interface over which the join message was sent.
On the send side, I create a single send socket, an call socket.connect to associate the socket with multicast IP address 224.0.1.1 and UDP port 20001.
I then call socket.send once to send a single test message. Because of the two separate join messages, the sent test message appears TWICE on the wire, once with destination Ethernet address 01:00:52:00:01:01 and once with destination Ethernet address a8:66:7f:3a:2b:1a.
On the receiving side, both messages are received separately. Thus each sent message is received TWICE.
The question is: how can I prevent this from happening?
A minimal example that reproduces the behavior is as follows:
import socket
import struct
import time
mcast_ipv4_address = "224.0.1.1"
port = 20001
group = (mcast_ipv4_address, port)
txsock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
txsock.connect(group)
rxsock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
req = struct.pack("=4sl", socket.inet_aton(mcast_ipv4_address), socket.INADDR_ANY)
rxsock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, req)
rxsock.bind(group)
time.sleep(1.0)
print("Sending single message...")
msg = b"test"
txsock.send(msg)
print("Sent {}".format(msg))
print("Receiving first message...")
msg = rxsock.recv(65535)
print("Received {}".format(msg))
print("Receiving second message...")
msg = rxsock.recv(65535)
print("Received {}".format(msg))
time.sleep(0.1)
Additional details:
1) The operating system is macOS High Sierra 10.13.5
2) The Python version is 3.5.1
3) The first sleep is essential; without it the problem does not occur because it takes some time for the join messages to be sent
4) The second sleep is not essential; it is there to make sure both test messages are seen by wireshark before the program terminates and the leave messages are sent.
5) I tried using the real IP address of the outgoing interface in stead of INADDR_ANY in the req structure, but it does not make any difference.
I found the answer to my own question:
If you disable the IP_MULTICAST_LOOP option on the sending socket, then:
1) Wireshark will STILL report two IGMPv2 join messages, same as before
2) Wireshark will STILL report two UDP multicast messages, same as before
3) However, the receiving socket will only receive a single UDP multicast message (the example program will block on "Receiving second message...")
Here is the updated code for macOS:
txsock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM,
socket.IPPROTO_UDP)
txsock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_LOOP, 0) # <<< FIX
txsock.connect(group)
Annoyingly, the behavior on Linux is the opposite:
If you leave IP_MULTICAST_LOOP to its default value of enabled, as in the original example program, you will receive exactly one copy of the sent packet.
If you disable IP_MULTICAST_LOOP, as in the "fixed" example program, you will not receive any copy of the sent packet (at least not on AWS).
After further investigation, I found that the behavior does not depend on the platform on which the code runs (macOS vs Linux) but on the router to which the platform is connected.
I was debugging a python program, the application can't receive udp packets as expected. Finally I found it was the UdpSocket.connect cause the UdpSocket to lose these packets. See below code:
def main:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.connect((server_ip, 9)) #This results in the issue
#Use sock.bind((local_ip, 12345))
#instead would solve the problem
localip, localport = sock.getsockname()
packet = GetRegisterData(localip, localport)
sock.sendto(packet, (server_ip, 5000)) #the host (server_ip, 5000) will
#always send back two udp packets
#to respond this statement
sleep(1)
while True:
response = sock.recv(1024) #packets from (server_ip, 5000) reached
#machine but this statement never return
if not len(response)
break
print response
I am very new to Python, and don't understand why this would happen. Any body helps explain this?
[Update]
I use tcpdump to capture packets, only to find the lost packets have reached the machine, but due to unkown reason the sock.recv just doesn't retuern. I want somebody to help explain why the sock.recv doesn't return everytime here.
You didn't mention where the packets that you expect to receive (but fail to) are coming from. I'm guessing they're not coming from the address you connected to, though. See the man page for connect(2) - which is what you're calling when you use this Python API - for information about why this matters. In particular:
If the socket sockfd is of type SOCK_DGRAM then addr is the address to which datagrams are sent by default, and the only address from which datagrams are received.
(emphasis mine).
I have two functions. One sends a UDP packet to a port number and returns the port number if it gets a response. The second cycles through addresses calling the first function repeatedly incrementing the port number. It stops when a port number is returned. I'm basically spamming an IP to find which port number is being used.
All works well with very long time outs but I'm trying to speed up the rate at which I send my test packets. For example I might get a packet back from port 27018 even though the spam port function is sending to 27022. It then incorrectly reports that 27022 is the port to use. This happens even when I return the returned packet info from the first function since you can't tell the arguments which were used originally.
def check_port(s, serverip, serverport):
payload = "ffffffff54536f7572636520456e67696e6520517565727900".decode('hex')
s.sendto(payload, (serverip, serverport))
while 1:
try:
s.settimeout(0.1) # time to wait for response
d = s.recvfrom(1400)
data = d[0]
addr = d[1]
if len(data) > 1:
break
except socket.error:
return False
return addr[1]
def spam_ports(serverip):
s = serverComms.port_config()
start_highport = 27015
end_highport = 28015
start_lowport = 2299
end_lowport = 4000
port = start_highport
while check_port(s,serverip, port) == False:
port += 1
if port == end_highport:
port = start_lowport
if port == end_lowport:
return 'no query ports found in range'
else:
return check_port(s,serverip, port)
I really appreciate any help with this.
I think I know what happens.
It takes some time for the server to reply. If the delay is shorter than that, your application becomes confused. Let me explain:
You send packages to port 1000, 1001, 1002, ...
Say port 1010 produces a reply. But lets assume the server needs a full second to reply. Your application has progressed well bejond 1010 since the timeout is less then a second. By the time the reply arrives your application is already at, say, 1020. Now it looks like the received package is the result of sending something to 1020.
Possible approch
What you need is a way to know to which port a received reply belongs. Here it gets tricky:
Each package has a source and a destination port. With the packages you send the destination port is incremented. The source port is probably arbitrarly assigned. When the server answers it will send a package with an arbitrary source port and the destination port equal the source port of your package.
What you could do is check with the documentation and see how you can control the source port of the packages you're sending. With that you make sure that each sent package has a different source port. This identifies each package uniquely. Now you can use the destination port of the reply to know were it belongs to.
Then you can use, for example, two threads. One sending out packages and one receiving the answers. You keep a dict that maps my_source_port to server_port. The sender fills the dict and the receiver reads it. You can let the sender send as fast as it can, now timeout required. Once the sender is done you give the receiver thread a second or so to catch up.
port_map = {}
active_ports = []
def send(min, max):
for dest_port in range(min,max):
src_port = get_free_port()
port_map[src_port] = dest_port
send_package(src_port, dest_port, 'somedata')
def recv():
while True:
src_port, dest_port, data = receive_package()
port = port_map[dest_port]
active_ports.append(port)
This is not a working example, just the basic idea. The methods don't exist in that form, thread synchronization is missing and recv() would run forever, but it shows the basic idea.
You probably have to create a socket for each package you send.