I can't send pickle object from raspberry with Linux through socket as a client to an online windows server while I can send it from my computer without any problem. The weird thing that I can send the message length (the server receive the message length ) but can't send the pickle object and the server is stuck to the while loop after receiving that message.
client code:
message = pickle.dumps(faceBlob)
message_header = bytes(f"{len(message):<{HEADER_LENGTH}}", "utf-8")
client_socket.send(message_header + message)
server code:
try:
# Receive our "header" containing message length, it's size is defined and constant
message_header = client_socket.recv(HEADER_LENGTH)
# If we received no data, client gracefully closed a connection, for example using socket.close() or socket.shutdown(socket.SHUT_RDWR)
if not len(message_header):
return False
# Convert header to int value
message_length = int(message_header.decode('utf-8').strip())
fragments = []
print(message_length)
received_len = 0
while received_len < message_length:
chunk = client_socket.recv(message_length - received_len)
if not chunk:
break
fragments.append(chunk)
received_len += len(chunk)
data_arr = b"".join(fragments)
# Return an object of message header and message data
return {'header': message_header, 'data': data_arr}
except:
return False
Related
I'm trying to send a json containing text fields and a buffer in a bytearray, from a micro-controller to a Windows server
msg = {"some_stuff": "some_stuff", "buf": bytearray(b'\xfe\xc2\xf1\xfe\xd5\xc0 ...')}
Note that the buffer is quite long (so that I can't put it here as reference) len(buf) -> 35973
I'm sending the length of the message before to the server so that it knows how long is the message to be received
def send_json(conn, msg):
msg = json.dumps(msg).encode('utf-8')
msg_length = len(msg)
header = str(msg_length).encode('utf-8')
header += b' ' * (64 - len(header))
conn.send(header)
conn.send(msg)
The receiving function is then
def receive_json(conn) -> dict:
msg_length = int(
conn.recv(64).decode('utf-8').replace(' ', '')
)
msg_b = conn.recv(msg_length)
msg_s = msg_b.decode('utf-8')
try:
msg_d = json.loads(msg_s)
except:
msg_d = eval(msg_s)
return msg_d
The problem is that the received message is truncated.
msg_b = b'{"buf": bytearray(b\'\\xfe\\xc2\\xf1 ... \\x06u\\xd0\\xff\\xb'
It's worth mentioning that while in debug, if I stop for a while with a breakpoint on line msg_b = conn.recv(msg_length), before running it, the received message is complete.
So it seems that in the receiving function the conn.recv(msg_length) instruction does not wait to receive a message of the specified length (msg_length)
Why is it the case? What can I do to receive a complete message?
I could introduce time.sleep between receiving the length of the message and the message, but how to know how much to wait depending on the message length?
Thank you
My solution was to check for how much of the message is missing and iterate till the message is complete
def receive_json(conn) -> dict:
msg_length = int(
conn.recv(64).decode('utf-8').replace(' ', '')
)
buf = bytearray(b'')
while len(buf) < msg_length:
missing_length = msg_length - len(buf)
packet = conn.recv(missing_length)
buf.extend(packet)
msg_s = buf.decode('utf-8')
try:
msg_d = json.loads(msg_s)
except:
msg_d = eval(msg_s)
return msg_d
TCP is a streaming protocol that guarantees delivery of bytes in the order sent, but not with the same send breaks. You need to define a protocol (which you have, as a 64-byte header of message size, then the message data), and then buffer reads until you have a complete message.
Python sockets have a .makefile method that handles the buffering for you, where you can .read(n) a specific number of bytes or .readline() to read a newline-terminated line. With this you can implement the following client and server:
server.py
import socket
import json
import time
s = socket.socket()
s.bind(('',5000))
s.listen()
while True:
c,a = s.accept()
print(f'{a} connected')
# wrap socket in a file-like buffer
with c, c.makefile('rb') as r: # read binary so .read(n) gets n bytes
while True:
header = r.readline() # read header up to a newline
if not header: break # if empty string, client closed connection
size = int(header)
data = json.loads(r.read(size)) # read exactly "size" bytes and decode JSON
print(f'{a}: {data}')
print(f'{a} disconnected')
client.py
import socket
import json
def send_json(conn, msg):
# smaller data size if non-ASCII used.
data = json.dumps(msg, ensure_ascii=False).encode()
msg_length = len(data) # length in encoded bytes
# send newline-terminated header, then data
conn.sendall(f'{msg_length}\n'.encode())
conn.sendall(data)
s = socket.socket()
s.connect(('localhost',5000))
with s:
send_json(s, {'name':'马克'}) # check to support non-ASCII properly
send_json(s, [1,2,3])
Start server.py, then run client.py a couple of times:
Output:
('127.0.0.1', 26013) connected
('127.0.0.1', 26013): {'name': '马克'}
('127.0.0.1', 26013): [1, 2, 3]
('127.0.0.1', 26013) disconnected
('127.0.0.1', 26015) connected
('127.0.0.1', 26015): {'name': '马克'}
('127.0.0.1', 26015): [1, 2, 3]
('127.0.0.1', 26015) disconnected
I am attempting to complete a challenge that states:
We've noticed that the aliens are sending messages between their ships, we think they're using XOR to encrypt the messages, and we've intercepted a key.
Set up a server listening on ("localhost", 10000) to intercept one of the alien messages. When you do perform a bitwise XOR on the message with the key "attackthehumans" and then respond with the encrypted data.
Tip: Read the response to get the flag.
After some research, I was able to come up with the following code. However, when I run it in the challenge's code editor, the only feedback I receive is "Error trying to connect to your server and recieving message back."
import socket
# Function to xor strings
def xor_string(string):
key = "attackthehumans"
bit_key = ''.join(format(ord(i), 'b') for i in key)
bit_data = ''.join(format(ord(i), 'b') for i in string)
xor_string = str(0)
for i in range(len(bit_key)):
if bit_data[i] == bit_key[i]:
xor_string = xor_string + str(0)
else:
xor_string = xor_string + str(1)
return xor_string
# Sets up server on localhost and port 10000
print("Setting up server...")
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("localhost", 10000))
print("Server set up. Listening for connections...")
server.listen()
conn, addr = server.accept()
print("Connected by: {}".format(addr))
# Once connection is established, server receives data, XOR's it, sends it back, and
# waits for response to get the flag.
with conn:
print("Receiving data from connection... ")
data = conn.recv()
data = data.decode()
print("Data received: {}".format(data.decode()))
xor_data = xor_string(data)
conn.sendall(xor_data.encode())
response = conn.recv()
response = response.decode()
print(response)
I am not sure what I'm doing wrong. I can't figure out if it's the socket or the xor_string function. Any help would be appreciated.
This should be enough to do your XORing:
def xor_string(string):
key = b"attackthehumans"
key = key * (len(string)//len(key)+1)
res = bytes([k^s for (k,s) in zip(key,string)])
return res
Then your main code becomes:
print("Receiving data from connection... ")
data = conn.recv()
print("Data received:", data)
xor_data = xor_string(data)
conn.sendall(xor_data)
print(conn.recv())
I have this code, that print the http server response, but now I'm trying to get the only the status code, and from there make decisions.
like :
Code:200 - print ok
code:404 - print page not found
etc
PS: cant use http library
from socket import *
#constants variables
target_host = 'localhost'
target_port = 80
target_dir = 'dashboard/index.html'
# create a socket object
client = socket(AF_INET, SOCK_STREAM) # create an INET (IPv4), STREAMing socket (TCP)
# connect the client
client.connect((target_host,target_port))
# send some data
request = "GET /%s HTTP/1.1\r\nHost:%s\r\n\r\n" % (target_dir, target_host)
#Send data to the socket.
client.send(request.encode())
# receive some data
data = b''
while True: #while data
buffer = client.recv(2048) #recieve a 2048 bytes data from socket
if not buffer: #no more data, break
break
data += buffer #concatenate buffer data
client.close() #close buffer
#display the response
print(data.decode())
I would change the reception loop as below: extract the first line, split it, interpret the second word as an integer.
line = b''
while True:
c = client.recv(1)
if not c or c=='\n':
break
line += c
status = -1
line = line.split()
if len(line)>=2:
try:
status = int(line[1])
except:
pass
print(status)
If we heavily rely on try we can simplify the second part
try:
status = int(line.split()[1])
except:
status = -1
print(status)
I have a server-client setup that works as follows:
The client connects to the server.
The client sends the server a 64-byte message telling the server how much data to read
The server reads that many bytes of data, responds, and the process repeats.
When the client is finished, it sends the server a null message
The server sees that the message length is 0 and closes the connection.
This seems to work fine for the first pass. After the server responds though, it doesn't wait for the client to send more data. Instead the server immediately reads 64 bytes. Since the client hasn't responded, the length of the message is 0 and the connection is closed.
I'm unsure why the server is not pausing until the client sends more data.
Here is the server loop:
self.__stop = False
while not self.__stop:
if self.client_sock:
# Check if the client is still connected and if data is available
try:
rdy_read, rdy_write, sock_err = select.select(
[self.client_sock, ], [self.client_sock, ], [], 5)
except select.error as err:
self.stop()
return
if len(rdy_read) > 0:
# msg length will be sent as bytes
read_data = self.client_sock.recv(64)
# Check if the socket has been closed
if read_data == 0:
self.stop()
else:
msg_length = int(read_data)
msg = self.client_sock.recv(msg_length)
response = f"{[12, 15, 66]}\n"
msg_size = padded_size_of_msg(response)
self.client_sock.send(msg_size)
self.client_sock.send(f"{response}".encode('utf-8'))
else:
print(
f"[THREAD {self.number}] No client connected.")
self.stop()
self.close()
The function padded_size_of_msg() is to calculate the length of the message, pad that number to be 64-bytes, then send that to the client:
def padded_size_of_msg(msg):
msg_length = len(msg)
send_length = str(msg_length).encode('utf-8')
send_length += b' ' * (64- len(send_length))
return send_length
The complete class declaration is below:
class ServerSocketThread(Thread):
def __init__(self, client_sock, client_addr, number):
Thread.__init__(self)
self.client_sock = client_sock
self.client_addr = client_addr
self.number = number
def run(self):
self.__stop = False
while not self.__stop:
if self.client_sock:
# Check if the client is still connected and if data is available
try:
rdy_read, rdy_write, sock_err = select.select(
[self.client_sock, ], [self.client_sock, ], [], 5)
except select.error as err:
print(
f"[THREAD {self.number}] Select() failed on socket with {self.client_addr}")
self.stop()
return
if len(rdy_read) > 0:
# msg length will be sent as bytes
read_data = self.client_sock.recv(64)
# Check if the socket has been closed
if read_data == 0:
print(
f"[THREAD {self.number}] {self.client_addr} closed the socket")
self.stop()
else:
msg_length = int(read_data)
# Client will send msg as bytes. No need to decode to str
msg = self.client_sock.recv(msg_length)
response = f"{[12, 15, 66]}\n"
# Send outputs to client as bytes
msg_size = padded_size_of_msg(response)
self.client_sock.send(msg_size)
self.client_sock.send(f"{response}".encode('utf-8'))
else:
print(
f"[THREAD {self.number}] No client connected.")
self.stop()
self.close()
def stop(self):
self.__stop = True
def close(self):
if self.client_sock:
print(
f"[THREAD {self.number}] Closing conn with {self.client_addr}")
self.client_sock.close()
This ended up being an issue with using nc from the command line.
If I were to try and send "test" from the client, it would register the size as 4-bytes and tell the server to read 4-bytes. From the terminal, nc would instead send "test\n". The newline character '\n' would not be read and instead waited in the queue. When the second recv() was called, it immediately read the '\n' and took that as an indication to close the connection.
I'm trying to make a server-client program, where server will listen to client's messages and depends on the message, will response. I send a message from client with username and content, server accept it and print a message to sending to client Till here is everything fine. But when it comes to sending a message server will throw and error:
`TypeError: byte indices must be integers or slices, not str`
It looks like this line is the problem, but I'm not sure....
`clientsocket.send(msg['header'] + msg['data'])`
here is a whole server code. Please let me know, if client code is necessary too please.
import socket
import time
import pickle
import select
HEADERSIZE = 10
IP = "127.0.0.1"
PORT = 1234
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((IP, PORT))
s.listen()
sockets_list = [s]
clients = {}
# Handles message receiving
def receive_message(clientsocket):
try:
message_header = clientsocket.recv(HEADERSIZE)
if not len(message_header):
return False
message_length = int(message_header.decode('utf-8').strip())
return {'header': message_header, 'data': clientsocket.recv(message_length)}
except:
return False
while True:
read_sockets, _, exception_socket = select.select(sockets_list, [], sockets_list)
for notified_socket in read_sockets:
if notified_socket == s:
clientsocket, address = s.accept()
user = receive_message(clientsocket)
if user is False:
continue
sockets_list.append(clientsocket)
clients[clientsocket] = user
print(f"Connection from {address[0]}:{address[1]} has been estabilished! User:{user['data'].decode('utf-8')}")
else:
message = receive_message(notified_socket)
if message is False:
print(f"Close connection from {clients[notified_socket]['data'].decode('utf-8')}")
sockets_list.remove(notified_socket)
del clients[notified_socket]
continue
user = clients[notified_socket]
#message_decoded = message['data'].decode('utf-8')
print(f'Received message from {user["data"].decode("utf-8")}: {message["data"].decode("utf-8")}')
for clientsocket in clients:
if clientsocket == notified_socket:
if message["data"].decode("utf-8") == "y":
#d = {1: "Hey", 2: "there"}
msg = pickle.dumps("th.jpeg")
print(msg)
msg = bytes(f'{len(msg):<{HEADERSIZE}}', "utf-8") + msg
clientsocket.send(msg['header'] + msg['data'])
else:
d = {1: "Hey", 2: "there"}
msg = pickle.dumps(d)
print(msg)
# msg = bytes(f'{len(msg):<{HEADERSIZE}}', "utf-8") + msg
clientsocket.send(msg['header'] + msg['data'])
for notified_socket in exception_socket:
sockets_list.remove(notified_socket)
del clients[notified_socket]
HERE is a whole error code:
Connection from 127.0.0.1:48480 has been estabilished! User:j
Received message from j: y
b'\x80\x03X\x07\x00\x00\x00th.jpegq\x00.'
Traceback (most recent call last):
File "server.py", line 69, in <module>
clientsocket.send(msg['header'] + msg['data'])
TypeError: byte indices must be integers or slices, not str
As You can see, it works till sending the message line
msg = pickle.dumps("th.jpeg") will encode the string "th.jpeg" as a bytes-object.
msg = bytes(f'{len(msg):<{HEADERSIZE}}', "utf-8") + msg just adds that bytes object to another bytes-object.
So msg is a simple bytes-object, not any kind of server packet or similar. Therefor it is not possible to subscribe msg with msg['header'] or any other string.
Your code seems a little weird but maybe just try this line:
clientsocket.send(msg)
Since you are already converting msg to a bytes-object, it can be sent to the client directly. You just have to decode it properly in the client.