Develop Reference

sending multiple images using socket python get sent as one to client

I am capturing screenshots from the server, then sending it to the client, but the images get all sent as one big file to the client that keeps expanding in size. This only happens when i send from one machine to another (I am working on a local netwrok) but when running both client and server from my machine they work fine.
Note: for the client on the other machine, I packaged it into an exe using pyinstaller, since this machine does not have python.
server code:
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
import time
import pyautogui
from socket import *
import os
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
print("Listening for connections.. ")
q,addr=s.accept()
i = 0
while True:
screenshot = pyautogui.screenshot()
screenshot.save(str(i) + ".jpg")
with open(str(i) + ".jpg", "rb") as f:
data = f.read(4096)
while data:
q.send(data)
data = f.read(4096)
q.send(b"full")
i += 1
time.sleep(0.3)
client code:
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
from multiprocessing.reduction import recv_handle
from socket import * # Imports socket module
s=socket(AF_INET, SOCK_STREAM) # Creates a socket
s.connect((host,port))
i = 0
while True:
with open(str(i) + "s.jpg", "wb") as f:
recv_data = s.recv(4096)
while recv_data:
f.write(recv_data)
recv_data = s.recv(4096)
if(recv_data == b"full"):
break
i += 1

There various wrong assumptions here which lead to the problem you see. The wrong assumptions are:
that send(data) will write all data It might send less. You need to check the return value or use sendall to be sure.
that a single send in the sender is matched by exactly a single recv in the recipientTCP is only an unstructured byte stream. send does not add message semantics, so a single send might lead to multiple recv, multiple send might lead to a single recv etc. Specifically send("data") followed by send("full") might be recv(4096) as "datafull", thus missing your code to detect end of image.
As for why does it work on the local machine but not on the remote - the chance in the latter case is higher that send get combined together and recv as one.

As stated by Steffen Ulrich you should use sendall for sending and for receiving we create a specialized function my_recv that will repeatedly call socket.recv until the expected number of bytes have been received. Also, a 4-byte header (you can make the length greater if your file sizes warrant this) that contains a binary representation of the file length precedes the sending of the actual file data. In this way the client knows exactly how much data it should receive for each file.
Server Code
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
import time
import pyautogui
from socket import *
import os
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
s.listen(1) # This should be called
print("Listening for connections.. ")
q,addr=s.accept()
i = 0
while True:
screenshot = pyautogui.screenshot()
screenshot.save(str(i) + ".jpg")
with open(str(i) + ".jpg", "rb") as f:
# Get length by positioning to end of file
image_length = f.seek(0, 2)
f.seek(0, 0) # Seek back to beginning of file
# Convert image length to a 4-byte array:
image_length_bytes = image_length.to_bytes(4, 'big')
q.sendall(image_length_bytes)
data = f.read(4096)
while len(data):
q.sendall(data)
data = f.read(4096)
i += 1
Client Code
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
from multiprocessing.reduction import recv_handle
from socket import * # Imports socket module
s=socket(AF_INET, SOCK_STREAM) # Creates a socket
s.connect((host,port))
def my_recv(msg_length):
chunks = []
bytes_to_recv = msg_length
while bytes_to_recv:
chunk = s.recv(bytes_to_recv)
if chunk == b'':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_to_recv -= len(chunk)
return b''.join(chunks)
i = 0
while True:
image_length_bytes = my_recv(4)
image_length = int.from_bytes(image_length_bytes, 'big')
with open(str(i) + "s.jpg", "wb") as f:
bytes_to_recv = image_length
while bytes_to_recv:
recv_data = my_recv(min(4096, bytes_to_recv))
f.write(recv_data)
bytes_to_recv -= len(recv_data)
i += 1

Python send <\n> over tcp socket

We have a string which should be send to a service via a websocket. This string containt some data which is seperated by some tags. One row of data always ends with <\r><\n>.
So we want to send each row of data to this service which is implemented as a TCPServer. This is working perfectly fine in case we use normal strings, but as soon as we try to incorporate the tags I mentioned, Python just keeps appending them in the infinite loop and sends them as one huge string when the loop is exited:
Consider the following implementation:
import random
from datetime import datetime
import time
import socket
import sys
HOST = "xx.xx.xx.xx"
PORT = 12345
srv = (HOST, PORT)
cls = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
cls.connect(srv)
# DATA:
# Variable to change status text
dec = 0
while 1:
data = "foo\\nbar"
size = cls.send(data.encode())
print(size)
time.sleep(1)
Then the string is built up in python until I quit the loop manually to: foo\\nbarfoo\\nbarfoo\\nbarfoo\\nbarfoo\\nbarfoo\\nbarfoo\\nbar and sent to the socket.
What we want to accomplish is, that those strings are sent after each other like:
foo\nbar
foo\nbar
foo\nbar
Is there any possibility to send those special characters with the string together?

Download several files from a local server to a client

The following codes let me download from server to client three files called tmp.bsp, tmp.seq and tmp.dms. However, just the first file tmp.dms is completely downloaded. The other one tmp.seq is filled up with the informations of tmp.bsp and tmp.bsp stay 0KB.
client:
import socket
import socket
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
sData = "Temp"
sData2 = "Temp"
sData3 = "Temp"
while True:
sData = skClient.recv(1024)
fDownloadFile = open("tmp.dms","wb")
sData2 = skClient.recv(1024)
fDownloadFile2 = open("tmp.seq","wb")
sData3 = skClient.recv(1024)
fDownloadFile3 = open("tmp.bsp","wb")
while sData:
fDownloadFile.write(sData)
sData = skClient.recv(1024)
fDownloadFile.close()
fDownloadFile2.write(sData2)
sData2 = skClient.recv(1024)
fDownloadFile2.close()
fDownloadFile3.write(sData3)
sData3 = skClient.recv(1024)
fDownloadFile3.close()
print "Download over"
break
skClient.close()
n is a counter and the prints are for debugging.
sFileName is to download one file, and used to work but since I want three files I just commented it.
server:
import socket
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
while True:
Content,Address = skServer.accept()
print Address
files = "C:\Users\Name_user\Desktop\Networking\Send_Receive/"
fUploadFile = open(files+str('tmp.dms'),"rb")
sRead = fUploadFile.read(1024)
fUploadFile2 = open(files+str('tmp.seq'),"rb")
sRead2 = fUploadFile2.read(1024)
fUploadFile3 = open(files+str('tmp.bsp'),"rb")
sRead3 = fUploadFile3.read(1024)
while sRead:
Content.send(sRead)
sRead = fUploadFile.read(1024)
Content.send(sRead2)
sRead2 = fUploadFile2.read(1024)
# Content.send(sRead3)
# sRead3 = fUploadFile3.read(1024)
Content.close()
print "Sending is over"
break
skServer.close()
files I'm using:
server2.py is my server
Execution

The main issue with your code is that you're sending / receiving an arbitrary number of data. If your buffer (1024) is smaller than the file size then the client's file will contain less information, and if it's larger the file may contain more information (data from the next file).
You could solve this issue by sending a value that signifies the end of a file. The problem with this method is that this value can't be contained in any file, and the client must be scanning the received data for this value.
Another possible solution is to calculate the file size and send that infomation in front of the file data. This way the cilent will know how many data to expect for each file.
Using struct.pack we can create a minimal four bytes header with the file size.
def send_file(soc, path):
with open(path, 'rb') as f:
data = f.read()
size = struct.pack('!I', len(data))
soc.send(size + data)
Tthen the client can get the file size by reading four bytes and unpacking to int.
def recv_file(soc, path):
size_header = soc.recv(4)
size = struct.unpack('!I', size_header)[0]
data = soc.recv(size)
with open(path, 'wb') as f:
f.write(data)
Note that sending/receiving files with one call may raise a socket error if the file size is larger than the socket buffer. In that case you'll have to read the data in smaller chunks in a loop, or increase the buffer size with socket.setsockopt.
Here is a modified version of the above functions that can handle large files:
import struct
import os.path
def send_file(soc, path):
file_size = os.path.getsize(path)
size_header = struct.pack('!Q', file_size)
soc.send(size_header)
with open(path, 'rb') as f:
while True:
data = f.read(1024)
if not data:
break
soc.send(data)
def recv_file(soc, path):
size_header = soc.recv(8)
file_size = struct.unpack('!Q', size_header)[0]
chunks = [1024 for i in range(file_size / 1024)]
with open(path, 'wb') as f:
for chunk in chunks:
f.write(soc.recv(chunk))
f.write(soc.recv(file_size % 1024))
I haven't tested this code thoroughly, but it should work for files of any size.
An example using the send_file function in your server:
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
Content,Address = skServer.accept()
print Address
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
send_file(Content, file)
Content.close()
print "Sending is over"
skServer.close()
Using recv_file in the client:
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
recv_file(skClient, file)
print "Download over"
skClient.close()

Yes you are right, I did run your program and found exactly same issue. I dont have enough time to work more on this issue but I found few key points which might lead you to the right work around.
https://docs.python.org/2/howto/sockets.html
The above official doc says:
When a recv returns 0 bytes, it means the other side has closed (or is in the process of closing) the connection. You will not receive any more data on this connection. Ever. You may be able to send data successfully
This is what it is happening when the third file returns 0 bytes.
But why 2nd and 3rd file is merged, I guess its because sockets are just buffered files and we might need to try making sure buffer is clear before sending another.
Read this,
Now there are two sets of verbs to use for communication. You can use send and recv, or you can transform your client socket into a file-like beast and use read and write. The latter is the way Java presents its sockets. I’m not going to talk about it here, except to warn you that you need to use flush on sockets. These are buffered “files”, and a common mistake is to write something, and then read for a reply. Without a flush in there, you may wait forever for the reply, because the request may still be in your output buffer.
But if you plan to reuse your socket for further transfers, you need to realize that there is no EOT on a socket. I repeat: if a socket send or recv returns after handling 0 bytes, the connection has been broken. If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there’s nothing more to read (for now). Now if you think about that a bit, you’ll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection. The choice is entirely yours, (but some ways are righter than others).
Hope this helps.

I'm not totally fluent in Python, but I think your while statement should be something like:
while: sData or sData2 or sData3
I may have the syntax wrong, but currently it looks like you will stop when "sData" is done and stop downloading sData2 and aData3 at that time even if they haven't finished.
Hmm--either that or the "While" isn't looping at all and it's just being used as an "if"? hard to tell without knowing the API.

Python client socket hangs

I have found one benign yet persistent problem with my socket client, it seems to hang and never receive a response from the server. Put differently, there is an indefinite wait for any response.
This only happens for cases where the server socket, has run say a minute or so method/function whose result is supposed to be returned to the client.
Put simply if you try to send plain text back and forth, this works fine, however making calls to a function that may need the client end to wait bit for a response cause it to hang.
Any help will be appreciated.
Below is some code...thanks to #Robᵩ for providing an earlier solution that worked with text like input and response
client
import socket
s = socket.socket()
s.connect(("localhost",3000))
f=open ("tranmit.jpg", "rb")
l = f.read(1024)
while (l):
s.send(l)
l = f.read(1024)
s.shutdown(socket.SHUT_WR)
reply =s.recv(1024) # Feedback: would like the receive feedback from the server.
print reply
s.close()
server
import socket
import sys
s = socket.socket()
s.bind(("localhost",3000))
s.listen(10)
i=1
# def somefunction(x):
# ''' does some computationally semi intensive work, say last about 120s and returns a result for the client'''
while True:
sc, address = s.accept()
print address
f = open('transmit.jpg','wb') #open in binary
l = 1
while(l):
l = sc.recv(1024)
while (l):
f.write(l)
l = sc.recv(1024)
f.close()
result =somefunction('transmit.jpg') # does something with the image from the client
sc.send(str(result)) # Would like to send back a response
sc.close()
s.close()

My chat client freezes up after beginning threads

I made a better chat client following help from people:
They told me that if I didn't want to be blocked on .recv when waiting for messages, I would need to use threads, classes, functions, and queues to do so.
So I followed some help a specific person gave me where I created a thread from a class and then defined a function that was supposed to read incoming messages and print them.
I also created a function that allows you to enter stuff to be sent off.
Thing is, when I run the program. Nothing happens.
Can somebody help point out what is wrong? (I've asked questions and researched for 3 days, without getting anywhere, so I did try)
from socket import *
import threading
import json
import select
print("Client Version 3")
HOST = input("Connect to: ")
PORT = int(input("On port: "))
# Create Socket
s = socket(AF_INET,SOCK_STREAM)
s.connect((HOST,PORT))
print("Connected to: ",HOST,)
#-------------------Need 2 threads for handling incoming and outgoing messages--
# 1: Create out_buffer:
Buffer = []
rlist,wlist,xlist = select.select([s],Buffer,[])
class Incoming(threading.Thread):
# made a function a thread
def Incoming_messages():
while True:
for i in rlist:
data = i.recv(1024)
if data:
print(data.decode())
# Now for outgoing data.
def Outgoing():
while True:
user_input=("Your message: ")
if user_input is True:
Buffer += [user_input.encode()]
for i in wlist:
s.sendall(Buffer)
Buffer = []
Thanks for taking a look, thanks also to Tony The Lion for suggesting this

Take a look at this revised version of your code: (in python3.3)
from socket import *
import threading
import json
import select
print("client")
HOST = input("connect to: ")
PORT = int(input("on port: "))
# create the socket
s = socket(AF_INET, SOCK_STREAM)
s.connect((HOST, PORT))
print("connected to:", HOST)
#------------------- need 2 threads for handling incoming and outgoing messages--
# 1: create out_buffer:
out_buffer = []
# for incoming data
def incoming():
rlist,wlist,xlist = select.select([s], out_buffer, [])
while 1:
for i in rlist:
data = i.recv(1024)
if data:
print("\nreceived:", data.decode())
# now for outgoing data
def outgoing():
global out_buffer
while 1:
user_input=input("your message: ")+"\n"
if user_input:
out_buffer += [user_input.encode()]
# for i in wlist:
s.send(out_buffer[0])
out_buffer = []
thread_in = threading.Thread(target=incoming, args=())
thread_out = threading.Thread(target=outgoing, args=())
thread_in.start() # this causes the thread to run
thread_out.start()
thread_in.join() # this waits until the thread has completed
thread_out.join()
in your program you had various problems, namely you need to call the threads; to just define them isn't enough.
you also had forgot the function input() in the line: user_input=input("your message: ")+"\n".
the "select()" function was blocking until you had something to read, so the program didn't arrive to the next sections of the code, so it's better to move it to the reading thread.
the send function in python doesn't accept a list; in python 3.3 it accepts a group of bytes, as returned by the encoded() function, so that part of the code had to be adapted.