I'm trying to transfer files with python socket module, I can transfer and write the file but I can't stop the function when there are no more bytes to write
FORMAT="utf-8"
HEADER=512
CLIENT CODE
FORMAT="utf-8"
HEADER=512
def File(path):
name=os.path.basename(path)
client.send("file".encode(FORMAT))
client.send(name.encode(FORMAT))
print(client.recv(HEADER).decode(FORMAT))
f=open(path,"rb")
l=f.read(HEADER)
while (l):
client.send(l)
l=f.read(HEADER)
f.close()
print("Finish")
SERVER CODE
def Save(conn):
name=(conn.recv(HEADER).decode(FORMAT))
conn.send(f"Saving {name}".encode(FORMAT))
print(name)
Writing=True
with open(PATH+name,"wb") as f:
print("Writing file")
while Writing:
data=conn.recv(HEADER)
if not data:
Writing=False
f.write(data)
f.close()
print("File written")
conn.close()
To know when a socket has finished, take a look at How does the python socket.recv() method know that the end of the message has been reached?.
Anyway, receiving constant HEADER size is a bad practice, as the received amount can be smaller (due to network issues);
You'd better keep up with the actual amount received with respect to the desired amount, like in this example:
def receive(self, size):
from_client = b''
remaining_to_recive = size
while 0 < remaining_to_recive:
data = self.sock.recv(remaining_to_recive)
if not data: break
from_client += data
remaining_to_recive -= len(data)
if 0 < remaining_to_recive:
raise RuntimeError(Connection.NOT_ALL_DATA_RECEIVED_ERROR)
return from_client
Related
I a beginner in python and am having trouble trying to store an output of a PCAP file from wireshark into a text file. The code works, but it takes way too long to read the PCAP file as it is 300+ MB in size with 157167 lines of text.
Is there a way I can make it run faster? Here is the source code:
import pyshark
from datetime import datetime
def network_conversation(packet):
try:
protocol = packet.transport_layer
source_address = packet.ip.src
source_port = packet[packet.transport_layer].srcport
destination_address = packet.ip.dst
destination_port = packet[packet.transport_layer].dstport
packet_time = packet.sniff_time
packet_length = []
packet_length = int(packet.length)
return (f'{packet_time} {protocol} {source_address}:{source_port} --> {destination_address}:{destination_port} {packet_length}')
except AttributeError as e:
pass
capture = pyshark.FileCapture('YouTube.pcap')
date=datetime.now().strftime('%Y-%m-%d %H-%M-%S')
conversations = []
for packet in capture:
results = network_conversation(packet)
if results != None:
conversations.append(results)
# this sorts the conversations by protocol
# TCP and UDP
for item in sorted(conversations):
with open(f"capture{date}.txt", "a+") as f:
print(item, file=f)
Thanks
I have the following socket listening on my local network:
def recvall(sock):
BUFF_SIZE = 4096 # 4 KiB
fragments = []
while True:
chunk = sock.recv(BUFF_SIZE)
fragments.append(chunk)
# if the following line is removed, data is omitted
time.sleep(0.005)
if len(chunk) < BUFF_SIZE:
break
data = b''.join(fragments)
return data
def main():
pcd = o3d.geometry.PointCloud()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('192.168.0.22', 2525))
print("starting listening...")
s.listen(1)
counter = 0
while True:
clientsocket, address = s.accept()
print(f"Connection from {address} has been established!")
received_data = recvall(clientsocket)
clientsocket.send(bytes(f"response nr {counter}!", "utf-8"))
counter += 1
print(len(received_data))
if __name__ == "__main__":
main()
To this port, I'm sending byte data with a length of 172800 bytes from an app on my mobile phone.
As one can see, I'm printing the amount of data received. The amount is only correct, if I use time.sleep() as shown in the code above. If I don't use this method, only a part of the data is received.
Obviously this is some timing issue, the question is: How can I be sure to receive all the data all the time without using time.sleep() (since this is also not 100% certain to work, depending on the sleeping time set)
sock.recv() returns the data that is available. The relevant piece from the man page of recv(2) is:
The receive calls normally return any data available, up to the requested amount,
rather than waiting for receipt of the full amount requested.
In your case, time.sleep(0.005) seems to allow for all the remaining data of the message to arrive and be stored in the buffer.
There are some options to eliminate the need for time.sleep(0.005). Which one is the most appropriate depends on your needs.
If the sender sends data, but does not expect a response, you can have the sender close the socket after it sends the data, i.e., sock.close() after sock.sendall(). recv() will return an empty string that can be used to break out of the while loop on the receiver.
def recvall(sock):
BUFF_SIZE = 4096
fragments = []
while True:
chunk = sock.recv(BUFF_SIZE)
if not chunk:
break
fragments.append(chunk)
return b''.join(fragments)
If the sender sends messages of fixed length, e.g., 172800 bytes, you can use recv() in a loop until the receiver receives an entire message.
def recvall(sock, length=172800):
fragments = []
while length:
chunk = sock.recv(length)
if not chunk:
raise EOFError('socket closed')
length -= len(chunk)
fragments.append(chunk)
return b''.join(fragments)
Other options include a. adding a delimiter, e.g., a special character that cannot be part of the data, at the end of the messages that the sender sends; the receiver can then run recv() in a loop until it detects the delimiter and b. prefixing the messages on the sender with their length; the receiver will then know how many bytes to expect for each message.
I am connecting to a pre-configured server that serves four different file formats with different sizes. Each file is appended with the file size...
Example: lighthouse.jpg
561276ÿØÿà JFIF ` ` ÿî Adobe
The "561276" is the file size and needs to be pruned before saving the file.
Example: randomText.txt
45711111111111111111111111111111111111111111111111111111111
222222222222222222222222222222222222222222222222222222222
33333333333333333333333333333333333333333333333333
44444444444444444444444444444444444444444444444444444444
66666666666666666666
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
cvccccccccccccccccccccccccccccccccccccccccccccccc
ddddddddddddddddddddddddddddddddddddddddddddddddddd
The "457" is the file size and needs to be pruned before saving the file.
Some files have a size that is only 3-digits long. Some have a file size that is 6-digits long (as seen here). I want to make my code size-agnostic; regardless of how many digits is in the size.
I've tried using:
while len(buf) < 4:
buf += sock.recv(4 - len(buf))
size = struct.unpack('!i', buf)
but this only prunes the first four digits.
AND
I've tried using
len = sock.recv(4)
data = sock.recv(len)
but once again... only prunes the first four digits
Here is what I have so far:
def get_size():
buf = ''
while len(buf) < 4:
buf += sock.recv(4 - len(buf))
size = struct.unpack('!i', buf)
print "[*] Receiving %s bytes" % size
def download_pic():
size = get_size()
fname = 'tst.jpg'
with open(fname, 'wb') as img:
while True:
data = sock.recv(1024)
if not data:
break
img.write(data)
print '[*] {0} received!'.format(fname)
def main():
doconnectionstuffandprinttoconsole() #establishes connection
answer = input("[>] Your Selection: ")
sock.send(str(answer))
if answer == 2:
download_pic()
sock.close()
Any help in pruning the size from the file(s) would be greatly appreciated!
Jason Harper's suggestion (#jasonharper) got me thinking. When I ran repr(data) on the chunks from randomText.txt, I saw that it had a break in it that looked like...
'457''1111111111111111111111...
The server was attempting to send two different chunks (one at at time) but it kept getting merged into one chunk. So, I increased my sock.recv(64) up to sock.recv(256). And for some reason, it send two chunks!
'457'
'111111111...' [truncated]
NEW AND IMPROVED CODE!
import socket
import sys
import struct
import os
user1 = {'user1': 91827364}
user2 = {'user2': 19283746}
user3 = {'user3': 46372819}
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('127.0.0.1', 2058))
def print_data():
data_rcv = sock.recv(1024)
print "[-] {0}".format(data_rcv)
def download_file(format):
fname = 'download'
fullname = fname + '.' + format
try:
with open(fullname, 'wb') as txt:
len = sock.recv(256)
while True:
data = sock.recv(int(len))
if not data:
break
txt.write(data)
print("[*] {0} successfully downloaded with a length of {1} characters!".format(fullname, len))
except Exception:
print("[!] Error receiving file. Please try again.")
def connect():
print("[*] Sending Length")
sock.send("5")
my_struct = struct.pack('5s i', 'user1', 91827364)
print("[*] Sending User1 Struct")
sock.send(my_struct)
print_data()
def main():
print_data()
connect()
print_data()
answer = input("[>] Your Selection: ")
sock.send(str(answer))
if answer == 2: # Option to download Lighthouse.jpg
download_file("jpg")
elif answer == 4: # Option to download randomText.txt
download_file("txt")
sock.close()
if __name__ == "__main__":
main()
MY OUTPUT
[-] Please enter credentials
[*] Sending Length
[*] Sending User1 Struct
[-] Authenticated
[-] Choose a file to retrieve from the following list (enter the number):
1. photo.png
2. Lighthouse.jpg
3. npp.6.8.5.Installer.exe
4. randomText.txt
[>] Your Selection: 2
[*] download.jpg successfully downloaded with a length of 561276 characters!
I'm recently studiying sockets trying to make them work inside a Python script (Python3), inside Windows.
Here the Python script of the server side.
import socket
import time
MSGLEN = 2048
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('localhost', 8000))
server.listen(1)
while 1:
#accept connections from outside
(clientsocket, address) = server.accept()
chunks = []
bytes_recd = 0
while bytes_recd < MSGLEN:
chunk = clientsocket.recv(min(MSGLEN - bytes_recd, 2048)) #should enough this row without checks if transmission guaranteed inside buffer dimension
#print(chunk)
#i=0
chunk = chunk.decode()
bytes_recd = bytes_recd + len(chunk)
chunks.append(chunk)
for i in range(bytes_recd):
if(chunk[i] == "_"):
print("Breaking(_Bad?)")
break
buff_str = chunk[:i]
print(buff_str)
if chunk == '':
print("Server notification: connection broken")
break
mex = ''.join(chunks)
print("Server notification: \n\tIncoming data: " + mex)
i=1;
while i==1:
chunk = clientsocket.recv(128)
chunk = chunk.decode()
if chunk == '':
i = 0
totalsent = 0
msg = "Server notification: data received"
while totalsent < MSGLEN:
sent = clientsocket.send(bytes(msg[totalsent:], 'UTF-8'))
if sent == 0 :
print ("Server notification: end transmitting")
break
totalsent = totalsent + sent
I'm checking when a "_" is received and make some decision in it. This because I'm using blocking sockets. You should forget the very last part and the whole program functionality since I'm working on it and the incriminated part is here:
for i in range(bytes_recd):
if(chunk[i] == "_"):
print("Breaking(_Bad?)")
break
buff_str = chunk[:i]
Something weird happens: the check works fine and break the loop by printing the rest at the right index value. BUT! This wild and apparently non-sense error appears:
>>>
Breaking(_Bad?), i: 2
13
Traceback (most recent call last):
File "C:\Users\TheXeno\Dropbox\Firmwares\Altri\server.py", line 24, in <module>
if(chunk[i] == "_"):
IndexError: string index out of range
As you can see from the console output, it finds the number before the "_", in this case is the string "13" and is located at i = 2, which is compliant with the receiving string format form the socket: "charNumber_String". But seems to keep counting until exit from bounds.
EDIT: I will not rename the variables, but next time, better use improved names, and not "chunk" and "chunks".
Let's look at this block of code:
while bytes_recd < MSGLEN:
chunk = clientsocket.recv(min(MSGLEN - bytes_recd, 2048))
chunk = chunk.decode()
bytes_recd = bytes_recd + len(chunk)
chunks.append(chunk)
for i in range(bytes_recd):
if(chunk[i] == "_"):
print("Breaking(_Bad?)")
break
Lets say you read 100 bytes, and lets assume that the decoded chunk is the same length as the encoded chunk. bytes_recd will be 100, and your for loop goes from zero to 99, and all is well.
Now you read another 100 bytes. chunk is again 100 bytes long, and chunks (with an "s") is 200 bytes. bytes_recd is now 200. Your for loop now goes from 0 to 199, and you're checking chunk[i]. chunk is only 100 bytes long, so when i gets past 99, you get the error.
Maybe you meant to compare chunks[i] (with an "s")?
try:
for i, chunk in enumerate(chunks):
if(chunk == "_"):
print("Breaking(_Bad?)")
break
This way you never go out of bounds. So one error less :)
I have a bunch of code that looks like this:
with tempfile.NamedTemporaryFile() as tmpfile:
tmpfile.write(fileobj.read()) # fileobj is some file-like object
tmpfile.flush()
try:
self.sftp.put(tmpfile.name, path)
except IOError:
# error handling removed for ease of reading
pass
Is it possible to do an upload like this without having to write the file out somewhere?
Update As of Paramiko 1.10, you can use putfo:
self.sftp.putfo(fileobj, path)
Instead of using paramiko.SFTPClient.put, you can use paramiko.SFTPClient.open, which opens a file-like object. You can write to that. Something like this:
f = self.sftp.open(path, 'wb')
f.write(fileobj.read())
f.close()
Note that it may be worthwhile to feed paramiko data in 32 KiB chunks, since that's the largest chunk underlying SSH protocol can handle without breaking it into multiple packets.
Is StringIO what you're looking for? (doc page)
SFTPClient's get() and put() functions take paths and not file-handles, which makes things a bit awkward.
You could write a wrapper for paramiko.SFTPClient to give it the functionality that you want.
Here's my best untested attempt:
from paramiko import SFTPClient
class SFTPClient2(SFTPClient):
def put(self, local_file, remotepath, callback=None, confirm=True):
fl = source_file
file_size = os.fstat(fl.fileno()).st_size
try:
fr = self.file(remotepath, 'wb')
fr.set_pipelined(True)
size = 0
try:
while True:
data = fl.read(32768)
if len(data) == 0:
break
fr.write(data)
size += len(data)
if callback is not None:
callback(size, file_size)
finally:
fr.close()
finally:
fl.close()
if confirm:
s = self.stat(remotepath)
if s.st_size != size:
raise IOError('size mismatch in put! %d != %d' % (s.st_size, size))
else:
s = SFTPAttributes()
return s
def get(self, remotepath, local_file, callback=None):
fr = self.file(remotepath, 'rb')
file_size = self.stat(remotepath).st_size
fr.prefetch()
try:
fl = local_file
try:
size = 0
while True:
data = fr.read(32768)
if len(data) == 0:
break
fl.write(data)
size += len(data)
if callback is not None:
callback(size, file_size)
finally:
fl.close()
finally:
fr.close()
s = os.fstat(fl.fileno())
if s.st_size != size:
raise IOError('size mismatch in get! %d != %d' % (s.st_size, size))
If it works, the get and put functions should now take local file-handles rather than paths.
All I had to do was get rid of the code that opens the file from the path, and change the code that gets the size of the file to use os.fstat instead of os.stat.