In the below code I am trying to receive a file from Python Sockets and write it to a local file
I have following code
chunk=clientDtSocket.recv(1024)
while chunk:
print("In Chunk"+str(chunk))
incomingFile.write(chunk)
chunk=clientDtSocket.recv(1024)
I get following
In Chunkb'Sohail Khan'
But the file size remains same.
Also how can I count the no of bytes I have recieved.
Make sure the file is closed after the loop.
You can check the received bytes count using len function:
chunk = clientDtSocket.recv(1024)
while chunk:
print("received {} bytes".format(len(chunk))) # <-----
print("In Chunk " + str(chunk))
incomingFile.write(chunk)
chunk = clientDtSocket.recv(1024)
incomingFile.close() # <----
Instead of manually closing the file, consider using with statement:
with open('/path/to/localfile', 'wb') as incomingFile:
....
When you open file set write buffer size to 0:
bufsize = 0
incomingFile = open('....', 'w', bufsize)
It is normal behevior, that data are saved to the file not immidiately after calling write function, but afted write buffer will be totaly filled. But if you setup buffer size to 0 as in axample above, yoor data will be written immidiatel. Writting data from write buffer to file often callaed "flushing"
Flushing also occured when you close file:
incomingFile.close()
Related
Given some file that contain some text. How can I read Y bytes from this file after X bytes and print them?
I thought to use these functions: file = open("my_file", 'rb') and file.read(..) but I don't sure how to do it with these functions.
You almost have it, you are just missing seek to select the position to read from:
file = open("my_file", 'rb')
file.seek(X)
content = file.read(Y)
file.close()
print(content)
However, if an error happened, your file would be left open for longer than necessary, so almost always you should use the with syntax instead, which will automatically dispose of file at the end of the block:
with open("my_file", 'rb') as file:
file.seek(X)
content = file.read(Y)
print(content)
Note that content will be bytes, not text.
I am new to python,actually I tried a simple program. I am reading a text file which is updating data every time from logs so I need to copy or append that updated data only from that text file to another text file and do some process.For this I am done with below code.
with open("Log.txt") as f:
data = f.readlines()
i=len(data)#find length of file
def check(i):
with open("Log.txt") as f2:
count=f2.readlines()
x=len(count)#finds length of file after update
print(x)
j=i
i=x
while(j<x):
with open("Log.txt") as f1:
count=f1.readlines()
msg=count[j]#get updated text
j=j+1
with open('Output1.txt', 'a') as f3:
f3.write(str(msg))#append updated text to file
process()#calling a function which do some process on text file
while True:
check(i)
By using above code I am getting updated data but the problem is its getting slow down while doing infinite loops.Means Initially If I have a data in Log.text file upto 12:00pm it will append all data,After looping for 5 min the data in log.text file will increase but after same 5 min time I will get only 3 min data into output file. it was slow there is so much delay in getting data from text file.why??how can I get same updated text into output file instantly.
Try following code:
def read_and_copy():
with open("/var/log/messages") as input:
with open('/tmp/output', 'a+') as output:
while True:
# read line by line
data = input.readline()
# check if line is not empty
# also if needed - provide necessary checks
if data:
# write to another file
output.write(data)
# Flush the write buffers of the stream if applicable.
# This does nothing for read-only and non-blocking streams.
output.flush()
read_and_copy()
Keep in mind, that each call of read_and_copy will read entire file again and overwrite output file.
I think what you should use is follow, which follows a file. Whenever new lines are added to Log.txt, the below code updates it to Output1.txt.
Try this
import os, time
logfile = "Log.txt"
thefile = open(logfile)
def follow(thefile):
thefile.seek(0,2) # Go to the end of the file
while True:
line = thefile.readline()
if not line:
time.sleep(1) # Sleep briefly
else:
with open('Output1.txt', 'a') as f3:
f3.write(line)
loglines = follow(thefile)
I am using the following code for downloading a csv file from a remote http server:
with contextlib.closing(requests.get("http://some_url/file.csv", stream= True)) as response, open('/tmp/archivo.csv', 'wb') as output_file:
for chunk in response.iter_lines(chunk_size=1024*8):
if chunk:
print(chunk)
output_file.write(chunk)
Everything seems to work, the output on the screen is:
b'Code,Description'
b'"1","Some"'
b'"2","Few"'
b'"3","human"'
....
But the file is:
$ cat /tmp/archivo.csv
Code,Description"1","Some""2","Few""3","human"...
I think that the issue is related to the new line encoding between Windows, GNU/Linux and Mac, but I don't know how to specify that in the code just showed.
The problem is not related to opening the file in 'wb' mode, is I made the changes needed
with contextlib.closing(requests.get("http://some_url/file.csv", stream= True)) as response, open('/tmp/archivo.csv', 'w') as output_file:
for chunk in response.iter_lines(chunk_size=1024*8):
if chunk:
print(chunk.decode('utf-8'))
output_file.write(chunk.decode('utf-8'))
The output on the screen changes,
'Code,Description'
'"1","Some"'
'"2","Few"'
'"3","human"'
...
but the file doesn't
Any ideas?
You are seeing newlines because print() automatically terminates every output with a newline. Compare your output to
print(chunk.decode('utf-8'), end='')
which specifically does not terminate with newlines.
To correctly insert newlines into output_file you can use
output_file.write(chunk.decode('utf-8') + '\n')
or iterate over chunks instead of lines using
for chunk in response.iter_content(chunk_size=1024*8):
if chunk:
output_file.write(chunk)
I found this code here which monitors the progress of downloads. -
import urllib2
url = "http://download.thinkbroadband.com/10MB.zip"
file_name = url.split('/')[-1]
u = urllib2.urlopen(url)
f = open(file_name, 'wb')
meta = u.info()
file_size = int(meta.getheaders("Content-Length")[0])
print "Downloading: %s Bytes: %s" % (file_name, file_size)
file_size_dl = 0
block_sz = 8192
while True:
buffer = u.read(block_sz)
if not buffer:
break
file_size_dl += len(buffer)
f.write(buffer)
status = r"%10d [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
status = status + chr(8)*(len(status)+1)
print status,
f.close()
I do not see the block size being modified at any point in the while loop, so, to me, buffer = u.read(block_sz) should keep reading the same thing over and over again.
We know this doesn't happen, is it because read() in while loop has a built in pointer that starts reading from where you left off last time?
What about write()? Does it keep appending after where it last left off, even though the file is not opened in append mode?
File objects and network sockets and other forms of I/O communication are data streams. Reading from them always produces the next section of data, calling .read() with a buffer size does not re-start the stream from the beginning.
So yes, there is a virtual 'position' in streams where you are reading from.
For files, that position is called the file pointer, and this pointer advances both for reading and writing. You can alter the position by using seeking, or by simply re-opening the file. You can ask a file object to tell you the current position, too.
For network sockets however, you can only go forward; the data is received from outside your computer and reading consumes that data.
I have a text file with data such as
b'\x00\x09\x00\xfe'
This was piped into a text file from a TCP socket stream. Call this text file 'stream.txt'. I opened this file with the following code:
f = open("stream.txt", "rb")
bytes_read = f.read()
When I open this file within another Python script, I get a '\' for each '\' in the original file. On top of this, I cannot access the bytes array as such, since it appears to have become a string. That is, 'bytes_read' is now
'b"\\x00\\x09\\x00\\xfe"'
How can I recover this string as a bytes array?
The client code I used to capture this data is the following script:
from socket import *
clientsock = socket(AF_INET, SOCK_STREAM)
clientsock.connect(('1.2.3.4', 2000)) # Open the TCP socket
clientsock.sendall(b'myCommand') # Send a command to the server
data = clientsock.recv(16) # Wait for the response
print(data) # For piping to 'stream.txt'
clientsock.close()
As the data was printed to the terminal, I redirected it to a file:
$ python3 client.py > stream.txt
My goal is to bypass the redirect into text file and pipe directly into a plotter... But first I wanted to get this to work.
Was able to solve this by writing directly to a file. So rather than using 'print(data)', and redirecting to a file, I tried this:
file = open("rawData", "wb")
...
file.write(data)
...
file.close()
Was able to process "rawData" as expected.