I have to write 7231 bytes into a file using python script. In a client-server program, my python script act like client and it received 7231 bytes from server. If I check in TCP-Dump, its shows complete data. But when I try to write into a file; I am missing the content.
My script:
def SendOnce(self, req='/gpsData=1',method="GET"):
conn = httplib.HTTPConnection(self.proxy)
self.Logresponse("\nConnection Open\n<br />")
conn.request(method,req)
Log="\nRequest Send: %s\n<br \>\n" %req
self.Logresponse(Log)
response = conn.getresponse()
Log = "\nResponse Code: %s\n<br \>\n" %response.status
self.Logresponse(Log)
Log = "\nSarav -- Get Header: %s \n version= %s <br \>\n" %(response.msg,response.version)
self.Logresponse(Log)
if (response.status==200):
Log = response.read()
self.Logresponse(Log)
conn.close()
self.Logresponse("\nConnection Close\n<br \>")
return response
this "self.Logresponse(Log)" is writing into file. If i receive 1023 bytes, its writing full content into that. Please help me out how to write complete data.
Note: I am writing Hexa Format data.
First of all, 7231 bytes is not exactly huge...
With the limited info you gave, I would guess that you might have forgotten to take the OS's write buffer into account. You probably try to read the file before all the content was written to it.
Python generally uses the system's standard buffer (you can change that). You can decrease that buffer, or force a flush yourself.
I'm just guessing, it might be that the .read() function doesn't return all data in one chunk; can you try to modify the inner part like this:
if (response.status==200):
while 1:
Log = response.read()
if not Log:
break
self.Logresponse(Log)
Related
I am trying to read some data streams using protobuf in python, and i want to use trio to make the client for reading the streams. The protobuf has some method calls, and i find they do not work when i use trio streams.
Python client on a linux machine.
import DTCProtocol_pb2 as Dtc
async def parent(addr, encoding, heartbeat_interval):
print(f"parent: connecting to 127.0.0.1:{addr[1]}")
client_stream = await trio.open_tcp_stream(addr[0], addr[1])
# encoding request
print("parent: spawing encoding request ...")
enc_req = create_enc_req(encoding) # construct encoding request
await send_message(enc_req, Dtc.ENCODING_REQUEST,client_stream, 'encoding request') # send encoding request
log.debug('get_reponse: started')
response = await client_stream.receive_some(1024)
m_size = struct.unpack_from('<H', response[:2]) # the size of message
m_type = struct.unpack_from('<H', response[2:4]) # the type of the message
m_body = response[4:]
m_resp = Dtc.EncodingResponse()
m_body would be some bytes data, which I dont know how to decode. Dtc.EncodingResponse() is the protobuf method which would give a Dtc object which contains the response in a readable format. (Dtc is the protobuf file). But I get nothing here. When I did this script without trio, Dtc.EncodingResponse() would give the full response in readable format.
I am guessing the problem is that the "client_stream" is a trio stream object that only reads bytes, and so I probably need to use a ReceiveChannel object instead. But if this is true, I dont know how to do this.
UPDATE:
The answer below by Nathaniel J. Smith solves my problem.
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_body)
I feel so silly, but I did not ParseFromString the data previously, and that was all it took. Extremely grateful to all who gave replies. Hope this helps someone out there.
Like #shmee said in the comment, I think your code got mangled some by the edits... you should double-check.
When I did this script without trio, Dtc.EncodingResponse() would give the full response in readable format
I think you might have dropped a line when switching to Trio? Dtc.EncodingResponse() just creates a new empty EncodingResponse object. If you want to parse the data from m_body into your new object, you have to do that explicitly, with something like:
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_body)
However, there's another problem... the reason it's called receive_some is that it receives some bytes, but might not receive all the bytes you asked for. Your code is assuming that a single call to receive_some will fetch all the bytes in the response, and that might be true when you're doing simple test, but in general it's not guaranteed. If you don't get enough data on the first call to receive_some, you might need to keep calling it repeatedly until you get all the data.
This is actually very standard... sockets work the same way. That's why the first thing your server is sending an m_size field at the beginning – it's so you can tell whether you've gotten all the data or not!
Unfortunately, as of June 2019, Trio doesn't provide a helper to do this loop for you – you can track progress on that in this issue. In the mean time, it's possible to write your own. I think something like this should work:
async def receive_exactly(stream, count):
buf = bytearray()
while len(buf) < count:
new_data = await stream.receive_some(count - len(buf))
if not new_data:
raise RuntimeError("other side closed the connection unexpectedly")
buf += new data
return buf
async def receive_encoding_response(stream):
header = await receive_exactly(stream, 4)
(m_size, m_type) = struct.unpack('<HH', header)
m_body = await receive_exactly(stream, m_size)
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_size)
return m_resp
I have an Apache server with python cgi (Python3). A client start a get request to get a virtual file, and I need to give him back the good one regarding his user-agent. I was able to do it with text files but when I try to serve back binairies files like images (.jpg) or .zip, the downloaded file seems corrupted.
When I parse it I can see b'\x00\x....' so I think the byte conversion went wrong somewhere.
I have tried with sys.stdout.write but it expects a str not bytes. I have tried also to "play" with the headers by changing the content type for example but it is not working.
reqFile = open(filePath,'rb')
content = reqFile.read()
print("Content-Type:image/jpg")
print("Accept-Rangers:byte")
print("Content-Length:"+str(len(content))
print()
print (content)
Thanks in advance !!
Ok, I have found that print() insert '\n' character and other stuff. So, for binairies file, I recommend to use sys.stdout.
file = open(filePath, "rb")
content = file.read()
length = len(content)
file.close()
print("Content-type:application/x-download")
print("Content-length:%d" % length)
print()
sys.stdout.flush()
sys.stdout.buffer.write(content)
Don't forget to do sys.stdout.flush() in order to have a clean output.
I've got a client/server program where the client sends plaintext to the server which then runs AES encryption and returns the ciphertext. I'm using the following algorithm for the encryption/decryption:
http://anh.cs.luc.edu/331/code/aes.py
When I get the results back from the encryption and print them on the server-side I see mostly gibberish in the terminal. I can save to a file immediately and get something along these lines:
tgâY†Äô®Ø8ί6ƒlÑÝ%ŠIç°´>§À¥0Ð
I can see that this is the correct output because if I immediately decrypt it on the server, I get the original plaintext back. If I run this through the socket, send it back to the client, and print(), I get something more like this:
\rtg\xe2Y\x86\x8f\xc4\xf4\xae\xd88\xce\xaf6\x83l\xd1\xdd%\x8aI\xe7\xb0\xb4>\xa7\xc0\x18\xa50\xd0
There's an obvious difference here. I'm aware that the \x represents a hex value. If I save on the client-side, the resulting text file still contains all \x instances (i.e., it looks exactly like what I displayed directly above). What must I do to convert this into the same kind of output that I'm seeing in the first example? From what I have seen so far, it seems that this is unicode and I'm having trouble...
Relevant code from server.py
key = aes.generateRandomKey(keysizes[len(key)%3])
encryptedText = aes.encryptData(key, text)
f = open("serverTest.txt", "w")
f.write(encryptedText)
f.close()
print(encryptedText)
decryptedText = aes.decryptData(key, encryptedText)
print(decryptedText)
conn.sendall(encryptedText)
Relevant code from client.py
cipherText = repr(s.recv(16384))[1:-1]
s.close()
cipherFile = raw_input("Enter the filename to save the ciphertext: ")
print(cipherText)
f = open(cipherFile, "w")
f.write(cipherText)
Edit: To put this simply, I need to be able to send that data to the client and have it display in the same way as it shows up on the server. I feel like there's something I can do with decoding, but everything I've tried so far doesn't work. Ultimately, I'll have to send from the client back to the server, so I'm sure the fix here will also work for that, assuming I can read it from the file correctly.
Edit2: When sending normally (as in my code above) and then decoding on the client-side with "string-escape", I'm getting identical output to the terminal on both ends. The file output also appears to be the same. This issue is close to being resolved, assuming I can read this in and get the correct data sent back to the server for decrypting.
Not sure I fully understood what you're up to, but one difference between client and server is that on the client you're getting the repr for the byte string, while on the server you print the byte string directly.
(if I got the issue right) I'd suggest replacing
repr(s.recv(16384))[1:-1]
with a plain
s.recv(16384)
I want to read specific bytes from a remote file using a python module. I am using urllib2. Specific bytes in the sense bytes in the form of Offset,Size. I know we can read X number of bytes from a remote file using urlopen(link).read(X). Is there any way so that I can read data which starts from Offset of length Size.?
def readSpecificBytes(link,Offset,size):
# code to be written
This will work with many servers (Apache, etc.), but doesn't always work, esp. not with dynamic content like CGI (*.php, *.cgi, etc.):
import urllib2
def get_part_of_url(link, start_byte, end_byte):
req = urllib2.Request(link)
req.add_header('Range', 'bytes=' + str(start_byte) + '-' + str(end_byte))
resp = urllib2.urlopen(req)
content = resp.read()
Note that this approach means that the server never has to send and you never download the data you don't need/want, which could save tons of bandwidth if you only want a small amount of data from a large file.
When it doesn't work, just read the first set of bytes before the rest.
See Wikipedia Article on HTTP headers for more details.
Unfortunately the file-like object returned by urllib2.urlopen() doesn't actually have a seek() method. You will need to work around this by doing something like this:
def readSpecificBytes(link,Offset,size):
f = urllib2.urlopen(link)
if Offset > 0:
f.read(Offset)
return f.read(size)
If a would-be-HTTP-server written in Python2.6 has local access to a file, what would be the most correct way for that server to return the file to a client, on request?
Let's say this is the current situation:
header('Content-Type', file.mimetype)
header('Content-Length', file.size) # file size in bytes
header('Content-MD5', file.hash) # an md5 hash of the entire file
return open(file.path).read()
All the files are .zip or .rar archives no bigger than a couple of megabytes.
With the current situation, browsers handle the incoming download weirdly. No browser knows the file's name, for example, so they use a random or default one. (Firefox even saved the file with a .part extension, even though it was complete and completely usable.)
What would be the best way to fix this and other errors I may not even be aware of, yet?
What headers am I not sending?
Thanks!
This is how I send ZIP file,
req.send_response(200)
req.send_header('Content-Type', 'application/zip')
req.send_header('Content-Disposition', 'attachment;'
'filename=%s' % filename)
Most browsers handle it correctly.
If you don't have to return the response body (that is, if you are given a stream for the response body by your framework) you can avoid holding the file in memory with something like this:
fp = file(path_to_the_file, 'rb')
while True:
bytes = fp.read(8192)
if bytes:
response.write(bytes)
else:
return
What web framework are you using?