I am trying to send multiple strings using the socket.send() and socket.recv() function.
I am building a client/server, and "in a perfect world" would like my server to call the socket.send() function a maximum of 3 times (for a certain server command) while having my client call the socket.recv() call 3 times. This doesn't seem to work the client gets hung up waiting for another response.
server:
clientsocket.send(dir)
clientsocket.send(dirS)
clientsocket.send(fileS)
client
response1 = s.recv(1024)
if response1:
print "\nRecieved response: \n" , response1
response2 = s.recv(1024)
if response2:
print "\nRecieved response: \n" , response2
response3 = s.recv(1024)
if response3:
print "\nRecieved response: \n" , response3
I was going through the tedious task of joining all my strings together then reparsing them in the client, and was wondering if there was a more efficient way of doing it?
edit:
My output of response1 gives me unusual results. The first time I print response1, it prints all 3 of the responses in 1 string (all mashed together). The second time I call it, it gives me the first string by itself. The following calls to recv are now glitched/bugged and display the 2nd string, then the third string. It then starts to display the other commands but as if it was behind and in a queue.
Very unusual, but I will likely stick to joining the strings together in the server then parsing them in the client
You wouldn't send bytes/strings over a socket like that in a real-world app.
You would create a messaging protocol on-top of the socket, then you would put your bytes/strings in messages and send messages over the socket.
You probably wouldn't create the messaging protocol from scratch either. You'd use a library like nanomsg or zeromq.
server
from nanomsg import Socket, PAIR
sock = Socket(PAIR)
sock.bind('inproc://bob')
sock.send(dir)
sock.send(dirS)
sock.send(fileS)
client
from nanomsg import Socket, PAIR
sock = Socket(PAIR)
sock.bind('inproc://bob')
response1 = sock.recv()
response2 = sock.recv()
response3 = sock.recv()
In nanomsg, recv() will return exactly what was sent by send() so there is a one-to-one mapping between send() and recv(). This is not the case when using lower-level Python sockets where you may need to call recv() multiple times to get everything that was sent with send().
TCP is a streaming protocol and there are no message boundaries. Whether a blob of data was sent with one or a hundred send calls is unknown to the receiver. You certainly can't assume that 3 sends can be matched with 3 recvs. So, you are left with the tedious job of reassembling fragments at the receiver.
One option is to layer a messaging protocol on top of the pure TCP stream. This is what zeromq does, and it may be an option for reducing the tedium.
The answer to this has been covered elsewhere.
There are two solutions to your problem.
Solution 1:
Mark the end of your strings. send(escape(dir) + MARKER) Your client then keeps calling recv() until it gets the end-of-message marker. If recv() returns multiple strings, you can use the marker to know where they start and end. You need to escape the marker if your strings contain it. Remember to escape on the client too.
Solution 2:
Send the length of your strings before you send the actual string. Your client then keeps calling recv() until its read all the bytes. If recv() returns multiple strings. You know where they start and end since you know how long they are. When sending the length of your string, make you you use a fixed number of bytes so you can distinguish the string lenght from the string in the byte stream. You will find struct module useful.
Related
I want to Hash a CAN message received from a vehicle. The following code written in Python is used to receive the CAN message(dev.recv()) from the vehicle and print the received message (dev.send()).I want to hash the CAN message present in dev.recv()function before sending the message using dev.send().Is this possible? If so how can it be done?
from canard.hw import socketcan
dev = socketcan.SocketCanDev(’can0’)
dev.start()
while True:
f = dev.recv()
dev.send(f)
`
I am not sure what the data type is for "f", the data you receive from the function recv.
I am guessing that SocketCanDev is just a wrapper for the device, and recv acts very similar to the function, read().
So, "f" in your code might be interpreted as an array of bytes, or chars.
Hashing is done to an array of bytes, regardless of the format of the
string.
And, the result of the hashing does not depend on the input format or data type.
Therefore, in your case,
while True:
f = dev.recv()
result = (hashFunction)(f)
dev.send(result) // result should in the data type that the function send can accept as a parameter
(hashFunction) can be replaced with an actual function from a hashing library, such as "hashlib".
If you are interested in cryptographic hashing, you should take a look at hashlib
With it you should be able to hash the message and send the hash like this
H = hashlib.new('sha256', message)
dev.send(H.digest())
If you want to still send the original message besides the hash, you could make two calls to send.
I'm coding to tcp client/server using python twisted
in order to replace for Java or C#.
I have to parse length prefixed string messages based on ANS(alpha numeric string) in connected permanent session.
like this :
message format : [alpha numeric string:4byte][message data]
example-1 : 0004ABCD ==> ABCD
example-2 : 0002AB0005HELLO ==> AB, HELLO
it can't be solved by IntNProtocol, NetStringProtocol.
And if a client send a 2kb message from application layer, the kernel split message data by MSS(maximum segment size) and send packets are splitted.
in TCP send/receive environment, it often raise like this :
example : 1000HELLO {not yet arrived 995 byte data}
so it has to wait for receiving spare data using array, queue...
in the twisted, I don't know how to parse multiple large-message.
Anybody help me to give some information or URL?
class ClientProtocol(protocol.Protocol):
def dataReceived(self, data):
# how can I code to parse multiple large message?
# is there solution to read specific size for data ?
It looks like you can implement this protocol using StatefulProtocol as a base. Your protocol basically has two states. In the first state, you're waiting for 4 bytes which you will interpret as a zero-padded base 10 integer. In the second state, you're waiting for a number of bytes equal to the integer read in the first state.
from twisted.protocols.stateful import StatefulProtocol
class ANSProtocol(StatefulProtocol):
def getInitialState(self):
return (self._state_length, 4)
def _state_length(self, length_bytes):
length = int(length_bytes)
return self._state_content, length
def _state_content(self, content):
self.application_logic(content)
return self.getInitialState()
def application_logic(self, content):
# Application logic operating on `content`
# ...
I found this code to detect the length of encrypted data in the frame :
header = self.request.recv(5)
if header == '':
#print 'client disconnected'
running = False
break
(content_type, version, length) = struct.unpack('>BHH', header)
data = self.request.recv(length)
Souce :
https://github.com/EiNSTeiN-/poodle/blob/master/samples/poodle-sample-1.py
https://gist.github.com/takeshixx/10107280
https://gist.github.com/ixs/10116537
This code, listen the connection between a client and a server. When the client talk to the server, self.request.recv(5) can get you the length of the header in the frame. Then we use that length to take the data.
If we print the exchange between the client and the server :
Client --> [proxy] -----> Server
length : 24 #why 24 ?
Client --> [proxy] -----> Server
length: 80 #length of the data
Client <-- [proxy] <----- Server
We can see that the client will send two packet to the server.
If i change
data = self.request.recv(length)
to
data = self.request.recv(4096)
Only one exchange is made.
Client --> [proxy] -----> Server
length: 109 #length of the data + the header
Client <-- [proxy] <----- Server
My question is why we only need to take a size of 5 to get the lenght, content_type informations ? Is there an understandable doc about this ?
Why there is two request: one with 24 and another with the lenght of our data ?
why we only need to take a size of 5 to get the lenght, content_type
informations ?
Because obviously that's the way the protocol was designed.
Binary streams only guarantee that when some bytes are put into one end of the stream, they arrive in the same order on the other end of the stream. For message transmission through binary streams the obvious problem is: where are the message boundaries? The classical solution to this problem is to add a prefix to messages, a so-called header. This header has a fixed size, known to both communication partners. That way, the recipient can safely read header, message, header, message (I guess you grasp the concept, it is an alternating fashion). As you see, the header does not contain message data -- it is just communication "overhead". This overhead should be kept small. The most efficient (space-wise) way to store such information is in binary form, using some kind of code that must, again, be known to both sides of the communication. Indeed, 5 bytes of information is quite a lot.
The '>BHH' format string indicates that this 5 byte header is built up like this:
unsigned char (1 Byte)
unsigned short (2 Bytes)
unsigned short (2 Bytes)
Plenty of room for storing information such as length and content type, don't you think? This header can encode 256 different content types, 65536 different versions, and a message length between 0 and 65535 bytes.
Why there is two request: one with 24 and another with the lenght of
our data ?
If your network forensics / traffic analysis does not correspond to what you have inferred from code, one of both types of analyses is wrong/incomplete. In this case, I guess that your traffic analysis is correct, but that you have not understood all relevant code for this kind of communication. Note that I did not look at the source code you linked to.
I currently have an smtplib client using sendmail. I need to trap the message id given to the sent message by the receiving server - and from what I've read here I need to use smtplib's docmd command. What I've worked out so far is fairly easy:
import smtplib
svr = smtplib.SMTP("localhost",26)
r = svr.docmd("ehlo", "me")
r = svr.docmd("mail from:","me#home.com")
r = svr.docmd("rcpt to:","you#work.com")
r = svr.docmd("data","")
But now when I try to do svr.docmd("send","Hello there") it just hangs? Similarly, I assume I should do r = svr.docmd("send",".") to send the terminating "." character to send the mail, and get the server response (in theory including msg id!) as part of the returned tuple?
Would really appreciate it if someone could point me at where I'm going wrong in these final steps?
Or am I completely misunderstanding - is using docmd NOT the way to get the server response when the mail is sent? If I am wrong, what should I be doing instead?
smtplibmodule cannot work like that. It is a higher level library and you should not try to send simple mail with docmd. docmd only helps to pass special commands. You should simply use :
svr.sendmail('me#home', 'you#work', 'msg')
If you really want to do it the hard way, you should send all the data in only single docmd :
...
r = svr.docmd("data")
r = svr.docmd("line 1\r\nline 2\r\n.\r\n")
That way it a a complete command that receives its SMTP response
Edit :
The return of low-level commands is the content of the SMTP response. The RFC 2821 specifies that this replies consist in a 3 digit number followed by an textual explaination. It can be multiline if the number is followed by a -. But all that is hidden from you by smtplib module. The main point of interest if you use low level commands is first digit from the number :
1yz Positive Preliminary reply
2yz Positive Completion reply
3yz Positive Intermediate reply
4yz Transient Negative Completion reply
5yz Permanent Negative Completion reply
If it is 4 or 5 the command was rejected, 1,2, or 3 it was successful
If you use directly sendmail command, if you do not get an exception, the mail was successfuly received by the server and accepted for at least on recipient. The return value is a dictionary, with one entry for each recipient that was refused. Each entry contains a tuple of the SMTP error code and the accompanying error message sent by the server (extract from the module documentation).
I advice you to use the sendmail command unless you want to learn the SMTP protocol or have other special requirement.
As you know sometimes we can't know what the size of the data(if there is no Content-Length in http response header).
What is the best way to receive http response data(use socket)?
The follow code can get all the data but it will blocking at buf = sock.recv(1024).
from socket import *
import sys
sock = socket(AF_INET, SOCK_STREAM)
sock.connect(('www.google.com', 80))
index = "GET / HTTP/1.1\r\nHOST:www.google.com\r\nConnection:keep-alive\r\n\r\n"
bdsock.send(index)
data = ""
while True:
buf = bdsock.recv(1024)
if not len(buf):
break
data += buf
I'm assuming you are writing the sender as well.
A classic approach is to prefix any data sent over the wire with the length of the data. On the receive side, you just greedily append all data received to a buffer, then iterate over the buffer each time new data is received.
So if I send 100 bytes of data, I would prefix an int 100 to the beginning of the packet, and then transmit. Then, the receiver knows exactly what it is looking for. IF you want to get fancy, you can use a special endline sequence like \x00\x01\x02 to indicate the proper end of packet. This is an easily implemented form of error checking.
Use a bigger size first, do a couple of tests, then see what is the lenght of those buffers, you will then have an idea about what would the maximum size be. Then just use that number +100 or so just to be sure.
Testing different scenarios will be your best bet on finding your ideal buf size.
It would also help to know what protocol you are using the sockets for, then we would have a better idea and response for you.
Today I got the same question again.
And I found the simple way is use httplib.
r = HTTPResponse(sock)
r.begin()
# now you can use HTTPResponse method to get what you want.
print r.read()