My problem: Ok, I made a little chat program thing where I am basically using sockets in order to send messages over a network.
It works great, but when I decided to take it a step further, I ran into a problem.
I decided to add some encryption to the strings I was sending over the network, and so I went ahead and wrote the script that did that.
The problem is that apparently you can't just send a dictionary through sockets as you might with a string.
I did some research first, and I found this stuff about Pickles. Unfortunately, I couldn't find out exactly how I could use them to convert strings, aside from having it exporting the dictionary to a file, but I can't do that without changing my program.
Can anyone help explain how I am to do this? I've looked around everywhere but I can't seem to find out how.
I've uploaded what I've got so far here, if that comes of any interest to anybody.
print("\n\t\t Fill out the following fields:")
HOST = input("\nNet Send Server Public IP: ")
PORT = int(input("\nNet Send Server Port: "))
#------------------------------------------------
#Assessing Validity of Connection
#------------------------------------------------
try:
s = socket(AF_INET,SOCK_STREAM)
s.connect((HOST,PORT))
print("Connected to server:",HOST,)
except IOError:
print("\n\n\a\t\tUndefined Connection Error Encountered")
input("Press Enter to exit, then restart the script")
sys.exit()
#-------------------------------------------------
#Now Sending and recieving mesages
#-------------------------------------------------
i = True
while i is True:
try:
User_input = input("\n Enter your message: ")
Lower_Case_Conversion = User_input.lower()
#Tdirectory just stores the translated letters
Tdirectory = []
# x is zero so that it translates the first letter first, evidently
x = 0
COUNTLIMIT = len(Lower_Case_Conversion)
while x < COUNTLIMIT:
for letter in Lower_Case_Conversion[x]:
if letter in TRvalues:
Tdirectory += [TRvalues[Lower_Case_Conversion[x]]]
x = x + 1
message = input('Send: ')
s.send(message.encode())
print("\n\t\tAwaiting reply from: ",HOST,)
reply = s.recv(1024)
print(HOST,"\n : ",reply)
except IOError:
print("\n\t\aIOError Detected, connection most likely lost.")
input("\n\nPress Enter to exit, then restart the script")
Oh, and if your wondering what TRvalues is. It's the dictionary that contains the 'translations' for encrypting simple messages.
try:
TRvalues = {}
with open(r"C:\Users\Owatch\Documents\Python\FunStuff\nsed.txt", newline="") as f:
reader = csv.reader(f, delimiter=" ")
TRvalues = dict(reader)
(The translations are held in a .txt it imports)
You have to serialize your data. there would be many ways to do it, but json and pickle will be the likely way to go for they being in standard library.
for json :
import json
data_string = json.dumps(data) #data serialized
data_loaded = json.loads(data) #data loaded
for pickle(or its faster sibling cPickle):
import cPickle as pickle
data_string = pickle.dumps(data, -1)
#data serialized. -1, which is an optional argument, is there to pick best the pickling protocol
data_loaded = pickle.loads(data) #data loaded.
also, please don't write
i= True
while i is True:
#do_something
because simple while True: would suffice.
You need to serialize your data first. There are several ways to do this, the most common probably JSON, XML and (python specific) pickles. Or your own custom serialization.
The basic idea is: Serialize your data, send it, receive it, deserialize it again.
If you want to use pickle you can use the loads and dumps functions.
import pickle
a_dict = { x:str(x) for x in range(5) }
serialized_dict = pickle.dumps(a_dict)
# Send it through the socket and on the receiving end:
a_dict = pickle.loads(the_received_string)
You can also use JSON in a similar fashion. I like JSON because it is human readable and isn't python specific.
import json
a_dict = { x:str(x) for x in range(5) }
serialized_dict = json.dumps(a_dict)
# Send it through the socket and on the receiving end:
a_dict = json.loads(the_received_string)
You can use pickle and python remote object (or pyro only), to send full objects and data over networks (Internet included). For instance, if you want send object (dict, list, class, objects, etc. ) use python remote objects for it.
It very useful for you want to do.
There is more information in this link http://pythonhosted.org/Pyro4/
And this starter manual can be useful to know what you send or execute on network pcs http://pythonhosted.org/Pyro4/intro.html#simple-example
I hope it will help you
Using JSON to serialize your data is the way I prefer to do it. I actually made a library that does just that for you: jsonsocket library. It will do the serialization/deserialization automatically for you. It also handles big amounts of data efficiently.
You can also use zmqObjectExchanger (https://github.com/ZdenekM/zmq_object_exchanger). It wraps pickle and zmq to transfer python objects over network.
Related
I want to use boto3 to run a command on an ECS Fargate container which generates a lot of binary output, and stream that output into a file on my local machine.
My attempt is based on the recommendation here, and looks like this:
import json
import uuid
import boto3
import construct as c
import websocket
# Define Structs
AgentMessageHeader = c.Struct(
"HeaderLength" / c.Int32ub,
"MessageType" / c.PaddedString(32, "ascii"),
)
AgentMessagePayload = c.Struct(
"PayloadLength" / c.Int32ub,
# This only works with my test command. It won't work with my real command that returns binary data
"Payload" / c.PaddedString(c.this.PayloadLength, "ascii"),
)
# Define initial payload
init_payload = {
"MessageSchemaVersion": "1.0",
"RequestId": str(uuid.uuid4()),
"TokenValue": session["tokenValue"],
}
# Define the container you want to talk to
cluster = "..."
task = "..."
container = "..."
# Send command with large response (large enough to span multiple messages)
result = client.execute_command(
cluster=cluster,
task=task,
container=container,
# This is a sample command that returns text. My real command returns hundreds of megabytes of binary data
command="python -c 'for i in range(1000):\n print(i)'",
interactive=True,
)
# Get session info
session = result["session"]
# Create websocket connection
connection = websocket.create_connection(session["streamUrl"])
try:
# Send initial response
connection.send(json.dumps(init_payload))
while True:
# Receive data
response = connection.recv()
# Decode data
message = AgentMessageHeader.parse(response)
payload_message = AgentMessagePayload.parse(response[message.HeaderLength:])
if 'channel_closed' in message.MessageType:
raise Exception('Channel closed before command output was received')
# Print data
print("Header:", message.MessageType)
print("Payload Length:", payload_message.PayloadLength)
print("Payload Message:", payload_message.Payload)
finally:
connection.close()
This almost works, but has a problem - I can't tell when I should stop reading.
If you read the final message from aws, and call connection.recv() again, aws seems to loop around and send you the initial data - the same data you would have received the first time you called connection.recv().
One semi-hackey way to try to deal with this is by adding an end marker to the command. Sort of like:
result = client.execute_command(
...
command="""bash -c "python -c 'for i in range(1000):\n print(i)'; echo -n "=== END MARKER ===""""",
)
This idea works, but to be used properly, becomes really difficult to use. There's always a chance that the end marker text gets split up between two messages, and dealing with that becomes a pain, since you can no longer write a payload immediately to disk until you verify that the end of the payload, along with the beginning of the next payload, isn't your end marker.
Another hackey way is to checksum the first payload, and every subsequent payload, comparing the checksum of each payload to the checksum of the first payload. That will tell you if you've looped around. Unfortunately, this also has a chance of having a collision, if the binary data in 2 messages just happens to repeat, although the chances of that in practice would probably be slim.
Is there a simpler way to determine when to stop reading?
Or better yet, a simpler way to have boto3 give me a stream of binary data from the command I ran?
I am trying to read some data streams using protobuf in python, and i want to use trio to make the client for reading the streams. The protobuf has some method calls, and i find they do not work when i use trio streams.
Python client on a linux machine.
import DTCProtocol_pb2 as Dtc
async def parent(addr, encoding, heartbeat_interval):
print(f"parent: connecting to 127.0.0.1:{addr[1]}")
client_stream = await trio.open_tcp_stream(addr[0], addr[1])
# encoding request
print("parent: spawing encoding request ...")
enc_req = create_enc_req(encoding) # construct encoding request
await send_message(enc_req, Dtc.ENCODING_REQUEST,client_stream, 'encoding request') # send encoding request
log.debug('get_reponse: started')
response = await client_stream.receive_some(1024)
m_size = struct.unpack_from('<H', response[:2]) # the size of message
m_type = struct.unpack_from('<H', response[2:4]) # the type of the message
m_body = response[4:]
m_resp = Dtc.EncodingResponse()
m_body would be some bytes data, which I dont know how to decode. Dtc.EncodingResponse() is the protobuf method which would give a Dtc object which contains the response in a readable format. (Dtc is the protobuf file). But I get nothing here. When I did this script without trio, Dtc.EncodingResponse() would give the full response in readable format.
I am guessing the problem is that the "client_stream" is a trio stream object that only reads bytes, and so I probably need to use a ReceiveChannel object instead. But if this is true, I dont know how to do this.
UPDATE:
The answer below by Nathaniel J. Smith solves my problem.
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_body)
I feel so silly, but I did not ParseFromString the data previously, and that was all it took. Extremely grateful to all who gave replies. Hope this helps someone out there.
Like #shmee said in the comment, I think your code got mangled some by the edits... you should double-check.
When I did this script without trio, Dtc.EncodingResponse() would give the full response in readable format
I think you might have dropped a line when switching to Trio? Dtc.EncodingResponse() just creates a new empty EncodingResponse object. If you want to parse the data from m_body into your new object, you have to do that explicitly, with something like:
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_body)
However, there's another problem... the reason it's called receive_some is that it receives some bytes, but might not receive all the bytes you asked for. Your code is assuming that a single call to receive_some will fetch all the bytes in the response, and that might be true when you're doing simple test, but in general it's not guaranteed. If you don't get enough data on the first call to receive_some, you might need to keep calling it repeatedly until you get all the data.
This is actually very standard... sockets work the same way. That's why the first thing your server is sending an m_size field at the beginning – it's so you can tell whether you've gotten all the data or not!
Unfortunately, as of June 2019, Trio doesn't provide a helper to do this loop for you – you can track progress on that in this issue. In the mean time, it's possible to write your own. I think something like this should work:
async def receive_exactly(stream, count):
buf = bytearray()
while len(buf) < count:
new_data = await stream.receive_some(count - len(buf))
if not new_data:
raise RuntimeError("other side closed the connection unexpectedly")
buf += new data
return buf
async def receive_encoding_response(stream):
header = await receive_exactly(stream, 4)
(m_size, m_type) = struct.unpack('<HH', header)
m_body = await receive_exactly(stream, m_size)
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_size)
return m_resp
I've got a client/server program where the client sends plaintext to the server which then runs AES encryption and returns the ciphertext. I'm using the following algorithm for the encryption/decryption:
http://anh.cs.luc.edu/331/code/aes.py
When I get the results back from the encryption and print them on the server-side I see mostly gibberish in the terminal. I can save to a file immediately and get something along these lines:
tgâY†Äô®Ø8ί6ƒlÑÝ%ŠIç°´>§À¥0Ð
I can see that this is the correct output because if I immediately decrypt it on the server, I get the original plaintext back. If I run this through the socket, send it back to the client, and print(), I get something more like this:
\rtg\xe2Y\x86\x8f\xc4\xf4\xae\xd88\xce\xaf6\x83l\xd1\xdd%\x8aI\xe7\xb0\xb4>\xa7\xc0\x18\xa50\xd0
There's an obvious difference here. I'm aware that the \x represents a hex value. If I save on the client-side, the resulting text file still contains all \x instances (i.e., it looks exactly like what I displayed directly above). What must I do to convert this into the same kind of output that I'm seeing in the first example? From what I have seen so far, it seems that this is unicode and I'm having trouble...
Relevant code from server.py
key = aes.generateRandomKey(keysizes[len(key)%3])
encryptedText = aes.encryptData(key, text)
f = open("serverTest.txt", "w")
f.write(encryptedText)
f.close()
print(encryptedText)
decryptedText = aes.decryptData(key, encryptedText)
print(decryptedText)
conn.sendall(encryptedText)
Relevant code from client.py
cipherText = repr(s.recv(16384))[1:-1]
s.close()
cipherFile = raw_input("Enter the filename to save the ciphertext: ")
print(cipherText)
f = open(cipherFile, "w")
f.write(cipherText)
Edit: To put this simply, I need to be able to send that data to the client and have it display in the same way as it shows up on the server. I feel like there's something I can do with decoding, but everything I've tried so far doesn't work. Ultimately, I'll have to send from the client back to the server, so I'm sure the fix here will also work for that, assuming I can read it from the file correctly.
Edit2: When sending normally (as in my code above) and then decoding on the client-side with "string-escape", I'm getting identical output to the terminal on both ends. The file output also appears to be the same. This issue is close to being resolved, assuming I can read this in and get the correct data sent back to the server for decrypting.
Not sure I fully understood what you're up to, but one difference between client and server is that on the client you're getting the repr for the byte string, while on the server you print the byte string directly.
(if I got the issue right) I'd suggest replacing
repr(s.recv(16384))[1:-1]
with a plain
s.recv(16384)
I'm new to Python and want to read my smart meters P1 port using a Raspberry Pi and Python. Problem: the input looks like some component is drunk.
I'm sure it's pretty simple to fix, but after several hours of searching and trying, had to seek help.
When reading the P1 port with CU etc. everything is fine so the hardware etc. is OK. Using a serial to USB converter from dx.com (this one)
Command and (part of) the output: cu -l /dev/ttyUSB0 -s 9600 --parity=none
0-0:96.1.1(205A414246303031363631323463949271)
1-0:1.8.1(03118.000*kWh)
However, when trying to read it from Python, the input becomes gibberish (but at least sort of consistant):
0-0:96.±.±(²05A´±´²´630303±39363±3²3´639·3±3²©
±-0:±.¸.±(03±±¸.000ªë×è©
How to fix this? The code I'm using is:
import serial
ser = serial.Serial()
ser.baudrate = 9600
ser.bytesize=serial.SEVENBITS
ser.parity=serial.PARITY_EVEN
ser.stopbits=serial.STOPBITS_ONE
ser.xonxoff=0
ser.rtscts=0
ser.timeout=20
ser.port="/dev/ttyUSB0"
ser.close()
ser.open()
print ("Waiting for P1 output on " + ser.portstr)
counter=0
#read 20 lines
while counter < 20:
print ser.readline()
counter=counter+1
try:
ser.close()
print ("Closed serial port.")
except:
sys.exit ("Couldn't close serial port.")
Have already tried messing with baudrate etc. but that doesn't make any difference.
I'm not very familiar with the serial module, but I noticed that your cu command assumes there is no parity bit (--parity=none), but your python script assumes there is an even parity bit (ser.parity=serial.PARITY_EVEN). I would try
ser.parity=serial.PARITY_NONE
And if there's no parity bit, you'll also probably want
ser.bytesize=serial.EIGHTBITS
UPDATE: found a workaround by replacing the naughty characters.
This may work for others with the same problem, but I dont know if the bad characters are exactly the same. So the replacement part may need some work to make it work for others.
It's not exactly a solution as the incoming telegram is still messed up, but the following code will work around that. My telegram is completely clean now.
Relevant part of the code I'm using now:
#Define 2 variables
P1_numbers = {'±':'1', '²':'2', '´':'4', '·':'7', '¸':'8'}
P1_rest = {'¯':'/', 'ª':'*', '©':')', 'Æ':'F', 'ë':'k', '×':'W', 'è':'h', 'í':'m'}
# Define function to read the telegram. Calls a function to clean it.
def P1_read(stack):
counter = 0
while counter < TelegramLength:
stack.append(P1_clean(ser.readline()))
counter=counter+1
return stack
# Define function to clean up P1 output
def P1_clean(line):
for i, j in P1_numbers.iteritems():
line = line.replace(i, j)
for i, j in P1_rest.iteritems():
line = line.replace(i, j)
return line
I quess you have a smart meter with P1 protocol: DSMR 3.0?
Then these are the correct serial port settings, which you already had:
serialport = serial.Serial( # Configure Serial communication port
baudrate = 9600,
timeout = 11,
bytesize = serial.SEVENBITS,
parity = serial.PARITY_EVEN,
stopbits = serial.STOPBITS_ONE )
Probably some encoding or interpretation of the data is going wrong at your side. Here is an other method the read the smart meter:
To make the readout of the p1 protocol as easy as possible I'd suggest to use TextIOWrapper, this way you can read the serial port with the readline method. The "!" always ends the P1 telegram, so that can be used to detect the end of the message. when a full telegram has been received, the telegram can be processed. Example:
import io
p1port = io.TextIOWrapper(io.BufferedReader(serialport, buffer_size=1), newline='\n', encoding='ascii')
P1Message = []
while True:
try:
rawline = self.p1port.readline()
except UnicodeDecodeError:
print "Encode error on readline"
if '!' in rawline:
# Process your P1Message here
P1Message = [] # Clear message, wait for new one
else:
P1Message.append(rawline)
The OP is looooong gone, but the problem is of sufficiently general interest, so here's a fresh answer. User #Brionius is right: A look at the bit patterns involved shows that it's definitely a parity problem. Here's how to inspect the bit pattern of the characters "1" and "±":
>>> "{0:b}".format(ord("1"))
'110001'
>>> "{0:b}".format(ord("±"))
'10110001'
Get it? The characters are getting corrupted by having their high (8th) bit turned on. Or you can see this by setting the high bit of ascii "1":
>>> chr(ord("1") | 0b10000000)
'±'
Now, "1", "2" and "4" have three bits set (odd parity), and are corrupted. "0", "3", "5", etc. have even parity (2 or 4 bits set), and are preserved. So the communication channel is using even parity, which is not decoded properly at the receiving end.
I had the same problem, also in the context of the P1 Smart Meter port, and it took me quite a while to find it.
'cu' was displaying the data correctly, but Python wasn't (nor were some other programs). Apparently the parity bit is somehow not handled correctly. The following solves this problem:
p1_raw = ser.readline()
print 'raw:', p1_raw
decoded = ''.join(chr(ord(ch) & 0x7f) for ch in p1_raw)
print 'decoded:', decoded
I still find it strange that this is happening, because this was actually happening when I was trying to read the output of a Smart Meter for the second time. I already had a script successfully monitoring another Smart Meter at a different house for a couple of years and I never ran into this problem.
Perhaps there's a small difference in the USB-Serial adapters that causes this?!?
I'm attempting to send an HTTP Request to a website and read the data it returns. The first website I tried worked successfully. It returned about 4 packets of data and then returned a 0 packet which the script caught and terminated.
However, attempting to load http://www.google.com/ does not work this way. Instead, it returns about 10 packets of the same length, a final smaller packet, and then proceeds to time out. Is it normal for this to happen? Does it all just depend on what server the host is using?
If anyone could recommend an alternative way to reading with socket.recv() that would take into account that a final null packet is not always sent, it would be greatly appreciated. Thanks.
try:
data = s.recv(4096)
while True:
more = s.recv(4096)
print len(more)
if not more:
break
else:
data += more
except socket.timeout:
errMsg = "Connection timed-out while connecting to %s. Request headers were as follows: %s", (parsedUrl.netloc, rHeader.headerContent)
self.logger.exception(errMsg)
raise Exception
For HTTP, use requests rather than writing your own.
> ipython
In [1]: import requests
In [2]: r = requests.get('http://www.google.com')
In [3]: r.status_code
Out[3]: 200
In [4]: r.text[:80]
Out[4]: u'<!doctype html><html itemscope="itemscope" itemtype="http://schema.org/WebPage">'
In [5]: len(r.text)
Out[5]: 10969
TCP does not give you "packets", but sequential bytes sent from the other side. It is a stream. recv() gives you chunks of that stream that are currently available. You stitch them back together and parse the stream content.
HTTP is rather involved protocol to work out by hand, so you probably want to start with some existing library like httplib instead.
It could be that Google uses Keep-Alive to keep the socket open in order to serve a further request. This would require parsing of the header and reading the exact number of bytes.
Depending on which version of HTTP you use, you have to add Connection: Keep-Alive to your headers or not. (This might be the simplest solution: just use HTTP/1.0 instead of 1.1.)
If you use that feature nevertheless, you would have to receive your first chunk of data and
parse if there is a '\r\nContent-Length: ' inside, and if so, take the bytes between that and the next '\r\n' and convert them to a number. That is your size.
Have a look if you have a '\r\n\r\n' in your data. If so, that is the end of your header. From here, you must read the exact number of bytes mentionned above.
Example:
import socket
s = socket.create_connection(('www.google.com', 80))
s.send("GET / HTTP/1.1\r\n\r\n")
x = s.recv(10000)
poscl = x.lower().find('\r\ncontent-length: ')
poseoh = x.find('\r\n\r\n')
if poscl < poseoh and poscl >= 0 and poseoh >= 0:
# found CL header
poseocl = x.find('\r\n',poscl+17)
cl = int(x[poscl+17:poseocl])
realdata = x[poseoh+4:]
Now, you have the content length in cl and the (start of the) payload data in realdata. The number of bytes missing of this request is missing = cl - len(realdata). If it is 0, you've got everything; if not, do s.read(missing) and recalculate missing until it is 0.
The code above is a simppe start of the job to be done; there are some places where you might need to recv() further before you can proceed.
This is quite compliated. By far easier ways would be
to use HTTP 1.1's Connection: close header in the request,
to use HTTP 1.0,
to use one of the libraries crafted for this task and not to reinvent the wheel.