Develop Reference

Bidirectional Communication between Python and C#

I'm working on a project, where I have to connect Python and C# in a rather flexible way, with information being send from one to the other frequently. Most of the time, the data goes back and forth (A works with data, sends result to B, B works with result, answers A) but not necessary always.
Named pipes seemed like the way to go but I'm struggling with keeping it all synchronous.
Currently I'm just starting a server on one of them and let it wait for a client connection, while the other creates a server as well, and connects as a client when the server is ready.
It's a lot of creating servers and clients and trying to not break it, which happens as soon as a client tries to connect to an not existing server..
I'd like to do it with a pipe that runs in a separate thread, but couldn't find a way to keep it alive, it simply closes, when the client is done reading.
Are named pipes just too simplistic for such a task, or is there a way to make this work robustly?
Update:
For reference, how my code looks like (reduced to the minimum here):
First the NamedPipe Server and Client in C#
public static void SendDataToPython(string payload)
{
using (var server = new NamedPipeServerStream("Test"))
{
server.WaitForConnection();
using (var stream = new MemoryStream())
using (var writer = new BinaryWriter(stream))
{
writer.Write(payload);
server.Write(stream.ToArray(), 0, stream.ToArray().Length);
}
server.Disconnect();
}
}
public static string WaitForMessage()
{
string message = "";
string servername = ".";
string pipeName = "CSServer";
using (var pipeClient = new NamedPipeClientStream(servername, pipeName, PipeDirection.In))
{
pipeClient.Connect();
And in Python:
class PipeServer():
def __init__(self, pipeName):
self.pipe = win32pipe.CreateNamedPipe(
r'\\.\pipe\\' + pipeName,
win32pipe.PIPE_ACCESS_DUPLEX,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE | win32pipe.PIPE_WAIT,
1, 65536, 65536,
0,
None)
def connect(self):
win32pipe.ConnectNamedPipe(self.pipe, None)
def write(self, message):
win32file.WriteFile(self.pipe, message.encode() + b'\n')
def close(self):
win32file.CloseHandle(self.pipe)
using (StreamReader sr = new StreamReader(pipeClient))
{
string temp;
while ((temp = sr.ReadLine()) != null)
{
message += temp;
}
}
}
return message;
}
class PipeClient:
def __init__(self, pipeName):
self.pipeName = pipeName
def receive(self):
file_handle = win32file.CreateFile(
f"\\\\.\\pipe\\{self.pipeName}",
win32file.GENERIC_READ | win32file.GENERIC_WRITE,
0,
None,
win32file.OPEN_EXISTING,
0,
None)
left, data = win32file.ReadFile(file_handle, 4096)
print(data.decode("utf-8")[1:])

How to decode C++ Google Protocol Buffers WriteVarint32 from Python

Right now we are working on a multiplayer game with a client in python and a server in C++.
We send the size of the header, a header with an object Id and the object. When parsing, we read the size of the header, parse the header and then start parsing the object it indicates. Everything works fine until I try to send a response from C++ to the python client, then _DecodeVarInt32 reads only a byte and determines that the size of the header is 0.
We suspect its because of the int encoding, but we haven't found any material regarding this issue and protocol buffers
Socket send on C++
int sz = header.size()+4+obj.size();
char* pkg = new char[sz];
google::protobuf::io::ArrayOutputStream aos(pkg,sz);
CodedOutputStream *cos= new CodedOutputStream(&aos);
cos->WriteVarint32(header.size());
cos->WriteString(header.data());
cos->WriteString(obj.data());
int rc = sendto(sd,(void*)pkg,sz,0,&sock.sa,sock.sa_len);
Socket receive in C++
google::protobuf::uint32 sz;
google::protobuf::io::ArrayInputStream ais(buffer,MAX_MESSAGE_SIZE);
CodedInputStream cis(&ais);
cis.ReadVarint32(&sz);
auto lim = cis.PushLimit(sz);
sample.from_bin(cis);
cis.PopLimit(lim);
obj.from_bin(cis);
Socket send on python
header = Header(headerId)
header.to_bin()
object.to_bin()
headerSize = _VarintBytes(header._size)
print(len(headerSize))
print(header._size)
print(object._size)
btarray = headerSize + header._data + object._data
print(len(btarray))
self.sock.sendto(btarray,(self.host,self.port))
And socket receive in python
def recvHeader(self):
buf = self.sock.recv(2000,socket.MSG_PEEK)
print(buf)
rc = Header(serialMsg.MessageID.LOGINPETITION)
n = 0
hdSize, n =_DecodeVarint32(buf,n)
rc.from_bin(buf[n:n+hdSize])
n+=hdSize
return (rc,n)
def loadObject(self,object,startPos):
print('start load')
buf = self.sock.recv(2000)
object.from_bin(buf[startPos:])
print('loaded')

i want to receive string from python server

I'm making an application using android and Python.
android is client
python is server
i send image file to python and want to receive string from server
but No strings are coming from the server.
socketIn.readLine() This part does not work.
try { // 소켓을 생성하고 입출력 스트립을 소켓에 연결
clientSocket = Socket(ip , port)
Log.d("Socket>>>>>>", "ip and port open Success!!!!!")
//val inputStream = clientSocket.getInputStream()
val tempfile = file
try{
socketIn = BufferedReader(InputStreamReader(clientSocket.getInputStream(), "UTF-8"))
//socketOut = PrintWriter(BufferedWriter(OutputStreamWriter(clientSocket.getOutputStream())),true)
dis = DataInputStream(FileInputStream(tempfile))
dos = DataOutputStream(clientSocket.getOutputStream())
val buf = ByteArray(1024)
var read_length : Int = 0
do {
read_length = dis.read(buf)
if(read_length == -1)
break
dos.write(buf)
dos.flush()
} while(read_length > 0)
var line : String?
var StringBuilder = StringBuilder()
do {
line = socketIn.readLine()
if(line == null)
break
StringBuilder.append(line)
}while(line != null)
onApiResult(line)
} catch (e : Exception){
Log.d("error", "${e}")
onApiFailed()
} finally {
clientSocket.close()
}
this is my android client code. client send the image to python server using tcp.
The image is sent well but the string does not come.
There is an error here line = socketIn.readLine()
please tell me how to fix it
from socket import *
serverPort = 8000
serverSocket = socket(AF_INET, SOCK_STREAM)
serverSocket.bind(('123.234.345.456', serverPort))
serverSocket.listen(1)
print('The server is ready to receive')
msg = "hi"
while True:
connectionSocket, addr = serverSocket.accept()
img_file = open('hi.jpg', "wb")
while True:
sentence = connectionSocket.recv(1024)
data = sentence
img_file.write(sentence)
if sentence:
print("recving IMg....")
print(sentence)
sentence = connectionSocket.recv(1024)
img_file.write(sentence)
else:
print('Done')
img_file.close()
break
connectionSocket.sendall(bytes(msg, 'UTF-8'))
connectionSocket.close()

Just a guess -- you are sending binary data, but your client code uses java Reader/Writer classes (dedicated to reading text-like data). Use Streams, instead of Reader/Writer. There is no notion of 'endOfLine' when reading binary data. Also note, that client call to 'readLine()' assumes client platform dependent end-of-line, whatever it may be. If server and client platform differ, it will never work.

What is the proper way to exchange data between QTcpSocket and python socket?

I am sending .jpg images from Qt client to python server. Qt client sends as follows:
void Sender::sendToServer(QByteArray imageData)
{
QDataStream stream(serverSocket);
int t = imageData.size();
stream << t;
stream << imageData;
serverSocket->waitForBytesWritten(-1);
qDebug() << "Frame size:" << imageData.size();
}
And the server is:
unpacker = struct.Struct('!i')
conn, addr = s.accept()
bool = True
data = b''
while True:
while len(data) < 4:
try:
data += conn.recv(100)
except:
break
img_size = unpacker.unpack(data[:4])[0]
print('size: ', img_size)
print('Data: ',data)
print('Image: ',data[:8]) # deleting preceding 8 bytes
# image processing
The output is:
//Client:
Frame size: 49993
//Server:
size: 49993
Data: b'\x00\x00\xc3\r\x00\x00\xc3\r\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01...
Image: b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00...
In the output, .jpg file starts from '\xff\xd8'. As C++ int is 4 byte, the server expects "{4 byte}\xff\xd8...". However, preceding bytes are always of different size. Sometimes it's 8 bytes, but sometimes even different (such as 6 or 9 bytes). I could not figure out the reason behind it. What is the problem here?
Platform: win7, Qt 5.9.1, Python 3.4.
Edit
Following #Frank's answer, I changed the code as follows and everything works fine:
Qt client send method:
void Sender::sendToServer(QByteArray imageData)
{
QDataStream stream(serverSocket);
QString size = QString::number(data.size());
QString size_8_byte = size.rightJustified(8, ' '); // add spaces to make it exactly 8 bytes
imageData.prepend(size_8_byte.toUtf8());
stream.writeRawData((char*)(imageData.data()), imageData.size());
serverSocket->waitForBytesWritten(-1);
}
Python server:
unpacker = struct.Struct('8s')
conn, addr = s.accept()
data = b''
bool = True
while True:
while len(data) < 8:
try:
data += conn.recv(100)
except:
break
img_size = unpacker.unpack(data[:8])[0]
img_size = int(img_size.decode("utf-8").strip()) #remove spaces and convert to int
data = data[8:] #correct data
#image processing

QDataStream is not a simple binary dumper, it formats the data in some internally defined way. This helps with stuff like endianness correction and the like,
However, there is a very simple fix available to you: QDataStream provides the writeRawData() function, which bypasses all formatting operations.
Simply change your output operations to using it and you should be good to go.

One serial communication - multiple read and write between Python script and Arduino

I would like to set up a serial communication between a Python daemon and an Arduino.
At first, the Python daemon sets up a serial connection that will last for the whole lifetime of the daemon. Through this connection, I would like to send data to the Arduino and receive back data in the acks variable every time the Python daemon receives commands.
The problem is that while the first time the communication goes well, nothing is sent through serial afterwards. If I make the a new connection for every request it works, but it makes the program very slow, which I'd like to avoid.
edit: the real issue is when send a correct string to the arduio evrything goes well but when i send a wrong one the serial port block and it will never reconize corrct strings again( the problem is in the arduino code)
Python code:
import serial
import time
import sys
from socket import *
import threading
import thread
def handler(clientsock,addr):
while 1:
#arduino.flush()
data = clientsock.recv(BUFSIZ)
if not data:
break
print data
print data
#time.sleep(3)
arduino.write(data)
#time.sleep(3)
ack = arduino.readline(1)
arduino.flush()
clientsock.send(ack+"\n")
clientsock.close()
if __name__=='__main__':
HOST = '0.0.0.0'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
arduino = serial.Serial('/dev/ttyACM0',9600,timeout=6)
serversock = socket(AF_INET, SOCK_STREAM)
serversock.bind(ADDR)
serversock.listen(2)
while 1:
print 'waiting for connection...'
clientsock, addr = serversock.accept()
print '...connected from:', addr
thread.start_new_thread(handler, (clientsock, addr))
Arduino code:
int relayPinCH1 = 7; // pin de commande du relais 1
char inData[20]; // Allocate some space for the string
char inChar=-1; // Where to store the character read
byte index = 0; // Index into array; where to store the character
void setup()
{
pinMode(relayPinCH1, OUTPUT);
Serial.begin(9600);
}
char Comp(char* This) {
while (Serial.available() > 0) // Don't read unless
// there you know there is data
{
if(index < 19) // One less than the size of the array
{
inChar = Serial.read(); // Read a character
inData[index] = inChar; // Store it
index++; // Increment where to write next
inData[index] = '\0'; // Null terminate the string
}
}
Serial.flush();
if (strcmp(inData,This) == 0) {
for (int i=0;i<19;i++) {
inData[i]=0;
}
index=0;
return(0);
}
else {
return(1);
}
}
void loop()
{
//Serial.println("Hello Pi");
if (Comp("l11\n")==0)
{
Serial.flush();
digitalWrite(relayPinCH1, HIGH);
Serial.println("y");
}
if (Comp("l10\n")==0)
{
Serial.flush();
digitalWrite(relayPinCH1, LOW);
Serial.println("n");
}
delay(1000);
}

In your Arduino code, your logic is kind of funky - so, I'm not sure, but are you clearing index to 0 before you start the loop again? It looks like once index == 19, it may or may not get reset to 0 depending upon later logic. If you enter Comp() a second time and index >= 19 then you'll never read the serial port again.

I think #Zeus is entirely right (and hence I upvoted that answer), but there are also other problems. To reiterate what #Zeus is saying:
index is only reset to 0 if the comparison succeeds. So your buffer is full, the string you are looking for isn't there, and index never goes back to 0 again.
Once index reaches 19, no more reading is done. As a result, whatever is in inData stays in inData and all the future comparisons will fail, meaning index will never get reset to 0.
There are a number of other problems in the code, but the main issue is that the design is very fragile, and prone to exactly the sort of error you are experiencing. For instance if the newlinews your Python script is sending are CR+LF for newlines, but you are expecting CR only, you'll have the same sort of failure you have now: first time communications work, but never again.
I would suggest reorganizing your code like this:
Your function for reading serial port reads a line from a serial port and returns that to the caller (without the newlines), regardless of the content of the communications.
The caller compares the line received from the serial port with the list of known commands and executes them accordingly.
This might look rougly as follows
char strCommand[0xFF];
int idxCommandChar;
// Read a command from serial, returning the command size
// This function BLOCKS, i.e., doesn't return until a command is available
int readSerialCommand() {
// We reset the index to zero on every read: the command is overwritten every time
idxCommandChar = 0;
// Read serial characters and store them in strCommand
// until we get a newline
int in = Serial.read();
while (in!='\n') {
strCommand[idxCommandChar++] = in;
in = Serial.read();
}
// Add the string terminator
strCommand[idxCommandChar++] = '\0';
// Return command size
return idxCommandChar;
}
// Get command from serial, and process it.
void processCommand() {
readSerialCommand();
if (strcmp(strCommand, "CMD1")==0) {
// do something
} else if (strcmp(strCommand, "CMD2")==0) {
// do something else
} else {
// Unknown command
Serial.println("Unknown command");
}
}
void loop() {
processCommand();
delay(1000);
}
This code blocks on serial, i.e. doesn't return until a newline is detected. You could easily modify the code to be non-blocking, perhaps like this:
/* Read serial characters, if available and store them in strCommand
until we get a newline
Returns 0 if no command is available */
int readSerialCommand() {
idxCommandChar = 0;
while (Serial.available()) {
int in = Serial.read();
while (in!='\n') {
strCommand[idxCommandChar++] = in;
in = Serial.read();
}
strCommand[idxCommandChar++] = '\0';
return idxCommandChar;
}
return 0;
}
// Get command from serial (if available), and process it.
void processCommand() {
if (readSerialCommand()) {
....
In either case you might loose serial characters while you are waiting, so you may want to rethink that strategy.