Develop Reference

Python and C pipeline: Python does not write from buffer when I flush

This is part of a university assignment but the problem I have does not relate to the assignment itself.
I have a C file that is waiting for two inputs from terminal, there are two gets(info1) and later a gets(info2). I have a buffered writed that I use to do the following inside python:
sys.stdout.buffer.write("Hello 1".encode("ascii"))
sys.stdout.flush()
sys.stdout.buffer.write("Hello 2".encode("ascii"))
sys.stdout.flush()
I then have a pipeline that sends these outputs from the python code to the input of the C program using the terminal. When I run the command I can see on the prints that both "Hello 1" and "Hello 2" is sent to gets(info1). Why is this happening? Since I flush the buffer should it not send the first input to the terminal, getting catched by the C first gets(info1) and "Hello 2" getting catched in the second gets(info2)? I even introduced a sleep function after the first flush but it sleeps then sends both the outputs to the first gets(info1). The pipeline obviously works since the C program is able to get the output from terminal produced by the python program. But why am I only gettings inputs to the first function even though I flush the buffer after the first string is written?
When I do
sys.stdout.buffer.write("Hello 1".encode("ascii"))
sys.stdout.buffer.write("\n".encode("ascii"))
sys.stdout.flush()
sys.stdout.buffer.write("Hello 2".encode("ascii"))
sys.stdout.flush()
It sends it properly. However, I need the output to be very specific

I got it working. See stdout buffering. I needed to feed stdout a newline instead of flushing, all good! :)

Why is Pycharm failing to convert string to integer?

num = 0
for i in range(5):
ask = int(input())
if abs(ask) > 0:
num = ask
print(num)
When I run the code, it lets me input the first string. But once I enter the second string, the program crashes and says "pythonProject\main.py", line 3, in
ask = int(input())
ValueError: invalid literal for int() with base 10: ''"
What's going on?
My input: I just type 1, press enter, type 2, press enter and then it crashes. I am sure it is not an error where I click enter too quickly, or I accidentally type in an empty string because I've ran the code multiple times.
What have I tried so far?
Creating a new project and pasting the code -> didn't work
Asking my friend to copy the code onto his PyCharm and run it -> worked fine on his computer
'Edit configurations', uncheck 'emulate code in output console' -> didn't work, it was already unchecked
Checked that I was running the correct file in the project -> didn't work, I was running the right file
EDIT:
FIXED, just needed to check 'Emulate code in output console' rather than uncheck it. Not sure why this works though, or how I can keep it checked for all future projects - rather than having to manually check it every time.

FIXED, just needed to check 'Emulate code in output console' rather than uncheck it. Not sure why this works though, or how I can keep it checked for all future projects - rather than having to manually check it every time.

The problem is with the input, as you are either pressing enter without any input (empty string) or you are entering a float value. This thread might help you in this case. The code is working fine when I input an integer and gives the same error when entered empty string or a float value.

To get it work you need to check "Emulate code in output console".
I've answered in the comments section and I'm glad it worked out, here is an explanation:
You need to know a concept of "terminal emulator" to understand why and how this works. When a program is ran (at least on UNIX-like operating systems), it has three I/O streams: stdin, stdout and stderr. The stdin is used to input data, and two others are for output.
Input or output stream is just a buffer used to communicate with the program back and forth. Once something is written to the buffer, it can be read from there. If the buffer is empty, an attempt to read from there will cause stall until the buffer has something in it. More about stdio: https://en.wikipedia.org/wiki/Standard_streams
When the program is ran through the terminal emulator, I/O streams are connected to this emulator, so whatever you type in the terminal window is written to your stdin by default. Whatever your program writes to the stdout and stderr is displayed on the screen. (However, this behavior may be changed using pipes, so you can pass data from some file to the stdin and also you can redirect the output to the file)
Here is the history behind terminal emulators, to understand, why is it implemented this way: https://en.wikipedia.org/wiki/Terminal_emulator#Computer_terminals
For example, you have a simple program:
s = input('Enter string: ')
print(f'stdout: {s}')
If you run it from the terminal and type "TEST":
$ python3 test.py
TEST
stdout: TEST
But you also can, for example, pass data directly to stdin, and redirect output to the file:
$ echo "ABCDEF" | python3 test.py > OUTPUT.txt
there will be no text in the terminal, but OUPUT.txt will appear. It will contain:
stdout: ABCDEF
Now, about PyCharm:
By default, when it runs your script, it does not automatically emulate terminal in the output window. It simply does not send anything to the stdin and it won't react to pressed keys. When your program gets to the line with input(), it starts to read the stdin stream until it gets \n character from the stream (indicating that user has pressed Return key). As nothing gets sent to the stream, it will wait infinitely.
Useful tip: for testing, instead of just typing something into the terminal every time, you can also check "Redirect input from:" and choose an input file.

RPi - Python Curses program running at boot does not have keyboard focus

I am trying to write a program that will run when my raspberry pi starts up and will allow me to immediately begin typing things with my keyboard and have it be picked up by the program. I don't want to have to manually start the program when the pi starts. I need to use curses (or a similar unbuffered keyboard input library) because I display what I am typing on a 2x16 I2C LCD, but I also need everything that I am typing to be recorded to a text file.
Right now, I am auto-starting the program at boot by putting a line in rc.local. This works, and the I2C display is correctly showing program output, but it does not respond to keyboard input, and the keyboard input is instead displayed (when I connect the pie to a screen, the goal is to run headless) on an odd console layout that exits when I press enter and says -bash: 'whatever I just typed' command not found.
I have already tried:
Setting a timer at the beginning of the program to wait until the pi has fully booted before initializing the curses window and keyboard capture
Creating a seperate python program to wait until the pi has fully booted and then running the main script by importing it
Neither of these methods works though, I get the same problem with slight differences.
To be clear, the program works flawlessly if I run it manually from the command line. But there is no keyboard input to the program (or at least not where it is supposed to be inputted) when I autostart the script with rc.local.
My code:
#!/usr/bin/python
import I2C_LCD_driver, datetime, sys
from time import *
from subprocess import call
mylcd = I2C_LCD_driver.lcd()
for x in range(30): #waits for raspberry pi to boot up
mylcd.lcd_display_string("Booting Up: "+str(x), 1)
sleep(1)
import curses
key = curses.initscr()
curses.cbreak()
curses.noecho()
key.keypad(1)
key.nodelay(1)
escape=0
while escape==0:
#variable initialization
while 1:
k=key.getch()
if k>-1: #runs when you hit any key. getch() returns -1 until a key is pressed
if k==27: #exits the program when you hit Esc
break
elif k==269:
# a couple other special Function key cases are here
else:
inpt=chr(k)
mylcd.lcd_display_string(inpt,2,step) #writes the last character to the display
#some more code that handles writing the text to the LCD, which works flawlessly when run manually.
file.write("%s\r\n" % entry)
file.close()
mylcd.lcd_display_string("Saved ",2)
mylcd.lcd_display_string("F1 New F2 PwrOff",1)
while 1:
k=key.getch()
if k>-1:
if k==265: #do it again! with F1
mylcd.lcd_clear()
break
elif k==266: #shut down with F2
escape=1
break
curses.nocbreak()
key.keypad(0)
curses.echo()
curses.endwin()
call("sudo shutdown -h now", shell=True)
The line that I have in /etc/rc.local is as follows if that is important:
sudo python3 journal.py &
and it is followed by the 'exit 0' line.
Thanks for any help you can provide. I know this is a very specific problem and will be tedious to reproduce, but if anyone knows anything about autostarting functions I would be very appreciative of any tips.

Ok, literally all I had to do (which I did find after some more research on stackexchange, this is the thread that contained the answer I was looking for) was run my program from ~/.bashrc instead of /etc/rc.local. This method works perfectly, exactly what I wanted.

This should be because of how you called the program:
python3 journal.py &
You may want to check out JOB CONTROL of bash (or your shell) man page:
Only foreground
processes are allowed to read from ... the terminal. Background processes which
attempt to read from ... the
terminal are sent a SIGTTIN ... signal by the kernel's terminal
driver, which, unless caught, suspends the process.
In short, once curses (or anything for that matter) try to read from stdin your process is likely stopped (after it may have already written to your display). Keep it in the foreground to be able to have it use stdin (and by extension keyboard).
Side note: Not sure about distro and details of implementation of rc.local in your case, but aren't init scripts normally run with uid/gid 0 already (without wrapping individual calls through sudo?)

Printing to (and overwriting) same line for a status indicator

I am running the following on Mac OSX and Python 3. I am running the snippet using Sublime Text 3's default build system for Python 3 and displaying the output in the same program's output window.
print('Starting test.')
for i in range(1,100):
print('This is a test. ' + str(i), end='\r')
print('Test done!')
My desired output is basically a repetition of that same line, overwriting the previous line in the console each time. By the end, I should ideally only see the following in my output window.
Starting test.
This is a test. 99
Test done!
I would expect the end='\r' gets me to this behavior, but it doesn't. Instead, I see:
Starting test.
This is a test. 1
This is a test. 2
This is a test. 3
[...]
This is a test. 99
Test done!

For background, if you use View Package File from the command palette to view Default/exec.py you can examine the code that's being used to execute programs and display their output into the panel.
As a short synopsis, the program is executed in a background thread, and the output of stdout and stderr are read from using this code:
def read_fileno(self, fileno, execute_finished):
decoder_cls = codecs.getincrementaldecoder(self.listener.encoding)
decoder = decoder_cls('replace')
while True:
data = decoder.decode(os.read(fileno, 2**16))
if len(data) > 0:
if self.listener:
self.listener.on_data(self, data)
else:
try:
os.close(fileno)
except OSError:
pass
if execute_finished and self.listener:
self.listener.on_finished(self)
break
That is, whenever they read any data, they invoke on_data() in the object that's listening for results, which in this case is the exec command itself. The implementation of that method looks like this:
def on_data(self, proc, data):
# Normalize newlines, Sublime Text always uses a single \n separator
# in memory.
data = data.replace('\r\n', '\n').replace('\r', '\n')
self.append_string(proc, data)
Internally Sublime uses \n as the line separator, so on windows \r\n is interpreted as \n, and on MacOS \r is interpreted as \n as well.
So on the one hand this doesn't do what you expect because your \r is effectively treated the same as \n is.
Aside of that, if you trace the call to append_string() through the guard code that keeps it thread safe and allows it to interact with the Sublime core, it ends up in a line that looks like this:
self.output_view.run_command(
'append',
{'characters': characters, 'force': True, 'scroll_to_end': True})
That is, whatever characters you provide, they're directly appended to the output buffer without regard to what they might contain.
If you were to provide a custom build target (a somewhat advanced plugin but still possible) this behaviour could be modified, although it might be tricky to get it to work on MacOS since there would be no way to distinguish what \r means on that platform.

The code displays correctly in a normal command line window. The problem is with Sublime's output window, which doesn't seem to be displaying the carriage return correctly (it replaces it with a newline).

Actually you need to add '\r' at the beginning of the string.
print('Starting test.')
for i in range(1,100):
print('\rThis is a test.'+str(i), end="")
print('\nTest done!')
Output
Starting test.
This is a test.99
Test done!

Python - program stops to read standard input

I'm writting a python script which receive several parameters on his standard input.
I use the raw_input() function which works for the firsts parameters but totally freeze when I call it inside a while loop.
Here is my code:
def launch_trade(logger):
Kerviel = Trader()
param = raw_input() #Works fine
Kerviel.Capital = float(param)
param = raw_input() #Works fine
Kerviel.NbDays = int(param)
param = raw_input() #Works Fine
while (param != '--end--'):
Kerviel.action(float(param), logger)
Kerviel.Cours.append(param)
param = raw_input() #Here it infinite wait
Actually this program works when I send all parameters myself in my console. But it is supposed to be called by a php script which sends it parameters on his stdin.
Why does this last raw_input() doesn't work when parameters are sent by a php script ?
Thanks for your answers and sorry for bad english.

Each input or raw_input call waits for a line ending with '\n'. A live user hits keys ending with the Enter key. A program has to send strings to stdout, connected to stdin, ending with '\n'. So the problem must be that the php program is not sending enough, or that not eveything it sends gets to your python program. If the latter, it could be that the php program needs to 'flush' stdout to push the last string to python. When a file is closed, flush is automatic, but stdout would not ordinarily be closed.