Develop Reference

When using Popen how can I read from stdout?

I'm trying to use Popen to start a shell process, execute commands, print the commands to output, and print the output of the commands (if any). Writing to stdin is working fine, but trying to read from stdout causes the program to freeze.
Here's my program:
with Popen(["/bin/sh"], stdin=PIPE, stdout=PIPE, stderr=STDOUT, text=True, bufsize=0) as proc:
with open("script.txt") as scriptFile:
for line in scriptFile.readlines():
proc.stdin.write(line)
print(f"$ {line.strip()}")
# Reading from proc.stdout locks the program.
for outputLine in proc.stdout.readlines():
print(outputLine)
And here's a simplified script.txt. (The real one creates a git repository and illustrates the use of various git commands.)
mkdir project
cd project
echo "This is line 1.\nThis is line 2." > text1.txt
cat text1.txt
If I don't try to read from stdout all of the commands in script.txt (and my real version with multiple git commands) work fine.
Is there any way to get the output of commands from stdout interspersed with writing them to stdin?

Open shell environment and run a series of command using Python

Seems duplicate of Using Python to open a shell environment, run a command and exit environment. I want to run the ulimit command in the shell environment in Redhat. Procedure: Open shell environment, run ulimit commands on shell, get the result and exit the shell environmnet. Referencing the above solution, I tried:
from subprocess import Popen, PIPE
def read_limit():
p = Popen('sh', stdin=PIPE)
file_size = p.communicate('ulimit -n')
open_files = p.communicate('ulimit -f')
file_locks = p.communicate('ulimit -x')
return file_size, open_files, file_locks
But I got error: ValueError: I/O operation on closed file.

The documentation for communicate() says:
send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
After that, the pipe will be closed.
You can use
p.stdin.write("something")
p.stdin.flush()
result = p.stdout.readline()
for your three commands and then
p.stdin.close()
p.wait()
at the end to terminate it

How to get stdout and stderr from a tmux session?

I am writing a sample python program in linux system. I am using tmux to create a session and execute another script within the tmux-session. I would like to get the stdout and stderr out of the tmux session to the parent script but that somehow does not work.
Code snippet:
cmd = "tmux new-session -d -s 'test' '/my_dir/fibonacci.py __name:=fibonacci_0'"
proc = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
(stdout, stderr) = proc.communicate()
print(stderr)
I have came across answers to use show-buffer and pipe-pane. But that did not help. Maybe I need to modify the tmux command itself.

Thank you for your support. After digging a bit, I came up with a workaround. I am just adding it here for someone with similar needs.
What I have done is created a named pipe, redirect the output of tmux session to named pipe and then finally read from it.
# this attach if a session exists or creates one, then exits from the session
call("tmux new-session -A -s test \; detach", shell=True)
# to avoid conflict, remove existing named pipe and then create named pipe
call("rm -f /tmp/mypipe && mkfifo /tmp/mypipe && tmux pipe-pane -t test -o 'cat > /tmp/mypipe'", shell=True)
# feed the pipe to the stdout and stderr
poc = Popen(['cat', '/tmp/mypipe'], stdout=PIPE, stderr=PIPE)
# finally execute the command in tmux session
Popen(['tmux', 'send-keys', '-t', '/my_dir/fibonacci.py', 'C-m'])
(stdout, stderr) = proc.communicate()
print(stderr)
Hope this is helpful.

TLDR: tmux is like sandboxed cmd, you can't get in the session and reach sdtout or stderr
tmux creates a dispatched process. stdout and stderr is tmux would be the text messages that tmux provies, not the commands you run in a tmux session.
So you can't get your commands' output from a dispatched process.
In order to get the stdout and stderr, you have to change how fibonacci.py dumps text. Python logging framework can help you in this situation. You can write all stdout to stdout.txt and get content of that file.

Snort command run through python script

I am trying to read the snort alert in the console for one of my project.
What I did is, I wrote a following code to run the snort to listen to my interface.
import subprocess
command = 'snort -l /var/log/snort -c /etc/snort/snort.conf -A console -i s1-eth1'
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
process.wait()
print process.returncode
I run the above script through another python script as below
#!/usr/bin/env python
import os
import sys
from subprocess import Popen, PIPE, STDOUT
script_path = os.path.join(os.getcwd() +'/', 'comm.py')
p = Popen([sys.executable, '-u', script_path],
stdout=PIPE, stderr=STDOUT, bufsize=1)
with p.stdout:
for line in iter(p.stdout.readline, b''):
print line,
p.wait()
The output of the script takes me where snort listens. but when doing experiment that triggers the rule of snort file to alert, its not printing the output in terminal where i ran the python script.
when I run the snort command in the normal terminal the alert message prints all fine.
does any know any work around for this.
Thanks in advance.

How to terminate a Python subprocess started with shell=True if the command being run is setsid

I'm starting a Python subprocess with this command:
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True, start_new_session=True)
As per this answer, start_new_session=True (equvalent to preexec_fn=os.setsid) allows me to terminate the process with os.killpg(p.pid, signal.SIGTERM).
However, if command happens to be something like setsid sleep 10000, the process isn't terminated. In Python, how can I terminate subprocesses like these as well?