Develop Reference

How to run pytest from a python script subprocess

I need to run pytest test_start.py and keep running the program.
My cod:
import subprocess
subprocess.run(['pytest', r'C:\Python\test_start.py'], shell=True)
print('hello')
But when I run the script, pytest starts executing and print waits for it to finish.
How can I run py test and go ahead to execute the script?
UPD: When i used subprocess.Popen - I see that print has been executed, but I don't see the execution of pytest

subprocess.run specifically waits for the process to finish. If you don't want to wait, use subprocess.Popen

I solved this problem by simply adding
time.sleep(5) after subprocess
subprocess.Popen(['pytest', r'E:\Parser\Python\test_start.py'], shell=True)
time.sleep(5)
print('hello')
Apparently pytest just didn't have time to start)

Waiting for GNU Parallel processes to finish when using & and spawning via Python

In my current application, I have a Python 2.7 script called main.py that launches another Python 2.7 script called calculator.py using GNU Parallel like the following:
os.system("seq 10000 | parallel -N0 -j 50 nohup python calculator.py &")
print "Done"
This works pretty well, with one exception: I need to resume executing other commands in main.py (that is, after the os.system call, e.g. the print "Done" line) just after all the 10000 instances spawned with GNU Parallel finish running.
Is there a proper way to do that? Solutions with os.spawn and Python 2.7 subprocess are both welcome, but using GNU Parallel is absolutely mandatory.
EDIT: Here are my requirements:
1) it is crucial to me that the many instances of calculator.py that are spawned keep running if the terminal closes (hence the nohup)
2) I need it to not block current terminal session (hence the &)
3) I need it to print "Done" in the example above gets executed only after the 10000 jobs finish
If achieving all above at the same time is not possible, I think I could then manually keep a log of all launched processes and then manually force the rest of the code "main.py" code to continue after all those processes end. This, of course, is a cumbersome last-resource option.

Run python script from another python script but not as an child process

Is it possible to run a python script from another python script without wating for termination.
Parent process will terminate immediately after creation of child process.
I tried:
subprocess.Popen([sys.executable, "main.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
and also:
os.system(...)

If you know that the other Python script has a main method, you could simply in you code call that other script:
import main
...
exit(main.main())
But here the other script executes in the context of calling script. If you want to avoid it, you could use the os.exec... functions, by launching a new Python interpretor:
import os
...
os.execl(sys.executable, "python", 'main.py')
The exec family functions will replace (under Unix-Linux) the current Python interpretor with a new one.

You can just add & to start script in background:
import os
os.system('/path/to/script.sh &')
exit()
In this case launched shell script will continue working even after main Python script exits.
But keep in mind that it can cause zombie processes appearance in our system.

Running one process in parallel linux

I need to make sure to run two processes (python scripts) almost at the same time. But I want the program to continue until one of them is finished. I am running these processes from a C++ program using system.
Is this the right way to run script1 and script2 at the same time and continue just after script2 is finished?
python ./script1.py & python ./script2.py
Thank you!

Your snippet won't work because it will continue as soon as script2 finishes. script1 may still be working at the background.
If you are using bash shell you can do the following:
python ./script1.py &
PID1=$!
python ./script2.py
wait $PID1
$! has the process id of the previously background command. So we run script1 in the background, then we run script2 until completion, and then we wait for script1 to finish (if not already finished).

python: subprocess calling a script which runs a background process hanging

I have an issue using subprocess.call when calling scripts which in turn run background processes.
I am calling a bash script from a python script.
python 2.7.3.
#!/bin/python
from subprocess import call
.
.
call(["run_exp",file_name])
print "exp complete!"
.
.
run_exp is a bash script which runs a process in the background.
#!/bin/bash
.
.
run_task auto_output 2>/dev/null &
.
.
echo "run_exp finished!"
The run task command is another bash script. This is always completed by the time run_exp has finished.
Running run_exp from command line I see expected behaviour and all processes finish completed.
An issue arises when i call the run_exp command using python call. When using call I see the output "run_exp finished!" but never "exp complete!". If I remove the run_task operation (and associated code with its operation in run_exp) from run_exp, the call command runs to completion as expected. This leads me to believe there is an issue using call when the script called runs processes in the background.
Can anyone shed any light on why this might occur.
Thanks!

The output of the background scripts is still going to the same file descriptor as the child script. That's why the parent script is still waiting for it to finish.
You should close all file descriptors in your background scripts if you want to demonize them:
(run_task auto_output >/dev/null 2>&1) &
(The parentheses do this in a subshell which I sometimes found was needed.)
Also it can help to wait explicitly at the end of your child script for the background process:
run_task auto_output 2>/dev/null & backgroundPid=$!
...
echo "run_exp finished!"
wait "$backgroundPid"
And maybe combining both strategies should also tried if both fail alone.