Develop Reference

Subprocesses not running with pytest Error Processing Test

I am currently converting my unittest setup over to pytest, everything is working however on certain tests I run a command line process in order to load code into the hardware I have attached via USB. this process works fine with unittest however when using pytest or nose2 I get the response
------------------------------------------------ Captured stderr call -------------------------------------------------
Error processing Test
this happens just when my process begins to run? I get no error message am an unsure why one is not output? The command is fine as it runs on cmd and unittest is there something I am missing in order to make it work with pytest?
For reference my class I am running
class LoadCode():
def __init__(self, comport):
''' Constructor
'''
self.filename = None
self.code_comport = comport
self.code_loaded = False
self.logger = logging.getLogger(__name__)
def set_code_filename(self, new_file_name):
''' Sets the filename parameter for loading the code
'''
if (self.filename != new_file_name):
self.filename = new_file_name
self.logger.info("Setting code File to " + self.filename)
self.code_loaded = False
else:
self.logger.info("Code File Name Is Already Set !")
def write_code(self):
REBOOT_TIME = 50 #approximatly 50 seconds if enough for a reboot after loading boot and main and enough time for
SUCCESSFUL_RETURNCODE = 0 # 0 is a successful return code for subprocess
if(self.filename != None and self.code_comport != None):
#set up command line to run
command = <<COMMAND>>
self.logger.info("Running: " + command)
#run command line as subprocess (thread will wait for command line to finish)
load_code = subprocess.run(command)
#successful returncode = 0 anything else means an error has occured during subprocess
subprocess.CompletedProcess(args=[command], returncode = SUCCESSFUL_RETURNCODE)
if (load_code.returncode == SUCCESSFUL_RETURNCODE ):
self.code_loaded = True
self.logger.info(self.filename)
time.sleep(REBOOT_TIME) #Allow reboot
else:
raise AssertionError("ERROR: No code File Set/No Comport Set")
self.is_code_loaded()
def is_code_loaded(self):
'''check the bool of code to ensure it has successfully ran
'''
if self.code_loaded == False:
Print("Failed")
raise AssertionError("Code Was Not Loaded ..")
else:
print("WORKED")

subprocess.CompletedProcess(args=[command], returncode = SUCCESSFUL_RETURNCODE)
this line of code is not needed as it returns from subproccess.run(). thanks #Masklinn for pointing that out.
details: https://python.readthedocs.io/en/latest/library/subprocess.html?highlight=CompletedProcess
The path that was generated contained spaces in the middle and did not have one at the start which is why the command did not run and the error was returned! now my subprocess work perfectly fine with pytest and nose2 ! :)

Not able to write output of python script to text file in Windows

I am using a python script to extract the tweets from twitter. I want to write the output into a text file.
When I run the .py file from Anaconda command prompt, it shows me the output in the command prompt. But when I try to write the same output to a file, it doesn't write anything.
C:\Users\akjain>python c:\Akash\TweetExtract.py >> twitter_data.txt
I have tried to open the Ananconda as an Administrator too. Also created the text file in the same folder where I have python script before running the script.
I also tried the below code but this also did not work.
C:\windows\system32>python c:\Akash\GartnerTweetExtract.py > c:\Akash\twitter_data.txt
Edit:
Code to print the ouput to command prompt which is inside my python script is as follows:
#This is a basic listener that just prints received tweets to stdout.
class StdOutListener(StreamListener):
def on_data(self, data):
print (data)
return True
def on_error(self, status):
print (status)
Any help would be really appreciated.
Regards,
Akash

So my best guess is that you are facing permission issues. Why don't you copy the file to C:\Users\akjain> and run
python TweetExtract.py > your-file.txt

You can store your output in text file using following code snippet, this way you can also check what error it's throwing in case of failure.
class StdOutListener(StreamListener):
def on_data(self, data):
print (data)
with open('twitter_data.txt', 'w') as f:
f.write(data)
return True
def on_error(self, status):
print (status)

Redirecting Output From a Program to a File with Python: Specific Bug

I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')

I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code

Python cannot find the file path to write the output

I have the following static method in Python.
import subprocess
class ProcessUtility:
#staticmethod
def execute_command(url):
process = None
process = subprocess.check_output(["phantomas " + url + " --har=test.har"])
return process
It command basically writes the output to test.har file. I have created the test.har file in the same directory as that of the script and given it read,write and execute permissions.
Upon executing I get the error.
OSError: [Errno 2] No such file or directory
Any ideas why I keep getting this.

Most subprocess functions take a list of arguments as opposed to a string, if shell=False (the default). Try
process = subprocess.check_output(['phantomas', url, '--har=test.har'])

Run a .bat program in the background on Windows

I am trying to run a .bat file (which acts as a simulator) in a new window, so it must always be running in the background. I think that creating a new process is the only option that I have. Basically, I want my code to do something like this:
def startSim:
# open .bat file in a new window
os.system("startsim.bat")
# continue doing other stuff here
print("Simulator started")
I'm on Windows so I can't do os.fork.

Use subprocess.Popen (not tested on Windows, but should work).
import subprocess
def startSim():
child_process = subprocess.Popen("startsim.bat")
# Do your stuff here.
# You can terminate the child process after done.
child_process.terminate()
# You may want to give it some time to terminate before killing it.
time.sleep(1)
if child_process.returncode is None:
# It has not terminated. Kill it.
child_process.kill()
Edit: you could also use os.startfile (Windows only, not tested too).
import os
def startSim():
os.startfile("startsim.bat")
# Do your stuff here.

Looks like you want "os.spawn*", which seems to equate to os.fork, but for Windows.
Some searching turned up this example:
# File: os-spawn-example-3.py
import os
import string
if os.name in ("nt", "dos"):
exefile = ".exe"
else:
exefile = ""
def spawn(program, *args):
try:
# check if the os module provides a shortcut
return os.spawnvp(program, (program,) + args)
except AttributeError:
pass
try:
spawnv = os.spawnv
except AttributeError:
# assume it's unix
pid = os.fork()
if not pid:
os.execvp(program, (program,) + args)
return os.wait()[0]
else:
# got spawnv but no spawnp: go look for an executable
for path in string.split(os.environ["PATH"], os.pathsep):
file = os.path.join(path, program) + exefile
try:
return spawnv(os.P_WAIT, file, (file,) + args)
except os.error:
pass
raise IOError, "cannot find executable"
#
# try it out!
spawn("python", "hello.py")
print "goodbye"

On Windows, a background process is called a "service". Check this other question about how to create a Windows service with Python: Creating a python win32 service

import subprocess
proc = subprocess.Popen(['/path/script.bat'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
Using subprocess.Popen() will run the given .bat path ( or any other executable).
If you do wish to wait for the process to finish just add proc.wait():
proc.wait()