I am using subprocess.call to run an executable from a python script
works fine under linux, the stdout from the executable is printed to the terminal as desired
when running the same subprocess.call on windows (7) the command executes successfully but i get no stdout to the cmd window
my subprocess.call looks like this:
call([self.test_dict["exe"],
"-p",
self.test_dict["pulse_input"],
"-l",
self.test_dict["library_input"],
"-c",
self.test_dict["config"],
"-r",
self.test_dict["pulse_output"],
"-s",
self.test_dict["stream_output"],
"-u",
self.test_dict["library_output"],
track_output_flag,
track_output_string,
socket_output_flag,
socket_output_string,
"-f",
self.test_dict["frame_size"],
"-d",
self.test_dict["ddi_enable"],
self.test_dict["verbose"],
])
how can i get the executables stdout displayed in the windows cmd window, i.e. how can i get the same behaviour i observe under linux?
I can now see some output using the Popen approach suggested below. my Popen call looks like this:
print(subprocess.Popen([self.test_dict["exe"],
"-p",
self.test_dict["pulse_input"],
"-l",
self.test_dict["library_input"],
"-c",
self.test_dict["config"],
"-r",
self.test_dict["pulse_output"],
"-s",
self.test_dict["stream_output"],
"-u",
self.test_dict["library_output"],
track_output_flag,
track_output_string,
socket_output_flag,
socket_output_string,
"-f",
self.test_dict["frame_size"],
"-d",
self.test_dict["ddi_enable"],
self.test_dict["verbose"]],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate())
On linux:
(' Test Harness - Version 2.2.2\n \n\nFrame 1 of 5: 1000 of 4032
pdws Rx: 0.206, G2: 32.337 (kpps) lib: ddi_n 0 residue: 11 \nFrame 2
of 5: 2000 of 4032 pdws Rx: 0.174, G2: 35.396 (kpps) lib: ddi_n 0
residue: 18 \nFrame 3 of 5: 3000 of 4032 pdws Rx: 0.197, G2: 31.913
(kpps) lib: ddi_n 0 residue: 25 \nFrame 4 of 5: 4000 of 4032 pdws Rx:
0.139, G2: 33.908 (kpps) lib: ddi_n 0 residue: 13 \nFrame 5 of 5: 4032 of 4032 pdws Rx: 2.581, G2: 20.164 (kpps) lib: ddi_n 0 residue: 0
\nlibrary status: lib.n: 52 lib.ddi_n: 0 lib.orig_n: 52\nelapsed time:
0.300253 (s)\n\n', '')
On Windows:
('', '')
the first bit of the linux output is what i would like to see printed to stdout on windows
i was mistaken that adding stdout=subprocess.PIPE was fixing the problem on windows, it wasn't - apologies for the confusion
Related
Im translating a model done on weka to python-weka-wrapper3 and i dont know how to an evaluator and search options on attributeselectedclassifier.
This is the model on weka:
weka.classifiers.meta.AttributeSelectedClassifier -E "weka.attributeSelection.CfsSubsetEval -P 1 -E 1" -S "weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1" -W weka.classifiers.meta.MultiSearch -- -E FM -search "weka.core.setupgenerator.MathParameter -property classifier.classifier.classifier.numOfBoostingIterations -min 5.0 -max 50.0 -step 1.0 -base 10.0 -expression I" -class-label 1 -algorithm "weka.classifiers.meta.multisearch.DefaultSearch -sample-size 100.0 -initial-folds 2 -subsequent-folds 10 -initial-test-set . -subsequent-test-set . -num-slots 1" -log-file /Applications/weka-3-8-3 -S 1 -W weka.classifiers.meta.Bagging -- -P 100 -S 1 -num-slots 1 -I 100 -W weka.classifiers.meta.FilteredClassifier -- -F "weka.filters.supervised.instance.SMOTE -C 0 -K 3 -P 250.0 -S 1" -S 1 -W weka.classifiers.meta.CostSensitiveClassifier -- -cost-matrix "[0.0 1.0; 1.0 0.0]" -S 1 -W weka.classifiers.trees.ADTree -- -B 10 -E -3 -S 1
and I have this right now:
base = Classifier(classname="weka.classifiers.trees.ADTree", options=["-B", "10", "-E", "-3", "-S", "1"])
cls = SingleClassifierEnhancer(classname="weka.classifiers.meta.CostSensitiveClassifier",
options =["-cost-matrix", "[0.0 1.0; 1.0 0.0]", "-S", "1"])
cls.classifier = base
smote = Filter(classname="weka.filters.supervised.instance.SMOTE", options=["-C", "0", "-K", "3", "-P", "250.0", "-S", "1"])
fc = FilteredClassifier()
fc.filter = smote
fc.classifier = cls
bagging_cls = Classifier(classname="weka.classifiers.meta.Bagging",
options=["-P", "100", "-S", "1", "-num-slots", "1", "-I", "100"])
bagging_cls.classifier = fc
multisearch_cls = MultiSearch(
options = ["-S", "1"])
multisearch_cls.evaluation = "FM"
multisearch_cls.log_file = "/home/pablo/Escritorio/TFG/OUTPUT.txt"
multisearch_cls.search = ["-sample-size", "100", "-initial-folds", "2", "-subsequent-folds", "10",
"-initial-test-set", ".", "-subsequent-test-set", ".", "-num-slots", "1"]
mparam = MathParameter()
mparam.prop = "numOfBoostingIterations"
mparam.minimum = 5.0
mparam.maximum = 50.0
mparam.step = 1.0
mparam.base = 10.0
mparam.expression = "I"
multisearch_cls.parameters = [mparam]
multisearch_cls.classifier = bagging_cls
AttS_cls = AttributeSelectedClassifier()
AttS_cls.evaluator = "weka.attributeSelection.CfsSubsetEval -P 1 -E 1"
AttS_cls.search = "weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1"
AttS_cls.classifier = multisearch_cls
train, test = data_modelos_1_2.train_test_split(70.0, Random(1))
AttS_cls.build_classifier(train)
evl = Evaluation(train)
evl.crossvalidate_model(AttS_cls, test, 10, Random(1))
print(AttS_cls)
#graph.plot_dot_graph(AttS_cls.graph)
print("")
print("=== Setup ===")
print("Classifier: " + AttS_cls.to_commandline())
print("Dataset: ")
print(test.relationname)
print("")
print(evl.summary("=== " + str(10) + " -fold Cross-Validation ==="))
print(evl.class_details())
plcls.plot_roc(evl, class_index=[0, 1], wait=True)
but when I do
AttS_cls.evaluator = "weka.attributeSelection.CfsSubsetEval -P 1 -E 1"
AttS_cls.search = "weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1"
it reach me this error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/tmp/ipykernel_40724/2750622902.py in <module>
30
31 AttS_cls = AttributeSelectedClassifier()
---> 32 AttS_cls.search = "weka.attributeSelection.GreedyStepwise"
33 AttS_cls.classifier = multisearch_cls
34
/usr/local/lib/python3.8/dist-packages/weka/classifiers.py in search(self, search)
435 :type search: ASSearch
436 """
--> 437 javabridge.call(self.jobject, "setSearch", "(Lweka/attributeSelection/ASSearch;)V", search.jobject)
438
439
AttributeError: 'str' object has no attribute 'jobject'
I understand that I have to set them as objects because it raise this error because i try to set them as strings but I dont know how.
You need to instantiate ASSearch and ASEvaluation objects. If you have command-lines, you can use the from_commandline helper method like this:
from weka.core.classes import from_commandline, get_classname
from weka.attribute_selection import ASSearch
from weka.attribute_selection import ASEvaluation
search = from_commandline('weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1', classname=get_classname(ASSearch))
evaluation = from_commandline('weka.attributeSelection.CfsSubsetEval -P 1 -E 1', classname=get_classname(ASEvaluation))
The second argument of the from_commandline method is the classname of the wrapper that you want to use instead of OptionHandler. For simplicity, I import the correct wrappers and then use the get_classname method to return the dot notation of the wrapper's class. That way I can avoid accidental typos in the classname strings.
Also, by using single quotes, you won't have to worry about Weka's quotes in the command-lines and you can just use the Weka command-line string verbatim.
You can also use the same approach for instantiating the AttributeSelectedClassifier wrapper itself, without having to go through instantiating search and evaluation separately:
from weka.core.classes import from_commandline, get_classname
from weka.classifiers import AttributeSelectedClassifier
cls = from_commandline('weka.classifiers.meta.AttributeSelectedClassifier -E "weka.attributeSelection.CfsSubsetEval -P 1 -E 1" -S "weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1" -W weka.classifiers.meta.MultiSearch -- -E FM -search "weka.core.setupgenerator.MathParameter -property classifier.classifier.classifier.numOfBoostingIterations -min 5.0 -max 50.0 -step 1.0 -base 10.0 -expression I" -class-label 1 -algorithm "weka.classifiers.meta.multisearch.DefaultSearch -sample-size 100.0 -initial-folds 2 -subsequent-folds 10 -initial-test-set . -subsequent-test-set . -num-slots 1" -log-file /Applications/weka-3-8-3 -S 1 -W weka.classifiers.meta.Bagging -- -P 100 -S 1 -num-slots 1 -I 100 -W weka.classifiers.meta.FilteredClassifier -- -F "weka.filters.supervised.instance.SMOTE -C 0 -K 3 -P 250.0 -S 1" -S 1 -W weka.classifiers.meta.CostSensitiveClassifier -- -cost-matrix "[0.0 1.0; 1.0 0.0]" -S 1 -W weka.classifiers.trees.ADTree -- -B 10 -E -3 -S 1', get_classname(AttributeSelectedClassifier))
I am working on a python2.7 project. I set a process pool with 10 processes. My code structure is:
pool = Pool(10)
pool.map(ProcessJson, jsonFiles)
def ProcessJson(jsonpath):
# doing something and get (int)numUrls
for idx in xrange(numUrls):
flag = DownloadVideo(paras).run()
if flag == 0:
continue
class DownloadVideo():
def __init__(para):
# init
def run(self):
try:
videopath = os.path.join(folder, sname.encode("utf-8"))
try:
cmd = "wget -q -c --limit-rate=1M --tries=3 -T10 -P %s --output-document=%s \"%s\"" % (folder, videopath, url)
ret = os.system(cmd)
if (ret >> 8) != 0:
logger.error("cmd error---" + str(ret >> 8) + "---" + cmd + "\n")
return 0
except Exception as e:
logger.error("python system cmd error---", e)
return 0
except Exception as e1:
logger.error("exception with downloading---", e1)
return 0
After sometime, the process are all sleep. I use sudo strace -p pid and lsof -p pid to find the problem. What I get are showing:
sudo strace -p 27723
Process 27723 attached
wait4(-1,
Then i found its child process 27724
sudo strace -p 27724
Process 27724 attached
read(6,
cat wchan 27724
do_wait_data
:/proc/27724$ ls -l ./fd
l-wx------ 1 cwz domain^users 64 7月 12
16:20 3 -> Documents/tools/log/__main__-1.log
lr-x------ 1 cwz domain^users 64 7月 12
16:20 4 -> pipe:[8180487]
l-wx------ 1 cwz domain^users 64 7月 12 16:20 5 ->
chenweizhao/Documents/tools/GoogleImages-
1/Tim_Faraday/xXP4smHFrfLw9M.jpg
lrwx------ 1 cwz domain^users 64 7月 12 16:20 6 -> socket:
[8286859]
l-wx------ 1 cwz domain^users 64 7月 12
16:20 7 -> pipe:[8180490]
lr-x------ 1 cwz domain^users 64 7月 12
16:20 8 -> /dev/null
and
lsof -p 27724
wget 27724 cwz 3w REG 8,2 427352 5786641
chenweizhao/tools/log/__main__-1.log
wget 27724 cwz 4r FIFO 0,8 0t0 8180487 pipe
wget 27724 cwz 5w REG 8,2 24404 9971506
chenweizhao/tools/GoogleImages-1/Tim_Faraday/xXP4smHFrfLw9M.jpg
wget 27724 cwz 6u IPv4 8286859 0t0
TCP myIP:47496->ec2-13-56-87-172.us-west-1.compute.amazonaws.com:https
(ESTABLISHED)
wget 27724 cwz 7w FIFO 0,8 0t0
8180490 pipe
wget 27724 cwz 8r CHR 1,3 0t0 1029
/dev/null
How can I solve the problem? Thank you very much!
I'm trying to work with opts but cannot get it to work in other PC because arguments it's always empty. Below is my code.
import getopt
import sys
try:
print getopt.getopt(sys.argv[1:], "f::c::")
opts, args = getopt.getopt(sys.argv[1:], "f::c::")
except getopt.GetoptError, err:
# print help information and exit:
print str(err) # will print something like "option -a not recognized"
sys.exit(2)
print opts
print args
funcion = None
tipo = None
for opt, arg in opts:
if opt in ('-f'):
funcion = arg
if opt in ('-c'):
tipo = arg
print funcion
print tipo
usage test:
python test.py –f import_dbs –c 1
PC A result:
([('-f', 'imports'), ('-c', '1')], [])
[('-f', 'imports'), ('-c', '1')]
[]
imports
1
PC B result:
([], ['\x96f', 'import_dbs', '\x96c', '1'])
[]
['\x96f', 'import_dbs', '\x96c', '1']
None
None
the problem is with your cli command, not your code. you have dashes (or some sort of unicode) instead of hyphens
$ python test.py –f import_dbs –c 1
None
None
$ python test.py -f import_dbs –c 1
import_dbs
None
$ python test.py -f import_dbs -c 1
import_dbs
1
$ echo "python test.py –f import_dbs –c 1" | od -c
0000000 p y t h o n t e s t . p y –
0000020 ** ** f i m p o r t _ d b s –
0000040 ** ** c 1 \n
0000046
$ echo "python test.py -f import_dbs -c 1" | od -c
0000000 p y t h o n t e s t . p y -
0000020 f i m p o r t _ d b s - c
0000040 1 \n
0000042
probably caused by cutting and pasting or weird keyboard mapping.
I am trying to concatenate a couple of files together and add a header.
import subprocess
outpath = "output.tab"
with open( outpath, "w" ) as outf :
"write a header"
if header is True:
p1 = subprocess.Popen(["head", "-n1", files[-1] ], stdout= outf, )
if type(header) is str:
p1 = subprocess.Popen(["head", "-n1", header ], stdout= outf,)
for fl in files:
print( fl )
p1 = subprocess.Popen(["tail", "-n+2", fl], stdout= outf, )
for some reason some files (fl) are printed only partially and next file starts amid of a string from a previous file:
awk '{print NF}' output.tab | uniq -c
108 11
1 14
69 11
1 10
35 11
1 16
250 11
1 16
Is there any way to fix it in Python?
An example of messed up lines:
$tail -n+108 output.tab | head -n1
CENPA chr2 27008881.0 2701ABCD3 chr1 94883932.0 94944260.0 0.0316227766017 0.260698861451 0.277741584016 0.302602378581 0.4352790705329718 56 16
$grep -n A1 'CENPA' file1.tab
109:CENPA chr2 27008881.0 27017455.0 1.0 0.417081004817 0.0829327365256 0.545205239241 0.7196619496326693 95 3
110-CENPO chr2 25016174.0 25045245.0 1000.0 0.151090930896 -0.0083671250883 0.50882773122 0.0876177652747541 82 0
$grep -n 'ABCD3' file2.tab
2:ABCD3 chr1 94883932.0 94944260.0 0.0316227766017 0.260698861451 0.277741584016 0.302602378581 0.4352790705329718 56 16
I think the issue here is that subprocess.Popen() runs asynchronously by default, and you seem to want it to run synchronously. So really, all of your head and tail commands are running at the same time, directing into the output file.
To fix this, you probably want to just add .wait():
import subprocess
outpath = "output.tab"
with open( outpath, "w" ) as outf :
"write a header"
if header is True:
p1 = subprocess.Popen(["head", "-n1", files[-1] ], stdout= outf, )
p1.wait() # Pauses the script until the command finishes
if type(header) is str:
p1 = subprocess.Popen(["head", "-n1", header ], stdout= outf,)
p1.wait()
for fl in files:
print( fl )
p1 = subprocess.Popen(["tail", "-n+2", fl], stdout= outf, )
p1.wait()
from itertools import permutations
from itertools import combinations
import itertools
import random
import os
def split(arr, size):
arrs = []
while len(arr) > size:
pice = arr[:size]
arrs.append(pice)
arr = arr[size:]
arrs.append(arr)
return arrs
print("Enter the GCAP utility command:")
N=raw_input()
A= [N]
A1=[]
S7=[]
S6=[]
X=[]
X1=[]
m1=[]
G=0
A1=0
G1=[]
g1=[]
u=[]
def HELP():
print('SYNTAX : ABC')
def comp():
for x11, y11 in zip(Y, m1[G-1]):
if x11 == y11:
X1 = sorted(m1[G-1])
A1= X1
return A1
def my_containsAll(str1, Y):
for c in Y:
if c not in str1: return 0;
return 1;
def mysplit(s):
head = s.rstrip('0123456789')
tail = s[len(head):]
return head, tail
def imcon(u,U3):
U1= ['imconfig','-i']
U2=U1+u+U3
return U2
if N == 'GCOA.01': ## for first GCOA.01 command
print("Enter the command in this format ")
S=raw_input()
print ("Enter the user interface for example : ")
U= raw_input()
for s in ['GCOA.01']:
g = mysplit(s)
g1.append('-c')
g1.append(g[1])
u.append(U)
#print(g1)
if S == "GCOA.01": ## if no other parameter typed
U4= imcon(u,g1)
ss= ' '.join(U4)
#print(ss)
else:
S=S.split()
S1 =[i.split('\t', 1)[0] for i in S] ## spliting the command into list
#print(S1)
G1= S1[0]
#print(G1)
for s in ['GCOA.01']:
g = mysplit(s)
#print(g[1])
S11=S1[0]
S5=[S11]
del S1[0]
for x in S1: ## separate the parameters from the whole command
print(x)
X.append(x[:2])
Y=list(X)
S2=['-c','-a','-h']
S12 =sorted(S2)
str1 = ''.join(S12)
#print(str1)
A2= my_containsAll(str1, Y)
if A2 == 1:
#print(S2)
S3=[]
for i in range(1, len(S2)+1): ## creating the permutation and combinations of the parameters
S3.extend(itertools.combinations(S2, i))
if i >= 1:
for k in S3:
for c in permutations(k):
S6.append(c)
for l in S6:
if l not in S7:
S7.append(l)
# print(S7)
for m in S7: ## creating the list of the different possible combination of P&C
m=list(m)
m1.append(m)
#print (m1)
for i1 in m1: ## for finding the index of the parameters given by the user from the P&C list of parameters
G=G+1
if Y == i1:
#print (G)
break;
g1=g1+S1
#print(g1)
if Y == m1[0]:
R=imcon(u,g1)
ss=' '.join(R)
print(ss) ## this print are basically command which is to direct on terminal window
elif Y == m1[1]:
R=imcon(u,g1)
ss=' '.join(R)
print(ss)
else:
HELP()
Output :
imconfig -i th0 -c 01 -h -a
example: (in command prompt)
c:/python34> python ssm9.py
imconfig -i th0 -c 01 - h -a
c:/python34> imconfig -i th0 -c 01 -h -a
/// i want something like that means the above printed value redirect here. but in linux terminal window
I wrote a sample code like this
vivek#vivekUbuntu:~$ cat sample.py
#! /usr/bin/python
import socket
hostname = socket.gethostname()
def getPingCommand(host):
return "ping %s" %host
print getPingCommand(hostname)
And then executed like this:
vivek#vivekUbuntu:~$ `python sample.py`
PING vivekUbuntu (127.0.1.1) 56(84) bytes of data.
64 bytes from vivekUbuntu (127.0.1.1): icmp_seq=1 ttl=64 time=0.020 ms
64 bytes from vivekUbuntu (127.0.1.1): icmp_seq=2 ttl=64 time=0.027 ms
64 bytes from vivekUbuntu (127.0.1.1): icmp_seq=3 ttl=64 time=0.050 ms
64 bytes from vivekUbuntu (127.0.1.1): icmp_seq=4 ttl=64 time=0.050 ms
64 bytes from vivekUbuntu (127.0.1.1): icmp_seq=5 ttl=64 time=0.049 ms
^C
--- vivekUbuntu ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 0.020/0.039/0.050/0.013 ms
vivek#vivekUbuntu:~$
Even the option mentioned by S0III0s is also correct. It works for me.
`python ssm9.py`
or
$(python ssm9.py)
or
eval `python ssm9.py`
or
eval $(python ssm9.py)
Should all work.
(You can try it like this:)
> $(echo "echo test")
test