Develop Reference

How to run External collector from Scollector?

I am trying to run external sample.py script in /path-to-scollector/collectors/0 folder from scollector.
scollector.toml:
Host = "localhost:0"
ColDir="//path-to-scollector//collectors//"
BatchSize=500
DisableSelf=true
command to run scollector:
scollector-windows-amd64.exe -conf scollector.toml -p
But I am not getting the sample.py metrics in the output. It is expected to run continuosly and print output to cnosole. Also when I am running:
scollector-windows-amd64.exe -conf scollector.toml -l
my external collector is not listed.

In your scollector.toml, You should one line as below,
Filter=["sample.py "] .
in your sample.py, you need this line
#!/usr/bin/python

For running scollector on linux machine the above solution works well. But with windows its a bit tricky. Since scollector running on windows can only identify batch files. So we need to do a little extra work for windows.
create external collector :-
It can be written in any language python,java etc. It contains the main code to get the data and print to console.
Example my_external_collector.py
create a wrapper batch script :-
wrapper_external_collector.bat.
Trigger my_external_collector.py inside wrapper_external_collector.bat.
python path_to_external/my_external_collector.py
You can pass arguments to the script also.Only disadvantage is we need to maintain two scripts.

Execute batch file in different directory

I have a a file structure like the following (Windows):
D:\
dir_1\
batch_1.bat
dir_1a\
batch_2.bat
dir_2\
main.py
For the sake of this question, batch_1.bat simply calls batch_2.bat, and looks like:
cd dir_1a
start batch_2.bat %*
Opening batch_1.bat from a command prompt indeed opens batch_2.bat as it's supposed to, and from there on, everything is golden.
Now I want my Python file, D:\dir_2\main.py, to spawn a new process which starts batch_1.bat, which in turn should start batch_2.bat. So I figured the following Python code should work:
import subprocess
subprocess.Popen(['cd "D:/dir_1"', "start batch_1.bat"], shell=True)
This results in "The system cannot find the path specified" being printed to my Python console. (No error is raised, of course.) This is due to the first command. I get the same result even if I cut it down to:
subprocess.Popen(['cd "D:/"'], shell=True)
I also tried starting the batch file directly, like so:
subprocess.Popen("start D:/dir_1/batch_1.bat", shell=True)
For reasons that I don't entirely get, this seems to just open a windows command prompt, in dir_2.
If I forego the start part of this command, then my Python process is going to end up waiting for batch_1 to finish, which I don't want. But it does get a little further:
subprocess.Popen("D:/dir_1/batch_1.bat", shell=True)
This results in batch_1.bat successfully executing... in dir_2, the directory of the Python script, rather than the directory of batch_1.bat, which results in it not being able to find dir_1a\ and hence, batch_2.bat is not executed at all.
I am left highly confused. What am I doing wrong, and what should I be doing instead?

Your question is answered here: Python specify popen working directory via argument
In a nutshell, just pass an optional cwd argument to Popen:
subprocess.Popen(["batch_1.bat"], shell=True, cwd=r'd:\<your path>\dir1')

Django Python Script Doesn't Output Files

I have a script that executes the following code through a Django view:
# generate dag file
try:
commandString = [
'python',
os.path.join('/srv/nfsshare/transcode50', userFolder, directory, 'condor_execute.py')
]
subprocess.check_call(commandString,
stdout=open('/srv/nfsshare/transcode50/output2.txt', 'w'),
stderr=subprocess.STDOUT)
except Exception, e:
open('/tmp/test_exception.txt', 'w').write(str(e))
raise
What that is supposed to do is execute a Python file I have generated. If I run that exact command from the server's command prompt (i.e., python /srv/nfsshare/transcode50/cogden/algorithms/condor_execute.py), it works just fine and produces the needed files it's supposed to within that directory. However, if I run it from Django, it produces no error messages, produces the correct console output (a message that basically says "Created file blah at blah"), but produces no files whatsoever, when it normally generates two. I'm getting no permissions errors or anything and have ensured the directory is chmodded appropriately.

I suspect your process is never getting around to closing the file (as you can't do it explicitly as you have no reference to it - while run as a script - the interpreter would have stopped - but Django works different with loaded processes etc...) and it's not big enough to be flushed - try moving out your open.
with open('/srv/nfsshare/transcode50/output2.txt', 'w') as stdout:
subprocess.check_call(commandString, stdout=stdout, stderr=subprocess.STDOUT)
Also, make sure that when you make changes to make sure the Django server is restarted/similar to make sure code is reloaded and changes are taken into effect.

Runing bcdedit from python in Windows 2008 SP2

I do not know windows well, so that may explain my dilemma ...
I am trying to run bcdedit in Windows 2008R2 from Python 2.6.
My Python routine to run a command looks like this:
def run_program(cmd_str):
"""Run the specified command, returning its output as an array of lines"""
dprint("run_program(%s): entering" % cmd_str)
cmd_args = cmd_str.split()
subproc = subprocess.Popen(cmd_args, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
(outf, errf) = (subproc.stdout, subproc.stderr)
olines = outf.readlines()
elines = errf.readlines()
if Options.debug:
if elines:
dprint('Error output:')
for line in elines:
dprint(line.rstrip())
if olines:
dprint('Normal output:')
for line in olines:
dprint(line.rstrip())
errf.close()
outf.close()
res = subproc.wait()
dprint('wait result=', res)
return (res, olines)
I call this function thusly:
(res, o) = run_program('bcdedit /set {current} MSI forcedisable')
This command works when I type it from a cmd window, and it works when I put it in a batch file and run it from a command window (as Administrator, of course).
But when I run it from Python (as Administrator), Python claims it can't find the command, returning:
bcdedit is not recognized as an internal or external command,
operable program or batch file
Also, if I trying running my batch file from Python (which works from the command line), it also fails. I've also tried it with the full path to bcdedit, with the same results.
What is it about calling bcdedit from Python that makes it not found?
Note that I can call other EXE files from Python, so I have some level of confidence that my Python code is sane ... but who knows.
Any help would be most appreciated.

Windows 2008 R2 is 64-bit-only, yes? Python's a 32-bit process. When a 32-bit app runs something from C:\Windows\System32, Windows actually looks in C:\Windows\SysWOW64. Use C:\Windows\SysNative.

Perhaps the path to bcdedit.exe isn't in your system path when Python is running for some reason (a different user account, for example). You can find this out by printing:
os.environ.get("PATH")
It's semicolon-delimited, so os.environ.get("PATH").split(';') might be more useful.
I can't see any reason why it wouldn't be there, but just in case, you should be looking for C:\Windows\System32, where C is your Windows drive letter.

Check your PATH variable and see if C:\windows\system32 is there. (use set in DOS)

For some reason I experiment the same trouble from c#. If I list the files it was not here, but when I was looking from Explorer it was there. maybe it is some kind of protected file. To call bcdedit.exe, I manually copied it from system32 to my application folder and it worked. There is also another one in windows\winsxs folder. I can start it from my application, but I`m not sure it is the same path on all computers.
Hope it helps!

Python Popen difficulties: File not found

I'm trying to use python to run a program.
from subprocess import Popen
sa_proc = Popen(['C:\\sa\\sa.exe','--?'])
Running this small snippit gives the error:
WindowsError: [Error 2] The system cannot find the file specified
The program exists and I have copy and pasted directly from explorer the absolute path to the exe. I have tried other things and have found that if I put the EXE in the source folder with the python script and use './sa.exe' then it works. The only thing I can think of is that I'm running the python script (and python) from a separate partition (F:).
Any ideas?
Thanks

As the docs say, "On Windows: the Popen class uses CreateProcess() to execute the child program, which operates on strings. If args is a sequence, it will be converted to a string using the list2cmdline() method.". Maybe that method is messing things up, so why not try the simpler approach of:
sa_proc = Popen('C:\\sa\\sa.exe --?')
If this still fails, then: what's os.environ['COMSPEC'] just before you try this? What happens if you add , shell=True to Popen's arguments?
Edit: turns out apparently to be a case of simple mis-spellling, as 'sa' was actually the program spelled SpamAssassin -- double s twice -- and what the OP was writing was spamassasin -- one double s but a single one the second time.

You may not have permission to execute C:\sa\sa.exe. Have you tried running the program manually?