I have a script that executes the following code through a Django view:
# generate dag file
try:
commandString = [
'python',
os.path.join('/srv/nfsshare/transcode50', userFolder, directory, 'condor_execute.py')
]
subprocess.check_call(commandString,
stdout=open('/srv/nfsshare/transcode50/output2.txt', 'w'),
stderr=subprocess.STDOUT)
except Exception, e:
open('/tmp/test_exception.txt', 'w').write(str(e))
raise
What that is supposed to do is execute a Python file I have generated. If I run that exact command from the server's command prompt (i.e., python /srv/nfsshare/transcode50/cogden/algorithms/condor_execute.py), it works just fine and produces the needed files it's supposed to within that directory. However, if I run it from Django, it produces no error messages, produces the correct console output (a message that basically says "Created file blah at blah"), but produces no files whatsoever, when it normally generates two. I'm getting no permissions errors or anything and have ensured the directory is chmodded appropriately.
I suspect your process is never getting around to closing the file (as you can't do it explicitly as you have no reference to it - while run as a script - the interpreter would have stopped - but Django works different with loaded processes etc...) and it's not big enough to be flushed - try moving out your open.
with open('/srv/nfsshare/transcode50/output2.txt', 'w') as stdout:
subprocess.check_call(commandString, stdout=stdout, stderr=subprocess.STDOUT)
Also, make sure that when you make changes to make sure the Django server is restarted/similar to make sure code is reloaded and changes are taken into effect.
Related
There isn't a particular point in the script where it hangs (I've seen it getting stuck at random points in the script), so checking the logs didn't yield much insight. It doesn't even throw an exception or an error. It just keeps running while stuck.
I'm basically calling this python script from a powershell file (which later gets called by Task scheduler).
$python = "C:\Python34\python.exe"
$python_path = "C:\Source\main.py"
cd (split-path $python_path)
while($true)
{
& $python $python_path
}
Is there something I need to do to make sure it doesn't get stuck?
You Will have to do your path strings like this path ="c:\\test\\file"
Because it ignorer the first backslash
I have a a file structure like the following (Windows):
D:\
dir_1\
batch_1.bat
dir_1a\
batch_2.bat
dir_2\
main.py
For the sake of this question, batch_1.bat simply calls batch_2.bat, and looks like:
cd dir_1a
start batch_2.bat %*
Opening batch_1.bat from a command prompt indeed opens batch_2.bat as it's supposed to, and from there on, everything is golden.
Now I want my Python file, D:\dir_2\main.py, to spawn a new process which starts batch_1.bat, which in turn should start batch_2.bat. So I figured the following Python code should work:
import subprocess
subprocess.Popen(['cd "D:/dir_1"', "start batch_1.bat"], shell=True)
This results in "The system cannot find the path specified" being printed to my Python console. (No error is raised, of course.) This is due to the first command. I get the same result even if I cut it down to:
subprocess.Popen(['cd "D:/"'], shell=True)
I also tried starting the batch file directly, like so:
subprocess.Popen("start D:/dir_1/batch_1.bat", shell=True)
For reasons that I don't entirely get, this seems to just open a windows command prompt, in dir_2.
If I forego the start part of this command, then my Python process is going to end up waiting for batch_1 to finish, which I don't want. But it does get a little further:
subprocess.Popen("D:/dir_1/batch_1.bat", shell=True)
This results in batch_1.bat successfully executing... in dir_2, the directory of the Python script, rather than the directory of batch_1.bat, which results in it not being able to find dir_1a\ and hence, batch_2.bat is not executed at all.
I am left highly confused. What am I doing wrong, and what should I be doing instead?
Your question is answered here: Python specify popen working directory via argument
In a nutshell, just pass an optional cwd argument to Popen:
subprocess.Popen(["batch_1.bat"], shell=True, cwd=r'd:\<your path>\dir1')
I am trying to move my django project from a development server to a production server. I have ironed out almost everything with one (BIG) exception. When I run the following code in the terminal (using python manage.py shell) it works fine, however running through my apache server (with mod_wsgi) it does not run fine.
My code:
...
blastn_cline = NcbiblastnCommandline(query=filepath, db=db, evalue=0.1, outfmt=5, out=out, task="blastn-short", dust="no")
process = subprocess.Popen(str(blastn_cline),shell=True,stdout = subprocess.PIPE, stderr = subprocess.PIPE)
proc_out, proc_err = process.communicate()
err_log = open('/Users/basehunt/logs/ncbi_error_log.log', 'a+')
err_log.write("\n"+str(datetime.datetime.now())+": "+str(proc_err))
err_log.close()
...
when I look at my log file ncbi_error_log.log after I run through terminal I get (as an example):
2011-12-17 12:30:54.771292:
so no error. However, when I run through my apache server I get:
2011-12-17 12:28:59.755323: /bin/sh: blastn: command not found
I have tried to search extensively for a solution to this problem but can't find anything that gives a fix - though I hope I am missing something glaringly obvious so I can quickly sort this out.
Additional info:
OS X Snow Leopard
python version is 2.7.2
django 1.3
PATH contains the directory with blastn
If there is any additional code you want to see, let me know.
SOLVED:
by changing
process = subprocess.Popen(str(blastn_cline),shell=True,stdout = subprocess.PIPE, stderr = subprocess.PIPE)
to
process = subprocess.Popen('/Users/basehunt/BLAST/ncbi-blast-2.2.25+/bin/'+str(blastn_cline),shell=True,stdout = subprocess.PIPE, stderr = subprocess.PIPE)
in order to point absolutely to the function. Many thanks.
When running under Apache/mod_wsgi you MUST use a full path name to program being run, or any files being access for that matter. This is because your user PATH is not inherited or used by Apache. The current working directory of the process could also be anything, so can't rely on relative paths either.
So, instead of just 'blastn', use '/some/path/blastn', replacing '/some/path/' with the full path to where the program is located.
I've encountered and weired behaviour when running python in Tomcat-CGI. All things workfine expect calling a this command
subprocess.Popen('"C:\Program Files\AutoIt3\Aut2Exe\Aut2exe.exe" /in "C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps\python\install.au3" /out "C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps\python\install.exe"')
Running this code generates an exe as expected, however, it also puts the following in the HTML
<subprocess.Popen object at 0x0094BC10>
If I call the same inside an batch file, it prints the entire output in the HTML and doesn't create the exe too.
Any Ideas?
I do not hnow much about TomCat and your environment, but I would say that your
<subprocess.Popen object at 0x0094BC10>
is returnvalue of subprocess.Popen() call.
I would try to move the subprocess.Popen() somewhere, where its returnvalue is not captured into your html (if what you want is eliminate the returnvalue from your html). Just my first idea, hope it helps.
edit:OK, I could swear that the way I'd tested it showed that the getcwd was also causing the exception, but now it appears it's just the file creation. When I move the try-except blocks to their it actually does catch it like you'd think it would. So chalk that up to user error.
Original Question:
I have a script I'm working on that I want to be able to drop a file on it to have it run with that file as an argument. I checked in this question, and I already have the mentioned registry keys (apparently the Python 2.6 installer takes care of it.) However, it's throwing an exception that I can't catch. Running it from the console works correctly, but when I drop a file on it, it throws an exception then closes the console window. I tried to have it redirect standard error to a file, but it threw the exception before the redirection occurred in the script. With a little testing, and some quick eyesight I saw that it was throwing an IOError when I tried to create the file to write the error to.
import sys
import os
#os.chdir("C:/Python26/Projects/arguments")
try:
print sys.argv
raw_input()
os.getcwd()
except Exception,e:
print sys.argv + '\n'
print e
f = open("./myfile.txt", "w")
If I run this from the console with any or no arguments, it behaves as one would expect. If I run it by dropping a file on it, for instance test.txt, it runs, prints the arguments correctly, then when os.getcwd() is called, it throws the exception, and does not perform any of the stuff from the except: block, making it difficult for me to find any way to actually get the exception text to stay on screen. If I uncomment the os.chdir(), the script doesn't fail. If I move that line to within the except block, it's never executed.
I'm guessing running by dropping the file on it, which according to the other linked question, uses the WSH, is somehow messing up its permissions or the cwd, but I don't know how to work around it.
Seeing as this is probably not Python related, but a Windows problem (I for one could not reproduce the error given your code), I'd suggest attaching a debugger to the Python interpreter when it is started. Since you start the interpreter implicitly by a drag&drop action, you need to configure Windows to auto-attach a debugger each time Python starts. If I remember correctly, this article has the needed information to do that (you can substitute another debugger if you are not using Visual Studio).
Apart from that, I would take a snapshot with ProcMon while dragging a file onto your script, to get an idea of what is going on.
As pointed out in my edit above, the errors were caused by the working directory changing to C:\windows\system32, where the script isn't allowed to create files. I don't know how to get it to not change the working directory when started that way, but was able to work around it like this.
if len(sys.argv) == 1:
files = [filename for filename in os.listdir(os.getcwd())
if filename.endswith(".txt")]
else:
files = [filename for filename in sys.argv[1:]]
Fixing the working directory can be managed this way I guess.
exepath = sys.argv[0]
os.chdir(exepath[:exepath.rfind('\\')])