I'm using subprocess to run a script.R file from test.py. My test.py goes as:
import subprocess
import pandas as pd
subprocess.call(["/usr/bin/Rscript", "--vanilla", "script.R"]) #line 3
df=pd.read_csv("output.csv") #line 4
script.R goes as:
library(limma)
df <- read.csv("input.csv")
df<-normalizeCyclicLoess(df)
write.csv(df,"output.csv")
When I run the above file (test.py), I get an error:
FileNotFoundError: [Errno 2] File b'output.csv' does not exist: b'output.csv'
I understand that this error is because a output.csv file doesn't exist in my working directory. But, I assume that it would be created by script.R, which isn't happening probably because before the execution of line 3 finishes, python goes to line 4. We used help from here for this, and as mentioned, we are using call for this. What's going wrong then? Thanks...
EDIT: I noticed that if I don't import any libraries in my code (like limma above), everything works fine. I can write any lengthy code as I want, and it doesn't give me any error and proceeds to completion. But, as soon as I import any library, subprocess.call(....) gives me a non-zero result (zero means process completed). For a test, I changed my script.R to library(limma) (and nothing else) (tried for other libraries too- got the same results), and it gave me a non-zero result. Thus, I think, there's some problem with the import of libraries using subprocess. Please note that I'm able to run this R code directly, and nothing is wrong with my code/library. Please give me some hints on what might be going wrong here...
Apologies, my initial answer was completely wrong.
Your subprocess call:
subprocess.call(["/usr/bin/Rscript", "--vanilla", "script.R"])
will return a number - this is the exit code of the process. If it succeeds, without error, it should be zero. You could check this to make sure your R code ran correctly.
Have you tried running your R code directly? Does it produce the output.csv? And if so, does it produce it in the right place?
Related
I´ve spent the last few hours on the multiprocessing tool in python. Last thing I did was to set up a Pool with a few workers to execute a function. Everything was doing fine and multiprocessing was working.
After I switched to another file to implement my multiprocessing insights in my actual project, the code was running, but the .get() invocation following the apply_async(func, args) never returned anything.
I thought it was due to a false implementation, but turned out to be something else. I went back to the first file where I did my experiments and tried to run this one again. Suddenly also here multiprocessing didn´t work anymore. Apart from trying to implement the multiprocessing in my project I didn´t really do anything else on the code or the environment. The only thing maybe correlated to the problem is that I tried to execute an older version of the __main__ program in the same environment, leading to an error because the __main__ file took to newer project files to execute.
Here´s the code I used for the simple experiment:
import torch.multiprocessing as mp
import torch
import time
*#thats code from my project*
import data.preprocessor as preprocessor
import eco_neat_t1_cfg as cfg
def f(x):
return x*x
if __name__ == "__main__":
p = mp.Pool(1)
j = p.apply_async(f, (5,))
r = j.get(timeout=10)
print(r)
So nothing to complicated here, it should actually work.
However, there is no result being returned. I notice the CPU is working according to the number of processes invoked, so I assume it is caught in some kind of loop.
I am running on Win10, with Python running over Anaconda Spyder with the IPython Console.
When I close Spyder and all Python processes, restart and execute it again it works, but only if I am not importing my own code. Once I import the eco_neat_t1_cfg as cfg, which is basically only a dataclass with a few other imports, the program wont return anything and gets stuck in the execution.
I am getting a Reloaded modules: data, data.preprocessor, data.generator error. This is probably because cfg also imports data.preprocessor. The weird thing is, if I now delete all imports from the source code, save the file and run it, it will work again, although the imports are not even specified in the code anymore... however, every time I restart Python and Spyder I need to specify the modules again ofc.
Also, the simple code is working after deletion of all imports, but the more complicated I actually want to execute is not, although it was perfectly fine before all this happened. Now I am getting a PicklingError: Can't pickle <class 'neat.genotype.genome.Pickleable_Genome'>: it's not the same object as neat.genotype.genome.Pickleable_Genome error. As I said, there were no problems with the pickling before. That´s the "more complicated" piece of code I pass to apply_async.
def create_net(genome):
net = feedforward.Net("cpu", genome, 1.0)
inp = torch.ones(2, 5)
input = torch.FloatTensor(inp).view(1, -1).squeeze()
res = net(input)
return res
What I did was to delete all pycache in my project, also reset Spyder settings and update Anaconda and Spyder. Also I tried to look at every piece of code of the cfg file, but even if I delete every line, also no imports, the program will not work. Nothing helps, I really don´t have any idea of what´s the problem and how to solve it. Maybe one of you has an idea.
One more thing to say, if I manually interrupt (CTRL + C) the execution, in the stacktrace I noticed the processes to be waiting.
File "C:/Users/klein/Desktop/bak/test/eco_neat_t1/blabla.py", line 72, in <module>
r = j.get(timeout=10)
File "C:\Users\klein\Anaconda3\envs\eco_neat\lib\multiprocessi\pool.py", line 651, in get
self.wait(timeout)
File "C:\Users\klein\Anaconda3\envs\eco_neat\lib\multiprocessing\pool.py", line 648, in wait
self._event.wait(timeout)
File "C:\Users\klein\Anaconda3\envs\eco_neat\lib\threading.py", line 552, in wait
signaled = self._cond.wait(timeout)
Cheers
SOLUTION: The bug was my directory structure that somehow led python to fail. I had both of my __main__ test files running in the root/tests/eco_neat_t1 directory. I two new files t1_cfg and t1 with the same content in the root directory. Suddenly everthing works fine...weird world
I figured out the zip code in same line out. It's sys.argv[1], I had other code I neglected to comment out when trying out [1] that gave me the error. All I need help with now is getting weather.py to run without having to call the whole file path.
I will preface with I'm not very experienced with python and may get certain names wrong or think something might work that obviously doesn't, bear with me I tried to word this to make as much sense as possible.
So I need to run a program using the command line. The program is complete and 100% functioning when ran in PyDev. The program is called weather.py, and what needs to trigger it in cmd is
python weather.py (5 digit zip)
I cannot get the program to run using just 'python weather.py' first off. I have added C:\python27 to PATH as well as C:\python27\python.exe (not sure if that does anything). Getting the .py to run via those two keywords doesn't seem to work with what I've tried. I also need to be able to add a zip code to the same line to trigger the program. I was told about
zipcode = sys.argv[0]
to allow the zip code to be automatically initialized as a variable, but I get
IndexError: list index out of range
when I run the program using
python C:\python27\weather.py
I tried replacing 0 with 1 or 2 because I'm unfamiliar with .argv but neither of those worked either. Any help getting the program to run using just python weather.py OR getting the zip code input to function on the same line is greatly appreciated.
Make sure you import sys in your code.
import sys
zipCode = sys.argv[1]
and actually provide an argument
EDIT:
For clarity, if sys was not imported, you would get NameError and not an IndexError. Additionally, when passing args in from the command line, the indexing actually begins at 0 where sys.argv[0] is always the program name and the provided args begin at 1. So, in this case, the zip code would be at sys.argv[1]
EDIT2:
variable name to avoid using reserve words :)
I want to call a program multiple times from a python code, and save the output of that program in a text file. My first problem right now is just calling the other code. I have to redirect to a different directory and call ./rank on output.txt. This is how Im trying to do it:
TheCommand = "~/src/rank-8-9-2011/rank output.txt"
os.system(TheCommand)
but im getting a parsing error.
[Parsing error on line ]Unknown error: 0
Im running python2.7 on Mac OS 10.5.8. Im not sure what the problem is. I also tried using subprocess:
subprocess.call(["~/src/rank-8-9-2011/rank", "output.txt"])
This does not find the directory (I have a feeling Im using the subprocess incorrectly), but I dont know what is wrong with the os.system.
the name of the program in the first argument to subprocess.Popen must not contain ~ as it doesn't pass the string to the shell for processing (which like always using parameterized queries in sql, protects one from string injection attacks, e.g. if instead of output.text one had ;rm -rf /, the system version would run rank and then run rm -rf . but the subprocess.Popen would only have rank open a file named ;rm -rf .), so one should expand it by calling os.path.expanduser:
subprocess.Popen([os.path.expanduser('~/src/rank-8-9-2011/rank'), "output.txt"])
although it is possible to turn shell processing on by passing shell=True, it is not recommended for the aforementioned reason.
you should try http://docs.python.org/library/os.path.html#os.path.expanduser
import os.path
subprocess.call([os.path.expanduser("~/src/rank-8-9-2011/rank"), "output.txt"])
I'm fairly certain your parsing error is coming from rank, not from your os.system command, as nothing there looks weird. What happens if you run rank manually?
subprocess seems to have a problem with '~', although I'm not immediately sure why. Put the full path and it should work (although you'll likely get that parsing error if it is indeed a problem with rank).
edit:OK, I could swear that the way I'd tested it showed that the getcwd was also causing the exception, but now it appears it's just the file creation. When I move the try-except blocks to their it actually does catch it like you'd think it would. So chalk that up to user error.
Original Question:
I have a script I'm working on that I want to be able to drop a file on it to have it run with that file as an argument. I checked in this question, and I already have the mentioned registry keys (apparently the Python 2.6 installer takes care of it.) However, it's throwing an exception that I can't catch. Running it from the console works correctly, but when I drop a file on it, it throws an exception then closes the console window. I tried to have it redirect standard error to a file, but it threw the exception before the redirection occurred in the script. With a little testing, and some quick eyesight I saw that it was throwing an IOError when I tried to create the file to write the error to.
import sys
import os
#os.chdir("C:/Python26/Projects/arguments")
try:
print sys.argv
raw_input()
os.getcwd()
except Exception,e:
print sys.argv + '\n'
print e
f = open("./myfile.txt", "w")
If I run this from the console with any or no arguments, it behaves as one would expect. If I run it by dropping a file on it, for instance test.txt, it runs, prints the arguments correctly, then when os.getcwd() is called, it throws the exception, and does not perform any of the stuff from the except: block, making it difficult for me to find any way to actually get the exception text to stay on screen. If I uncomment the os.chdir(), the script doesn't fail. If I move that line to within the except block, it's never executed.
I'm guessing running by dropping the file on it, which according to the other linked question, uses the WSH, is somehow messing up its permissions or the cwd, but I don't know how to work around it.
Seeing as this is probably not Python related, but a Windows problem (I for one could not reproduce the error given your code), I'd suggest attaching a debugger to the Python interpreter when it is started. Since you start the interpreter implicitly by a drag&drop action, you need to configure Windows to auto-attach a debugger each time Python starts. If I remember correctly, this article has the needed information to do that (you can substitute another debugger if you are not using Visual Studio).
Apart from that, I would take a snapshot with ProcMon while dragging a file onto your script, to get an idea of what is going on.
As pointed out in my edit above, the errors were caused by the working directory changing to C:\windows\system32, where the script isn't allowed to create files. I don't know how to get it to not change the working directory when started that way, but was able to work around it like this.
if len(sys.argv) == 1:
files = [filename for filename in os.listdir(os.getcwd())
if filename.endswith(".txt")]
else:
files = [filename for filename in sys.argv[1:]]
Fixing the working directory can be managed this way I guess.
exepath = sys.argv[0]
os.chdir(exepath[:exepath.rfind('\\')])
I'm trying to use python to run a program.
from subprocess import Popen
sa_proc = Popen(['C:\\sa\\sa.exe','--?'])
Running this small snippit gives the error:
WindowsError: [Error 2] The system cannot find the file specified
The program exists and I have copy and pasted directly from explorer the absolute path to the exe. I have tried other things and have found that if I put the EXE in the source folder with the python script and use './sa.exe' then it works. The only thing I can think of is that I'm running the python script (and python) from a separate partition (F:).
Any ideas?
Thanks
As the docs say, "On Windows: the Popen class uses CreateProcess() to execute the child program, which operates on strings. If args is a sequence, it will be converted to a string using the list2cmdline() method.". Maybe that method is messing things up, so why not try the simpler approach of:
sa_proc = Popen('C:\\sa\\sa.exe --?')
If this still fails, then: what's os.environ['COMSPEC'] just before you try this? What happens if you add , shell=True to Popen's arguments?
Edit: turns out apparently to be a case of simple mis-spellling, as 'sa' was actually the program spelled SpamAssassin -- double s twice -- and what the OP was writing was spamassasin -- one double s but a single one the second time.
You may not have permission to execute C:\sa\sa.exe. Have you tried running the program manually?