So I have written a piece of code which first runs a powershell command to generate a UTF-8 version of a DAT file (have been having special character issues with the original file, hence the step). Following which I try to open the newly created file. But the issue is, I keep getting 'FileNotFoundError: [Errno 2]' Initially I was only trying with the file name since the newly created file was in the same folder, but then i tried to generate the absolute path as well.
import os
import subprocess
subprocess.Popen('powershell.exe -Command "Get-Content .\Own.DAT | Set-Content -Encoding utf8 Own1.dat"')
filepath = __file__
filepath = filepath[:-7]
with open(filepath+"Own1.dat", "r") as f:
I can confirm that filepath+"Own1.dat" is fetching the correct filepath. Yet can't figure out what the issue could be.
Edit: Someone asked for confirmation, here is the message i am getting:
C:\Users\Debojit\MiniConda3\python.exe "E:/My Documents/Work/essbase/ownership/test.py"
Traceback (most recent call last):
File "E:/My Documents/Work/essbase/ownership/test.py", line 18, in <module>
with open(filepath+"Own1.dat", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'E:/My Documents/Work/essbase/ownership/Own1.dat'
Process finished with exit code 1
Note: Curiously enough if i put the powershell command into a separate batch file, write a code in the python script to run it, the works without any issues. Here is the code i am talking about:
import os
import subprocess
from subprocess import Popen
p = Popen("conversion.bat", cwd=r"E:\My Documents\Work\essbase\ownership")
stdout, stderr = p.communicate()
filepath = __file__
filepath = filepath[:-7]
with open(filepath+"Own1.dat", "r") as f:
The conversion.bat file contains the following
powershell.exe -Command "Get-Content .\Own.DAT | Set-Content -Encoding utf8 Own1.DAT"
But I don't want to include a separate batch file to go with the python script.
Any idea what might be causing the issue?
Your error is unrelated to powershell. Popen runs asynchronously. In one command, you are using communicate(), but in the other, you are not.
You're using Popen() incorrectly.
If you want run a command and also pass arguments to it, you have to pass them as a list, like so:
subprocess.Popen(['powershell.exe', '-Command', ...])
In your code, popen tries to run a command literally named powershell.exe -Command "Get-Content ... which of course doesn't exist.
To use a simpler example, this code won't work:
subprocess.Popen('ls -l')
because it's trying to run a command literally named ls -l.
But this does work:
subprocess.Popen(['ls', '-l'])
I still couldn't figure out why the error was happening. But I found a workaround
with open("conversion.bat","w") as f:
f.writelines("powershell.exe -Command \"Get-Content '" +fileName+ "' | Set-Content -Encoding utf8 Own1.dat\"")
from subprocess import Popen
p = Popen("conversion.bat", cwd=os.path.dirname(os.path.realpath(__file__)))
stdout, stderr = p.communicate()
os.remove("conversion.bat")
Basically I would create the batch file, run it and then delete it once the file has been created. Don't why I have to use this route, but it works.
Related
I have a C file say, myfile.c.
Now to compile I am doing : gcc myfile.c -o myfile
So now to run this I need to do : ./myfile inputFileName > outputFileName
Where inputFileName and outputFileName are 2 command line inputs.
Now I am trying to execute this within a python program and I am trying this below approach but it is not working properly may be due to the >
import subprocess
import sys
inputFileName = sys.argv[1];
outputFileName = sys.argv[2];
subprocess.run(['/home/dev/Desktop/myfile', inputFileName, outputFileName])
Where /home/dev/Desktop is the name of my directory and myfile is the name of the executable file.
What should I do?
The > that you use in your command is a shell-specific syntax for output redirection. If you want to do the same through Python, you will have to invoke the shell to do it for you, with shell=True and with a single command line (not a list).
Like this:
subprocess.run(f'/home/dev/Desktop/myfile "{inputFileName}" > "{outputFileName}"', shell=True)
If you want to do this through Python only without invoking the shell (which is what shell=True does) take a look at this other Q&A: How to redirect output with subprocess in Python?
You can open the output file in Python, and pass the file object to subprocess.run().
import subprocess
import sys
inputFileName = sys.argv[1];
outputFileName = sys.argv[2];
with open(outputFileName, "w") as out:
subprocess.run(['/home/dev/Desktop/myfile', inputFileName], stdout=out)
I would like to loop over files using subprocess.run(), something like:
import os
import subprocess
path = os.chdir("/test")
files = []
for file in os.listdir(path):
if file.endswith(".bam"):
files.append(file)
for file in files:
process = subprocess.run("java -jar picard.jar CollectHsMetrics I=file", shell=True)
How do I correctly call the files?
shell=True is insecure if you are including user input in it. #eatmeimadanish's answer allows anybody who can write a file in /test to execute arbitrary code on your machine. This is a huge security vulnerability!
Instead, supply a list of command-line arguments to the subprocess.run call. You likely also want to pass in check=True – otherwise, your program would finish without an exception if the java commands fails!
import os
import subprocess
os.chdir("/test")
for file in os.listdir("."):
if file.endswith(".bam"):
subprocess.run(
["java", "-jar", "picard.jar", "CollectHsMetrics", "I=" + file], check=True)
Seems like you might be over complicating it.
import os
import subprocess
path = os.chdir("/test")
for file in os.listdir(path):
if file.endswith(".bam"):
subprocess.run("java -jar picard.jar CollectHsMetrics I={}".format(file), shell=True)
I have a Python 3 script that I am writing to do three things:
1) Determine which Retrosheets data files are supposed to be downloaded
2) Create wget commands to retrieve the files and download them
3) Unzip the files after they have been downloaded.
When testing each function in the Python Console, I have no problems. But, when I try to do everything automatically, I get the following output:
Start Decade: 1930
End Decade: 1950
Creating wget commands...
Commands created...
Downloaded 3 files.
Unzipping files...
Traceback (most recent call last):
File "import_pbp.py", line 54, in <module>
unzip_data(decade_files)
File "import_pbp.py", line 39, in unzip_data
with zipfile.ZipFile('zip' + file, 'r') as zip_ref:
File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5 /lib/python3.5/zipfile.py", line 1009, in __init__
self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: 'zip1930seve.zip'
The files are downloaded after this output to the console. This would seem to indicate that the unzip function is running before the files are downloaded. How do I make sure that my files are downloaded before the unzip function is called? Code below:
Download function:
# define function to execute the download commands
def download_data(commands):
for command in commands:
os.popen(command)
print('Downloaded ' + str(len(commands)) + ' files.')
Unzip Function:
# Unzip the data files into the 'unparsed' folder.
def unzip_data(file_list):
print('Unzipping files...')
for file in file_list:
with zipfile.ZipFile('zip' + file, 'r') as zip_ref:
zip_ref.extractall('unparsed/')
print(file + ' unzipped')
print('All files unzipped...')
EDIT: I looked at the response in this thread but it didn't quite explain what I needed like tdelaney did below. They are similar, but for my purposes, different. Especially since that question is 6 years old and I'm guessing there may have been significant changes to the language since then.
EDIT 2: Removed non-essential code to shorten the post.
os.popen doesn't wait for the process to complete so you launch all of the commands at once then try the unzips before they are done. Since you don't read the stdout pipe returned from os.popen, you also risk the program hanging if the output pipe fills.
The subprocess module has several functions for calling programs. Assuming you really do want all of the commands to run in parallel and that you just want to discard any output data from the commands, you could reimplement that function as
import subprocess as subp
import os
# define function to execute the download commands and unzip
def download_data(commands):
procs = []
for command in commands:
procs.append(subp.Popen(command, shell=True,
stdout=open(os.devnull, 'wb')))
for proc in procs:
proc.wait()
print('Downloaded ' + str(len(commands)) + ' files.')
I expect a directory to be created and then a file to be opened within it for writing to when I execute my code below in Python 2.6.6,
import subprocess
def create_output_dir(work_dir):
output_dir = '/work/m/maxwell9/some_name5/'
subprocess.Popen(['mkdir', output_dir])
return output_dir
if __name__ == '__main__':
work_dir = '/work/m/maxwell9/'
output_dir = create_output_dir(work_dir)
#output_dir = '/work/m/maxwell9/some_name5/'
filename = output_dir + 'bt.sh'
with open(filename, 'w') as script:
print('there')
but instead I get the error,
Traceback (most recent call last):
File "slurm_test.py", line 13, in <module>
with open(filename, 'w') as script:
IOError: [Errno 2] No such file or directory: '/work/m/maxwell9/some_name5/bt.sh'
If I run the script, I can then see that the directory is created. If I then uncomment the line,
#output_dir = '/work/m/maxwell9/some_name5/'
and comment the line,
output_dir = create_output_dir(work_dir)
then the file is output fine. So there is something about creating the folder and then writing to it in the same script that is causing an error.
subprocess.Popen starts up an external process but doesn't wait for it to complete unless you tell it to (e.g. by calling .wait on the returned Popen instance). Most likely, mkdir is in the process of creating a directory while open(filename, 'w') attempts to create a file in that directory. This is an example of a "race condition".
The solution is to .wait on the open process (as noted above), or you can use one of the convenience wrappers subprocess.check_output, subprocess.check_call or (even better), you can avoid subprocess entirely by using os.mkdir or os.makedirs.
You could use the os library instead of subprocess, which makes for a more straightforward implementation. Try swapping out your create_output_dir function with this:
import os
def create_output_dir(work_dir):
try:
os.makedirs(work_dir)
except OSError:
pass
return work_dir
Opening a file available in present working directory's temp folder in python
I tried
pwdir=os.getcwd()
tempdir=pwdir+"/temp/test.txt"
f=open(tempdir,'r+')
When I print the path of tempdir, it is showing up correctly and also the contents of file are also read.
When I try to combine this operation from an Applescript, which calls this python script. I get an error like this
f=open(pwdir1,'r+')
IOError: [Errno 2] No such file or directory: '//temp/test.txt'" number 1
EDIT:
I am using Shell script from Applescript to call this pythonscript
do shell script "/Users/mymac/Documents/'Microsoft User Data'/test.py"
EDIT:
Python Code:
tempdir = os.path.join(os.getcwd(),'temp','htmlinput.html')
print tempdir
with open(tempdir) as f:
html=f.read()
Python output from terminal:(works perfectly fine)
/Users/mymac/Documents/Microsoft User Data/Outlook Script Menu Items/temp/htmlinput.html
I am also able to see the file contents.
Applescript Code:
do shell script "/Users/mymac/Documents/'Microsoft User Data'/'Outlook Script Menu Items'/new.py"
Applescript Error:
error "Microsoft Outlook got an error: Traceback (most recent call last):
File \"/Users/mymac/Documents/Microsoft User Data/Outlook Script Menu Items/new.py\", line 12, in <module>
with open(tempdir) as f:
IOError: [Errno 2] No such file or directory: '/temp/htmlinput.html'" number 1
I don't know Applescript -- or OS X in general. It looks like the script is being run from the root folder, and os.getcwd() returns '/'. The directory of the script itself is sys.path[0] or the dirname of the current module -- dirname(__file__) -- if it's a single script instead of a package. Try one of the following
import os, sys
tempdir = os.path.join(sys.path[0], 'temp', 'temp.txt')
or
import os
tempdir = os.path.join(os.path.dirname(__file__), 'temp', 'temp.txt')
The double slash is your problem. The right way to join file and path names in Python is with os.path.join. Try:
tempdir = os.path.join(os.getcwd(), 'temp', 'test.txt')
Also, you should probably be doing:
with open(tempdir) as f:
which will make sure tempdir gets closed even if you have an error.
Edit:
We need to see what tempdir is when the script is invoked by the AppleScript, not when it is invoked from the terminal. If you do
tempdir = os.path.join(os.getcwd(),'temp','htmlinput.html')
with open('/Users/mymac/Documents/temp.txt', 'w') as fwrite:
fwrite.write(tempdir)
What exactly ends up in the file /Users/mymac/Documents/temp.txt?