My script generates multiple files that contain random names based on the info it extracts. I created this test to try and move all new files created while running into a new directory named after the file being ran.
When I use os.popen("mv " + moveFiles +' ' + filename + "_dir") it works just fine, but os.popen is considered insecure due to shellshock
When switching to cmd = Popen(["mv", str(moveFiles), filename + "_dir"]), I get the following error
mv: cannot stat '/home/test/testing/TestFile1.txt
/home/test/testing/TestFile2.txt': No such file or directory
I believe this is due to it adding the apostrophe at the beginning and end of the moveFiles variable which tries to move it as 1 file rather than 2. So it works when a single file is created, but anymore results in the error. Is there a way to remove this?
'/home/test/testing/TestFile1.txt /home/test/testing/TestFile2.txt'
def createDir(filename):
"""
createDir creates the folder of the file/argument given (Example.txt_dir)
"""
Dir = str(filename) + "_dir"
cmd = Popen(["mkdir", Dir], stdout=PIPE, stderr=PIPE)
def createFiles(filename):
"""
createFiles creates test files to move into Example.txt_dir
"""
with open('TestFile1.txt', 'w') as m:
cmd = Popen(["file", filename], stdout=m, stderr=PIPE)
print('Saved as TestFile1.txt')
with open('TestFile2.txt', 'w') as m:
cmd = Popen(["file", filename], stdout=m, stderr=PIPE)
print('Saved as TestFile2.txt')
def dirDifference(dir1, dir2):
"""
dirDifference compares 2 paths, 1 before being ran and 1 after to get a list of all new files to be moved
"""
#Compares Directory before and after running
dif = [i for i in dir1 + dir2 if i not in dir1 or i not in dir2]
separator = ' '
x = separator.join(map(str, dif))
return x
def moveDir(filename, moveFiles):
"""
moveDir: Moves the new files to the directory.
"""
Dir = str(filename) + "_dir"
cmd = Popen(["mv", moveFiles, filename + "_dir"])
Your suspicion is correct: the problem is that you have two filenames joined together with a space. Since you're using Popen() and not os.popen() you're bypassing shell interpretation of the arguments, which means that individual filenames aren't getting separated. This is the same as if you had used quotes on the command line:
mv 'file1 file2' destination
mv: cannot stat 'file1 file2': No such file or directory
You've asked it to move a single file whose name has a space in the middle. What you need to do is make each filename a separate element in the list in Popen():
cmd = Popen(["mv", file1, file2, destination])
In the case of your code above, instead of dirDifference() returning filenames joined together with spaces, it could simply return a list, which you could use with Popen():
cmd = Popen(["mv"] + moveFiles + [filename + "_dir"])
(making sure that moveFiles is a non-empty list of course)
Related
I have a bunch of wave files from an outdoor bird recorder that are broken up into 1 hour segments. Each days worth of audio is in a single folder and I have 30 days worth of folders. I am trying to iterate through the folders an merge each days audio into one file and export it with the folder name but each time i try to run it the print statements indicate that each for loop runs to completion before the merge function can be called, or it runs properly and the merge funtion throws a write error.
import wave
import os
#creates an empty object for the first folder name
rootfiles= ""
#sets the path for the starting location
path = "I:\SwiftOne_000"
#lists all folders in the directory "path"
dir_list = os.listdir(path)
print("Files and directories in '", path, "' :")
#iterates through folders in path
for i in dir_list:
#adds file name to original path
rootfiles = ( path + "\\" + i)
prefix = i
# define outfiles for waves
out_name = prefix
print("first loop completed")
for x in rootfiles:
myfiles= []
paths = rootfiles
ext = (".wav")
#print(paths)
dir_lists = os.listdir(paths)
#print(dir_lists)
#print("Files and directories in '", paths, "' :")
print("second loop completed")
for x in dir_lists:
myfiles.append( paths + "\\" + x)
#print (myfiles)
outfile= "D:\SwiftD\prefix" + prefix + ".wav"
wav_files = myfiles
print("third loop completed")
from contextlib import closing
with closing(wave.open(outfile, 'wb')) as output:
# find sample rate from first file
with closing(wave.open(wav_files[0])) as w:
output.setparams(w.getparams())
# write each file to output
for infile in wav_files:
with closing(wave.open(infile)) as w:
output.writeframes(w.readframes(w.getnframes()))
I think you want something like this, assuming your folder structure is:
- Swift (directory)
- Day1 (directory)
- File1
- File2
- File3
import os, wave
src = r'I:\SwiftOne_000'
output_folder = r'I:\OutputFolder'
input_data = {}
for d_name, d_path in [(d, path) for d in os.listdir(src) if os.path.isdir(path := os.path.join(src, d))]:
input_data[d_name] = [path for f in os.listdir(d_path) if f.lower().endswith('.wav') and os.path.isfile(path := os.path.join(d_path, f))]
print(input_data)
for d_name, paths in input_data.items():
with wave.open(os.path.join(output_folder, f'{d_name}.wav'), 'wb') as output:
params_written = False
for path in paths:
with wave.open(path, 'rb') as data:
if not params_written:
output.setparams(data.getparams())
params_written = True
output.writeframes(data.readframes(data.getnframes()))
There are a few issues with your code. It better to use os.path.join to concatentate paths rather than constructing the string yourself as it makes it platform independent (although you probably don't care). os.listdir will return files and folders so you should check the type with os.path.isfile or os.path.isdir to be sure. The case for the file extension isn't always in lower case so your extension check might not work; using .lower() means you can always check for .wav.
I'm pretty sure you don't need contentlib closing as the with block will already take care of this for you.
You are using the outfile variable to write to the file, however, you overwrite this each time you loop around the third loop, so you will only ever get one file corresponding to the last directory.
Without seeing the stack trace, I'm not sure what the write error is likely to be.
Please bear with me, I've not used python before, and I'm trying to get some rendering done as quick as possible and getting stopped in my tracks with this.
I'm outputting the .ifd files to a network drive (Z:), and they are stored in a folder structure like;
Z:
- \0001
- \0002
- \0003
I need to iterate over the ifd files within a single folder, but the number of files is not static so there also needs to be a definable range (1-300, 1-2500, etc). The script therefore has to be able to take an additional two arguments for a start and end range.
On each iteration it executes something called 'mantra' using this statement;
mantra -f file.FRAMENUMBER.ifd outputFile.FRAMENUMBER.png
I've found a script on the internet that is supposed to do something similar;
import sys, os
#import command line args
args = sys.argv
# get args as string
szEndRange = args.pop()
szStartRange = args.pop()
#convert args to int
nStartRange = int(szStartRange, 10);
nEndRange = int(szEndRange, 10);
nOrd = len(szStartRange);
#generate ID range
arVals = range(nStartRange, nEndRange+1);
for nID in arVals:
szFormat = 'mantra -V a -f testDebris.%%(id)0%(nOrd)dd.ifd' % {"nOrd": nOrd};
line = szFormat % {"id": nID};
os.system(line);
The problem I'm having is that I can't get it to work. It seems to iterate, and do something - but it looks like it's just spitting out ifds into a different folder somewhere.
TLDR;
I need a script which will at least take two arguments;
startFrame
endFrame
and from those create a frameRange, which is then used to iterate over all ifd files executing the following command;
mantra -f fileName.currentframe.ifd fileName.currentFrame.png
If I were able to specify the filename and the files directory and output directory that'd be great too. I've tried manually doing that but there must be some convention to that I don't know as it was coming up with errors when I tried (stopping at the colon).
If anyone could hook me up or point me in the right direction that'd be swell. I know I should try and learn python, but I'm at my wits end with the rendering and need a helping hand.
import os, subprocess, sys
if len(sys.argv) != 3:
print('Must have 2 arguments!')
print('Correct usage is "python answer.py input_dir output_dir" ')
exit()
input_dir = sys.argv[1]
output_dir = sys.argv[2]
input_file_extension = '.txt'
cmd = 'currentframe'
# iterate over the contents of the directory
for f in os.listdir(input_dir):
# index of last period in string
fi = f.rfind('.')
# separate filename from extension
file_name = f[:fi]
file_ext = f[fi:]
# create args
input_str = '%s.%s.ifd' % (os.path.join(input_dir, file_name), cmd)
output_str = '%s.%s.png' % (os.path.join(output_dir + file_name), cmd)
cli_args = ['mantra', '-f', input_str, output_str]
#call function
if subprocess.call(cli_args, shell=True):
print('An error has occurred with command "%s"' % ' '.join(cli_args))
This should be sufficient for you to either use currently or with slight modification.
Instead of specifically inputting a start and end range you could just do:
import os
path, dirs, files = os.walk("/Your/Path/Here").next()
nEndRange = len(files)
#generate ID range
arVals = range(1, nEndRange+1);
The command os.walk() counts the # of files in the folder that you specified.
Although, an even easier way of getting your desired output is like this:
import os
for filename in os.listdir('dirname'):
szFormat = 'mantra -f ' + filename + ' outputFile.FRAMENUMBER.png'
line = szFormat % {"id": filename}; # you might need to play around with this formatting
os.system(line);
Because os.listdir() iterates through the specified directory and filename is every file in that directory, so you don't even need to count them.
a little help building the command.
for nID in arVals:
command = 'mantra -V a -f '
infile = '{0}.{1:04d}.ifd '.format(filename, id)
outfile = '{0}.{1:04d}.png '.format(filename, id)
os.system(command + infile + outfile);
and definitely use os.walk or os.listdir like #logic recommends
for file in os.listdir("Z:"):
filebase = os.path.splitext(file)[0]
command = 'mantra -V a -f {0}.ifd {0}.png'.format(filebase)
According to all the sources I've read, the open method creates a file or overwrites one with an existing name. However I am trying to use it and i get an error:
File not found - newlist.txt (Access is denied)
I/O operation failed.
I tried to read a file, and couldn't. Are you sure that file exists? If it does exist, did you specify the correct directory/folder?
def getIngredients(path, basename):
ingredient = []
filename = path + '\\' + basename
file = open(filename, "r")
for item in file:
if item.find("name") > -1:
startindex = item.find("name") + 5
endindex = item.find("<//name>") - 7
ingredients = item[startindex:endindex]
ingredient.append(ingredients)
del ingredient[0]
del ingredient[4]
for item in ingredient:
printNow(item)
file2 = open('newlist.txt', 'w+')
for item in ingredient:
file2.write("%s \n" % item)
As you can see i'm trying to write the list i've made into a file, but its not creating it like it should. I've tried all the different modes for the open function and they all give me the same error.
It looks like you do not have write access to the current working directory. You can get the Python working directory with import os; print os.getcwd().
You should then check whether you have write access in this directory. This can be done in Python with
import os
cwd = os.getcwd()
print "Write access granted to current directory", cwd, '>', os.access(cwd, os.W_OK)
If you get False (no write access), then you must put your newfile.txt file somewhere else (maybe at path + '/newfile.txt'?).
Are you certain the directory that you're trying to create the folder in exists?
If it does NOT... Then the OS won't be able to create the file.
This looks like a permissions problem.
either the directory does not exist or your user doesn't have the permissions to write into this directory .
I guess the possible problems may be:
1) You are passing the path and basename as parameters. If you are passing the parameters as strings, then you may get this problem:
For example:
def getIngredients(path, basename):
ingredient = []
filename = path + '\\' + basename
getIngredients("D","newlist.txt")
If you passing the parameters the above way, this means you are doing this
filename = "D" + "\\" + "newlist.txt"
2) You did not include a colon(:) after the path + in the filename.
3) Maybe, the file does not exist.
This is the way i call a .py file within a folder from a .pyw file outside, i am able to do that successfully, but want to know how can i pass variable from this .pyw file to the .py file within a folder.
if int(verfyUser) == int(Username):
path = self.application.applicationDirectory
name = "User_UI"
path = os.path.join(self.application.applicationDirectory, name)
if os.path.exists(os.path.join(path, name + ".pyw")):
filename = os.path.join(path, name + ".pyw")
else:
filename = os.path.join(path, name + ".py")
args = []
if ' ' in filename:
filename = '"' + filename + '"'
python = sys.executable
self.close()
if ' ' in python:
pythonQuoted = '"' + python + '"'
else:
pythonQuoted = python
os.spawnv(os.P_NOWAIT, python, [pythonQuoted, filename] + args)
i tried to pass the variables in args[], but that's not the way. Please to help me.
Want to find a way to pass the variable from .pyw to the calling py file.
Hope i am clear.....
How do you want to pass the variables? Just the values? In that case your approach seems good: you can call the other Python script with the values as command-line arguments. Your other script should then parse the command-line arguments that it has received and utilize them in the code.
But, just to be sure, you cannot write variable = 3 in the current script, then spawn a separate Python script and then use variable right away somehow.
I want to create symlinks for each file in a nested directory structure, where all symlinks will be put in one large flat folder, and have the following code by now:
# loop over directory structure:
# for all items in current directory,
# if item is directory, recurse into it;
# else it's a file, then create a symlink for it
def makelinks(folder, targetfolder, cmdprocess = None):
if not cmdprocess:
cmdprocess = subprocess.Popen("cmd",
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
print(folder)
for name in os.listdir(folder):
fullname = os.path.join(folder, name)
if os.path.isdir(fullname):
makelinks(fullname, targetfolder, cmdprocess)
else:
makelink(fullname, targetfolder, cmdprocess)
#for a given file, create one symlink in the target folder
def makelink(fullname, targetfolder, cmdprocess):
linkname = os.path.join(targetfolder, re.sub(r"[\/\\\:\*\?\"\<\>\|]", "-", fullname))
if not os.path.exists(linkname):
try:
os.remove(linkname)
print("Invalid symlink removed:", linkname)
except: pass
if not os.path.exists(linkname):
cmdprocess.stdin.write("mklink " + linkname + " " + fullname + "\r\n")
So this is a top-down recursion where first the folder name is printed, then the subdirectories are processed. If I run this now over some folder, the whole thing just stops after 10 or so symbolic links.
The program still seems to run but no new output is generated. It created 9 symlinks for some files in the # tag & reencode and the first three files in the ChillOutMix folder. The cmd.exe Window is still open and empty, and shows in its title bar that it is currently processing the mklink command for the third file in ChillOutMix.
I tried to insert a time.sleep(2) after each cmdprocess.stdin.write in case Python is just too fast for the cmd process, but it doesn't help.
Does anyone know what the problem might be?
Why not just execute mklink directly?
Try this at the end:
if not os.path.exists(linkname):
fullcmd = "mklink " + linkname + " " + fullname + "\r\n"
print fullcmd
cmdprocess.stdin.write(fullcmd)
See what commands it prints. You may see a problem.
It may need double quotes around mklink's arg, since it contains spaces sometimes.