I want to create symlinks for each file in a nested directory structure, where all symlinks will be put in one large flat folder, and have the following code by now:
# loop over directory structure:
# for all items in current directory,
# if item is directory, recurse into it;
# else it's a file, then create a symlink for it
def makelinks(folder, targetfolder, cmdprocess = None):
if not cmdprocess:
cmdprocess = subprocess.Popen("cmd",
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
print(folder)
for name in os.listdir(folder):
fullname = os.path.join(folder, name)
if os.path.isdir(fullname):
makelinks(fullname, targetfolder, cmdprocess)
else:
makelink(fullname, targetfolder, cmdprocess)
#for a given file, create one symlink in the target folder
def makelink(fullname, targetfolder, cmdprocess):
linkname = os.path.join(targetfolder, re.sub(r"[\/\\\:\*\?\"\<\>\|]", "-", fullname))
if not os.path.exists(linkname):
try:
os.remove(linkname)
print("Invalid symlink removed:", linkname)
except: pass
if not os.path.exists(linkname):
cmdprocess.stdin.write("mklink " + linkname + " " + fullname + "\r\n")
So this is a top-down recursion where first the folder name is printed, then the subdirectories are processed. If I run this now over some folder, the whole thing just stops after 10 or so symbolic links.
The program still seems to run but no new output is generated. It created 9 symlinks for some files in the # tag & reencode and the first three files in the ChillOutMix folder. The cmd.exe Window is still open and empty, and shows in its title bar that it is currently processing the mklink command for the third file in ChillOutMix.
I tried to insert a time.sleep(2) after each cmdprocess.stdin.write in case Python is just too fast for the cmd process, but it doesn't help.
Does anyone know what the problem might be?
Why not just execute mklink directly?
Try this at the end:
if not os.path.exists(linkname):
fullcmd = "mklink " + linkname + " " + fullname + "\r\n"
print fullcmd
cmdprocess.stdin.write(fullcmd)
See what commands it prints. You may see a problem.
It may need double quotes around mklink's arg, since it contains spaces sometimes.
Related
My script generates multiple files that contain random names based on the info it extracts. I created this test to try and move all new files created while running into a new directory named after the file being ran.
When I use os.popen("mv " + moveFiles +' ' + filename + "_dir") it works just fine, but os.popen is considered insecure due to shellshock
When switching to cmd = Popen(["mv", str(moveFiles), filename + "_dir"]), I get the following error
mv: cannot stat '/home/test/testing/TestFile1.txt
/home/test/testing/TestFile2.txt': No such file or directory
I believe this is due to it adding the apostrophe at the beginning and end of the moveFiles variable which tries to move it as 1 file rather than 2. So it works when a single file is created, but anymore results in the error. Is there a way to remove this?
'/home/test/testing/TestFile1.txt /home/test/testing/TestFile2.txt'
def createDir(filename):
"""
createDir creates the folder of the file/argument given (Example.txt_dir)
"""
Dir = str(filename) + "_dir"
cmd = Popen(["mkdir", Dir], stdout=PIPE, stderr=PIPE)
def createFiles(filename):
"""
createFiles creates test files to move into Example.txt_dir
"""
with open('TestFile1.txt', 'w') as m:
cmd = Popen(["file", filename], stdout=m, stderr=PIPE)
print('Saved as TestFile1.txt')
with open('TestFile2.txt', 'w') as m:
cmd = Popen(["file", filename], stdout=m, stderr=PIPE)
print('Saved as TestFile2.txt')
def dirDifference(dir1, dir2):
"""
dirDifference compares 2 paths, 1 before being ran and 1 after to get a list of all new files to be moved
"""
#Compares Directory before and after running
dif = [i for i in dir1 + dir2 if i not in dir1 or i not in dir2]
separator = ' '
x = separator.join(map(str, dif))
return x
def moveDir(filename, moveFiles):
"""
moveDir: Moves the new files to the directory.
"""
Dir = str(filename) + "_dir"
cmd = Popen(["mv", moveFiles, filename + "_dir"])
Your suspicion is correct: the problem is that you have two filenames joined together with a space. Since you're using Popen() and not os.popen() you're bypassing shell interpretation of the arguments, which means that individual filenames aren't getting separated. This is the same as if you had used quotes on the command line:
mv 'file1 file2' destination
mv: cannot stat 'file1 file2': No such file or directory
You've asked it to move a single file whose name has a space in the middle. What you need to do is make each filename a separate element in the list in Popen():
cmd = Popen(["mv", file1, file2, destination])
In the case of your code above, instead of dirDifference() returning filenames joined together with spaces, it could simply return a list, which you could use with Popen():
cmd = Popen(["mv"] + moveFiles + [filename + "_dir"])
(making sure that moveFiles is a non-empty list of course)
Please bear with me, I've not used python before, and I'm trying to get some rendering done as quick as possible and getting stopped in my tracks with this.
I'm outputting the .ifd files to a network drive (Z:), and they are stored in a folder structure like;
Z:
- \0001
- \0002
- \0003
I need to iterate over the ifd files within a single folder, but the number of files is not static so there also needs to be a definable range (1-300, 1-2500, etc). The script therefore has to be able to take an additional two arguments for a start and end range.
On each iteration it executes something called 'mantra' using this statement;
mantra -f file.FRAMENUMBER.ifd outputFile.FRAMENUMBER.png
I've found a script on the internet that is supposed to do something similar;
import sys, os
#import command line args
args = sys.argv
# get args as string
szEndRange = args.pop()
szStartRange = args.pop()
#convert args to int
nStartRange = int(szStartRange, 10);
nEndRange = int(szEndRange, 10);
nOrd = len(szStartRange);
#generate ID range
arVals = range(nStartRange, nEndRange+1);
for nID in arVals:
szFormat = 'mantra -V a -f testDebris.%%(id)0%(nOrd)dd.ifd' % {"nOrd": nOrd};
line = szFormat % {"id": nID};
os.system(line);
The problem I'm having is that I can't get it to work. It seems to iterate, and do something - but it looks like it's just spitting out ifds into a different folder somewhere.
TLDR;
I need a script which will at least take two arguments;
startFrame
endFrame
and from those create a frameRange, which is then used to iterate over all ifd files executing the following command;
mantra -f fileName.currentframe.ifd fileName.currentFrame.png
If I were able to specify the filename and the files directory and output directory that'd be great too. I've tried manually doing that but there must be some convention to that I don't know as it was coming up with errors when I tried (stopping at the colon).
If anyone could hook me up or point me in the right direction that'd be swell. I know I should try and learn python, but I'm at my wits end with the rendering and need a helping hand.
import os, subprocess, sys
if len(sys.argv) != 3:
print('Must have 2 arguments!')
print('Correct usage is "python answer.py input_dir output_dir" ')
exit()
input_dir = sys.argv[1]
output_dir = sys.argv[2]
input_file_extension = '.txt'
cmd = 'currentframe'
# iterate over the contents of the directory
for f in os.listdir(input_dir):
# index of last period in string
fi = f.rfind('.')
# separate filename from extension
file_name = f[:fi]
file_ext = f[fi:]
# create args
input_str = '%s.%s.ifd' % (os.path.join(input_dir, file_name), cmd)
output_str = '%s.%s.png' % (os.path.join(output_dir + file_name), cmd)
cli_args = ['mantra', '-f', input_str, output_str]
#call function
if subprocess.call(cli_args, shell=True):
print('An error has occurred with command "%s"' % ' '.join(cli_args))
This should be sufficient for you to either use currently or with slight modification.
Instead of specifically inputting a start and end range you could just do:
import os
path, dirs, files = os.walk("/Your/Path/Here").next()
nEndRange = len(files)
#generate ID range
arVals = range(1, nEndRange+1);
The command os.walk() counts the # of files in the folder that you specified.
Although, an even easier way of getting your desired output is like this:
import os
for filename in os.listdir('dirname'):
szFormat = 'mantra -f ' + filename + ' outputFile.FRAMENUMBER.png'
line = szFormat % {"id": filename}; # you might need to play around with this formatting
os.system(line);
Because os.listdir() iterates through the specified directory and filename is every file in that directory, so you don't even need to count them.
a little help building the command.
for nID in arVals:
command = 'mantra -V a -f '
infile = '{0}.{1:04d}.ifd '.format(filename, id)
outfile = '{0}.{1:04d}.png '.format(filename, id)
os.system(command + infile + outfile);
and definitely use os.walk or os.listdir like #logic recommends
for file in os.listdir("Z:"):
filebase = os.path.splitext(file)[0]
command = 'mantra -V a -f {0}.ifd {0}.png'.format(filebase)
I have a python script that doesn't seem to be opening the files.
The folder in the script is defined like this:
logdir = "C:\\Programs\\CommuniGate Files\\SystemLogs\\"
submitdir = "C:\\Programs\\CommuniGate Files\\Submitted\\"
This is how the paths are being used:
filenames = os.listdir(logdir)
fnamewithpath = logdir + fname
I'm running this script in Windows 7 sp1
Does this look correct?
Is there something I can put into the code to debug it to make sure the files are opening?
Thank you,
Docfxit
Edited to provide more clarification:
The actual code to open and close the file is here:
# read all the log parts
for fname in logfilenames :
fnamewithpath = logdir + fname
try :
inputFile = open(fnamewithpath, "r")
except IOError as reason :
print("Error: " + str(reason))
return
if testing :
print("Reading file '%s'" % (fname))
reporter.munchFile(inputFile)
inputFile.close()
# open output file
if testing :
outfilename = fullLognameWithPath + ".summary"
fullOutfilename = outfilename
else :
outfilename = submitdir + "ls" + str(time.time()) + "-" + str(os.getpid())
fullOutfilename = outfilename + ".tmp"
try :
outfile = open(fullOutfilename, "w")
except IOError :
print("Can't open output file " + fullOutfilename)
return
if not testing :
# add the mail headers first
outfile.write("To: " + reportToAddress + "\n")
outfile.write("From: " + reportFromAddress + "\n")
outfile.write("Subject: CGP Log Summary new for " + logname + "\n")
if useBase64 :
outfile.write("Content-Transfer-Encoding: base64\n")
outfile.write("\n")
# save all this as a string so that we can base64 encode it
outstring = ""
outstring += "Summary of file: " + fullLogname + partAddendum + "\n"
outstring += "Generated " + time.asctime() + "\n"
outstring += reporter.generateReport()
if useBase64 :
outstring = base64.encodestring(outstring)
outfile.write(outstring)
outfile.close()
if not testing :
# rename output file to submit it
try :
os.rename(outfilename + ".tmp", outfilename + ".sub")
except OSError :
print("Can't rename mail file to " + outfilename + ".sub")
I was originally wondering if the double back slashes included in the path were correct.
I can't figure out why it isn't producing the output correctly.
Just in case someone would like to see the entire script I posted it:
The first half is at:
http://pastebin.ws/7ipf3
The second half is at:
http://pastebin.ws/2fuu3n
It was too large to post all in one.
This is being run in Python 3.2.2
Thank you very much for looking at it,
Docfxit
The code as written above does not actually open either file.
os.listdir returns a list of the files (technically entries, as non-files like . and .. are also included) in the specified path, but does not actually open them. For that you would need to call the open function on one of the paths.
If you wanted to open all the files in filenames for write, you could do something like this:
fileList = []
for f in filenames:
if os.path.isfile(fullPath):
fullPath = os.path.join(logdir, f)
fileList.append(open(fullPath, 'w')
After this, the list fileList would contain the open file handles for all of the files, which could then be iterated over (and, for example, all written to if you wanted to multiplex the output).
Note that when done, you should loop through the list and explicitly close them all (the with syntax that automatically closes them has additional complexities/limitations when it comes to dynamically sized lists, and is best avoided here, IMO).
For more info on files, see:
Reading and Writing Files
Also, it's best to use os.path.join to combine components of a path. That way it can be portable across supported platforms, and will automatically use the correct path separators, and such.
Edit in response to comment from OP:
I would recommend you step through the code using a debugger to see exactly what's going wrong. I use pudb, which is an enhanced command-line debugger, and find it invaluable. You can install it via pip into your system/virtualenv Python environment.
In my Python script, I need to create a new file in a sub directory without changing directories, and I need to continually edit that file from the current directory.
My code:
os.mkdir(datetime+"-dst")
for ip in open("list.txt"):
with open(ip.strip()+".txt", "a") as ip_file: #this file needs to be created in the new directory
for line in open("data.txt"):
new_line = line.split(" ")
if "blocked" in new_line:
if "src="+ip.strip() in new_line:
#write columns to new text file
ip_file.write(", " + new_line[11])
ip_file.write(", " + new_line[12])
try:
ip_file.write(", " + new_line[14] + "\n")
except IndexError:
pass
Problems:
The path for the directory and file will not always be the same, depending on what server I run the script from. Part of the directory name will be the datetime of when it was created ie time.strftime("%y%m%d%H%M%S") + "word" and I'm not sure how to call that directory if the time is constantly changing. I thought I could use shutil.move() to move the file after it was created, but the datetime stamp seems to pose a problem.
I'm a beginner programmer and I honestly have no idea how to approach these problems. I was thinking of assigning variables to the directory and file, but the datetime is tripping me up.
Question: How do you create a file within a sub directory if the names/paths of the file and sub directory aren't always the same?
Store the created directory in a variable. os.mkdir throws if a directory exists by that name.
Use os.path.join to join path components together (it knows about whether to use / or \).
import os.path
subdirectory = datetime + "-dst"
try:
os.mkdir(subdirectory)
except Exception:
pass
for ip in open("list.txt"):
with open(os.path.join(subdirectory, ip.strip() + ".txt"), "a") as ip_file:
...
first convert the datetime to something the folder name can use
something like this could work
mydate_str = datetime.datetime.now().strftime("%m-%d-%Y")
then create the folder as required
- check out
Creating files and directories via Python
Johnf
According to all the sources I've read, the open method creates a file or overwrites one with an existing name. However I am trying to use it and i get an error:
File not found - newlist.txt (Access is denied)
I/O operation failed.
I tried to read a file, and couldn't. Are you sure that file exists? If it does exist, did you specify the correct directory/folder?
def getIngredients(path, basename):
ingredient = []
filename = path + '\\' + basename
file = open(filename, "r")
for item in file:
if item.find("name") > -1:
startindex = item.find("name") + 5
endindex = item.find("<//name>") - 7
ingredients = item[startindex:endindex]
ingredient.append(ingredients)
del ingredient[0]
del ingredient[4]
for item in ingredient:
printNow(item)
file2 = open('newlist.txt', 'w+')
for item in ingredient:
file2.write("%s \n" % item)
As you can see i'm trying to write the list i've made into a file, but its not creating it like it should. I've tried all the different modes for the open function and they all give me the same error.
It looks like you do not have write access to the current working directory. You can get the Python working directory with import os; print os.getcwd().
You should then check whether you have write access in this directory. This can be done in Python with
import os
cwd = os.getcwd()
print "Write access granted to current directory", cwd, '>', os.access(cwd, os.W_OK)
If you get False (no write access), then you must put your newfile.txt file somewhere else (maybe at path + '/newfile.txt'?).
Are you certain the directory that you're trying to create the folder in exists?
If it does NOT... Then the OS won't be able to create the file.
This looks like a permissions problem.
either the directory does not exist or your user doesn't have the permissions to write into this directory .
I guess the possible problems may be:
1) You are passing the path and basename as parameters. If you are passing the parameters as strings, then you may get this problem:
For example:
def getIngredients(path, basename):
ingredient = []
filename = path + '\\' + basename
getIngredients("D","newlist.txt")
If you passing the parameters the above way, this means you are doing this
filename = "D" + "\\" + "newlist.txt"
2) You did not include a colon(:) after the path + in the filename.
3) Maybe, the file does not exist.