Develop Reference

Save created file into new directory in loop

I am currently updating some files I have in a directory, with a loop, and I would like to save those file in a different directory.
Here is what I have:
from astropy.io import fits
from mpdaf.obj import Spectrum, WaveCoord
import os, glob
ROOT_DIR=input("Enter root directory : ")
os.chdir(ROOT_DIR)
destination=input("Enter destination directory : ")
fits_files = glob.glob("*.fits")
for spect in fits_files:
spe = Spectrum(filename= spect, ext=[0,1])
(spect_name, ext) = os.path.splitext(spect)
sperebin = spe.rebin(57)
sperebin.write(spect_name + "-" + "rebin" + ".fits")
With the last line sperebin.write(spect_name + "-" + "rebin" + ".fits") it is currently writing the file in the directory I'm in, and I would like it to write it directly into the destination directory, any idea how to proceed?

With os.path.join you can combine directory with filename to get the file path.
sperebin.write(os.path.join(destination, pect_name + "-" + "rebin" + ".fits"))
Also with os you can check if the directory exists and create it if you want to.
if not os.path.exists(destination):
os.makedirs(destination)

You don't need to or want to change directories in your script. Instead, use pathlib to simplify the creation of the new filename.
from pathlib import Path
root_dir = Path(input("Enter root directory : "))
destination_dir = Path(input("Enter destination directory : "))
# Spectrum wants a string for the filename argument
# so you need to call str on the Path object
for pth in root_dir.glob("*.fits"):
spe = Spectrum(filename=str(pth), ext=[0,1])
sperebin = spe.rebin(57)
dest_pth = destination_dir / pth.stem / "-rebin" / pth.suffix
sperebin.write(str(dest_pth))

Need help creating a txt file in a folder which, is in the same directory

I'm pretty new to python. I have a homwork problem where we need to analyze corpora and then compare them. We also have to save the files as a .txt file after is has been processed with an attribute, the size.
So I need to create a .txt file in a seperate folder called trigram-models.
This folder is in the same directory as my python file. I think i have to use the os module but i'm not sure how.
Here is my code:
from langdetect import read_trigrams, trigram_table, write_trigrams
import os
def make_profiles(datafolder, profilefolder, size):
filelist = []
for file in os.listdir('./training'):
filelist.append(file)
print(filelist)
for file in filelist:
filen = "./training/"+file
print("fi", filen)
maketable = trigram_table(filen, size)
readdata = read_trigrams(filen)
#print("re", readdata)
splitname = str(file).split('-')
newname = splitname[0] + "." + str(size) + '.txt'
endtable = write_trigrams(readdata, newname)
return (endtable)
make_profiles("./training", "./trigram-models", 20)

To create a directory, I would use the following format which relies on a try / catch and prevents an error if the directory already exists:
dirName = 'tempDir'
try:
# Create target Directory
os.mkdir(dirName)
print("Directory " , dirName , " Created ")
except FileExistsError:
print("Directory " , dirName , " already exists")
To change your directory you can use the following:
os.chdir(directoryLocation)
I recommend reading chapter 8 in automating the boring stuff with python.
I hope this helps. If you have any questions please don't hesitate to ask.

First of all, be sure to indent all the code in your method for it to be appropriately enclosed.
You are also passing relative paths (datafolder, profilefolder) for your folders as method arguments, so you should use them inside the method instead.
Lastly, to create a file in a folder, I would recommend using the following algorithm:
file_path = '/'.join(profilefolder, newname)
with open(file_path, 'w') as ouf:
ouf.write(endtable)
You will probably need to replace "endtable" with a string representation of your data.
Hope it helps.

As to clarify on toti08's answer, you should replace os.absdir with os.path.absdir.
filelist = [os.path.abspath(f) for f in os.listdir(data_folder)]
instead of
filelist = [os.abspath(f) for f in os.listdir(data_folder)]

Your function is not using the argument profileFolder, where you specify the name of the output directory. So first of all you should use this information for creating a folder before processing your files.
So first thing would be to create this output directory.
Second is to save your files there, and to do that you need to append the file name to the output directory. Something like this:
def make_profiles(data_folder, output_folder, size):
filelist = []
for file in os.listdir(data_folder):
filelist.append(file)
# Create output folder
if not os.path.exists(output_folder):
os.mkdir(output_folder)
for file in filelist:
filen = "./training/"+file
#print("fi", filen)
splitname = str(file).split('-')
# Create new file by appending name to output_folder
newname = os.path.join(output_folder, splitname[0] + "." + str(size) + '.txt')
return (endtable)
make_profiles(./training, './trigram-models', 20)
Note that you can also specify the relative folder name (i.e. "trigram-models" only) and then create the output directory by appending this name to the current path:
output_folder = os.path.join(os.getcwd(), output_folder)
Also (not related to the question) this piece of code could be optimized:
filelist = []
for file in os.listdir(data_folder):
filelist.append(file)
os.listdir already returns a list, so you could directly write:
filelist = os.listdir(data_folder)
But since you're interested in the absolute path of each file you could better do:
filelist = [os.path.abspath(f) for f in os.listdir(data_folder)]
where you basically take each file returned by os.listdir and you append its absolute path to your file list. Doing this you could avoid the line filen = "./training/"+file.
So in the end your code should look something like this:
def make_profiles(data_folder, output_folder, size):
filelist = [os.abspath(f) for f in os.listdir(data_folder)]
# Create output folder
if not os.path.exists(output_folder):
os.mkdir(output_folder)
for file in filelist:
splitname = str(file).split('-')
# [...add other pieces of code]
# Create new file by appending name to output_folder
newname = os.path.join(output_folder, splitname[0] + "." + str(size) + '.txt')
# [...add other pieces of code]
return (endtable)
make_profiles(./training, './trigram-models', 20)

python: get file path and create a new file in the same directory

So I know how to get a location of the needed file, I'm doing this with
file_name = os.path.splitext(os.path.basename(file_full_name))[0]
but what I need to do is:
Get the file name, which I have
Get path of the file, which I also know how to get
Create a file in the same directory with a modified name, for example lets say I have a file called "data.csv" on Desktop. I would need to create a file called "data - results.csv" to Desktop.
I have only tried printing the new name, but the only result I have got with this code:
myresultcsvfile = os.path.splitext(os.path.basename(file_name))[0] + " - results.csv"
is this:
myfile: ('Book1 - Copy', ' - results.csv')
I'm clearly doing something wrong but I cant figure out what. And that's just the file name, I also need to add full path of the parent file to it (so that the end result would be "C:\users[username]\desktop\Book1 - copy - results.csv" in this case)

Try this:
full = 'C:\\ .. put here your path .. \\data.csv'
dir_name = os.path.dirname(full)
file_name = os.path.splitext(os.path.basename(full))[0]
output_file = dir_name + '\\' + file_name + ' - results.csv'

Python: How to copy specific files from one location to another and keep directory structure

Hi I'm struggling with some python code for to copy specific files in a folder to another folder whilst keeping the directory structure.
I'm learning so this code is put together using various code snippets I've found, I couldn't find anything that exactly matched my circumstance and I don't understand python enough yet to understand where I've gone wrong
def filtered_copy(src_dir, dest_dir, filter):
print 'Copying files named ' + filter + ' in ' + src_dir + ' to ' + dest_dir
ignore_func = lambda d, files: [f for f in files if isfile(join(d, f)) and f != filter]
if os.path.exists(dest_dir):
print 'deleting existing data'
shutil.rmtree(dest_dir)
copytree(src_dir, dest_dir, ignore=ignore_func)
Executing this code like this
filtered_copy(c:\foldertosearch, c:\foldertocopyto, 'settings.xml')
does copy across the file I want but does not copy across the parent folder i.e. the src_dir, so the result I'm trying to achieve is:
c:\foldertocopyto\foldertosearch\settings.xml
*** Edit - to clarify this is a script that will be used on multiple operating systems
So if the folder structure was more complex i.e.
Parent folder
-subfolder
--subsubfolder
----subsubsubfolder
------settings.xml
and I ran
filtered_copy(subsubsubfolder, foldertocopyto, 'settings.xml')
I would want the new folder structure to be
foldertocopyto (there could be more parent folders above this or not)
--subsubsubfolder
----settings.xml
In other words the folder I search for a specific file in should also be copied across and if that folder already exists it should be deleted before the folder and file is copied across
I assumed copytree() would do this part - but obviously not!
*** end of Edit
*** Latest code changes
This works but I'm sure it's a long-winded way, also it copies blank folders, presumably because of the copytree() execution, I'd prefer just the folder that's being searched and the filtered file...
def filtered_copy(src_dir, dest_dir, filter):
foldername = os.path.basename(os.path.normpath(src_dir))
print 'Copying files named ' + filter + ' in ' + src_dir + ' to ' + dest_dir + '/' + foldername
ignore_func = lambda d, files: [f for f in files if isfile(join(d, f)) and f != filter]
if os.path.exists(dest_dir + '/' + foldername):
print 'deleting existing data'
shutil.rmtree(dest_dir)
copytree(src_dir, dest_dir + '/' + foldername, ignore=ignore_func)
*** end of latest code changes

You can use distutils.dir_util.copy_tree. It works just fine and you don't have to pass every argument, only src and dst are mandatory.
However in your case you can't use a similar tool like shutil.copytree because it behaves differently: as the destination directory must not exist this function can't be used for overwriting its contents
Give a try to a sample code below:
def recursive_overwrite(src, dest, ignore=None):
if os.path.isdir(src):
if not os.path.isdir(dest):
os.makedirs(dest)
files = os.listdir(src)
if ignore is not None:
ignored = ignore(src, files)
else:
ignored = set()
for f in files:
if f not in ignored:
recursive_overwrite(os.path.join(src, f),
os.path.join(dest, f),
ignore)
else:
shutil.copyfile(src, dest)

Why is my write function not creating a file?

According to all the sources I've read, the open method creates a file or overwrites one with an existing name. However I am trying to use it and i get an error:
File not found - newlist.txt (Access is denied)
I/O operation failed.
I tried to read a file, and couldn't. Are you sure that file exists? If it does exist, did you specify the correct directory/folder?
def getIngredients(path, basename):
ingredient = []
filename = path + '\\' + basename
file = open(filename, "r")
for item in file:
if item.find("name") > -1:
startindex = item.find("name") + 5
endindex = item.find("<//name>") - 7
ingredients = item[startindex:endindex]
ingredient.append(ingredients)
del ingredient[0]
del ingredient[4]
for item in ingredient:
printNow(item)
file2 = open('newlist.txt', 'w+')
for item in ingredient:
file2.write("%s \n" % item)
As you can see i'm trying to write the list i've made into a file, but its not creating it like it should. I've tried all the different modes for the open function and they all give me the same error.

It looks like you do not have write access to the current working directory. You can get the Python working directory with import os; print os.getcwd().
You should then check whether you have write access in this directory. This can be done in Python with
import os
cwd = os.getcwd()
print "Write access granted to current directory", cwd, '>', os.access(cwd, os.W_OK)
If you get False (no write access), then you must put your newfile.txt file somewhere else (maybe at path + '/newfile.txt'?).

Are you certain the directory that you're trying to create the folder in exists?
If it does NOT... Then the OS won't be able to create the file.

This looks like a permissions problem.
either the directory does not exist or your user doesn't have the permissions to write into this directory .

I guess the possible problems may be:
1) You are passing the path and basename as parameters. If you are passing the parameters as strings, then you may get this problem:
For example:
def getIngredients(path, basename):
ingredient = []
filename = path + '\\' + basename
getIngredients("D","newlist.txt")
If you passing the parameters the above way, this means you are doing this
filename = "D" + "\\" + "newlist.txt"
2) You did not include a colon(:) after the path + in the filename.
3) Maybe, the file does not exist.