I am having an issue of trying to load CSV files in a python script when trying to run the file from outside the directory of the script called main.py and the CSV files. (CSV files and main are in the same directory) I think the same issues as this SO post which doesn't appear to contain a solution.
If I run this from terminal:
$ python /home/bbartling/Desktop/Load-Shredder/Load-Shift/main.py
I get some CSV file loading errors [Errno 2] No such file or directory: 'addresses_jci.csv'
But if I run the script in the directory of Load-Shift $ python main.py it works just fine.
How do I fix my script to accommodate this? I added this to the top of my script:
script_dir = os.path.abspath( os.path.dirname( __file__ ) )
print("script directory: ",script_dir)
which prints:
script directory: /home/bbartling/Desktop/Load-Shredder/Load-Shift
But still no luck. Any ideas to try?
Edit
CSV file loading function in main.py
dir_path = os.path.dirname(os.path.realpath(__file__))
filename = os.path.join(dir_path, f'log_{dt_string}.log')
script_dir = os.path.abspath( os.path.dirname( __file__ ) )
print("script directory: ",script_dir)
def load_addresses(csv_file):
try:
print(f"Loading csv file: {csv_file}!")
os.path.join(script_dir, f'{csv_file}.csv')
with open(f'{csv_file}', newline='') as f:
reader = csv.reader(f)
data = list(reader)
# flattens list of lists
csv_file = sum(data, [])
print(f"{csv_file} addresses loaded: ",csv_file)
except Exception as error:
print(f"CSV file load error: {error}")
csv_file = [] # errors out
return csv_file
The problem in your code, I think you did not update the csv_file variable update:
csv_file = os.path.join(script_dir, f'{csv_file}.csv')
Though it can be more simpler:
import os
script_path = os.path.dirname(__file__)
filename = 'sample'
file_path = os.path.join(script_path, f'{filename}.csv')
print(file_path)
filepaths in general are a bit confusing in programming languages, so this is a very common problem for beginners.
I typically have a fixed working directory for my projects, and tend to access all files relative to that working directory. If you can implement it, this is the simplest solution.
$ cd /home/bbartling/Desktop/Load-Shredder/Load-Shift/
$ python main.py
can give the desired solution, but depending upon your project this may or may not be feasible. This is by no means a hard-code or a hot fix, and perfectly valid for my projects. However, if you're making a reusable shell script, you probably don't want this.
I've used Path(__file__).parent before to get the directory the file is running in, but IDK how the speed compares to the way you compute it.
I've used this a couple times before V
import os
HERE = os.path.dirname(os.path.realpath(__file__))
with open(os.path.join(HERE, 'mkdocs.yml')) as fl:
print(fl.read())
You can change the filename here to your script's path, and that will work. If you know where it will be stored relative to your main.py dir, you can hard-code it. otherwise get it from the command line or console, whichever you prefer. Use the relative path from your main.py directory to your csv here.
This is working! in the function to load a file:
THIS WORKS with specifing object to reference for the os.path.join:
full_csv_path = os.path.join(dir_path, f'{csv_file}')
THIS DOESNT WORK without specifying an object for the program to reference:
os.path.join(dir_path, f'{csv_file}')
Example below that also includes logging:
import csv, os, logging
# datetime object containing current date and time
now = datetime.now()
dt_string = now.strftime("%m_%d_%Y %H_%M_%S")
dir_path = os.path.dirname(os.path.realpath(__file__))
filename = os.path.join(dir_path, f'log_{dt_string}.log')
# Logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
file_handler = logging.FileHandler(filename)
file_handler.setLevel(logging.INFO)
file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
logger.addHandler(file_handler)
print("dir_path is: ",dir_path)
def load_addresses(csv_file):
try:
print(f"Loading csv file: {csv_file}!")
full_csv_path = os.path.join(dir_path, f'{csv_file}')
print("full_csv_path: ",full_csv_path)
with open(full_csv_path, newline='') as f:
reader = csv.reader(f)
data = list(reader)
# flattens list of lists
csv_file = sum(data, [])
print(f"{csv_file} addresses loaded: ",csv_file)
except Exception as error:
print(f"CSV file load error: {error}")
csv_file = [] # errors out
return csv_file
You can use pandas , os and fnmatch for this purpose as given below,
import pandas as pd
import os,fnmatch
# change path to the folder from another folder
os.chdir("/path_to_the_folder_having_files_from_another_folder")
# print the current working directory to check if you have arrived
# in the right folder/directory
print(os.getcwd())
# then use fnmatch package to get the CSV files only
# or you can make some other pattern using some prefix or suffix
# like *this.csv or *that.csv
files = fnmatch.filter(os.listdir('.'), '*.csv')
# print the files to check that you have captured the right files
files
# read your files using pandas
# be careful with `sep`, you need to check specifically for your
# files
csv_files = [pd.read_csv(f, low_memory=False,sep = "\t",header=0 for f in files]
Related
I have been attempting to change all files in a folder of a certain type to another and then save them to another folder I have created.
In my example the files are being changed from '.dna' files to '.fasta' files. I have successfully completed this via this code:
files = Path(directory).glob('*.dna')
for file in files:
record = snapgene_file_to_seqrecord(file)
fasta = record.format("fasta")
print(fasta)
My issue is now with saving these files to a new folder. My attempt has been to use this:
save_path = Path('/Users/user/Documents...')
for file in files:
with open(file,'w') as a:
record = snapgene_file_to_seqrecord(a)
fasta = record.format("fasta").read()
with open(save_path, file).open('w') as f:
f.write(fasta)
No errors come up but it is definitely not working. I can see that there may be an issue with how I am writing this but I can't currently think of a better way to do it.
Thank you in advance!
Hi, You can use os lib to rename the file with the new extension (type)
import os
my_file = 'my_file.txt'
base = os.path.splitext(my_file)[0]
os.rename(my_file, base + '.bin')
And you can use shutil lib to move the file to a new directory.
import shutil
# absolute path
src_path = r"E:\pynative\reports\sales.txt"
dst_path = r"E:\pynative\account\sales.txt"
shutil.move(src_path, dst_path)
Hope that can be of help.
I am currently accessing a script that opens a file in the directory it's located in. I am accessing this file from both the main.py file located in the same directory, as well as a testfile which is located in a "Test" subdirectory. Trying to use a file from the Test subdirectory to call the function that opens the file causes the script to try and open it from the Test directory instead of the super directory, since I am opening the file simply by calling it as following:
with open(filename,"w") as f:
Is there a way to define the location of the file in a way that makes it possible for the script opening it to be called from anywhere?
Use __file__ to get the path to the current script file, then find the file relative to that:
# In main.py: find the file in the same directory as this script
import os.path
open(os.path.join(os.path.dirname(__file__), 'file.txt'))
# In Test/test.py: find the file one directory up from this script
import os.path
open(os.path.join(os.path.dirname(__file__), '..', 'file.txt'))
just give the absolute file path instead of giving a relative one
for eg
abs_path = '/home/user/project/file'
with open(abs_path, 'r') as f:
f.write(data)
Try specifying the path:
import os
path = 'Your path'
path = os.path.abspath(path)
with open(path, 'w') as f:
f.write(data)
From what I understood your file is in a directory parent_directory/file_name.txt
and in another folder parent_directory/sub_directory/file_name.txt. All you have to do is paste the below code in both parent and sub directories.
import os
file_name = 'your_file_name'
# if the file is in current directory set the path to file_name
if file_name in os.listdir(os.getcwd()):
path = file_name
# if the path is not in current directory go back to parent directory set the path to parent directory
else:
path = os.path.abspath(os.path.join(os.getcwd(), os.pardir))
print('from',os.getcwd())
with open(path, 'r') as filename:
print(filename.read())
I'm struggling with reading a file in python, the py file and CSV file are in the same folder but the VSCode makes an error and can't find the file:
import csv
with open('file.csv','r') as f:
reader = reader(f)
...
how can I fix this??
and the error is:
Exception has occurred: FileNotFoundError
[Errno 2] No such file or directory: 'file.csv'
If you run:
import os
os.getcwd()
You'll find out your current working directory which I assume is not the one you were expecting. If you're running the python script through VS code it could be using it could be the directory which you have open on the left hand side.
So either run the python using the correct working directory or use an absolute path like this:
import csv
with open('pathname/file.csv','r') as f:
reader = reader(f)
There might be an issue with your relative path settings.
Try this:
import os
import csv
dir = os.path.dirname(__file__)
filename = os.path.join(dir, 'file.csv')
with open(filename,'r') as f:
reader = reader(f)
Are you using spyder?
If so, please check if the current working path is the path your py file locates.
import csv
with open('file.csv','r') as f:
reader = csv.reader(f)
in this case your file.csv should be in folder where is your python script (current working folder)
or, instead of 'file.csv' you can put absolute path
I currently have a program that collects data from .txt files in a folder and then saves that data to a csv file. Due to how I am planning on distributing this program, I need the Python file to live in the folder where these .txt files are located. However, I need the .csv files to be thrown to an absolute file path rather than being created in the same folder as the Python script and .txt documents. Here is what I have currently coded,
def write_to_csv(journal_list):
#writes a list of journal dictionaries to a csv file.
import csv
username = "Christian"
csv_name = username + ".csv"
myFile = open(csv_name, 'w')
with myFile:
myFields = ["filename", "username", "project_name", "file_path",
"date", "start_time", "end_time", "length_of_revit_session",
"os_version", "os_build", "revit_build", "revit_branch",
"cpu_name", "cpu_clockspeed", "gpu_name", "ram_max", "ram_avg", "ram_peak",
"sync_count", "sync_time_total", "sync_time_peak", "sync_time_avg",
"commands_total", "commands_hotkey_percentage", "commands_unique",
"commands_dynamo", "commands_escape_key", "commands_most_used"]
writer = csv.DictWriter(myFile, fieldnames=myFields)
writer.writeheader()
for item in journal_list:
try:
writer.writerow(item)
except:
print("error writing data to:", item)
I appreciate the help.
USing os.path.join() you can select your desire path for your file to be written. Here is an example:
import os
desier_path = '/home/foo/'
file_path = os.path.join(dest_dir, csv_name)
with open(file_path, 'w'):
...
Consider asking for the path from the script, and setting a default if not passed in. This would make your script a lot more flexible than having the path coded in it.
You can use the click package which simplifies this a bit.
import os
import click
def write_to_csv(path, journal_list):
# .. your normal code
file_path = os.path.join(path, '{}.csv'.format(username))
# .. rest of your code here
#click.command()
#click.option('--output_dir', default='/home/foo/bar/', help='Default path to save files')
def main(output_dir):
write_to_csv(output_dir)
if __name__ == '__main__':
main()
Normally, executing the following code will pickle an object to a file in my current directory:
fp = open('somefile.txt', 'wb')
pickle.dump(object, fp)
How do I re-direct the output from pickle.dump to a different directory?
with open('/full/path/to/file', 'wb') as f:
pickle.dump(object, f)
How about combination of pathlib and with, I think it more flexible and safer.
# from python 3.4
from pathlib import Path
my_path = Path("{path to you want to set root like . or ~}") / "path" / "to" / "file"
with my_path.open('wb') as fp:
pickle.dump(object, fp)
If you wish to save the file to a sub-folder located inside the folder containing your code you can use the pathlib module. This will allow the code to work even if its location is moved on your computer, or you code is added to a different machine.
import the module:
from pathlib import Path
Set root equal to your current folder:
root = Path(".")
Create a path to your sub-folder and file name:
my_path = root / "my_sub_folder" / "my_file_name"
Open your file, dump data to your file, close your file:
my_file = open(my_path, 'wb')
my_file = pickle.dump("data_to_save", my_file)
my_file.close()
Note if your my_file doesn't currently exist you will want to create it before running this code. *
You can just try
import pickle
filepath = r'<FolderName>/<FileName>.pkl'
pickle.dump(<model>, open(filepath, 'wb'))
so final code look likes
If your Folder name is Models
and you wanted to pickle randomForest Model
import pickle
filepath = r'Models/rfPickle.pkl'
pickle.dump(randomForest, open(filepath, 'wb'))