Normally, executing the following code will pickle an object to a file in my current directory:
fp = open('somefile.txt', 'wb')
pickle.dump(object, fp)
How do I re-direct the output from pickle.dump to a different directory?
with open('/full/path/to/file', 'wb') as f:
pickle.dump(object, f)
How about combination of pathlib and with, I think it more flexible and safer.
# from python 3.4
from pathlib import Path
my_path = Path("{path to you want to set root like . or ~}") / "path" / "to" / "file"
with my_path.open('wb') as fp:
pickle.dump(object, fp)
If you wish to save the file to a sub-folder located inside the folder containing your code you can use the pathlib module. This will allow the code to work even if its location is moved on your computer, or you code is added to a different machine.
import the module:
from pathlib import Path
Set root equal to your current folder:
root = Path(".")
Create a path to your sub-folder and file name:
my_path = root / "my_sub_folder" / "my_file_name"
Open your file, dump data to your file, close your file:
my_file = open(my_path, 'wb')
my_file = pickle.dump("data_to_save", my_file)
my_file.close()
Note if your my_file doesn't currently exist you will want to create it before running this code. *
You can just try
import pickle
filepath = r'<FolderName>/<FileName>.pkl'
pickle.dump(<model>, open(filepath, 'wb'))
so final code look likes
If your Folder name is Models
and you wanted to pickle randomForest Model
import pickle
filepath = r'Models/rfPickle.pkl'
pickle.dump(randomForest, open(filepath, 'wb'))
Related
f = open("demofile3.txt", "w")
f.writelines(["\nSee you soon!", "\nOver and out."])
f.close()
`
I have been trying to create or open a file and write something in it but it does not seem to do anything.
it's about pwd.
But here is a method: use full path.
For example:
from pathlib import Path
folder = Path(__file__).parent
your_txt = folder / "demofile3.txt"
with open(your_txt, "w") as f:
f.writelines(["\nSee you soon!", "\nOver and out."])
I am having an issue of trying to load CSV files in a python script when trying to run the file from outside the directory of the script called main.py and the CSV files. (CSV files and main are in the same directory) I think the same issues as this SO post which doesn't appear to contain a solution.
If I run this from terminal:
$ python /home/bbartling/Desktop/Load-Shredder/Load-Shift/main.py
I get some CSV file loading errors [Errno 2] No such file or directory: 'addresses_jci.csv'
But if I run the script in the directory of Load-Shift $ python main.py it works just fine.
How do I fix my script to accommodate this? I added this to the top of my script:
script_dir = os.path.abspath( os.path.dirname( __file__ ) )
print("script directory: ",script_dir)
which prints:
script directory: /home/bbartling/Desktop/Load-Shredder/Load-Shift
But still no luck. Any ideas to try?
Edit
CSV file loading function in main.py
dir_path = os.path.dirname(os.path.realpath(__file__))
filename = os.path.join(dir_path, f'log_{dt_string}.log')
script_dir = os.path.abspath( os.path.dirname( __file__ ) )
print("script directory: ",script_dir)
def load_addresses(csv_file):
try:
print(f"Loading csv file: {csv_file}!")
os.path.join(script_dir, f'{csv_file}.csv')
with open(f'{csv_file}', newline='') as f:
reader = csv.reader(f)
data = list(reader)
# flattens list of lists
csv_file = sum(data, [])
print(f"{csv_file} addresses loaded: ",csv_file)
except Exception as error:
print(f"CSV file load error: {error}")
csv_file = [] # errors out
return csv_file
The problem in your code, I think you did not update the csv_file variable update:
csv_file = os.path.join(script_dir, f'{csv_file}.csv')
Though it can be more simpler:
import os
script_path = os.path.dirname(__file__)
filename = 'sample'
file_path = os.path.join(script_path, f'{filename}.csv')
print(file_path)
filepaths in general are a bit confusing in programming languages, so this is a very common problem for beginners.
I typically have a fixed working directory for my projects, and tend to access all files relative to that working directory. If you can implement it, this is the simplest solution.
$ cd /home/bbartling/Desktop/Load-Shredder/Load-Shift/
$ python main.py
can give the desired solution, but depending upon your project this may or may not be feasible. This is by no means a hard-code or a hot fix, and perfectly valid for my projects. However, if you're making a reusable shell script, you probably don't want this.
I've used Path(__file__).parent before to get the directory the file is running in, but IDK how the speed compares to the way you compute it.
I've used this a couple times before V
import os
HERE = os.path.dirname(os.path.realpath(__file__))
with open(os.path.join(HERE, 'mkdocs.yml')) as fl:
print(fl.read())
You can change the filename here to your script's path, and that will work. If you know where it will be stored relative to your main.py dir, you can hard-code it. otherwise get it from the command line or console, whichever you prefer. Use the relative path from your main.py directory to your csv here.
This is working! in the function to load a file:
THIS WORKS with specifing object to reference for the os.path.join:
full_csv_path = os.path.join(dir_path, f'{csv_file}')
THIS DOESNT WORK without specifying an object for the program to reference:
os.path.join(dir_path, f'{csv_file}')
Example below that also includes logging:
import csv, os, logging
# datetime object containing current date and time
now = datetime.now()
dt_string = now.strftime("%m_%d_%Y %H_%M_%S")
dir_path = os.path.dirname(os.path.realpath(__file__))
filename = os.path.join(dir_path, f'log_{dt_string}.log')
# Logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
file_handler = logging.FileHandler(filename)
file_handler.setLevel(logging.INFO)
file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
logger.addHandler(file_handler)
print("dir_path is: ",dir_path)
def load_addresses(csv_file):
try:
print(f"Loading csv file: {csv_file}!")
full_csv_path = os.path.join(dir_path, f'{csv_file}')
print("full_csv_path: ",full_csv_path)
with open(full_csv_path, newline='') as f:
reader = csv.reader(f)
data = list(reader)
# flattens list of lists
csv_file = sum(data, [])
print(f"{csv_file} addresses loaded: ",csv_file)
except Exception as error:
print(f"CSV file load error: {error}")
csv_file = [] # errors out
return csv_file
You can use pandas , os and fnmatch for this purpose as given below,
import pandas as pd
import os,fnmatch
# change path to the folder from another folder
os.chdir("/path_to_the_folder_having_files_from_another_folder")
# print the current working directory to check if you have arrived
# in the right folder/directory
print(os.getcwd())
# then use fnmatch package to get the CSV files only
# or you can make some other pattern using some prefix or suffix
# like *this.csv or *that.csv
files = fnmatch.filter(os.listdir('.'), '*.csv')
# print the files to check that you have captured the right files
files
# read your files using pandas
# be careful with `sep`, you need to check specifically for your
# files
csv_files = [pd.read_csv(f, low_memory=False,sep = "\t",header=0 for f in files]
I am creating a zip file using the zipfile module. It works like a charm. but that's file, saved in the executed script place.
my script path is a:
[b]c:/User/Administrator/[/b]script.py
and the zipfile saved in:
[b]c:/User/Administrator/[/b]backup.zip
but I want, [b]creating a zipfile, in another path[/b], like this:
[b]d:/backups/[/b]backup.zip
my code like this:
import zipfile
zip_file = zipfile.ZipFile("backup.zip", 'w')
with zip_file:
for file in filePaths:
zip_file.write(file)
my question is a how can I adding custom path for saving the zip_file. because I have not an enough space in C:
tnx a lot.
Give the path you want to ZipFile function.
When you give only the name of the file, it will save the file in the current directory which the program is running.
Do this instead:
import zipfile
# For example you want to save it in drive 'D'
path = "D:\\PathToYourDir\\backup.zip"
zip_file = zipfile.ZipFile(path, 'w')
with zip_file:
for file in filePaths:
zip_file.write(file)
I'm struggling with reading a file in python, the py file and CSV file are in the same folder but the VSCode makes an error and can't find the file:
import csv
with open('file.csv','r') as f:
reader = reader(f)
...
how can I fix this??
and the error is:
Exception has occurred: FileNotFoundError
[Errno 2] No such file or directory: 'file.csv'
If you run:
import os
os.getcwd()
You'll find out your current working directory which I assume is not the one you were expecting. If you're running the python script through VS code it could be using it could be the directory which you have open on the left hand side.
So either run the python using the correct working directory or use an absolute path like this:
import csv
with open('pathname/file.csv','r') as f:
reader = reader(f)
There might be an issue with your relative path settings.
Try this:
import os
import csv
dir = os.path.dirname(__file__)
filename = os.path.join(dir, 'file.csv')
with open(filename,'r') as f:
reader = reader(f)
Are you using spyder?
If so, please check if the current working path is the path your py file locates.
import csv
with open('file.csv','r') as f:
reader = csv.reader(f)
in this case your file.csv should be in folder where is your python script (current working folder)
or, instead of 'file.csv' you can put absolute path
I have a folder that stores a json file in my django application folder, ie, test_data/data.json.
In my tests.py, I am trying to read this file using the following code:
with open('/test_data/data.json', 'r') as f:
self.response_data = json.load(f)
However, I keep on getting the following error:
FileNotFoundError: [Errno 2] No such file or directory: '/test_data/data.json'
What am I doing wrong? Thanks.
Edit: I tried removing the leading slash, yet I still get the same error.
import os
try this
with open(os.getcwd() + '/test_data/data.json', 'r') as f:
self.response_data = json.load(f)
If you're opening files in directories close to where your code is, it is common to place
import os
DIRNAME = os.path.dirname(__file__) # the directory of this file
at the top of the file.
Then you can open files in a test_data subdirectory with
with open(os.path.join(DIRNAME, 'test_data', 'data.json'), 'rb') as fp:
self.response_data = json.load(fp)
you probably want to open json files, which should be utf-8 encoded, in 'rb' (read-binary) mode.