I am reading a yaml file like so in Python3:
def get_file():
file_name = "file.yml"
properties_stream = open(file_name, 'r')
properties_file = yaml.safe_load(properties_stream)
properties_stream.close()
print(properties_file)
return properties_file
When I update file.yml by adding keys or deleting all the contents of the file, the "print(properties_file)" statement is still retaining the contents of the yaml file the first time I ran this code.
Any ideas why this might be happening?
Anthon was correct. Turned out I had two folders with the same name in my project directory in a different subdirectory and it was reading from the wrong file.
Related
I am trying to open a CSV file that I recently saved using Python. Here is global directory:
So the folder is called Parsing Data Using Python, and there are 3 files inside, we only concern ourselves with the codealong.py file, which is some Python code that I want to run to open the 'patrons.csv' file.
Here is the code in the codealong.py file:
import csv
html_output = ''
with open('patrons.csv','r') as data_file:
csv_data = csv.reader(data_file)
print(list(csv_data))
When I run this, I get the Error2: No such file or directory: 'patrons.csv'
Any ideas why I am getting this? Because I am saving patrons.csv in the same directory as codealong.py I thought the file would be detected!
One approach would be to set the working directory to be the same location as where your script is. To do this for any script, add this to the top:
import os
os.chdir(os.path.dirname(os.path.abspath(__file__)))
This takes the full name of where your script is located, takes just the path element and sets the current working directory to that.
You then would not need to specify the full path to your file.
Another approach using pathlib.
import csv
from pathlib import Path
file = Path(__file__).parent / "patrons.csv"
with file.open("r", encoding="utf-8") as data_file:
csv_data = csv.reader(data_file)
print(list(csv_data))
SO members...how can i read latest json file in a directory one time only (if no new file print something). So far I can only read the latest file ...The sample script (run every 45mins) below open and read latest json file in a directory. In this case latest file is file3.json (json file created every 30mins). Thus, if file4 is not created for some reason (for example server fail to create new json file). If the script run again.. it will still read the same last file3.
files in directory
file1.json
file2.json
file3.json
The script below able to open and read latest json file created in the directory.
import glob
import os
import os.path
import datetime, time
listFiles = glob.iglob('logFile/*.json')
latestFile = max(listFiles, key=os.path.getctime)
with open(latestFile, 'r') as f:
mydata = json.load(f)
print(mydata)
To ensure the script will only read newest file and read the newest file one time only...aspect something below:-
listFiles = glob.iglob('logFile/*.json')
latestFile = max(listFiles, key=os.path.getctime)
if latestFile newer than previous open/read file: # Not sure to compare the latest file with the previous file.
with open(latestFile, 'r') as f:
mydata = json.load(f)
print(mydata)
else:
print("no new file created")
Thank you for your help. Example solution would be good to share.
I can't figure out the solution...seems simple but few days try n error without any luck.
(1)Make sure read latest file in directory
(2)Make sure read file/s that may miss to read (due to script fail to run)
(3)Only read once all the files and if no new file give warning.
Thank you.
After SO discussion and suggestion, I got few methods to resolve or at least to accommodate some of the requirement. I just move files that have been processed. If no file create, script will run nothing and if script fail and once normalize it will run and read all related files available. I think its good for now. Thank you guyz...
Below is the answer rather an approach, I would like to propose:
The idea is as follows:
Every log file that is written to a directory can have a key-val in it called "creation_time": timestamp (fileX.json that gets stored in the server). Now, your script runs at 45min to obtain the file which is dumped to a directory. In normal cases, you must be able to read the file, and finally, when you exit the script you can store the last read filename and the creation_time taken from the fileX.json into a logger.json.
An example for a logger.json is as follows:
{
"creation_time": "03520201330",
"file_name": "file3.json"
}
Whenever a server fail or any delay occurs, there could be a rewritten of the fileX.json or new fileX's.json would have been created in the directory. In these situations, you would first open the logger.json and obtain both the timestamp and last filename as shown in the example above. By using the last filename, you can compare the old timestamp that is present in logger with the new timestamp in fileX.json. If they match basically there is no change you only read ahead files and rewrite the logger.
If that is not the case you would re-read the last fileX.json again and proceed to read other ahead files.
I'm using configuration files via configparser in python3.7.1.
I have a variable in one of the files that I would like to modify at runtime an was wondering what is the best way to do it.
Right now I rewrite the entire file as so:
config = configparser.ConfigParser()
config.read('config.ini')
config['Categorie']['new_variable'] = variable
with open('config.ini', 'w') as configfile:
config.write(configfile)
I have 2 concerns with this method:
potentially losing the config on an error
deleting comments in the file each time
configparser doesn't store comments, so you're stuck on that one unless you don't use this module... Or you could extract the comments and reinject them in the file afterwards (losing the position of the comments, but keeping the contents)
To avoid losing the configuration in case of an error (disk full or other) you could save as another name, delete the current file and rename.
conf_file = 'config.ini'
with open(conf_file+'.bak', 'w') as configfile:
config.write(configfile)
if os.path.exists(conf_file):
os.remove(conf_file) # else rename won't work when target exists
os.rename(conf_file+'.bak',conf_file)
this method is safe. If file cannot be written, the previous file isn't clobbered. Worst case is (if someone pulls the plug at the exact moment when the original file is removed) the .bak file (with proper new contents) remains.
An alternative would be to rename existing .ini file, write the new one, and delete the old .ini file when the file has been successfully written.
All operations are performed on the same drive, so there's no more disk access (except for the renaming), even if the file is huge.
config.ini
[DEFAULT]
ServerAliveInterval = 45
Compression = yes
CompressionLevel = {somelevel}
ForwardX11 = yes
python code
content = open('config.ini').read()
somelevel = 34
filled = content.format(**locals())
gg = ConfigParser()
gg.read_string(filled)
However, do NOT use **locals() for real code.
Do try to look for a better way, this is dirty.
edit : I tried do do some simple f string hacks, they fail due to escaping issues.
Could you give me any explanation, please
I have a part of the code using ConfigParser
File that I am reading in in the directory ~/ui/config.cfg
after I am calling function below and I get a new file in the directory where my module presents which is (~/ui/helper/config.cfg)
class CredentialsCP:
def __init__(self, cloud_name=None):
self.config = ConfigParser.ConfigParser()
self.cloud_name = cloud_name
def rewrite_pass_in_config(self, cloud, new_pass):
if new_pass:
self.config.read('config.cfg')
self.config.set(cloud, 'password', new_pass)
with open('config.cfg', 'wb') as configfile:
self.config.write(configfile)
else:
return False
It creates a new file in the directory where I am running my code from, but I need the same file to be re-written. How can I do that? And why I keep getting the same behavior ?
Since you're using the same file name (config.cfg) when reading and writing (and also, not altering the working dir), you are operating on the same file. Since you're writing the ~/ui/helper/config.cfg file (it gets created after running the code), that's the one that you're reading from too.
So, you are not opening (for reading) the file that you think you are. From [Python]: read(filenames, encoding=None)
If a file named in filenames cannot be opened, that file will be ignored.
...
If none of the named files exist, the ConfigParser instance will contain an empty dataset.
You're reading from a file that doesn't exist, which yields an empty config, and that is the config you're writing in your (desired) file. In order to fix your problem, specify the desired file by its full or relative name. You could have something like:
In __init__ :
self.file_name = os.path.expanduser("~/ui/config.cfg") # Or any path processing code
In rewrite_pass_in_config:
Read:
self.config.read(self.file_name)
Write
with open(self.file_name, "wb") as configfile:
self.config.write(configfile)
I was working on saving text to different files. so, now I already created several files and each text file has some texts/paragraph in it. Now, I just want to save these files to a directory. I already created a self-defined directory, but now it is empty. I want to save these text files into my directory.
The partial code is below:
for doc in root:
docID = doc.find('DOCID').text.strip()
text = doc.find('TEXT').text,strip()
f = open("%s" %docID, 'w')
f.write(str(text))
Now, I created all the files with text in it. and I also have a blank folder/directory now. I just don't know how to put these files into the directory.
I would be appreciate it.
========================================================================
[Solved] Thank you guys for your all helping! I figured it out. I just edit my summary here. I got a few problems.
1. my docID was saved as tuple. I need to convert to string without any extra symbol. here is the reference i used: https://stackoverflow.com/a/17426417/9387211
2. I just created a new path and write the text to it. i used this method: https://stackoverflow.com/a/8024254/9387211
Now, I can share my updated code and there is no more problem here. Thanks everyone again!
for doc in root:
docID = doc.find('DOCID').text.strip()
did = ''.join(map(str,docID))
text = doc.find('TEXT').text,strip()
txt = ''.join(map(str,docID))
filename = os.path.join(dst_folder_path, did)
f = open(filename, 'w')
f.write(str(text))
Suppose you have all the text files in home directory (~/) and you want to move them to /path/to/dir folder.
from shutil import copyfile
import os
docid_list = ['docid-1', 'docid-2']
for did in docid_list:
copyfile(did, /path/to/folder)
os.remove(did)
It will copy the docid files in /path/to/folder path and remove the files from the home directory (assuming you run this operation from home dir)
You can frame the file path for open like
doc_file = open(<file path>, 'w')