I'm quite new to Python and Pi'ing, and would like some help with appending a text file on a Raspberry Pi. My script will call a GET or a POST REST API and write the time, status and reason for each call.
I got the call information from grepit's comment in Simple URL GET/POST function in Python and it works great.
For appending my file, I use the following code:
#...Some working code...
dateNow = datetime.datetime.now()
string = ("\n" + dateNow.strftime("%c") + " - " + str(response.status) +
": " + response.reason + "\n")
with open('MyCallLog.txt', 'a+') as file:
file.write(string)
What I have read regarding similar issues, is that the file is not closed or flushed. However, if I try to debug using print(file.read()) outside the 'with' I get an error that is file is already closed and debugging inside the with displays nothing. I also tried it without the with and specifically stating file.close(). I have debugged the string variable using print(string) and it displays as intended.
Any suggestions?
Final notes:
I am aware that opening a file as 'a+' does open it in read and write mode. This is currently only for my debugging purposes.
When a file is opened in append mode using "a+" the cursor is positioned at the end of the file. That is why a call to .write() will append to the end of the file instead of overwriting it.
When you call file.read() in the with block, it is reading the file from the last character onwards, which is why your print output is empty.
To print the content you need to seek to the beginning of the file.
with open("myfile.txt", "a+") as file:
file.write("some_text")
file.seek(0)
print(file.read()) # "some_text"
Better yet, just open the file again for your debugging.
with open("myfile.txt", "a+") as file:
file.write("some_text")
with open("myfile.txt", "r") as file:
print(file.read())
Your code to append was actually correct. There should be a file in the CWD with all of your attempts in it.
Also, the reason that you get an error when you try to call .read() outside of the with block is because file.close() is implicitly called when the block exits.
open() returns a context manager. You can read about context managers in python here. They are very useful and great to know about. I write new context managers often at my job.
Related
I'm trying to create a webscraping script in Python where I follow a bunch of links and insert them into a .txt file. However, I want to do this only if the website already doesn't exist in the file.
I have written this code to insert the given website link into the file, so far (not working):
def writeSite(site):
file = open("websites.txt", 'a+')
# print(site)
if site in file.read():
return
file.write(site + "\n")
file.close()
Thanks in advance.
You were pretty close, but because you open the file to append to it, it starts with the file pointer at the end. You need to seek to the start to read its contents again:
def writeSite(site):
file = open("websites.txt", 'a+')
file.seek(0)
# print(site)
if site in file.read():
return
file.write(site + "\n")
file.close()
However, keep in mind that site in file.read() is very crude.
For example, imagine you already have 'http://somesite.com/page/' in the file but now you want to add 'http://somesite.com/' - the URL as a whole is not in the file, but your test will find it.
If you want to check whole lines (and be sure you deal with the file nicely), this would be better:
def writeSite(site):
site += '\n'
with open("websites.txt", 'a+') as f:
f.seek(0)
if site in f.readlines():
return
f.write(site)
It adds a newline to the name of the site to separate the URLs in the file and uses readlines to make use of that fact to check for the whole URL. Using with ensures the file always gets closed.
And since you want to read before writing anyway, you could use 'r+' as a mode, and skip the seek - but only if you can be sure the file already exists. I assume you chose 'a+' because that isn't the case.
(in case you worry that this changes the value of site - that's only true for the parameter inside the function. Whatever value you passed in outside the function will remain unaffected)
Started Python a week ago and I have some questions to ask about reading and writing to the same files. I've gone through some tutorials online but I am still confused about it. I can understand simple read and write files.
openFile = open("filepath", "r")
readFile = openFile.read()
print readFile
openFile = open("filepath", "a")
appendFile = openFile.write("\nTest 123")
openFile.close()
But, if I try the following I get a bunch of unknown text in the text file I am writing to. Can anyone explain why I am getting such errors and why I cannot use the same openFile object the way shown below.
# I get an error when I use the codes below:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
readFile = openFile.read()
print readFile
openFile.close()
I will try to clarify my problems. In the example above, openFile is the object used to open file. I have no problems if I want write to it the first time. If I want to use the same openFile to read files or append something to it. It doesn't happen or an error is given. I have to declare the same/different open file object before I can perform another read/write action to the same file.
#I have no problems if I do this:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
openFile2 = open("filepath", "r+")
readFile = openFile2.read()
print readFile
openFile.close()
I will be grateful if anyone can tell me what I did wrong here or is it just a Pythong thing. I am using Python 2.7. Thanks!
Updated Response:
This seems like a bug specific to Windows - http://bugs.python.org/issue1521491.
Quoting from the workaround explained at http://mail.python.org/pipermail/python-bugs-list/2005-August/029886.html
the effect of mixing reads with writes on a file open for update is
entirely undefined unless a file-positioning operation occurs between
them (for example, a seek()). I can't guess what
you expect to happen, but seems most likely that what you
intend could be obtained reliably by inserting
fp.seek(fp.tell())
between read() and your write().
My original response demonstrates how reading/writing on the same file opened for appending works. It is apparently not true if you are using Windows.
Original Response:
In 'r+' mode, using write method will write the string object to the file based on where the pointer is. In your case, it will append the string "Test abc" to the start of the file. See an example below:
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\n'
>>> f.write("foooooooooooooo")
>>> f.close()
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\nfoooooooooooooo'
The string "foooooooooooooo" got appended at the end of the file since the pointer was already at the end of the file.
Are you on a system that differentiates between binary and text files? You might want to use 'rb+' as a mode in that case.
Append 'b' to the mode to open the file in binary mode, on systems
that differentiate between binary and text files; on systems that
don’t have this distinction, adding the 'b' has no effect.
http://docs.python.org/2/library/functions.html#open
Every open file has an implicit pointer which indicates where data will be read and written. Normally this defaults to the start of the file, but if you use a mode of a (append) then it defaults to the end of the file. It's also worth noting that the w mode will truncate your file (i.e. delete all the contents) even if you add + to the mode.
Whenever you read or write N characters, the read/write pointer will move forward that amount within the file. I find it helps to think of this like an old cassette tape, if you remember those. So, if you executed the following code:
fd = open("testfile.txt", "w+")
fd.write("This is a test file.\n")
fd.close()
fd = open("testfile.txt", "r+")
print fd.read(4)
fd.write(" IS")
fd.close()
... It should end up printing This and then leaving the file content as This IS a test file.. This is because the initial read(4) returns the first 4 characters of the file, because the pointer is at the start of the file. It leaves the pointer at the space character just after This, so the following write(" IS") overwrites the next three characters with a space (the same as is already there) followed by IS, replacing the existing is.
You can use the seek() method of the file to jump to a specific point. After the example above, if you executed the following:
fd = open("testfile.txt", "r+")
fd.seek(10)
fd.write("TEST")
fd.close()
... Then you'll find that the file now contains This IS a TEST file..
All this applies on Unix systems, and you can test those examples to make sure. However, I've had problems mixing read() and write() on Windows systems. For example, when I execute that first example on my Windows machine then it correctly prints This, but when I check the file afterwards the write() has been completely ignored. However, the second example (using seek()) seems to work fine on Windows.
In summary, if you want to read/write from the middle of a file in Windows I'd suggest always using an explicit seek() instead of relying on the position of the read/write pointer. If you're doing only reads or only writes then it's pretty safe.
One final point - if you're specifying paths on Windows as literal strings, remember to escape your backslashes:
fd = open("C:\\Users\\johndoe\\Desktop\\testfile.txt", "r+")
Or you can use raw strings by putting an r at the start:
fd = open(r"C:\Users\johndoe\Desktop\testfile.txt", "r+")
Or the most portable option is to use os.path.join():
fd = open(os.path.join("C:\\", "Users", "johndoe", "Desktop", "testfile.txt"), "r+")
You can find more information about file IO in the official Python docs.
Reading and Writing happens where the current file pointer is and it advances with each read/write.
In your particular case, writing to the openFile, causes the file-pointer to point to the end of file. Trying to read from the end would result EOF.
You need to reset the file pointer, to point to the beginning of the file before through seek(0) before reading from it
You can read, modify and save to the same file in python but you have actually to replace the whole content in file, and to call before updating file content:
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
I needed a function to go through all subdirectories of folder and edit content of the files based on some criteria, if it helps:
new_file_content = ""
for directories, subdirectories, files in os.walk(folder_path):
for file_name in files:
file_path = os.path.join(directories, file_name)
# open file for reading and writing
with io.open(file_path, "r+", encoding="utf-8") as edit_file:
for current_line in edit_file:
if condition in current_line:
# update current line
current_line = current_line.replace('john', 'jack')
new_file_content += current_line
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
# delete actual file content
edit_file.truncate()
# rewrite updated file content
edit_file.write(new_file_content)
# empties new content in order to set for next iteration
new_file_content = ""
edit_file.close()
I am doing Learn Python the Hard Way and am on exercise 16. The study drill says to write a script using read and argv.
My code is as follows:
from sys import argv
script, file_name, pet_name = argv
print "Ah, your pet's name is %r." %pet_name
print "This will write your pet's name in a text file."
print "First, this will delete the file. "
print "Proceeding..."
writefile = open(file_name, 'w')
writefile.truncate()
writefile.write(pet_name)
writefile.close
raw_input("Now it will read. Press ENTER to continue.")
readfile = open(file_name, "r")
print readfile.read()
The code works until the end. When it says to print the file, the command line gives a blank line.
PS C:\Users\[redacted]\lpthw> python ex16study.py pet.txt jumpy
Ah, your pet's name is 'jumpy'.
This will write your pet's name in a text file.
First, this will delete the file.
Proceeding...
Now it will read. Press ENTER to continue.
PS C:\Users\[redacted]\lpthw>
I am not sure why the script is just printing a blank file.
You never called the writefile.close() method:
writefile.write(pet_name)
writefile.close
# ^^
Without closing the file, the memory buffer to help speed writing never gets flushed and the file effectively remains empty.
Either call the method:
writefile.write(pet_name)
writefile.close()
or use the file as a context manager (with the with statement) to have Python close it for you:
with open(file_name, 'w') as writefile:
writefile.write(pet_name)
Note that the writefile.truncate() call is entirely redundant. Opening a file in write mode ('w') always truncates the file already.
I know this question has been asked before quiet a lot on SO and elsewhere too. I still couldn't get it done. And im sorry if my English is bad
Removing file in linux was much more simpler. Just os.remove(my_file) did the job, But in windows it gives
os.remove(my_file)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: (file-name)
my code :
line_count = open(my_file, mode='r') #
t_lines = len(line_count.readlines()) # For total no of lines
outfile = open(dec_file, mode='w')
with open(my_file, mode='r') as contents:
p_line = 1
line_infile = contents.readline()[4:]
while line_infile:
dec_of_line = baseconvert(line_infile.rstrip(),base16,base10)
if p_line == t_lines:
dec_of_line += str(len(line_infile)).zfill(2)
outfile.write(dec_of_line + "\r\n")
else:
outfile.write(dec_of_line + "\r\n")
p_line += 1
line_infile = contents.readline()[4:]
outfile.close()
os.remove(my_file)
Here my_file is a variable that contains complete path structure of a file. Like wise dec_file also contains path, but to a new file. And the file im trying to remove is the file that's being used under read mode. Need some help please.
my try's :
Tried closing the file my_file.close(). The corresponding error i got was AttributeError: 'str' object has no attribute 'close'. I knew, when a file is in
read mode it automatically closes when it comes to the end of the
file. But still i gave it a try
Also tried by os.close(my_file) as per https://stackoverflow.com/a/1470388/3869739. i got error as TypeError: an integer is required
Or, am i getting this error just because i have opened the file
twice (for counting the line and to read the file content),..?
Pythonic way of reading from or writing to a file is by using a with context.
To read a file:
with open("/path/to/file") as f:
contents = f.read()
#Inside the block, the file is still open
# Outside `with` here, f.close() is automatically called.
To write:
with open("/path/to/file", "w") as f:
print>>f, "Goodbye world"
# Outside `with` here, f.close() is automatically called.
Now, if there's no other process reading or writing to the file, and assuming you have all the permission you should be able to close the file. There is a very good chance that there's a resource leak (file handle not being closed), because of which Windows will not allow you to delete a file. Solution is to use with.
Further, to clarify on few other points:
Its the garbage collector that causes the closure of the stream when the object is destroyed. The file is not auto-closed upon complete reading. That wouldn't make sense if the programmer wanted to rewind, would it?
os.close(..) internally calls the C-API close(..) that takes an integer file descriptor. Not string as you passed.
Started Python a week ago and I have some questions to ask about reading and writing to the same files. I've gone through some tutorials online but I am still confused about it. I can understand simple read and write files.
openFile = open("filepath", "r")
readFile = openFile.read()
print readFile
openFile = open("filepath", "a")
appendFile = openFile.write("\nTest 123")
openFile.close()
But, if I try the following I get a bunch of unknown text in the text file I am writing to. Can anyone explain why I am getting such errors and why I cannot use the same openFile object the way shown below.
# I get an error when I use the codes below:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
readFile = openFile.read()
print readFile
openFile.close()
I will try to clarify my problems. In the example above, openFile is the object used to open file. I have no problems if I want write to it the first time. If I want to use the same openFile to read files or append something to it. It doesn't happen or an error is given. I have to declare the same/different open file object before I can perform another read/write action to the same file.
#I have no problems if I do this:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
openFile2 = open("filepath", "r+")
readFile = openFile2.read()
print readFile
openFile.close()
I will be grateful if anyone can tell me what I did wrong here or is it just a Pythong thing. I am using Python 2.7. Thanks!
Updated Response:
This seems like a bug specific to Windows - http://bugs.python.org/issue1521491.
Quoting from the workaround explained at http://mail.python.org/pipermail/python-bugs-list/2005-August/029886.html
the effect of mixing reads with writes on a file open for update is
entirely undefined unless a file-positioning operation occurs between
them (for example, a seek()). I can't guess what
you expect to happen, but seems most likely that what you
intend could be obtained reliably by inserting
fp.seek(fp.tell())
between read() and your write().
My original response demonstrates how reading/writing on the same file opened for appending works. It is apparently not true if you are using Windows.
Original Response:
In 'r+' mode, using write method will write the string object to the file based on where the pointer is. In your case, it will append the string "Test abc" to the start of the file. See an example below:
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\n'
>>> f.write("foooooooooooooo")
>>> f.close()
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\nfoooooooooooooo'
The string "foooooooooooooo" got appended at the end of the file since the pointer was already at the end of the file.
Are you on a system that differentiates between binary and text files? You might want to use 'rb+' as a mode in that case.
Append 'b' to the mode to open the file in binary mode, on systems
that differentiate between binary and text files; on systems that
don’t have this distinction, adding the 'b' has no effect.
http://docs.python.org/2/library/functions.html#open
Every open file has an implicit pointer which indicates where data will be read and written. Normally this defaults to the start of the file, but if you use a mode of a (append) then it defaults to the end of the file. It's also worth noting that the w mode will truncate your file (i.e. delete all the contents) even if you add + to the mode.
Whenever you read or write N characters, the read/write pointer will move forward that amount within the file. I find it helps to think of this like an old cassette tape, if you remember those. So, if you executed the following code:
fd = open("testfile.txt", "w+")
fd.write("This is a test file.\n")
fd.close()
fd = open("testfile.txt", "r+")
print fd.read(4)
fd.write(" IS")
fd.close()
... It should end up printing This and then leaving the file content as This IS a test file.. This is because the initial read(4) returns the first 4 characters of the file, because the pointer is at the start of the file. It leaves the pointer at the space character just after This, so the following write(" IS") overwrites the next three characters with a space (the same as is already there) followed by IS, replacing the existing is.
You can use the seek() method of the file to jump to a specific point. After the example above, if you executed the following:
fd = open("testfile.txt", "r+")
fd.seek(10)
fd.write("TEST")
fd.close()
... Then you'll find that the file now contains This IS a TEST file..
All this applies on Unix systems, and you can test those examples to make sure. However, I've had problems mixing read() and write() on Windows systems. For example, when I execute that first example on my Windows machine then it correctly prints This, but when I check the file afterwards the write() has been completely ignored. However, the second example (using seek()) seems to work fine on Windows.
In summary, if you want to read/write from the middle of a file in Windows I'd suggest always using an explicit seek() instead of relying on the position of the read/write pointer. If you're doing only reads or only writes then it's pretty safe.
One final point - if you're specifying paths on Windows as literal strings, remember to escape your backslashes:
fd = open("C:\\Users\\johndoe\\Desktop\\testfile.txt", "r+")
Or you can use raw strings by putting an r at the start:
fd = open(r"C:\Users\johndoe\Desktop\testfile.txt", "r+")
Or the most portable option is to use os.path.join():
fd = open(os.path.join("C:\\", "Users", "johndoe", "Desktop", "testfile.txt"), "r+")
You can find more information about file IO in the official Python docs.
Reading and Writing happens where the current file pointer is and it advances with each read/write.
In your particular case, writing to the openFile, causes the file-pointer to point to the end of file. Trying to read from the end would result EOF.
You need to reset the file pointer, to point to the beginning of the file before through seek(0) before reading from it
You can read, modify and save to the same file in python but you have actually to replace the whole content in file, and to call before updating file content:
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
I needed a function to go through all subdirectories of folder and edit content of the files based on some criteria, if it helps:
new_file_content = ""
for directories, subdirectories, files in os.walk(folder_path):
for file_name in files:
file_path = os.path.join(directories, file_name)
# open file for reading and writing
with io.open(file_path, "r+", encoding="utf-8") as edit_file:
for current_line in edit_file:
if condition in current_line:
# update current line
current_line = current_line.replace('john', 'jack')
new_file_content += current_line
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
# delete actual file content
edit_file.truncate()
# rewrite updated file content
edit_file.write(new_file_content)
# empties new content in order to set for next iteration
new_file_content = ""
edit_file.close()