I'm trying to pull some data from Facebook pages for a product and dump it all into a text file, but I find that the file keeps overwriting itself with the data. I'm not sure if it's a pagination issue or if I have to make several files.
Here's my code:
#Modules
import requests
import facebook
import json
def some_action(post):
print posts['data']
print post['created_time']
#Token
access_token = 'INSERT ACCESS TOKEN'
user = 'walkers'
#Posts
graph = facebook.GraphAPI(access_token)
profile = graph.get_object(user)
posts = graph.get_connections(profile['id'], 'posts')
#Write
while True:
posts = requests.get(posts['paging']['next']).json()
#print posts
with open('test121.txt', 'w') as outfile:
json.dump(posts, outfile)
Any idea as to why this is happening?
w overwrites, open with a to append or open the file once outside the loop:
append:
while True:
posts = requests.get(posts['paging']['next']).json()
#print posts
with open('test121.txt', 'a') as outfile:
json.dump(posts, outfile)
Open once outside the loop:
with open('test121.txt', 'w') as outfile:
while True:
posts = requests.get(posts['paging']['next']).json()
#print posts
json.dump(posts, outfile)
It makes more sense to use the second option, if you are going to be running the code multiple times then you can open with a outside the loop also, if the file does not exist it will be created, if it does data will be appended
This is because you are using the file operator with w mode, you are overwriting the content. You can use the a append mode:
It can be done like this
Modification:
with open('test121.txt', 'w') as outfile:
while True:
posts = requests.get(posts['paging']['next']).json()
json.dump(posts, outfile)
w overwrites on the existing file
i.e)
File1.txt:
123
code:
with open("File1.txt","w") as oup1:
oup1.write("2")
File1.txt after python run:
2
Its value is overwritten
a appends to the existing file
i.e)
File1.txt:
123
code:
with open("File1.txt","a") as oup1:
oup1.write("2")
File1.txt after python run:
1232
The written content is appended to the end.
Opening and Closing Files
to use actual data files reading and writing to the standard input and output.
Python provides basic functions and methods necessary to manipulate files by default. You can do your most of the file manipulation using a file object.
The open Function
Before you can read or write a file, you have to open it using Python's built-in open() function. This function creates a file object, which would be utilized to call other support methods associated with it.
Syntax
file object = open(file_name [, access_mode][, buffering])
Here are parameter details:
file_name: The file_name argument is a string value that contains the name of the file that you want to access.
access_mode: The access_mode determines the mode in which the file has to be opened, i.e., read, write, append, etc. A complete list of possible values is given below in the table. This is optional parameter and the default file access mode is read (r).
buffering: If the buffering value is set to 0, no buffering takes place. If the buffering value is 1, line buffering is performed while accessing a file. If you specify the buffering value as an integer greater than 1, then buffering action is performed with the indicated buffer size. If negative, the buffer size is the system default(default behavior).
Here is a list of the different modes of opening a file −
Modes and Description
r= Opens a file for reading only. The file pointer is placed at the beginning of the file. This is the default mode.
rb= Opens a file for reading only in binary format. The file pointer is placed at the beginning of the file. This is the default mode.
r+= Opens a file for both reading and writing. The file pointer placed at the beginning of the file.
rb+= Opens a file for both reading and writing in binary format. The file pointer placed at the beginning of the file.
w= Opens a file for writing only. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.
wb= Opens a file for writing only in binary format. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.
w+= Opens a file for both writing and reading. Overwrites the existing file if the file exists. If the file does not exist, creates a new file for reading and writing.
wb+= Opens a file for both writing and reading in binary format. Overwrites the existing file if the file exists. If the file does not exist, creates a new file for reading and writing.
a= Opens a file for appending. The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing.
ab= Opens a file for appending in binary format. The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing.
a+= Opens a file for both appending and reading. The file pointer is at the end of the file if the file exists. The file opens in the append mode. If the file does not exist, it creates a new file for reading and writing.
ab+= Opens a file for both appending and reading in binary format. The file pointer is at the end of the file if the file exists. The file opens in the append mode. If the file does not exist, it creates a new file for reading and writing.
Reading and Writing Files
The file object provides a set of access methods to make our lives easie with the use of read() and write() methods to read and write files.
The write() Method
The write() method writes any string to an open file. It is important to note that Python strings can have binary data and not just text.
The write() method does not add a newline character ('\n') to the end of the string −
Syntax
fileObject.write(string);
Here, passed parameter is the content to be written into the opened file.
Example
# Open a file
fo = open("file.txt", "wb")
fo.write( "Python is a great language");
# Closeopend file
fo.close()
The above method would create foo.txt file and would write given content in that file and finally it would close that file. If you would open this file, it would have following content.
Python is a great language.
The read() Method
The read() method reads a string from an open file. It is important to note that Python strings can have binary data. apart from text data.
Syntax
fileObject.read([count]);
Here, passed parameter is the number of bytes to be read from the opened file. This method starts reading from the beginning of the file and if count is missing, then it tries to read as much as possible, maybe until the end of file.
Example
Let's take a file foo.txt, which we created above.
# Open a file
fo = open("foo.txt", "r+")
str = fo.read(10);
print "Read String is : ", str
# Close opend file
fo.close()
This produces the following result −
Read String is : Python is
Related
I have this code:
line1 = []
line1.append("xyz ")
line1.append("abc")
line1.append("mno")
file = open("File.txt","w")
for i in range(3):
file.write(line1[i])
file.write("\n")
for line in file:
print(line)
file.close()
But when I try it, I get an error message like:
File "...", line 18, in <module>
for line in file:
UnsupportedOperation: not readable
Why? How do I fix it?
You are opening the file as "w", which stands for writable.
Using "w" you won't be able to read the file. Use the following instead:
file = open("File.txt", "r")
Additionally, here are the other options:
"r" Opens a file for reading only.
"r+" Opens a file for both reading and writing.
"rb" Opens a file for reading only in binary format.
"rb+" Opens a file for both reading and writing in binary format.
"w" Opens a file for writing only.
"a" Open for writing. The file is created if it does not exist.
"a+" Open for reading and writing. The file is created if it does not exist.
Use a+ to open a file for reading, writing and create it if it doesn't exist.
a+ Opens a file for both appending and reading. The file pointer is at
the end of the file if the file exists. The file opens in the append
mode. If the file does not exist, it creates a new file for reading
and writing. -Python file modes
with open('"File.txt', 'a+') as file:
print(file.readlines())
file.write("test")
Note: opening file in a with block makes sure that the file is properly closed at the block's end, even if an exception is raised on the way. It's equivalent to try-finally, but much shorter.
There are few modes to open file (read, write etc..)
If you want to read from file you should type file = open("File.txt","r"), if write than file = open("File.txt","w"). You need to give the right permission regarding your usage.
more modes:
r. Opens a file for reading only.
rb. Opens a file for reading only in binary format.
r+ Opens a file for both reading and writing.
rb+ Opens a file for both reading and writing in binary format.
w. Opens a file for writing only.
you can find more modes in here
This will let you read, write and create the file if it don't exist:
f = open('filename.txt','a+')
f = open('filename.txt','r+')
Often used commands:
f.readline() #Read next line
f.seek(0) #Jump to beginning
f.read(0) #Read all file
f.write('test text') #Write 'test text' to file
f.close() #Close file
Sreetam Das's table is nice but needs a bit of an update according to w3schools and my own testing. Unsure if this due to the move to python 3.
"a" - Append - will append to the end of the file and will create a file if the specified file does not exist.
"w" - Write - will overwrite any existing content and will create a file if the specified file does not exist.
"x" - Create - will create a file, returns an error if the file exist.
I would have replied directly but I do not have the rep.
https://www.w3schools.com/python/python_file_write.asp
I have this code:
line1 = []
line1.append("xyz ")
line1.append("abc")
line1.append("mno")
file = open("File.txt","w")
for i in range(3):
file.write(line1[i])
file.write("\n")
for line in file:
print(line)
file.close()
But when I try it, I get an error message like:
File "...", line 18, in <module>
for line in file:
UnsupportedOperation: not readable
Why? How do I fix it?
You are opening the file as "w", which stands for writable.
Using "w" you won't be able to read the file. Use the following instead:
file = open("File.txt", "r")
Additionally, here are the other options:
"r" Opens a file for reading only.
"r+" Opens a file for both reading and writing.
"rb" Opens a file for reading only in binary format.
"rb+" Opens a file for both reading and writing in binary format.
"w" Opens a file for writing only.
"a" Open for writing. The file is created if it does not exist.
"a+" Open for reading and writing. The file is created if it does not exist.
Use a+ to open a file for reading, writing and create it if it doesn't exist.
a+ Opens a file for both appending and reading. The file pointer is at
the end of the file if the file exists. The file opens in the append
mode. If the file does not exist, it creates a new file for reading
and writing. -Python file modes
with open('"File.txt', 'a+') as file:
print(file.readlines())
file.write("test")
Note: opening file in a with block makes sure that the file is properly closed at the block's end, even if an exception is raised on the way. It's equivalent to try-finally, but much shorter.
There are few modes to open file (read, write etc..)
If you want to read from file you should type file = open("File.txt","r"), if write than file = open("File.txt","w"). You need to give the right permission regarding your usage.
more modes:
r. Opens a file for reading only.
rb. Opens a file for reading only in binary format.
r+ Opens a file for both reading and writing.
rb+ Opens a file for both reading and writing in binary format.
w. Opens a file for writing only.
you can find more modes in here
This will let you read, write and create the file if it don't exist:
f = open('filename.txt','a+')
f = open('filename.txt','r+')
Often used commands:
f.readline() #Read next line
f.seek(0) #Jump to beginning
f.read(0) #Read all file
f.write('test text') #Write 'test text' to file
f.close() #Close file
Sreetam Das's table is nice but needs a bit of an update according to w3schools and my own testing. Unsure if this due to the move to python 3.
"a" - Append - will append to the end of the file and will create a file if the specified file does not exist.
"w" - Write - will overwrite any existing content and will create a file if the specified file does not exist.
"x" - Create - will create a file, returns an error if the file exist.
I would have replied directly but I do not have the rep.
https://www.w3schools.com/python/python_file_write.asp
If I want to open a file, unpickle an object inside it, then overwrite it later, is it okay to just use
data = {} #Its a dictionary in my code
file = open("filename","wb")
data = pickle.load(file)
data["foo"] = "bar"
pickle.dump(data,file)
file.close()
Or would I have to use "rb" first and then use "wb" later (using with statements for each) which is what I am doing now. Note that in my program, there is a hashing algorithm in between opening the file and closing it, which is where the dictionary data comes from, and I basically want to be able to only open the file once without having to do two with statements
If you want to read, then write the file, do not use modes involving w at all; all of them truncate the file on opening it.
If the file is known to exist, use mode "rb+", which opens an existing file for both read and write.
Your code only needs to change a tiny bit:
# Open using with statement to ensure prompt/proper closing
with open("filename","rb+") as file:
data = pickle.load(file) # Load from file (moves file pointer to end of file)
data["foo"] = "bar"
file.seek(0) # Move file pointer back to beginning of file
pickle.dump(data, file) # Write new data over beginning of file
file.truncate() # If new dump is smaller, make sure to chop off excess data
You can use wb+ which opens the file for both reading and writing
This question is helpful for understanding the differences between each of pythons read and write conditions, but adding + at the end usually always opens the file for both read and write
Confused by python file mode "w+"
In python's OS module there is a method to open a file and a method to read a file.
The docs for the open method say:
Open the file file and set various flags according to flags and
possibly its mode according to mode. The default mode is 0777 (octal),
and the current umask value is first masked out. Return the file
descriptor for the newly opened file.
The docs for the read method say;
Read at most n bytes from file descriptor fd. Return a string
containing the bytes read. If the end of the file referred to by fd
has been reached, an empty string is returned.
I understand what it means to read n bytes from a file. But how does this differ from open?
"Opening" a file doesn't actually bring any of the data from the file into your program. It just prepares the file for reading (or writing), so when your program is ready to read the contents of the file it can do so right away.
Opening a file allows you to read or write to it (depending on the flag you pass as the second argument), whereas reading it actually pulls the data from a file that is typcially saved into a variable for processing or printed as output.
You do not always read from a file once it is opened. Opening also allows you to write to a file, either by overwriting all the contents or appending to the contents.
To read from a file:
>>> myfile = open('foo.txt', 'r')
>>> myfile.read()
First you open the file with read permission (r)
Then you read() from the file
To write to a file:
>>> myfile = open('foo.txt', 'r')
>>> myfile.write('I am writing to foo.txt')
The only thing that is being done in line 1 of each of these examples is opening the file. It is not until we actually read() from the file that anything is changed
open gets you a fd (file descriptor), you can read from that fd later.
One may also open a file for other purpose, say write to a file.
It seems to me you can read lines from the file handle without invoking the read method but I guess read() truly puts the data in the variable location. In my course we seem to be printing lines, counting lines, and adding numbers from lines without using read().
The rstrip() method needs to be used, however, because printing the line from the file handle using a for in statement also prints the invisible line break symbol at the end of the line, as does the print statement.
From Python for Everybody by Charles Severance, this is the starter code.
"""
7.2
Write a program that prompts for a file name,
then opens that file and reads through the file,
looking for lines of the form:
X-DSPAM-Confidence: 0.8475
Count these lines and extract the floating point
values from each of the lines and compute the
average of those values and produce an output as
shown below. Do not use the sum() function or a
variable named sum in your solution.
You can download the sample data at
http://www.py4e.com/code3/mbox-short.txt when you
are testing below enter mbox-short.txt as the file name.
"""
# Use the file name mbox-short.txt as the file name
fname = input("Enter file name: ")
fh = open(fname)
for line in fh:
if not line.startswith("X-DSPAM-Confidence:") :
continue
print(line)
print("Done")
When using this code in python:
f = open('ping.log', 'r+')
f.write("["+time.ctime()+"]"+"Status")
f.close()
My file always gets overwritten. And only has one line in it, like this:
[Fri Sep 02 16:30:56 2011]Status
Why is it getting overwritten?
It's failing because you are effectively recreating the file each time as you are overwriting the first N bytes every time. If you wrote less bytes you'd see the "old" information still there.
You need to open the file for "append"
'a' opens the file for appending
Source
r+ sets the initial file pointer to the beginning. Either seek to the end or use a mode.
Check this question. Open the file with the "a" mode:
f = open("ping.log","a")
...
http://docs.python.org/tutorial/inputoutput.html#reading-and-writing-files
The first argument is a string containing the filename. The second
argument is another string containing a few characters describing the
way in which the file will be used. mode can be 'r' when the file will
only be read, 'w' for only writing (an existing file with the same
name will be erased), and 'a' opens the file for appending; any data
written to the file is automatically added to the end. 'r+' opens the
file for both reading and writing. The mode argument is optional; 'r'
will be assumed if it’s omitted.
so use
f = open('ping.log', 'a')
f.write("["+time.ctime()+"]"+"Status")
f.close()