In my code, I write a file to my hard disk. After that, I need to import the generated file and then continue processing it.
for i in xrange(10):
filename=generateFile()
# takes some time, I wish to freeze the program here
# and continue once the file is ready in the system
file=importFile(filename)
processFile(file)
If I run the code snippet in one go, most likely file=importFile(filename) will complain that that file does not exist, since the generation takes some time.
I used to manually run filename=generateFile() and wait before running file=importFile(filename).
Now that I'm using a for loop, I'm searching for an automatic way.
You could use time.sleep and I would expect that if you are loading a module this way you would need to reload rather than import after the first import.
However, unless the file is very large why not just generate the string and then eval or exec it?
Note that since your file generation function is not being invoked in a thread it should be blocking and will only return when it thinks it has finished writing - possibly you can improve things by ensuring that the file writer ends with outfile.flush() then outfile.close() but on some OSs there may still be a time when the file is not actually available.
for i in xrange(10):
(filename, is_finished)=generateFile()
while is_finished:
file=importFile(filename)
processFile(file)
continue;
I think you should use a flag to test if the file is generate.
Related
So I know how to execute a Python file within another Python file, exec(open('file.py').read()) But the file that I want to loop has a while(True): loop within the file. What I want to do is loop the opening of the file containing the while true loop, while having that file run in the background.
The Opening Code:
loopCount=1
maxCount=100
while(loopcount<=maxCount):
exec(open('whileTrue.py').read())
I would think that you would do something like this, but instead of opening the file, allowing the file to run in the background, and opening the file again, etc; the code opens the file once then has the file run without opening the next file until after the file has been closed.
Any help with this is greatly appreciated.
Firstly, the exec(...) command is unsafe and really should be avoided. An alternative method would be to import your other Python file as if it were a module.
To be able to run the second file multiple times in parallel, you could use the threading module built into Python.
Say we have two files, FileA.py and FileB.py.
FileA.py:
import threading, FileB
for i in range(5):
currentThread = threading.Thread(target=FileB.Begin)
currentThread.start()
FileB.py:
import time
def Begin():
for i in range(10):
print("Currently on the {} second...".format(i))
time.sleep(1)
FileA.py will start 5 different threads, each running the Begin() function from FileB.py. You may replace the contents of the Begin() function, rename it or do whatever you like, but be sure to update the threads target in FileA.py.
It should be noted that opening too many threads is a bad idea as you may exceed the amount of memory your computer has. As well as that, you should ensure that your threads eventually end and are not a true 'infinite loop', as you may find you have to force your version of FileA.py to close in order to end the threads.
For more detail on the threading module you can see the python documentation: https://docs.python.org/3/library/threading.html
Or you could fine many guides related to the topic by searching for Python threading on Google.
I hope this helps.
I am using seek function to extract new lines in an updated file. My code looks like this:
read_data=open('path-to-myfile','r')
read_data.seek(0,2)
while True:
time.sleep(sometime)
new_data=read_data.readlines()
do something with new_data
myfile is a csv file that will be constantly updated
The problem is that usually after several loops inside the while, new_data return nothing. It can be different loop numbers. While I checked myfile, it is still updating..... So any problem I have on my code ? Or is there any other way to do this ?
Any help appreciated !!
You have two programs accessing the same file on disk? If that is the case, then the resource may be locking. I set up an example script that writes to a file, and another file that reads for changes based on the code you provided.
So in one instance of python:
import time
while True:
time.sleep(2)
with open('test.txt','a') as read_data:
read_data.seek(0,2)
read_data.write("bibbity boopity\n")
And in another instance of python
import time
read_data=open('test.txt','r')
read_data.seek(0,2)
while True:
time.sleep(1)
new_data=read_data.readlines()
print(new_data)
In this case, the resource is updating slower than its being read, so changes printed by the bottom prog will be blank. But if I speed up the changes per second, well I still see them. But there are some instances where not all the updates are seen.
You may want to use asynchronous file reading to catch all the changes. Python 3 asyncio library doesn't support async file read/write, but curio does.
See also this question
Is it possible to create temporary files in python that will be deleted after some time? I checked tempfile library which generates temporary files and directories.
tempfile.TemporaryFile : This function creates tempfile which will be destroyed as soon as closed.
tempfile.NamedTemporaryFile: This function accepts a parameter delete. If delete is given the value false while calling the function the file will not be deleted on close.
What I need is a temp file which I should be able to read with its name for some time even after I close it. However, It should be deleted automatically after some time.
What is the easiest way to do this in Python 2.7?
If you only need to run the file at intervals within your Python program, you can use it as a context manager:
def main(file_object):
# Can read and write to file_object at any time within this function
if __name__ == '__main__':
with tempfile.NamedTemporaryFile() as file_object:
main(file_object)
# Can still use file_object
# file_object is referencing a closed and deleted file here
If you need to be able to open it outside of the Python program, say after main is over, like in a process that runs asyncronously, you should probably look for a more permenant solution (e.g. allowing the user to specify the file name.)
If you really needed to have a temporary auto-deleting file, you can start a threading.Timer to delete it after some time, but make sure that your program deletes it even if it is stopped before the timer ends.
If I wanted to close a file in a program with an infinite loop, I could do the following:
file = open('abc', 'w')
while True:
try:
file.write('a\n')
except KeyboardInterrupt:
break
file.close()
But if I just left it like this:
file = open('abc', 'w')
while True:
file.write('a\n')
Would the file not get closed on exit?
Python has automatic garbage collection. Garbage collection is the automatic clearing away of unused file handles or data in memory. As a python program halts, if it has halted cleanly, it will free up the memory and file handles it has used as it closes. It's prudent to have a familiarity with how garbage collection works when writing programs that run for extended periods of time, or that access a lot of information or files.
As mentioned above by #Mitch Jackson, if a python program halts cleanly, it will free up the memory. In your case, since you are just opening a file in a while loop. The program won't be able to close already opened file when it halts unless you explicitly include exception handling like a try-catch block or a with statement which wraps it up.
Here are the documentation on with statement: docs
This code works fine for me. Appends data at the end.
def writeFile(dataFile, nameFile):
fob = open(nameFile,'a+')
fob.write("%s\n"%dataFile)
fob.close()
But the problem is when I close the program and later run again I found that all the previous data were lost. Process is started to write from the start and there is no data in the file.
But during the run it perfectly add a line at the end of file.
I can't understand the problem. Please some one help.
NB: I am using Ubuntu-10.04 with python 2.6
There is nothing wrong with the code you posted here... I tend to agree with other the comments that this file is probably being overwritten elsewhere in your code.
The only suggestion I can think of to test this explicitly (if your use case can tolerate it) is to throw in an exit() statement at the end of the function and then open the file externally (aka in gedit) and see if the last change took.
Alternatively to the exit, you could run the program in the terminal and include a call to pdb at the end of this function which would interrupt the program without killing it:
import pdb; pdb.set_trace()
You will then have to hit c to continue the program each time this runs.
If that checks out, do a search for other places this file might be getting opened.