Develop Reference

How to include another python file from the same folder

I was wondering if it's possible for me to include another python file in the actual one?
I'm running an app.py from the terminal with Flask and I want, when I click on submit that python run 5 lines of code, like nothing. But when I do the variable from the first python file can't be read on the second and even if I put the same variable into the second python file then it still doesn't work.
I want to run a "for", this is the code
for line in fin:
line = re.sub('APP:NAME', name, line)
line = re.sub('APP:USERNAME', username, line)
line = re.sub('APP:TEXT', tweet, line)
fout.write(line)
I checked all the forums and I didn't find the solution.
Thank you

Method 1
I think your need is to include a python file to another python file in a same directory.
You can import a python file as shown below
import filename
here, the filename is the filename of the file in the same location without file extension.
and you can call functions inside that file by calling like this.
filename.function()
Method 2
if you which to rename that filename and use it as something else:
import filename as fn
here, the filename is the filename of the file in the same location without file extension.
and you can call functions inside that file by calling like this.
fn.functionName()
Method 3
or you can simply use
from filename import *
and call the functions in that file as normal functions in the current python file like
functionname()
but Better method is method 1.

Specifying file name with mkstemp, file not found

I need to be able to create a temporary file with a specified file name and write data to it, then zip said file with filename up along with other files:
fd, path = tempfile.mkstemp(".bin", "filename", "~/path/to/working/directory/")
try:
with os.fdopen(fd, "wb") as tmp:
tmp.write(data)
with ZipFile("zip.zip", "w") as zip:
zip.write("filename")
zip.writestr("file2", file2_str)
zip.writestr("file3", file3_str)
# ...
finally:
os.remove(path)
I think I must be misunderstanding how mkstemp works, I get the error at the first line of code here:
FileNotFoundError: [Errno 2] No such file or directory: '~/path/to/working/directory/filenameq5st7dey.bin'
It looks like a bunch of garbage gets added to the file name before the suffix is put on the file. I've tried this without a suffix and I still get garbage at the end of the file name.
Aside from the garbage in the file name, why do I get a file not found error instead of having a temporary file created in my directory with that name (plus garbage)?

You supplied this argument:
"~/path/to/working/directory/"
Perfectly natural, it makes sense why you would supply it. But it is wrong. If you ls . you likely will not find a ~ directory.
What you were hoping for was expansion to ${HOME}, as the Bourne shell does. In python we must call this function:
os.path.expanduser("~/path/to/working/directory/")
Print the result it returns and you'll see why it's essential.
Some folks prefer to have pathlib do the work for them:
from pathlib import Path
Path("~/path/to/working/directory/").expanduser()

shutil.move deletes the file on windows when new name contains a colon (and an extension)

Try:
import os, shutil
wd = os.path.abspath(os.path.curdir)
newfile = os.path.join(wd, 'testfile')
print str(newfile)
with open(newfile, 'w') as f: f.write('Hello bugs')
shutil.move(newfile, os.path.join(wd, 'testfile:.txt')) # note the :
Now check the directory - newfile is deleted and no other file is created - Process finished with exit code 0.
If however you issue:
shutil.move(newfile, os.path.join(wd, 'testfile:')) # note no extension
it blows with:
Traceback (most recent call last):
File "C:/Users/MrD/.PyCharm40/config/scratches/scratch_3", line 9, in <module>
shutil.move(newfile, os.path.join(wd, 'testfile:'))
File "C:\_\Python27\lib\shutil.py", line 302, in move
copy2(src, real_dst)
File "C:\_\Python27\lib\shutil.py", line 130, in copy2
copyfile(src, dst)
File "C:\_\Python27\lib\shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 22] invalid mode ('wb') or filename: 'C:\\Users\\MrD\\.PyCharm40\\config\\scratches\\testfile:'
as it should.
Is it a bug ?
Context: I was testing the behavior of my code when illegal filenames were given (: is illegal in windows filenames) when to my amazement my program deleted the original file (bad!) and created a zero size file with the attributes of the original (yes in my case the file was created, just empty) and filename the filename given up to the : - soo a filename like textfile:.jpg gave me a zero byte textfile. It took a lot of debugging - here is the little critter inside the Python27\lib\shutil.py copyfile() (the line that blows above and did not blow):
I don't know why in my case the file was created though while when running the script no.

This isn't a bug in Python's shutil or os modules, it's just a weirdness in Windows. Peter Wood's link in the comments discusses "Advanced Data Streams" -- a Windows filesystem mechanism that attaches a hidden file containing metadata to a regular, visible file. A key word there is attached; The hidden file is deleted if the file it is attached to is deleted.
It appears that a colon is used to separate the path of the regular file from the hidden file. For example, if in the command line you write:
> notepad foo
Then close notepad, and write
> notepad foo.txt:bar
Notepad will open the hidden file. Go ahead and write something in it, save, and close. Typing > dir and the command line will only show foo.txt, not foo.txt:bar.txt. But sure enough, if you write
> notepad foo.txt:bar.txt
the file you just edited will appear, and your changes will be intact.
So what is happening with your Python code? The documentation for shutil.move says:
src is copied (using shutil.copy2()) to dst and then removed.
So when you move testfile to testfile:.txt, Python first copies testfile to the hidden testfile:.txt. But then it removes testfile, and by doing so removes the hidden testfile:.txt. Therefore it appears to you that the original file has been deleted, and no new file has been created.
The following snippet of code might make this clearer (I've saved it as demo.py, and I'm running it in the same, other-wise empty directory):
import os, shutil
with open('test', 'w') as f:
f.write('Hello bugs')
shutil.copy2('test', 'test:foo.txt')
with open('test:foo.txt') as f:
print(f.read())
print 'test: exists? ', os.path.exists('test')
print 'test:foo.txt exists? ', os.path.exists('test:foo.txt')
print os.listdir('.')
print('removing...')
os.remove('test')
print 'test: exists? ', os.path.exists('test')
print 'test:foo.txt exists? ', os.path.exists('test:foo.txt')
print os.listdir('.')
This prints:
Hello bugs
test exists? True
test:foo.txt exists? True
['demo.py', 'test']
removing...
test: exists? False
test:foo.txt exists? False
['demo.py']
This shows that we can create a normal file, write to it, and copy that normal file to its hidden stream, open, and read it just fine, and the result is as expected. Then we see that os.path.exists shows that both test and it's hidden attachment test:foo.txt exist, even though os.listdir only shows test. Then we delete test and we see that test:foo.txt no longer exists as well.
Lastly, you can't create a hidden data stream without a name, therefore test: is an invalid path. Python correctly throws an exception in this case.
So the Python code is actually functioning as it should under Windows -- "Alternate Data Streams" are just such a little-known "feature" that this behavior is surprising.

Copy a file line by line in python

I am writing a python program to copy a file line by line into a new file. The code I have is below in which I am using a loop to copy the file line by line.
However since the number of lines in the file may change, is there a way to copy a file line by line in python without using a loop which relies on numbers, and instead relies on the something like the EOF character to terminate the loop?
import os
import sys
i = 0
f = open("C:\\Users\\jgr208\\Desktop\\research_12\\sap\\beam_springs.$2k","r")
copy = open("C:\\Users\\jgr208\\Desktop\\research_12\\sap\\copy.$2k","wt")
#loop that copies file line by line and terminates loop when i reaches 10
while i < 10:
line = f.readline()
copy.write(str(line))
i = i +1
f.close()
copy.close()

You can iterate over lines in a file object in Python by iterating over the file object itself:
for line in f:
copy.write(line)
From the docs on file objects:
An alternative approach to reading lines is to loop over the file object. This is memory efficient, fast, and leads to simpler code:
>>> for line in f:
print line,

Files can be iterated directly, without the need for an explicit call to readline:
f = open("...", "r")
copy = open("...", "w")
for line in f:
copy.write(line)
f.close()
copy.close()

See shutil module for better ways of doing this than copying line-by-line:
shutil.copyfile(src, dst)
Copy the contents (no metadata) of the
file named src to a file named dst. dst must be the complete target
file name; look at shutil.copy() for a copy that accepts a target
directory path. If src and dst are the same files, Error is raised.
The destination location must be writable; otherwise, an IOError
exception will be raised. If dst already exists, it will be replaced.
Special files such as character or block devices and pipes cannot be
copied with this function. src and dst are path names given as
strings.
Edit: Your question says you are copying line-by-line because the source file is volatile. Something smells wrong about your design. Could you share more details regarding the problem you are solving?

Writing line by line can be slow when working with large data. You can accelerate the read/write operations by reading/writing a bunch of lines all at once. Please refer to my answer to a similar question here

Using with statements:
with open("input.txt", "r", encoding="utf-8") as input_file:
with open("output.txt", "w", encoding="utf-8") as output_file:
for input_line in input_file:
output_line = f(input_line) # You can change the line here
output_file.write(output_line)
Note that input_line contains the end-of-line character(s) (\n or \r\n), if there are any.

Validating a zip file coming from stdin

After some frustration with unzip(1L), I've been trying to create a script that will unzip and print out raw data from all of the files inside a zip archive that is coming from stdin. I currently have the following, which works:
import sys, zipfile, StringIO
stdin = StringIO.StringIO(sys.stdin.read())
zipselect = zipfile.ZipFile(stdin)
filelist = zipselect.namelist()
for filename in filelist:
print filename, ':'
print zipselect.read(filename)
When I try to add validation to check if it truly is a zip file, however, it doesn't like it.
...
zipcheck = zipfile.is_zipfile(zipselect)
if zipcheck is not None:
print 'Input is not a zip file.'
sys.exit(1)
...
results in
File "/home/chris/simple/zipcat/zipcat.py", line 13, in <module>
zipcheck = zipfile.is_zipfile(zipselect)
File "/usr/lib/python2.7/zipfile.py", line 149, in is_zipfile
result = _check_zipfile(fp=filename)
File "/usr/lib/python2.7/zipfile.py", line 135, in _check_zipfile
if _EndRecData(fp):
File "/usr/lib/python2.7/zipfile.py", line 203, in _EndRecData
fpin.seek(0, 2)
AttributeError: ZipFile instance has no attribute 'seek'
I assume it can't seek because it is not a file, as such?
Sorry if this is obvious, this is my first 'go' with Python.

You should pass stdin to is_zipfile, not zipselect. is_zipfile takes a path to a file or a file object, not a ZipFile.
See the zipfile.is_zipfile documentation
You are correct that a ZipFile can't seek because it isn't a file. It's an archive, so it can contain many files.

To do this entirely in memory will take some work. The AttributeError message means that the is_zipfile method is trying to use the seek method of the file handle you provide. But standard input is not seekable, and therefore your file object for it has no seek method.
If you really, really can't store the file on disk temporarily, then you could buffer the entire file in memory (you would need to enforce a size limit for security), and then implement some "duck" code that looks and acts like a seekable file object but really just uses the byte-string in memory.
It is possible that you could cheat and buffer only enough of the data for is_zipfile to do its work, but I seem to recall that the table-of-contents for ZIP is at the end of the file. I could be wrong about that though.

Your 2011 python2 fragment was: StringIO.StringIO(sys.stdin.read())
In 2018 a python3 programmer might phrase that as: io.StringIO(...).
What you wanted was the following python3 fragment: io.BytesIO(...).
Certainly that works well for me when using the requests module to download binary ZIP files from webservers:
zf = zipfile.ZipFile(io.BytesIO(req.content))