The documentation reads:
ast.parse(source, filename='<unknown>', mode='exec')
Equivalent to compile(source, filename, mode, ast.PyCF_ONLY_AST).
compile(source, filename, mode[, flags[, dont_inherit]])
The filename argument should give the file from which the code was read;
pass some recognizable value if it wasn’t read from a file
('<string>' is commonly used).
But it doesn't tell me much how to get this filename back from the AST node. Or how is this filename parameter is used. Is it just a stub?
It sets the co_filename attribute on the code object, which is used to display the filename in tracebacks. Besides that, it's not really significant what value you pass.
>>> c = compile('raise Exception("spam")', 'eggs', 'exec')
>>> eval(c)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "eggs", line 1, in <module>
Exception: spam
Related
I often use print() instead of file.write() when writing to files in Python. Recently, I had to write out some binary data, and tried a similar pattern:
f = open('test.txt', 'w')
f.write('abcd')
print('efgh', file=f)
f.close()
f = open('test.bin', 'wb')
f.write(b'abcd') # works as expected
print(b'efgh', file=f)
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# TypeError: a bytes-like object is required, not 'str'
f.close()
My first thought was the default newline-appending behaivor of print() might be at fault:
print(b'efgh', file=f, end='')
# same error
print(b'efgh', file=f, end=None)
# same error
print(b'efgh', file=f, end=b'')
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# TypeError: end must be None or a string, not bytes
print(b'efgh', file=f, end=b'\n')
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# TypeError: end must be None or a string, not bytes
Unfortunately, none of the permutations of changing the end worked.
Why is this happening? I suspect that this use of print() might not be supported, but in that case the error message is quite confusing.
From python3 documentation: https://docs.python.org/3/library/functions.html#print
All non-keyword arguments are converted to strings like str() does and written to the stream, separated by sep and followed by end. Both sep and end must be strings; they can also be None, which means to use the default values. If no objects are given, print() will just write end.
As I understand, you can't write bytes with print() because the object you want to write is passed only as positional argument and positional arguments are converted to strings(hence the error).
I am creating a program that loads and runs python scripts from a compressed file. Along with those python scripts, I have a config file that I previously used configparser to load info from in an uncompressed version of the program.
Is it possible to directly read config files in zip files directly with configparser? or do I have to unzip it into a temp folder and load it from there?
I have tried directly giving the path:
>>> sysconf = configparser.ConfigParser()
>>> sysconf.read_file("compressed.zip/config_data.conf")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
self._read(f, source)
File "/usr/local/lib/python3.4/configparser.py", line 1058, in _read
raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: '<???>', line: 1
Didn't work. no surprises there.
Then I tried using zipfile
>>> zf = zipfile.ZipFile("compressed.zip")
>>> data = zf.read("config_data.conf")
>>> sysconf = configparser.ConfigParser()
>>> sysconf.read_file(data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
self._read(f, source)
File "/usr/local/lib/python3.4/configparser.py", line 1009, in _read
if line.strip().startswith(prefix):
AttributeError: 'int' object has no attribute 'strip'
and found that it didn't work either.
so I've resorted to creating a temp folder, uncompressing to it, and reading the conf file there. I would really like to avoid this if possible as the conf files are the only limiting factor. I can (and am) loading the python modules from the zip file just fine at this point.
I can get the raw text of the file if there's a way to pass that directly to configparser, but searching the docs I came up empty handed.
Update:
I tried using stringIO as a file object, and it seems to work somewhat.
configparser doesn't reject it, but it doesn't like it either.
>>> zf = zipfile.ZipFile("compressed.zip")
>>> data = zf.read(config_data.conf)
>>> confdata = io.StringIO(str(data))
>>> sysconf = configparser.ConfigParser()
>>> sysconf.readfp(confdata)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/configparser.py", line 736, in readfp
self.read_file(fp, source=filename)
File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
self._read(f, source)
File "/usr/local/lib/python3.4/configparser.py", line 1058, in _read
raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: '<???>', line: 1
(continues to spit out the entire contents of the file)
If I use read_file instead, it doesn't error out, but doesn't load anything either.
>>> zf = zipfile.ZipFile("compressed.zip")
>>> data = zf.read(config_data.conf)
>>> confdata = io.StringIO(str(data))
>>> sysconf = configparser.ConfigParser()
>>> sysconf.read_file(confdata)
>>> sysconf.items("General") #(this is the main section in the file)
Traceback (most recent call last):
File "/usr/local/lib/python3.4/configparser.py", line 824, in items
d.update(self._sections[section])
KeyError: 'General'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/configparser.py", line 827, in items
raise NoSectionError(section)
configparser.NoSectionError: No section: 'General'
can get the raw text of the file if there's a way to pass that directly to configparser
Try configparser.ConfigParser.read_string
When coupled with an appropriate ZIP file, this code works for me:
import zipfile
import configparser
zf = zipfile.ZipFile("compressed.zip")
zf_config = zf.open("config_data.conf", "rU")
zf_config_data = zf_config.read().decode('ascii')
config = configparser.ConfigParser()
config.read_string(zf_config_data)
assert config['today']['lunch']=='cheeseburger'
Upon reflection, the following might be more appropriate:
import zipfile
import configparser
import io
zf = zipfile.ZipFile("compressed.zip")
zf_config = zf.open("config_data.conf", "rU")
zf_config = io.TextIOWrapper(zf_config)
config = configparser.ConfigParser()
config.read_file(zf_config)
assert config['today']['lunch']=='cheeseburger'
As written in comments, #matthewatabet answer won't work with Python 3.4 (and newer vesions). It's because ZipFile.open now returns a "bytes-like" object and not a "file-like" object anymore. You can use:
codecs.getreader("utf-8")(config_file)
To convert the config_file bytes-like object into a file-like object using the UTF-8 encoding. The code is now:
import zipfile, configparser, codecs
# Python >= 3.4
with zipfile.ZipFile("compressed.zip") as zf:
config_file = zf.open("config_data.conf") # binary mode
sysconfig = configparser.ConfigParser()
sysconfig.read_file(codecs.getreader("utf-8")(config_file))
That seems more satisfactory than creating a string, but I don't know if it's more efficient...
EDIT Since Python 3.9, the zipfile module provides a zipfile.Path.open method that can handle text and binary modes. Default is text mode. The following code works fine:
# Python >= 3.9
with zipfile.ZipFile("compressed.zip") as zf:
zip_path = zipfile.Path(zf)
config_path = zip_path / "config_data.conf"
config_file = config_path.open() # text mode
sysconfig = configparser.ConfigParser()
sysconfig.read_file(config_file)
ZipFile not only supports read but also open, which returns a file-like object. So, you could do something like this:
zf = zipfile.ZipFile("compressed.zip")
config_file = zf.open("config_data.conf")
sysconfig = configparser.ConfigParser()
sysconfig.readfp(config_file)
I'm running python 2.7.3 and doing some basic stuff involving the os module.
import os
def main():
f= os.popen('cat > out', 'w',1)
os.write(f, 'hello pipe')
os.close(f)
main()
Based on examples I've seen I would expect the code to work, but the interpreter gives this error:
Traceback (most recent call last):
File "./test.py", line 11, in <module>
main()
File "./test.py", line 8, in main
os.write(f, 'hello pipe')
TypeError: an integer is required
Okay, off to the documentation. The help page says:
write(...)
write(fd, string) -> byteswritten
Write a string to a file descriptor.
fd seems to stand for file descriptor. Presumably this is what you get when you do something like:
file = open('test.py')
No surprise, online documentation says the exact same thing.
What's going on here?
No, a "file descriptor" is an integer, not the file object. To go from a file object to a file descroptor, call file.fileno(). To wit:
>>> f = open("tmp.txt", "w")
>>> help(f.fileno)
Help on built-in function fileno:
fileno(...)
fileno() -> integer "file descriptor".
This is needed for lower-level file interfaces, such os.read().
>>> f.fileno()
4
Instead of using that, though, you probably just want to do the following, unless you really need to use the low-level functions for some reason:
f = os.popen('cat > out', 'w',1)
f.write('hello pipe')
f.close()
I am reading the bottle source code and see:
eval(compile('def _raise(*a): raise a[0], a[1], a[2]', '<py3fix>', 'exec'))
I read the documentation about compile and it only tells me that <string> is commonly used. I also googled and searched on stackoverflow but can not find the related info.
So can anyone tell me how <py3fix> affects compilation? And is there any other filename, where can I find the related documentation?
Thanks in advance.
It doesn't affect it at all. It's just a name that it's used to identify where the compiled code is coming from, so you can use string you want.
Like the docs say:
compile(source, filename, mode[, flags[, dont_inherit]])
The filename argument should give the file from which the code was
read; pass some recognizable value if it wasn’t read from a file
('< string>' is commonly used).
in the case where the source is not being read from a file (like here) they suggest that you use <string> so that you know that this code is compiled from a written string.
The person who comited the code, did it when fixing some Bottle Python 2/3 bugs. So I'm guessing that he used <py3fix> as a way to identify the assertion was raised from the def _raise he compiles when the user is running 2.x:
>>> eval(compile('def _raise(*a): raise a[0], a[1], a[2]', '<py3fix>', 'exec'))
>>> _raise(Exception, "error message", None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<py3fix>", line 1, in _raise
Exception: error message
>>> eval(compile('def _raise(*a): raise a[0], a[1], a[2]', '<my_source_file>', 'exec'))
>>> _raise(Exception, "error message", None)Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<my_source_file>", line 1, in _raise
Exception: error message
I know this may seem rude or mean or unpolite, but I need some help to try to figure out why I cant call window.loadPvmFile("f:\games#DD.ATC3.Root\common\models\a300\amu\dummy.pvm") exactly like that as a string. Instead of doing that, it gives me a traceback error:
Traceback (most recent call last):
File "F:\Python Apps\pvmViewer_v1_1.py", line 415, in <module>
window.loadPvmFile("f:\games\#DD.ATC3.Root\common\models\a300\amu\dummy.pvm")
File "F:\Python Apps\pvmViewer_v1_1.py", line 392, in loadPvmFile
file1 = open(path, "rb")
IOError: [Errno 22] invalid mode ('rb') or filename:
'f:\\games\\#DD.ATC3.Root\\common\\models\x07300\x07mu\\dummy.pvm'
Also notice, that in the traceback error, the file path is different. When I try a path that has no letters in it except for the drive letter and filename, it throws this error:
Traceback (most recent call last):
File "F:\Python Apps\pvmViewer_v1_1.py", line 416, in <module>
loadPvmFile('f:\0\0\dummy.pvm')
File "F:\Python Apps\pvmViewer_v1_1.py", line 393, in loadPvmFile
file1 = open(path, "r")
TypeError: file() argument 1 must be encoded string without NULL bytes, not str
I have searched for the place that the encode function is called or where the argument is encoded and cant find it. Flat out, I am out of ideas, frustrated and I have nowhere else to go. The source code can be found here: PVM VIEWER
Also note that you will not be able to run this code and load a pvm file and that I am using portable python 2.7.3! Thanks for everyone's time and effort!
\a and \0 are escape sequences. Use r'' (or R'') around the string to mark it as a raw string.
window.loadPvmFile(r"f:\games#DD.ATC3.Root\common\models\a300\amu\dummy.pvm")