I am creating a program that loads and runs python scripts from a compressed file. Along with those python scripts, I have a config file that I previously used configparser to load info from in an uncompressed version of the program.
Is it possible to directly read config files in zip files directly with configparser? or do I have to unzip it into a temp folder and load it from there?
I have tried directly giving the path:
>>> sysconf = configparser.ConfigParser()
>>> sysconf.read_file("compressed.zip/config_data.conf")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
self._read(f, source)
File "/usr/local/lib/python3.4/configparser.py", line 1058, in _read
raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: '<???>', line: 1
Didn't work. no surprises there.
Then I tried using zipfile
>>> zf = zipfile.ZipFile("compressed.zip")
>>> data = zf.read("config_data.conf")
>>> sysconf = configparser.ConfigParser()
>>> sysconf.read_file(data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
self._read(f, source)
File "/usr/local/lib/python3.4/configparser.py", line 1009, in _read
if line.strip().startswith(prefix):
AttributeError: 'int' object has no attribute 'strip'
and found that it didn't work either.
so I've resorted to creating a temp folder, uncompressing to it, and reading the conf file there. I would really like to avoid this if possible as the conf files are the only limiting factor. I can (and am) loading the python modules from the zip file just fine at this point.
I can get the raw text of the file if there's a way to pass that directly to configparser, but searching the docs I came up empty handed.
Update:
I tried using stringIO as a file object, and it seems to work somewhat.
configparser doesn't reject it, but it doesn't like it either.
>>> zf = zipfile.ZipFile("compressed.zip")
>>> data = zf.read(config_data.conf)
>>> confdata = io.StringIO(str(data))
>>> sysconf = configparser.ConfigParser()
>>> sysconf.readfp(confdata)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/configparser.py", line 736, in readfp
self.read_file(fp, source=filename)
File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
self._read(f, source)
File "/usr/local/lib/python3.4/configparser.py", line 1058, in _read
raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: '<???>', line: 1
(continues to spit out the entire contents of the file)
If I use read_file instead, it doesn't error out, but doesn't load anything either.
>>> zf = zipfile.ZipFile("compressed.zip")
>>> data = zf.read(config_data.conf)
>>> confdata = io.StringIO(str(data))
>>> sysconf = configparser.ConfigParser()
>>> sysconf.read_file(confdata)
>>> sysconf.items("General") #(this is the main section in the file)
Traceback (most recent call last):
File "/usr/local/lib/python3.4/configparser.py", line 824, in items
d.update(self._sections[section])
KeyError: 'General'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/configparser.py", line 827, in items
raise NoSectionError(section)
configparser.NoSectionError: No section: 'General'
can get the raw text of the file if there's a way to pass that directly to configparser
Try configparser.ConfigParser.read_string
When coupled with an appropriate ZIP file, this code works for me:
import zipfile
import configparser
zf = zipfile.ZipFile("compressed.zip")
zf_config = zf.open("config_data.conf", "rU")
zf_config_data = zf_config.read().decode('ascii')
config = configparser.ConfigParser()
config.read_string(zf_config_data)
assert config['today']['lunch']=='cheeseburger'
Upon reflection, the following might be more appropriate:
import zipfile
import configparser
import io
zf = zipfile.ZipFile("compressed.zip")
zf_config = zf.open("config_data.conf", "rU")
zf_config = io.TextIOWrapper(zf_config)
config = configparser.ConfigParser()
config.read_file(zf_config)
assert config['today']['lunch']=='cheeseburger'
As written in comments, #matthewatabet answer won't work with Python 3.4 (and newer vesions). It's because ZipFile.open now returns a "bytes-like" object and not a "file-like" object anymore. You can use:
codecs.getreader("utf-8")(config_file)
To convert the config_file bytes-like object into a file-like object using the UTF-8 encoding. The code is now:
import zipfile, configparser, codecs
# Python >= 3.4
with zipfile.ZipFile("compressed.zip") as zf:
config_file = zf.open("config_data.conf") # binary mode
sysconfig = configparser.ConfigParser()
sysconfig.read_file(codecs.getreader("utf-8")(config_file))
That seems more satisfactory than creating a string, but I don't know if it's more efficient...
EDIT Since Python 3.9, the zipfile module provides a zipfile.Path.open method that can handle text and binary modes. Default is text mode. The following code works fine:
# Python >= 3.9
with zipfile.ZipFile("compressed.zip") as zf:
zip_path = zipfile.Path(zf)
config_path = zip_path / "config_data.conf"
config_file = config_path.open() # text mode
sysconfig = configparser.ConfigParser()
sysconfig.read_file(config_file)
ZipFile not only supports read but also open, which returns a file-like object. So, you could do something like this:
zf = zipfile.ZipFile("compressed.zip")
config_file = zf.open("config_data.conf")
sysconfig = configparser.ConfigParser()
sysconfig.readfp(config_file)
Related
I recieve some files with .ini file with them. I have to recieve file names from [FILES] section.
Sometimes there is an extra witespace in another section of .ini-file which raises exception in ConfigParser module
The example of "bad" ini-file:
[LETTER]
SUBJECT=some text
some text
and text with whitespace in the beggining
[FILES]
0=file1.txt
1=file2.doc
My code(Python 3.7):
import configparser
def get_files_from_ini_file(info_file):
ini = configparser.ConfigParser(allow_no_value=True)
ini.read(info_file) # ERROR is here
if ini.has_section("FILES"):
pocket_files = [ini.get("FILES", i) for i in ini.options("FILES")]
return pocket_files
print(get_files_from_ini_file("D:\\bad.ini"))
Traceback (most recent call last):
File "D:/test.py", line 10, in <module>
print(get_files_from_ini_file("D:\\bad.ini"))
File "D:/test.py", line 5, in get_files_from_ini_file
ini.read(info_file) # ERROR
File "C:\Users\ap\AppData\Local\Programs\Python\Python37-32\lib\configparser.py", line 696, in read
self._read(fp, filename)
File "C:\Users\ap\AppData\Local\Programs\Python\Python37-32\lib\configparser.py", line 1054, in _read
cursect[optname].append(value)
AttributeError: 'NoneType' object has no attribute 'append'
I can't influence on files I recieve so that is there any way to ignore this error? In fact I need only [FILES] section to parse.
Have tried empty_lines_in_values=False with no result
May be that's invalid ini file and I should write my own parser?
If you only need the "FILES" part, a simple way is to:
open the file and read into a string
get the part after "[FILES]" using .split() method
add "[FILES]" before the string
use the configparser read_string method on the string
This is a hacky solution but it should work:
import configparser
def get_files_from_ini_file(info_file):
with open(info_file, 'r') as file:
ini_string = file.read()
useful_part = "[FILES]" + ini_string.split("[FILES]")[-1]
ini = configparser.ConfigParser(allow_no_value=True)
ini.read_string(useful_part) # ERROR is here
if ini.has_section("FILES"):
pocket_files = [ini.get("FILES", i) for i in ini.options("FILES")]
return pocket_files
print(get_files_from_ini_file("D:\\bad.ini"))
I`ve got an stream which contains .tar file conten, so I work with it using tarfile.open('r|')
What I need to do - is to look into list of files inside it and read some of them, then upload whole tar into another place.
When I try to tarfile.extractfile() after tarfile.getnames() it raises an tarfile.StreamError. But I cannot extract file which name I dont know.
How can I get list of files without crushing tarfile? I cannot save the whole tar into RAM\disk, because some files inside it can be larger than 10GB.
>>> tf = tarfile.open(fileobj=open('Downloads/clean-alpine.ova', 'rb'), mode='r|')
>>> tfn = tf.getnames()
>>> tfn
['clean-alpine.ovf', 'clean-alpine.mf', 'clean-alpine-disk1.vmdk']
>>> tf.fileobj
<tarfile._Stream object at 0x7ff878dac7b8>
>>> tf.fileobj.pos
33595392
>>> ovf = tf.extractfile('clean-alpine.ovf')
>>> ovf
<ExFileObject name=''>
>>> d = ovf.read().decode()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/tarfile.py", line 696, in read
self.fileobj.seek(offset + (self.position - start))
File "/usr/lib/python3.6/tarfile.py", line 522, in seek
raise StreamError("seeking backwards is not allowed")
tarfile.StreamError: seeking backwards is not allowed
Looking at the source of TarFile.extractall() the important bit is to use TarFile as an iterable, like I did in my use case:
for member in tf:
if not member.isfile():
continue
dest = Path.cwd() / member.name # This is vulnerable to, like, 5 things
with tf.extractfile(member) as tfobj:
dest.write_bytes(tfobj.read())
How do I turn this format of TXT file into a CSV file?
Date,Open,high,low,close
1/1/2017,1,2,1,2
1/2/2017,2,3,2,3
1/3/2017,3,4,3,4
I am sure you can understand? It already has the comma -eparated values.
I tried using numpy.
>>> import numpy as np
>>> table = np.genfromtxt("171028 A.txt", comments="%")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Smith\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\lib\npyio.py", line 1551, in genfromtxt
fhd = iter(np.lib._datasource.open(fname, 'rb'))
File "C:\Users\Smith\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\lib\_datasource.py", line 151, in open
return ds.open(path, mode)
File "C:\Users\Smith\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\lib\_datasource.py", line 501, in open
raise IOError("%s not found." % path)
OSError: 171028 A.txt not found.
I have (S&P) 500 txt files to do this with.
You can use csv module. You can find more information here.
import csv
txt_file = 'mytext.txt'
csv_file = 'mycsv.csv'
in_txt = csv.reader(open(txt_file, "r"), delimiter=',')
out_csv = csv.writer(open(csv_file, 'w+'))
out_csv.writerows(in_txt)
Per #dclarke's comment, check the directory from which you run the code. As you coded the call, the file must be in that directory. When I have it there, the code runs without error (although the resulting table is a single line with four nan values). When I move the file elsewhere, I reproduce your error quite nicely.
Either move the file to be local, add a local link to the file, or change the file name in your program to use the proper path to the file (either relative or absolute).
The issue is that I'm pulling data from one source and I want to save it to dropbox as a pickle file. I can't save it in a directory, because I'm running the code on a server (iron.io).
import tempfile
import pickle
def SFDCDropboxSync(Data):
f = tempfile.NamedTemporaryFile(delete=False)
pickle.dump(Data,open(f,'wb'))
client = dropbox.client.DropboxClient(access_token)
client.put_file(filename, f)
This is the error I get:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Shippy/RecurringDataDump/SFDCDropboxUpload.py", line 38, in <module>
if __name__ == "__main__": main() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Shippy/RecurringDataDump/SFDCDropboxUpload.py", line 31, in main
print SFDCDropboxUploadDownload().SFDCDropboxSync(lst) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Shippy/RecurringDataDump/SFDCDropboxUpload.py", line 26, in SFDCDropboxSync
pkl = self.SaveListtoPickle(lst) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Shippy/RecurringDataDump/SFDCDropboxUpload.py", line 20, in SaveListtoPickle
pickle.dump(lst,open(f,'wb')) TypeError: coercing to Unicode: need string or buffer, instance found [Finished in 0.7s with exit code 1] [shell_cmd: python -u "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Shippy/RecurringDataDump/SFDCDropboxUpload.py"] [dir: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Shippy/RecurringDataDump] [path: /usr/bin:/bin:/usr/sbin:/sbin]
In your code, the NamedTemporaryFile f is not a string. It is a file object, similar to the output of open(file_path).
From the documentation: This file-like object can be used in a with statement, just like a normal file.
If you want to path to the created file, use tmp_file.name
For example, this works: (tested on python 3.6.2)
def SFDCDropboxSync(Data):
with tempfile.NamedTemporaryFile() as tmp_file:
pickle.dump(Data, tmp_file)
tmp_file.flush()
print(pickle.load(open(tmp_file.name, 'rb')))
This will delete the file when it exits the while (file closes).
Warning for Windows: you might have trouble reading the file while it is open. Instead, use something similar to this:
with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
pickle.dump(Data, open(tmp_file.name, 'wb'))
tmp_filename = tmp_file.name
pickle.load(open(tmp_filename, 'rb'))
os.remove(tmp_filename)
I'm having an issue with web2py. I have a text file called defVals.txt that's in the modules folder. I try to read from it, using open("defVals.txt") (in a Module in the same director as defVals.txt), but I get the error:
Traceback (most recent call last):
File "/home/jordan/web2py/gluon/restricted.py", line 212, in restricted
exec ccode in environment
File "/home/jordan/web2py/applications/randommotif/controllers/default.py", line 67, in <module>
File "/home/jordan/web2py/gluon/globals.py", line 188, in <lambda>
self._caller = lambda f: f()
File "/home/jordan/web2py/applications/randommotif/controllers/default.py", line 13, in index
defaultData = parse('defVals.txt')
File "applications/randommotif/modules/defaultValParser.py", line 6, in parse
lines = open(fileName)
IOError: [Errno 2] No such file or directory: 'defVals.txt'
What am I doing wrong? Where should I place defVals.txt
I'm using Ubuntu 12.10
Thanks,
Jordan
Update:
This is the source code to defaultValParser.py:
import itertools
import string
import os
from gluon import *
from gluon.custom_import import track_changes; track_changes(True)
#this returns a dictionary with the variables in it.
def parse(fileName):
moduleDir = os.path.dirname(os.path.abspath('defaultValParser.py'))
filePath = os.path.join(moduleDir, fileName)
lines = open(filePath, 'r')
#remove lines that are comments. Be sure to remove whitespace in the beginning and end of line
real = filter(lambda x: (x.strip())[0:2] != '//', lines)
parts = (''.join(list(itertools.chain(*real)))).split("<>")
names = map(lambda x: (x.split('=')[0]).strip(), parts)
values = map(lambda x: eval(x.split('=')[1]), parts)
return dict(zip(names, values))
It works fine if I import it and call it from a terminal (provided I comment out the gluon imports), but if I call it from a web2py controller, it fails completely:
Traceback (most recent call last):
File "/home/jordan/web2py/gluon/restricted.py", line 212, in restricted
exec ccode in environment
File "/home/jordan/web2py/applications/randommotif/controllers/default.py", line 71, in <module>
File "/home/jordan/web2py/gluon/globals.py", line 188, in <lambda>
self._caller = lambda f: f()
File "/home/jordan/web2py/applications/randommotif/controllers/default.py", line 17, in index
defaultData = parse('defVals.txt')
File "applications/randommotif/modules/defaultValParser.py", line 6, in parse
IOError: [Errno 2] No such file or directory: 'defVals.txt'
Use an absolute path based on the __file__ path of the module:
moduledir = os.path.dirname(os.path.abspath('__file__'))
# ..
defaultData = parse(os.path.join(moduledir, 'defVals.txt'))
__file__ is the filename of the current module, using the .dirname() of that gives you the directory the module is in. I used .abspath() to make sure you have an absolute path for your module file at all times, heading off some edgecases you could hit otherwise.
moduledir is a global in your module.