I have data stored in a shelf file created with python 2.7
When I try to access the file from python 3.4, I get an error:
>>> import shelve
>>> population=shelve.open('shelved.shelf')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python34\lib\shelve.py", line 239, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "C:\Python34\lib\shelve.py", line 223, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "C:\Python34\lib\dbm\__init__.py", line 88, in open
raise error[0]("db type could not be determined")
dbm.error: db type could not be determined
I'm still able to access the shelf with no problem in python 2.7, so there seems to be a backward-compatibility issue. Is there any way to directly access the old format with the new python version?
As I understand now, here is the path that lead to my problem:
The original shelf was created with Python 2 in Windows
Python 2 Windows defaults to bsddb as the underlying database for shelving, since dbm is not available on the Windows platform
Python 3 does not ship with bsddb. The underlying database is dumbdbm in Python 3 for Windows.
I at first looked into installing a third party bsddb module for Python 3, but it quickly started to turn into a hassle. It then seemed that it would be a recurring hassle any time I need to use the same shelf file on a new machine. So I decided to convert the file from bsddb to dumbdbm, which both my python 2 and python 3 installations can read.
I ran the following in Python 2, which is the version that contains both bsddb and dumbdbm:
import shelve
import dumbdbm
def dumbdbm_shelve(filename,flag="c"):
return shelve.Shelf(dumbdbm.open(filename,flag))
out_shelf=dumbdbm_shelve("shelved.dumbdbm.shelf")
in_shelf=shelve.open("shelved.shelf")
key_list=in_shelf.keys()
for key in key_list:
out_shelf[key]=in_shelf[key]
out_shelf.close()
in_shelf.close()
So far it looks like the dumbdbm.shelf files came out ok, pending a double-check of the contents.
The shelve module uses Python's pickle, which may require a protocol version when being accessed between different versions of Python.
Try supplying protocol version 2:
population = shelve.open('shelved.shelf', protocol=2)
According to the documentation:
Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes. Refer to PEP 307 for information about improvements brought by protocol 2.
This is most likely the protocol used in the original serialization (or pickling).
Edited: You may need to rename your database. Read on...
Seems like pickle is not the culprit here. shelve relies also in anydbm (Python 2.x) or dbm (Python 3) to create/open a database and store the pickled information.
I created (manually) a database file using the following:
# Python 2.7
import anydbm
anydbm.open('database2', flag='c')
and
# Python 3.4
import dbm
dbm.open('database3', flag='c')
In both cases, it creates the same kind of database (may be distribution dependent, this is on Debian 7):
$ file *
database2: Berkeley DB (Hash, version 9, native byte-order)
database3.db: Berkeley DB (Hash, version 9, native byte-order)
anydbm can open database3.db without problems, as expected:
>>> anydbm.open('database3')
<dbm.dbm object at 0x7fb1089900f0>
Notice the lack of .db when specifying the database name, though. But dbm chokes on database2, which is weird:
>>> dbm.open('database2')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.4/dbm/__init__.py", line 88, in open
raise error[0]("db type could not be determined")
dbm.error: db type could not be determined
unless I change the name of the name of the database to database2.db:
$ mv database2 database2.db
$ python3
>>> import dbm
>>> dbm.open('database2')
<_dbm.dbm object at 0x7fa7eaefcf50>
So, I suspect a regression on the dbm module, but I haven't checked the documentation. It may be intended :-?
NB: Notice that in my case, the extension is .db, but that depends on the database being used by dbm by default! Create an empty shelf using Python 3 to figure out which one are you using and what is it expecting.
I don't think it's possible to use a Python 2 shelf with Python 3's shelve module. The underlying files are completely different, at least in my tests.
In Python 2*, a shelf is represented as a single file with the filename you originally gave it.
In Python 3*, a shelf consists of three files: filename.bak, filename.dat, and filename.dir. Without any of these files present, the shelf cannot be opened by the Python 3 library (though it appears that just the .dat file is sufficient for opening, if not actual reading).
#Ricardo Cárdenes has given an overview of why this may be--it's likely an issue with the underlying database modules used in storing the shelved data. It's possible that the databases are backwards compatible, but I don't know and a quick search hasn't turned up any obvious answers.
I think it's likely that some of the possible databases implemented by dbm are backwards-compatible, whereas others are not: this could be the cause of the discrepancy between answers here, where some people, but not all, are able to open older databases directly by specifying a protocol.
*On every machine I've tested, using Python 2.7.6 vs Pythons 3.2.5, 3.3.4, and 3.4.1
The berkeleydb module includes a retro-compatible
implementation of the shelve object. (You will also need to install Oracle Berkeley DB)
You just need to:
import berkeleydb
from berkeleydb import dbshelve as shelve
population = shelve.open('shelved.shelf')
Related
I am exploring OpenVas tool for a project requirement, openVas is currently managed by Greenbone. I am getting error when I try to use remote scanner using python api.
I did all initial configuration, setup the required gui account etc and was able to scan the required systems manually however when I try to do the same using Python Api its not working. There isn't any example available on internet nor in there manual to verify my code.
I have used [https://pypi.org/project/python-gvm/] api.
I wrote simple code but its not working..
from gvm.connections import SSHConnection
from gvm.protocols.latest import Gmp
from gvm.transforms import EtreeTransform
from gvm.xml import pretty_print
connection = SSHConnection(hostname='192.168.1.84',username='alex',password='alex#123')
gmp = Gmp(connection)
gmp.authenticate('admin', 'admin')
# Retrieve current GMP version
version = gmp.get_version()
# Prints the XML in beautiful form
pretty_print(version)
I am getting error-
/usr/bin/python3.7 /home/punshi/PycharmProjects/nessus_api/openvas-greenbone.py
/usr/local/lib/python3.7/dist-packages/paramiko/kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
m.add_string(self.Q_C.public_numbers().encode_point())
/usr/local/lib/python3.7/dist-packages/paramiko/kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
self.curve, Q_S_bytes
/usr/local/lib/python3.7/dist-packages/paramiko/kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
hm.add_string(self.Q_C.public_numbers().encode_point())
Traceback (most recent call last):
File "/home/punshi/PycharmProjects/nessus_api/openvas-greenbone.py", line 8, in <module>
gmp.authenticate('admin', 'admin')
File "/usr/local/lib/python3.7/dist-packages/gvm/protocols/gmpv7.py", line 211, in authenticate
response = self._read()
File "/usr/local/lib/python3.7/dist-packages/gvm/protocols/base.py", line 54, in _read
return self._connection.read()
File "/usr/local/lib/python3.7/dist-packages/gvm/connections.py", line 126, in read
raise GvmError('Remote closed the connection')
gvm.errors.GvmError: Remote closed the connection
Process finished with exit code 1
I have tested SSH connection manually so the problem is either with my code or some other.
Additional Detail-
Ubuntu 16,
Greenbone Security Assistant 7.0.3 (gui)
Open Vas - 9.0.3
I've exactly the same problem that I solved it with TLSConnection instead of SSHConnection. Here is your code:
import gvm
from gvm.protocols.latest import Gmp
from gvm.transforms import EtreeTransform
from gvm.xml import pretty_print
connection =gvm.connections.TLSConnection(hostname='192.168.1.84')
gmp = Gmp(connection)
gmp.authenticate('admin', 'admin')
# Retrieve current GMP version
version = gmp.get_version()
# Prints the XML in beautiful form
pretty_print(version)
I am exploring OpenVas tool for a project requirement, openVas is currently managed by Greenbone.
Just a side note. OpenVAS is developed by Greenbone since many years now. Therefore we did rename the project to Greenbone Vulnerability Management (GVM) with version 10. Only the actual scanner component will be still named after OpenVAS. See https://community.greenbone.net/t/is-openvas-manager-and-gvmd-the-same/1777/3 for more details.
Using the SSHConnection needs some additional setup at the remote server. Using TLSConnection might be easier but needs also changes in the settings of gvmd/openvasmd because it is only listening on a unix socket by default.
I am trying to connect to a SQL Anywhere database via python. I have created the DSN and I can use command prompt to connect to the database using dbisql - c "DNS=myDSN". When I try to connect through python using con = sqlanydb.connect(DSN= "myDSN")
I get
`Traceback (most recent call last):
File "<pyshell#5>", line 1, in <module>
con = sqlanydb.connect(DSN= "RPS Integration")
File "C:\Python27\lib\site-packages\sqlanydb.py", line 522, in connect
return Connection(args, kwargs)
File "C:\Python27\lib\site-packages\sqlanydb.py", line 538, in __init__
parent = Connection.cls_parent = Root("PYTHON")
File "C:\Python27\lib\site-packages\sqlanydb.py", line 464, in __init__
'libdbcapi_r.dylib')
File "C:\Python27\lib\site-packages\sqlanydb.py", line 456, in load_library
raise InterfaceError("Could not load dbcapi. Tried: " + ','.join(map(str, names)))
InterfaceError: (u'Could not load dbcapi. Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)`
I was able to resolve the problem. I was never able to use the sqlanydb.connect. I ended up using pyodbc. So my final connection string was con = pydobc.connect(dsn="myDSN"). I think that sqlanydb is only fully functional with sqlanydb 17 and I was using a previous version.
Depending on how Sybase is locally installed, it could also be that the Python module really cannot find the (correct) library when looking up dbcapi.dll in your PATH. A quick fix for that is to create a new SQLANY_API_DLL environment variable (that's the initial None in the names the error message says it tried using; it takes priority over all the others) containing the correct path, usually something like %SQLANY16%\Bin64\dbcapi.dll depending on what version you have installed (Sybase usually creates an environment variable pointing to its installation folder on a per version basis).
One way to encounter the backtrace shown by the OP is running 64-bit Python interpreter, which is unable to load the 32-bit dbcapi.dll. Try launching with "py -X-32" to force a 32-bit engine, or reinstall using a 32-bit python engine.
Unfortunately sqlanydb's code that tries to be smart about finding dbcapi also ends up swallowing the exceptions thrown during loading. The sqlanydb author probably assumed that failure to load implies file not found, which isn't always the case.
I was stoked in this same problem, both windows system32 and 64.
I'm using Python 3.9 and I found that since Python 3.8 the cdll.LoadLibrary function does no longer search PATH to find library files.
To fix this, I created the enviroment variable SQLANY_API_DLL pointing to the route of sqlanywhere installation, in my case version 17, bin32 or bin64 depending on your system.
Using the shelve standard lib on Python 2.7.2, I wrote an extremely simple test to create a persistent data file and then immediately open it for printing:
import os
import shelve
shelf_filename = str(__file__.split('.')[0] + '.dat')
#delete the shelf file if it exists already.
try:
os.remove(shelf_filename)
print "DELETED LEFTOVER SHELF FILE", shelf_filename
except OSError:
pass
#create a new shelf, write some data, and flush it to disk
shelf_handle = shelve.open(shelf_filename)
print "OPENED", shelf_filename, "WITH SHELF"
shelf_handle['foo'] = 'bar'
shelf_handle.close()
print "FLUSHED AND CLOSED THE SHELF"
#re-open the shelf we just wrote, read/print the data, and close it
shelf_handle = shelve.open(shelf_filename)
print "RE-OPENED", shelf_filename, "WITH SHELF"
print 'foo:', shelf_handle.get('foo')
shelf_handle.close()
#delete the shelf file
os.remove(shelf_filename)
However this script fails unexpectedly when it tries to re-open the shelf it just created:
DELETED LEFTOVER SHELF FILE shelve_test.dat
OPENED shelve_test.dat WITH SHELF
FLUSHED AND CLOSED THE SHELF
Traceback (most recent call last):
File "shelve_test.py", line 21, in <module>
shelf_handle = shelve.open(shelf_filename)
File ".../shelve.py", line 239, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File ".../shelve.py", line 223, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
File ".../anydbm.py", line 82, in open
raise error, "db type could not be determined"
anydbm.error: db type could not be determined
This is about the most basic possible usage of shelve so I really don't understand what I'm missing, and AFAIK I'm following the docs to the letter. Can somebody with a little shelve experience please tell me what's going on?
UPDATE: This script apparently works in certain platforms and not others, which is really kind of surprising for a Python standard lib. Definitely confirmed this does NOT work on:
Python 2.7.2 (default, May 15 2013, 13:46:05)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
This version of Python shelve is using a deprecated dbm package, and the issue can be solved by specifying a different dbm as follows:
import anydbm
anydbm._defaultmod = __import__('dumbdbm')
If those two lines are added to the script above then everything works as expected. This really needs to be fixed in shelve.
shelve doesn't expect you to specify the extension on the filename. Instead it gives the file an extension based on which db was used to create the file.
So when you call shelve.open("shelve_test.dat"), you are creating a file that shelve expects to be created in using dumbdbm, but is instead created with a different (better) database manager.
Instead, you should be calling shelve.open("shelve_test"), that way shelve can choose to save the file using the correct extension. This also applies when opening an already created file.
I am hoping for a little advice on shelves/databases in Python.
Problem: I have a database created on the mac, that I want to use on windows 7. I use Python 3.2, MacOS 10.7, and win 7.
When I open and save my shelve on the mac all is good and well. I get a file with a ".db" extension. On my windows-python it is not recognized. I can however create a new db on the pc and get files with ".bak, dat, .dir" extensions.
I am guessing that the python on the pc does not have the same underlying database that my mac-python uses?
I am not sure which is the correct approach here, but maybe I could:
Change the default-db that my systems uses?
Find out which db my mac-python uses and add that on the pc?
Change the way I store my data all together?
Speed is not an issue, the datasize is a few megabytes, and it's not accessed very often.
Hope to find a helping hand out there. Thanks in advance - any help is much appreciated.
/Esben
What I am doing:
Import shelve
db = shelve.open('mydb')
entries = db['list']
db.close
It's pretty straight forward, I have a working db-file called "mydb.db" on the mac but when I try to open it on the pc-python i get:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/dbm/init.py", line 107, in whichdb
f = io.open(filename + ".pag", "rb")
IOError: [Errno 2] No such file or directory: 'mydb.pag'
Thank you for your reply!
I seems that shelves in python are not easily forced to use a specific db, however pickles works like a charm. At least from mac os -> windows 7.
So short answer: If you want portability, don't use shelves, use pickles directly.
/Esben
sqlite3 module is a cross platform module that is even supported by many other languages and tools.
pickle module is simpler, but also cross platform. You give it an object and it dumps it to a file. No tables or rows like sqlite.
I ran into the same issue and implemented a dict-based class that supports loading and writing the contents of the dict from and to disk.
from pathlib import Path
import pickle
class DiskDict(dict):
def __init__(self, sync_path: Path):
self.path = sync_path
if self.path.exists():
with open(self.path, "rb") as file:
tmp_dct = pickle.load(file)
super().update(tmp_dct)
print(f"loaded DiskDict with {len(tmp_dct)} items from {self.path}")
def sync_to_disk(self):
with open(self.path, "wb") as file:
tmp_dct = super().copy()
pickle.dump(tmp_dct, file)
print(f"saved DiskDict with {len(tmp_dct)} items to {self.path}")
You might try using pysos, it's a lightweight shelve alternative that is also cross platform.
Install using pip: pip install pysos
Example usage:
import pysos
db = pysos.Dict('myfile.db')
db['hello'] = 'persistence!'
db.close()
The advantage is also that everything is contained in this single file myfile.db so you can easely just copy it around.
I ran across an odd problem while trying to transfer a project to a windows machine.
In my project I use a session handler (http://gaeutilities.appspot.com/session) it works fine on my mac but on windows I get:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\webapp__init__.py", line 510, in call
handler.get(*groups)
File "C:\Development\Byggmax.Affiliate\bmaffiliate\admin.py", line 29, in get
session = Session()
File "C:\Development\Byggmax.Affiliate\bmaffiliate\appengine_utilities\sessions.py", line 547, in init
self.cookie.load(string_cookie)
File "C:\Python26\lib\Cookie.py", line 628, in load
for k, v in rawdata.items():
AttributeError: 'unicode' object has no attribute 'items'
Anyone familiar with the Session Handler that knows anything of this? All help are welcome!
..fredrik
The bug is pretty clear by glancing at the sources, although perfectly OS-independent. In sessions.py lines 544-547:
string_cookie = os.environ.get(u"HTTP_COOKIE", u"")
self.cookie = Cookie.SimpleCookie()
self.output_cookie = Cookie.SimpleCookie()
self.cookie.load(string_cookie)
lines 544 makes it very likely that string_cookie is unicode (though it might be a byte string from the environment, those u"" mean that the sessions.py author is trying hard to get it as unicode!-). Meanwhile in Cookie.py lines 624-628:
if type(rawdata) == type(""):
self.__ParseString(rawdata)
else:
# self.update() wouldn't call our custom __setitem__
for k, v in rawdata.items():
line 624 only parses a byte string: anything else (including a unicode string!) is treated like a dict (whence the crash).
Obviously this Cookie.py is emphatically not the one this sessions.py was developed for. So what can possibly have happened...? Well, we do know of course that App Engine is strictly Python 2.5 and the Cookie.py we showed is that of Python 2.6. So let's look at Cookie.py in 2.5 (lines 618-621 in this version):
if type(rawdata) == type(""):
self.__ParseString(rawdata)
else:
self.update(rawdata)
so in 2.5, given an empty unicode string, the cookie (which subclasses dict) does self.update(u'')... which is an innocuous no-op. The comment in 2.6 shows why the maintainer of Cookie.py switched to the current loop... which breaks when rawdata's u''.
Long story short: install Python 2.5 on your Windows machine, and use that version with the GAE SDK, not the 2.6 you're currently using -- unless you truly love debugging of extremely subtle version differences where an incorrect use was innocuous in 2.5 but breaks in 2.6, like this one;-). Also enter a bug about this in the gaeutilities tracker, since that call to load with an empty unicode string is simply wrong, even though in 2.5 it happens to be innocuous.
Specifically, get the appropriate Windows msi of 2.5.4 from here, depending on whether you have a 32-bit or 64-bit version of Windows.
Followed the link to this post from the issue posted on the projects issue tracker. As posted there, my response is I won't be focusing on applying updates to make the project work with Python 2.6. However, I will put a little more emphasis on taking a look at the call to load with an empty unicode string.