I have a program where I am reading in a JSON file, and executing some SQL based on parameters specified in the file. The
load_json_file()
method loads the json file to a Python object first (not seen here but works correctly)
The issue is with the piece of the code here:
class TestAutomation:
def __init__(self):
self.load_json_file()
# connect to Teradata and load session to be used for execution
def connection(self):
con = self.load_json_file()
cfg_dsn = con['config']['dsn']
cfg_usr = con['config']['username']
cfg_pwd = con['config']['password']
udaExec = teradata.UdaExec(appName="DataAnalysis", version="1.0", logConsole=False)
session = udaExec.connect(method="odbc", dsn=cfg_dsn, username=cfg_usr, password=cfg_pwd)
return session
the init_ method first loads the JSON file, and then I store that in 'con'. I am getting an error though that reads:
cfg_dsn = con['config']['dsn']
E TypeError: 'NoneType' object is not subscriptable
The JSON file looks like this:
{
"config":{
"src":"C:/Dev\\path",
"dsn":"XYZ",
"sheet_name":"test",
"out_file_prefix":"C:/Dev\\test\\OutputFile_",
"password":"pw123",
"username":"user123",
"start_table":"11",
"end_table":"26",
"skip_table":"1,13,17",
"spot_check_table":"77"
}
}
the load_json_file() is defined like this:
def load_json_file(self):
if os.path.isfile(os.path.dirname(os.path.realpath(sys.argv[0])) + '\dwconfig.json'):
with open('dwconfig.json') as json_data_file:
cfg_data = json.load(json_data_file)
return cfg_data
Any ideas why I am seeing the error?
problem is that you're checking if the configuration file exists, then read it.
If it doesn't, your function returns None. This is wrong in many ways because os.path.realpath(sys.argv[0]) can return an incorrect value, for instance if the command is run with just the base name, found through the system path ($0 returns the full path in bash but not in python or C).
That's not how you get the directory of the current command.
(plus afterwards you're going to do with open('dwconfig.json') as json_data_file: which is now the name of the file, without the full path, wrong again)
I would skip this test, but compute the config file path properly. And if it doesn't exist, let the program crash instead of returning None that will crash later.
def load_json_file(self):
with open(os.path.join(os.path.dirname(__file__),'dwconfig.json')) as json_data_file:
cfg_data = json.load(json_data_file)
return cfg_data
So... cfg_dsn = con['config']['dsn']
something in there is set to None
you could be safe and write it like
(con or {}).get('config',{}).get('dsn')
or make your data correct.
Python 2.7
Download a test file from: www.py4inf.com/code/mbox.txt
Briefly, I need list all lines that start with 'From' and take only the mail address. Selecting line by line.
If the condition is true, write in other file the ( only) mail address (result). I could wrote the code and it is working. But It would be better if I use functions. I crashed when I tried to pass the parameters. I have a problem when I have a function that receive a parameter and send one or two.
The result is: copy line by line ALL input file in output file almost like a recursion. No search nothing and the file is very big.
At last, have you any page to read about funtions, paramt, passing paramt, pass reference, and other. Ask is easy and I prefer read and try to understand and if I have a problem, light a cande in the middle of the night!.
#Li is the input paramet. Line from fileRead(the pointer of the file).
#if the condition is true select all lines that start with From:
def funFormat(li):
if li.startswith('From:'):
li = li.rstrip('')
li = li.replace('From: ',' \n')
return li?
fileRead=open('mbox_small.txt','r')
for eachline in fileRead:
result=funFormat(eachline)
fileWrite =open('Resul73.txt', 'a')
fileWrite.write( result )
fileRead.close()
fileWrite.close()
You're opening a file every time you need to write, and closing it just at the end. Maybe that's what's messing it up? Try this and let me know if it works -
#Li is the input paramet. Line from fileRead(the pointer of the file).
#if the condition is true select all lines that start with From:
def funFormat(li):
if li.startswith('From:'):
li = li.rstrip('')
li = li.replace('From: ',' \n')
return li
else:
return None
fileRead = open('mbox_small.txt', 'r')
fileWrite = open('Resul73.txt', 'a')
for eachline in fileRead:
result=funFormat (eachline)
if result:
fileWrite.write (result)
fileRead.close()
fileWrite.close()
Also, I suggest you to read up on with blocks. This will help you work with files more efficiently. As for functions, there are enough resources online.
Running signtool.exe verify /a /v C:\Windows\notepad.exe I can see the signature for notepad.exe is in C:\Windows\system32\CatRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\ntexe.cat. How does signtool know that is where the signature exists for this pe file?
I am trying to replicate this signtool behaviour in python. Once I have the catalog file I can get the certificate information with the code below but I cannot see how windows links the file to the catalog.
import win32com.client
catpath = "C:\\Windows\\system32\\CatRoot\\{F----E}\\nt5.cat"
signedCode = win32com.client.Dispatch('capicom.signedcode')
signedCode.FileName=catpath
signedCode.Verify()
certs = signedCode.Certificates
for cert in certs:
print cert.Archived
print cert.IssuerName
print cert.SerialNumber
print cert.SubjectName
print cert.Thumbprint
print cert.ValidFromDate
print cert.ValidToDate
print cert.Version
But how do I get which security catalog file the executable is in?
Disclaimer: the followings is a rough guess based on testing as the exact process is undocumented.
Windows scans through every cat file in System32\CatRoot\{F7--EE}, add them to the system catalog database and roughly sort them by each entry's file hash/tag value.
(revealed by CatRoot2\dberr.txt which contains the log for the database process)
The database is the file System32\CatRoot2\{F7--EE}\catdb.
Inside catdb, a file hash is followed by its cat file name in CatRoot\{F7--EE}.
Note the hash excludes PE checksum & Certificate Table Entry.
The hash can be obtained from SignTool verify /v or this.
I am trying to create a function in Python 2.7.3 to open a SQLite database.
This is my code at the moment:
import sqlite3 as lite
import sys
db = r'someDb.sqlite'
def opendb(db):
try:
conn = lite.connect(db)
except sqlite3.Error:
print "Error open db.\n"
return False
cur = conn.cursor()
return [conn, cur]
I have tried the code above and I have observed that the sqlite3 library opens the database declared if exists, or creates a new database if this one doesn't exist.
Is there a way to check if the database exists with sqlite3 methods or I have to use file operation like os.path.isfile(path)?
In Python 2, you'll have to explicitly test for the existence using os.path.isfile:
if os.path.isfile(db):
There is no way to force the sqlite3.connect function to not create the file for you.
For those that are using Python 3.4 or newer, you can use the newer URI path feature to set a different mode when opening a database. The sqlite3.connect() function by default will open databases in rwc, that is Read, Write & Create mode, so connecting to a non-existing database will cause it to be created.
Using a URI, you can specify a different mode instead; if you set it to rw, so Read & Write mode, an exception is raised when trying to connect to a non-existing database. You can set different modes when you set the uri=True flag when connecting and pass in a file: URI, and add a mode=rw query parameter to the path:
from urllib.request import pathname2url
try:
dburi = 'file:{}?mode=rw'.format(pathname2url(db))
conn = lite.connect(dburi, uri=True)
except sqlite3.OperationalError:
# handle missing database case
See the SQLite URI Recognized Query Parameters documentation for more details on what parameters are accepted.
os.path.isfile() is just telling you if a file exists, not if it exists AND is a SQLite3 database! Knowing http://www.sqlite.org/fileformat.html, you could do this :
def isSQLite3(filename):
from os.path import isfile, getsize
if not isfile(filename):
return False
if getsize(filename) < 100: # SQLite database file header is 100 bytes
return False
with open(filename, 'rb') as fd:
header = fd.read(100)
return header[:16] == 'SQLite format 3\x00'
and subsequently use it like :
for file in files:
if isSQLite3(file):
print "'%s' is a SQLite3 database file" % file
else:
print "'%s' is not a SQLite3 database file" % file
Yes, there is a way to do what you want with Python 3.4+.
Use the sqlite3.connect() function to connect, but pass it a URI instead of a file path, and add mode=rw to its query string.
Here is a complete working code example:
import sqlite3
con = sqlite3.connect('file:aaa.db?mode=rw', uri=True)
This will open an existing database from a file named aaa.db in the current folder, but will raise an error in case that file can not be opened or does not exist:
Traceback (most recent call last):
File "aaa.py", line 2, in <module>
con = sqlite3.connect('file:aaa.db?mode=rw', uri=True)
sqlite3.OperationalError: unable to open database file
Python sqlite.connect() docs state that:
If uri is true, database is interpreted as a URI. This allows you to specify options. For example, to open a database in read-only mode you can use:
db = sqlite3.connect('file:path/to/database?mode=ro', uri=True)
More information about this feature, including a list of recognized options, can be found in the SQLite URI documentation.
Here's an excerpt of all the relevant URI option information collected from http://www.sqlite.org/c3ref/open.html:
mode: The mode parameter may be set to either "ro", "rw", "rwc", or "memory". Attempting to set it to any other value is an error. If "ro" is specified, then the database is opened for read-only access, just as if the SQLITE_OPEN_READONLY flag had been set in the third argument to sqlite3_open_v2(). If the mode option is set to "rw", then the database is opened for read-write (but not create) access, as if SQLITE_OPEN_READWRITE (but not SQLITE_OPEN_CREATE) had been set. Value "rwc" is equivalent to setting both SQLITE_OPEN_READWRITE and SQLITE_OPEN_CREATE. If the mode option is set to "memory" then a pure in-memory database that never reads or writes from disk is used. It is an error to specify a value for the mode parameter that is less restrictive than that specified by the flags passed in the third parameter to sqlite3_open_v2().
The sqlite3_open_v2() interface works like sqlite3_open() except that it accepts two additional parameters for additional control over the new database connection. The flags parameter to sqlite3_open_v2() can take one of the following three values, optionally combined with the SQLITE_OPEN_NOMUTEX, SQLITE_OPEN_FULLMUTEX, SQLITE_OPEN_SHAREDCACHE, SQLITE_OPEN_PRIVATECACHE, and/or SQLITE_OPEN_URI flags:
SQLITE_OPEN_READONLY
The database is opened in read-only mode. If the database does not already exist, an error is returned.
SQLITE_OPEN_READWRITE
The database is opened for reading and writing if possible, or reading only if the file is write protected by the operating system. In either case the database must already exist, otherwise an error is returned.
SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE
The database is opened for reading and writing, and is created if it does not already exist. This is the behavior that is always used for sqlite3_open() and sqlite3_open16().
For convenience, here's also a Python 3.4+ function for converting a regular path to an URI usable by sqlite.connect():
import pathlib
import urllib.parse
def _path_to_uri(path):
path = pathlib.Path(path)
if path.is_absolute():
return path.as_uri()
return 'file:' + urllib.parse.quote(path.as_posix(), safe=':/')
This is a fork (using Python 3) based on Tom Horen's answer, which presents a solution more complete and reliable that the elected answer.
The elected answer, does not evaluate any content, header, etc., in order to determine whether the file actually contains any data related to a SQLite3 database or not.
I tried to present something more pragmatic here:
#!/usr/bin/python3
import os
import sys
if os.path.isfile('test.sqlite3'):
if os.path.getsize('test.sqlite3') > 100:
with open('test.sqlite3','r', encoding = "ISO-8859-1") as f:
header = f.read(100)
if header.startswith('SQLite format 3'):
print("SQLite3 database has been detected.")
Building on a couple of other answers above. Here is a clean solution that works in Python 3.7.7:
def isSqlite3Db(db):
if not os.path.isfile(db): return False
sz = os.path.getsize(db)
# file is empty, give benefit of the doubt that its sqlite
# New sqlite3 files created in recent libraries are empty!
if sz == 0: return True
# SQLite database file header is 100 bytes
if sz < 100: return False
# Validate file header
with open(db, 'rb') as fd: header = fd.read(100)
return (header[:16] == b'SQLite format 3\x00')
Usage:
if isSqlite3Db('<path_to_db>'):
# ... <path_to_db> is a Sqlite 3 DB
Notes:
The answers checking file size is > 100 does not work as a new sqlite3 db created in recent python creates an file with length of 0.
Other examples reading file header returned bytes in Python 3.7.7 and not string so comparison would fail.
Examples that use sqlite3.connect(dburl, uri=True) did not work for me in Python 3.7.7 as it gave false positives.
I am using a function like the following at the beginning of my script so that I can try and figure out why a sqlite3 db script might not be working. Like the comments say, it uses 3 phases, checks if a path exist, checks if the path is a file, checks if that file's header is a sqlite3 header.
def checkdbFileforErrors():
#check if path exists
try:
with open('/path/to/your.db'): pass
except IOError:
return 1
#check if path if a file
if not isfile('/path/to/your.db'):
return 2
#check if first 100 bytes of path identifies itself as sqlite3 in header
f = open('/path/to/your.db', "rx")
ima = f.read(16).encode('hex')
f.close()
#see http://www.sqlite.org/fileformat.html#database_header magic header string
if ima != "53514c69746520666f726d6174203300":
return 3
return 0
In my application I have a method for users to convert a report to a PDF document. This works perfectly - once. If the user clicks the button again, the conversion hangs.
This is my code:
def print_report(self):
web = QtWebKit.QWebView()
filename = "reporttemplate.html"
file = open(filename,'r')
html = file.read()
file.close()
web.setHtml(html)
#web.show()
printer = QtGui.QPrinter()
printer.setPageSize(QtGui.QPrinter.Letter)
printer.setOutputFormat(QtGui.QPrinter.PdfFormat)
# ---- BROKEN ----
# This next line is where it hangs on the second call to this function.
# The first time it works, and generates the PDF as expected.
# ---- BROKEN ON THE NEXT LINE! ----
printer.setOutputFileName(r'C:\path\to\report\directory\file.pdf')
def convertIt():
web.print_(printer)
print "Pdf generated"
web.close()
QtCore.QObject.connect(web, QtCore.SIGNAL("loadFinished(bool)"), convertIt)
My thought is that the printer still has the file open. If that's the case, how can I close the file?
It works if I relaunch the application and the file already exists. For that reason, I don't believe it's hanging because the file already exists.
Testing your code I noticed that for me it only works when I put web.setHtml(html) at the end (last statement) in the print_report method. Doing that I was able to generate file.pdf as many times as I wanted to.