I am diving down into the world of Python, practicing by writing a simple inventory based application for my dvd collection as a means to get aquainted to working with sqlite3.
As part of my project, I am using a ini file for settings, and then reading the values from that in a shared library that is called from another file. I am curious as to feedback on my methods, especially my use of the config file, and best coding practices around it.
The config is formatted as follows, named config.ini
[main]
appversion = 0.1.0
datasource = data\database.db
my utils library is then formatted as follows:
import os
import sqlite3
from configparser import ConfigParser
CONFIG_PATH = os.path.join(os.path.dirname(__file__), 'config/config.ini')
def get_settings(config_path=CONFIG_PATH):
config = ConfigParser()
config.read(config_path)
return config
def db_connect():
config = get_settings()
con = sqlite3.connect(config.get('main', 'datasource'))
return con
Finally, my test lookup, which does function is:
from utils import db_connect
def asset_lookup():
con = db_connect()
cur = con.cursor()
cur.execute("SELECT * FROM dvd")
results = cur.fetchall()
for row in results:
print(row)
My biggest question is in regards to my building of the data connection from within utils.py. First I'm reading the file, then form within the same script, building the data connection from a setting within the ini file. This is then read by other files. This was my method of trying to be modular, but I was not sure if its proper.
Thanks in advance.
To directly answer your question, you could do something like this to cache your objects so you don't create/open them over and over again whenever you call one of the functions in utils.py:
import os
import sqlite3
from configparser import ConfigParser
CONFIG_PATH = os.path.join(os.path.dirname(__file__), 'config/config.ini')
config = None
con = None
def get_settings(CONFIG_PATH):
global config
if config is None:
config = ConfigParser()
config.read(CONFIG_PATH)
return config
def db_connect():
global con
if con is None:
config = get_settings()
con = sqlite3.connect(config.get('main', 'datasource'))
return con
While this might solve your problem, it relies heavily on global variables, which might cause problems elsewhere. Typically, that's where you switch to classes as containers for your code parts that belong together. For example:
import os
import sqlite3
from configparser import ConfigParser
class DVDApp:
CONFIG_PATH = os.path.join(os.path.dirname(__file__), 'config/config.ini')
def __init__(self):
self.config = ConfigParser()
self.config.read(self.CONFIG_PATH)
self.con = sqlite3.connect(self.config.get('main', 'datasource'))
def asset_lookup(self):
cur = self.con.cursor()
cur.execute("SELECT * FROM dvd")
results = cur.fetchall()
for row in results:
print(row)
Initializing config and connection objects held in self reduces to just 3 lines of code. Thereby making it almost unnecessary to split your code over several files. And even if so, it would be enough to share the one instance of DVDApp between modules, which then holds all the other shared objects.
Related
Let's assume, that there are following minimalistic python classes inside one module, e.g. Module:
module/
__init__.py
db.py
document.py
db.py
import yaml
class DB(object):
config = {}
#classmethod
def load_config(cls, config_path):
cls.config = yaml.load(open(config_path, 'r').read())
and document.py
from .db import DB
class Document(object):
db = None
def __init__(self):
self.db = DB()
End-user is going to use such Module as follows:
from Module import DB, Document
DB.load_config('/path/to/config.yml')
Document.do_some_stuff()
doc1 = Document()
doc2 = Document.find(...)
doc2.update_something(...)
doc2.save()
It is expected that Document class and every instance of it will have internally an access to class DB with a config specified by user. However, since Document performs an internal import of DB class (from .db import DB) it receives a 'fresh' DB class with default config.
I did a lot of searches, most of questions and answers are about module-wide configs, but not specified by the end user.
How can I achieve such functionality? I guess that there is some architectural problem here, but what is the most simple way to solve it?
Perhaps this isn't the most appropriate answer, but a few months back I wrote a module called aconf for this exact purpose. It's a memory-based global configuration module for Python written in 8 lines. The idea is you can do the following:
You create a Config object to force the user to input the configuration your program requires (in this case it's inside config.py):
""" 'Config' class to hold our desired configuration parameters.
Note:
This is technically not needed. We do this so that the user knows what he/she should pass
as a config for the specific project. Note how we also take in a function object - this is
to demonstrate that one can have absolutely any type in the global config and is not subjected
to any limitations.
"""
from aconf import make_config
class Config:
def __init__(self, arg, func):
make_config(arg=arg, func=func)
You consume your configuration throughout your module (in this case, inside functionality.py):
""" Use of the global configuration through the `conf` function. """
from aconf import conf
class Example:
def __init__(self):
func = conf().func
arg = conf().arg
self.arg = func(arg)
And then use it (in this case inside main.py):
from project.config import Config
from project.functionality import Example
# Random function to demonstrate we can pass _anything_ to 'make_config' inside 'Config'.
def uppercase(words):
return words.upper()
# We create our custom configuration without saving it.
Config(arg="hello world", func=uppercase)
# We initialize our Example object without passing the 'Config' object to it.
example = Example()
print(example.arg)
# >>> "HELLO WORLD"
The entire aconf module is the following:
__version__ = "1.0.1"
import namedtupled
def make_config(**kwargs):
globals()["aconf"] = kwargs
conf = lambda: namedtupled.map(globals()["aconf"])
config = lambda: globals()["aconf"]
... in essence, you just save your configuration to globals() during runtime.
It's so stupid it makes me wonder if you should even be allowed to do this. I wrote aconf for fun, but have never personally used it in a big project. The reality is, you might run into the problem of making your code weird for other developers.
I have developed few libraries for robot framework for my feature testing, for these libraries all variables are coming from a variables.py file. Below is the code block for variables.py:
#!/usr/bin/env python
import sys
import os
import optparse
import HostProperties
import xml.etree.ElementTree as ET
from robot.api import logger
testBed = 748
tree = ET.parse('/home/p6mishra/mybkp/testLibs/TestBedProperties.xml')
class raftGetTestBedProp(object):
def GetTestBedNumber(self):
_attributeDict = {}
root = tree.getroot()
for _tbProperties in root:
for _tbNumber in _tbProperties:
get_tb = _tbNumber.attrib
if get_tb['name']== str(testBed):
get_tb2 = _tbNumber.attrib
return root, get_tb2['name']
def GetTestBedProperties(self, root, testBedNumber):
propertyList = []
for _tbProperties in root:
get_tb = _tbProperties.attrib
for _tbProperty in _tbProperties:
get_tb1 = _tbProperty.attrib
if get_tb1['name']== str(testBedNumber):
for _tbPropertyVal in _tbProperty:
get_tb2 = _tbPropertyVal.attrib
if 'name' in get_tb2.keys():
propertyList.append(get_tb2['name'])
return propertyList
def GetIPNodeType(self, root, testBedNumber):
for tbNumber1 in root.findall('tbproperties'):
for tbNumber in tbNumber1:
ipv4support = tbNumber.find('ipv4support').text
ipv6support = tbNumber.find('ipv6support').text
lbSetup = tbNumber.find('lbSetup').text
name = tbNumber.get('name')
if name==str(testBedNumber):
return ipv4support, ipv6support, lbSetup
obj1, obj2 = raftGetTestBedProp().GetTestBedNumber()
ipv4support, ipv6support, lbSetup = raftGetTestBedProp().GetIPNodeType(obj1, obj2)
AlltestBedProperties = raftGetTestBedProp().GetTestBedProperties(obj1, obj2)
HostPropertyDict = {}
for testBedProperty in AlltestBedProperties:
try:
val1 = getattr(HostProperties, testBedProperty)
HostPropertyDict[testBedProperty] = val1
except:
logger.write("Error in the Configuration data. Please correct and then proceed with the testing", 'ERROR')
for indexVal in range(len(AlltestBedProperties)):
temp = AlltestBedProperties[indexVal]
globals()[temp] = HostPropertyDict[temp]
This variables.py file returns all variables defined in HostProperties.py file based on testbed number.
If i import this library like from variables import * in other libraries it gives me the required variables.
But the problem is here I have hardcoaded 748 so it works fine for me but i want to pass this testbed number information from pybot command and make it available for my Robot testcase as well as all the developed libraries.
Can you post Robot Framework code you use to call these Python files? I think you could use pybot -v testBed:748 and pass it as a parameter to __init__ your class. I am not sure without seeing how you start your Python variables.
A bit different way is to use environment variables:
#!/usr/bin/env python
import sys
import os
import optparse
import HostProperties
import xml.etree.ElementTree as ET
from robot.api import logger
testBed = os.environ['testbed']
tree = ET.parse('/home/p6mishra/mybkp/testLibs/TestBedProperties.xml')
Before starting pybot just define this environment parameter:
export testbed=748
pybot tests.txt
Here's the code I have. Basically I have the Shebang line in there because the psycopg2 wasn't working without it.
But now when I have this line in there it doesn't allow me to run the database, it just says "no module named 'flask'"
#!/usr/bin/python3.4
#
# Small script to show PostgreSQL and Pyscopg together
#
from flask import Flask, render_template
from flask import request
from flask import *
from datetime import datetime
from functools import wraps
import time
import csv
import psycopg2
app = Flask(__name__)
app.secret_key ='lukey'
def getConn():
connStr=("dbname='test' user='lukey' password='lukey'")
conn=psycopg2.connect(connStr)
return conn
#app.route('/')
def home():
return render_template(index.html)
#app.route('/displayStudent', methods =['GET'])
def displayStudent():
residence = request.args['residence']
try:
conn = None
conn = getConn()
cur = conn.cursor()
cur.execute('SET search_path to public')
cur.execute('SELECT stu_id,student.name,course.name,home_town FROM student,\
course WHERE course = course_id AND student.residence = %s',[residence])
rows = cur.fetchall()
if rows:
return render_template('stu.html', rows = rows, residence = residence)
else:
return render_template('index.html', msg1='no data found')
except Exception as e:
return render_template('index.html', msg1='No data found', error1 = e)
finally:
if conn:
conn.close()
##app.route('/addStudent, methods =['GET','POST']')
#def addStudent():
if __name__ == '__main__':
app.run(debug = True)
This is an environment problem, not a flask, postgres or shebang problem. A specific version of Python is being called, and it is not being given the correct path to its libraries.
Depending on what platform you are on, changing you shebang to #! /usr/bin/env python3 can fix the problem, but if not (very likely not, though using env is considered better/portable practice these days), then you may need to add your Python3 libs location manually in your code.
sys.path.append("/path/to/your/python/libs")
If you know where your Python libs are (or maybe flask is installed somewhere peculiar?) then you can add that to the path and imports following the line where you added to the path will include it in their search for modules.
I have a third-party module (cx_Oracle) that I'd like to import whose location is unknown from environment to environment. I am currently using pythons configparser so I thought it would be a neat trick to set the location of the module within the config parser, append that location to path, and then import the third-party module from there.
This worked all fine and dandy until I began to refactor my code and started to split out logic into their own class/methods:
class Database:
def __init__(self, config):
self.CONFIG=config
sys.path.append(self.CONFIG.cx_oracle_path)
from cx_Oracle import cx_Oracle
self.open()
def open(self):
self.CONNECTION = cx_Oracle.Connection(self.CONFIG.username,
self.CONFIG.password,
self.CONFIG.db_sid)
self.CURSOR = self.CONNECTION.cursor()
....
....
....
Of course, the open method does not know what to do because cx_Oracle was defined in init and so the open method cannot see it.
I can't picture the proper way to do this, so I'm assuming I am over thinking this. What should I do instead so that open (and all other methods within the Database class) can see the imported module?
Thank you.
If you only need to use cx_Oracle within that class, you can just set it as an attribute on this instance, for example:
class Database:
def __init__(self, config):
self.CONFIG=config
sys.path.append(self.CONFIG.cx_oracle_path)
from cx_Oracle import cx_Oracle
self.cx_Oracle = cx_Oracle
self.open()
def open(self):
self.CONNECTION = self.cx_Oracle.Connection(self.CONFIG.username,
self.CONFIG.password,
self.CONFIG.db_sid)
self.CURSOR = self.CONNECTION.cursor()
As a side note, if you are creating multiple Database instances this is an odd approach, since you would end up adding multiple identical entries to sys.path.
i'm trying to make a group of defs in one file so then i just can import them whenever i want to make a script in python
i have tried this:
def get_dblink( dbstring):
"""
Return a database cnx.
"""
global psycopg2
try
cnx = psycopg2.connect( dbstring)
except Exception, e:
print "Unable to connect to DB. Error [%s]" % ( e,)
exit( )
but i get this error: global name 'psycopg2' is not defined
in my main file script.py
i have:
import psycopg2, psycopg2.extras
from misc_defs import *
hostname = '192.168.10.36'
database = 'test'
username = 'test'
password = 'test'
dbstring = "host='%s' dbname='%s' user='%s' password='%s'" % ( hostname, database, username, password)
cnx = get_dblink( dbstring)
can anyone give me a hand?
You just need to import psycopg2 in your first snippet.
If you need to there's no problem to 'also' import it in the second snippet (Python makes sure the modules are only imported once). Trying to use globals for this is bad practice.
So: at the top of every module, import every module which is used within that particular module.
Also: note that from x import * (with wildcards) is generally frowned upon: it clutters your namespace and makes your code less explicit.