Related
I have a module that needs to update new variable values from the web, about once a week. I could place those variable values in a file & load those values on startup. Or, a simpler solution would be to simply auto-update the code.
Is this possible in Python?
Something like this...
def self_updating_module_template():
dynamic_var1 = {'dynamic var1'} # some kind of place holder tag
dynamic_var2 = {'dynamic var2'} # some kind of place holder tag
return
def self_updating_module():
dynamic_var1 = 'old data'
dynamic_var2 = 'old data'
return
def updater():
new_data_from_web = ''
new_dynamic_var1 = new_data_from_web # Makes API call. gets values.
new_dynamic_var2 = new_data_from_web
# loads self_updating_module_template
dynamic_var1 = new_dynamic_var1
dynamic_var2 = new_dynamic_var2
# replace module place holders with new values.
# overwrite self_updating_module.py.
return
I would recommend that you use configparser and a set of default values located in an ini-style file.
The ConfigParser class implements a basic configuration file parser
language which provides a structure similar to what you would find on
Microsoft Windows INI files. You can use this to write Python programs
which can be customized by end users easily.
Whenever the configuration values are updated from the web api endpoint, configparser also lets us write those back out to the configuration file. That said, be careful! The reason that most people recommend that configuration files be included at build/deploy time and not at run time is for security/stability. You have to lock down the endpoint that allows updates to your running configuration in production and have some way to verify any configuration value updates before they are retrieved by your application:
import configparser
filename = 'config.ini'
def load_config():
config = configparser.ConfigParser()
config.read(filename)
if 'WEB_DATA' not in config:
config['WEB_DATA'] = {'dynamic_var1': 'dynamic var1', # some kind of place holder tag
'dynamic_var2': 'dynamic var2'} # some kind of place holder tag
return config
def update_config(config):
new_data_from_web = ''
new_dynamic_var1 = new_data_from_web # Makes API call. gets values.
new_dynamic_var2 = new_data_from_web
config['WEB_DATA']['dynamic_var1'] = new_dynamic_var1
config['WEB_DATA']['dynamic_var2'] = new_dynamic_var2
def save_config(config):
with open(filename, 'w') as configfile:
config.write(configfile)
Example usage::
# Load the configuration
config = load_config()
# Get new data from the web
update_config(config)
# Save the newly updated configuration back to the file
save_config(config)
I have written a code where parameters are taken from a config file.
My first parameter in the config is for setting the debug level.
config = ConfigParser.RawConfigParser()
config.read('config.cfg')
log_level = config.get('Logger','log_level' )
There are other sections in the config which gives the server IP and password to scan each sections.
Main Code:
for section in config.sections():
components = dict() #start with empty dictionary for each section
env.user = config.get(section, 'server.user_name')
env.password = config.get(section, 'server.password')
host = config.get(section, 'server.ip')
From my config
[Logger]
#Possible values for logging are INFO, DEBUG and ERROR
log_level = DEBUG
[server1]
server.user_name = root
server.password = password
server.ip = 172.19.41.21
[server2]
server.user_name = root
server.password = password
server.ip = 172.19.41.21
Now my code says to check each section to retrieve the username and password. Since the first section doesn't contain these values, it's failing. How can i check each section for username and password and if its not there just go to the next section. I tried by checking for NONE and go to the next section. But that code is ugly and it's failing. Something like this:
if env.user=='':
next
Can someone help me to proceed further?
Add this code to the beginning of your for loop:
if not config.has_option(section, 'server.user_name'):
continue
Since only the first section doesn't contain these values you can use the iter function.
sections = iter(config.sections())
next(sections)
for section in sections:
# something(section)
Or as #tjohnson mentioned:
for section in config.sections()[1:]:
# something(section)
Another way to do is to just catch the exception.
for section in config.sections():
components = dict() #start with empty dictionary for each section
try:
env.user = config.get(section, 'server.user_name')
env.password = config.get(section, 'server.password')
host = config.get(section, 'server.ip')
except ConfigParser.NoOptionError as e:
continue # At least one required option is missing in the section, skip
The advantage is that if any option is missing, the section will be ignored.
However, if you need to be atomic (for example, if it is a problem to set
env.user if the section is finally ignored due to server.ip not present), you might
want something like this.
for section in config.sections():
components = dict() #start with empty dictionary for each section
try:
tmp_user = config.get(section, 'server.user_name')
tmp_password = config.get(section, 'server.password')
tmp_host = config.get(section, 'server.ip')
except ConfigParser.NoOptionError as e:
continue # At least one required option is missing in the section, skip
else:
env.user = tmp_user
env.password = tmp_password
host = tmp_host
In that case, maybe it is easier to use has_option 3 times.
I'm using Alembic with SQLAlchemy. With SQLAlchemy, I tend to follow a pattern where I don't store the connect string with the versioned code. Instead I have file secret.py that contains any confidential information. I throw this filename in my .gitignore so it doesn't end up on GitHub.
This pattern works fine, but now I'm getting into using Alembic for migrations. It appears that I cannot hide the connect string. Instead in alembic.ini, you place the connect string as a configuration parameter:
# the 'revision' command, regardless of autogenerate
# revision_environment = false
sqlalchemy.url = driver://user:pass#localhost/dbname
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembi
I fear I'm going to accidentally commit a file with username/password information for my database. I'd rather store this connect string in a single place and avoid the risk of accidentally committing it to version control.
What options do I have?
I had the very same problem yesterday and found a following solution to work.
I do the following in alembic/env.py:
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# this will overwrite the ini-file sqlalchemy.url path
# with the path given in the config of the main code
import config as ems_config
config.set_main_option('sqlalchemy.url', ems_config.config.get('sql', 'database'))
ems_config is an external module that holds my configuration data.
config.set_main_option(...) essentially overwrites the sqlalchemy.url key in the [alembic] section of the alembic.ini file. In my configuration I simply leave it black.
The simplest thing I could come up with to avoid commiting my user/pass was to a) add in interpolation strings to the alembic.ini file, and b) set these interpolation values in env.py
alembic.ini
sqlalchemy.url = postgresql://%(DB_USER)s:%(DB_PASS)s#35.197.196.146/nozzle-website
env.py
import os
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic import context
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# here we allow ourselves to pass interpolation vars to alembic.ini
# fron the host env
section = config.config_ini_section
config.set_section_option(section, "DB_USER", os.environ.get("DB_USER"))
config.set_section_option(section, "DB_PASS", os.environ.get("DB_PASS"))
...
Alembic documentation suggests using create_engine with the database URL (instead of modifying sqlalchemy.url in code).
Also you should modify run_migrations_offline to use the new URL. Allan Simon has an example on his blog, but in summary, modify env.py to:
Provide a shared function to get the URL somehow (here it comes from the command line):
def get_url():
url = context.get_x_argument(as_dictionary=True).get('url')
assert url, "Database URL must be specified on command line with -x url=<DB_URL>"
return url
Use the URL in offline mode:
def run_migrations_offline():
...
url = get_url()
context.configure(
url=url, target_metadata=target_metadata, literal_binds=True)
...
Use the URL in online mode by using create_engine instead of engine_from_config:
def run_migrations_online():
...
connectable = create_engine(get_url())
with connectable.connect() as connection:
...
So what appears to work is reimplementing engine creation in env.py, which is apparently a place for doing this kind of customizing Instead of using the sqlalchemy connect string in the ini:
engine = engine_from_config(
config.get_section(config.config_ini_section),
prefix='sqlalchemy.',
poolclass=pool.NullPool)
You can replace and specify your own engine configuration:
import store
engine = store.engine
Indeed the docs seems to imply this is ok:
sqlalchemy.url - A URL to connect to the database via SQLAlchemy. This key is in fact only referenced within the env.py file that is specific to the “generic” configuration; a file that can be customized by the developer. A multiple database configuration may respond to multiple keys here, or may reference other sections of the file.
I was looking for a while how to manage this for mutli-databases
Here is what I did. I have two databases : logs and ohlc
According to the doc,
I have setup the alembic like that
alembic init --template multidb
alembic.ini
databases = logs, ohlc
[logs]
sqlalchemy.url = postgresql://botcrypto:botcrypto#localhost/logs
[ohlc]
sqlalchemy.url = postgresql://botcrypto:botcrypto#localhost/ohlc
env.py
[...]
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
logger = logging.getLogger('alembic.env')
# overwrite alembic.ini db urls from the config file
settings_path = os.environ.get('SETTINGS')
if settings_path:
with open(settings_path) as fd:
settings = conf.load(fd, context=os.environ) # loads the config.yml
config.set_section_option("ohlc", "sqlalchemy.url", settings["databases"]["ohlc"])
config.set_section_option("logs", "sqlalchemy.url", settings["databases"]["logs"])
else:
logger.warning('Environment variable SETTINGS missing - use default alembic.ini configuration')
[...]
config.yml
databases:
logs: postgresql://botcrypto:botcrypto#127.0.0.1:5432/logs
ohlc: postgresql://botcrypto:botcrypto#127.0.0.1:5432/ohlc
usage
SETTINGS=config.yml alembic upgrade head
Hope it can helps !
In the case of MultiDB settings (the same for SingleDB), you can use config.set_section_option('section_name', 'variable_name', 'db_URL') to modify the values of the database URL in the alembic.ini file.
For example:
alembic.init
[engine1]
sqlalchemy.url =
[engine2]
sqlalchemy.url =
Then,
env.py
config = context.config
config.set_section_option('engine1', 'sqlalchemy.url', os.environ.get('URL_DB1'))
config.set_section_option('engine2', 'sqlalchemy.url', os.environ.get('URL_DB2'))
env.py:
from alembic.config import Config
alembic_cfg = Config()
alembic_cfg.set_main_option("sqlalchemy.url", getenv('PG_URI'))
https://alembic.sqlalchemy.org/en/latest/api/config.html
I was bumping into this problem as well since we're running our migrations from our local machines. My solution is to put environment sections in the alembic.ini which stores the database config (minus the credentials):
[local]
host = localhost
db = dbname
[test]
host = x.x.x.x
db = dbname
[prod]
host = x.x.x.x
db = dbname
Then I put the following in the env.py so the user can pick their environment and be prompted for the credentials:
from alembic import context
from getpass import getpass
...
envs = ['local', 'test', 'prod']
print('Warning: Do not commit your database credentials to source control!')
print(f'Available migration environments: {", ".join(envs)}')
env = input('Environment: ')
if env not in envs:
print(f'{env} is not a valid environment')
exit(0)
env_config = context.config.get_section(env)
host = env_config['host']
db = env_config['db']
username = input('Username: ')
password = getpass()
connection_string = f'postgresql://{username}:{password}#{host}/{db}'
context.config.set_main_option('sqlalchemy.url', connection_string)
You should store your credentials in a password manager that the whole team has access to, or whatever config/secret store you have available. Though, with this approach the password is exposed to your local clip board - an even better approach would be to have env.py directly connect to your config/secret store API and pull out the username/password directly but this adds a third party dependency.
Another solution is to create a template alembic.ini.dist file and to track it with your versionned code, while ignoring alembic.ini in your VCS.
Do not add any confidential information in alembic.ini.dist:
sqlalchemy.url = ...
When deploying your code to a platform, copy alembic.ini.dist to alembic.ini (this one won't be tracked by your VCS) and modify alembic.ini with the platform's credentials.
As Doug T. said you can edit env.py to provide URL from somewhere else than ini file. Instead of creating new engine you can pass an additional url argument to the engine_from_config function (kwargs are later merged to options taken from ini file). In that case you could e.g. store encrypted password in ini file and decrypt it in runtime by passphrase stored in ENV variable.
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix='sqlalchemy.',
poolclass=pool.NullPool,
url=some_decrypted_endpoint)
An option that worked for me was to use set_main_option and leave the sqlalchemy.url = blank in alembic.ini
from config import settings
config.set_main_option(
"sqlalchemy.url", settings.database_url.replace("postgres://", "postgresql+asyncpg://", 1))
sttings is a class in config file that I use to get variables in env file check this os.environ.get() does not return the Environment Value in windows? for more detail, another option is to use os.environ.get but make sure that you export the varibale to prevent errors like sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string
Based on the answer of TomDotTom I came up with this solution
Edit the env.py file with this
config = context.config
config.set_section_option("alembic", "sqlalchemy.url",
os.environ.get("DB_URL", config.get_section_option("alembic", "sqlalchemy.url"))) # type: ignore
This will override the sqlalchemy.url option from the alembic section with DB_URL environment variable if such environment variable exists, otherwise will use what else is in the alembic.ini file
Then I can run the migrations pointing to another database like this
DB_URL=driver://user:pass#host:port/dbname alembic upgrade head
And keep using alembic upgrade head during my development flow
I've tried all the answer here, but not working. Then I try to deal by myself, as below:
.ini file:
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = alembic
# template used to generate migration files
file_template = %%(rev)s_%%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d_%%(minute).2d_%%(second).2d
# timezone to use when rendering the date
# within the migration file as well as the filename.
# string value is passed to dateutil.tz.gettz()
# leave blank for localtime
# timezone =
# max length of characters to apply to the
# "slug" field
#truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; this defaults
# to alembic/versions. When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
databases = auth_engine
[auth_engine]
sqlalchemy.url = mysql+mysqldb://{}:{}#{}:{}/auth_db
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
.env file(it is in the root folder of my project):
DB_USER='root'
DB_PASS='12345678'
DB_HOST='127.0.0.1'
DB_PORT='3306'
env.py file:
from __future__ import with_statement
import os
import re
import sys
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic import context
DB_USER = os.getenv("DB_USER")
DB_PASS = os.getenv("DB_PASS")
DB_HOST = os.getenv("DB_HOST")
DB_PORT = os.getenv("DB_PORT")
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
# gather section names referring to different
# databases. These are named "engine1", "engine2"
# in the sample .ini file.
db_names = config.get_main_option('databases')
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
sys.path.append(os.path.join(os.path.dirname(__file__), "../../../"))
from db_models.auth_db import auth_db_base
target_metadata = {
'auth_engine': auth_db_base.auth_metadata
}
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
engines = {}
for name in re.split(r',\s*', db_names):
engines[name] = rec = {}
section = context.config.get_section(name)
url = section['sqlalchemy.url'].format(DB_USER, DB_PASS, DB_HOST, DB_PORT)
section['sqlalchemy.url'] = url
rec['url'] = url
# rec['url'] = context.config.get_section_option(name, "sqlalchemy.url")
for name, rec in engines.items():
print("Migrating database %s" % name)
file_ = "%s.sql" % name
print("Writing output to %s" % file_)
with open(file_, 'w') as buffer:
context.configure(url=rec['url'], output_buffer=buffer,
target_metadata=target_metadata.get(name),
compare_type=True,
compare_server_default=True
)
with context.begin_transaction():
context.run_migrations(engine_name=name)
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
engines = {}
for name in re.split(r',\s*', db_names):
engines[name] = rec = {}
section = context.config.get_section(name)
url = section['sqlalchemy.url'].format(DB_USER, DB_PASS, DB_HOST, DB_PORT)
section['sqlalchemy.url'] = url
rec['engine'] = engine_from_config(
section,
prefix='sqlalchemy.',
poolclass=pool.NullPool)
for name, rec in engines.items():
engine = rec['engine']
rec['connection'] = conn = engine.connect()
rec['transaction'] = conn.begin()
try:
for name, rec in engines.items():
print("Migrating database %s" % name)
context.configure(
connection=rec['connection'],
upgrade_token="%s_upgrades" % name,
downgrade_token="%s_downgrades" % name,
target_metadata=target_metadata.get(name),
compare_type=True,
compare_server_default=True
)
context.run_migrations(engine_name=name)
for rec in engines.values():
rec['transaction'].commit()
except:
for rec in engines.values():
rec['transaction'].rollback()
raise
finally:
for rec in engines.values():
rec['connection'].close()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
Wish can help someone else.
In env.py just add
config.set_main_option('sqlalchemy.url', os.environ['DB_URL'])
after
config = context.config
like
config = context.config
config.set_main_option('sqlalchemy.url', os.environ['DB_URL'])
and then execute like that:
DB_URL="mysql://atuamae:de4#127.0.0.1/db" \
alembic upgrade head
Given the following format (.properties or .ini):
propertyName1=propertyValue1
propertyName2=propertyValue2
...
propertyNameN=propertyValueN
For Java there is the Properties class that offers functionality to parse / interact with the above format.
Is there something similar in python's standard library (2.x) ?
If not, what other alternatives do I have ?
I was able to get this to work with ConfigParser, no one showed any examples on how to do this, so here is a simple python reader of a property file and example of the property file. Note that the extension is still .properties, but I had to add a section header similar to what you see in .ini files... a bit of a bastardization, but it works.
The python file: PythonPropertyReader.py
#!/usr/bin/python
import ConfigParser
config = ConfigParser.RawConfigParser()
config.read('ConfigFile.properties')
print config.get('DatabaseSection', 'database.dbname');
The property file: ConfigFile.properties
[DatabaseSection]
database.dbname=unitTest
database.user=root
database.password=
For more functionality, read: https://docs.python.org/2/library/configparser.html
For .ini files there is the configparser module that provides a format compatible with .ini files.
Anyway there's nothing available for parsing complete .properties files, when I have to do that I simply use jython (I'm talking about scripting).
I know that this is a very old question, but I need it just now and I decided to implement my own solution, a pure python solution, that covers most uses cases (not all):
def load_properties(filepath, sep='=', comment_char='#'):
"""
Read the file passed as parameter as a properties file.
"""
props = {}
with open(filepath, "rt") as f:
for line in f:
l = line.strip()
if l and not l.startswith(comment_char):
key_value = l.split(sep)
key = key_value[0].strip()
value = sep.join(key_value[1:]).strip().strip('"')
props[key] = value
return props
You can change the sep to ':' to parse files with format:
key : value
The code parses correctly lines like:
url = "http://my-host.com"
name = Paul = Pablo
# This comment line will be ignored
You'll get a dict with:
{"url": "http://my-host.com", "name": "Paul = Pablo" }
A java properties file is often valid python code as well. You could rename your myconfig.properties file to myconfig.py. Then just import your file, like this
import myconfig
and access the properties directly
print myconfig.propertyName1
if you don't have multi line properties and a very simple need, a few lines of code can solve it for you:
File t.properties:
a=b
c=d
e=f
Python code:
with open("t.properties") as f:
l = [line.split("=") for line in f.readlines()]
d = {key.strip(): value.strip() for key, value in l}
If you have an option of file formats I suggest using .ini and Python's ConfigParser as mentioned. If you need compatibility with Java .properties files I have written a library for it called jprops. We were using pyjavaproperties, but after encountering various limitations I ended up implementing my own. It has full support for the .properties format, including unicode support and better support for escape sequences. Jprops can also parse any file-like object while pyjavaproperties only works with real files on disk.
This is not exactly properties but Python does have a nice library for parsing configuration files. Also see this recipe: A python replacement for java.util.Properties.
i have used this, this library is very useful
from pyjavaproperties import Properties
p = Properties()
p.load(open('test.properties'))
p.list()
print(p)
print(p.items())
print(p['name3'])
p['name3'] = 'changed = value'
Here is link to my project: https://sourceforge.net/projects/pyproperties/. It is a library with methods for working with *.properties files for Python 3.x.
But it is not based on java.util.Properties
This is a one-to-one replacement of java.util.Propeties
From the doc:
def __parse(self, lines):
""" Parse a list of lines and create
an internal property dictionary """
# Every line in the file must consist of either a comment
# or a key-value pair. A key-value pair is a line consisting
# of a key which is a combination of non-white space characters
# The separator character between key-value pairs is a '=',
# ':' or a whitespace character not including the newline.
# If the '=' or ':' characters are found, in the line, even
# keys containing whitespace chars are allowed.
# A line with only a key according to the rules above is also
# fine. In such case, the value is considered as the empty string.
# In order to include characters '=' or ':' in a key or value,
# they have to be properly escaped using the backslash character.
# Some examples of valid key-value pairs:
#
# key value
# key=value
# key:value
# key value1,value2,value3
# key value1,value2,value3 \
# value4, value5
# key
# This key= this value
# key = value1 value2 value3
# Any line that starts with a '#' is considerered a comment
# and skipped. Also any trailing or preceding whitespaces
# are removed from the key/value.
# This is a line parser. It parses the
# contents like by line.
You can use a file-like object in ConfigParser.RawConfigParser.readfp defined here -> https://docs.python.org/2/library/configparser.html#ConfigParser.RawConfigParser.readfp
Define a class that overrides readline that adds a section name before the actual contents of your properties file.
I've packaged it into the class that returns a dict of all the properties defined.
import ConfigParser
class PropertiesReader(object):
def __init__(self, properties_file_name):
self.name = properties_file_name
self.main_section = 'main'
# Add dummy section on top
self.lines = [ '[%s]\n' % self.main_section ]
with open(properties_file_name) as f:
self.lines.extend(f.readlines())
# This makes sure that iterator in readfp stops
self.lines.append('')
def readline(self):
return self.lines.pop(0)
def read_properties(self):
config = ConfigParser.RawConfigParser()
# Without next line the property names will be lowercased
config.optionxform = str
config.readfp(self)
return dict(config.items(self.main_section))
if __name__ == '__main__':
print PropertiesReader('/path/to/file.properties').read_properties()
If you need to read all values from a section in properties file in a simple manner:
Your config.properties file layout :
[SECTION_NAME]
key1 = value1
key2 = value2
You code:
import configparser
config = configparser.RawConfigParser()
config.read('path_to_config.properties file')
details_dict = dict(config.items('SECTION_NAME'))
This will give you a dictionary where keys are same as in config file and their corresponding values.
details_dict is :
{'key1':'value1', 'key2':'value2'}
Now to get key1's value :
details_dict['key1']
Putting it all in a method which reads that section from config file only once(the first time the method is called during a program run).
def get_config_dict():
if not hasattr(get_config_dict, 'config_dict'):
get_config_dict.config_dict = dict(config.items('SECTION_NAME'))
return get_config_dict.config_dict
Now call the above function and get the required key's value :
config_details = get_config_dict()
key_1_value = config_details['key1']
-------------------------------------------------------------
Extending the approach mentioned above, reading section by section automatically and then accessing by section name followed by key name.
def get_config_section():
if not hasattr(get_config_section, 'section_dict'):
get_config_section.section_dict = dict()
for section in config.sections():
get_config_section.section_dict[section] =
dict(config.items(section))
return get_config_section.section_dict
To access:
config_dict = get_config_section()
port = config_dict['DB']['port']
(here 'DB' is a section name in config file
and 'port' is a key under section 'DB'.)
create a dictionary in your python module and store everything into it and access it, for example:
dict = {
'portalPath' : 'www.xyx.com',
'elementID': 'submit'}
Now to access it you can simply do:
submitButton = driver.find_element_by_id(dict['elementID'])
My Java ini files didn't have section headers and I wanted a dict as a result. So i simply injected an "[ini]" section and let the default config library do its job.
Take a version.ini fie of the eclipse IDE .metadata directory as an example:
#Mon Dec 20 07:35:29 CET 2021
org.eclipse.core.runtime=2
org.eclipse.platform=4.19.0.v20210303-1800
# 'injected' ini section
[ini]
#Mon Dec 20 07:35:29 CET 2021
org.eclipse.core.runtime=2
org.eclipse.platform=4.19.0.v20210303-1800
The result is converted to a dict:
from configparser import ConfigParser
#staticmethod
def readPropertyFile(path):
# https://stackoverflow.com/questions/3595363/properties-file-in-python-similar-to-java-properties
config = ConfigParser()
s_config= open(path, 'r').read()
s_config="[ini]\n%s" % s_config
# https://stackoverflow.com/a/36841741/1497139
config.read_string(s_config)
items=config.items('ini')
itemDict={}
for key,value in items:
itemDict[key]=value
return itemDict
This is what I'm doing in my project: I just create another .py file called properties.py which includes all common variables/properties I used in the project, and in any file need to refer to these variables, put
from properties import *(or anything you need)
Used this method to keep svn peace when I was changing dev locations frequently and some common variables were quite relative to local environment. Works fine for me but not sure this method would be suggested for formal dev environment etc.
import json
f=open('test.json')
x=json.load(f)
f.close()
print(x)
Contents of test.json:
{"host": "127.0.0.1", "user": "jms"}
I have created a python module that is almost similar to the Properties class of Java ( Actually it is like the PropertyPlaceholderConfigurer in spring which lets you use ${variable-reference} to refer to already defined property )
EDIT : You may install this package by running the command(currently tested for python 3).
pip install property
The project is hosted on GitHub
Example : ( Detailed documentation can be found here )
Let's say you have the following properties defined in my_file.properties file
foo = I am awesome
bar = ${chocolate}-bar
chocolate = fudge
Code to load the above properties
from properties.p import Property
prop = Property()
# Simply load it into a dictionary
dic_prop = prop.load_property_files('my_file.properties')
Below 2 lines of code shows how to use Python List Comprehension to load 'java style' property file.
split_properties=[line.split("=") for line in open('/<path_to_property_file>)]
properties={key: value for key,value in split_properties }
Please have a look at below post for details
https://ilearnonlinesite.wordpress.com/2017/07/24/reading-property-file-in-python-using-comprehension-and-generators/
you can use parameter "fromfile_prefix_chars" with argparse to read from config file as below---
temp.py
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('--a')
parser.add_argument('--b')
args = parser.parse_args()
print(args.a)
print(args.b)
config file
--a
hello
--b
hello dear
Run command
python temp.py "#config"
You could use - https://pypi.org/project/property/
eg - my_file.properties
foo = I am awesome
bar = ${chocolate}-bar
chocolate = fudge
long = a very long property that is described in the property file which takes up \
multiple lines can be defined by the escape character as it is done here
url=example.com/api?auth_token=xyz
user_dir=${HOME}/test
unresolved = ${HOME}/files/${id}/${bar}/
fname_template = /opt/myapp/{arch}/ext/{objid}.dat
Code
from properties.p import Property
## set use_env to evaluate properties from shell / os environment
prop = Property(use_env = True)
dic_prop = prop.load_property_files('my_file.properties')
## Read multiple files
## dic_prop = prop.load_property_files('file1', 'file2')
print(dic_prop)
# Output
# {'foo': 'I am awesome', 'bar': 'fudge-bar', 'chocolate': 'fudge',
# 'long': 'a very long property that is described in the property file which takes up multiple lines
# can be defined by the escape character as it is done here', 'url': 'example.com/api?auth_token=xyz',
# 'user_dir': '/home/user/test',
# 'unresolved': '/home/user/files/${id}/fudge-bar/',
# 'fname_template': '/opt/myapp/{arch}/ext/{objid}.dat'}
I did this using ConfigParser as follows. The code assumes that there is a file called config.prop in the same directory where BaseTest is placed:
config.prop
[CredentialSection]
app.name=MyAppName
BaseTest.py:
import unittest
import ConfigParser
class BaseTest(unittest.TestCase):
def setUp(self):
__SECTION = 'CredentialSection'
config = ConfigParser.ConfigParser()
config.readfp(open('config.prop'))
self.__app_name = config.get(__SECTION, 'app.name')
def test1(self):
print self.__app_name % This should print: MyAppName
This is what i had written to parse file and set it as env variables which skips comments and non key value lines added switches to specify
hg:d
-h or --help print usage summary
-c Specify char that identifies comment
-s Separator between key and value in prop file
and specify properties file that needs to be parsed eg : python
EnvParamSet.py -c # -s = env.properties
import pipes
import sys , getopt
import os.path
class Parsing :
def __init__(self , seprator , commentChar , propFile):
self.seprator = seprator
self.commentChar = commentChar
self.propFile = propFile
def parseProp(self):
prop = open(self.propFile,'rU')
for line in prop :
if line.startswith(self.commentChar)==False and line.find(self.seprator) != -1 :
keyValue = line.split(self.seprator)
key = keyValue[0].strip()
value = keyValue[1].strip()
print("export %s=%s" % (str (key),pipes.quote(str(value))))
class EnvParamSet:
def main (argv):
seprator = '='
comment = '#'
if len(argv) is 0:
print "Please Specify properties file to be parsed "
sys.exit()
propFile=argv[-1]
try :
opts, args = getopt.getopt(argv, "hs:c:f:", ["help", "seprator=","comment=", "file="])
except getopt.GetoptError,e:
print str(e)
print " possible arguments -s <key value sperator > -c < comment char > <file> \n Try -h or --help "
sys.exit(2)
if os.path.isfile(args[0])==False:
print "File doesnt exist "
sys.exit()
for opt , arg in opts :
if opt in ("-h" , "--help"):
print " hg:d \n -h or --help print usage summary \n -c Specify char that idetifes comment \n -s Sperator between key and value in prop file \n specify file "
sys.exit()
elif opt in ("-s" , "--seprator"):
seprator = arg
elif opt in ("-c" , "--comment"):
comment = arg
p = Parsing( seprator, comment , propFile)
p.parseProp()
if __name__ == "__main__":
main(sys.argv[1:])
Lightbend has released the Typesafe Config library, which parses properties files and also some JSON-based extensions. Lightbend's library is only for the JVM, but it seems to be widely adopted and there are now ports in many languages, including Python: https://github.com/chimpler/pyhocon
You can use the following function, which is the modified code of #mvallebr. It respects the properties file comments, ignores empty new lines, and allows retrieving a single key value.
def getProperties(propertiesFile ="/home/memin/.config/customMemin/conf.properties", key=''):
"""
Reads a .properties file and returns the key value pairs as dictionary.
if key value is specified, then it will return its value alone.
"""
with open(propertiesFile) as f:
l = [line.strip().split("=") for line in f.readlines() if not line.startswith('#') and line.strip()]
d = {key.strip(): value.strip() for key, value in l}
if key:
return d[key]
else:
return d
this works for me.
from pyjavaproperties import Properties
p = Properties()
p.load(open('test.properties'))
p.list()
print p
print p.items()
print p['name3']
I followed configparser approach and it worked quite well for me. Created one PropertyReader file and used config parser there to ready property to corresponding to each section.
**Used Python 2.7
Content of PropertyReader.py file:
#!/usr/bin/python
import ConfigParser
class PropertyReader:
def readProperty(self, strSection, strKey):
config = ConfigParser.RawConfigParser()
config.read('ConfigFile.properties')
strValue = config.get(strSection,strKey);
print "Value captured for "+strKey+" :"+strValue
return strValue
Content of read schema file:
from PropertyReader import *
class ReadSchema:
print PropertyReader().readProperty('source1_section','source_name1')
print PropertyReader().readProperty('source2_section','sn2_sc1_tb')
Content of .properties file:
[source1_section]
source_name1:module1
sn1_schema:schema1,schema2,schema3
sn1_sc1_tb:employee,department,location
sn1_sc2_tb:student,college,country
[source2_section]
source_name1:module2
sn2_schema:schema4,schema5,schema6
sn2_sc1_tb:employee,department,location
sn2_sc2_tb:student,college,country
You can try the python-dotenv library. This library reads key-value pairs from a .env (so not exactly a .properties file though) file and can set them as environment variables.
Here's a sample usage from the official documentation:
from dotenv import load_dotenv
load_dotenv() # take environment variables from .env.
# Code of your application, which uses environment variables (e.g. from `os.environ` or
# `os.getenv`) as if they came from the actual environment.
In Google App Engine, an entity has a Key. A key can be made from a path, in which case str(key) is an opaque hex string. Example:
from google.appengine.ext import db
foo = db.Key.from_path(u'foo', u'bar', _app=u'baz')
print foo
gives
agNiYXpyDAsSA2ZvbyIDYmFyDA
if you set up the right paths to run the code.
So, how can one take the hex string and get the path back? I thought the answer would be in Key or entity group docs, but I can't see it.
from google.appengine.ext import db
k = db.Key('agNiYXpyDAsSA2ZvbyIDYmFyDA')
_app = k.app()
path = []
while k is not None:
path.append(k.id_or_name())
path.append(k.kind())
k = k.parent()
path.reverse()
print 'app=%r, path=%r' % (_app, path)
when run in a Development Console, this outputs:
app=u'baz', path=[u'foo', u'bar']
as requested. A shorter alternative is to use the (unfortunately, I believe, undocumented) to_path method of Key instances:
k = db.Key('agNiYXpyDAsSA2ZvbyIDYmFyDA')
_app = k.app()
path = k.to_path()
print 'app=%r, path=%r' % (_app, path)
with the same results. But the first, longer version relies only on documented methods.
Once you have the Key object (which can be created by passing that opaque identifier to the constructor), use Key.to_path() to get the path of a Key as a list. For example:
from google.appengine.ext import db
opaque_id = 'agNiYXpyDAsSA2ZvbyIDYmFyDA'
path = db.Key(opaque_id).to_path()