Related
I'm using jsonnet to build json objects that will be used by Python code, calling jsonnet from Python using the bindings. I want to set up my directory structure so that the jsonnet files are in a subdirectory or subdirectories relative to where the Python code is run, something like:
foo.py
jsonnet/
jsonnet/bar.jsonnet
jsonnet/baz.libsonnet
Running foo.py should then be able to use _jsonnet.evaluate_snippet() on strings read from files in jsonnet/ that import other files from jsonnet/. What's the best way to do this?
The default importer uses paths relative to the file from which they are imported. In case of evaluate_snippet you need to pass the path manually. This way jsonnet knows where to look for imported files.
If your intention is to process the files you can use a custom importer. (Digression: jsonnet tries to avoid the need to preprocess the source files, so there is probably a better way or a missing feature in jsonnet.)
Below is the complete, working example on how to use custom importers in Python (adjusted to the directory structure provided):
import os
import unittest
import _jsonnet
# Returns content if worked, None if file not found, or throws an exception
def try_path(dir, rel):
if not rel:
raise RuntimeError('Got invalid filename (empty string).')
if rel[0] == '/':
full_path = rel
else:
full_path = dir + rel
if full_path[-1] == '/':
raise RuntimeError('Attempted to import a directory')
if not os.path.isfile(full_path):
return full_path, None
with open(full_path) as f:
return full_path, f.read()
def import_callback(dir, rel):
full_path, content = try_path(dir, rel)
if content:
return full_path, content
raise RuntimeError('File not found')
class JsonnetTests(unittest.TestCase):
def setUp(self):
self.input_filename = os.path.join(
"jsonnet",
"bar.jsonnet",
)
self.expected_str = '{\n "num": 42,\n "str": "The answer to life ..."\n}\n'
with open(self.input_filename, "r") as infile:
self.input_snippet = infile.read()
def test_evaluate_file(self):
json_str = _jsonnet.evaluate_file(
self.input_filename,
import_callback=import_callback,
)
self.assertEqual(json_str, self.expected_str)
def test_evaluate_snippet(self):
json_str = _jsonnet.evaluate_snippet(
"jsonnet/bar.jsonnet",
self.input_snippet,
import_callback=import_callback,
)
self.assertEqual(json_str, self.expected_str)
if __name__ == '__main__':
unittest.main()
Note: it's a modified version of an example from jsonnet repo.
I don't fully get why you would use evaluate_snippet() (maybe mask the actual filenames via loading them from python into strings + evaluate_snippet("blah", str) ? ), instead of evaluate_file() - in any case that structure should just work ok.
Example:
jsonnet_test.py:
import json:
import _jsonnet
jsonnet_file = "jsonnet/bar.jsonnet"
data = json.loads(_jsonnet.evaluate_file(jsonnet_file))
print("{str} => {num}".format(**data))
jsonnet/bar.jsonnet:
local baz = import "baz.libsonnet";
{
str: "The answer to life ...",
num: baz.mult(6, 7),
}
jsonnet/baz.libsonnet:
{
mult(a, b):: (
a * b
),
}
Output:
$ python jsonnet_test.py
The answer to life ... => 42
it's a little bit I'm out of python syntax and I have a problem in reading a .ini file with interpolated values.
this is my ini file:
[DEFAULT]
home=$HOME
test_home=$home
[test]
test_1=$test_home/foo.csv
test_2=$test_home/bar.csv
Those lines
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('config.ini')
print parser.get('test', 'test_1')
does output
$test_home/foo.csv
while I'm expecting
/Users/nkint/foo.csv
EDIT:
I supposed that the $ syntax was implicitly included in the so called string interpolation (referring to the manual):
On top of the core functionality, SafeConfigParser supports
interpolation. This means values can contain format strings which
refer to other values in the same section, or values in a special
DEFAULT section.
But I'm wrong. How to handle this case?
First of all according to the documentation you should use %(test_home)s to interpolate test_home. Moreover the key are case insensitive and you can't use both HOME and home keys. Finally you can use SafeConfigParser(os.environ) to take in account of you environment.
from ConfigParser import SafeConfigParser
import os
parser = SafeConfigParser(os.environ)
parser.read('config.ini')
Where config.ini is
[DEFAULT]
test_home=%(HOME)s
[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv
You can write custom interpolation in case of Python 3:
import configparser
import os
class EnvInterpolation(configparser.BasicInterpolation):
"""Interpolation which expands environment variables in values."""
def before_get(self, parser, section, option, value, defaults):
value = super().before_get(parser, section, option, value, defaults)
return os.path.expandvars(value)
cfg = """
[section1]
key = value
my_path = $PATH
"""
config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])
If you want to expand some environment variables, you can do so using os.path.expandvars before parsing a StringIO stream:
import ConfigParser
import os
import StringIO
with open('config.ini', 'r') as cfg_file:
cfg_txt = os.path.expandvars(cfg_file.read())
config = ConfigParser.ConfigParser()
config.readfp(StringIO.StringIO(cfg_txt))
the trick for proper variable substitution from environment is to use the ${} syntax for the environment variables:
[DEFAULT]
test_home=${HOME}
[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv
ConfigParser.get values are strings, even if you set values as integer or True. But ConfigParser has getint, getfloat and getboolean.
settings.ini
[default]
home=/home/user/app
tmp=%(home)s/tmp
log=%(home)s/log
sleep=10
debug=True
config reader
>>> from ConfigParser import SafeConfigParser
>>> parser = SafeConfigParser()
>>> parser.read('/home/user/app/settings.ini')
>>> parser.get('defaut', 'home')
'/home/user/app'
>>> parser.get('defaut', 'tmp')
'/home/user/app/tmp'
>>> parser.getint('defaut', 'sleep')
10
>>> parser.getboolean('defaut', 'debug')
True
Edit
Indeed you could get name values as environ var if you initialize SafeConfigParser with os.environ. Thanks for the Michele's answer.
Quite late, but maybe it can help someone else looking for the same answers that I had recently. Also, one of the comments was how to fetch Environment variables and values from other sections. Here is how I deal with both converting environment variables and multi-section tags when reading in from an INI file.
INI FILE:
[PKG]
# <VARIABLE_NAME>=<VAR/PATH>
PKG_TAG = Q1_RC1
[DELIVERY_DIRS]
# <DIR_VARIABLE>=<PATH>
NEW_DELIVERY_DIR=${DEL_PATH}\ProjectName_${PKG:PKG_TAG}_DELIVERY
Python Class that uses the ExtendedInterpolation so that you can use the ${PKG:PKG_TAG} type formatting. I add the ability to convert the windows environment vars when I read in INI to a string using the builtin os.path.expandvars() function such as ${DEL_PATH} above.
import os
from configparser import ConfigParser, ExtendedInterpolation
class ConfigParser(object):
def __init__(self):
"""
initialize the file parser with
ExtendedInterpolation to use ${Section:option} format
[Section]
option=variable
"""
self.config_parser = ConfigParser(interpolation=ExtendedInterpolation())
def read_ini_file(self, file='./config.ini'):
"""
Parses in the passed in INI file and converts any Windows environ vars.
:param file: INI file to parse
:return: void
"""
# Expands Windows environment variable paths
with open(file, 'r') as cfg_file:
cfg_txt = os.path.expandvars(cfg_file.read())
# Parses the expanded config string
self.config_parser.read_string(cfg_txt)
def get_config_items_by_section(self, section):
"""
Retrieves the configurations for a particular section
:param section: INI file section
:return: a list of name, value pairs for the options in the section
"""
return self.config_parser.items(section)
def get_config_val(self, section, option):
"""
Get an option value for the named section.
:param section: INI section
:param option: option tag for desired value
:return: Value of option tag
"""
return self.config_parser.get(section, option)
#staticmethod
def get_date():
"""
Sets up a date formatted string.
:return: Date string
"""
return datetime.now().strftime("%Y%b%d")
def prepend_date_to_var(self, sect, option):
"""
Function that allows the ability to prepend a
date to a section variable.
:param sect: INI section to look for variable
:param option: INI search variable under INI section
:return: Void - Date is prepended to variable string in INI
"""
if self.config_parser.get(sect, option):
var = self.config_parser.get(sect, option)
var_with_date = var + '_' + self.get_date()
self.config_parser.set(sect, option, var_with_date)
Based on #alex-markov answer (and code) and #srand9 comment, the following solution works with environment variables and cross-section references.
Note that the interpolation is now based on ExtendedInterpolation to allow cross-sections references and on before_read instead of before_get.
#!/usr/bin/env python3
import configparser
import os
class EnvInterpolation(configparser.ExtendedInterpolation):
"""Interpolation which expands environment variables in values."""
def before_read(self, parser, section, option, value):
value = super().before_read(parser, section, option, value)
return os.path.expandvars(value)
cfg = """
[paths]
foo : ${HOME}
[section1]
key = value
my_path = ${paths:foo}/path
"""
config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])
It seems in the last version 3.5.0, ConfigParser was not reading the env variables, so I end up providing a custom Interpolation based on the BasicInterpolation one.
class EnvInterpolation(BasicInterpolation):
"""Interpolation as implemented in the classic ConfigParser,
plus it checks if the variable is provided as an environment one in uppercase.
"""
def _interpolate_some(self, parser, option, accum, rest, section, map,
depth):
rawval = parser.get(section, option, raw=True, fallback=rest)
if depth > MAX_INTERPOLATION_DEPTH:
raise InterpolationDepthError(option, section, rawval)
while rest:
p = rest.find("%")
if p < 0:
accum.append(rest)
return
if p > 0:
accum.append(rest[:p])
rest = rest[p:]
# p is no longer used
c = rest[1:2]
if c == "%":
accum.append("%")
rest = rest[2:]
elif c == "(":
m = self._KEYCRE.match(rest)
if m is None:
raise InterpolationSyntaxError(option, section,
"bad interpolation variable reference %r" % rest)
var = parser.optionxform(m.group(1))
rest = rest[m.end():]
try:
v = os.environ.get(var.upper())
if v is None:
v = map[var]
except KeyError:
raise InterpolationMissingOptionError(option, section, rawval, var) from None
if "%" in v:
self._interpolate_some(parser, option, accum, v,
section, map, depth + 1)
else:
accum.append(v)
else:
raise InterpolationSyntaxError(
option, section,
"'%%' must be followed by '%%' or '(', "
"found: %r" % (rest,))
The difference between the BasicInterpolation and the EnvInterpolation is in:
v = os.environ.get(var.upper())
if v is None:
v = map[var]
where I'm trying to find the var in the enviornment before checking in the map.
Below is a simple solution that
Can use default value if no environment variable is provided
Overrides variables with environment variables (if found)
needs no custom interpolation implementation
Example:
my_config.ini
[DEFAULT]
HOST=http://www.example.com
CONTEXT=${HOST}/auth/
token_url=${CONTEXT}/oauth2/token
ConfigParser:
import os
import configparser
config = configparser.ConfigParser(interpolation=configparser.ExtendedInterpolation())
ini_file = os.path.join(os.path.dirname(__file__), 'my_config.ini')
# replace variables with environment variables(if exists) before loading ini file
with open(ini_file, 'r') as cfg_file:
cfg_env_txt = os.path.expandvars(cfg_file.read())
config.read_string(cfg_env_txt)
print(config['DEFAULT']['token_url'])
Output:
If no environtment variable $HOST or $CONTEXT is present this config will take the default value
user can override the default value by creating $HOST, $CONTEXT environment variable
works well with docker container
In C++, I can print debug output like this:
printf(
"FILE: %s, FUNC: %s, LINE: %d, LOG: %s\n",
__FILE__,
__FUNCTION__,
__LINE__,
logmessage
);
How can I do something similar in Python?
There is a module named inspect which provides these information.
Example usage:
import inspect
def PrintFrame():
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename) # __FILE__ -> Test.py
print(info.function) # __FUNCTION__ -> Main
print(info.lineno) # __LINE__ -> 13
def Main():
PrintFrame() # for this line
Main()
However, please remember that there is an easier way to obtain the name of the currently executing file:
print(__file__)
For example
import inspect
frame = inspect.currentframe()
# __FILE__
fileName = frame.f_code.co_filename
# __LINE__
fileNo = frame.f_lineno
There's more here http://docs.python.org/library/inspect.html
Building on geowar's answer:
class __LINE__(object):
import sys
def __repr__(self):
try:
raise Exception
except:
return str(sys.exc_info()[2].tb_frame.f_back.f_lineno)
__LINE__ = __LINE__()
If you normally want to use __LINE__ in e.g. print (or any other time an implicit str() or repr() is taken), the above will allow you to omit the ()s.
(Obvious extension to add a __call__ left as an exercise to the reader.)
You can refer my answer:
https://stackoverflow.com/a/45973480/1591700
import sys
print sys._getframe().f_lineno
You can also make lambda function
I was also interested in a __LINE__ command in python.
My starting point was https://stackoverflow.com/a/6811020 and I extended it with a metaclass object. With this modification it has the same behavior like in C++.
import inspect
class Meta(type):
def __repr__(self):
# Inspiration: https://stackoverflow.com/a/6811020
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
# print(info.filename) # __FILE__ -> Test.py
# print(info.function) # __FUNCTION__ -> Main
# print(info.lineno) # __LINE__ -> 13
return str(info.lineno)
class __LINE__(metaclass=Meta):
pass
print(__LINE__) # print for example 18
wow, 7 year old question :)
Anyway, taking Tugrul's answer, and writing it as a debug type method, it can look something like:
def debug(message):
import sys
import inspect
callerframerecord = inspect.stack()[1]
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename, 'func=%s' % info.function, 'line=%s:' % info.lineno, message)
def somefunc():
debug('inside some func')
debug('this')
debug('is a')
debug('test message')
somefunc()
Output:
/tmp/test2.py func=<module> line=12: this
/tmp/test2.py func=<module> line=13: is a
/tmp/test2.py func=<module> line=14: test message
/tmp/test2.py func=somefunc line=10: inside some func
import inspect
.
.
.
def __LINE__():
try:
raise Exception
except:
return sys.exc_info()[2].tb_frame.f_back.f_lineno
def __FILE__():
return inspect.currentframe().f_code.co_filename
.
.
.
print "file: '%s', line: %d" % (__FILE__(), __LINE__())
Here is a tool to answer this old yet new question!
I recommend using icecream!
Do you ever use print() or log() to debug your code? Of course, you
do. IceCream, or ic for short, makes print debugging a little sweeter.
ic() is like print(), but better:
It prints both expressions/variable names and their values.
It's 40% faster to type.
Data structures are pretty printed.
Output is syntax highlighted.
It optionally includes program context: filename, line number, and parent function.
For example, I created a module icecream_test.py, and put the following code inside it.
from icecream import ic
ic.configureOutput(includeContext=True)
def foo(i):
return i + 333
ic(foo(123))
Prints
ic| icecream_test.py:6 in <module>- foo(123): 456
To get the line number in Python without importing the whole sys module...
First import the _getframe submodule:
from sys import _getframe
Then call the _getframe function and use its' f_lineno property whenever you want to know the line number:
print(_getframe().f_lineno) # prints the line number
From the interpreter:
>>> from sys import _getframe
... _getframe().f_lineno # 2
Word of caution from the official Python Docs:
CPython implementation detail: This function should be used for internal and specialized purposes only. It is not guaranteed to exist in all implementations of Python.
In other words: Only use this code for personal testing / debugging reasons.
See the Official Python Documentation on sys._getframe for more information on the sys module, and the _getframe() function / submodule.
Based on Mohammad Shahid's answer (above).
Given a string with a module name, how do you import everything in the module as if you had called:
from module import *
i.e. given string S="module", how does one get the equivalent of the following:
__import__(S, fromlist="*")
This doesn't seem to perform as expected (as it doesn't import anything).
Please reconsider. The only thing worse than import * is magic import *.
If you really want to:
m = __import__ (S)
try:
attrlist = m.__all__
except AttributeError:
attrlist = dir (m)
for attr in attrlist:
globals()[attr] = getattr (m, attr)
Here's my solution for dynamic naming of local settings files for Django. Note the addition below of a check to not include attributes containing '__' from the imported file. The __name__ global was being overwritten with the module name of the local settings file, which caused setup_environ(), used in manage.py, to have problems.
try:
import socket
HOSTNAME = socket.gethostname().replace('.','_')
# See http://docs.python.org/library/functions.html#__import__
m = __import__(name="settings_%s" % HOSTNAME, globals=globals(), locals=locals(), fromlist="*")
try:
attrlist = m.__all__
except AttributeError:
attrlist = dir(m)
for attr in [a for a in attrlist if '__' not in a]:
globals()[attr] = getattr(m, attr)
except ImportError, e:
sys.stderr.write('Unable to read settings_%s.py\n' % HOSTNAME)
sys.exit(1)
The underlying problem is that I am developing some Django, but on more than one host (with colleagues), all with different settings. I was hoping to do something like this in the project/settings.py file:
from platform import node
settings_files = { 'BMH.lan': 'settings_bmh.py", ... }
__import__( settings_files[ node() ] )
It seemed a simple solution (thus elegant), but I would agree that it has a smell to it and the simplicity goes out the loop when you have to use logic like what John Millikin posted (thanks). Here's essentially the solution I went with:
from platform import node
from settings_global import *
n = node()
if n == 'BMH.lan':
from settings_bmh import *
# add your own, here...
else:
raise Exception("No host settings for '%s'. See settings.py." % node())
Which works fine for our purposes.
It appears that you can also use dict.update() on module's dictionaries in your case:
config = [__import__(name) for name in names_list]
options = {}
for conf in config:
options.update(conf.__dict__)
Update: I think there's a short "functional" version of it:
options = reduce(dict.update, map(__import__, names_list))
I didn't find a good way to do it so I took a simpler but ugly way from http://www.djangosnippets.org/snippets/600/
try:
import socket
hostname = socket.gethostname().replace('.','_')
exec "from host_settings.%s import *" % hostname
except ImportError, e:
raise e
Given a string with a module name, how do you import everything in the module as if you had called:
from module import *
i.e. given string S="module", how does one get the equivalent of the following:
__import__(S, fromlist="*")
This doesn't seem to perform as expected (as it doesn't import anything).
Please reconsider. The only thing worse than import * is magic import *.
If you really want to:
m = __import__ (S)
try:
attrlist = m.__all__
except AttributeError:
attrlist = dir (m)
for attr in attrlist:
globals()[attr] = getattr (m, attr)
Here's my solution for dynamic naming of local settings files for Django. Note the addition below of a check to not include attributes containing '__' from the imported file. The __name__ global was being overwritten with the module name of the local settings file, which caused setup_environ(), used in manage.py, to have problems.
try:
import socket
HOSTNAME = socket.gethostname().replace('.','_')
# See http://docs.python.org/library/functions.html#__import__
m = __import__(name="settings_%s" % HOSTNAME, globals=globals(), locals=locals(), fromlist="*")
try:
attrlist = m.__all__
except AttributeError:
attrlist = dir(m)
for attr in [a for a in attrlist if '__' not in a]:
globals()[attr] = getattr(m, attr)
except ImportError, e:
sys.stderr.write('Unable to read settings_%s.py\n' % HOSTNAME)
sys.exit(1)
The underlying problem is that I am developing some Django, but on more than one host (with colleagues), all with different settings. I was hoping to do something like this in the project/settings.py file:
from platform import node
settings_files = { 'BMH.lan': 'settings_bmh.py", ... }
__import__( settings_files[ node() ] )
It seemed a simple solution (thus elegant), but I would agree that it has a smell to it and the simplicity goes out the loop when you have to use logic like what John Millikin posted (thanks). Here's essentially the solution I went with:
from platform import node
from settings_global import *
n = node()
if n == 'BMH.lan':
from settings_bmh import *
# add your own, here...
else:
raise Exception("No host settings for '%s'. See settings.py." % node())
Which works fine for our purposes.
It appears that you can also use dict.update() on module's dictionaries in your case:
config = [__import__(name) for name in names_list]
options = {}
for conf in config:
options.update(conf.__dict__)
Update: I think there's a short "functional" version of it:
options = reduce(dict.update, map(__import__, names_list))
I didn't find a good way to do it so I took a simpler but ugly way from http://www.djangosnippets.org/snippets/600/
try:
import socket
hostname = socket.gethostname().replace('.','_')
exec "from host_settings.%s import *" % hostname
except ImportError, e:
raise e