I print out (m x n) table of values for debugging, however, I do not want the debug messages to be printed out in non-debugging mode. In C, it can be done with "#ifdef _DEBUG" in code and define _DEBUG in preprocessor definition. May I know what is equivalent way in Python?
Python has module called "logging"
See this question:
Using print statements only to debug
Or the basic tutorial:
http://docs.python.org/2/howto/logging.html
You could define a global variable someplace, if that's what you want. However, probably the cleaner and more standard way is to read a config file (easy because you can write a config file in plain Python) and define DEBUG in there. So you've got a config file that looks like this:
# program.cfg
# Other comments
# And maybe other configuration settings
DEBUG = True # Or False
And then in your code, you can either import your config file (if it's in a directory on the Python path and has a Python extension), or else you can execfile it.
cfg = {}
execfile('program.cfg', cfg) # Execute the config file in the new "cfg" namespace.
print cfg.get('DEBUG') # Access configuration settings like this.
try this:
import settings
if settings.DEBUG:
print testval
This prints testval if, and only if, DEBUG=True in settings.py
Related
I am looking for a solution to loading an envvars.sh file within a django application. I would prefer to do this within the settings module and keep a file outside of it. Here is what I currently have:
SITE_ROOT = os.path.dirname(os.path.realpath(__file__))
def source_envvars(fname='envvars.sh'):
"""Equivalent of `$ source {fname}` with lines in the form of `export key=value`"""
envvars = os.path.join(SITE_ROOT, fname)
if not os.path.exists(envvars): return
with open(fname, 'r') as f:
for line in f:
if line.startswith('export '):
line = line.strip()
key = line.split('=')[0].split()[1]
value = line.split('=')[1].strip('"\'')
os.environ[key] = value
if not os.environ.get('DB_HOST'): source_envvars()
# rest of settings.py file...
Are there any downsides of using this approach? I know this doesn't support complicated bash-type exports, but all I have are basic exports of the form:
export DB_HOST=rds.amazon.com/12345
Does the above approach properly instantiate all the variables, or does it seem to be missing something or insecure in some way or other?
Your implementation can only handle fixed formats. A shell script can potentially have conditional statements and loops, etc., none of which can be handled properly without actually sourcing it through a shell.
A more robust method is therefore to use subprocess.check_output to run a shell with . to source envvars.sh and the env command to output the environment variables, the output of which can be then used to create a generator expression to update os.environ with:
os.environ.update(
line.rstrip().split('=', 1)
for line in subprocess.check_output(['sh', '-c', '. envvars.sh; env']).splitlines()
)
Edit: My first attempt at asking this might be a bit unfocused/poorly worded here's a better explanation of what I'm trying to do:
I'm trying to modify the default behavior of the print function for the entire environment python is running in without having to modify each file that's being run.
I'm attempting to decorate the print function (I know there are many ways to do this such as overriding it but that's not really the question I'm asking) so I can have it print out some debugging information and force it to always flush. I did that like so:
def modify_print(func):
# I made this so that output always gets flushed as it won't by default
# within the environment I'm using, I also wanted it to print out some
# debugging information, doesn't really matter much in the context of this
# question
def modified_print(*args,**kwargs):
return func(f"some debug prefix: ",flush=True,*args,**kwargs)
return modified_print
print = modify_print(print)
print("Hello world") # Prints "some debug prefix Hello World"
However what I'm trying to do is modify this behavior throughout my entire application. I know I can manually decorate/override/import the print function in each file however I'm wondering if there is some way I can globally configure my python environment to decorate this function everywhere. The only way I can think to do this would be to edit the python source code and build the modified version.
EDIT:
Here's the behavior I wanted implemented, thank you Match for your help.
It prints out the line number and filename everywhere you call a print function within your python environment. This means you don't have to import or override anything manually in all of your files.
https://gist.github.com/MichaelScript/444cbe5b74dce2c01a151d60b714ac3a
import site
import os
import pathlib
# Big thanks to Match on StackOverflow for helping me with this
# see https://stackoverflow.com/a/48713998/5614280
# This is some cool hackery to overwrite the default functionality of
# the builtin print function within your entire python environment
# to display the file name and the line number as well as always flush
# the output. It works by creating a custom user script and placing it
# within the user's sitepackages file and then overwriting the builtin.
# You can disable this behavior by running python with the '-s' flag.
# We could probably swap this out by reading the text from a python file
# which would make it easier to maintain larger modifications to builtins
# or a set of files to make this more portable or to modify the behavior
# of more builtins for debugging purposes.
customize_script = """
from inspect import getframeinfo,stack
def debug_printer(func):
# I made this so that output always gets flushed as it won't by default
# within the environment I'm running it in. Also it will print the
# file name and line number of where the print occurs
def debug_print(*args,**kwargs):
frame = getframeinfo(stack()[1][0])
return func(f"{frame.filename} : {frame.lineno} ", flush=True,*args,**kwargs)
return debug_print
__builtins__['print'] = debug_printer(print)
"""
# Creating the user site dir if it doesn't already exist and writing our
# custom behavior modifications
pathlib.Path(site.USER_SITE).mkdir(parents=True, exist_ok=True)
custom_file = os.path.join(site.USER_SITE,"usercustomize.py")
with open(custom_file,'w+') as f:
f.write(customize_script)
You can use usercustomize script from the site module to achieve something like this.
First, find out where your user site-packages directory is:
python3 -c "import site; print(site.USER_SITE)"
/home/foo/.local/lib/python3.6/site-packages
Next, in that directory, create a script called usercustomize.py - this script will now be run first whenever python is run.
One* way to replace print is to override the __builtins__ dict and replace it with a new method - something like:
from functools import partial
old_print = __builtins__['print']
__builtins__['print'] = partial(old_print, "Debug prefix: ", flush=True)
Drop this into the usercustomize.py script and you should see all python scripts from then on being overridden. You can temporarily disable calling this script by calling python with the -s flag.
*(Not sure if this is the correct way of doing this - there may be a better way - but the main point is that you can use usercustomize to deliver whatever method you choose).
There's no real reason to define a decorator here, because you are only intending to apply it to a single, predetermined function. Just define your modified print function directly, wrapping it around __builtins__.print to avoid recursion.
def print(*args, **kwargs):
__builtins.__print(f"some debug prefix: ", flush=True, *args, **kwargs)
print("Hello world") # Prints "some debug prefix Hello World"
You can use functools.partial to simplify this.
import functools
print = functools.partial(__builtins.__print, f"some debug prefix: ", flush=True)
Some background (not mandatory, but might be nice to know): I am writing a Python command-line module which is a wrapper around latexdiff. It basically replaces all \cite{ref1, ref2, ...} commands in LaTeX files with written-out and properly formatted references before passing the files to latexdiff, so that latexdiff will properly mark changes to references in the text (otherwise, it treats the whole \cite{...} command as a single "word"). All the code is currently in a single file which can be run with python -m latexdiff-cite, and I have not yet decided how to package or distribute it. To make the script useful for anybody else, the citation formatting needs to be configurable. I have implemented an optional command-line argument -c CONFIGFILE to allow the user to point to their own JSON config file (a default file resides in the module folder and is loaded if the argument is not used).
Current implementation: My single-file command-line Python module currently parses command-line arguments in if __name__ == '__main__', and loads the config file (specified by the user in -c CONFIGFILE) here before running the main function of the program. The config variable is thus available in the entire module and all is well. However, I'm considering publishing to PyPI by following this guide which seems to require me to put the command-line parsing in a main() function, which means the config variable will not be available to the other functions unless passed down as arguments to where it's needed. This "passing down by arguments" method seems a little cluttered to me.
Question: Is there a more pythonic way to set some configuration globals in a module or otherwise accomplish what I'm trying to? (I don't want to rely on 3rd party modules.) Am I perhaps completely off the tracks in some fundamental way?
One way to do it is to have the configurations defined in a class or a simple dict:
class Config(object):
setting1 = "default_value"
setting2 = "default_value"
#staticmethod
def load_config(json_file):
""" load settings from config file """
with open(json_file) as f:
config = json.load(f)
for k, v in config.iteritems():
setattr(Config, k, v)
Then your application can access the settings via this class: Config.setting1 ...
In python ,There is a reload method to reload an imported module , but is there a method to reload the currently running script without restarting it, this would be very helpful in debugging a script and changing the code live while the script is running. In visual basic I remember a similar functionality was called "apply code changes", but I need a similar functionality as a function call like "refresh()" which will apply the code changes instantly.
This would work smoothly when an independent function in the script is modified and we need to immediately apply the code change without restarting the script.
Inshort will:
reload(self)
work?
reload(self) will not work, because reload() works on modules, not live instances of classes. What you want is some logic external to your main application, which checks whether it has to be reloaded. You have to think about what is needed to re-create your application state upon reload.
Some hints in this direction:
Guido van Rossum wrote once this: xreload.py; it does a bit more than reload() You would need a loop, which checks for changes every x seconds and applies this.
Also have a look at livecoding which basically does this. EDIT: I mistook this project for something else (which I didn't find now), sorry.
perhaps this SO question will help you
Perhaps you mean something like this:
import pdb
import importlib
from os.path import basename
def function():
print("hello world")
if __name__ == "__main__":
# import this module, but not as __main__ so everything other than this
# if statement is executed
mainmodname = basename(__file__)[:-3]
module = importlib.import_module(mainmodname)
while True:
# reload the module to check for changes
importlib.reload(module)
# update the globals of __main__ with the any new or changed
# functions or classes in the reloaded module
globals().update(vars(module))
function()
pdb.set_trace()
Run the code, then change the contents of function and enter c at the prompt to run the next iteration of the loop.
test.py
class Test(object):
def getTest(self):
return 'test1'
testReload.py
from test import Test
t = Test()
print t.getTest()
# change return value (test.py)
import importlib
module = importlib.import_module(Test.__module__)
reload(module)
from test import Test
t = Test()
print t.getTest()
Intro
reload is for imported modules. Documentation for reload advises against reloading __main__.
Reloading sys, __main__, builtins and other key modules is not recommended.
To achieve similar behavior on your script you will need to re-execute the script. This will - for normal scripts - also reset the the global state. I propose a solution.
NOTE
The solution I propose is very powerful and should only be used for code you trust. Automatically executing code from unknown sources can lead to a world of pain. Python is not a good environment for soapboxing unsafe code.
Solution
To programmatically execute a python script from another script you only need the built-in functions open, compile and exec.
Here is an example function that will re-execute the script it is in:
def refresh():
with open(__file__) as fo:
source_code = fo.read()
byte_code = compile(source_code, __file__, "exec")
exec(byte_code)
The above will in normal circumstances also re-set any global variables you might have. If you wish to keep these variables you should check weather or not those variables have already been set. This can be done with a try-except block covering NameError exceptions. But that can be tedious so I propose using a flag variable.
Here is an example using the flag variable:
in_main = __name__ == "__main__"
first_run = "flag" not in globals()
if in_main and first_run:
flag = True
None of these answers did the job properly for me, so I put together something very messy and very non-pythonic to do the job myself. Even after running it for several weeks, I am finding small issues and fixing them. One issue will be if your PWD/CWD changes.
Warning this is very ugly code. Perhaps someone will make it pretty, but it does work.
Not only does it create a refresh() function that properly reloads your script in a manner such that any Exceptions will properly display, but it creates refresh_<scriptname> functions for previously loaded scripts just-in-case you need to reload those.
Next I would probably add a require portion, so that scripts can reload other scripts -- but I'm not trying to make node.js here.
First, the "one-liner" that you need to insert in any script you want to refresh.
with open(os.path.dirname(__file__) + os.sep + 'refresh.py', 'r') as f: \
exec(compile(f.read().replace('__BASE__', \
os.path.basename(__file__).replace('.py', '')).replace('__FILE__', \
__file__), __file__, 'exec'))
And second, the actual refresh function which should be saved to refresh.py in the same directory. (See, room for improvement already).
def refresh(filepath = __file__, _globals = None, _locals = None):
print("Reading {}...".format(filepath))
if _globals is None:
_globals = globals()
_globals.update({
"__file__": filepath,
"__name__": "__main__",
})
with open(filepath, 'rb') as file:
exec(compile(file.read(), filepath, 'exec'), _globals, _locals)
def refresh___BASE__():
refresh("__FILE__")
Tested with Python 2.7 and 3.
Take a look at reload. You just need to install the plugin an use reload ./myscript.py, voilà
If you are running in an interactive session you could use ipython autoreload
autoreload reloads modules automatically before entering the execution of code typed at the IPython prompt.
Of course this also works on module level, so you would do something like:
>>>import myscript
>>>myscript.main()
*do some changes in myscript.py*
>>>myscript.main() #is now changed
I am working on a Pylons app that runs on top of Apache with mod_wsgi. I would like to send logging messages that my app generates to files in my app's directory, instead of to Apache's logs. Further, I would like to specify the location of logfiles via a relative path so that it'll be easier to deploy my app on other people's servers. Right now I can log to files, but only via a fragile absolute path.
Here is the relevant part of my development.ini file:
# Logging configuration
[loggers]
keys = root, routes, myapp, sqlalchemy, debugging-logger
[handlers]
keys = console, debugging-logger-file
[formatters]
keys = generic
[logger_debugging-logger]
level = DEBUG
handlers = debugging-logger-file
qualname = myapp.controllers.logging-test-controller.debugging-logger
[handler_debugging-logger-file]
class = FileHandler
args = ('/var/pylons/myapp/logs/myapp-debugging-errors.log', 'a')
level = DEBUG
formatter = generic
Although the .ini helpfully advises using %(here)s to refer to the current path, using %(here)s in the "args = ('foo')" line of the error handler does not behave the way that I expect it to. The syntax of this ini file is documented on the Paste Deploy site, but does not specify how %(here)s can be used in relation to quoted strings.
What syntax should I use in the "args = ('foo')" line to specify the current path?
The problem is that Paste Deploy creates one ConfigParser object to store the 'here' tag in it's set of defaults, and logging.config.fileConfig() is never passed that set of defaults. Therefore, when fileConfig() reads the .ini file, it doesn't have access to the 'here' tag, and the ConfigParser's interpolation can't find it.
You could do something like this:
[DEFAULT]
my_log_dir = '/var/pylons/myapp/logs'
...
[handler_debugging-logger-file]
args = (%(my_log_dir)s + '/myapp-debugging-errors.log', 'a')
Not exactly what you're looking for, but a tiny bit more configurable.
Another possibility is:
args = (os.getcwd() + '/myapp-debugging-errors.log', 'a')
(This works because 'os' is a valid variable in the logging module's namespace when it calls eval() on the args value. But this is an implementation detail of the logging package that may not be reliable long term.) But this most likely won't give you what you want-- it will most likely use the Apache process's working directory.
You could even set an environment variable outside the program, and use it like:
args = (os.environ['MY_LOG_DIR'] + '/myapp-debugging-errors.log', 'a')
And yet another possibility is overriding the behavior of some of the functions or class methods in the logging module or paste package.
Hope those give you some ideas.
Configuration files for Paste Deploy allow a 'here' tag to indicate directory where configuration file is. You can then work relative to that.