Is there a way to enable/disable logging quickly using a global switch? I am using following code to setup logging
import logging
logging.basicConfig(filename='ordersLogs.log', level=logging.DEBUG)
And I am logging using this in many files.
I know we can use disable function on logging and it will disable logging for that level but to do that I need to make changes to all those files. So is there a better way of doing it?
logging.disable is probably what you are looking for. It provides an application-level setting (applies to all the loggers you have defined), with an optional level argument that you can use to suppress logging up to a certain level (typically, you will want to get rid of nuisance logging, but keep the higher criticality lines).
I'm looking for recommendations on a python logging framework python within a microservice. There's the built in logging system provided by python, there's structlogger. Currently I use structlogger with an ELK stack with filebeat instead of logstash. Please let me know what you would recommend and why? My usual criterial is popularity on stackoverflow (I'm not kidding), as it makes it a lot easier to get over technical issues or bugs.
Use the builtin logging module.
It does pretty much anything you need. structlogger isn't really a different framework and more of a default configuration for the builtin logging module. Also if you need something other than just logging to files or stdout the builtin module has a lot of handlers, and there exist a lot of third party handlers that work with the builtin module. (e.g. graylog)
I'm developing a reusable Python module (for Python 2.7 if it matters). I am wondering what the best practices are with regard to logging for others who wish to include my module in a larger framework which has its own logging approach.
Is there a standard way to set up logging within my module to use whatever loggers an external calling program has defined?
Here is a great blog post that outlines some best practices, which I have tried to adopt as my own: http://eric.themoritzfamily.com/learning-python-logging.html
He goes through all the details and rationale, but essentially it comes down to a couple simple principles.
Use getLogger based on your module's __name__ and start logging:
import logging
logger = logging.getLogger(__name__)
logger.info( ... )
logger.debug( ... )
Don't define handlers or configure the appropriate log-level in your module, because they may interfere with the "external calling program" you have in mind. You (or your users) can set up the handlers and the desired level of logging detail for all the sub-modules as needed, in the main application code.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I've noticed a few Python packages that use config files written in Python. Apart from the obvious privilege escalation, what are the pros and cons of this approach?
Is there much of a precedence for this? Are there any guides as to the best way to implement this?
Just to clarify: In my particular use case, this will only be used by programmers or people who know what they're doing. It's not a config file in a piece of software that will be distributed to end users.
The best example I can think of for this is the django settings.py file, but I'm sure there are tons of other examples for using a Python file for configuration.
There are a couple of key advantages for using Python as config file over other solutions, for example:
There is no need to parse the file: Since the file is already Python, you don't have to write or import a parser to extract the key value pairs from the file.
Configuration settings can be more than just key/values: While it would be folly to have settings define their own classes, you can use them to define tuples, lists or dictionaries of settings allowing for more options and configuration than other options. This is especially true with django, where the settings file has to accommodate for all manner of plug-ins that weren't originally known by the framework designers.
Writing configuration files is easy: This is spurious, but since the configuration is a Python file it can be edited and debugged within the IDE of the program itself.
Implicit error-checking: If your program requires an option called FILE_NAME and that isn't in the settings the program will throw an exception. This means that settings become mandatory and error handling of the settings can be more explicit. This can be a double edged sword, but manually changing config files should be for power editors who should be able to handle the consequences of exceptions.
Config options are easily accessed and namespaces: Once you go import settings you can wildly start calling settings.UI_COLOR or settings.TIMEOUT. These are clear, and with the right IDE, tracking where these settings are made becomes easier than with flat files.
But the most powerful reason: Overrides, overrides, overrides. This is quite an advanced situation and can be use-case specific, but one that is encouraged by django in a few places.
Picture that you are building a web application, where there is a development and production server. Each of these need their own settings, but 90% of them are the same. In that case you can do things like define a config file that covers all of development and make it (if its safer) the default settings, and then override if its production, like so:
PORT = 8080
HOSTNAME = "dev.example.com"
COLOR = "0000FF"
if SITE_IS_LIVE:
import * from production_settings.py
Doing an import * from will cause any settings that have been declared in the production_settings.py file to override the declarations in the settings file.
I've not seen a best practise guideline or PEP document that covers how to do this, but if you wanted some general guidelines, the django settings.py is a good example to follow.
Use consistent variable names, preferably UPPER CASE as they are understood to be settings or constants.
Expect odd data structures, if you are using Python as the configuration language, then try to handle all basic data types.
Don't try and make an interface to change settings, that isn't a simple text editor.
When shouldn't you use this approach? When you are dealing with simple key/value pairs that need to be changed by novice users. Python configs are a power user option only. Novice users will forget to end quotes or lists, not be consistent, will delete options they think don't apply and will commit the unholiest of unholies and will mix tabs and spaces spaces only. Because you are essentially dealing with code not config files, all off these will break your program. On the otherside, writing a tool that would parse through through a python file to find the appropriate options and update them is probably more trouble than it is worth, and you'd be better of reusing an existing module like ConfigParser
I think Python code gets directly used for configuration mostly because it's just so easy, quick, powerful and flexible way to get it done. There is currently no other tool in the Python ecosystem that provides all these benefits together. The ConfigParserShootout cat give you enough reasons why it may be better to roll Python code as config.
There are some security considerations that can be worked around either by defensive code evaluation or by policies such as properly setting the filesystem permissions in deployment.
I've seen so much struggle with rather complex configuration being done in various formats, using various parsers, but in the end being the easiest when done in code.
The only real downside I came upon is that people managing the configuration have to be somewhat aware of Python, at least the syntax, to be able to do anything and not to brake anything. May or may not matter case by case.
Also the fact that some serious projects, such as Django and Sphinx, are using this very approach should be soothing enough:
https://docs.djangoproject.com/en/dev/topics/settings/
http://sphinx-doc.org/config.html
There are many options for writing configuration files, with well written parsers:
ini
json
yaml
xml
csv
there's no good reason to have any kind of configuration be parsed as a python script directly. That could led to many kind of problems, from the security aspects to the hard to debug errors, that could be raised late in the run of the program life.
There's even discussions to build an alternative to the setup.py for python packages, which is pretty close to a python source code based configuration from a python coder's point of view.
Otherwise, you may just have seen python objects exported as strings, that looks a bit like json, though a little more flexible… Which is then perfectly fine as long as you don't eval()/exec() them or even import them, but pass it through a parser, like 'ast.literal_eval' or parsing, so you can make sure you only load static data not executable code.
The only few times I'd understand having something close to a config file written in python, is a module included in a library that defines constants used by that library designed to be handled by the user of the library. I'm not even sure that would be a good design decision, but I'd understand such a thing.
edit:
I wouldn't consider django's settings.py an example of good practice, though I consider it's part of what I'm consider a configuration file for coding-literate users that works fine because django is aimed at being used mostly by coders and sysadmins. Also, django offers a way of configuration through a webpage.
To take #lego's arguments:
There is no need to parse the file
there's no need to explicitly parse it, though the cost of parsing is anecdotic, even more given the safety and the extra safety and the ability to detect problems early on
Configuration settings can be more than just key/values
ini files apart, you can define almost any fundamental python type using json/yaml or xml. And you don't want to define classes, or instanciate complex objects in a configuration file…
Writing configuration files is easy:
but using a good editor, json/yaml or even xml syntax can be checked and verified, to have a perfectly parsable file.
Implicit error-checking:
not an argument neither, as you say it's double sworded, you can have something that parses fine, but causes an exception after many hours of run.
Config options are easily accessed and namespaces:
using json/yaml or xml, options can easily be namespaced, and used as python objects naturally.
But the most powerful reason: Overrides, overrides, overrides
It's not a good argument neither in favor of python code. Considering your code is made of several modules that are interdendant and use a common configuration file, and each of them have their own configuration, then it's pretty easy to load first the main configuration file as a good old python dictionary, and the other configuration files just loaded by updating the dictionary.
If you want to track changes, then there are many recipes to organize a hierarchy of dicts that fallbacks to another dict if it does not contain the value.
And finally, configuration values changed at runtime can't be (actually shouldn't be) serialized in python correctly, as doing so would mean changing the currently running program.
I'm not saying you shouldn't use python to store configuration variables, I'm just saying that whatever syntax you choose, you should get it through a parser before getting it as instances in your program. Never, ever load user modifiable content without double checking. Never trust your users!
If the django people are doing it, it's because they've built a framework that only makes sense when gathering many plugins together to build an application. And then, to configure the application, you're using a database (which is a kind of configuration file… on steroids), or actual files.
HTH
I've done this frequently in company internal tools and games. Primary reason being simplicity: you just import the file and don't need to care about formats or parsers. Usually it has been exactly what #zmo said, constants meant for non programmers in the team to modify (say the size of the grid of the game level. or the display resolution).
Sometimes it has been useful to be able to have logic in the configuration. For example alternative functions that populate the initial configuration of the board in the game. I've found this a great advantage actually.
I acknowledge that this could lead to hard to debug problems. Perhaps in these cases those modules have been more like game level init modules than typical config files. Anyhow I've been really happy about the straightforward way to make clear textual config files with the ability to have logic there too and haven't gotten bit by it.
This is yet another config file option. There are several quite adequate config file formats available.
Please take a moment to understand the system administrator's viewpoint or some 3rd party vendor supporting your product. If there is yet another config file format they might drop your product. If you have a product that is of monumental importance then people will go through the hassle of learning the syntax just to read your config file. (like X.org, or apache)
If you plan on another programming language accessing/writing the config file info then a python based config file would be a bad idea.
Answer Credit to Leenert Regebro:
Apparently my instincts were right, and this is impossible. And obvious as well since there was only one answer in two hours.
Thanks for all the coments as well.
My Google-foo is failing me.
I am writing a library of custom exceptions as a module, for use in multiple projects under a single publisher. I may have no say in the other projects, or I may have a say. So it could be in use both by me and others. The "and others" is the problem here. Within my exceptions module there will be specific functions for outputting tracebacks etc. to log files using the logging module. This is fine for me, because I use the logging module.
But if someone else, not using logging, uses the exceptions library, I need to skip the logging part. A try...except resolves this problem, but what if they ARE using logging? In this case I need to be able to determine their logging scheme (console/file/stream, file names etc.) This is so that I can create a sub-logger, which will write to their file (or console or what have you):
<snip>
their_logger = THE_FUNCTION_I_CANNOT_FIGURE_OUT_HOW_TO_WRITE()
temp_var = their_logger.name + ".ExceptionLogger"
myLogger = logging.getLogger(temp_var)
</snip>
Obviously I could create a separate class or function it instantiate my module and have it receive a parameter of type logging.logger, but I would prefer to idiot proof this, if it is even possible.
I cant even check a global or the globals() dict for a value that I know of, because the other programmer might not use one.
Is there any way to do this? (Assuming my library has been imported, and possibly not by the top level application...) I personally have never tried to get data from upstream in the namespaces to be available in a lower namespace without explicit passing, and I doubt it is even possible, but there are a lot of programmers out there, any one ever achieved this?
It's a bad idea to include optional configuration by default. Instead of adding the logging specifics by default and then make some sort of wild guess hidden by a try/except to exclude it, put that part of code into a function, and call it from your code explicitly.
You can not idiot-proof things. In fact, the more magic and hidden logic you have, the less idiot-proof it will be, and soon it will instead be intelligence-proof where it becomes really difficult to understand the magic.
So go with your idea of making a function and passing in the logger instead.