Why does python logging module use old string formatting? - python

Python has two main string formatting options % and str.format. logging module has a lazy feature.
logging.debug('The value is %s', huge_arg)
This does not construct the string if the log line is not going to be print. However, this feature works only if the sting uses old style % format. Is there a way to use str.format with this lazy feature? There could be a named arg like:
logging.debug('The value is {}', fmt_arg=(huge_arg))

The only answer to your question is that - so far - nobody has volunteered to change the logging code to support the newer format feature. If you're volunteering, why not ask about it on the Python-Dev mailing list? I expect it's trickier than you realize (e.g., there may be calls to logging functions already that happen to pass a fmt_arg keyword argument). Good luck ;-)

I'm a little scared to contradict the TimBot, but I have another answer :-)
The logging module uses % formatting because it pre-dates the appearance of {}-formatting in Python (logging appeared in 2.3, str.format in 2.6). Logging has not been converted over to {}-formatting because:
You can't just switch over without breaking a lot of existing code in third party libraries and applications, so %-formatting is here to stay.
When {}-formatting arrived, it was a bit slower that %-formatting (AFAIK, it still is, though it offers more control over output), and people regard logging as an overhead as it is, never mind adding to that overhead ;-)
There is already support for {}-formatting and even $-style formatting (string.Template), as described in this post from 2010. The approach described supports logging's lazy formatting.

Related

Is there any way to divide an equation by another in python? [duplicate]

How can I (easily) take a string such as "sin(x)*x^2" which might be entered by a user at runtime and produce a Python function that could be evaluated for any value of x?
Python's own internal compiler can parse this, if you use Python notation.
If your change the notation slightly, you'll be happier.
import compiler
eq= "sin(x)*x**2"
ast= compiler.parse( eq )
You get an abstract syntax tree that you can work with.
EDIT parser is deprecated in Python 3.9: https://docs.python.org/3/whatsnew/3.9.html#new-parser
You can use Python parser:
import parser
from math import sin
formula = "sin(x)*x**2"
code = parser.expr(formula).compile()
x = 10
print(eval(code))
It performs better than pure eval and, of course, avoids code injection!
f = parser.parse('sin(x)*x^2').to_pyfunc()
Where parser could be defined using PLY, pyparsing, builtin tokenizer, parser, ast.
Don't use eval on user input.
pyparsing might do what you want (http://pyparsing.wikispaces.com/) especially if the strings are from an untrusted source.
See also http://pyparsing.wikispaces.com/file/view/fourFn.py for a fairly full-featured calculator built with it.
To emphasize J.F. Sebastian's advice, 'eval' and even the 'compiler' solutions can be open to subtle security holes. How trustworthy is the input? With 'compiler' you can at least filter out things like getattr lookups from the AST, but I've found it's easier to use PLY or pyparsing for this sort of thing than it is to secure the result of letting Python help out.
Also, 'compiler' is clumsy and hard to use. It's deprecated and removed in 3.0. You should use the 'ast' module (added in 2.6, available in 2.5 as '_ast').
Sage is intended as matlab replacement and in intro videos it's demonstrated how similar to yours cases are handled. They seem to be supporting a wide range of approaches. Since the code is open-source you could browse and see for yourself how the authors handle such cases.
In agreement with vartec. I would use SymPy - in particular the lambdify function should do exactly what you want.
See: http://showmedo.com/videotutorials/video?name=7200080&fromSeriesID=720
for a very nice explanation of this.
Best wishes,

proper replacement of QString().arg method in python3

When it comes to internationalization - using python2 and PyQt4 - the "proposed way" to format a translated string is using the QString.arg() method:
from PyQt4.QtGui import QDialog
#somewhere in a QDialog:
self.tr("string %1 %2").arg(arg1).arg(arg2)
But QString() doesn't exist in python3-PyQt4.
So my question is, what is the best way to format any translated strings in python3? Should I use the standard python method str.format() or maybe there is something more suitable?
The QString::arg method is really there as a workaround for C++'s limited string formatting support, to make sure you don't use sprintf with all the problems that entails (not handling placeholders that are in different orders in different languages, buffer overruns, etc.). Because Python doesn't have any such problems, there's no good reason to use it.
In fact, there's very little reason to ever use QString explicitly. In PyQt4, it wasn't phased out completely, but by PyQt5, it was. (Technically, PyQt4 supports "string API v2" in both Python 2.x and 3.x, but only enables it by default in 3.x; PyQt4 enables v2 by default in both, and hides the ability to switch back to v1.) See Python Strings, Qt Strings and Unicode in the documentation for some added info.
There is one uncommon, but major if it affects you, exception: If you're writing an app that's partly in Qt/C++ and partly in PyQt, you're going to have problems sharing I18N data when some of them are in "string %1 %2" format and others are in "string {1} {2}" format. (The first time you ship one of your file out to an outsourced translation company, they're going to get it wrong, guaranteed.)
Yes, just use standard python string formatting.
QString is gone because it's pretty much interchangable with python's unicode strings (which are str in python3 and unicode in python2), so PyQt takes care of converting one into the other as needed.
QString being disabled isn't limited to python3, it's just the default there. You can get the same on python2 by doing this befor importing anything from PyQt4:
import sip
sip.setapi('QString', 2)

Python Core Library and PEP8

I was trying to understand why Python is said to be a beautiful language. I was directed to the beauty of PEP 8... and it was strange. In fact it says that you can use any convention you want, just be consistent... and suddenly I found some strange things in the core library:
request()
getresponse()
set_debuglevel()
endheaders()
http://docs.python.org/py3k/library/http.client.html
The below functions are new in the Python 3.1. What part of PEP 8 convention is used here?
popitem()
move_to_end()
http://docs.python.org/py3k/library/collections.html
So my question is: is PEP 8 used in the core library, or not? Why is it like that?
Is there the same situation as in PHP where I cannot just remember the name of the function because there are possible all ways of writing the name?
Why PEP 8 is not used in the core library even for the new functions?
PEP 8 recommends using underscores as the default choice, but leaving them out is generally done for one of two reasons:
consistency with some other API (e.g. the current module, or a standard interface)
because leaving them out doesn't hurt readability (or even improves it)
To address the specific examples you cite:
popitem is a longstanding method on dict objects. Other APIs that adopt it retain that spelling (i.e. no underscore).
move_to_end is completely new. Despite other methods on the object omitting underscores, it follows the recommended PEP 8 convention of using underscores, since movetoend is hard to read (mainly because toe is a word, so most people's brains will have to back up and reparse once they notice the nd)
set_debuglevel (and the newer set_tunnel) should probably have left the underscore out for consistency with the rest of the HTTPConnection API. However, the original author may simply have preferred set_debuglevel tosetdebuglevel (note that debuglevel is also an argument to the HTTPConnection constructor, explaining the lack of a second underscore) and then the author of set_tunnel simply followed that example.
set_tunnel is actually another case where dropping the underscore arguably hurts readability. The juxtaposition of the two "t"s in settunnel isn't conducive to easy parsing.
Once these inconsistencies make it into a Python release module, it generally isn't worth the hassle to try and correct them (this was done to de-Javaify the threading module interface between Python 2 and Python 3, and the process was annoying enough that nobody else has volunteered to "fix" any other APIs afflicted by similar stylistic problems).
From PEP8:
But most importantly: know when to be
inconsistent -- sometimes the style
guide just doesn't apply. When in doubt, use your best judgment. Look
at other examples and decide what looks best. And don't hesitate to
ask!
What you have mentioned here is somewhat consistent with the PEP8 guidelines; actually, the main inconsistencies are in other parts, usually with CamelCase.
The Python standard library is not as tightly controlled as it could be, and the style of modules varies. I'm not sure what your examples are meant to illustrate, but it is true that Python's library does not have one voice, as Java's does, or Win32. The language (and library) are built by an all-volunteer crew, with no corporation paying salaries to people dedicated to the language, and it sometimes shows.
Of course, I believe other factors outweigh this negative, but it is a negative nonetheless.

What's better, ConfigObj or ConfigParser?

Which is better for creating a settings file for Python programs, the built-in module (ConfigParser), or the independent project (ConfigObj)?
I recently switched from configparser to configobj, and I'm thrilled to have done so.
For me, the big difference is configobj's validator. It lets me very easily/succinctly (1) define the type and acceptable values for each entry, and (2) set defaults.
Those two features save me a lot of code and prevent a lot from going wrong. Plus, there's really no extra overhead to using configobj in terms of the complexity of my code, and the library is quite small.
Depending on your Python version, it may be contentious whether this answers your question, but after a short look at ConfigParser and ConfigObj, I settled for configparser, the Python 3 version of ConfigParser. There’s also a backported version on PyPI.
ConfigParser seemed cumbersome to me, maybe even—dare I say it—unpythonic, and with ConfigObj I encountered an esoteric problem with lists (I’d prefer to be able to align them vertically, since mine get really long), and it didn’t help that the latest version was published more than two years ago. The API of configparser, however, looked as spiffy as that of ConfigObj (albeit not as feature-rich), and when we move to Python 3, I can probably switch to the built-in version easily.
By the way, it works great so far.
Thus far, I found ConfigParser sufficient any time I used it. Plus, it's an included battery and not a third party library.
Though I have to admit, the code samples in the ConfigObj documentation make ConfigParser look really sucksy in comparision. For a script which heavily manipulates .ini files, I'd propably prefer it.

Is "safe_eval" really safe?

I'm looking for a "safe" eval function, to implement spreadsheet-like calculations (using numpy/scipy).
The functionality to do this (the rexec module) has been removed from Python since 2.3 due to apparently unfixable security problems. There are several third-party hacks out there that purport to do this - the most thought-out solution that I have found is
this Python Cookbok recipe, "safe_eval".
Am I reasonably safe if I use this (or something similar), to protect from malicious code, or am I stuck with writing my own parser? Does anyone know of any better alternatives?
EDIT: I just discovered RestrictedPython, which is part of Zope. Any opinions on this are welcome.
Depends on your definition of safe I suppose. A lot of the security depends on what you pass in and what you are allowed to pass in the context. For instance, if a file is passed in, I can open arbitrary files:
>>> names['f'] = open('foo', 'w+')
>>> safe_eval.safe_eval("baz = type(f)('baz', 'w+')", names)
>>> names['baz']
<open file 'baz', mode 'w+' at 0x413da0>
Furthermore, the environment is very restricted (you cannot pass in modules), thus, you can't simply pass in a module of utility functions like re or random.
On the other hand, you don't need to write your own parser, you could just write your own evaluator for the python ast:
>>> import compiler
>>> ast = compiler.parse("print 'Hello world!'")
That way, hopefully, you could implement safe imports. The other idea is to use Jython or IronPython and take advantage of Java/.Net sandboxing capabilities.
Writing your own parser could be fun! It might be a better option because people are expecting to use the familiar spreadsheet syntax (Excel, etc) and not Python when they're entering formulas. I'm not familiar with safe_eval but I would imagine that anything like this certainly has the potential for exploitation.
If you simply need to write down and read some data structure in Python, and don't need the actual capacity of executing custom code, this one is a better fit:
http://code.activestate.com/recipes/364469-safe-eval/
It garantees that no code is executed, only static data structures are evaluated: strings, lists, tuples, dictionnaries.
Although that code looks quite secure, I've always held the opinion that any sufficiently motivated person could break it given adequate time. I do think it will take quite a bit of determination to get through that, but I'm relatively sure it could be done.
Daniel,
Jinja implements a sandboxe environment that may or may not be useful to you. From what I remember, it doesn't yet "comprehend" list comprehensions.
Sanbox info
The functionality you want is in the compiler language services, see
http://docs.python.org/library/language.html
If you define your app to accept only expressions, you can compile the input as an expression and get an exception if it is not, e.g. if there are semicolons or statement forms.

Categories

Resources