I came across a built-in function compile today. Though i read the documentation but still do not understand it's usage or where it is applicable. Please can anyone explain with example the use of this function. I will really appreciate examples.
From the documentation, the function takes some parameters as shown below.
compile(source, filename, mode[, flags[, dont_inherit]])
It is not that commonly used. It is used when you have Python source code in string form, and you want to make it into a Python code object that you can keep and use. Here's a trivial example:
>>> codeobj = compile('x = 2\nprint "X is", x', 'fakemodule', 'exec')
>>> exec(codeobj)
X is 2
Basically, the code object converts a string into an object that you can later call exec on to run the source code in the string. (This is for "exec" mode; the "eval" mode allows use of eval instead, if the string contains code for a single expression.) This is not a common task, which is why you may never run across a need for it.
The main use for it is in metaprogramming or embedding situations. For instance, if you have a Python program that allows users to script its behavior with custom Python code, you might use compile and exec to store and execute these user-defined scripts.
Another reason compile is rarely used is that, like exec, eval, and their ilk, compile is a potential security hole. If you take user code in string form and compile it and later exec it, you could be running unsafe code. (For instance, imagine that in my example above the code was formatYourHardDrive() instead of print x.)
compile is a lower level version of exec and eval. It does not execute or evaluate your statements or expressions, but returns a code object that can do it. The modes are as follows:
compile(string, '', 'eval') returns the code object that would have been executed had you done eval(string). Note that you cannot use statements in this mode; only a (single) expression is valid. Used for a single expression.
compile(string, '', 'exec') returns the code object that would have been executed had you done exec(string). You can use any number of statements here. Used for an entire module.
compile(string, '', 'single') is like the exec mode, but it will ignore everything except for the first statement. Note that an if/else statement with its results is considered a single statement. Used for one single statement.
Take a look that the documentation. There is also an awesome (well, dumbed down) explanation at http://joequery.me/code/python-builtin-functions/#compile with an excellent example of usage.
What specifically don't you understand? The documentation explains that it will:
Compile the source into a code or AST object. Code objects can be executed by an exec statement or evaluated by a call to eval(). source can either be a Unicode string, a Latin-1 encoded string or an AST object. Refer to the ast module documentation for information on how to work with AST objects.
So it takes python code, and returns on of those two things
exec will execute the python code
eval will evaluate an expression, which is less functional than exec
ast allows you to navigate the Abstract Syntax Tree that the code generates
Related
I have a string stored in a database stands for a class instance creation for example module1.CustomHandler(filename="abc.csv", mode="rb"), where CustomHandler is a class defined in module1.
I would like to evaluate this string to create a class instance for a one time use. Right now I am using something like this
statement = r'module1.CustomHandler(filename="abc.csv", mode="rb")' # actually read from db
exec(f'from parent.module import {statement.split(".")[0]}')
func_or_instance = eval(statement) # this is what I need
Only knowledgable developers can insert such records into database so I am not worried about eval some unwanted codes. But I've read several posts saying eval is unsafe and there is always a better way. Is there a way I can achieve this without using eval?
You might want to take a look at the ast Python module, which stands for abstract syntax trees. It's mainly used when you need to process the grammar of the programming language, work with code in string format, and so much more functions available in the official documentation.
In this case eval() function looks like the best solution, clear and readable, but safe only under certain conditions. For example, if you try to evaluate a code that contains a class not implemented in the code, it will throw an exception. That's the main reason why eval is sometimes unsafe.
Why is this a valid statement in Python?
"Guido"
This tripped me up with a multi-line string where I did not properly use parens:
# BAD
message = "Guido"
" van Rossum"
# GOOD
message = ("Guido"
" van Rossum")
Is it purely for the repl or is there some other reasoning for it?
Expressions are statements in Python (and most other imperative languages) for several good reasons:
What if a function, like foo() does something useful but also returns a value? foo() would be an invalid statement if the language didn't allow us to implicitly discard the return value.
Docstrings! They are just strings that happen to be the first statement of a module/class/function, so no special syntax was needed to support them.
In general, it's hard for the interpreter to determine whether an expression might have side effects. So it's been designed to not even try; it simply evaluates the expression, even if it's a simple constant, and discards the result.
To check for such mistakes as you mentioned, and many others, pylint can be helpful. It has a specific warning for this very case. However, it seems to not catch the mistake in your exact example code (using PyLint version 2.4.4); might be a bug.
I have a class and I need to use string input to create it and I've heard about exec(), so I tried using that and I put the string in properly yet it gives me errors, this is the exec line:
exec(" ".join(args[2:])).toString()
The first 2 parts of the list are not relevant. I debugged this string just to see it is correct and even tried to hardcode it and it worked, but it didn't when I used exec.
What is wrong with this and how can I make this right?
Appreciating all the comment :)
Edit:
The error I get is AttributeError saying it is a NoneType, although if I just hardcode it it works perfectly fine.
Solution 1: instead of passing a Python expression, either use argparse's subcommands or define your own easily-parsable mini-language. You can store relevant classes and functions in a dict to get them from strings (better than relying on globals as it lets you whitelist only what you want to expose)
Solution 2: use the full power of the ast module to parse the Python expression into an ast then write your own visitors to safely evaluate the ast.
Solution 3: keep on using exec or eval and wait until some script kiddie erases your full hard-drive or worse.
When writing code in IDLE, sometimes when I insert a function like re.sub( in the example, a window pops up explaining the function and the inputs it needs. I find this very helpful and would like to have this window pop up every time. I googled and tried different key combinations but I can't see to find how you do this.
Can somebody help me with this?
It is pretty simple if you want a key combination to show calltip.
Just type "ctrl+\" as shown in the picture here:
One thing to remember it will only work when you have already typed the opening parenthesis and not before.
As shown in the screenshots below:
Parenthesis opened
Inside the Parenthesis
Without opening parenthesis
Your question is specific to the python IDLE. In IDLE, you have this functionality enabled by default. For it to work, the function (or method) has to be available in the current namespace.That means it has to either be defined in the running environment, or imported in to the running environment.
For example:
>>> def foo(x)
"""the foo function"""
return x
when you type >>> foo( in the prompt after the definition, you will see the explanation which really is the documentation contained in the docstring (the stuff between the triple quotes).
If a function or method does not have any documentation, then you will not see any explanation. For example
>>> def bar(y):
return y
In this case when you type in bar( at the prompt, IDLE will just show y, this is because the function does not have any documentation.
Some built in functions (called builtins) do not have docstrings, often this is because they are implemented in the C programming language. For example
>>> from functools import reduce
>>> reduce(
In this case IDLE will not give any hint because the function does not have any docstring for it to display.
A great companion to learning is the python standard reference. You can lookup built in function definitions there for clear explanations about what they do. On the other hand, when writing your own functions, remember to put docstrings as they will help you as you go on.
IDLE's calltips contain the function signature (if directly available) followed by the beginning of the docstring (if there is one). For builtins that have not gotten the 'Argument Clinic' treatment, the signature is the beginning of the docstring. This is the case for reduce. In 3.6 and 3.7, when I type reduce( after the import and prompt, the calltip contains the signature as given in the docstring. To see the entire reduce() docstring, use >>> help(reduce) or enter reduce.__doc__.
To see more calltips when editing in the editor, run your code after entering the import statements. For instance, if you start IDLE and immediate edit a new file and enter
import reduce
reduce(
you see no calltip, as you described in your question. But if you hit F5 after the import and return to the editor, you will. Similarly, if you want to see calltips for your own functions, run the file occasionally after defining them.
If you checked Python C- API documentation about running python code via the C calls, you will always find mention to PyCompilerFlags, but nothing really describes what is it except the last portion of documentation and says nothing about its possible values and their effect on execution.
PyCompilerFlags is the C API equivalent to the flags argument passed to compile and related functions in Python. This probably isn't at all obvious if you don't already know the Python docs forward and backward before looking at the CPython C-API docs.
From compile:
The optional arguments flags and dont_inherit control which future statements affect the compilation of source. If neither is present (or both are zero) the code is compiled with those future statements that are in effect in the code that is calling compile(). If the flags argument is given and dont_inherit is not (or is zero) then the future statements specified by the flags argument are used in addition to those that would be used anyway. If dont_inherit is a non-zero integer then the flags argument is it – the future statements in effect around the call to compile are ignored.
Future statements are specified by bits which can be bitwise ORed together to specify multiple statements. The bitfield required to specify a given feature can be found as the compiler_flag attribute on the _Feature instance in the __future__ module.
Following the link to future statements gives more details on how they work, and the link to the __future__ has a chart showing the list of future statements available.
Another thing that may not be obvious: each future feature flag corresponds to a flag that ends up in the co_flags attribute of a code object. So:
code = compile('1 <> 2', '', 'eval', flags=__future__.barry_as_FLUFL.compiler_flag)
assert code.co_flags & CO_FUTURE_BARRY_AS_BDFL
In C, if you pass struct PyCompilerFlags flags = { CO_FUTURE_BARRY_AS_BDFL } to get the same effect.
If you want to see the actual numeric values for those flags, you have to look up the corresponding CO_* constants in the C source or in the __future__ source.
Things are slightly different in the C API, in a few ways.
Rather than passing both flags and dont_inherit, you only pass flags, which is a complete set of all of the future statements you want to be in effect during the PyRun_* or PyCompile_* call.
Most of the functions take a PyCompile_Flags struct holding an int, instead of a raw int. This is just for the purpose of type checking; in memory, a struct holding an int is stored the same way as an int.
Many functions take their flags by pointer, so you can retrieve the possibly-updated set of flags after running the code.
Let's look at a complete example. I'll use Python 2.7 even though I've been linking to 3.7 docs, just because an example using print is simpler than one using forward annotations.
This code prints an empty tuple:
print()
But if you run the first one with PyRun_SimpleStringFlags, passing CO_FUTURE_PRINT_FUNCTION (0x10000) as the flags`, it will print a blank line, a la Python 3.
If you run this code:
from __future__ import print_function
print()
… then whether you passed in 0 or CO_FUTURE_PRINT_FUNCTION, it will print a blank line. And after the call, if you look at the flags you passed in by reference, it will have that CO_FUTURE_PRINT_FUNCTION or'd onto it. So, if you're compiling and running a chunk at a time, you can pass that same value along to the next string, and it'll inherit that future flag. (Much like when you write a future statement in the interactive interpreter, it affects all the statements you interpret after that.)