If you checked Python C- API documentation about running python code via the C calls, you will always find mention to PyCompilerFlags, but nothing really describes what is it except the last portion of documentation and says nothing about its possible values and their effect on execution.
PyCompilerFlags is the C API equivalent to the flags argument passed to compile and related functions in Python. This probably isn't at all obvious if you don't already know the Python docs forward and backward before looking at the CPython C-API docs.
From compile:
The optional arguments flags and dont_inherit control which future statements affect the compilation of source. If neither is present (or both are zero) the code is compiled with those future statements that are in effect in the code that is calling compile(). If the flags argument is given and dont_inherit is not (or is zero) then the future statements specified by the flags argument are used in addition to those that would be used anyway. If dont_inherit is a non-zero integer then the flags argument is it – the future statements in effect around the call to compile are ignored.
Future statements are specified by bits which can be bitwise ORed together to specify multiple statements. The bitfield required to specify a given feature can be found as the compiler_flag attribute on the _Feature instance in the __future__ module.
Following the link to future statements gives more details on how they work, and the link to the __future__ has a chart showing the list of future statements available.
Another thing that may not be obvious: each future feature flag corresponds to a flag that ends up in the co_flags attribute of a code object. So:
code = compile('1 <> 2', '', 'eval', flags=__future__.barry_as_FLUFL.compiler_flag)
assert code.co_flags & CO_FUTURE_BARRY_AS_BDFL
In C, if you pass struct PyCompilerFlags flags = { CO_FUTURE_BARRY_AS_BDFL } to get the same effect.
If you want to see the actual numeric values for those flags, you have to look up the corresponding CO_* constants in the C source or in the __future__ source.
Things are slightly different in the C API, in a few ways.
Rather than passing both flags and dont_inherit, you only pass flags, which is a complete set of all of the future statements you want to be in effect during the PyRun_* or PyCompile_* call.
Most of the functions take a PyCompile_Flags struct holding an int, instead of a raw int. This is just for the purpose of type checking; in memory, a struct holding an int is stored the same way as an int.
Many functions take their flags by pointer, so you can retrieve the possibly-updated set of flags after running the code.
Let's look at a complete example. I'll use Python 2.7 even though I've been linking to 3.7 docs, just because an example using print is simpler than one using forward annotations.
This code prints an empty tuple:
print()
But if you run the first one with PyRun_SimpleStringFlags, passing CO_FUTURE_PRINT_FUNCTION (0x10000) as the flags`, it will print a blank line, a la Python 3.
If you run this code:
from __future__ import print_function
print()
… then whether you passed in 0 or CO_FUTURE_PRINT_FUNCTION, it will print a blank line. And after the call, if you look at the flags you passed in by reference, it will have that CO_FUTURE_PRINT_FUNCTION or'd onto it. So, if you're compiling and running a chunk at a time, you can pass that same value along to the next string, and it'll inherit that future flag. (Much like when you write a future statement in the interactive interpreter, it affects all the statements you interpret after that.)
Related
The docs states that:
It is important to realize that scopes are determined textually: the global scope of a function defined in a module is that module’s namespace, no matter from where or by what alias the function is called. On the other hand, the actual search for names is done dynamically, at run time
I understand the first part: scopes are determined textually. But what does it mean that the actual search for names is done dynamically at run time? As opposed to what?
Let's try to compare this to what happens in C for instance, as I understand that this is the opposite of what happens in Python.
In C, consider the following code:
int a = 5
printf("The value of a is: %d\n", a);
So in C, the actual search for names is done at compile time - that means that the compiled machine code for the printf function will contain reference to the memory address of a whereas in Python
a = 5
print(a)
The compiled code of the print(a) will contain instructions for going looking in the namespace dictionary for what is pointed to by a and then access it.
Is that correct?
It means that a name can suddenly start resolving to something else, because it was redefined during the execution of the program. The alternative would be to resolve names when the program is read and parsed, and stick to this interpretation. (Which would be somewhat faster allow considerable additional optimization, e.g. by "knowing" things about the default behavior of Python built-in functions; but it is not how the language was designed.)
Here's an example that suddenly changes behavior:
for n in range(3):
print(max([10, 20, 30]))
max = min
This loop will print what you expect on the first iteration, but from then on the identifier max will refer to the local variable max, and will resolve to the builtin min(). Silly, but realistic use cases are a different question...
As opposed to being done statically at compile-time.
For instance in a language like C or Rust, by default symbols are looked up at compile-time, and at runtime the code just goes to whatever was resolved during compilation.
In Python however, every time you call a function the interpreter will look for that name in the relevant scope(s), then will use whatever's bound to that name at that point in time. Semantically if not necessarily technically.
So if e.g. you swap the object assigned to that name, then the code will call the remplacement instead of the original. Even if the replacement is not callable at all.
I'm writing a shared object in Go (c-shared) which will be loaded and run from python. Everything is working fine, until the Go code needs to return an error. I am converting the error to string using error.Error() but when trying to return that to python, cgo is hitting:
panic: runtime error: cgo result has Go pointer
Which is very odd, since this is a string and not a pointer supposedly. I know there are no issues with returning go strings via shared object exported function, as I do that in several other places without any issue.
The Go code looks like:
package main
import "C"
//export MyFunction
func MyFunction() string {
err := CallSomethingInGo()
if err != nil {
return err.Error()
}
return ""
}
func main() {}
The go code is compiled to .so using buildmode=c-shared and then In the python code, I have something like this:
from ctypes import *
lib = cdll.LoadLibrary("./mygocode.so")
class GoString(Structure):
_fields_ = [("p", c_char_p),("n", c_longlong)]
theFunction = lib.MyFunction
theFunction.restype = GoString
err = theFunction()
When the last line executes and the golang code returns NO error then everything is fine and it works! But, if the golang code tries to return an error (e.g. CallSomethingInGo fails and returns err) then the python code fails with:
panic: runtime error: cgo result has Go pointer
I've tried manually returning strings from go to python and it works fine, but trying to return error.Error() (which should be a string per my understanding) fails. What is the correct way to return the string representation of the error to python?
One more piece of info - from golang, I did a printf("%T", err) and I see the type of the error is:
*os.PathError
I also did printf("%T", err.Error()) and confirmed the type returned by err.Error() was 'string' so I am still not sure why this isn't working.
Even stranger to me...I tried modifying the go functions as shown below for a test, and this code works fine and returns "test" as a string back to python...
//export MyFunction
func MyFunction() string {
err := CallSomethingInGo()
if err != nil {
// test
x := errors.New("test")
return x.Error()
}
return ""
}
I'm so confused! How can that test work, but not err.Error() ?
As I said in a comment, you're just not allowed to do that.
The rules for calling Go code from C code are outlined in the Cgo documentation, with this particular issue described in this section, in this way (though I have bolded a few sections in particular):
Passing pointers
Go is a garbage collected language, and the garbage collector needs to know the location of every pointer to Go memory. Because of this, there are restrictions on passing pointers between Go and C.
In this section the term Go pointer means a pointer to memory allocated by Go (such as by using the & operator or calling the predefined new function) and the term C pointer means a pointer to memory allocated by C (such as by a call to C.malloc). Whether a pointer is a Go pointer or a C pointer is a dynamic property determined by how the memory was allocated; it has nothing to do with the type of the pointer.
Note that values of some Go types, other than the type's zero value, always include Go pointers. This is true of string, slice, interface, channel, map, and function types. A pointer type may hold a Go pointer or a C pointer. Array and struct types may or may not include Go pointers, depending on the element types. All the discussion below about Go pointers applies not just to pointer types, but also to other types that include Go pointers.
Go code may pass a Go pointer to C provided the Go memory to which it points does not contain any Go pointers. The C code must preserve this property: it must not store any Go pointers in Go memory, even temporarily. When passing a pointer to a field in a struct, the Go memory in question is the memory occupied by the field, not the entire struct. When passing a pointer to an element in an array or slice, the Go memory in question is the entire array or the entire backing array of the slice.
C code may not keep a copy of a Go pointer after the call returns. This includes the _GoString_ type, which, as noted above, includes a Go pointer; _GoString_ values may not be retained by C code.
A Go function called by C code may not return a Go pointer (which implies that it may not return a string, slice, channel, and so forth). A Go function called by C code may take C pointers as arguments, and it may store non-pointer or C pointer data through those pointers, but it may not store a Go pointer in memory pointed to by a C pointer. A Go function called by C code may take a Go pointer as an argument, but it must preserve the property that the Go memory to which it points does not contain any Go pointers.
Go code may not store a Go pointer in C memory. C code may store Go pointers in C memory, subject to the rule above: it must stop storing the Go pointer when the C function returns.
These rules are checked dynamically at runtime. The checking is controlled by the cgocheck setting of the GODEBUG environment variable. The default setting is GODEBUG=cgocheck=1, which implements reasonably cheap dynamic checks. These checks may be disabled entirely using GODEBUG=cgocheck=0. Complete checking of pointer handling, at some cost in run time, is available via GODEBUG=cgocheck=2.
It is possible to defeat this enforcement by using the unsafe package, and of course there is nothing stopping the C code from doing anything it likes. However, programs that break these rules are likely to fail in unexpected and unpredictable ways.
This is what you are seeing: you have a program that breaks several rules, and now it fails in unexpected and unpredictable ways. In particular, your lib.MyFunction is
a Go function called by C code
since Python's cdll handlers count as C code. You can return nil, as that's the zero-value, but you are not allowed to return Go strings. The fact that the empty-string constant (and other string constants from some other error types) is not caught at runtime is a matter of luck.1
1Whether this is good luck or bad luck depends on your point of view. If it failed consistently, perhaps you would have consulted the Cgo documentation earlier. Instead, it fails unpredictably, but not in your most common case. What's happening here is that the string constants were compiled to text (or rodata) sections and therefore are not actually dynamically allocated. However, some—not all, but some—errors' string bytes are dynamically allocated. Some os.PathErrors point into GC-able memory, and these are the cases that are caught by the
reasonably cheap dynamic checks
mentioned in the second-to-last paragraph.
I am currently trying to pass a string to a Fortran library. I have gotten other functions from this library to work, but this particular one seems to be unique in that it wants a string passed to it as an argument to the function.
Looking at the source code, the function requires three arguments
SUBROUTINE EZVOLLIB(VOLEQI,DBHOB,HTTOT,VOL)
and the arguments are defined:
IMPLICIT NONE
CHARACTER*(*) VOLEQI
CHARACTER*10 VOLEQ
REAL DBHOB,HTTOT,TOPD, VOL(15), MHT
INTEGER REGN,ERRFLG
In Python my call to the function looks like
from ctypes import *
mylib = cdll.LoadLibrary('/home/bryce/Programming/opencompile/libvollib.so')
dbhob = c_float(42.2)
vol = (c_float * 15)()
voleqi = c_char_p("101DVEW119 ")
mylib.ezvollib_(voleqi, dbhob, vol)
This runs without a segmentation fault, but does not seem to "fill" the variable vol with the desired 15 float values.
Is there any way to get vol to retrieve the values being returned from the EZVOLLIB function?
There are many similar questions here, but it is hard to find an exact duplicate. There are several possible ways to do that with different degrees of universal correctness and portability.
The most correct way is to use modern Fortran to C interoperability as explained in fortran77, iso_c_binding and c string That requires writing more Fortran wrapping code.
There are people who are strictly against writing any more Fortran, even-though that is the only portable solution. In that case they must explore what is the actual calling convention for Fortran strings in their compiler. Usually a hidden integer argument with the string length is passed to the subroutine. For gfortran see https://gcc.gnu.org/onlinedocs/gfortran/Argument-passing-conventions.html
Your ctypes interface could employ these compiler-specific calling conventions but then the interface will be, well, compiler-specific. But you are already relying on the specific name mangling ezvollib_ and that is compiler-specific as well. You can find examples here on SO where people were bitten by relying on that.
Also note, as noted by High Performance Mark, that the subroutine in question has four arguments, not three: EZVOLLIB(VOLEQI,DBHOB,HTTOT,VOL). Calling it with just three as in mylib.ezvollib_(voleqi, dbhob, vol) is an error. You are missing the HTTOT argument.
win32 API QueryServiceConfig2 function supports the SERVICE_CONFIG_TRIGGER_INFO structure to get event(s) that trigger the service startup. However, python's win32service.QueryServiceConfig2() does not list such value as a parameter option. Is it possible to get that information with the win32service module?
Unfortunately, no. Here's a simple code snippet ran under Python 3.5 and PyWin32 v221:
#!/usr/bin/env python3
import win32service
if __name__ == "__main__":
for name in dir(win32service):
if name.startswith("SERVICE_CONFIG_"):
print(name, getattr(win32service, name))
Output:
(py35x64_test) e:\Work\Dev\StackOverflow\q046916726>"c:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" a.py
SERVICE_CONFIG_DELAYED_AUTO_START_INFO 3
SERVICE_CONFIG_DESCRIPTION 1
SERVICE_CONFIG_FAILURE_ACTIONS 2
SERVICE_CONFIG_FAILURE_ACTIONS_FLAG 4
SERVICE_CONFIG_PRESHUTDOWN_INFO 7
SERVICE_CONFIG_REQUIRED_PRIVILEGES_INFO 6
SERVICE_CONFIG_SERVICE_SID_INFO 5
I've also checked win32con (which is another PyWin32 module, that only contains constant definitions), but no luck either.
Then, I took a look at ${PYWIN32_SRC_DIR}/pywin32-221/win32/src/win32service.i, and noticed that for ChangeServiceConfig2 (and also QueryServiceConfig2), the InfoLevel argument is specifically checked against the above constants, and thus passing its value (8) directly, would raise an exception (NotImplementedError).
Before going further, let's spend a little bit understanding what happens when calling such a Python wrapper (like win32service.QueryServiceConfig2):
Arguments (Python style - if any) are converted to C style
C function is called with the above converted arguments
Function result (or output arguments) - if any - are converted back to Python
For [MS.Docs]: ChangeServiceConfig2W function, data is transferred back and forth via arguments (depending on dwInfoLevel value, lpInfo can have various meanings):
Let's take a look to a value that is supported: e.g. SERVICE_CONFIG_PRESHUTDOWN_INFO:
lpInfo is a pointer to a [MS.Docs]: SERVICE_PRESHUTDOWN_INFO structure which only has a dwPreshutdownTimeout member (which is a simple DWORD)
On the other hand, SERVICE_CONFIG_TRIGGER_INFO:
lpInfo is a pointer to a [MS.Docs]: SERVICE_TRIGGER_INFO structure
pTriggers member is a pointer to a [MS.Docs]: SERVICE_TRIGGER structure
pDataItems member is a pointer to a [MS.Docs]: SERVICE_TRIGGER_SPECIFIC_DATA_ITEM structure
which is waaay more complex (and note that all involved structures have other members as well, I only listed the ones that increase the nesting level).
Adding support for all those arguments is not exactly a trivial task, so they aren't handled (at least for the moment).
There are more examples like this one, I guess it's a matter of priority, as not many people requested the functionality, combined with MS's (unfortunate?) design decision to have functions with such complex behaviors.
As an alternative (a quite complex one), you can use [Python 3.Docs]: ctypes - A foreign function library for Python, but you'll have to define all the structures that I listed above in Python (to extend ctypes.Structure).
As a side note, I was in the exact situation once: I had to call [MS.Docs]: LsaLogonUser function (!!! 14 freaking arguments !!!). Obviously, it wasn't exported by PyWin32, so I called it via ctypes, but I had to write:
~170 lines of code to define structures (only the ones that I needed for my scenario)
~50 lines to populate them and call the function
I am quite new to python programming and working on a quite large python module using IntelliJ. I was wondering if I have a static method that is called from multiple places and I change the signature of this method to accept a different no. of arguments. I might fix the actual method calls in some places but might miss changing the calls in some other. In java, I will receive a compile time error but since python is interpreted I will only figure out that sometime during runtime(probably in production). Till now I have been using the command 'python -m compileall' but I was wondering like in Java is there any way to get syntax errors in IntelliJ.
Unit tests and static code analysis tools such as pylint will help. pylint is able to detect incorrect number of arguments being passed to functions.
If you're using Python 3, function annotations might be useful, and mypy can type check annotated function calls (apparently, I've not used it).
In general the strategy for changing existing function signatures without breaking dependent code is to use keyword arguments. For example, if you wanted to add a new argument to a function, add it as a keyword argument:
#def f(a):
# """Original function"""
# print(a)
def f(a, b=None):
"""New and improved function"""
print(a)
if b is not None:
print(b)
Now calls with and without the new argument will work:
>>> f('blah')
blah
>>> f('blah', 'cough')
blah
cough
Of course this will not always work, e.g. if argument(s) are removed, or if the semantics of the function are changed in a way that breaks existing code.