It is written in a documentation:
Such extension modules can do two things that can’t be done directly
in Python: they can implement new built-in object types, and they can
call C library functions and system calls.
Syscalls
I cannot see why "system calls" are special here. I know what it is syscall. I didn't see why it is special and why it cannot be done directly in Python.
Especially, we can use open in Python to open a file. It must be a underlying syscall to get descriptor for file ( in Unix systems).
It was just open. Besides that we can use: call(["ls", "-l"]) and it also must use syscall like execve or something like that.
Functions
Why is calling C library function is special? After all:
ctypes is a foreign function library for Python. It provides C
compatible data types, and allows calling functions in DLLs or shared
libraries. It can be used to wrap these libraries in pure Python.
Essentially system calls interact with the underlying system services(that is the Kernel for Linux). C functions on the other hand run on user space exclusively. To that sense system call is more "special".
Related
There is a patch to add support for the POSIX openat functions (and other *at functions like fstatat) to the python standard library that is marked as closed with resolution fixed, but the os, posix and platform modules do not currently include any of these methods.
These methods are the standard way of solving problems like this in C and other languages efficiently and without race conditions.
Are these included in the standard library currently somewhere? And if not, are there plans to include this in the future.
Yes, this is supported by passing the dir_fd argument to various functions in the standard os module. See for example os.open():
Open the file path and set various flags [...]
This function can support paths relative to directory descriptors with the dir_fd parameter.
If you want to use high-level file objects such as those returned by the builtin open() function, that function's documentation provides example code showing how to do this using the opener parameter to that function. Note that open() and os.open() are entirely different functions and should not be confused. Alternatively, you could open the file with os.open() and then pass the file descriptor number to os.fdopen() or to open().
It should also be noted that this currently only works on Unix; the portable and future-proof way to check for dir_fd support is to write code such as the following:
if os.open in os.supports_dir_fd:
# Use dir_fd.
else:
# Don't.
On the other hand, I'm not entirely sure Windows even allows opening a directory in the first place. You certainly can't do it with _open()/_wopen(), which are documented to fail if "the given path is a directory." To be safe, I recommend only trying to open the directory after you check for dir_fd support.
I have my own C++ library project(with source) written in Qt and it uses QTcpsocket, QUdpSocket, QSerialPort signals and slots.
I would like to support this library in Python as well.
What is the preferred way to do this?
Writing a wrapper in Python, if so does it have obstacles?
Dont know if PyQt is just for this purpose?
Or do you thnink is it better to rewrite the lib in Python by just implementing the logic used in C++ library project.
As this is library is part of a SDK, same applies for supporting QT dll with .NET as well in fact, as a second step after supporting Python.
Example API of Qt.
quint16 SendCommandAsync(CmdBaseSpv1* pcommand,
ConnectionArg connectionArg,
emitLogOptions::emitLogOption logOption,
LogInfo &txLogInfo,
LogInfo &rxLogInfo);
I want to call this function from Python.
Function parameters CmdBaseSpv1, ConnectionArg, emitLogOption, LogInfo are all Qt classes.
Some of these arguments are using the QOBJECT base class.
As you see from the function name; it is an Asynchronous function call. Result will emit a signal so I need to get async result as well.
I'll write down what I know about wrapping C++ libraries and will try to source it, but as a huge disclaimer, I have only used this for something very, very simple myself.
You ask about rewriting the library in Python. I would say this depends. If the code is trivial, then I don't see why not. If it is larger and has to be kept up-to-date with other code (as you imply with .Net), I wouldn't. It makes sense to reuse the same code for both.
My suggestion
From what I see of your code I would try to wrap it using boost::python or SWIG.
How to wrap
The main trouble is going to be to create CmdBaseSpv1, ConnectionArg, etc. in Python.
If you don't need any Qt-classes to instantiate your classes, this should be straightforward. However, in case you need the Qt types inside of Python (e.g. because the constructor of CmdBaseSpv1 requires a QString), your task is a lot more complicated because you need a way to convert a Python-string into a QString. If you can, you should only use stl-types.
Everything in Python
The simplest way to wrap a small C library is to use the cffi module (or ctypes). You can write the full binding in Python. However, this is a lot of manual work if your API is large and can get difficult.
There is another problem: ctypes is only compatible with C, not C++. So you'd need to change your interface to be compatible with C, internally you could still use C++ and Qt.
Wrap by hand
An alternative is to wrap the library calls yourself. You can either do this by using the Python API. There are also a few libraries that help you create the bindings. Boost::python seems especially promising and works with C++.
Binding generators
If your API is very large, you should use a binding generator which parses the C++ code and generates the bindings itself. For example sip is one of them. It is used to create the bindings for the whole Qt library. There are a few binding generators out there, one mentioned in the Python docs is SWIG. PySide uses Shiboken and also has a nice description of it on their website.
SWIG has the additional advantage, that you can create bindings for multiple languages, including C#.
PyQt
PyQt is a binding generated from Qt using sip. You'll probably not need it, unless you need to access the full power of Qt from inside Python. If this is the case, consider using sipfor generating the bindings, so things like the signal-slot mechanism are compatible between your library and PyQt.
Challenges with bindings
Bindings come with a few challenges because Python and C++ are different in some key areas.
Memory-management
Memory management in Python is almost automatic, in C++ you're required to do it manually. For example
def myfunc():
mywidget = QWidget()
at the end of myfunc() mywidget gets garbage collected. In C++ however
void myfunc() {
auto mywidget = new QWidget();
}
mywidget is still around. This means that even when inside Python, you need to take care of the C++ memory management. The problems I've seen are memory leaks and dangling pointers. Watch out for this when using callbacks, you don't want Python to garbage collect the callback while C++ thinks it's still alive.
Exceptions
Not all programming languages have exceptions or deal with them the same way. For example, it would be nice if an exception inside C++ can be caught inside Python.
Links to related question
How to wrap a c++ library for python? (example of boost::python)
Exposing a C++ API to Python (discussion about boost::python, SWIG and more)
https://stackoverflow.com/a/5686873 (discusses Cython, another choice)
I've read here about importing a module in python. There is an option to not import a whole module (e.g. sys) and to only import a part of it (e.g. sys.argv). Is that possible in C? Can I include only the implementation of printf or any other function instead of the whole stdio.h library?
I ask this because it seems very inefficient to include a whole file where I need only several lines of code.
I understand that there is a possibility that including only the function itself won't work because it depends on other functions, other includes, defines, and globals. I only ask in order to use this for whole code blocks that contain all the data that are needed in order to execute.
C does not have anything that is equivalent to, or even similar to Python's "from ... import" mechanism.
I ask this because it seems very inefficient to include a whole file where I need only several lines of code.
Actually, what normally happens when you #include a file is that you import the declarations for macros, or functions declared somewhere else. You don't import any executable code ... so the "unnecessary" inclusions have ZERO impact on runtime code size or efficiency.
If you use (i.e. "call") a macro, then that causes the macro body to expanded, which adds to the executable code size.
If you call a function whose declaration you have included, that will add the code ... for the call statement itself. The function does not expanded though. Instead, an "external reference" is added to your ".o" file, which the loader resolves when you create the executable from the ".o" files and the dependent libraries.
Python: "There is an option to not import a whole module" : I think you misunderstand what is going on here. When you specify the names to import, it means that only those names go into you namespace. The "whole" module is compiled, and any code outside functions is run, even when you specify just one name.
C: I am going to assume that you are using an operating system like UNIX/Linux/OS X or Windows (the following does not apply to embedded systems).
The closest C has to import is dynamic runtime linking. That is not part of standard C, it is defined by the operating system. So POSIX has one mechanism and Windows has another. Most people call these library files "DLLs", but strictly speaking that is a Microsoft term, they are "shared objects" (.so) on UNIX type systems.
When a process attaches to a DLL or .so then it is "mapped" into the virtual memory of the process. The detail here varies between operating systems, but essentially the code is split into "pages", the size of which varies, but 4kb for 32-bit systems and 16kb for 64-bit is typical. Only those pages that are required are loaded into memory. When a page is required then a so-called "page-fault" occurs and the operating system will get the page from either the executable file or the swap area (depending on the OS).
One of the advantages of this mechanism is that code pages can be shared between processes. So if you have 50 processes all using the same DLL (like the C run-time library, for example), then only one copy is actually loaded into memory. They all share the one set of pages (they can because they are read-only).
There is no sharing mechanism like that in Python - unless the module is itself written in C and is a DLL (.pyd).
All this occurs without the knowledge of the program.
EDIT: looking at other's answers I realise you might be thinking of the #include pre-processor directive to merge a header file into the source code. Assuming these are standard header files, then they make no difference to the size of your executable, they should be "idempotent". That is, they only contain information of use by the pre-processor, compiler, or linker. If there are definitions in the header file that are not used there should be no side effect.
Linking libraries (-l directive to the compiler) that are not used will make the executable larger, which makes the page tables larger, but aside from that if they are not used then they shouldn't make any significant difference. That is because of the on-demand page-loading described above (the concept was invented in the 1960s in Manchester UK).
I am developing a tool for some numerical analysis of user-defined functions.
The idea is to make a convenient UI in Python, where user can enter C function, then press a button - and receive some output data.
Computations can take minutes or hours, so Numpy-only performance is not acceptable.
I have tried the following approach: the Python-based UI calls gcc, compiles dll from user functions that is than used by my core C-based algorithms in Cython wrappings. It works, but since there is no way to fully unload the python module, I can not recompile user-defined function until the whole UI program is closed and run again.
The only way I see now is to separate the computational core and UI processes and then make them interact via shared memory/messaging. As user wants to update his function, the program terminates the core, recompiles dll and starts the core again.
Can you suggest any common practice in such cases?
Thank you!
Python has really good multiprocessing support (and really not very good threading support), so you could spawn a new python process for each expression to be evaluated, compile and load the dll in the new process, then have it communicate the results back to the parent process. When the spawned process exits, everything should be unloaded.
Each time the user enters a function, you could generate a new module with a random, unique name (as you might do for a temp file). Then compile that module, load it, and call the function.
Over time, this will of course result in a lot of modules being loaded. If users don't run too many functions in a single session, you might get away with it. Alternatively, you could compile the function within a plain DLL instead of a Python module, and load it with ctypes. When you're done with the DLL, simply discard any references to it -- when the ctypes handle is eventually gced, it should unload the DLL (see
How can I unload a DLL using ctypes in Python?).
You will also probably want to delete your DLLs from disk. If you're using Windows, that might be tricky, since it's hard to predict when ctypes will unload it, and Windows won't let you delete it will it's in use.
Have you considered Weave or Instant (less up-to-date)? I've only used instant myself, but it seems like either would be perfect for what you're describing.
They can both automatically compile and cache c code at runtime from a string. I've only ever used instant to inline c, but I'm sure it or weave would work just as well for compiling user input.
Here's an example of how to use weave, from the tutorial.
>>> a = 'string'
>>> def protected_printf(a):
... assert(type(a) == type(1))
... weave.inline(r'printf("%d\n",a);',['a'])
>>> protected_printf(1)
1
>>> protected_printf('string')
AssertError...
Well given a C code , is there a way that i can use other languages like python to execute the C code . What i am trying to say is , there are soo many modules which are built using a language , but also offer access via different languages , is there any way to do that ?
Of course, it's called "extending" in the Python world. The official documentation is here. A short excerpt:
This document describes how to write
modules in C or C++ to extend the
Python interpreter with new modules.
Those modules can define new functions
but also new object types and their
methods. The document also describes
how to embed the Python interpreter in
another application, for use as an
extension language. Finally, it shows
how to compile and link extension
modules so that they can be loaded
dynamically (at run time) into the
interpreter, if the underlying
operating system supports this
feature.
An even easier way for Python would be using the ctypes standard package to run code in DLLs.
Many ways. Generically, this is often called a Foreign Function Interface. That Wikipedia page says the following about Python:
* The major dynamic languages, such as Python, Perl, Tcl, and Ruby,
all provide easy access to native code
written in C/C++ (or any other
language obeying C/C++ calling
conventions).
o Python additionally provides the Ctypes module 2, which
can load C functions from shared
libraries/DLLs on-the-fly and
translate simple data types
automatically between Python and C
semantics. For example:
import ctypes libc = ctypes.CDLL('/lib/libc.so.6' ) # under Linux/Unix
t = libc.time(None) # equivalent C code: t = time(NULL)
print t
A popular choice that supports many languages is SWIG