Develop Reference

Disable code inspection with Cython and compilation

I have a python script which I have transformed into a pyd file using Cython and Gcc. I would like it to be a black box, so that people cannot inspect it easily.
The source code after cythonization and compilation is hidden. However, it seems like you can still inspect it quite easily when you import the module in a python console and use the dir magic method. Is there a way to prevent code inspection from happening?

The first thing to emphasise is that Cython isn't really designed to help you hide your code - it's more an unintentional consequence. If Cython could get a 1% speed-up by making the source code for a compiled module completely visible then it'd probably choose to do that rather than hide it (however that's pretty unlikely... I can't see a mechanism that'd happen by).
Having some things visible by dir is pretty unavoidable. Cython is trying to make things behave like Python as far as possible and Python works by looking names up in a module dictionary. All dir does is print the keys from the module dictionary. However there are some things you can try:
use cdef functions rather than def functions. cdef functions are only not callable (or visible) from Python. They have some limitations though - you can't use *args or **kwds as arguments for example. Some sub-notes:
you will probably need one entry-point that can be called from Python.
if you need to share cdef functions between modules then you will need to create a pxd file. shared cdef functions will appear in the __pyx_capi__ special attribute of your module where their name, C signature, and possibly function pointer will be able to anyone that looks.
Turning off the binding and embedsignature compiler directives will make regular def functions a little less easily introspected.
You can tag a cdef class with the cython.internal decorator. This makes it only easily accessible from within your module. Notes:
cdef classes aren't quite a substitute for Python classes (for example, all attributes must be pre-declared at compile-time),
If you return one of these classes to Python then the user can introspect it,
The class will leave some evidence that it exists in the names automatically generated pickle functions (visible in your module). You can turn that off with the auto_pickle compiler directive.

Choose Python classes to instantiate at runtime based on either user input or on command line parameters

I am starting a new Python project that is supposed to run both sequentially and in parallel. However, because the behavior is entirely different, running in parallel would require a completely different set of classes than those used when running sequentially. But there is so much overlap between the two codes that it makes sense to have a unified code and defer the parallel/sequential behavior to a certain group of classes.
Coming from a C++ world, I would let the user set a Parallel or Serial class in the main file and use that as a template parameter to instantiate other classes at runtime. In Python there is no compilation time so I'm looking for the most Pythonic way to accomplish this. Ideally, it would be great that the code determines whether the user is running sequentially or in parallel to select the classes automatically. So if the user runs mpirun -np 4 python __main__.py the code should behave entirely different than when the user calls just python __main__.py. Somehow it makes no sense to me to have if statements to determine the type of an object at runtime, there has to be a much more elegant way to do this. In short, I would like to avoid:
if isintance(a, Parallel):
m = ParallelObject()
elif ifinstance(a, Serial):
m = SerialObject()
I've been reading about this, and it seems I can use factories (which somewhat have this conditional statement buried in the implementation). Yet, using factories for this problem is not an option because I would have to create too many factories.
In fact, it would be great if I can just "mimic" C++'s behavior here and somehow use Parallel/Serial classes to choose classes properly. Is this even possible in Python? If so, what's the most Pythonic way to do this?
Another idea would be to detect whether the user is running in parallel or sequentially and then load the appropriate module (either from a parallel or sequential folder) with the appropriate classes. For instance, I could have the user type in the main script:
from myPackage.parallel import *
or
from myPackage.serial import *
and then have the parallel or serial folders import all shared modules. This would allow me to keep all classes that differentiate parallel/serial behavior with the same names. This seems to be the best option so far, but I'm concerned about what would happen when I'm running py.test because some test files will load parallel modules and some other test files would load the serial modules. Would testing work with this setup?

You may want to check how a similar issue is solved in the stdlib: https://github.com/python/cpython/blob/master/Lib/os.py - it's not a 100% match to your own problem, nor the only possible solution FWIW, but you can safely assume this to be a rather "pythonic" solution.
wrt/ the "automagic" thing depending on execution context, if you decide to go for it, by all means make sure that 1/ both implementations can still be explicitely imported (like os.ntpath and os.posixpath) so they are truly unit-testable, and 2/ the user can still manually force the choice.
EDIT:
So if I understand it correctly, this file you points out imports modules depending on (...)
What it "depends on" is actually mostly irrelevant (in this case it's a builtin name because the target OS is known when the runtime is compiled, but this could be an environment variable, a command line argument, a value in a config file etc). The point was about both conditional import of modules with same API but different implementations while still providing direct explicit access to those modules.
So in a similar way, I could let the user type from myPackage.parallel import * and then in myPackage/init.py I could import all the required modules for the parallel calculation. Is this what you suggest?
Not exactly. I posted this as an example of conditional imports mostly, and eventually as a way to build a "bridge" module that can automagically select the appropriate implementation at runtime (on which basis it does so is up to you).
The point is that the end user should be able to either explicitely select an implementation (by explicitely importing the right submodule - serial or parallel and using it directly) OR - still explicitely - ask the system to select one or the other depending on the context.
So you'd have myPackage.serial and myPackage.parallel (just as they are now), and an additional myPackage.automagic that dynamically selects either serial or parallel. The "recommended" choice would then be to use the "automagic" module so the same code can be run either serial or parallel without the user having to care about it, but with still the ability to force using one or the other where it makes sense.
My fear is that py.test will have modules from parallel and serial while testing different files and create a mess
Why and how would this happen ? Remember that Python has no "process-global" namespace - "globals" are really "module-level" only - and that python's import is absolutely nothing like C/C++ includes.
import loads a module object (can be built directly from python source code, or from compiled C code, or even dynamically created - remember, at runtime a module is an object, instance of the module type) and binds this object (or attributes of this object) into the enclosing scope. Also, modules are garanteed (with a couple caveats, but those are to be considered as error cases) to be imported only once for a given process (and then cached) so importing the same module twice in a same process will yield the same object (IOW a module is a singleton).
All this means that given something like
# module A
def foo():
return bar(42)
def bar(x):
return x * 2
and
# module B
def foo():
return bar(33)
def bar(x):
return x / 2
It's garanteed that however you import from A and B, A.foo will ALWAYS call A.bar and NEVER call B.bar and B.foo will only ever call B.bar (unless you explicitely monkeyptach them of course but that's not the point).
Also, this means that within a module you cannot have access to the importing namespace (the module or function that's importing your module), so you cannot have a module depending on "global" names set by the importer.
To make a long story short, you really need to forget about C++ and learn how Python works, as those are wildly different languages with wildly different object models, execution models and idioms. A couple interesting reads are http://effbot.org/zone/import-confusion.htm and https://nedbatchelder.com/text/names.html
EDIT 2:
(about the 'automagic' module)
I would do that based on whether the user runs mpirun or just python. However, it seems it's not possible (see for instance this or this) in a portable way without a hack. Any ideas in that direction?
I've never ever had anything to do with mpi so I can't help with this - but if the general consensus is that there's no reliable portable way to detect this then obviously there's your answer.
This being said, simple stupid solutions are sometimes overlooked. In your case, explicitly setting an environment variable or passing a command-line switch to your main script would JustWork(tm), ie the user should for example use
SOMEFLAG=serial python main.py
vs
SOMEFLAG=parallel mpirun -np4 python main.py
or
python main.py serial
vs
mpirun -np4 python main.py parallel
(whichever works best for you needs - is the most easily portable).
This of course requires a bit more documentation and some more effort from the end-user but well...

I'm not really what you're asking here. Python classes are just (callable/instantiable) objects themselves, so you can of course select and use them conditionally. If multiple classes within multiple modules are involved, you can also make the imports conditional.
if user_says_parallel:
from myPackage.parallel import ParallelObject
ObjectClass = ParallelObject
else:
from myPackage.serial import SerialObject
ObjectClass = SerialObject
my_abstract_object = ObjectClass()
If that's very useful depends on your classes and the effort it takes to make sure they have the same API so they're compatible when replacing each other. Maybe even inheritance à la ParallelObject => SerialObject is possible, or at least a common (virtual) base class to put all the shared code. But that's just the same as in C++.

How do I make independently compiled cython packages use a shared random number generator?

I have an experimental programming language where programs are compiled to c. I've written a cython wrapper that wraps around the compiled c code and allows it to be callable from python. This lets you use compiled programs as fast low-level functions from within python. It's often the case that we want to use multiple such programs within the same python program. Then the pipeline for generating and importing each program is:
Compile the program to c using the compiler.
Compile the c code to a .so shared object using gcc.
Generate a .pyx wrapper which can access the c functions we want to use from python.
Compile the .pyx wrapper with cythonize to generate a .so.
Import the .so shared object using python's import feature.
In practice, steps 1-4 are actually merged into a single external call to make using sys, with a generated Makefile performing each of the 4 steps. This lets us call make via an external call with sys and then import the compiled program without ever leaving python.
A compiled program may have probabilistic constructs. In particular, branching decisions are governed by random numbers. To do this, calls are made to c's native
rand()
function. When a wrapper compiled program is imported in python, an import call is made to the generated .so shared object which is produced using cythonize. So far I've tried calling
srand(<long int>time(NULL)
from within the .pyx file that wraps each compiled program. As far as I can tell, each imported .so will effectively be using its own random number generator. But it's not at all clear to me from the docs whether this is the case.
Ultimately I want the different .so's to be using the same random number generator, but I have no idea how to go about this. Any guidance would be greatly appreciated. Much of the code is too long to include here, but if you'd like to see any snippets (e.g. 'how do you do x component?') I will happily oblige.
Even if all you can offer is an explanation of how calls to rand() will interact between different shared objects generated with cythonize, that might give me enough to work out a solution.
Thanks in advance!

I'm not sure it's well-defined in the C specification whether the random seed is shared between .so files or individual (that said - I haven't read the C standard, so I'm guessing slightly here). Therefore what behaviour you see may depend on the platform you're on.
The simplest thing here would be to write a small Cython module that's sole purpose is to handle random number generation:
# cy_rand.pxd
cpdef void srand(unsigned int)
cpdef int rand()
# cy_rand.pyx
from libc cimport stdlib
cpdef void srand(unsigned int seed):
stdlib.srand(seed)
cpdef int rand():
return stdlib.rand()
I've made the functions cpdef so that you can call them from Python too. If you don't care about being able to do this then just make them cdef.
You need to compile this module in the normal way. In your other modules you can just do:
cimport cy_rand
cy_rand.srand(1) # some seed
rand_val = cy_rand.rand()
This way you know that random numbers are only being generated in one .so file. This adds a small layer of indirection so will be slightly slower than just calling it directly. Therefore it might be a good idea to add helper functions to generate random numbers in bulk (for speed).
Be aware that other libraries could call srand or rand themselves, and because it's possibly global state this could affect you - this is one of the reasons why the C standard library random number generator isn't hugely robust...

Ctypes - basic explanation

I'm trying to speed up an integration (scipy.integrate.quad) using Ctypes. I have never used C and don't understand the Ctypes documentation. Could someone give a basic explanation of what Ctypes is actually doing using as few computing terms as possible. Essentially please explain it like I'm 5!
Thanks

A computer runs any program by following very simple steps, known as machine code or native code. At that level, anything is a number of a handful of widths, and there are millions of memory slots to store them in. When writing a program, a higher level of abstraction is usually desired, allowing us to name variables and subroutines, keep track of what memory holds what value, and so on. Native code itself does not reveal that information, but stored program files (whether libraries or executables) often have some clues, such as subroutine names. C source code supplies this information in declarations, and some libraries like SciPy have wrappers to preserve the information another layer up, in this case Python. Python objects always hold information on their types, unlike C variables. Ctypes permits to look up names and describe the missing type information, so native variables and subroutines can be accessed from Python. In the scipy.integrate.quad example, Ctypes is used to create a Python function object from a native subroutine named func.
>>> import ctypes
>>> lib = ctypes.CDLL('/home/.../testlib.*') #use absolute path
>>> lib.func.restype = ctypes.c_double
>>> lib.func.argtypes = (ctypes.c_int,ctypes.c_double)
In C terms, this function is declared as extern double func(int, double);. In general, native routines are faster than Python ones, because Python has to figure out what to do with each operation by the objects it handles, while that information is statically determined in C. A middle ground can be reached with just in time compilers, of which PyPy is a good example.

What happens when you import a package?

For efficiency's sake I am trying to figure out how python works with its heap of objects (and system of namespaces, but it is more or less clear). So, basically, I am trying to understand when objects are loaded into the heap, how many of them are there, how long they live etc.
And my question is when I work with a package and import something from it:
from pypackage import pymodule
what objects get loaded into the memory (into the object heap of the python interpreter)? And more generally: what happens? :)
I guess the above example does something like:
some object of the package pypackage was created in the memory (which contains some information about the package but not too much), the module pymodule was loaded into the memory and its reference was created in the local name space. The important thing here is: no other modules of the pypackage (or other objects) were created in the memory, unless it is stated explicitly (in the module itself, or somewhere in the package initialization tricks and hooks, which I am not familiar with). At the end the only one big thing in the memory is pymodule (i.e. all the objects that were created when the module was imported). Is it so? I would appreciate if someone clarified this matter a little bit. Maybe you could advice some useful article about it? (documentation covers more particular things)
I have found the following to the same question about the modules import:
When Python imports a module, it first checks the module registry (sys.modules) to see if the module is already imported. If that’s the case, Python uses the existing module object as is.
Otherwise, Python does something like this:
Create a new, empty module object (this is essentially a dictionary)
Insert that module object in the sys.modules dictionary
Load the module code object (if necessary, compile the module first)
Execute the module code object in the new module’s namespace. All variables assigned by the code will be available via the module object.
And would be grateful for the same kind of explanation about packages.
By the way, with packages a module name is added into the sys.modules oddly:
>>> import sys
>>> from pypacket import pymodule
>>> "pymodule" in sys.modules.keys()
False
>>> "pypacket" in sys.modules.keys()
True
And also there is a practical question concerning the same matter.
When I build a set of tools, which might be used in different processes and programs. And I put them in modules. I have no choice but to load a full module even when all I want is to use only one function declared there. As I see one can make this problem less painful by making small modules and putting them into a package (if a package doesn't load all of its modules when you import only one of them).
Is there a better way to make such libraries in Python? (With the mere functions, which don't have any dependencies within their module.) Is it possible with C-extensions?
PS sorry for such a long question.

You have a few different questions here. . .
About importing packages
When you import a package, the sequence of steps is the same as when you import a module. The only difference is that the packages's code (i.e., the code that creates the "module code object") is the code of the package's __init__.py.
So yes, the sub-modules of the package are not loaded unless the __init__.py does so explicitly. If you do from package import module, only module is loaded, unless of course it imports other modules from the package.
sys.modules names of modules loaded from packages
When you import a module from a package, the name is that is added to sys.modules is the "qualified name" that specifies the module name together with the dot-separated names of any packages you imported it from. So if you do from package.subpackage import mod, what is added to sys.modules is "package.subpackage.mod".
Importing only part of a module
It is usually not a big concern to have to import the whole module instead of just one function. You say it is "painful" but in practice it almost never is.
If, as you say, the functions have no external dependencies, then they are just pure Python and loading them will not take much time. Usually, if importing a module takes a long time, it's because it loads other modules, which means it does have external dependencies and you have to load the whole thing.
If your module has expensive operations that happen on module import (i.e., they are global module-level code and not inside a function), but aren't essential for use of all functions in the module, then you could, if you like, redesign your module to defer that loading until later. That is, if your module does something like:
def simpleFunction():
pass
# open files, read huge amounts of data, do slow stuff here
you can change it to
def simpleFunction():
pass
def loadData():
# open files, read huge amounts of data, do slow stuff here
and then tell people "call someModule.loadData() when you want to load the data". Or, as you suggested, you could put the expensive parts of the module into their own separate module within a package.
I've never found it to be the case that importing a module caused a meaningful performance impact unless the module was already large enough that it could reasonably be broken down into smaller modules. Making tons of tiny modules that each contain one function is unlikely to gain you anything except maintenance headaches from having to keep track of all those files. Do you actually have a specific situation where this makes a difference for you?
Also, regarding your last point, as far as I'm aware, the same all-or-nothing load strategy applies to C extension modules as for pure Python modules. Obviously, just like with Python modules, you could split things up into smaller extension modules, but you can't do from someExtensionModule import someFunction without also running the rest of the code that was packaged as part of that extension module.

The approximate sequence of steps that occurs when a module is imported is as follows:
Python tries to locate the module in sys.modules and does nothing else if it is found. Packages are keyed by their full name, so while pymodule is missing from sys.modules, pypacket.pymodule will be there (and can be obtained as sys.modules["pypacket.pymodule"].
Python locates the file that implements the module. If the module is part of the package, as determined by the x.y syntax, it will look for directories named x that contain both an __init__.py and y.py (or further subpackages). The bottom-most file located will be either a .py file, a .pyc file, or a .so/.pyd file. If no file that fits the module is found, an ImportError will be raised.
An empty module object is created, and the code in the module is executed with the module's __dict__ as the execution namespace.1
The module object is placed in sys.modules, and injected into the importer's namespace.
Step 3 is the point at which "objects get loaded into memory": the objects in question are the module object, and the contents of the namespace contained in its __dict__. This dict typically contains top-level functions and classes created as a side effect of executing all the def, class, and other top-level statements normally contained in each module.
Note that the above only desribes the default implementation of import. There is a number of ways one can customize import behavior, for example by overriding the __import__ built-in or by implementing import hooks.
1 If the module file is a .py source file, it will be compiled into memory first, and the code objects resulting from the compilation will be executed. If it is a .pyc, the code objects will be obtained by deserializing the file contents. If the module is a .so or a .pyd shared library, it will be loaded using the operating system's shared-library loading facility, and the init<module> C function will be invoked to initialize the module.