Now I am trying to rewrite the fortran code to python script.
In original fortran code, it declares real number as:
real a(n),b(n),D(64)
Here how can I convert this D(64) in python code?
a(n), b(n) are values from the data which I used, but D(64) is not.
I need to put this into sub-module fortran code which I wrapped with f2py.
That sub-module code looks like below. M is just a integer which will be define in main code.
subroutine multires (a,b,M, D)
implicit none
real a(*),b(*),D(*)
.
.
if(nw.gt.1) D(ms+1)=(sumab/nw)
If you code is looking for good performance, stay away from python lists. Instead you want to use numpy arrays.
import numpy as np
D = np.zeros((64), np.float32)
This will construct a numpy ndarray with 64 elements of 32 bit reals initialized to 0. Using these kind of arrays rather than lists can greatly improve the performance of your python code over lists. They also give you finer control over the typing for interoperability and you incur less overhead, especially if you get into cython.
Python is dynamically typed, so you have not to declare variables with their type. A Fortran array can be translated in a Python list, but as you should not want to dynamically add elements to the list, you should initialize it to its size. So Fortran real D(64) could become in Python:
D = [ 0. for i in range(64) ]
(declares D to be a list and initializes it with 64 0. values)
Of course, if you can use numpy and not just plain Python, you could use numpy.array type. Among other qualities (efficiency, type control, ...), it can be declared with F (for Fortran) order, meaning first-index varies the fastest, as Fortran programmers are used to.
Related
Say I have a large NumPy array of dtype int32
import numpy as np
N = 1000 # (large) number of elements
a = np.random.randint(0, 100, N, dtype=np.int32)
but now I want the data to be uint32. I could do
b = a.astype(np.uint32)
or even
b = a.astype(np.uint32, copy=False)
but in both cases b is a copy of a, whereas I want to simply reinterpret the data in a as being uint32, as to not duplicate the memory. Similarly, using np.asarray() does not help.
What does work is
a.dtpye = np.uint32
which simply changes the dtype without altering the data at all. Here's a striking example:
import numpy as np
a = np.array([-1, 0, 1, 2], dtype=np.int32)
print(a)
a.dtype = np.uint32
print(a) # shows "overflow", which is what I want
My questions are about the solution of simply overwriting the dtype of the array:
Is this legitimate? Can you point me to where this feature is documented?
Does it in fact leave the data of the array untouched, i.e. no duplication of the data?
What if I want two arrays a and b sharing the same data, but view it as different dtypes? I've found the following to work, but again I'm concerned if this is really OK to do:
import numpy as np
a = np.array([0, 1, 2, 3], dtype=np.int32)
b = a.view(np.uint32)
print(a) # [0 1 2 3]
print(b) # [0 1 2 3]
a[0] = -1
print(a) # [-1 1 2 3]
print(b) # [4294967295 1 2 3]
Though this seems to work, I find it weird that the underlying data of the two arrays does not seem to be located the same place in memory:
print(a.data)
print(b.data)
Actually, it seems that the above gives different results each time it is run, so I don't understand what's going on there at all.
This can be extended to other dtypes, the most extreme of which is probably mixing 32 and 64 bit floats:
import numpy as np
a = np.array([0, 1, 2, np.pi], dtype=np.float32)
b = a.view(np.float64)
print(a) # [0. 1. 2. 3.1415927]
print(b) # [0.0078125 50.12387848]
b[0] = 8
print(a) # [0. 2.5 2. 3.1415927]
print(b) # [8. 50.12387848]
Again, is this condoned, if the obtained behaviour is really what I'm after?
Is this legitimate? Can you point me to where this feature is documented?
This is legitimate. However, using np.view (which is equivalent) is better since it is compatible with a static analysers (so it is somehow safer). Indeed, the documentation states:
It’s possible to mutate the dtype of an array at runtime. [...]
This sort of mutation is not allowed by the types. Users who want to write statically typed code should instead use the numpy.ndarray.view method to create a view of the array with a different dtype.
Does it in fact leave the data of the array untouched, i.e. no duplication of the data?
Yes. Since the array is still a view on the same internal memory buffer (a basic byte array). Numpy will just reinterpret it differently (this is directly done the C code of each Numpy computing function).
What if I want two arrays a and b sharing the same data, but view it as different dtypes? [...]
np.view can be used in this case as you did in your example. However, the result is platform dependent. Indeed, Numpy just reinterpret bytes of memory and theoretically the representation of negative numbers can change from one machine to another. Hopefully, nowadays, all mainstream modern processors use use the two's complement (source). This means that a np.in32 value like -1 will be reinterpreted as 2**32-1 = 4294967295 with a view of type np.uint32. Positive signed values are unchanged. As long as you are aware of this, this is fine and the behaviour is predictable.
This can be extended to other dtypes, the most extreme of which is probably mixing 32 and 64 bit floats.
Well, put it shortly, this is really like playing fire. In this case this certainly unsafe although it may work on your specific machine. Let us venturing into troubled waters.
First of all, the documentation of np.view states:
The behavior of the view cannot be predicted just from the superficial appearance of a. It also depends on exactly how a is stored in memory. Therefore if a is C-ordered versus fortran-ordered, versus defined as a slice or transpose, etc., the view may give different results.
The thing is Numpy reinterpret the pointer using a C code. Thus, AFAIK, the strict aliasing rule applies. This means that reinterpreting a np.float32 value to a np.float64 cause an undefined behaviour. One reason is that the alignment requirements are not the same for np.float32 (typically 4) and np.float32 (typically 8) and so reading an unaligned np.float64 value from memory can cause a crash on some architecture (eg. POWER) although x86-64 processors support this. Another reason comes from the compiler which can over-optimize the code due to the strict aliasing rule by making wrong assumptions in your case (like a np.float32 value and a np.float64 value cannot overlap in memory so the modification of the view should not change the original array). However, since Numpy is called from CPython and no function calls are inlined from the interpreter (probably not with Cython), this last point should not be a problem (it may be the case be if you use Numba or any JIT though). Note that this is safe to get an np.uint8 view of a np.float32 since it does not break the strict aliasing rule (and the alignment is Ok). This could be useful to efficiently serialize Numpy arrays. The opposite operation is not safe (especially due to the alignment).
Update about last section: a deeper analysis from the Numpy code show that some part of the code like type-conversion functions perform a safe type punning using the memmove C call, while some other functions like all basic unary operators or binary ones do not appear to do a proper type punning yet! Moreover, such feature is barely tested by users and tricky corner cases are likely to cause weird bugs (especially if you read and write in two views of the same array). Thus, use it at your own risk.
I would like to use something like a structarray in cython, and I would like this structarray as easily accessible in python as in cython.
Based on a whim I used a recarray using a dtype that looks like the struct that I would like to use. Curiously, it just works and allows me to use a c structarray that, over the hood ;), is a numpy recarray for the python user.
Here is my example
# This is a "structarray in cython with numpy recarrays" testfile
import numpy as np
cimport numpy as np
# My structarray has nodes with fields x and y
# This also works without packed, but I have seen packed used in other places where people asked similar questions
# I assume that for two doubles that is equivalent but is necessary for in8s in between
cdef packed struct node:
double x
double y
# I suppose that would be the equivalent numpy dtype?
# Note: During compilation it warns me about double to float downcasts, but I do not see where
nodetype = [('x' , np.float64),('y', np.float64)]
def fun():
# Make 10 element recarray
# (Just looked it up. A point where 1-based indexing would save a look in the docs)
mynode1 = np.recarray(10,dtype=nodetype)
# Recarray with cdef struct
mynode1 = np.recarray(10,dtype=nodetype)
# Fill it with non-garbage somewhere
mynode1[2].x=1.0
mynode1[2].y=2.0
# Brave: give recarray element to a c function assuming its equivalent to the struct
ny = cfuny(mynode1[2])
assert ny==2.0 # works!
# Test memoryview, assuming type node
cdef node [:] nview = mynode1
ny = cfunyv(nview,2)
assert ny==2.0 # works!
# This sets the numpy recarray value with a c function the gts a memoryview
cfunyv_set(nview,5,9.0)
assert mynode1[5].y==9.0 # alsow works!
return 0
# return node element y from c struct node
cdef double cfuny(node n):
return n.y
# give element i from memoryview of recarray to c function expecting a c struct
cdef double cfunyv(node [:] n, int i):
return cfuny(n[i])
# write into recarray with a function expecting a memoryview with type node
cdef int cfunyv_set(node [:] n,int i,double val):
n[i].y = val
return 0
Of course I am not the first to try this.
Here for example the same thing is done, and it even states that this usage would be part of the manual here, but I cannot find this on the page. I suspect it was there at some point. There are also several discussions involving the use of strings in such a custom type (e.g. here), and from the answers I gather that the possibility of casting a recarray on a cstruct is intended behaviour, as the discussion talks about incorporating a regression test about the given example and having fixed the string error at some point.
My question
I could not find any documentation that states that this should work besides forum answers. Can someone show me where that is documented?
And, for some additional curiosity
Will this likely break at any point during the development of numpy or cython?
From the other forum entries on the subject it seems that packed is necessary for this to work once more interesting datatypes are part of the struct. I am not a compiler expert and have never used structure packing myself, but I suspect that whether a structure gets packed or not depends on the compiler settings. Does that mean that someone who compiles numpy without packing structures needs to compile this cython code without the packed?
This doesn't seem to be directly documented. Best reference I can give you is the typed memoryview docs here.
Rather than specific cython support for numpy structured dtypes this instead seems a consequence of support for the PEP 3118 buffer protocol. numpy exposes a Py_buffer struct for its arrays, and cython knows how to cast those into structs.
The packing is necessary. My understanding is x86 is aligned on itemsize byte boundaries, whereas as a numpy structured dtype is packed into the minimum space possible. Probably clearest by example:
%%cython
import numpy as np
cdef struct Thing:
char a
# 7 bytes padding, double must be 8 byte aligned
double b
thing_dtype = np.dtype([('a', np.byte), ('b', np.double)])
print('dtype size: ', thing_dtype.itemsize)
print('unpacked struct size', sizeof(Thing))
dtype size: 9
unpacked struct size 16
Just answering the final sub-question:
From the other forum entries on the subject it seems that packed is necessary for this to work once more interesting datatypes are part of the struct. I am not a compiler expert and have never used structure packing myself, but I suspect that whether a structure gets packed or not depends on the compiler settings. Does that mean that someone who compiles numpy without packing structures needs to compile this cython code without the packed?
Numpy's behaviour is decided at runtime rather than compile-time. It will calculate the minimum amount of space a structure can need and allocate blocks of that. It won't be changed by any compiler settings so should be reliable.
cdef packed struct is therefore always needed to match numpy. However, it does not generate standards compliant C code. Instead, it uses extensions to GCC, MSVC (and others). Therefore it works fine on the major C compilers that currently exist, but in principle might fail on a future compiler. It looks like it should be possible to use the C11 standard alignas to achieve the same thing in a standards compliant way, so Cython could hopefully be modified to do that if needed.
I have a python code within which I want to manipulate a list using a Matlab function and return it as a new list to python.
To test matlab.engine, I've tried the following:
import matlab.engine
eng = matlab.engine.start_matlab()
eng.cd('~/Documents/someDirWithMatlabFunctions/')
a = testFnc(2)
where testFnc.m looks like
function [list2] = testFnc(list)
for i = 1:numel(list)
list(i) = 3*list(i)
end
list2 = list;
end
When I run the python code, I get the following output:
>>> a = eng.testFnc(4)
>>> a
12L
>>> print a
12
My first question is what is 12L? Furthermore, when I try to pass a list as an argument:
>>> a = eng.testFnc([1,2,3])
Undefined function 'mtimes' for input arguments of type 'cell'.
It then references the line of the Matlab function in which the multiplication takes place, as where the error occurs.
I had anticipated that this might be a problem, as lists and matrices are different things. How can I properly pass variables to and from Matlab?
What is 12L?
Python supports arbitrary precision integers, meaning you're able to represent larger numbers than a normal 32- or 64-bit integer type. The L tells you when a literal is of this type and not a regular integer.
Note, that L only shows up in the interpreter output, it's just signifying the type. That's why it doesn't show up when you print it.
How can I properly pass variables to and from Matlab?
Straight from MathWorks documentation:
The matlab Python package provides array classes to represent arrays of MATLAB numeric types as Python variables so that MATLAB arrays can be passed between Python and MATLAB.
The documentation goes on to give lots of helpful examples for how to pass variables from MATLAB.
To pass data back to MATLAB, I recommend using numpy/scipy. This answer explains more about how to do that.
This question already has answers here:
Simulating Pointers in Python
(11 answers)
Closed 9 years ago.
I'm a physics student, mostly programming in Python and Matlab, now getting into C to boost the performance of my simulations.
Simple question: are there pointers in Python? Are there pointers in Matlab? If yes, can I use them with similar "ease" as in C?
I know that behind the high-levelness in Python and Matlab, there are lots of pointers at work. The question is: can I do something like this C code explicitly in Python or Matlab:
int *p;
int someValue = 1;
p = &someValue;
doSomethingForMe = callSomeFunction(p);
I know there's probably no point in doing that anyway (for most applications). I just want to learn.
You will commonly hear that Python has names, not "variables" in the traditional sense. It also doesn't have "pointers" since such a concept is only meaningful when working at a low level, like with C. (For this reason, you can get "pointers" if you use a C compatibility library in Python, like ctypes. But, such pointers are meant only for interoperability with C.)
Most of the functionality you would use a pointer for in C is present in some other form in Python. For example:
Returning multiple values from a function. In C, you'd do int foo(int *, int *) to return two ints (and maybe an error code). In Python, just use tuple unpacking: x, y = foo() with return x, y in foo.
Passing in big objects. Since everything is pass-by-reference anyway, you can safely pass a reference to a huge object without having it get copied.
Modifying input parameters. Most of this is possible by choosing an appropriate mutable input. For example, you can pass a list into a function and modify the list within the function. In Python, since the references are the same, the input list will appear to be modified:
def change_list(l):
l[0] = 3
my_list = [1,2,3]
change_list(my_list)
print(my_list) # prints [3, 2, 3]
MATLAB has a different concept of variable than Python does. In MATLAB, a variable is a matrix with some dimensions. Writing x = y effectively copies y into x (setting xs dimensions appropriately). MATLAB internally does optimizations to avoid actually copying data unless necessary, but for the programmer it will seem as if x and y are separate matrices.
MATLAB also doesn't have pointers unless you are dealing with a compiled MEX extension (which is written in some other language like C). Like Python, you have mechanisms for passing things around without needing pointers (multiple input and output arguments, for example).
Python you can have pointers with ctypes library apparently. Also, you can simulate the notion of pointers in Python, like in this question.
In Matlab you don't have pointers either.
I have to iterate all items in two-dimensional array of integers and change the value (according to some rule, not important).
I'm surprised how significant difference in performance is there between python runtime and C# or java runtime. Did I wrote totally wrong python code (v2.7.2)?
import numpy
a = numpy.ndarray((5000,5000), dtype = numpy.int32)
for x in numpy.nditer(a.T):
x = 123
>python -m timeit -n 2 -r 2 -s "import numpy; a = numpy.ndarray((5000,5000), dtype=numpy.int32)" "for x in numpy.nditer(a.T):" " x = 123"
2 loops, best of 2: 4.34 sec per loop
For example the C# code performs only 50ms, i.e. python is almost 100 times slower! (suppose the matrix variable is already initialized)
for (y = 0; y < 5000; y++)
for (x = 0; x < 5000; x++)
matrix[y][x] = 123;
Yep! Iterating through numpy arrays in python is slow. (Slower than iterating through a python list, as well.)
Typically, you avoid iterating through them directly.
If you can give us an example of the rule you're changing things based on, there's a good chance that it's easy to vectorize.
As a toy example:
import numpy as np
x = np.linspace(0, 8*np.pi, 100)
y = np.cos(x)
x[y > 0] = 100
However, in many cases you have to iterate, either due to the algorithm (e.g. finite difference methods) or to lessen the memory cost of temporary arrays.
In that case, have a look at Cython, Weave, or something similar.
The example you gave was presumably meant to set all items of a two-dimensional NumPy array to 123. This can be done efficiently like this:
a.fill(123)
or
a[:] = 123
Python is a much more dynamic language than C or C#. The main reason why the loop is so slow is that on every pass, the CPython interpreter is doing some extra work that wastes time: specifically, it is binding the name x with the next object from the iterator, then when it evaluates the assignment it has to look up the name x again.
As #Sven Marnach noted, you can call a method function numpy.fill() and it is fast. That function is compiled C or maybe Fortran, and it will simply loop over the addresses of the numpy.array data structure and fill in the values. Much less dynamic than Python, which is good for this simple case.
But now consider PyPy. Once you run your program under PyPy, a JIT analyzes what your code is actually doing. In this example, it notes that the name x isn't used for anything but the assignment, and it can optimize away binding the name. This example should be one that PyPy speeds up tremendously; likely PyPy will be ten times faster than plain Python (so only one-tenth as fast as C, rather than 1/100 as fast).
http://pypy.org
As I understand it, PyPy won't be working with Numpy for a while yet, so you can't just run your existing Numpy code under PyPy yet. But the day is coming.
I'm excited about PyPy. It offers the hope that we can write in a very high-level language (Python) and yet get nearly the performance of writing things in "portable assembly language" (C). For examples like this one, the Numpy might even beat the performance of naive C code, by using SIMD instructions from the CPU (SSE2, NEON, or whatever). For this example, with SIMD, you could set four integers to 123 with each loop, and that would be faster than a plain C loop. (Unless the C compiler used a SIMD optimization also! Which, come to think of it, is likely for this case. So we are back to "nearly the speed of C" rather than faster for this example. But we can imagine trickier cases that the C compiler isn't smart enough to optimize, where a future PyPy might.)
But never mind PyPy for now. If you will be working with Numpy, it is a good idea to learn all the functions like numpy.fill() that are there to speed up your code.
C++ emphasizes machine time over programmer time.
Python emphasizes programmer time over machine time.
Pypy is a python written in python, and they have the beginnings of numpy; you might try that. Pypy has a nice JIT that makes things quite fast.
You could also try cython, which allows you to translate a dialect of Python to C, and compile the C to a Python C extension module; this allows one to continue using CPython for most of your code, while still getting a bit of a speedup. However, in the one microbenchmark I've tried comparing Pypy and Cython, Pypy was quite a bit faster than Cython.
Cython uses a highly pythonish syntax, but it allows you to pretty freely intermix Python datatypes with C datatypes. If you redo your hotspots with C datatypes, it should be pretty fast. Continuing to use Python datatypes is sped up by Cython too, but not as much.
The nditer code does not assign a value to the elements of a. This doesn't affect the timings issue, but I mention it because it should not be taken as a good use of nditer.
a correct version is:
for i in np.nditer(a, op_flags=[["readwrite"]]):
i[...] = 123
The [...] is needed to retain the reference to loop value, which is an array of shape ().
There's no point in using A.T, since its the values of the base A that get changed.
I agree that the proper way of doing this assignment is a[:]=123.
If you need to do operations on a multidimensional array that depend on the value of the array but don't depend on the position inside the array then .itemset is 5 times faster than nditer for me.
So instead of doing something like
image = np.random.random_sample((200, 200,3));
with np.nditer(image, op_flags=['readwrite']) as it:
for x in it:
x[...] = x*4.5 if x<0.2 else x
You can do this
image2 = np.random.random_sample((200, 200,3));
for i in range(0,image2.size):
x = image2.item(i)
image2.itemset(i, x*4.5 if x<0.2 else x);