Develop Reference

Ensure that calculations are done 64 bits (or at least warn of overflow)

I am using python and NumPy. I had the following basic quantity to compute:
(QL * (7**k))**2
Where
QL = 200003
k = 4
What puzzled me is that it returned a wrong (negative) number, which doesn't make sense. Then I realised after looking on the internet that the problem was because k was a 32-bit numpy integer.
A minimal working example can be the following:
QL = 200000
k = np.arange(10)[4]
print((QL * 7**k)**2)
This returns 406556672 instead of the correct answer 230592040000000000. The number is not negative here, but the same problem still occurs.
My question is:
How can I make sure that all the numbers used in my code are of the biggest possible integer size?
I don't want to explicitly specify it for each number that I create.
How can I at least force python to warn me when such things happen?

When you write QL = 200003; k = 4 in Python, the numbers are interpreted as ints. By default, if you were to convert these into numpy arrays or scalars, you would end up with whatever the default integer type is on your system.
Here is an example using one-element arrays:
QL = np.array([200003])
k = np.array([4])
On my system, I find that the dtype of both arrays is int32. You can change that by selecting your preferred dtype:
QL = np.array([200003], dtype=np.int64)
k = np.array([4], dtype=np.int64)
If you don't have access to the arrays at creation time, you can always convert them:
QL = QL.astype(np.int64)
k = k.astype(int64)
An option that is worth considering for integer math is skipping numpy altogether and using Python's infinite precision integers. If one of the numbers is a numpy scalar or one-element array, you can retrieve the corresponding Python object using the item method:
QL = QL.item()
k = k.item()
Numpy should raise at least a warning for overflow, but apparently this fails for some operations: https://github.com/numpy/numpy/issues/8987
TL;DR
In your case, k is a numpy scalar of type int32. You can do either one of the following:
For a numpy 64-bit result:
k = np.int64(k)
For an infinite-precision Python result:
k = k.item()
If you don't want to cast each k explicitly, you can create the range using the correct type:
k = np.arange(10, dtype=np.int64)[4]
There is no reliable way to set the default integer type for all new arrays without specifying it explicitly.

Error with numpy array calculations using int dtype (it fails to cast dtype to 64 bit automatically when needed)

I'm encountering a problem with incorrect numpy calculations when the inputs to a calculation are a numpy array with a 32-bit integer data type, but the outputs include larger numbers that require 64-bit representation.
Here's a minimal working example:
arr = np.ones(5, dtype=int) * (2**24 + 300) # arr.dtype defaults to 'int32'
# Following comment from #hpaulj I changed the first line, which was originally:
# arr = np.zeros(5, dtype=int)
# arr[:] = 2**24 + 300
single_value_calc = 2**8 * (2**24 + 300)
numpy_calc = 2**8 * arr
print(single_value_calc)
print(numpy_calc[0])
# RESULTS
4295044096
76800
The desired output is that the numpy array contains the correct value of 4295044096, which requires 64 bits to represent it. i.e. I would have expected numpy arrays to automatically upcast from int32 to int64 when the output requires it, rather maintaining a 32-bit output and wrapping back to 0 after the value of 2^32 is exceeded.
Of course, I can fix the problem manually by forcing int64 representation:
numpy_calc2 = 2**8 * arr.astype('int64')
but this is undesirable for general code, since the output will only need 64-bit representation (i.e. to hold large numbers) in some cases and not all. In my use case, performance is critical so forcing upcasting every time would be costly.
Is this the intended behaviour of numpy arrays? And if so, is there a clean, performant solution please?

Type casting and promotion in numpy is fairly complicated and occasionally surprising. This recent unofficial write-up by Sebastian Berg explains some of the nuances of the subject (mostly concentrating on scalars and 0d arrays).
Quoting from this document:
Python Integers and Floats
Note that python integers are handled exactly like numpy ones. They are, however, special in that they do not have a dtype associated with them explicitly. Value based logic, as described here, seems useful for python integers and floats to allow:
arr = np.arange(10, dtype=np.int8)
arr += 1
# or:
res = arr + 1
res.dtype == np.int8
which ensures that no upcast (for example with higher memory usage) occurs.
(emphasis mine.)
See also Allan Haldane's gist suggesting C-style type coercion, linked from the previous document:
Currently, when two dtypes are involved in a binary operation numpy's principle is that "the output dtype's range covers the range of both input dtypes", and when a single dtype is involved there is never any cast.
(emphasis again mine.)
So my understanding is that the promotion rules for numpy scalars and arrays differ, primarily because it's not feasible to check every element inside an array to determine whether casting can be done safely. Again from the former document:
Scalar based rules
Unlike arrays, where inspection of all values is not feasable, for scalars (and 0-D arrays) the value is inspected.
This would mean that you can either use np.int64 from the start to be safe (and if you're on linux then dtype=int will actually do this on its own), or check the maximum value of your arrays before suspect operations and determine if you have to promote the dtype yourself, on a case-by-case basis. I understand that this might not be feasible if you are doing a lot of calculations, but I don't believe there is a way around this considering numpy's current type promotion rules.

Numpy Vectorization - Weird issue

I am performing some vectorized calculation using numpy. I was investigating a bug I am having and I ended with this line:
(vertices[:,:,:,0]+vertices[:,:,:,1]*256)*4
The result was expected to be 100728 for the index vertices[0,0,17], however, I am getting 35192.
When I tried to change it into 4.0 instead of 4, I ended getting the correct value of 100728 and thus fixing my bug.
I would like to understand why the floating point matters here especially that I am using python 3.7 and it is multiplication, not even division.
Extra information:
vertices.shape=(203759, 12, 32, 3)
python==3.7
numpy==1.16.1
Edit 1:
vertices type is "numpy.uint8"
vertices[0, 0, 17] => [94, 98, 63]

The issue here is that you are using too small integers, and the number overflows and wraps around because numpy uses fixed width integers rather than infinite precision like python int's. Numpy will "promote" the type of a result based on the inputs, but it won't promote the result based on whether an overflow happens or not (it's done before the actual calculation.
In this case when you multiply: vertices[:,:,:,1]*256 (I shall call this A), 256 cannot be held in a uint8, so it goes to the next higher type: uint16 this allows the result of the multiplication to hold the correct value in this case, because the maximum possible value of any element in verticies is 255, so the largest value possible is 255*256, which fits just fine in a 16 bit uint.
Then you add vertices[:,:,:,0] + A (I shall call this B). if the largest value of A was 255*256, and the largest value of vertices[:,:,:,0] is 255 (again the largest value of a uint8), the largest sum of the two is equal to 216-1 (the largest value you can hold in a 16 bit unsigned int). This is still fine right up until you go for your last multiplication.
When you get to B * 4, numpy again has to decide what the return type should be. The integer 4 easily fits in a uint16, so numpy does not promote the type higher still to a uint32 or uint64 because it does not preemptively avoid overflows as previously described. This results in any multiplication products greater than 216-1 being returned as modulo 216.
If you instead use a floating point number (4. or 4.0), numpy sees this as a "higher" value type that cannot fit inside a uint16, so it promotes the result to floating point, which can accomodate much higher numbers without overflowing.
If you don't want to change the entire array: verticies to a larger dtype, you could simply take the result B and convert that before you multiply by 4 as such: B.astype(np.uint64) * 4. This will allow you to hold much larger values without overflowing (though it does not actually eliminate the problem if the value is larger than 4 ever).

Python error:only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

I understand that there are other questions with the same error message however I have looked at these and don't understand how it applies to my current situation. So I'm creating a matrix u=np.zeros(shape=(nt,nx)) and then I have also two arrays time=nt*[0] and middles=nx*[0]
this is the full code I'm trying to implement in regards to plotting the diffusion equation:
import numpy as np
import matplotlib.pyplot as plt
import math
D=1 #diffusion constant set equal to 1
C=1 #creation rate of neutrons, set equal to 1
L=math.pi
nx=101 #number of steps in x
nt=10002 #number of timesteps
dx=L/(nx-1) #step in x
dt=0.0001 # time step
Z=(D*dt)/(dx*dx) #constant for diffusion term
Z1=C*dt #constant for u term
x1=np.arange(-math.pi/2+0.03079992797, 0, 0.03079992797)
y=np.arange(0.06159985595,math.pi/2, 0.03079992797)
z = np.hstack((x1, y))
u=np.zeros(shape=(nt,nx))
time=nt*[0]
middles=nx*[0]
u[50,0]=1/dx #setting our delta function
for j in range(0,nt-1):
for i in range(2,nx-1):
u[j+1,i]=Z*(u[j,i+1]-2*u[j,i]+u[j,i-1])+Z1*u[j,i]+u[j,i]
u[j,1]=0
u[j,nx-1]=0
time[j]=dt*j
middles[j]=u[j,((nx-1)/2)]
if i==50 or i==100 or i==250 or i==500 or i==1000 or i==10000:
plt.plot(time,middles)
plt.title('Numerical Solution of the Diffusion Equation')
plt.xlabel('time')
plt.ylabel('middles')
plt.show()
however I keep getting this error message seen in the title only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices
The error message is in regards to the middles[j]=u[j,((nx-1)/2)]
line
I'm trying to convert this code from Matlabe if that explains things somewhat

You often get this error when you accidentally create a float with one of your calculations of an index value.
In this case:
middles[j] = u[j, ((nx-1)/2)]
...will create a float when (nx-1) is odd. So you could try:
middles[j] = u[j, int(np.round(((nx-1)/2), 0))]
(I'm using np.round here, which is overkill perhaps, but if you start dividing by a number other than 2, then this approach makes much more sense, as it will round up OR down. int() will always floor it.)
Python vs. Matlab
Two things to be aware of:
Matlab defaults to matrix multiplication, whereas NumPy defaults to element-wise multiplication - but that isn't an issue here.
Matlab uses 1-indexing whereas Python (and therefore NumPy) uses 0-indexing. Any code ported from R or Matlab will need indices shifting down by 1.

Never mind ... I see it: you're using a float as an index in your final line of code:
u[j,((nx-1)/2)]
Convert the second index to int:
u[j, int((nx-1)/2)]

Other reason for this type of error is when you store indices in set and try your ND array/dataframe through this

Sum of consecutive integers in numpy is incorrect

In summing the first 100,000,000 positive integers using the following:
import numpy as np
np.arange(1,100000001).sum()
I return: 987459712, which does not match the formula: N(N+1)/2 for N=100000000. Namely, the formula returns 5000000050000000.
Before posting, I wrote the following, which returns True:
np.arange(1,65536).sum() == ((65535+1) * 65535)/2
However, the number 65536 seems to be a critical point, as
np.arange(1,65537).sum() == ((65536+1) * 65536)/2
returns False.
For integers greater than 65536 the code returns False, whereas integers below this threshold return True.
Could someone explain either what I've done wrong in calculating the sum, or what is going on with the code?

Seems like numpy sometimes has a hard time guessing the correct datatype.
On my system, Win 10 64-bit, Python 3.4.4, numpy 1.13.1:
>> np.arange(1, 100000001).sum()
987459712
>> np.arange(1, 100000001).dtype
dtype('int32')
But, if we "help" numpy it gets the correct result:
>> np.arange(1, 100000001, dtype=np.int64).sum()
500000005000000
The wrong result is obviously due to 32-bit integer overflowing.

It isn't really that numpy has a hard time guessing things, it's just that the default int type is the same as C long type:
int_: Default integer type (same as C long; normally either int64 or int32)
For windows systems, longs are 32bit, even on 64bit builds (see here for more) so that's what's used by default, int32.
As DeepSpace suggested, setting dtype to int64 does the trick. This can be done either in arange or in the sum method.
Additionally, you wrote:
Before posting, I wrote the following, which returns True:
np.arange(1,65536).sum() == ((65535+1) * 65535)/2
However, the number 65536 seems to be a critical point, as
np.arange(1,65537).sum() == ((65536+1) * 65536)/2
returns False.
and this is explained by the fact that the second sum exceeds int32's max value while the first doesn't:
>> np.arange(1,65536).sum() < np.iinfo(np.int32).max
True
>>> np.arange(1,65537).sum() < np.iinfo(np.int32).max
False
of course the Python calculation is correct due to Python 3's arbitrary precision ints.
This is why many of us weren't able to reproduce. On most Unixes the default int size for 64bit machines is int64 (since the C long is 64bits) therefore the sum of those ints was equal to the expected value.