Develop Reference

Why does NumPy's irfft2 of rfft2 lead to a matrix with one less column when the original matrix has an odd second index?

I am confused by the following behavior of rfft2 and irfft2 in NumPy. If I start with a real matrix that is m x n where n is odd, then if I take rfft2 followed by irfft2, I end up with an m x (n-1) matrix. Since irfft2 is the inverse of rfft2, I would have expected to get back a matrix of size m x n. In addition, the values in the matrix are not what I started with -- see output below.
>>> import numpy as np
>>> x = np.ones((4, 3))
>>> ix = np.fft.rfft2(x)
>>> rx = np.fft.irfft2(ix)
>>> rx.shape
(4, 2)
>>> rx
array([[1.5, 1.5],
[1.5, 1.5],
[1.5, 1.5],
[1.5, 1.5]])
I would appreciate any feedback as to whether I am misinterpreting the results somehow or could this even possibly be a bug? I noticed that the same issue does not occur if the first index is odd and also there is no equivalent issue for rfft and irfft.
Note that I am using Python 3.8.8 with Anaconda distribution on an iMac Pro (2017) running macOS Mojave.

In order to make sure that irfft2 is in fact the inverse of rfft2, you need to let it know the exact shape of your input data when reversing the transformation.
Like so:
import numpy as np
x = np.ones((4, 3))
ix = np.fft.rfft2(x)
rx = np.fft.irfft2(ix, x.shape)
This is necessary precisely for the reason you highlight with your question: The way the transformed data (the "spectrum", ix in your example) is represented for real-valued input data (x) depends on whether the number of samples is odd or even in any of the dimensions.
The (i)rfft* family of functions are all tailored to the common use case where the input data is a series of real numbers, i.e. not complex numbers. The discrete Fourier transform of such an input is usually complex-valued, but has a special symmetry: the negative-frequency components are the complex conjugates of the corresponding positive-frequency components. That is, the spectrum contains essentially the same numbers twice, and half the spectrum already contains the information necessary to reconstruct the input data. Which makes sense: The spectrum is a series of complex numbers, which can be represented as two real numbers each, but the input data does not have that "complexity", as it is real-valued.
Then again, "half the spectrum" is not that clear a term when the length of the data (and thus of the full spectrum) may be odd or even. Mathematically, these two cases must be treated slightly differently. Which is why the length of the data is needed when reconstructing the input signal.
As the NumPy documentation of rfft notes for the one-dimensional case:
If n is even, [the last array element of the spectrum] contains the term representing both positive and negative Nyquist frequency (+fs/2 and -fs/2), and must also be purely real. If n is odd, there is no term at fs/2; [the last array element of the spectrum] contains the largest positive frequency (fs/2*(n-1)/n), and is complex in the general case.
And the documentation of irfft further explains:
The correct interpretation of the hermitian input depends on the length of the original data, as given by n. This is because each input shape could correspond to either an odd or even length signal. By default, irfft assumes an even output length which puts the last entry at the Nyquist frequency; aliasing with its symmetric counterpart. By Hermitian symmetry, the value is thus treated as purely real. To avoid losing information, the correct length of the real input must be given.
So an even-length signal is the default. Which is why you only run into this issue for odd lengths of the array dimension. The documentation of irfftn notes specifically that it is the inverse of rfftn only if called like irfftn(rfftn(x), x.shape).

FFTs in general do not handle odd-length inputs. They actually want powers of 2. When you FFT an odd-length vector of reals, you lose some information. If you try your experiment with a (4,4), you'll see that the output exactly matches the input.

Python won't show me np.exp marketshare values

I'm trying to estimate marketshares with the following formula:
c = np.exp(-Mu*a)/(np.exp(-Mu*a)+np.exp(-Mu*b))
in which a and b are 9x9 matrices with cell values that can be larger than 1000. Because the numbers are so small, Python returns NaN values. In order to enhance precision of the estimation i have already tried np.float128 but all this does is raise the error that numpy doesn't have an attribute called float128. I have also tried longdouble, again without success. Are there other ways to make Python show the actual values of the cells instead of NaN?

You have:
c = np.exp(-Mu*a)/(np.exp(-Mu*a)+np.exp(-Mu*b))
Multipying the numerator and denominator by e^(Mu*a), you get:
c = 1/(1+np.exp(Mu*(a-b)))
This is just a reformulation of the same formula.
Now, if the exp term is still too small, and you do not need a more precise result, then your c is approximately very close to 1. And if you still need to control precision, you can take log on both sides and use the Taylor expansion of log(1+x).

Conversion of float to int in Python (Hill Cipher)

I am getting different results for conversion of float to int in my Hill Cipher code (during decryption).
Code: https://github.com/krshrimali/Hill-Cipher/blob/master/hill_cipher.py
Issue: https://github.com/krshrimali/Hill-Cipher/issues/1
Code:
# create empty plain text string
plain_text = ""
# result is a matrix [[260. 574. 439.]]
# addition of 65 because inputs are uppercase letters
for i in range(dimensions):
plain_text += chr(int(result[0][i]) % 26 + 65)
Output: ABS
(the cipher text - encrypted text - was POH)
Result Matrix: (after multiplication of inverse with cipher key matrix)
[[ 260. 574. 539.]]
After conversion to int:
[260, 573, 538]
Can anyone explain why this happens and give a fix on this? Thanks.

The problem is that you're using int, which truncates toward zero.
Math with float values is inherently imprecise. If you don't understand why, the classic explanation is in What Every Computer Scientist Should Know About Floating-Point Numbers. But the short version is that every conversion and every intermediate calculation gets rounded to the nearest 52-bit fraction to the actual number. And that may mean that a calculation that would yield exactly 574 if performed with real numbers actually yields a number a tiny bit more or less than 574 when performed with floats. And if you end up with a number a tiny bit less than 574 and truncate it toward zero withint, you get 573.
In this case, what you want to do is use round instead, which rounds to the nearest integer. As long as you can be sure that your accumulated error is never as large as 0.5, that will do what you want. And, as long as you don't pick ridiculously huge key values (which would be pointless, because you don't get any more security that way), you can be sure of that.
However, there are two things worth considering here.
From a brief scan of the Hill cipher article at Wikipedia: It designed to be performed with quick pencil-and-paper operation. First, you don't need the inverse matrix, just a matrix that's inverse mod 26, which is easier to calculate, and means you stay in smaller numbers that are less likely to have this problem. And it means you can do all the math in integers, so the problem doesn't arise in the first place: create your matrix as an array with dtype=int, and there will be no rounding issues. And, as a bonus, if you do pick ridiculously huge key values, you'll get an error instead of incorrect results. (If you want to allow such values, you'd want to store Python unlimited-size int values in a dtype=object array. But if you don't need that, it just makes things slower and more complicated.)

wavfile.write: Identical arrays but only one works

I have two arrays that are identical (by design because I obtained the second one by doing an FFT and then inverse FFT of the first one). However, when I write the first one to a .wav file, I get sound-producing file as opposed to not when I do the same with the second one. I get no sound. Here is my code:
fs, data = wavfile.read(filename)
a = data.T[0]
c = fft(a)
y2 = fftp.ifft(c)
y2 = np.array([int(round(i)) for i in y2.real])
Now when I try:
sum(y2==a)==len(a)
I get True, which means the two arrays are identical. The only difference is that one has "dtype=int16":
In [322]: a
Out[322]: array([ 1, 1, 1, ..., 21, 20, 21], dtype=int16)
In [321]: y2
Out[321]: array([ 1, 1, 1, ..., 21, 20, 21])
How do I convert the second array to a format where it produces a valid .wav file as well?

That "only difference" is a huge difference.
The WAV format, by default, stores samples as signed little-endian 16-bit integers. So, when you write an array of int16 values as raw data, you get a playable WAV file (on a little-endian system, at least).
But when you write an array of int32 values, you get nonsense—each number turns into 2 samples, one of which is the high word of your data, the next of which is the low word. So, you've got your original audio samples at half speed, and interleaves with what's effectively random noise.
Or, alternatively, you can use a non-default WAV format. You didn't show enough of your code to show how you're handling this, but you can write WAV files in a variety of different formats, from 8-bit unsigned int to 32-bit float, and 32-bit signed ints are a valid format. WAV files can even handle compression (including MP3).
But less-common formats may not actually play with every tool; a lot of programs assume a WAV is 16-bit integers, and won't know what to do with anything else.
So, you're probably better off writing 16-bit integers.
Or, maybe you're already doing that—writing 32-bit int values with the right header—and maybe your player is handling them properly.
But you're writing 32-bit int values between -32768 and 32767. Which means you're only using 1/65536th of the dynamic range, so everything will be incredibly quiet. If you want to write 32-bit int values, you want to normalize them to the 32-bit int range, not the 16-bit int range.
The simplest solution to all of these problems is: convert the values back to int16 before writing them:
y3 = y2.astype(np.int16)

How do I represent a string as a number?

I need to represent a string as a number, however it is 8928313 characters long, note this string can contain more than just alphabet letters, and I have to be able to convert it back efficiently too. My current (too slow) code looks like this:
alpha = 'abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ,.?!#()+-=[]/*1234567890^*{}\'"$\\&#;|%<>:`~_'
alphaLeng = len(alpha)
def letterNumber(letters):
letters = str(letters)
cof = 1
nr = 0
for i in range(len(letters)):
nr += cof*alpha.find(letters[i])
cof *= alphaLeng
print(i,' ',len(letters))
return str(nr)

Ok, since other people are giving awful answers, I'm going to step in.
You shouldn't do this.
You shouldn't do this.
An integer and an array of characters are ultimately the same thing: bytes. You can access the values in the same way.
Most number representations cap out at 8 bytes (64-bits). You're looking at 8 MB, or 1 million times the largest integer representation. You shouldn't do this. Really.
You shouldn't do this. Your number will just be a custom, gigantic number type that would be identical under the hood.
If you really want to do this, despite all the reasons above, here's how...
Code
def lshift(a, b):
# bitwise left shift 8
return (a << (8 * b))
def string_to_int(data):
sum_ = 0
r = range(len(data)-1, -1, -1)
for a, b in zip(bytearray(data), r):
sum_ += lshift(a, b)
return sum_;
DONT DO THIS
Explanation
Characters are essentially bytes: they can be encoded in different ways, but ultimately you can treat them within a given encoding as a sequence of bytes. In order to convert them to a number, we can shift them left 8-bits for their position in the sequence, creating a unique number. r, the range value, is the position in reverse order: the 4th element needs to go left 24 bytes (3*8), etc.
After getting the range and converting our data to 8-bit integers, we can then transform the data and take the sum, giving us our unique identifier. It will be identical byte-wise (or in reverse byte-order) of the original number, but just "as a number". This is entirely futile. Don't do it.
Performance
Any performance is going to be outweighed by the fact that you're creating an identical object for no valid reason, but this solution is decently performant.
1,000 elements takes ~486 microseconds, 10,000 elements takes ~20.5 ms, while 100,000 elements takes about 1.5 seconds. It would work, but you shouldn't do it. This means it's scaled as O(n**2), which is likely due to memory overhead of reallocating the data each time the integer size gets larger. This might take ~4 hours to process all 8e6 elements (14365 seconds, calculated fitting the lower-order data to ax**2+bx+c). Remember, this is all to get the identical byte representation as the original data.
Futility
Remember, there are ~1e78 to 1e82 atoms in the entire universe, on current estimates. This is ~2^275. Your value will be able to represent 2^71426504, or about 260,000 times as many bits as you need to represent every atom in the universe. You don't need such a number. You never will.

If there are only ANSII characters. You can use ord() and chr().
built-in functions

There are several optimizations you can perform. For example, the find method requires searching through your string for the corresponding letter. A dictionary would be faster. Even faster might be (benchmark!) the chr function (if you're not too picky about the letter ordering) and the ord function to reverse the chr. But if you're not picky about ordering, it might be better if you just left-NULL-padded your string and treated it as a big binary number in memory if you don't need to display the value in any particular format.
You might get some speedup by iterating over characters instead of character indices. If you're using Python 2, a large range will be slow since a list needs to be generated (use xrange instead for Python 2); Python 3 uses a generator, so it's better.
Your print function is going to slow down output a fair bit, especially if you're outputting to a tty.
A big number library may also buy you speed-up: Handling big numbers in code

Your alpha.find() function needs to iterate through alpha on each loop.
You can probably speed things up by using a dict, as dictionary lookups are O(1):
alpha = 'abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ,.?!#()+-=[]/*1234567890^*{}\'"$\\&#;|%<>:`~_'
alpha_dict = { letter: index for index, letter in enumerate(alpha)}
print(alpha.find('$'))
# 83
print(alpha_dict['$'])
# 83

Store your strings in an array of distinct values; i.e. a string table. In your dataset, use a reference number. A reference number of n corresponds to the nth element of the string table array.