Develop Reference

unexpected results with np.hstack and np.c_ with regard to decimal points

I have a matrix (3x3) of
matrix = [[-8.51871377e-01 -5.22712676e-01 3.29641828e-02]
[ 5.22321761e-01 -8.52508933e-01 -2.02118648e-02]
[ 3.86672578e-02 -6.20947349e-10 9.99252295e-01]]
and I have a vector (3x1) of
vector = [[0.28407955]
[0.681005 ]
[4.12974119]]
When I concatenate the matrix with the vector via np.hstack or np.c_ then I get the following concatenation:
concat = [[-8.51871377e-01 -5.22712676e-01 3.29641828e-02 2.84079552e-01]
[ 5.22321761e-01 -8.52508933e-01 -2.02118648e-02 6.81005001e-01]
[ 3.86672578e-02 -6.20947349e-10 9.99252295e-01 4.12974119e+00]]
and not as expected:
concat = [[-8.51871377e-01 -5.22712676e-01 3.29641828e-02 0.28407955...]
[ 5.22321761e-01 -8.52508933e-01 -2.02118648e-02 0.681005...]
[ 3.86672578e-02 -6.20947349e-10 9.99252295e-01 4.12974119e+00]]
On the very right side of the concatenation it seems that the 0.'s are ignored in 0.28407955 and 0.681005 which means that the decimal point is shifted one place to the right and this leads to 2.84079552e-01 and 6.81005001e-01. Why is this happening and how can I get my expected results?

numpy vs pytorch precision

I've a numpy matrix a defined as:
>>> a
>>> array([[ 1.920941165 , 0.9518795607, 1.5358781432],
[-0.2418292026, 0.0851087409, -0.2760766872],
[-0.4161812806, 0.7409229185, -0.3248560283],
[-0.3439163186, 1.4052927665, -1.612850871 ],
[ 1.5810794171, 1.1820622504, 1.8063415367]])
If I typecast it to float32, it gives:
>>> a.astype(np.float32)
>>> array([[ 1.9209411 , 0.95187956, 1.5358782 ],
[-0.2418292 , 0.08510874, -0.27607667],
[-0.41618127, 0.7409229 , -0.32485604],
[-0.34391633, 1.4052927 , -1.6128509 ],
[ 1.5810794 , 1.1820623 , 1.8063415 ]], dtype=float32)
When I convert original a matrix to a tensor, I get:
>>> torch.tensor(a)
>>> tensor([[ 1.9209411650, 0.9518795607, 1.5358781432],
[-0.2418292026, 0.0851087409, -0.2760766872],
[-0.4161812806, 0.7409229185, -0.3248560283],
[-0.3439163186, 1.4052927665, -1.6128508710],
[ 1.5810794171, 1.1820622504, 1.8063415367]], dtype=torch.float64)
which looks correct as it retains original values from matrix a.
But when I convert float32-typecasted matrix to a tensor, I get different floating point numbers.
>>> torch.tensor(a.astype(np.float32))
>>> tensor([[ 1.9209411144, 0.9518795609, 1.5358781815],
[-0.2418292016, 0.0851087421, -0.2760766745],
[-0.4161812663, 0.7409229279, -0.3248560429],
[-0.3439163268, 1.4052927494, -1.6128509045],
[ 1.5810793638, 1.1820622683, 1.8063415289]])
Why can't the second tensor(tensor of type-casted matrix) be equal to the second matrix(type-casted one) provided above.

float32 has 24fraction bit (7.2 decimal point in decimal), what you see after that is not meaningful. ex: 1.920941165 (9 point).
this means if you want to retain all points you shoult represent as 64 float. however when you convert to 32, either in numpy or torch they should be same values, it is only printing is different. torch print till the number of floating that you have set, while numpy truncate only till valid points.
for example:
import numpy as np
np.set_printoptions(precision=10)
a = np.array([1.920941165],dtype=np.float32)
array([1.9209411], dtype=float32)**
t = torch.tensor(a , dtype=torch.float32)
tensor([1.9209411144])
however if you look into underlying memory of both (one uint32) are the same:
np.ndarray(1, dtype=np.uint32,buffer=a )
array([1073078630], dtype=uint32)
import ctypes
ptr = ctypes.cast(t.data_ptr(), ctypes.POINTER(ctypes.c_uint32))
ptr[0]
1073078630

python csv writer writes array in scientific notation

I have a matrix of float numbers which i'm trying to write into a csv file, however csv writer writes it in scientific notation, i would like to leave the numbers in the original notation, i tried adding "%.2f" % but that resulted in the following error:
"TypeError: only length-1 arrays can be converted to Python scalars"
for item in string_color_Values:
avg_color_values = [ float(item) for item in string_color_Values]
array1t = np.array(avg_color_values)
np.savetxt("test.csv", array1t, delimiter=",")
original:
[ 0.5258 1. ]
[ 0.528 1. ]
[ 0.5486 1. ]
[ 0.5545 1. ]
[ 0.732 1. ]
[ 0.7872 1. ]
[ 1. 1. ]]
Csv which i obtain:
1.2270000000035e-01,1.0000000000e+00
2.7639999999790e-01,1.0000000000e+00
etc..

You need to have fmt="%g" in the call to np.savetxt. That will write the numbers without trailing zeroes. Alternatively if you want "cleanly" formatted nubmers you could use e.g. fmt="%.4f" (or any other format that takes your fancy).
import numpy as np
avg_color_Values = np.random.random((10,3))
array1t = np.array(avg_color_Values)
np.savetxt("test.csv", array1t, delimiter=",", fmt = "%g")
By default np.savetxt defaults to a format of %.18e, which will allows float64s to be written without loss of precicion.

Using numpy's function loadtxt() to read a two column file then using polyfit() to return an array

afile is a given file, and z the degree of the polynomial. Been breaking my head over this for a while, frustrating how I'm basically given zero instructions on how to proceed.
This is what I thought it should be like:
import numpy as np
def newfile(afile,z):
x,y = np.loadtxt(afile)
d= np.polyfit(x,y,z)
return d
I have attempted to do it as
data = np.loadtxt(afile)
x = data[0:]
by printing "data" I'm given this format:
[[ 2. 888.8425]
[ 6. 888.975 ]
[ 14. 888.1026]
[ 17. 888.2071]
[ 23. 886.0479]
[ 26. 883.3316]
[ 48. 877.04 ]
[ 99. 854.3665]]
By printing "x" in this case just gives me the whole list (I'm thinking the issue lies in the lack of comma). In this case I'd want x to be an array of the left column.

I suppose you are getting an error when unpacking in this statement:
x,y = np.loadtxt(afile)
you should replace it for this:
x, y = zip(*np.loadtxt(afile))
the rest should work

How to force a ndarray show in normal way instead of scientific notation?

I'm trying to print a ndarray on the screen. But python always shows it in scientific notation, which I don't like. For a scalar we can use
>>> print '%2.4f' %(7.47212470e-01)
0.7472
But how to do that for a numpy.ndarray like this :
[[ 7.47212470e-01 3.71730070e-01 1.16736538e-01 1.22172891e-02]
[ 2.79279640e+00 1.31147152e+00 7.43946656e-02 3.08162255e-02]
[ 6.93657970e+00 3.14008688e+00 1.02851599e-01 3.96611266e-02]
[ 8.49295040e+00 3.94730094e+00 8.99398479e-02 7.60969188e-02]
[ 2.01849250e+01 8.62584092e+00 8.75722302e-02 6.17109672e-02]
[ 2.22570710e+01 1.00291292e+01 1.20918359e-01 1.07250131e-01]
[ 2.82496660e+01 1.27882133e+01 1.08438172e-01 1.58723714e-01]
[ 5.89170270e+01 2.55268510e+01 1.31990966e-01 1.61599514e-01]]
The method .astype(float) does not change the result, and .round(4) returns:
[[ 7.47200000e-01 3.71700000e-01 1.16700000e-01 1.22000000e-02]
[ 2.79280000e+00 1.31150000e+00 7.44000000e-02 3.08000000e-02]
[ 6.93660000e+00 3.14010000e+00 1.02900000e-01 3.97000000e-02]
[ 8.49300000e+00 3.94730000e+00 8.99000000e-02 7.61000000e-02]
[ 2.01849000e+01 8.62580000e+00 8.76000000e-02 6.17000000e-02]
[ 2.22571000e+01 1.00291000e+01 1.20900000e-01 1.07300000e-01]
[ 2.82497000e+01 1.27882000e+01 1.08400000e-01 1.58700000e-01]
[ 5.89170000e+01 2.55269000e+01 1.32000000e-01 1.61600000e-01]]
I just simply want the 0.7472 0.3717 etc.

The numpy.set_string_function function can be used to change the string representation of arrays.
You can also use numpy.set_print_options to change the precision used by default and turn off reporting of small numbers in scientific notation.
From the examples for set_print_options:
>>> np.set_printoptions(precision=4)
>>> print np.array([1.123456789])
[ 1.1235]

I don't know about the numpy arrays but I was simply facing the same problem while doing a project in Python.
Take a look at the Decimal class provided http://docs.python.org/library/decimal.html .
I don't know if it's provided in numpy though.