Develop Reference

How to save and use a Tensorflow dataset using the Experimental save and load mehods?

I wrote two python files create_save.py and load_use.py as shown below.
create_save.py is running good and saving tf dataset.
But load_use.py is giving errors shown below.
How to fix load_use.py errors please?
create_save.py
import os
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.data.experimental import save as tf_save
ds_dir = os.path.join('./', "save_load_tfds_dir")
ds = tf.data.Dataset.range(12)
tf_save(ds, ds_dir)
load_use.py
import os
import numpy as np
import pandas as pd
import tensorflow as tf
ds_dir = os.path.join('./', "save_load_tfds_dir")
new_ds = tf.data.experimental.load(ds_dir)
for elem in new_ds:
print(elem)
Above load_use.py program is giving following errors:
TypeError Traceback (most recent call last)
in
----> 1 new_ds = tf.data.experimental.load(ds_dir)
TypeError: load() missing 1 required positional argument:
'element_spec'
How to fix above error?

To load a previously saved dataset, you need to specify element_spec argument -- a type signature of the elements of the saved dataset, which can be obtained via tf.data.Dataset.element_spec. This requirement exists so that shape inference of the loaded dataset does not need to perform I/O.
import tempfile
path = os.path.join(tempfile.gettempdir(), "saved_data")
# Save a dataset
dataset = tf.data.Dataset.range(2)
tf.data.experimental.save(dataset, path)
new_dataset = tf.data.experimental.load(path,
tf.TensorSpec(shape=(), dtype=tf.int64)) # element_spec arg
for elem in new_dataset:
print(elem)
When you are creating a tf.data.Dataset, it has the attribute element_spec which is what you should be using while loading your saved file. (Refer: Dataset doc).
In the above example, the element_spec argument in the load() method is given as per the type specification of the data being saved in the code.
TF Data Load Documentation

pybinding cannot import name allow_rasterization

I just install pybinding and I'm trying to run the first example that is proposed in the documentation of this library.
import pybinding as pb
import numpy as np
import matplotlib.pyplot as plt
import pybinding as pb
d = 0.2 # [nm] unit cell length
t = 1 # [eV] hopping energy
# create a simple 2D lattice with vectors a1 and a2
lattice = pb.Lattice(a1=[d, 0], a2=[0, d])
lattice.add_sublattices(
('A', [0, 0]) # add an atom called 'A' at position [0, 0]
)
lattice.add_hoppings(
# (relative_index, from_sublattice, to_sublattice, energy)
([0, 1], 'A', 'A', t),
([1, 0], 'A', 'A', t)
)
lattice.plot()
plt.show()
I have already installed what is required in the documentation (for Windows OS) and the sicrpt run pretty well until it has to do the lattice.plot() throwing the following error
Traceback (most recent call last):
File "prueba.py", line 25, in <module>
lattice.plot()
File "C:\xampp7\Python\lib\site-packages\pybinding\lattice.py", line 463, in plot
axes=axes))
File "C:\xampp7\Python\lib\site-packages\pybinding\results.py", line 598, in plot
plot_sites(self.positions, self.sublattices, **props['site'])
File "C:\xampp7\Python\lib\site-packages\pybinding\system.py", line 285, in plot_sites
from pybinding.support.collections import CircleCollection
File "C:\xampp7\Python\lib\site-packages\pybinding\support\collections.py", line 2, in <module>
from matplotlib.collections import Collection, allow_rasterization
ImportError: cannot import name 'allow_rasterization'
I have already check and matplotlib is correctly installed (I try some plots recommended in the documentation of matplotlib and worked pretty well). Also a looked for the file collections.py in the pybinding library and the mistake is in the second line
import numpy as np
from matplotlib.collections import Collection, allow_rasterization
And looking at collections.py of the matplolib and searching for 'allow_rasterization' I found the duplicated 6 time the following function
#artist.allow_rasterization
def draw(self, renderer):
I am pretty new at python so I'm don't know if I'm looking at what I should. Thanks in advance

I got the same issue. I'm using Linux and the package pybinding are installed on
/usr/lib/python3.6/site-packages/pybinding/support/
You can correct the error changing the import modules of allow_rasterization module, changing the file colletion.py
/usr/lib/python3.6/site-packages/pybinding/support/colletion.py
the firsts lines from
import numpy as np
from matplotlib.collections import Collection, allow_rasterization
to
import numpy as np
from matplotlib.collections import Collection
from matplotlib.artist import allow_rasterization
You can correct the plot issue related to previous versions on matplotlib 1.1.2, because now the pybinding packages use matplotlib 2.2.2.

I uninstalled version 2.2.2 of matplotlib and installed version 1.1.2 of marplotlib.
I am now able to do put.show()

go into the file "C:\xampp7\Python\lib\site-packages\pybinding\support\collections.py" and modify the command line as:
"from matplotlib.collections import Collection#, allow_rasterization
from matplotlib.artist import allow_rasterization"

Go into the directory:
~/miniconda3/lib/python3.7/site-packages/pybinding/support
In the file: collections.py
change line 2:
from matplotlib.collections import Collection, allow_rasterization
to
from matplotlib.collections import Collection
from matplotlib.artist import allow_rasterization

Cannot import mean from statistics

I'm trying to import statistics module in python. It's giving me an error message when i execute the program.
Here is my code:
from statistics import mean
import numpy as np
import matplotlib.pyplot as plt
xs = np.array([1,2,3,4,5,6,7,8])
ys = np.array([2,8,5,0,5,7,3,6])
def best_fit_line(xs ,ys):
m = ( ((mean(xs)* mean(ys))- mean(xs*ys)) /
(mean(xs)*mean(xs))-(mean(xs*xs)))
return m
m = best_fit_line(xs,ys)
The error message:
Traceback (most recent call last):
File "/home/kudzai/Documents/Machine Learning/LinearRegAlg.py", line 1, in <module>
from statistics import mean
ImportError: No module named statistics

The statistics module was added in Python 3.4. Perhaps you're using an older Python version.
If you can't upgrade for whatever reason, you can also just use numpy's mean function: np.mean(xs) etc. For numpy arrays, it's probably faster too.

Using pyclblas for a simple example

I was trying to run a calculation using pyclblas (a python wrapper for clblas), but ran into some trouble.
Here's my code:
# imports (my python is 2.7)
from __future__ import absolute_import, print_function
import numpy as np
import pyopencl as cl
import pyclblas
# create some generic structures according to pyopencl tutorial
ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)
mf = cl.mem_flags
# create a vector and a buffer
c_np = np.random.rand(50000).astype(np.float)
c_g = cl.Buffer(ctx, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=c_np)
# use pyclblas to make a calculation with the vector
res = pyclblas.clblasSscal(len(c_np), 1.0, c_g, 0, 1, queue, None)
This gives me the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 540, in runfile
execfile(filename, namespace)
File "/home/name/.spyder2/.temp.py", line 49, in <module>
res = pyclblas.clblasSscal(len(c_np), 1.0, c_g, 0, 1, queue, None)
File "/usr/local/lib/python2.7/dist-packages/pyclblas.py", line 245, in clblasSscal
return pyclblas_swig.clblasSscal(N, alpha, X, offx, incx, commandQueues, eventWaitList)
TypeError: in method 'clblasSscal', argument 6 of type 'cl_uint'
The documentation says that queue should be pyopencl.CommandQueue, not 'cl_uint'.
Does any one know what the problem is?
Thanks!

It looks like I'm a bit late, but I'm the developer of the pyclblas package.
What version of the package were you trying to use when it broke? In some of the older versions you needed to wrap the command queue in a list or tuple for the SWIG interface to recognize your input. I've since updated the SWIG interface such that it works if you pass in a command queue directly. The version when you posted this, 0.8.1, also had a bug that kept it from being installed properly. The latest version 0.8.3 seems to install properly from pip and should meet your needs.
However, there were a few bugs in your script that will need to be corrected:
np.float needs to be np.float32 (because you're using Sscal instead of Dscal). Also, your c_g needs to be declared READ_WRITE instead of READ_ONLY because the Sscal kernel stores the modified result back into c_g.
This version works correctly for me using a fresh pip install:
#!/usr/bin/env python
import numpy as np
import pyopencl as cl
import pyclblas
import sys
alpha = float(sys.argv[1])
# create some generic structures according to pyopencl tutorial
ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)
mf = cl.mem_flags
# create a vector and a buffer
c_np = np.random.rand(50000).astype(np.float32)
c_g = cl.Buffer(ctx, mf.READ_WRITE, size=c_np.nbytes)
cl.enqueue_copy(queue, c_g, c_np)
# use pyclblas to make a calculation with the vector
res = pyclblas.clblasSscal(len(c_np), alpha, c_g, 0, 1, queue, None)
import scipy.linalg.blas
res_np = np.empty_like(c_np)
cl.enqueue_copy(queue, res_np, c_g)
exp_np = np.copy(c_np)
scipy.linalg.blas.sscal(alpha, exp_np)
print np.linalg.norm(c_np), np.linalg.norm(res_np), np.linalg.norm(exp_np)
print np.linalg.norm(exp_np - res_np)

Scipy installation, its installed why the traceback?

I am running the following code (this is the beginning snippet):
# Import necessary modules and functions
########################################
# time module, for timing it.
from time import time
global starttime
starttime = time()
# psutil for indentifying processor and memory usage
#import psutil
# csv module for loading csv files
import csv
# numpy, a standard module for working with arrays
import numpy as np
# for randomly shuffling arrays
from random import shuffle
# for computing mean squared errors
from sklearn import metrics
# importing modules that contain models we will use
from sklearn import linear_model, ensemble, gaussian_process
from sklearn import neural_network, svm
# importing memory profiler to enable us to determine peak memory usage
from memory_profiler import profile
# Helpful functions
###################
#psutilpercent = psutil.virtual_memory()
#print "\n", " --> Memory Check 1 Percent:", str(psutilpercent.percent) + "%\n"
# This function is for simplifying reading CSV files
def readCSV(path):
"""
Read a CSV file of floats, with no headder
"""
data = []
mycsv = csv.reader(open(path), delimiter="|")
for counter, row in enumerate(mycsv):
if counter != 0:
data.append(row)
return np.asarray(data, dtype=np.float32)
However I am receiving the traceback:
Traceback (most recent call last):
File "C:\Users\regression_v6.py", line 38, in <module>
from sklearn import metrics
File "C:\Python27\lib\site-packages\sklearn\__init__.py", line 32, in <module>
from .base import clone
File "C:\Python27\lib\site-packages\sklearn\base.py", line 10, in <module>
from scipy import sparse
ImportError: No module named scipy
But my computer shows Scipy as installed, here is a view of Control Panel\All Control Panel Items\Programs and Features:
Any ideas?

Did you install a special version of scipy? as your screenshot shows 'scipy_umfpack'
I can successfully import sklearn using the official version downloaded from sourceforge: http://sourceforge.net/projects/scipy/files/scipy/