I am trying to use Sequential model from keras of tensorflow. When I am executing following statement:
model.fit(x_train, y_train, epochs=20, verbose=True, validation_data=(x_dev, y_dev), batch_size=10)
I am getting following errors:
I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
W tensorflow/core/platform/profile_utils/cpu_utils.cc:126] Failed to get CPU frequency: 0 Hz
F tensorflow/core/grappler/costs/op_level_cost_estimator.cc:710] Check failed: 0 < gflops (0 vs. 0)type: "CPU"
I am not able to understand how to fix it. Can anyone please help me.
From this issue on github, I understood that device.frequency() returned 0 probably because NominalCPUFrequency() returned 1.
However, this information seems too abstract for me and I cannot understand.
First two ones are nothing to worry about.
The third one is a problem. You have installed an improper version of TensorFlow. Use one that supports the Mac M1 chip.
Run the following bash script to download and install TensorFlow.
#!/bin/bash
set -e
VERSION=0.1alpha3
INSTALLER_PACKAGE=tensorflow_macos-$VERSION.tar.gz
INSTALLER_PATH=https://github.com/apple/tensorflow_macos/releases/download/v$VERSION/$INSTALLER_PACKAGE
INSTALLER_SCRIPT=install_venv.sh
echo
# Check to make sure we're good to go.
if [[ $(uname) != Darwin ]] || [[ $(sw_vers -productName) != macOS ]] || [[ $(sw_vers -productVersion) != "11."* ]] ; then
echo "ERROR: TensorFlow with ML Compute acceleration is only available on macOS 11.0 and later."
exit 1
fi
# This
echo "Installation script for pre-release tensorflow_macos $VERSION. Please visit https://github.com/apple/tensorflow_macos "
echo "for instructions and license information."
echo
echo "This script will download tensorflow_macos $VERSION and needed binary dependencies, then install them into a new "
echo "or existing Python 3.8 virtual environment."
# Make sure the user knows what's going on.
read -p 'Continue [y/N]? '
if [[ ! $REPLY =~ ^[Yy]$ ]]
then
exit 1
fi
echo
echo "Downloading installer."
tmp_dir=$(mktemp -d)
pushd $tmp_dir
curl -LO $INSTALLER_PATH
echo "Extracting installer."
tar xf $INSTALLER_PACKAGE
cd tensorflow_macos
function graceful_error () {
echo
echo "Error running installation script with default options. Please fix the above errors and proceed by running "
echo
echo " $PWD/$INSTALLER_SCRIPT --prompt"
echo
echo
exit 1
}
bash ./$INSTALLER_SCRIPT --prompt || graceful_error
popd
rm -rf $tmp_dir
ref: https://github.com/apple/tensorflow_macos
I've done as follows on macOS 11.4 (Even though the ref says "OS Requirements macOS 12.0+"), python==3.8.2 and worked [ref: https://developer.apple.com/metal/tensorflow-plugin/]:
Create a venv on x86 terminal, i.e. Rosetta Terminal (see: https://dev.to/courier/tips-and-tricks-to-setup-your-apple-m1-for-development-547g)
i.e. Environment Setup:
x86 : AMD
Create venv: python3 -m venv ~/PATH/tensorflow-metal (Substitute PATH with your real PATH)
Activate the venv: source ~/PATH/tensorflow-metal/bin/activate
Update pip: python -m pip install -U pip
Install any library/package you need. For instance:
For instance: pip install matplotlib jupyterlab
Install base tensorflow:
python -m pip install tensorflow-macos
Install metal plugin:
python -m pip install tensorflow-metal
Good Luck & Cheers!
This might not help at all, but since I was running into the same problem I managed to get the model to train without the solutions provided here (that I will soon try), simply by changing my Y_test (0s and 1s) like this when making the train_test_split: (to_categorical(label). So:
X_train, X_test, Y_train, Y_test = train_test_split(
dataset,
to_categorical(label),
test_size=.2,
random_state=42
)
Then, when training the model, I get the following message - that I do not understand fully:
2022-04-03 23:10:08.941296: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
So, this is not really a solution, but more a temporary workaround - or it might give insight in where it goes wrong.
Related
I'm trying to calculate paths for a pip install from a internal devpi server. I'm running a self-hosted runner on a Windows server virtual machine. I'm trying to install the latest PIP package to the tool directory by calculating the path as follows;
- name: pip install xmlcli
env:
MYTOOLS: ${{ runner.tool_cache }}\mytools
run: |
echo "${{ runner.tool_cache }}"
$env:MYTOOLS
pip install --no-cache-dir --upgrade mytools.xmlcli --target=$env:MYTOOLS -i ${{secrets.PIP_INDEX_URL}}
echo "XMLCLI={$env:MYTOOLS}\mytools\xmlcli" >> $GITHUB_ENV`
- name: test xmlcli
run: echo "${{ env.XMLCLI }}"
As you can see; I've had some noob issues trying to output the env variable in windows. I came to the conclusion that under windows; the "run" command is being sent via powershell. Hence the "$env:MYTOOLS" usage.
The problem is the echo "XMLCLI=..." back to the git_env doesn't seem to be working properly as the test xmlcli step returns empty string.
I'm pretty sure I tried several different iterations of the echo command; but, haven't been successful.
Is there a video/docs/something that will clearly lays out the usage of "path arithmetic" from within the github action environment?
You need to append to $env:GITHUB_ENV, or you can set the script execution engine on your run action.
When using shell pwsh, then you can use:
"{environment_variable_name}={value}" >> $env:GITHUB_ENV
When using shell powershell
"{environment_variable_name}={value}" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
But in your case, if you're more familiar with bash, you can force the run action to always use bash
- run: |
// your stuff here
shell: bash
See:
run: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
Workflow commands for actions: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions?tool=bash
#jessehouwing gave me enough information to test a solution to my particular problem. Assuming Windows runners always runs on Powershell, the answer is:
- name: pip install xmlcli
env:
MYTOOLS: ${{ runner.tool_cache }}\mytools
run: |
echo "${{ runner.tool_cache }}"
$env:MYTOOLS
pip install --no-cache-dir --upgrade mytools.xmlcli --target=$env:MYTOOLS -i ${{secrets.PIP_INDEX_URL}}
"XMLCLI=$env:MYTOOLS\mytools\xmlcli" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
- name: test xmlcli
run: echo "${{ env.XMLCLI }}"
Incidentally; I was using
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-environment-variable
as a reference; it didn't seem to give the powershell equivalent.
I've got Python 3.9 (thru Anaconda) and Matlab R2021b installed on my Mac (Monterey). It is my personal computer, so I should have full administrator privileges. In following the instructions here, I ran open .bash_profile and found that when setting up Anaconda it had added the following:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('~/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "~/opt/anaconda3/etc/profile.d/conda.sh" ]; then
. "~/opt/anaconda3/etc/profile.d/conda.sh"
else
export PATH="~/opt/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
so that the Python interpreter should indeed be in-PATH (?). Then, in the Matlab terminal I found that my matlabroot is /Applications/MATLAB_R2021b.app so I ran
cd /Applications/MATLAB_R2021b.app/extern/engines/python
python setup.py install
without (I think) any issues. Back in Matlab, I ran
cd (fullfile(matlabroot,'extern','engines','python'))
system('python setup.py install')
but was met with
...
running install_lib
creating /Library/Python
error: could not create '/Library/Python': Permission denied
(the ... are other, presumably successful, copy operations, which I can add back if it would be helpful).
I'm very uncomfortable working with filesystem stuff and don't understand too well how to debug/navigate these kinds of issues, so any help with getting the Matlab API up and running would be enormously helpful. Thanks in advance.
when i run following code
import pyopencl as cl
cl.get_platforms()
I get error
clGetPlatformIDs failed: <unknown error -1001>
I am running python 3.6 pyopencl 2018.1.1 on aws ec2 Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-116-generic x86_64).
I have tried following things , but none of them work:
echo libnvidia-opencl.so.1 >> /etc/OpenCL/vendors/nvidia.icd
from root directory by doing sudo -i
after ssh into ubuntu ec2 instance. (initially this command wont work so i removed nvidia.icd file {rm nvidia.icd}and then this command worked. but it did not solve the error 1001 mentioned above.
echo libnvidia-opencl.so.384.111 >> /etc/OpenCL/vendors/nvidia.icd
sudo ln -s /opt/intel/opencl-1.2-3.2.1.16712/etc/intel64.icd /etc/OpenCL/vendors/nvidia.icd
sudo usermod -aG video your-user-name
sudo ln -s /usr/share/nvidia-331/nvidia.icd /etc/OpenCL/vendors
sudo ln -s /usr/share/nvidia-384/nvidia.icd /etc/OpenCL/vendors
optirun myopenclprogram
The easiest way to use OpenCL on EC2 is by using the Deep Learning Base Image, which comes with all necessary drivers and is already configured to work with P2 and P3 instance types. The image can be found at https://aws.amazon.com/marketplace/pp/B077GCH38C.
I've been trying to install nodejs on my VPS for a little while now. Since I'm using CentOs 5.6 I had to build it from source.
More so I need the python 2.7 as the default python on my box was 2.4.
I compiled python from source and it was installed successfully in /usr/local/bin/python2.7.
Now the problem is upon issuing make in the nodejs directory it reaches the following exceptions.
.
.
.
LD_LIBRARY_PATH=/root/node/out/Release/lib.host:/root/node/out/Release/lib.tar
get:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps/v8/tools/gyp; mkdir -p
/root/node/out/Release/obj/gen; python ../../tools/generate-trig-table.py "/root
/node/out/Release/obj/gen/trig-table.cc"
touch /root/node/out/Release/obj.host/deps/v8/tools/gyp/generate_trig_table.st
amp
LD_LIBRARY_PATH=/root/node/out/Release/lib.host:/root/node/out/Release/lib.tar
get:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps/v8/tools/gyp; mkdir -p
/root/node/out/Release/obj/gen; python ../../tools/js2c.py "/root/node/out/Relea
se/obj/gen/libraries.cc" CORE off ../../src/runtime.js ../../src/v8natives.js ..
/../src/array.js ../../src/string.js ../../src/uri.js ../../src/math.js ../../sr
c/messages.js ../../src/apinatives.js ../../src/debug-debugger.js ../../src/mirr
or-debugger.js ../../src/liveedit-debugger.js ../../src/date.js ../../src/json.j
s ../../src/regexp.js ../../src/arraybuffer.js ../../src/typedarray.js ../../src
/weak_collection.js ../../src/promise.js ../../src/object-observe.js ../../src/m
acros.py
File "../../tools/js2c.py", line 387
except Error as e:
^
SyntaxError: invalid syntax
make[1]: *** [/root/node/out/Release/obj/gen/libraries.cc] Error 1
make[1]: Leaving directory `/root/node/out'
make: *** [node] Error 2
Somewhere I read that the Exception syntax has changed from python 2.6 up and I figured it must be using the old python so I did the following but it made no difference:
PYTHON=/usr/local/bin/python2.7
export PYTHON
python2.7 configure && make && make install
Now I'm wondering how should I proceed?
IMO you need to place python2.7 first in path and then run:
export PATH=/usr/local/bin:${PATH}
python2.7 configure && make && make install
If this does not work, probably one of the Python scripts is looking for python. You can probably fix that by symlinking python, something like:
mkdir /tmp/py27
ln -s /usr/local/bin/python2.7 /tmp/py27/python
export PATH=/tmp/py27:${PATH}
python configure && make && make install
Is all caps PYTHON a valid environment variable?
http://www.wellho.net/resources/ex.php4?item=y115/penv.py
I would think you would rather create a sym link to the correct python interpreter.
ln -s /usr/local/bin/python2.7 /usr/local/bin/python
I have just installed python 2.7 using macports as:
sudo port install py27-numpy py27-scipy py27-matplotlib py27-ipython +notebook py27-pandas py27-sympy py27-nose
during the process it found some issues, mainly broken files related with py25-haslib that I managed to fix. Now it seems eveything is ok. I tested a few programs and they run as expected. Currently, I have two versions of python: 2.5 (Default, from when I worked in my former institution) and 2.7 (just installed):
which python
/usr/stsci/pyssg/Python-2.5.1/bin/python
which python2.7
/opt/local/bin/python2.7
The next move would be set the new python version 2.7 as default:
sudo port select --set python python27
sudo port select --set ipython ipython27
My question is: is there a way to go back to 2.5 in case something goes wrong?
I know a priori, nothing has to go wrong. But I have a few data reduction and analysis routines that work perfectly with the 2.5 version and I want to make sure I don´t mess up before setting the default.
if you want to revert, you can modify your .bash_profile or other login shell initialization to fix $PATH to not add "/Library/Frameworks/Python.framework/Versions/2.5/bin" to $PATH and/or to not have /usr/local/bin appear before /usr/bin on $PATH.
If you want to permanently remove the python.org installed version,
paste the following lines up to and including the chmod into a posix-
compatible shell:
tmpfile=/tmp/generate_file_list
cat <<"NOEXPAND" > "${tmpfile}"
#!/bin/sh
version="${1:-"2.5"}"
file -h /usr/local/bin/* | grep \
"symbolic link to ../../../Library/Frameworks/Python.framework/"\
"Versions/${version}" | cut -d : -f 1
echo "/Library/Frameworks/Python.framework/Versions/${version}"
echo "/Applications/Python ${version}"
set -- Applications Documentation Framework ProfileChanges \
SystemFixes UnixTools
for package do
echo "/Library/Receipts/Python${package}-${version}.pkg"
done
NOEXPAND
chmod ug+x ${tmpfile}
...excripted from troubleshooting question on python website