Develop Reference

Problems executing the Jupyter notebook using Sagemaker Lifecyle configuration

I have a set up a notebook in a sagemaker instance that collects data from a postgresql database and then updates a different one with the output. When running it manually, it works (I am using sqlalchemy's create_engine and then just using pandas' to_sql on the final dataframe).
I then set up a lifecycle configuration that would execute the notebook once the instance is triggered. I wanted to get around the 5-timeout issues so I used the nohup command when executing the notebook:
#!/bin/bash
set -e
sudo -u ec2-user -i <<'EOF'
# PARAMETERS
ENVIRONMENT=python3
NOTEBOOK_FILE=/home/ec2-user/SageMaker/HourlyRun.ipynb
source /home/ec2-user/anaconda3/bin/activate "$ENVIRONMENT"
pip install --upgrade psycopg2-binary
pip install gensim==3.8.3
nohup jupyter nbconvert --to html "$NOTEBOOK_FILE" --ExecutePreprocessor.kernel_name=python3 --ExecutePreprocessor.timeout=-1 --execute &
source /home/ec2-user/anaconda3/bin/deactivate
# PARAMETERS
IDLE_TIME=1800
wget https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/master/scripts/auto-stop-idle/autostop.py
if /usr/bin/python -c "import boto3" 2>/dev/null; then
PYTHON_DIR='/usr/bin/python'
elif /usr/bin/python3 -c "import boto3" 2>/dev/null; then
PYTHON_DIR='/usr/bin/python3'
else
exit 1
fi
(crontab -l 2>/dev/null; echo "*/5 * * * * $PYTHON_DIR $PWD/autostop.py --time $IDLE_TIME --ignore-connections >> /var/log/jupyter.log") | crontab -
EOF
The instance now starts and stops as scheduled with no errors but I noticed the database wasn't being updated. I checked the logs and it seems like an operational error when querying the database.
I searched around for the solution but I only found references to changing hot_standby_feedback to "on" but it's not clear how I can do this within the code in jupyter. Not sure how approach this or if there is a better solution.
Thanks in advance.

Cannot use keras models on Mac M1 with BigSur

I am trying to use Sequential model from keras of tensorflow. When I am executing following statement:
model.fit(x_train, y_train, epochs=20, verbose=True, validation_data=(x_dev, y_dev), batch_size=10)
I am getting following errors:
I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
W tensorflow/core/platform/profile_utils/cpu_utils.cc:126] Failed to get CPU frequency: 0 Hz
F tensorflow/core/grappler/costs/op_level_cost_estimator.cc:710] Check failed: 0 < gflops (0 vs. 0)type: "CPU"
I am not able to understand how to fix it. Can anyone please help me.
From this issue on github, I understood that device.frequency() returned 0 probably because NominalCPUFrequency() returned 1.
However, this information seems too abstract for me and I cannot understand.

First two ones are nothing to worry about.
The third one is a problem. You have installed an improper version of TensorFlow. Use one that supports the Mac M1 chip.
Run the following bash script to download and install TensorFlow.
#!/bin/bash
set -e
VERSION=0.1alpha3
INSTALLER_PACKAGE=tensorflow_macos-$VERSION.tar.gz
INSTALLER_PATH=https://github.com/apple/tensorflow_macos/releases/download/v$VERSION/$INSTALLER_PACKAGE
INSTALLER_SCRIPT=install_venv.sh
echo
# Check to make sure we're good to go.
if [[ $(uname) != Darwin ]] || [[ $(sw_vers -productName) != macOS ]] || [[ $(sw_vers -productVersion) != "11."* ]] ; then
echo "ERROR: TensorFlow with ML Compute acceleration is only available on macOS 11.0 and later."
exit 1
fi
# This
echo "Installation script for pre-release tensorflow_macos $VERSION. Please visit https://github.com/apple/tensorflow_macos "
echo "for instructions and license information."
echo
echo "This script will download tensorflow_macos $VERSION and needed binary dependencies, then install them into a new "
echo "or existing Python 3.8 virtual environment."
# Make sure the user knows what's going on.
read -p 'Continue [y/N]? '
if [[ ! $REPLY =~ ^[Yy]$ ]]
then
exit 1
fi
echo
echo "Downloading installer."
tmp_dir=$(mktemp -d)
pushd $tmp_dir
curl -LO $INSTALLER_PATH
echo "Extracting installer."
tar xf $INSTALLER_PACKAGE
cd tensorflow_macos
function graceful_error () {
echo
echo "Error running installation script with default options. Please fix the above errors and proceed by running "
echo
echo " $PWD/$INSTALLER_SCRIPT --prompt"
echo
echo
exit 1
}
bash ./$INSTALLER_SCRIPT --prompt || graceful_error
popd
rm -rf $tmp_dir
ref: https://github.com/apple/tensorflow_macos

I've done as follows on macOS 11.4 (Even though the ref says "OS Requirements macOS 12.0+"), python==3.8.2 and worked [ref: https://developer.apple.com/metal/tensorflow-plugin/]:
Create a venv on x86 terminal, i.e. Rosetta Terminal (see: https://dev.to/courier/tips-and-tricks-to-setup-your-apple-m1-for-development-547g)
i.e. Environment Setup:
x86 : AMD
Create venv: python3 -m venv ~/PATH/tensorflow-metal (Substitute PATH with your real PATH)
Activate the venv: source ~/PATH/tensorflow-metal/bin/activate
Update pip: python -m pip install -U pip
Install any library/package you need. For instance:
For instance: pip install matplotlib jupyterlab
Install base tensorflow:
python -m pip install tensorflow-macos
Install metal plugin:
python -m pip install tensorflow-metal
Good Luck & Cheers!

This might not help at all, but since I was running into the same problem I managed to get the model to train without the solutions provided here (that I will soon try), simply by changing my Y_test (0s and 1s) like this when making the train_test_split: (to_categorical(label). So:
X_train, X_test, Y_train, Y_test = train_test_split(
dataset,
to_categorical(label),
test_size=.2,
random_state=42
)
Then, when training the model, I get the following message - that I do not understand fully:
2022-04-03 23:10:08.941296: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
So, this is not really a solution, but more a temporary workaround - or it might give insight in where it goes wrong.

launch tensorboard in colab

i try launch tensorboard on colab, my code:
LOG_DIR = model_dir
get_ipython().system_raw(
'tensorboard --logdir {} --host 0.0.0.0 --port 6060 &'
.format(LOG_DIR)
)
get_ipython().system_raw('./ngrok http 6060 &')
! curl -s http://localhost:4040/api/tunnels | python3 -c \
"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
two days ago everything worked, but now such an error:
error

The error linked suggests the port you're using is 6006, but the code example you gave above has the port as 6060. So may just be a typo there.
It's also possible you want a TCP tunnel, not an HTTP tunnel.
In either case, might I suggest trying something like pyngrok to programmatically manage your ngrok tunnel for you? Full disclosure, I am the developer of it. Here are the docs if you're interest.

Run localhost server in Google Colab notebook

I am trying to implement Tacotron speech synthesis with Tensorflow in Google Colab using this code form a repo in Github, below is my code and working good till the step of using localhost server, how I can to run a localhost server in a notebook in Google Colab?
My code:
!pip install tensorflow==1.3.0
import tensorflow as tf
print("You are using Tensorflow",tf.__version__)
!git clone https://github.com/keithito/tacotron.git
cd tacotron
pip install -r requirements.txt
!curl https://data.keithito.com/data/speech/tacotron-20180906.tar.gz | tar xzC /tmp
!python demo_server.py --checkpoint /tmp/tacotron-20180906/model.ckpt #requires localhost
Unfortunately running in local mode from Google Colab will not help me because to do this I need to download the data in my machine which are too large.
Below is my last output and here I am supposed to open the localhost:8888 to complete the work, so as I mentioned before is there any way to run localhost in Google Colaboratory?

You can do this by using tools like ngrok or remote.it
They give you a URL that you can access from any browser to access your web server running on 8888
Example 1: Tunneling tensorboard running on
!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
!unzip ngrok-stable-linux-amd64.zip
get_ipython().system_raw('tensorboard --logdir /content/trainingdata/objectdetection/ckpt_output/trainingImatges/ --host 0.0.0.0 --port 6006 &')
get_ipython().system_raw('./ngrok http 6006 &')
! curl -s http://localhost:4040/api/tunnels | python3 -c \
"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
Running this install ngrok on colab, and makes a link like http://c11e1b53.ngrok.io/
Documentaion for NGROK

Another way of running a publicly accessible server using ngrok:
!pip install pyngrok --quiet
from pyngrok import ngrok
# Terminate open tunnels if exist
ngrok.kill()
# Setting the authtoken (optional)
# Get your authtoken from https://dashboard.ngrok.com/auth
NGROK_AUTH_TOKEN = ""
ngrok.set_auth_token(NGROK_AUTH_TOKEN)
# Open an HTTPs tunnel on port 5000 for http://localhost:5000
public_url = ngrok.connect(port="5000", proto="http", options={"bind_tls": True})
print("Tracking URL:", public_url)

You can use localtunnel to expose the port to the public internet.
Install localtinnel:
!npm install -g localtunnel
Start localtunnel:
!lt --port 8888
Navigate to the url it returns to access your web UI.

clGetPlatformIDs failed: <unknown error -1001>

when i run following code
import pyopencl as cl
cl.get_platforms()
I get error
clGetPlatformIDs failed: <unknown error -1001>
I am running python 3.6 pyopencl 2018.1.1 on aws ec2 Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-116-generic x86_64).
I have tried following things , but none of them work:
echo libnvidia-opencl.so.1 >> /etc/OpenCL/vendors/nvidia.icd
from root directory by doing sudo -i
after ssh into ubuntu ec2 instance. (initially this command wont work so i removed nvidia.icd file {rm nvidia.icd}and then this command worked. but it did not solve the error 1001 mentioned above.
echo libnvidia-opencl.so.384.111 >> /etc/OpenCL/vendors/nvidia.icd
sudo ln -s /opt/intel/opencl-1.2-3.2.1.16712/etc/intel64.icd /etc/OpenCL/vendors/nvidia.icd
sudo usermod -aG video your-user-name
sudo ln -s /usr/share/nvidia-331/nvidia.icd /etc/OpenCL/vendors
sudo ln -s /usr/share/nvidia-384/nvidia.icd /etc/OpenCL/vendors
optirun myopenclprogram

The easiest way to use OpenCL on EC2 is by using the Deep Learning Base Image, which comes with all necessary drivers and is already configured to work with P2 and P3 instance types. The image can be found at https://aws.amazon.com/marketplace/pp/B077GCH38C.