I am relatively new to GCP and am trying to schedule a notebook on GCP to run everyday. This notebook has dependencies in terms of libraries and other python modules/scripts. When I schedule this with the Cloud Scheduler (as shown in image), there are errors shown in logs at import statements of libraries and while importing other python modules.
I also created a requirements.txt file, but the scheduler doesn't seem to be reading it.
Am I doing something wrong?
Can anyone help or guide me with some possible solutions? Been stuck with this since a few days, any help would be highly appreciated.
PS- Cloud Functions would be by last option incase I'm not able to run this way.
The problem is that we have 2 different environments:
Notebook document itself
Docker container that Notebook Executor uses when you click on Execute: a Docker container is passed to Executor backend (Notebooks API + Vertex Custom Job) and since you are installing the dependencies in the Notebook itself (Managed Notebook underlying infra), these are not included in the container, hence this fails. You need to pass a container that includes Selenium.
If you need to build a custom container I would do the following:
Create a custom container
# Dockerfile.example
FROM gcr.io/deeplearning-platform-release/tf-gpu:latest
RUN pip install -y selenium
Then you’ll need to build and push it somewhere accessible.
PROJECT="my-gcp-project"
docker build . -f Dockerfile.example -t "gcr.io/${PROJECT}/tf-custom:latest"
gcloud auth configure-docker
docker push "gcr.io/${PROJECT}/tf-custom:latest"
Specify the container when launching the Execution "Custom Container"
The error means that you are missing the selenium module, you need to install it. You can use the following commands to install it:
python -m pip install -U selenium (you need pip installed)
pip install selenium
or depending on your permissions:
sudo pip install selenium
For python3:
sudo pip3 install selenium
Edit 1:
If you have selenium installed, check where you have Python located and where the Python looks for libraries/packages, including the ones installed using pip. Sometimes Python runs from a location, but looks for libraries in a different location. Make sure Python is looking for the libraries in the right directory.
Here is an answer that you can use to check if Python is configured correctly.
Related
I am trying, for the first time ever, to deploy a ML model, using Flask. I'm following the instructions from the link below.
https://towardsdatascience.com/deploy-a-machine-learning-model-using-flask-da580f84e60c
I created three separate and distinct .py files named 'model.py', 'server.py', and 'request.py'. I open my Anaconda Prompt end entered this: 'C:\Users\ryans>C:\Users\ryans\model.py'
Now, I get this.
I definitely have Numpy installed! Something must be wrong with my setup, or maybe the way I am starting the process is wrong, but I'm not sure what the issue is. Has anyone encountered this problem before.
I would suggest the following steps since you mentioned its your first time and when deploying a project for experimenting, it's good practice to put it in a virtual environment, which we can do with the virtualenv tool.
Assuming you already have pip installed, install virtualenv:
C:\> pip install virtualenv
Create a fresh working directory, switch to it and clone the repository from Github mentioned by the author of the article:
C:\your-working-directory\> git clone https://github.com/vyashemang/flask-salary-predictor.git
Start virtualenv in the repository's top directory
C:\your-working-directory\flask-salary-predictor\> virtualenv env
Activate the created batch file, \env\Scripts\activate.bat:
C:\Users\'Username'\venv\Scripts\activate.bat
Now in the virtual environment, install all requirements:
(env) C:\your-working-directory\flask-salary-predictor\> pip install -r requirements.txt
You can now run the model.py like you have above. I'd suggest that you run the server.py using gunicorn or on Windows use waitress according to this post so that instead of running request.py every time you want to send a request, you can use Insomnia or Postman to send API requests.
I am a beginner in python. I have done a website using django, flask, xml, wtforms and also i have used some API python modules too. The website was successfully created and working well in local machine.
But if i want to run in an another python available machine, i am in the need of install all my above mentioned modules manually.
Do we have something similar to gradle, maven or ant which will download/install the required modules during my first run?
Kindly help me.
One way is to freeze your current local python installations into a requirements.txt file and then install everything in one go in another machine.
$ pip freeze > requirements.txt
copy the requirements file into another machine,
install python and then ...
$ pip install -r requirements.txt
I installed some python packages in my gcp instance. python_speech_features was one of them. When I wrote pip list, it shows me that it is installed. But when I try to access it in my code, it says no module found with this name python_speech_features. I have attached screenshots of the error error and installed packages packages installed.
I tried to replicate a scenario like the one you currently have; however, I was able to use the ’python_speech_features’ library with this example on a GCE instance without issues. This is the procedure I did:
Create an GCE instance and connect it.
Create a virtual environment with this command: virtualenv -p python3 env
Activate the virtualenv with this command: source env/bin/activate
Install the following libraries:
numpy==1.18.5
python-speech-features==0.5
scipy==1.4.1
Download the example file: ‘example.wav’
Run the code: ‘python example.py’
I recommend trying this procedure to ensure that the library import works correctly.
I'm used to using pip to install Python packages into my Django projects' virtual environments.
When I am working with a Divio Docker project locally, this does not work.
There are two things you need to be aware of when installing Python packages into a Docker project:
the package must be installed in the correct environment
if you want to use the installed package in the future, it needs to be installed in a more permanent way
The details below describe using a Divio project, but the principle will be similar for other Docker installations.
Installation in the correct environment
To use pip on the command line to install a Python package into a Dockerised project, you need to be using pip inside the Docker environment, that is, inside the container.
It's not enough to be in the directory where you have access to the project's files. In this respect, it's similar to using a virtual environment - you need to have the virtualenv activated. (Otherwise, your package will be installed not in the virtual environment, but on your own host environment.)
To activate a virtual environment, you'd run something like source bin/activate on it.
To install a package within a Divio web container:
# start a bash prompt inside the project
docker-compose run --rm web bash
# install the package in the usual way
pip install rsa
rsa is now installed and available to use.
More permanent installation
So far however, the package will only be installed and available in that particular container. As soon as you exit the bash shell, the container will disappear. The next time you launch a web container, you will not find the rsa package there. That's because the container is launched each time from its image.
In order to have the package remain installed, you will need to include it in the image.
A Divio project includes a requirements.in file, listing Python packages that will be included in the image.
Add a new line containing rsa to the end of that file. Then run:
docker-compose build web
This will rebuild the Docker image. Next time you launch a container with (for example) docker-compose run --rm web bash, it will include that Python package.
(The Divio Developer Handbook has some additional guidance on using pip.)
Note: I am a member of the Divio team. This question is one that we see quite regularly via our support channels.
My Python version is 3.6.0 and my operating system is
Windows.
I want to install jupyter notebook using the order pip install jupyter.
But it failed, I got the following error:
Three Ways to Run Jupyter In Windows
The "Pure Python" Way
Make your way over to python.org, download and install the latest version (3.5.1 as of this writing) and make sure that wherever you install it, the directory containing python.exe is in your system PATH environment variable. I like to install it in the root of my C: drive, e.g. C:\Python35, so my PATH contains that directory.
Once that's installed, you'll want to create a virtual environment, a lightweight, disposable, isolated python installation where you can experiment and install 3rd party libraries without affecting your "main" installation. To do this, open up a Powershell window, and enter the following commands (where "myenv" is the name of the virtualenv we're going to create, you can use any name you like for this):
PS C:\> python -m venv myenv
PS C:\> myenv\Scripts\activate
Then, let's install jupyter and start up a notebook:
PS C:\> pip install jupyter
PS C:\> jupyter notebook
Incidentally, if you get a warning about upgrading pip, make sure to use the following incantation to upgrade (to prevent an issue on windows where pip is unable to upgrade its own executable in-place):
PS C:\> python -m pip install --upgrade pip
Advantages: Uses "pure" python, official tools, and no external dependencies. Well supported, with plenty of online documentation and support communities.
Disadvantages: While many popular data analysis or scientific python libraries can be installed by pip on windows (including Pandas and Matplotlib), some (for example SciPy) require a C compiler and the presence of 3rd party C libraries on the system which are difficult to install on Windows.
Who is it for? Python users comfortable with the command line and the tools that ship with Python itself.
The Python Distributions
Because of the difficulty mentioned above in getting packages like SciPy installed on Windows, a few commercial entities have put together pre-packaged Python "distributions" that contain most, if not all, of the commonly used libraries for data analysis and/or scientific computing.
Anaconda is an excellent option for this. Download their Python 3.5 installer for Windows, run it, and in your Start menu you'll have a bunch of neat new tools, including an entry for Jupyter Notebook. Click to start it up and it'll launch in the background and open up your browser to the notebook console. It doesn't get any easier than that.
Advantages: Simplest, fastest way to get started and it comes with probably everything you need for your scientific computing projects. And anything it doesn't ship with you can still instalAl via its built in conda package manager.
Disadvantages: No virtualenv support, although the conda package manager provides very similar functionality with the conda create command. Relies on a commercial 3rd party for support.
Who is it for? People who want the quickest, easiest way to get Jupyter notebook up and running (IE, most people).
Docker
Docker is a platform for running software in "containers", or self-contained, isolated processes. While it may sound similar in concept to python virtual environments, Docker containers are an entirely different kind of technology offering vast flexibility and power. Don't let the flexibility and power and confusing terminology put you off though -- Docker can be easy to get up and running on your PC and has some advantages of its own with respect to Python and Jupyter.
To get started on Windows, download the Docker Toolbox, which contains the tools you need to get up and running. Run the installer and make sure the checkbox to install Virtualbox is checked if you don't already have Virtualbox or another virtualization platform (like VMWare Workstation) installed.
Once installed, you'll have a "Docker Quickstart Terminal" shortcut in your Start Menu. Double click that shortcut and it will create your first Docker engine for you and set up everything you need automatically. Once you see a prompt in the terminal, you can use the docker run command to run Docker "images", which you can think of as pre-packaged bundles of software that will be automatically downloaded from the Docker Hub when you run them. There are many images on Docker Hub that offer Jupyter, including the official Jupyter Notebook image, and Anaconda itself if you want the full SciPy stack.
To run just the official Jupyter Notebook image in your Docker engine, type the following into the Docker Quickstart Terminal:
$ docker run --rm -it -p 8888:8888 -v "$(pwd):/notebooks" jupyter/notebook
After all the image's "layers" are downloaded, it will start up. Make a note of the IP address listed in the terminal (for example, 192.168.99.100), and point your browser at that IP address, port 8888 (e.g. http://192.168.99.100:8888) and you'll see the familiar Jupyter console, with both Python 2 and Python 3 kernels available.
Advantages: Use the flexibility and power of Docker! Honestly one of my favorite things about Docker is thinking of it as an open software distribution platform for things like the SciPy stack that are hard to install.
Disadvantages: Grapple with the flexibility and power of Docker! There are quite a few "gotchas" to be aware of when dealing with Docker, such as immutable containers, data volumes, arcane commands, and rapidly developing, occasionally buggy tooling.
I met the same problem as yours. I updated my Python to version 3.6.1 and re-install Jupyter notebook. I encounter the first problem UnicodeDecodeError when pip setting up tornado, my system is Windows 8.1 64bit.
Here is what I do to solve it:
I open the directory ...\Python\Python36\Lib\site-packages\tornado where new python installed
open the directory ...\Python\Python35\Lib\site-packages\tornado where old python installed
Then I copy and overwrite all *.py files in new tornado directory form old tornado (Note: my old tornado is the latest 4.4.2)
Then I run instruction pip install jupyter notebook again
then the same problem when pip setting up MarkupSafe, I do same as above. This time instruction pip install jupyter notebook do not throw error anymore. And Jupyter notebook works.
I think if you have the old version modules of Jupyter, you can try as what I do, Or think about download the error module manually and place them into right directory.
Hope this can help you.