I'm trying to deploy a model on the Google Cloud Platform. But I've been running into same issues I created the bucket and as specified on the docs I ran:
gcloud ai-platform local predict --model-dir gs://bucket/ \
--json-instances input.json \
--framework SCIKIT_LEARN
But for same reason it doesn't find the input file on the same bucket of the model. So I've followed the instructions on another question . I've tried coping the input.json into the main directory but for some other reason is not categorizing the json as a json file...
In reality the model was created using a library called simpletransformers that I've tried to install to test with no sucess.
I wish to know how is the best way to proceed?
input.json:
{ "document":{ "type":"PLAIN_TEXT", "content":"Protection plan costs, half of any delivery fee, and any Extras or young driver fee costs are always refunded."},"encodingType":"UTF8"}
As specified in the documentation, this command:
gcloud ai-platform local predict --model-dir local-or-cloud-storage-path-to-model-directory/ \
--json-instances local-path-to-prediction-input.json \
--framework name-of-framework
Is to test your model with local predictions and it's expecting to find your input.json file in your local machine rather than a GCS bucket. Based on what you've mentioned:
But for same reason it doesn't find the input file on the same bucket of the model
I'm assuming that you're expecting it to read it from GCS bucket. But it should actually be a local directory path; in your case, the command you executed doesn't specify a path so it's expecting to find your input.json file in the same directory where you've executed the command. I've just tried it and it worked fine for me.
I'm not sure what you mean by:
I've tried coping the input.json into the main directory but for some other reason is not categorizing the json as a json file...
But I'm assuming that you're referring to a GCS bucket as well, however, as mentioned before, your input.json file should be a local path rather than a GCS path.
Related
I have mounted a storage bucket on a VM using the command:
gcsfuse my-bucket /path/to/mount
After this I'm able to read files from the bucket in Python using Pandas, but I'm not able to write files nor create new folders. I have tried with Python and from the terminal using sudo but get the same error.
I have also tried Using the key_file from the bucket:
sudo mount -t gcsfuse -o implicit_dirs,allow_other,uid=1000,gid=1000,key_file=Notebooks/xxxxxxxxxxxxxx10b3464a1aa9.json <BUCKET> <PATH>
It does not through errors when I run the code, but still I'm not able to write in the bucket.
I have also tried:
gcloud auth login
But still have the same issue.
I ran into the same thing a while ago, which was really confusing. You have to set the correct access scope for the virtual machine so that anyone using the VM is able to call the storage API. The documentation shows that the default access scope for storage on a VM is read-only:
When you create a new Compute Engine instance, it is automatically
configured with the following access scopes:
Read-only access to Cloud Storage:
https://www.googleapis.com/auth/devstorage.read_only
All you have to do is change this scope so that you are also able to write to storage buckets from the VM. You can find an overview of different scopes here. To apply the new scope to your VM, you have to first shut it down. Then from your local machine execute the following command:
gcloud compute instances set-scopes INSTANCE_NAME \
--scopes=storage-rw \
--zone=ZONE
You can do the same thing from the portal if you go to the settings of your VM, scroll all the way down, and choose "Set Access for each API". You have the same options when you create the VM for the first time. Below is an example of how you would do this:
In this KF example https://github.com/kubeflow/examples/blob/master/financial_time_series/tensorflow_model/ml_pipeline.py an ML pipeline gets constructed that triggers Python functions via command line.
This means that all .py files that are being called (e.g. "python3 train.py --param value") should be in the directory where the process runs. What I don't understand is where exactly should I put the .py files in the context of GCP.
Should I just copy them using Cloud shell?
Or should I add git clone <repository with .py files> into my Dockerfile?
To kickstart KFP development using python, try the following tutorial: Data passing in python components
Should I just copy them using Cloud shell? Or should I add git clone <repository with .py files> into my Dockerfile?
Ideally, the files should be inside the container image (the Dockerfile method). This ensures maximum reproducibility.
For not very complex python scripts, the Lightweight python component feature allows you to create component from a python function. In this case the script code is store in the component command-line, so you do not need to upload the code anywhere.
Putting scripts somewhere remote (e.g. cloud storage or website) is possible, but can reduce reliability and reproducibility.
I trained a model in using Keras on Azure databricks (notebook). I would like to be able to save this model on an .h5 or .pkl file and download it to my local machine.
When I train the model locally I use the following to save the file inside a directory called models, but obviously this path does not exist on Azure.
model.save('models/cnn_w2v.h5')
I am new to Azure so any help will be greatly appreciated
Correct me if I'm wrong, you are executing this line on your DataBricks notebook:
model.save('models/cnn_w2v.h5')
Right?
So if that's the case, your model is saved, but it is stored on the Azure instance that is running behind.
You need to upload this file to Azure Storage (just add code to the notebook that does that).
Later, you will be able to download it to your local machine.
I have found the answer to my question above here: how to download files from azure databricks file store
Files stored in /FileStore are accessible in your web browser at https://.cloud.databricks.com/files/. For example, the file you stored in /FileStore/my-stuff/my-file.txt is accessible at:
"https://.cloud.databricks.com/files/my-stuff/my-file.txt"
Note If you are on Community Edition you may need to replace https://community.cloud.databricks.com/files/my-stuff/my-file.txt with https://community.cloud.databricks.com/files/my-stuff/my-file.txt?o=######where the number after o= is the same as in your Community Edition URL.
Refer: https://docs.databricks.com/user-guide/advanced/filestore.html
I have a python ML process which connects to BigQuery using a local json file which the env variable GOOGLE_APPLICATION_CREDENTIALS is pointing to (The file contains my keys supplied by google, see authentication getting-started )
When Running it locally its works great.
Im now looking to deploy my model through Google's Ml engine, specifically using the shell command gcloud ml-engine jobs submit training.
However, after i ran my process and looked at the logs in console.cloud.google.com/logs/viewer i saw that gcloud cant access Bigquery and i'm getting the following error:
google.auth.exceptions.DefaultCredentialsError: File:
/Users/yehoshaphatschellekens/Desktop/google_cloud_xgboost/....-.....json was not found.
Currently i don't think that the gcloud ml-engine jobs submit training takes the Json file with it (I thought that gcloud has access automatically to BigQuery, i guess not)
One optional workaround to this is to save my personal .json into my python dependancies in the other sub-package folder (see packaging-trainer) and import it.
Is this solution feasible / safe ?
Is there any other workaround to this issue?
What i did eventually is to upload the json to a gcloud storage bucket and then uploading it into my project each time i launch the ML-engine train process:
os.system('gsutil cp gs://secured_bucket.json .')
os.environ[ "GOOGLE_APPLICATION_CREDENTIALS"] = "......json"
the path should be absolute and with backslashes in Windows:
GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\[FILE_NAME].json"
set it this way in your Python code:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "C:\PATH.JSON"
Example with the Google Translate API here.
I am building a beam pipeline on Google cloud dataflow.
I am getting an error that cloud dataflow does not have permissions to write to a temp directory.
This is confusing since clearly dataflow has the ability to write to the bucket, it created a staging folder.
Why would I be able to write a staging folder, but not a temp folder?
I am running from within a docker container on a compute engine. I am fully authenticated with my service account.
PROJECT=$(gcloud config list project --format "value(core.project)")
BUCKET=gs://$PROJECT-testing
python tests/prediction/run.py \
--runner DataflowRunner \
--project $PROJECT \
--staging_location $BUCKET/staging \
--temp_location $BUCKET/temp \
--job_name $PROJECT-deepmeerkat \
--setup_file tests/prediction/setup.py
EDIT
In response to #alex amato
Does the bucket belong to the project or is it owned by another project?
Yes, when I go the home screen for the project, this is one of four buckets listed. I commonly upload data and interact with other google cloud services (cloud vision API) from this bucket.
Would you please provide the full error message.
"(8d8bc4d7fc4a50bd): Failed to write a file to temp location 'gs://api-project-773889352370-testing/temp/api-project-773889352370-deepmeerkat.1498771638.913123'. Please make sure that the bucket for this directory exists, and that the project under which the workflow is running has the necessary permissions to write to it."
"8d8bc4d7fc4a5f8f): Workflow failed. Causes: (8d8bc4d7fc4a526c): One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions."
Can you confirm that there isn't already an existing GCS object which matches the name of the GCS folder path you are trying to use?
Yes, there is no folder named temp in the bucket.
Could you please verify the permissions you have match the members you run as
Bucket permissions have global admin
which matches my gcloud auth
#chamikara was correct. Despite inheriting credentials from my service account, cloud dataflow needs its own credentials.
Can you also give access to cloudservices account (<project-number>#developer.gserviceaccount.com) as mentioned in cloud.google.com/dataflow/security-and-permissions.
Ran into the same issue with a different cause: I had set object retention policies, which prevents manual deletions. Given that renaming triggers a deletion, this error happened.
Therefore, if anyone runs into a similar issue, investigate your temp bucket's properties and potentially lift any retention policies.
I've got similar errors while moving from DirectRunner to DataflowRunner:
Staged package XXX.jar at location 'gs://YYY/staging/XXX.jar' is inaccessible.
After I've played with the permissions, this is what I did:
at Storage Browser, clicked on Edit Bucket Permissions (for the specific bucket), added the right Storage Permission for the member ZZZ-compute#developer.gserviceaccount.com
I hope this will save future time for other users as well.