Code:
from google.cloud import storage
client = storage.Client()
bucket = ['symbol_wise_nse', 'symbol_wise_final']
for i in bucket:
if client.get_bucket(i).exists():
BUCKET = client.get_bucket(i)
if the bucket exists i want to do client.get_bucket. How to check whether the bucket exists or not?
Another option that doesn't use try: except is:
from google.cloud import storage
client = storage.Client()
bucket = ['symbol_wise_nse', 'symbol_wise_final']
for i in bucket:
BUCKET = client.bucket(i)
if BUCKET.exists():
BUCKET = client.get_bucket(i)
There is no method to check if the bucket exists or not, however you will get an error if you try to access a non existent bucket.
I would recommend you to either list the buckets in the project with storage_client.list_buckets() and then use the response to confirm if the bucket exists in your code, or if you wish to perform the client.get_bucket in every bucket in your project, you can just iterate through the response directly.
Hope you find this information useful
You can use something like this:
from google.cloud import storage
client = storage.Client()
buckets = ['symbol_wise_nse', 'symbol_wise_final']
for i in buckets:
try:
bucket = client.get_bucket(i)
print(bucket)
except:
pass
The following worked for me (re-using params in question):
from google.cloud import storage
from google.cloud.storage import Bucket
client = storage.Client()
exists = Bucket(client, 'symbol_wise_nse').exists()
Related
I am using the code below to upload an image to Firebase storage, which is does successfully but without an access token. How would I add an access token? I have looked on stackoverflow but none of the implementations have worked for me
import os
from google.cloud import storage
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = 'C:///file.json'
storage_client = storage.Client()
bucket = storage_client.get_bucket('name.appspot.com')
imageBlob = bucket.blob("/")
imagePath = "C:\\\image.jpg"
imageBlob = bucket.blob("image.jpg")
imageBlob.upload_from_filename(imagePath)
I'm trying to make a cloud function using python, which reads json files containing schemas of tables from a directory in the cloud storage and from these schemas I need to create tables in bigquery.
I had some attempts to access cloud storage, but without success, previously I developed something similar in google colab, reading these schemas from a directory on the drive, but now things seem quite different.
Can someone help me?
You can check the Streaming data from Cloud Storage into BigQuery using Cloud Functions solution guide of GCP.
If you'd like a different approach you can refer to the download object guide at the GCP doc to retrieve the data from GCS, see the sample code below.
from google.cloud import storage
def download_blob(bucket_name, source_blob_name, destination_file_name):
"""Downloads a blob from the bucket."""
# bucket_name = "your-bucket-name"
# source_blob_name = "storage-object-name"
# destination_file_name = "local/path/to/file"
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(source_blob_name)
blob.download_to_filename(destination_file_name)
print(
"Blob {} downloaded to {}.".format(
source_blob_name, destination_file_name
)
)
You can create Cloud Function and read the download data from the file in Cloud Storage
def loader(event, context):
"""Triggered by a change to a Cloud Storage bucket.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
try:
file_name = event['name']
bucket_name = event['bucket']
client = storage.Client()
bucket = client.get_bucket(bucket_name)
file_blob = storage.Blob(file_name, bucket)
data = file_blob.download_as_string().decode()
Once you get the data you can create table in BigQuery.
What would be the client alternate for this?
import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('name')
I want to use client but that has a list_buckets function. Is there a way to pass a bucket name to client instead of getting the details from the result array??
This will give you the bucket names
import boto3
s3 = boto3.resource('s3')
buckets = s3.buckets.all()
for bucket in buckets:
print(bucket.name)
I don't think you can only get the names.
But there is a tricky way to use the client
import boto3
s3 = boto3.resource('s3')
s3.meta.client.list_buckets()['Buckets']
where it gives the client responses.
I know bucket name, I have access to it, I can browse it by web and by awscli.
How access it by Python's boto3? All examples assume accessing my own buckets:
import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket.name)
How to reach other's bucket?
If you have access to someone else's bucket and you know the name of that bucket you can access it like
import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('some-bucket-i-have-access-to')
for obj in bucket.objects.all():
print(obj.key)
I am Python and Google Cloud Storage newbie.
I am writing a python script to get a file list from Google Cloud Storage bucket using Google Cloud Python Client Library and list_blobs() function from Bucket class is not working as I expected.
https://googlecloudplatform.github.io/google-cloud-python/stable/storage-buckets.html
Here is my python code:
from google.cloud import storage
from google.cloud.storage import Blob
client = storage.Client.from_service_account_json(service_account_json_key_path, project_id)
bucket = client.get_bucket(bucket_id)
print(bucket.list_blobs())
If I understood the documentation correctly, print(bucket.list_blobs()) should print something like this: ['testfile.txt', 'testfile2.txt'].
However, my script printed this:
"google.cloud.storage.bucket._BlobIterator object at 0x7fdfdbcac590"
delete_blob() documentation has example code same as mine.
https://googlecloudplatform.github.io/google-cloud-python/stable/storage-buckets.html
I am not sure what I am doing wrong here.
Any pointers/examples/answers will be greatly appreciated. Thanks!
An example list function:
def list_blobs(bucket_name):
"""Lists all the blobs in the bucket."""
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blobs = bucket.list_blobs()
for blob in blobs:
print(blob.name)
What do you see if you:
for blob in bucket.list_blobs():
print(blob)
print(blob.name)