After following this tutorial, I am able to run a script that prints all the details of my database. However, I have no clue as to how to do something with said database! Here's my code:
from google.oauth2 import service_account
import googleapiclient.discovery
import json
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = 'credentials.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
sqladmin = googleapiclient.discovery.build('sqladmin', 'v1beta4', credentials=credentials)
response = sqladmin.instances().list(project='single-router-309308').execute()
print(json.dumps(
response, sort_keys=True, indent=2))
sqladmin.close()
Which prints all the info. I tried various things to reach my table, products, but I can't get it to work and keep getting a AttributeError: 'Resource' object has no attribute 'execute' (or 'list') exception. I tried stuff like this:
response = sqladmin.projects().list().execute()
To view my tables as well, but it doesn't work. I believe this is the correct approach since I can connect, but I haven't figured it out yet. Anybody know the answer?
As per the documentation, you should be able to get access to your table using the below code.
Note that you have an sql project, then an instance on the project, then a database in the instance, then your table is nested inside that database.
from pprint import pprint
# Project ID of the project that contains the instance.
project = 'single-router-309308'
# Database instance ID. You should have this from the above printout
instance = 'my-instance'
# Name of the database in the instance. You can look this up if you arent sure by logging into google cloud for your project. Your table is inside this database.
database = 'my-database'
request = service.databases().get(project=project, instance=instance, database=database)
response = request.execute() #returns a dictionary with the data
pprint(response)
I would suggest you take a look at the REST API references for CLoud SQL for MySQL for further reading (https://cloud.google.com/sql/docs/mysql/admin-api/rest/v1beta4/databases/get).
Related
I'm trying to set the environment variable from a dict but getting and error when connecting.
#service account pulls in airflow variable that contains the json dict with service_account credentials
service_account = Variable.get('google_cloud_credentials')
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=str(service_account)
error
PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.</Details></Error>'
when reading if I use and point to file then there are no issues.
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=/file/path/service_account.json
I'm wondering is there a way to convert the dict object to an os path like object? I don't want to store the json file on the container and airflow/google documentation isn't clear at all.
The Python stringio package lets you create a file-like object backed by a string, but that won't help here because the consumer of this environment variable is expecting a file path, not a file-like object. I don't think it's possible to do what you're trying to do. Is there a reason you don't want to just put the credentials in a file?
There is a way to do it, but the Google documentation is terrible. So I wrote a Github gist to document the recipe that I and a colleague (Imee Cuison) developed to use the key securely. Sample code below:
import json
from google.oauth2.service_account import Credentials
from google.cloud import secretmanager
def access_secret(project_id:str, secret_id:str, version_id:str="latest")->str:
"""Return the secret in string format"""
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
# Access the secret version.
response = client.access_secret_version(name=name)
# Return the decoded payload.
return response.payload.data.decode('UTF-8')
def get_credentials_from_token(token:str)->Credentials:
"""Given an authentication token, return a Credentials object"""
credential_dict = json.loads(secret_payload)
return Credentials.from_service_account_info(credential_dict)
credentials_secret = access_secret("my_project", "my_secret")
creds = get_credentials_from_token(credentials_secret)
# And now you can use the `creds` Credentials object to authenticate to an API
Putting the service account into the repository is not a good practice. As a best practice; You need to use authentication propagating from the default google auth within your application.
For instance, using Google Cloud Kubernetes you can use the following python code :
from google.cloud.container_v1 import ClusterManagerClient
credentials, project = google.auth.default(
scopes=['https://www.googleapis.com/auth/cloud-platform', ])
credentials.refresh(google.auth.transport.requests.Request())
cluster_manager = ClusterManagerClient(credentials=credentials)
Is there a way to retrieve data from Google Cloud Storage without third party apps?
I tried with python but I'm getting error code below.
import json
from httplib2 import Http
from oauth2client.client import SignedJwtAssertionCredentials
from apiclient.discovery import build
# Change these variables to fit your case
client_email = *******.iam.gserviceaccount.com
json_file = C:\******
cloud_storage_bucket = pubsite_prod_rev_********
report_to_download = installs_********_201901_app_version
private_key = json.loads(open(json_file).read())[my private key here]
credentials = SignedJwtAssertionCredentials(*******#gmail.com,my_private_key here),
storage = build('storage', 'v1', http=credentials.authorize(Http()))
print storage.objects().get()
bucket=cloud_storage_bucket
object=report_to_download).execute()
Python throws this error:
multiple statements found while compiling a single statement
The error message comes from your incorrect Python format, syntax or indentation.
Look at the last statement you want to execute, the bucket and object should be inside the get().
print storage.objects().get(
bucket=cloud_storage_bucket,
object=report_to_download).execute()
I need to update certain node in my Firebase DB, so this is what I'm doing:
from firebase_admin import db
def update_data_in_firebase(gid, account_id, location_id, data_to_update):
firebase_url = 'saved_locations/{}/accounts/{}/locations/{}'.format(gid, account_id, location_id)
ref = db.reference(path=firebase_url)
ref.update(data_to_update)
So, the code above is what I'm trying to do to update the data in the Firebase node, but I'm getting this error:
Invalid databaseURL option: "None". databaseURL must be a non-empty URL string.
Of course, I checked out the firebase URL and it matches, so the problem is not the URL, or, I'm missing something with the path, I mean, should I use absolute insted of relative path.
As mentioned in the comments of the question, the databaseURL was not defined.
Answer:
cred = credentials.Certificate('your_config.json')
firebase_admin = firebase_admin.initialize_app(cred, {'databaseURL': 'https://your-firebase-db'})
In the main docs of Firebase, I couldn't find the error on my app initialization:
Firebase Admin Docs
But in the Realtime Database - Admin (Get Started), there is a snippet where they initialize the Firebase App defining the databaseURL
I'm trying to use python to connect to Salesforce Bulk API. However, I don't want to test my code on the real salesforce. I want to test with my sandbox. However, I don't know how to connect to sandbox only... I've tried to add sandbox=True but it doesn't work...
import salesforce_bulk
bulk=salesforce_bulk.SalesforceBulk(username="username",password="password")
The advice here may be a bit deprecated. I was able to get the bulk uploads working by combining the Salesforce and SalesforceBulk libraries. Note the domain that I am passing to the api as well the sand_box name that needs to be appended to the username.
from simple_salesforce import Salesforce
from salesforce_bulk import SalesforceBulk
import json
from salesforce_bulk.util import IteratorBytesIO
from urllib.parse import urlparse
USER = "user#domain.com.<sandbox_name>"
PASS = "pass"
SEC_TOKEN = "token"
DOMAIN = "<domain>--<sandbox_name>.<instance>.my"
sf = Salesforce(username=USER, password=PASS, security_token=SEC_TOKEN, domain=DOMAIN)
bulk = SalesforceBulk(sessionId=sf.session_id, host=sf.sf_instance)
job = bulk.create_query_job("table", contentType='JSON')
batch = bulk.query(job, "select Id,LastName from table limit 5000")
bulk.close_job(job)
while not bulk.is_batch_done(batch):
sleep(10)
for result in bulk.get_all_results_for_query_batch(batch):
result = json.load(IteratorBytesIO(result))
for row in result:
print(row)
Old question but I had the same problem today, so maybe this will help someone.
This is a complete hack, but it works - probably a better hack would be to do this using salesforce-oauth-request (which does have a "sandbox=True" option), but I was logging in via beatbox anyway, so tried this first.
Gist is you log in to the sandbox using beatbox (which lets you specify your serverUrl) and then use that sessionId and instance_url to log in through salesforce_bulk.
import beatbox
from salesforce_bulk import SalesforceBulk
# log in to sandbox using beatbox
service = beatbox.PythonClient()
service.serverUrl = 'https://test.salesforce.com/services/Soap/u/20.0'
user = 'user#user.com'
password = 'secret'
token = '12345'
service.login(user, password+token)
# the _Client_serverUrl has the instance url + some
# SOAP stuff, so we need to strip that off
groups = service._Client__serverUrl.split('/')
instance_url = '/'.join(groups[:3])
# now we can use the instance_url and sessionId to
# log into Salesforce through SalesforceBulk
bulk = sfdc_bulk_connect(instance_url, service.sessionId)
Have you check the package is install
This library will use the salesforce-oauth-request package (which you must install) to run the Salesforce OAUTH2 Web flow and return an access token.
And the password did you have token or not ?
I would like to develop an app engine application that directly stream data into a BigQuery table.
According to Google's documentation there is a simple way to stream data into bigquery:
http://googlecloudplatform.blogspot.co.il/2013/09/google-bigquery-goes-real-time-with-streaming-inserts-time-based-queries-and-more.html
https://developers.google.com/bigquery/streaming-data-into-bigquery#streaminginsertexamples
(note: in the above link you should select the python tab and not Java)
Here is the sample code snippet on how streaming insert should be coded:
body = {"rows":[
{"json": {"column_name":7.7,}}
]}
response = bigquery.tabledata().insertAll(
projectId=PROJECT_ID,
datasetId=DATASET_ID,
tableId=TABLE_ID,
body=body).execute()
Although I've downloaded the client api I didn't find any reference to a "bigquery" module/object referenced in the above Google's example.
Where is the the bigquery object (from snippet) should be located?
Can anyone show a more complete way to use this snippet (with the right imports)?
I've Been searching for that a lot and found documentation confusing and partial.
Minimal working (as long as you fill in the right ids for your project) example:
import httplib2
from apiclient import discovery
from oauth2client import appengine
_SCOPE = 'https://www.googleapis.com/auth/bigquery'
# Change the following 3 values:
PROJECT_ID = 'your_project'
DATASET_ID = 'your_dataset'
TABLE_ID = 'TestTable'
body = {"rows":[
{"json": {"Col1":7,}}
]}
credentials = appengine.AppAssertionCredentials(scope=_SCOPE)
http = credentials.authorize(httplib2.Http())
bigquery = discovery.build('bigquery', 'v2', http=http)
response = bigquery.tabledata().insertAll(
projectId=PROJECT_ID,
datasetId=DATASET_ID,
tableId=TABLE_ID,
body=body).execute()
print response
As Jordan says: "Note that this uses the appengine robot to authenticate with BigQuery, so you'll to add the robot account to the ACL of the dataset. Note that if you also want to use the robot to run queries, not just stream, you need the robot to be a member of the project 'team' so that it is authorized to run jobs."
Here is a working code example from an appengine app that streams records to a BigQuery table. It is open source at code.google.com:
http://code.google.com/p/bigquery-e2e/source/browse/sensors/cloud/src/main.py#124
To find out where the bigquery object comes from, see
http://code.google.com/p/bigquery-e2e/source/browse/sensors/cloud/src/config.py
Note that this uses the appengine robot to authenticate with BigQuery, so you'll to add the robot account to the ACL of the dataset.
Note that if you also want to use the robot to run queries, not just stream, you need to robot to be a member of the project 'team' so that it is authorized to run jobs.