I tried to use some of the services provided by the Google cloud API, followed the tutorial, and after installing the SDK and authorization files, I started calling. But either way, the program gets stuck after it executes, and then it waits, with no results or any errors. Trying to use Java and python is the same thing. How do you troubleshoot this problem now
I use it in China, but I already use the global proxy model. Otherwise I wouldn't be able to download some of the python packages
import six
from google.cloud import language
from google.cloud.language import enums
from google.cloud.language import types
text = 'President Kennedy spoke at the White House.'
client = language.LanguageServiceClient()
if isinstance(text, six.binary_type):
text = text.decode('utf-8')
# Instantiates a plain text document.
document = types.Document(
content=text,
type=enums.Document.Type.PLAIN_TEXT)
# Detects syntax in the document. You can also analyze HTML with:
# document.type == enums.Document.Type.HTML
tokens = client.analyze_syntax(document).tokens
for token in tokens:
part_of_speech_tag = enums.PartOfSpeech.Tag(token.part_of_speech.tag)
print(u'{}: {}'.format(part_of_speech_tag.name,
token.text.content))
No useful information or error messages could be obtained, and the program did not interrupt, waiting for the result to be returned
China blocks many/most Google services (endpoints) from being accessed inside China. If you are doing this at home, this is almost a certainty.
Your best option is to signup with Alibaba Cloud and then use a VM to access Google services. Alibaba's network will "usually" get you thru the great firewall.
Related
I need to get the events for the current day from a personal Outlook calendar. I have found next to no feasible resources online besides maybe Microsoft's tutorial (https://learn.microsoft.com/en-us/graph/tutorials/python), but I do not want to build a Django app. Can anyone provide some other resources?
also: I have seen a lot of ppl calling APIs by using a GET <url> command. I cannot for the life of me understand how or where you can use this? Am I missing something crucial when it comes to using APIs?
First you should know that if you wanna call ms graph api, you need to get the access token first and add it to the request header like screenshot below. What I showed in the screenshot is create calendar events but they're similar. Therefore, you can't avoid to generate the token.
Then there're 2 ways lie in front of you, if you are composing a web app, then you can follow this section to find a suitable sample for you, and if you are composing a daemon application, that means you need to use clientcredentialflow here and you may refer to this section.
Anyway, whatever you use SDK or sending http request to call the api, you all need to choose a suitable flow to obtain access token.
For this purpose without using Microsoft Graph API via request in python, there is a PyPI package named O365.
By the following procedure you can easily read a Microsoft calendar:
install the package: pip install O365
register an application in the Microsoft Azure console and keep the application (client) id as well as client secret — this article can help you up.
check the signInAudience, it should be AzureADandPersonalMicrosoftAccount not PersonalMicrosoftAccount within Microsft Azure Manifest, otherwise, you can edit that.
next you should set delegated permission to what scopes you want, in your case it's Calendars.Read. Here's a snapshot of my configuration in Azure:
Now it's time to dive into the code:
from O365 import Account
CLIENT_ID = "xxx"
CLIENT_SECRET = "xxx"
credentials = (CLIENT_ID, CLIENT_SECRET)
scopes = ['Calendars.Read']
account = Account(credentials)
if not account.is_authenticated:
account.authenticate(scopes=scopes)
print('Authenticated!')
schedule = account.schedule()
calendar = schedule.get_default_calendar()
events = calendar.get_events(include_recurring=False)
for event in events:
print(event)
This is more of a design question on what to use within Google Cloud's infrastructure to obtain the results from a Python script.
Take the following scenario: we have over 60 projects and one central project for Stackdriver logging and the such.
It is from this central project I want to run a Python script (using Cloud Scheduler which then triggers the Cloud Function) to obtain a list of disks that haven't had their snapshot taken in the past 24 hours, those that aren't assigned to a snapshot schedule, and the snapshot schedules that have names that do not match our naming convention. I have the script already prepared, and it works very well from my workstation (producing a list of dictionaries of the desired results per project).
However, my question is: where should I send the results to? And how could I then have an email sent out to the appropriate people to action it?
I've played about with sending the object attributes to Pub/Sub within the central project, but this requires me to manually pull the messages, and I can't see any way of scheduling the pull request. I also don't see an option of sending out an email from Pub/Sub to an email address, and so the only option seems to be to create an email Cloud Function which is then triggered whenever one of the Subscriptions receives a new message from the first Cloud Function containing the original script.
I suppose I could simply set this up on one of our Windows VM instances and convert the script to PowerShell, but I was rather hoping to keep it out of a VM if at all possible.
Has anyone done this before? And if so, what did you use to get the desired results?
I think you can use Sendgrid API to send emails from your Cloud Function. It's very easy to set up it has a free plan which includes 12,000 per month and has an API for Python :D.
You can signup using the Google Cloud Marketplace selecting the free plan.
Then create an API key for your code here. If you only need to send mails I suggest you can select the option Restricted Access and for Mail Send give Full Access or the level you think will work for you.
Here's a code snippet for you:
import logging
from sendgrid import SendGridAPIClient
from sendgrid.helpers.mail import Mail, Email
from python_http_client.exceptions import HTTPError
def send_mail(request):
log = logging.getLogger(__name__)
SENDGRID_API_KEY = 'SG.blahblahblah'
sg = SendGridAPIClient(SENDGRID_API_KEY)
"""
Maybe here goes the code you use to check what you need
"""
APP_NAME = "Testing"
html_content = f"""
Here goes your mail body in HTML format
"""
message = Mail(
to_emails="dest#a.domain.com",
from_email=Email('sender#your.domain.com', "Your name or your app name"),
subject=f"Warning!!!!",
html_content=html_content
)
try:
response = sg.send(message)
log.info(f"email.status_code={response.status_code}")
return f'Your mail was sent!'
except HTTPError as e:
log.error(e)
And don't forget to add the sendgrid lib to your requirements.txt file:
# Function dependencies, for example:
# package>=version
sendgrid
Hope this can help you.
I have code below that was given to me to list Google Cloud Service Accounts for a specific Project.
import os
from googleapiclient import discovery
from gcp import get_key
"""gets all Service Accounts from the Service Account page"""
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
service = discovery.build('iam', 'v1')
project_id = 'projects/<google cloud project>'
request = service.projects().serviceAccounts().list(name=project_id)
response = request.execute()
accounts = response['accounts']
for account in accounts:
print(account['email'])
This code works perfectly and prints the accounts as I need them. What I'm trying to figure out is:
Where can I go to see how to construct code like this? I found a site that has references to the Python API Client, but I can't seem to figure out how to make the code above from it. I can see the Method to list the Service Accounts, but it's still not giving me enough information.
Is there somewhere else I should be going to educate myself. Any information you have is appreciated so I don't pull out the rest of my hair.
Thanks, Eric
Let me share with you this documentation page, where there is a detailed explanation on how to build a script such as the one you shared, and what does each line of code mean. It is extracted from the documentation of ML Engine, not IAM, but it is using the same Python Google API Client Libary, so just ignore the references to ML and the rest will be useful for you.
In any case, here it is a commented version of your code, so that you understand it better:
# Imports for the Client API Libraries and the key management
import os
from googleapiclient import discovery
from gcp import get_key
# Look for an environment variable containing the credentials for Google Cloud Platform
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
# Build a Python representation of the REST API
service = discovery.build('iam', 'v1')
# Define the Project ID of your project
project_id = 'projects/<google cloud project>'
"""Until this point, the code is general to any API
From this point on, it is specific to the IAM API"""
# Create the request using the appropriate 'serviceAccounts' API
# You can substitute serviceAccounts by any other available API
request = service.projects().serviceAccounts().list(name=project_id)
# Execute the request that was built in the previous step
response = request.execute()
# Process the data from the response obtained with the request execution
accounts = response['accounts']
for account in accounts:
print(account['email'])
Once you understand the first part of the code, the last lines are specific to the API you are using, which in this case is the Google IAM API. In this link, you can find detailed information on the methods available and what they do.
Then, you can follow the Python API Client Library documentation that you shared in order to see how to call the methods. For instance, in the code you shared, the method used depends on service, which is the Python representation of the API, and then goes down the tree of methods in the last link as in projects(), then serviceAccounts() and finally the specificlist() method, which ends up in request = service.projects().serviceAccounts().list(name=project_id).
Finally, just in case you are interested in the other available APIs, please refer to this page for more information.
I hope the comments I made on your code were of help, and that the documentation shared makes it easier for you to understand how a code like that one could be scripted.
You can use ipython having googleapiclient installed - with something like:
sudo pip install --upgrade google-api-python-client
You can go to interactive python console and do:
from googleapiclient import discovery
dir(discovery)
help(discovery)
dir - gives all entries that object has - so:
a = ''
dir(a)
Will tell what you can do with string object. Doing help(a) will give help for string object. You can do dipper:
dir(discovery)
# and then for instance
help(discovery.re)
You can call your script in steps, and see what is result print it, do some research, having something - do %history to printout your session, and have solution that can be triggered as a script.
I have a snippet of code using the pygoogle python module that allows me to programmatically search for some term in google succintly:
g = pygoogle(search_term)
g.pages = 1
results = g.get_urls()[0:10]
I just found out that this has been discontinued unfortunately and replaced by something called the google custom search. I looked at the other related questions on SO but didn't find anything I could use. I have two questions:
1) Does google custom search allow me to do exactly what I am doing in the three lines above?
2) If yes - where can I find example code to do exactly what I am doing above? If no then what is the alternative to do what I did using pygoogle?
It is possible to do this. The setup is... not very straightforward, but the end result is that you can search the entire web from python with few lines of code.
There are 3 main steps in total.
1st step: get Google API key
The pygoogle's page states:
Unfortunately, Google no longer supports the SOAP API for search, nor
do they provide new license keys. In a nutshell, PyGoogle is pretty
much dead at this point.
You can use their AJAX API instead. Take a look here for sample code:
http://dcortesi.com/2008/05/28/google-ajax-search-api-example-python-code/
... but you actually can't use AJAX API either. You have to get a Google API key. https://developers.google.com/api-client-library/python/guide/aaa_apikeys For simple experimental use I suggest "server key".
2nd step: setup Custom Search Engine so that you can search the entire web
Indeed, the old API is not available. The best new API that is available is Custom Search. It seems to support only searching within specific domains, however, after following this SO answer you can search the whole web:
From the Google Custom Search homepage ( http://www.google.com/cse/ ), click Create a Custom Search Engine.
Type a name and description for your search engine.
Under Define your search engine, in the Sites to Search box, enter at least one valid URL (For now, just put www.anyurl.com to get
past this screen. More on this later ).
Select the CSE edition you want and accept the Terms of Service, then click Next. Select the layout option you want, and then click
Next.
Click any of the links under the Next steps section to navigate to your Control panel.
In the left-hand menu, under Control Panel, click Basics.
In the Search Preferences section, select Search the entire web but emphasize included sites.
Click Save Changes.
In the left-hand menu, under Control Panel, click Sites.
Delete the site you entered during the initial setup process.
This approach is also recommended by Google: https://support.google.com/customsearch/answer/2631040
3rd step: install Google API client for Python
pip install google-api-python-client, more info here:
repo: https://github.com/google/google-api-python-client
more info: https://developers.google.com/api-client-library/python/apis/customsearch/v1
complete docs: https://api-python-client-doc.appspot.com/
4th step (bonus): do the search
So, after setting this up, you can follow the code samples from few places:
simple example: https://github.com/google/google-api-python-client/blob/master/samples/customsearch/main.py
cse() function docs: https://google-api-client-libraries.appspot.com/documentation/customsearch/v1/python/latest/customsearch_v1.cse.html
and end up with this:
from googleapiclient.discovery import build
import pprint
my_api_key = "Google API key"
my_cse_id = "Custom Search Engine ID"
def google_search(search_term, api_key, cse_id, **kwargs):
service = build("customsearch", "v1", developerKey=api_key)
res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
return res['items']
results = google_search(
'stackoverflow site:en.wikipedia.org', my_api_key, my_cse_id, num=10)
for result in results:
pprint.pprint(result)
After some tweaking you could write some functions that behave exactly like your snippet, but I'll skip this step here.
#mbdevpl's response helped me a lot, so all credit goes to them.
But there have been a few changes in the UI, so here is an update:
A. Install google-api-python-client
If you don't already have a Google account, sign up.
If you have never created a Google APIs Console project, read the Managing Projects page and create a project in the Google API Console.
Install the library.
B. To create an API key:
Navigate to the APIs & Services→Credentials panel in Cloud Console.
Select Create credentials, then select API key from the drop-down menu.
The API key created dialog box displays your newly created key.
You now have an API_KEY
C. Setup Custom Search Engine so you can search the entire web
Create a custom search engine in this link.
In Sites to search, add any valid URL (i.e. www.stackoverflow.com).
That’s all you have to fill up, the rest doesn’t matter. In the left-side menu, click Edit search engine → {your search engine name} → Setup
Set Search the entire web to ON.
Remove the URL you added from the list of Sites to search.
Under Search engine ID you’ll find the search-engine-ID.
Search example
from googleapiclient.discovery import build
my_api_key = "AIbaSyAEY6egFSPeadgK7oS/54iQ_ejl24s4Ggc" #The API_KEY you acquired
my_cse_id = "012345678910111213141:abcdef10g2h" #The search-engine-ID you created
def google_search(search_term, api_key, cse_id, **kwargs):
service = build("customsearch", "v1", developerKey=api_key)
res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
return res['items']
results = google_search('"god is a woman" "thank you next" "7 rings"', my_api_key, my_cse_id, num=10)
for result in results:
print(result)
Important! on the first run, you might have to enable the API in your account. The error message should contain the link to enable the API in. It will be something like:
https://console.developers.google.com/apis/api/customsearch.googleapis.com/overview?project={your project name}.
You’ll be asked to create a service name (It doesn’t matter what it is), and give it Roles.
I gave it Role Viewer and Service Usage Admin and it works.
Answer from 2020
Google aren't providing any API anymore for some reason, but https://github.com/bisoncorps/search-engine-parser is developing a python package for scraping Google.
Installation
pip install search-engine-parser
Usage
from search_engine_parser import GoogleSearch
def google(query):
search_args = (query, 1)
gsearch = GoogleSearch()
gresults = gsearch.search(*search_args)
return gresults['links']
google('Is it illegal to scrape google results')
I don't know how legal this is, but as long as you aren't commercializing your product I think you can get away with it. Besides Google haven't really sued anyone because of using their product, they have just banned their IP address.
For more information Is it ok to scrape data from Google results?
I have problems with the authentication in the Python Library of Google Cloud API.
At first it worked for some days without problem, but suddenly the API calls are not showing up in the API Overview of the Google CloudPlatform.
I created a service account and stored the json file locally. Then I set the environment variable GCLOUD_PROJECT to the project ID and GOOGLE_APPLICATION_CREDENTIALS to the path of the json file.
from google.cloud import speech
client = speech.Client()
print(client._credentials.service_account_email)
prints the correct service account email.
The following code transcribes the audio_file successfully, but the Dashboard for my Google Cloud project doesn't show anything for the activated Speech API Graph.
import io
with io.open(audio_file, 'rb') as f:
audio = client.sample(f.read(), source_uri=None, sample_rate=48000, encoding=speech.encoding.Encoding.FLAC)
alternatives = audio.sync_recognize(language_code='de-DE')
At some point the code also ran in some errors, regarding the usage limit. I guess due to the unsuccessful authentication, the free/limited option is used somehow.
I also tried the alternative option for authentication by installing the Google Cloud SDK and gcloud auth application-default login, but without success.
I have no idea where to start troubleshooting the problem.
Any help is appreciated!
(My system is running Windows 7 with Anaconda)
EDIT:
The error count (Fehler) is increasing with calls to the API. How can I get detailed information about the error?!
Make sure you are using an absolute path when setting the GOOGLE_APPLICATION_CREDENTIALS environment variable. Also, you might want to try inspecting the access token using OAuth2 tokeninfo and make sure it has "scope": "https://www.googleapis.com/auth/cloud-platform" in its response.
Sometimes you will get different error information if you initialize the client with GRPC enabled:
0.24.0:
speech_client = speech.Client(_use_grpc=True)
0.23.0:
speech_client = speech.Client(use_gax=True)
Usually it's an encoding issue, can you try with the sample audio or try generating LINEAR16 samples using something like the Unix rec tool:
rec --channels=1 --bits=16 --rate=44100 audio.wav trim 0 5
...
with io.open(speech_file, 'rb') as audio_file:
content = audio_file.read()
audio_sample = speech_client.sample(
content,
source_uri=None,
encoding='LINEAR16',
sample_rate=44100)
Other notes:
Sync Recognize is limited to 60 seconds of audio, you must use async for longer audio
If you haven't already, set up billing for your account
With regards to the usage problem, the issue is in fact that when you use the new google-cloud library to access ML APIs, it seems everyone authenticates to a project shared by everyone (hence it says you've used up your limit even though you've not used anything). To check and confirm this, you can call an ML API that you have not enabled by using the python client library, which will give you a result even though it shouldn't. This problem persists to other language client libraries and OS, so I suspect it's an issue with their grpc.
Because of this, to ensure consistency I always use the older googleapiclient that uses my API key. Here is an example to use the translate API:
from googleapiclient import discovery
service = discovery.build('translate', 'v2', developerKey='')
service_request = service.translations().list(q='hello world', target='zh')
result = service_request.execute()
print(result)
For the speech API, it's something along the lines of:
from googleapiclient import discovery
service = discovery.build('speech', 'v1beta1', developerKey='')
service_request = service.speech().syncrecognize()
result = service_request.execute()
print(result)
You can get the list of the discovery APIs at https://developers.google.com/api-client-library/python/apis/ with the speech one located in https://developers.google.com/resources/api-libraries/documentation/speech/v1beta1/python/latest/.
One of the other benefits of using the discovery library is that you get a lot more options compared to the current library, although often times it's a bit more of a pain to implement.