Develop Reference

Pythonanywhere: Twitter API authenticates from bash, but fails on scheduled task

I'm using the python-twitter (not tweepy) module in my application.
I have a script set up (maketweet.py) that, when I run it from bash / console, works successfully (i.e. it makes a tweet).
The problem I'm having is that when I run the same script as a scheduled task, I get an error:
Traceback (most recent call last):
File "/home/dir1/dir2/proj/maketweet.py", line 21, in <module>
api = create_api()
File "/home/dir1/dir2/proj/config.py", line 31, in create_api
if api.VerifyCredentials():
File "/home/dir1/.local/lib/python3.8/site-packages/twitter/api.py", line 4699, in VerifyCredentials
resp = self._RequestUrl(url, 'GET', data)
File "/home/dir1/.local/lib/python3.8/site-packages/twitter/api.py", line 4959, in _RequestUrl
raise TwitterError("The twitter.Api instance must be authenticated.")
twitter.error.TwitterError: The twitter.Api instance must be authenticated.
The other problem I'm having is that I can't conceive why it would make a difference that I'm using a scheduled task, rather than running the file directly from bash. Here are the contents of config.py:
#!/usr/bin/python3.8
import twitter
import os
import logging
from dotenv import load_dotenv
logger = logging.getLogger()
# project folder is one level up from file location
project_folder = pathlib.Path(__file__).parent.absolute()
load_dotenv(os.path.join(project_folder, '.env'))
# Authenticate to Twitter and create API object
TWITTER_CONSUMER_API_KEY = os.getenv("TWITTER_CONSUMER_API_KEY")
TWITTER_CONSUMER_API_SECRET = os.getenv("TWITTER_CONSUMER_API_SECRET")
TWITTER_ACCESS_TOKEN = os.getenv("TWITTER_ACCESS_TOKEN")
TWITTER_ACCESS_SECRET = os.getenv("TWITTER_ACCESS_SECRET")
def create_api():
# Create API object
api = twitter.Api(
access_token_key = TWITTER_ACCESS_TOKEN,
access_token_secret = TWITTER_ACCESS_SECRET,
consumer_key = TWITTER_CONSUMER_API_KEY,
consumer_secret = TWITTER_CONSUMER_API_SECRET,
sleep_on_rate_limit=True)
# test API object
if api.VerifyCredentials():
pass
else:
logger.error("Error creating API", exc_info=True)
raise Exception("Twitter user authentication error")
logger.info("API created")
return api
Obviously there's an error in creating the API. I imagine this has something to do with the environment variables and how they are accessed through scheduled tasks vs. bash. Just really not sure how to figure this one out...

mediaItems.search_next() returns 400

Unable to get all results of mediaItems.search:
photos = google.get_service(credentials, 'photoslibrary', 'v1')
request = photos.albums().list(pageSize=50)
while request is not None:
result = request.execute()
for album in result['albums']:
request2 = photos.mediaItems().search(body={'albumId': album['id']})
while request2 is not None:
result2 = request2.execute()
request2 = photos.mediaItems().search_next(request2, result2)
print('nextpageToken' in result2, request2)
request = photos.albums().list_next(request, result)
Running this fails on the first search_next() call with
[...]
File "/usr/local/lib/python3.7/dist-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/googleapiclient/http.py", line 851, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://photoslibrary.googleapis.com/v1/mediaItems:search?alt=json returned "Invalid JSON payload received. Unexpected end of string. Expected an object key or }.
That library isn't really supported it seems, so that might be the problem, or am I missing something here?

The google-api-python-client is generic client for all the Google's discovery based APIs and as such it supports all the API based on this protocol including the Photos one.
The correct way to use the services is to invoke the build method and after that use the available methods of the services.
Beside this you always want to use list_next as search_next does not exists.
Here is an example of the photos API working on my laptop (python 3.6)
import os
import pickle
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
SCOPES = ['https://www.googleapis.com/auth/photoslibrary.readonly', ]
# we check if we save the credentials in the past and we reuse them
if not os.path.exists('credentials.dat'):
# no credentials found, we run the standard auth flow
flow = InstalledAppFlow.from_client_secrets_file('client_id.json', SCOPES)
credentials = flow.run_local_server()
with open('credentials.dat', 'wb') as credentials_dat:
pickle.dump(credentials, credentials_dat)
else:
with open('credentials.dat', 'rb') as credentials_dat:
credentials = pickle.load(credentials_dat)
if credentials.expired:
credentials.refresh(Request())
photos_sdk = build('photoslibrary', 'v1', credentials=credentials)
# photos API
photos_albums_api = photos_sdk.albums()
photos_mediaitems_api = photos_sdk.mediaItems()
albums_list_params = {
'pageSize': 50,
}
# first request
albums_list_req = photos_albums_api.list(**albums_list_params)
while albums_list_req is not None:
photos_albums_list = albums_list_req.execute()
# print(photos_albums_list)
for album in photos_albums_list['albums']:
print(album['title'])
mediaitems_search_req = photos_mediaitems_api.search(body={'albumId': album['id']})
while mediaitems_search_req is not None:
mediaitems_search = mediaitems_search_req.execute()
print(mediaitems_search)
# mediaItems pagination management
mediaitems_search_req = photos_mediaitems_api.list_next(mediaitems_search_req, mediaitems_search)
# albums pagination handling
albums_list_req = photos_albums_api.list_next(albums_list_req, photos_albums_list)

If there are more results than the specified pageSize, the API returns a pageToken, you should use to request the next part. See the example here Access Google Photo API with Python using google-api-python-client

Google Client API v3 - update a file on drive using Python

I'm trying to update the content of a file from a python script using the google client api. The problem is that I keep receiving error 403:
An error occurred: <HttpError 403 when requesting https://www.googleapis.com /upload/drive/v3/files/...?alt=json&uploadType=resumable returned "The resource body includes fields which are not directly writable.
I have tried to remove metadata fields, but didn't help.
The function to update the file is the following:
# File: utilities.py
from googleapiclient import errors
from googleapiclient.http import MediaFileUpload
from googleapiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools
def update_file(service, file_id, new_name, new_description, new_mime_type,
new_filename):
"""Update an existing file's metadata and content.
Args:
service: Drive API service instance.
file_id: ID of the file to update.
new_name: New name for the file.
new_description: New description for the file.
new_mime_type: New MIME type for the file.
new_filename: Filename of the new content to upload.
new_revision: Whether or not to create a new revision for this file.
Returns:
Updated file metadata if successful, None otherwise.
"""
try:
# First retrieve the file from the API.
file = service.files().get(fileId=file_id).execute()
# File's new metadata.
file['name'] = new_name
file['description'] = new_description
file['mimeType'] = new_mime_type
file['trashed'] = True
# File's new content.
media_body = MediaFileUpload(
new_filename, mimetype=new_mime_type, resumable=True)
# Send the request to the API.
updated_file = service.files().update(
fileId=file_id,
body=file,
media_body=media_body).execute()
return updated_file
except errors.HttpError as error:
print('An error occurred: %s' % error)
return None
And here there is the whole script to reproduce the problem.
The goal is to substitute a file, retrieving its id by name.
If the file does not exist yet, the script will create it by calling insert_file (this function works as expected).
The problem is update_file, posted above.
from __future__ import print_function
from utilities import *
from googleapiclient import errors
from googleapiclient.http import MediaFileUpload
from googleapiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools
def get_authenticated(SCOPES, credential_file='credentials.json',
token_file='token.json', service_name='drive',
api_version='v3'):
# The file token.json stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
store = file.Storage(token_file)
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets(credential_file, SCOPES)
creds = tools.run_flow(flow, store)
service = build(service_name, api_version, http=creds.authorize(Http()))
return service
def retrieve_all_files(service):
"""Retrieve a list of File resources.
Args:
service: Drive API service instance.
Returns:
List of File resources.
"""
result = []
page_token = None
while True:
try:
param = {}
if page_token:
param['pageToken'] = page_token
files = service.files().list(**param).execute()
result.extend(files['files'])
page_token = files.get('nextPageToken')
if not page_token:
break
except errors.HttpError as error:
print('An error occurred: %s' % error)
break
return result
def insert_file(service, name, description, parent_id, mime_type, filename):
"""Insert new file.
Args:
service: Drive API service instance.
name: Name of the file to insert, including the extension.
description: Description of the file to insert.
parent_id: Parent folder's ID.
mime_type: MIME type of the file to insert.
filename: Filename of the file to insert.
Returns:
Inserted file metadata if successful, None otherwise.
"""
media_body = MediaFileUpload(filename, mimetype=mime_type, resumable=True)
body = {
'name': name,
'description': description,
'mimeType': mime_type
}
# Set the parent folder.
if parent_id:
body['parents'] = [{'id': parent_id}]
try:
file = service.files().create(
body=body,
media_body=media_body).execute()
# Uncomment the following line to print the File ID
# print 'File ID: %s' % file['id']
return file
except errors.HttpError as error:
print('An error occurred: %s' % error)
return None
# If modifying these scopes, delete the file token.json.
SCOPES = 'https://www.googleapis.com/auth/drive'
def main():
service = get_authenticated(SCOPES)
# Call the Drive v3 API
results = retrieve_all_files(service)
target_file_descr = 'Description of deploy.py'
target_file = 'deploy.py'
target_file_name = target_file
target_file_id = [file['id'] for file in results if file['name'] == target_file_name]
if len(target_file_id) == 0:
print('No file called %s found in root. Create it:' % target_file_name)
file_uploaded = insert_file(service, target_file_name, target_file_descr, None,
'text/x-script.phyton', target_file_name)
else:
print('File called %s found. Update it:' % target_file_name)
file_uploaded = update_file(service, target_file_id[0], target_file_name, target_file_descr,
'text/x-script.phyton', target_file_name)
print(str(file_uploaded))
if __name__ == '__main__':
main()
In order to try the example, is necessary to create a Google Drive API from https://console.developers.google.com/apis/dashboard,
then save the file credentials.js and pass its path to get_authenticated(). The file token.json will be created after the first
authentication and API authorization.

The problem is that the metadata 'id' can not be changed when updating a file, so it should not be in the body. Just delete it from the dict:
# File's new metadata.
del file['id'] # 'id' has to be deleted
file['name'] = new_name
file['description'] = new_description
file['mimeType'] = new_mime_type
file['trashed'] = True
I tried your code with this modification and it works

I also struggled a little bit with the function and found if you don't have to update the metadata then just remove them in the update function like :updated_file = service.files().update(fileId=file_id, media_body=media_body).execute()
At Least that worked for me

The problem is The resource body includes fields which are not directly writable. So try removing all of the metadata properties and then add them back one by one. The one I would be suspicious about is trashed. Even though the API docs say this is writable, it shouldn't be. Trashing a file has side effects beyond setting a boolean. Updating a file and setting it to trashed at the same time is somewhat unusual. Are you sure that's what you intend?

How to trigger a REST API based on a condition in Flask?

I have a Flask application where I let users access third party applications and fetch data from them and perform some visualizations.Now the user has to provide the application name and it's credentials in order to fetch the data.Now I want to avoid putting the application name in the url and rather all of the data should be sent as a POST request where I will parse the POST data, connect to the required app with the given credentials and perform some visualizations.This is what the user will send as a POST data
{
"application_name": "appdynamics",
"account_id": "sdf632sef",
"username": "kuhku86tg",
"password": "oihsd832"
}
Now I want to trigger my particular REST API class based on the application name provided by the user.
The way I planned was to create a seperate file that involves getting the POST data using request parser and then calling it in the main application where I will trigger my REST API class with a if condition based on the application name.Below is the file parse.py
from flask_restful import reqparse
# create a parser object
parser = reqparse.RequestParser()
# add agruments to the parser object
parser.add_argument('account_id', type=str, required=False, help="Please define 'account_id'")
parser.add_argument('username', type=str, required=False, help="Please define 'username'")
parser.add_argument('password', type=str, required=False, help="Please define 'password'")
parser.add_argument('application_name', type=str, required=False, help="Please define 'application name'")
data = parser.parse_args()
Now I call it in the main application app.py
from parser import data
from flask import Flask
from flask_restful import Api
app = Flask(__name__)
# create an API for the Flask app
api = Api(app)
# if the user demands info for appdynamics, trigger the Appdynamics API class
if data['application_name'] == "appdynamics":
api.add_resource(AppdynamicsAPI, "/<string:name>") # the string will contain the metric requirement
if __name__ == "__main__":
app.run(port=5000, debug=True)
Below is the section where the logic for the REST API is written
from parser import data
from flask_restful import Resource, reqparse
from fetch_data.appdynamics import fetch_all_apps, fetch_avg_resp_time, calls_per_min
from models.user import *
class AppdynamicsAPI(Resource):
# authenticate users
def post(self, name):
first_data = data
# if the user passes the credentials, insert it into the database otherwise use the last known credentials
# ensure you only insert valid credentials
if all([first_data['account_id'], first_data['password'], first_data['username']]):
users.update(first_data, {i: j for i, j in first_data.items()}, upsert=True)
print({i: j for i, j in first_data.items()})
credentials = users.find_one({})
print("Credentials", credentials)
account_id = credentials['account_id']
username = credentials['username']
password = credentials['password']
t_duration = first_data['t_duration']
if name == "allapps":
status_code, result = fetch_all_apps(account_id, username, password)
if status_code == 200:
return {"information": result}, status_code
return {"message": "Please enter correct credentials"}, status_code
However I receive the below error
Traceback (most recent call last):
File "/home/souvik/PycharmProjects/ServiceHandler/app.py", line 3, in <module>
from resource.appdynamics_resource import AppdynamicsAPI
File "/home/souvik/PycharmProjects/ServiceHandler/resource/appdynamics_resource.py", line 4, in <module>
from authentication.parser import data
File "/home/souvik/PycharmProjects/ServiceHandler/authentication/parser.py", line 14, in <module>
data = parser.parse_args()
File "/home/souvik/utorapp/lib/python3.5/site-packages/flask_restful/reqparse.py", line 302, in parse_args
req.unparsed_arguments = dict(self.argument_class('').source(req)) if strict else {}
File "/home/souvik/utorapp/lib/python3.5/site-packages/werkzeug/local.py", line 364, in <lambda>
__setattr__ = lambda x, n, v: setattr(x._get_current_object(), n, v)
File "/home/souvik/utorapp/lib/python3.5/site-packages/werkzeug/local.py", line 306, in _get_current_object
return self.__local()
File "/home/souvik/utorapp/lib/python3.5/site-packages/flask/globals.py", line 37, in _lookup_req_object
raise RuntimeError(_request_ctx_err_msg)
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.

You currently call data = parser.parse_args() in top-level code of a module. This runs at import time, but there is nothing to parse while your module gets imported, since this happens during startup and not while handling a request.
Call this from your view function (ie the code that runs while handling a request) instead. You will also need to restructure your code - calling api.add_resource() is something you do during startup/initialization time, not while handling a request.
The important thing to understand is that this is not PHP where all your code runs when a request is received. Instead, the Python modules are imported when you start your application (flask run, app.run(), or running it in a WSGI container). When a request is received only the code related to handling that request runs.

Google Adwords API authentication issue

Just getting started on the Adwords API, for some reason I can't seem to connect at all.
The code below, straight from the tutorial throws the error:
Traceback (most recent call last):
File "<pyshell#12>", line 1, in <module>
client = AdWordsClient(path=os.path.join('Users', 'ravinthambapillai', 'Google Drive', 'client_secrets.json'))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/adspygoogle/adwords/AdWordsClient.py", line 151, in __init__
self._headers = self.__LoadAuthCredentials()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/adspygoogle/adwords/AdWordsClient.py", line 223, in __LoadAuthCredentials
return super(AdWordsClient, self)._LoadAuthCredentials()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/adspygoogle/common/Client.py", line 94, in _LoadAuthCredentials
raise ValidationError(msg)
**ValidationError: Authentication data is missing.**
from adspygoogle.adwords.AdWordsClient import AdWordsClient
from adspygoogle.common import Utils
client = AdWordsClient(path=os.path.join('Users', 'this-user', 'this-folder', 'client_secrets.json'))

It looks like there's two issues. First, try removing the last path element, as far as I recall, the path parameter expects a directory that contains the authentication pickle, logs etc. This approach requires that you already have a valid auth_token.pkl.
Second, it appears that you're using OAuth2 for authentication (I'm guessing by the client_secrets.json file). For this to work, you'll need to use the oauth2client library and provide an oauth2credentials instance in the headers parameter to AdWordsClient.
The following is straight from the file examples/adspygoogle/adwords/v201302/misc/use_oauth2.py in the client distribution and should give you an idea how it works:
# We're using the oauth2client library:
# http://code.google.com/p/google-api-python-client/downloads/list
flow = OAuth2WebServerFlow(
client_id=oauth2_client_id,
client_secret=oauth2_client_secret,
# Scope is the server address with '/api/adwords' appended.
scope='https://adwords.google.com/api/adwords',
user_agent='oauth2 code example')
# Get the authorization URL to direct the user to.
authorize_url = flow.step1_get_authorize_url()
print ('Log in to your AdWords account and open the following URL: \n%s\n' %
authorize_url)
print 'After approving the token enter the verification code (if specified).'
code = raw_input('Code: ').strip()
credential = None
try:
credential = flow.step2_exchange(code)
except FlowExchangeError, e:
sys.exit('Authentication has failed: %s' % e)
# Create the AdWordsUser and set the OAuth2 credentials.
client = AdWordsClient(headers={
'developerToken': '%s++USD' % email,
'clientCustomerId': client_customer_id,
'userAgent': 'OAuth2 Example',
'oauth2credentials': credential
})

I am not familiar with the AdWordsClient api but are you sure your path is correct?
your current join produces a relative path, do you need an absolute one?
>>> import os
>>> os.path.join('Users', 'this-user')
'Users/this-user'
For testing you could hardcode the absoulte path in to make sure it is not a path issue
I would also make sure that 'client_secrets.json exists, and that it is readable by the user executing python