Develop Reference

Reading data from MySQL and writing in GCP Bucket using Apache Beam Python API

I'm trying to read data from MySQL database (located in GCP) and writing the same in GCP Bucket. I want to use Python SDK for the same. Below is the code I've written.
from __future__ import generators
import apache_beam as beam
import time
import jaydebeapi
import os
import argparse
from google.cloud import bigquery
import logging
import sys
from string import lower
from google.cloud import storage as gstorage
import pandas as pd
from oauth2client.client import GoogleCredentials
print("Import Successful")
class setenv(beam.DoFn):
def process(self,context):
os.system('gsutil cp gs://<MY_BUCKET>/mysql-connector-java-5.1.45.jar /tmp/')
logging.info('Enviornment Variable set.')
class readfromdatabase(beam.DoFn):
def process(self, context):
logging.info('inside readfromdatabase')
database_user='root'
database_password='<DB_PASSWORD>'
database_host='<DB_HOST>'
database_port='<DB_PORT>'
database_db='<DB_NAME>'
logging.info('reached readfromdatabase 1')
jclassname = "com.mysql.jdbc.Driver"
url = ("jdbc:mysql://{0}:{1}/{2}?user={3}&password={4}".format(database_host,database_port, database_db,database_user,database_password))
jars = ["/tmp/mysql-connector-java-5.1.45.jar"]
libs = None
cnx = jaydebeapi.connect(jclassname, url, jars=jars,
libs=libs)
logging.info('Connection Successful..')
cursor = cnx.cursor()
logging.info('Reading Sql Query from the file...')
query = 'select * from employees.employee_details'
logging.info('Query is %s',query)
logging.info('Query submitted to Database..')
for chunk in pd.read_sql(query, cnx, coerce_float=True, params=None, parse_dates=None, columns=None,chunksize=500000):
chunk.apply(lambda x: x.replace(u'\r', u' ').replace(u'\n', u' ') if isinstance(x, str) or isinstance(x, unicode) else x).WriteToText('gs://first-bucket-arpan/output2/')
logging.info("Load completed...")
return list("1")
def run():
try:
pcoll = beam.Pipeline()
dummy= pcoll | 'Initializing..' >> beam.Create(['1'])
logging.info('inside run 1')
dummy_env = dummy | 'Setting up Instance..' >> beam.ParDo(setenv())
logging.info('inside run 2')
readrecords=(dummy_env | 'Processing' >> beam.ParDo(readfromdatabase()))
logging.info('inside run 3')
p=pcoll.run()
logging.info('inside run 4')
p.wait_until_finish()
except:
logging.exception('Failed to launch datapipeline')
raise
def main():
logging.getLogger().setLevel(logging.INFO)
GOOGLE_APPLICATION_CREDENTIALS="gs://<MY_BUCKET>/My First Project-800a97e1fe65.json"
run()
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
main()
I'm running using below command:
python /home/aarpan_roy/readfromdatabase.py --region $REGION --runner DataflowRunner --project $PROJECT --temp_location gs://$BUCKET/tmp
While running, betting below output and no Dataflow job is being created:
INFO:root:inside run 2
INFO:root:inside run 3
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function annotate_downstream_side_inputs at 0x7fc1ea8a27d0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function fix_side_input_pcoll_coders at 0x7fc1ea8a28c0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x7fc1ea8a2938> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_sdf at 0x7fc1ea8a29b0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_gbk at 0x7fc1ea8a2a28> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sink_flattens at 0x7fc1ea8a2b18> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function greedily_fuse at 0x7fc1ea8a2b90> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function read_to_impulse at 0x7fc1ea8a2c08> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function impulse_to_input at 0x7fc1ea8a2c80> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fc1ea8a2e60> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function setup_timer_mapping at 0x7fc1ea8a2de8> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function populate_data_channel_coders at 0x7fc1ea8a2ed8> ====================
INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100
INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7fc1ea356ad0> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, '')
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (ref_AppliedPTransform_Initializing../Impulse_3)+((ref_AppliedPTransform_Initializing../FlatMap(<lambda at core.py:2632>)_4)+((ref_AppliedPTransform_Initializing../Map(decode)_6)+((ref_AppliedPTransform_Setting up Instance.._7)+(ref_AppliedPTransform_Processing_8))))
Copying gs://<MY_BUCKET>/mysql-connector-java-5.1.45.jar...
- [1 files][976.4 KiB/976.4 KiB]
Operation completed over 1 objects/976.4 KiB.
INFO:root:Enviornment Variable set.
INFO:root:inside run 4
Kindly help me resolving the issue and also guide how to extract data from MySql to GCP Bucket using Apache Beam Python API.
Thanks in advance.
=================================================================================
Hello All,
I changed the code a bit and running it from a shell script after exporting GOOGLE_APPLICATION_CREDENTIALS. But, when it is executing, getting below error:
WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: File /home/aarpan_roy/script/dataflowerviceaccount.json (pointed by GOOGLE_APPLICATION_CREDENTIALS environment variable) does not exist!
Connecting anonymously.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['True', '--service_account_name', 'dataflowserviceaccount', '--service_account_key_file', '/home/aarpan_roy/script/dataflowserviceaccount.json']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['True', '--service_account_name', 'dataflowserviceaccount', '--service_account_key_file', '/home/aarpan_roy/script/dataflowserviceaccount.json']
ERROR:root:Failed to launch datapipeline"
Below is the total log file:
aarpan_roy#my-dataproc-cluster-m:~/script/util$ sh -x dataflow_runner.sh
+ export GOOGLE_APPLICATION_CREDENTIALS=gs://<MY_BUCKET>/My First Project-800a97e1fe65.json
+ python /home/aarpan_roy/script/util/loadfromdatabase.py --config config.properties --productconfig cts.properties --env dev --sourcetable employee_details --sqlquery /home/aarpan_roy/script/sql/employee_details.sql --connectionprefix d3 --incrementaldate 1900-01-01
/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/__init__.py:82: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
/home/aarpan_roy/script/util/
INFO:root:Job Run Id is 22152
INFO:root:Job Name is load-employee-details-20200820
SELECT * FROM EMPLOYEE_DETAILS
WARNING:apache_beam.options.pipeline_options_validator:Option --zone is deprecated. Please use --worker_zone instead.
INFO:apache_beam.runners.portability.stager:Downloading source distribution of the SDK from PyPi
INFO:apache_beam.runners.portability.stager:Executing command: ['/usr/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/tmpEarTQ0', 'apache-beam==2.23.0', '--no-deps', '--no-binary', ':all:']
INFO:apache_beam.runners.portability.stager:Staging SDK sources from PyPI: dataflow_python_sdk.tar
INFO:apache_beam.runners.portability.stager:Downloading binary distribution of the SDK from PyPi
INFO:apache_beam.runners.portability.stager:Executing command: ['/usr/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/tmpEarTQ0', 'apache-beam==2.23.0', '--no-deps', '--only-binary', ':all:', '--python-version', '27', '--implementation', 'cp', '--abi', 'cp27mu', '--platform', 'manylinux1_x86_64']
INFO:apache_beam.runners.portability.stager:Staging binary distribution of the SDK from PyPI: apache_beam-2.23.0-cp27-cp27mu-manylinux1_x86_64.whl
WARNING:root:Make sure that locally built Python SDK docker image has Python 2.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python2.7_sdk:2.23.0. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: File /home/aarpan_roy/script/dataflowerviceaccount.json (pointed by GOOGLE_APPLICATION_CREDENTIALS environment variable) does not exist!
Connecting anonymously.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://<MY_BUCKET>/load-employee-details-20200820.1597916566.568419/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://<MY_BUCKET>/load-employee-details-20200820.1597916566.568419/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://<MY_BUCKET>/load-employee-details-20200820.1597916566.568419/pickled_main_session...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://<MY_BUCKET>/load-employee-details-20200820.1597916566.568419/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://<MY_BUCKET>/load-employee-details-20200820.1597916566.568419/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://<MY_BUCKET>/load-employee-details-20200820.1597916566.568419/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://<MY_BUCKET>/load-employee-details-20200820.1597916566.568419/apache_beam-2.23.0-cp27-cp27mu-manylinux1_x86_64.whl...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://<MY_BUCKET>/load-employee-details-20200820.1597916566.568419/apache_beam-2.23.0-cp27-cp27mu-manylinux1_x86_64.whl in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['True', '--service_account_name', 'dataflowserviceaccount', '--service_account_key_file', '/home/aarpan_roy/script/dataflowserviceaccount.json']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['True', '--service_account_name', 'dataflowserviceaccount', '--service_account_key_file', '/home/aarpan_roy/script/dataflowserviceaccount.json']
ERROR:root:Failed to launch datapipeline
Traceback (most recent call last)
File "/home/aarpan_roy/script/util/loadfromdatabase.py", line 105, in run
p=pcoll.run()
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 521, in run
allow_proto_holders=True).run(False)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 534, in run
return self.runner.run_pipeline(self, self._options)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 586, in run_pipeline
self.dataflow_client.create_job(self.job), self)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 681, in create_job
return self.submit_job_description(job)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 748, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 667, in Create
config, request, global_params=global_params)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
HttpForbiddenError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/turing-thought-277215/locations/asia-southeast1/jobs?alt=json>: response: <{'status': '403', 'content-length': '138', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Thu, 20 Aug 2020 09:42:47 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8', 'www-authenticate': 'Bearer realm="https://accounts.google.com/", error="insufficient_scope", scope="https://www.googleapis.com/auth/compute.readonly https://www.googleapis.com/auth/compute https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email email https://www.googleapis.com/auth/userinfo#email"'}>, content <{
"error": {
"code": 403,
"message": "Request had insufficient authentication scopes.",
"status": "PERMISSION_DENIED"
}
}
>
Traceback (most recent call last):
File "/home/aarpan_roy/script/util/loadfromdatabase.py", line 194, in <module>
args.sourcetable)
File "/home/aarpan_roy/script/util/loadfromdatabase.py", line 142, in main
run()
File "/home/aarpan_roy/script/util/loadfromdatabase.py", line 105, in run
p=pcoll.run()
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 521, in run
allow_proto_holders=True).run(False)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 534, in run
return self.runner.run_pipeline(self, self._options)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 586, in run_pipeline
self.dataflow_client.create_job(self.job), self)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 681, in create_job
return self.submit_job_description(job)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 748, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 667, in Create
config, request, global_params=global_params)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/home/aarpan_roy/.local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/turing-thought-277215/locations/asia-southeast1/jobs?alt=json>: response: <{'status': '403', 'content-length': '138', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Thu, 20 Aug 2020 09:42:47 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8', 'www-authenticate': 'Bearer realm="https://accounts.google.com/", error="insufficient_scope", scope="https://www.googleapis.com/auth/compute.readonly https://www.googleapis.com/auth/compute https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email email https://www.googleapis.com/auth/userinfo#email"'}>, content <{
"error": {
"code": 403,
"message": "Request had insufficient authentication scopes.",
"status": "PERMISSION_DENIED"
}
}
>
+ echo 1
1
I'm not able to understand where I'm making the mistake. Kindly help me to resolve the issue.
Thanks in advance.

I think that there might be a spelling mistake in your account credentials
/home/aarpan_roy/script/dataflowerviceaccount.json should be /home/aarpan_roy/script/dataflowserviceaccount.json. Can you check this ?

There seem to be two issues here:
On the one hand, the authentication issue, please perform the test provided by #Jayadeep to verify if it was an issue with the name of the credentials.
In the same way, there is probably an issue with the handling of credentials because you are running your code in a Dataproc instance, so I recommend you test your code in the Cloud Shell
On the other hand, I found another post where it is mentioned that the connection to Cloud SQL in python seems to be not as transparent as it is with Java (using jdbcIO).
At the same time, I found this another post where is mentioned a workaround to connect to cloud SQL but using psycopg2 instead of jaydebeapi. I recommend you try it.
import psycopg2
connection = psycopg2.connect(
host = host,
hostaddr = hostaddr,
dbname = dbname,
user = user,
password = password,
sslmode=sslmode,
sslrootcert = sslrootcert,
sslcert = sslcert,
sslkey = sslkey
)

Docusign Python SDK Authentication Example Error: Bad Request

I am trying to use the Docusign Rest API and attempted to follow the sample code on the Python SDK on Github here: https://github.com/docusign/docusign-python-client. I've replaced the necessary values with those obtained from Docusign (keys, urls etc) but the example seems to break down at the following line:
api_client.configure_jwt_authorization_flow(private_key_filename, oauth_base_url, integrator_key, user_id, 3600)
Before then, the code takes the given information and creates the url to authenticate as I understand it. I've manually accessed this link and it seems to work; it asks me for permission to allow the program to act on my behalf and then redirects me to the redirect uri with a code added to the uri. My problem is that when it executes the next line (the one I posted above), it results in this:
Traceback (most recent call last):
File "<ipython-input-40-84847e506c37>", line 1, in <module>
runfile('C:/Users/santi/Desktop/docusign/untitled0.py', wdir='C:/Users/santi/Desktop/docusign')
File "C:\Users\santi\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)
File "C:\Users\santi\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/santi/Desktop/docusign/untitled0.py", line 501, in <module>
api_client.configure_jwt_authorization_flow(private_key_filename, oauth_base_url, integrator_key, user_id, 3600)
File "C:\Users\santi\Anaconda3\lib\site-packages\docusign_esign\api_client.py", line 126, in configure_jwt_authorization_flow
post_params=self.sanitize_for_serialization({"assertion": assertion, "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer"}))
File "C:\Users\santi\Anaconda3\lib\site-packages\docusign_esign\api_client.py", line 430, in request
body=body)
File "C:\Users\santi\Anaconda3\lib\site-packages\docusign_esign\rest.py", line 244, in POST
body=body)
File "C:\Users\santi\Anaconda3\lib\site-packages\docusign_esign\rest.py", line 200, in request
raise ApiException(http_resp=r)
ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'private', 'Content-Type': 'text/html', 'X-AspNetMvc-Version': '5.2', 'X-DocuSign-TraceToken': '15fb83c2-9054-4522-b5ed-b77646fe1c4b', 'X-DocuSign-Node': 'DA2DFE4', 'Date': 'Sun, 05 Aug 2018 19:35:12 GMT', 'Content-Length': '11'})
HTTP response body: b'Bad Request'
I've googled the problem and found some similar errors on here but none that helped me out much. Following the traceback takes me into the api_client.py and rest.py files but I wasn't able to decipher much. In the end the origin in the error seems to be in line 200 of rest.py where an exception is raised because a value unknown to me is not within the proper range. I'm not sure what is causing the bad request on my end. Has anyone had a similar experience or might know what the issue is? I'd appreciate it a lot, thanks.

I was getting exactly the same error at the same line.
The account specific inputs to api_client.configure_jwt_authorization_flow() are:
the private_key_filename (which is processed correctly to get a JWT token)
the oauth_base_url (which is hard coded in the example as
"account-d.docusign.com")
the integrator_key (copy-pasted from my account)
the user_id
It turns out that the user_id value should the API Username code. That fixed the problem. I found the answer here:
DocuSign Python SDK - API Exception 400 - 'Bad Request'
Hope it solves your problem too!

Google Cloud Storage - PUT Fails on Development Server

On my dev server, Google Cloud Storage fails to save files during a PUT command.
This same command works fine on production. The response headers (see below) have little debug information. I have plenty of space on my hard-drive, and I have been running the dev server as an administrator, so it should have all necessary permissions.
Any ideas?
Below is the relevant Traceback (I changed the bucket name):
INFO 2015-08-07 14:14:41,415 module.py:809] default: "PUT /_ah/gcs/bucket/IPR2013-00219%2C%20No.%2016%20Institution%20Decision%20-%20Decis_2015-08-07%2018-14-25.100000.pdf?upload_id=encoded_gs_file%3AZG9ja2V0YWxhcm0tZGV2L0lQUjIwMTMtMDAyMTksIE5vLiAxNiBJbnN0aXR1dGlvbiBEZWNpc2lvbiAtIERlY2lzXzIwMTUtMDgtMDcgMTgtMTQtMjUuMTAwMDAwLnBkZg%3D%3D HTTP/1.1" 500 -
Traceback (most recent call last):
File "Project\libs\cloudstorage\storage_api.py", line 794, in __exit__
self.close()
File "Project\libs\cloudstorage\storage_api.py", line 787, in close
self._flush(finish=True)
File "Project\libs\cloudstorage\storage_api.py", line 841, in _flush
self._send_data(data, self._written, file_len)
File "Project\libs\cloudstorage\storage_api.py", line 874, in _send_data
{'upload_path': self._path_with_token})
File "Project\libs\cloudstorage\errors.py", line 141, in check_status
raise ServerError(msg)
ServerError: Expect status [200] from Google Storage. But got status 500.
Path: '/bucket/IPR2013-00219%2C%20No.%2016%20Institution%20Decision%20-%20Decis_2015-08-07%2018-14-25.100000.pdf'.
Request headers: {'accept-encoding': 'gzip, *', 'content-range': 'bytes 0-195060/195061', 'x-goog-api-version': '2'}.
Response headers: {'date': 'Fri, 07 Aug 2015 18:14:41 GMT', 'transfer-encoding': 'chunked', 'server': 'Development/2.0'}.
Body: ''.
Extra info: {'upload_path': '/bucket/IPR2013-00219%2C%20No.%2016%20Institution%20Decision%20-%20Decis_2015-08-07%2018-14-25.100000.pdf?upload_id=encoded_gs_file%3AZG9ja2V0YWxhcm0tZGV2L0lQUjIw
MTMtMDAyMTksIE5vLiAxNiBJbnN0aXR1dGlvbiBEZWNpc2lvbiAtIERlY2lzXzIwMTUtMDgtMDcgMTgtMTQtMjUuMTAwMDAwLnBkZg%3D%3D'}.
Update 2020
FYI, I raised this issue with Google here, and they said that they won't fix it because it has already been fixed, even though it clearly has not.
Update 2022
Still broken! See the proposed patch in the issue with Google.

How to tell python app I am returning JSON

I have a shopify app I use to sync quantities between shopify and our POS systems. Recently it started throwing a "406 not acceptable" error. I have tracked it down (i think) to when I make the actual call to update the quantity, this line:
return shopify.Variant(dict(id=288275460, inventory_quantity=3.0)).save()
I have hard-coded values that should work but the moment it tries to execute this command i get the traceback posted below.
From what I think is happening is that shopify stopped returning XML and now the returned data isn't acceptable to the python application on my webhost that is executing these commands. What I don't understand is how I would specify that my application should expect the proper content-type. Currently I am not even processing the returned data in any way just executing the command without even a return causes the same error. Just really confused which is unfortunately rather easy.
TRACEBACK:
Internal Server Error: /inventory/process/
Traceback (most recent call last):
File "/home/crabs/.virtualenvs/webhostname/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/crabs/.virtualenvs/webhostname/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/home/crabs/projects/webfaction/webdomain.com/apps/shopified/views.py", line 115, in process
dictUpdated = update(sku, arraydata[0], arraydata[1])
File "/home/crabs/projects/webfaction/webdomain.com/apps/shopified/inventory.py", line 72, in update
return simplejson.dumps(shopify.Variant(dict(id=288275460, inventory_quantity=3.0)).save())
File "/home/crabs/.virtualenvs/webhostname/local/lib/python2.7/site-packages/pyactiveresource/activeresource.py", line 775, in save
data=self.to_xml())
File "/home/crabs/.virtualenvs/webhostname/local/lib/python2.7/site-packages/pyactiveresource/connection.py", line 351, in put
return self._open('PUT', path, headers=headers, data=data)
File "/home/crabs/.virtualenvs/webhostname/local/lib/python2.7/site-packages/shopify/base.py", line 19, in _open
self.response = super(ShopifyConnection, self)._open(*args, **kwargs)
File "/home/crabs/.virtualenvs/webhostname/local/lib/python2.7/site-packages/pyactiveresource/connection.py", line 287, in _open
http_response = self._handle_error(err)
File "/home/crabs/.virtualenvs/webhostname/local/lib/python2.7/site-packages/pyactiveresource/connection.py", line 415, in _handle_error
raise ClientError(err)
ClientError: Response(code=406, body=" ", headers={'status': '406 Not Acceptable', 'x-request-id': '6be354d6-686b-4146-bcf3-03ed080756a2', 'x-xss-protection': '1; mode=block', 'x-shopify-asset-version': '1', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'x-shardid': '0', 'x-stats-apiclientid': '149305', 'connection': 'close', 'server': 'nginx', 'x-stats-apipermissionid': '2992571', 'x-shopid': '2117110', 'x-stats-userid': '0', 'x-ua-compatible': 'chrome=1', 'http_x_shopify_shop_api_call_limit': '1/40', 'date': 'Wed, 16 Jul 2014 19:06:48 GMT', 'x-shopify-shop-api-call-limit': '1/40', 'content-type': 'application/xml'}, msg="Not Acceptable")

How do I get the Google AdWords API Python Client Library to work?

Is there an up-to-date tutorial available for using the Python adwords client, or failing that does anyone know enough to unpick the following? Have I made some configuration error, or perhaps is Python 2.6 unsupported?
On Windows I've installed:
Python 2.6
setuptools-0.6c11.win32-py2.6.exe
PyXML-0.8.4
SOAPpy-0.12.4
ZSI-2.0-rc3
adwords_api_python_13.2.0
I've supplied the AdWords config program with the MCC account email address, its password, blank client email and id, and for the developer token the same email address with "+USD" appended.
Now, if I use SOAPpy as the SOAP library, I get an exception about an unexpected keyword parameter timeout, which has been supplied by one function in SOAPpy to another function in SOAPpy.
If I use ZSI as the SOAP library (which I believe is preferred), I get the following exception:
Traceback (most recent call last):
File "tutorial.py", line 36, in <module>
campaigns = campaign_service.Mutate(operations)[0]
File "c:\Python26\lib\site-packages\adspygoogle\adwords\CampaignService.py", l
ine 112, in Mutate
'Campaign', self._loc, request)
File "c:\Python26\lib\site-packages\adspygoogle\adwords\AdWordsWebService.py",
line 256, in CallMethod
self.__ManageSoap(buf, start_time, stop_time, error)
File "c:\Python26\lib\site-packages\adspygoogle\adwords\AdWordsWebService.py",
line 130, in __ManageSoap
raise Error(e)
adspygoogle.common.Errors.Error:
Traceback (most recent call last):
File "c:\Python26\lib\site-packages\adspygoogle\common\WebService.py", line 20
8, in CallMethod
eval('service.%s(request)' % method_name))
File "<string>", line 1, in <module>
File "c:\Python26\lib\site-packages\adspygoogle\adwords\zsi\v201008\CampaignSe
rvice_services.py", line 47, in mutateCampaign
self.binding.Send(None, None, request, soapaction="", **kw)
File "build\bdist.win32\egg\ZSI\client.py", line 267, in Send
self.SendSOAPData(soapdata, url, soapaction, **kw)
File "build\bdist.win32\egg\ZSI\client.py", line 301, in SendSOAPData
self.h.send(soapdata)
File "c:\Python26\lib\site-packages\adspygoogle\common\zsi\HttpsConnectionHand
ler.py", line 93, in send
httplib.HTTPSConnection.endheaders(self)
File "c:\Python26\lib\httplib.py", line 892, in endheaders
self._send_output()
File "c:\Python26\lib\httplib.py", line 764, in _send_output
self.send(msg)
File "c:\Python26\lib\site-packages\adspygoogle\common\zsi\HttpsConnectionHand
ler.py", line 93, in send
httplib.HTTPSConnection.endheaders(self)
File "c:\Python26\lib\httplib.py", line 890, in endheaders
raise CannotSendHeader()
CannotSendHeader [RAW DATA: _________________________________ Tue Mar 08 16:54:4
0 2011 REQUEST:
followed by some XML, which I've modified to remove identifying data:
<SOAP-ENV:Envelope xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xm
lns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ZSI="http://www.z
olera.com/schemas/ZSI/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance"><SOAP-ENV:Header><RequestHeader xmlns
="https://adwords.google.com/api/adwords/cm/v201008"><authToken>XYZ</authToken>
<userAgent>AwApi-Python-13.2.0|XYZ</userAgent><developerTok
en>XYZ#gmail.com+USD</developerToken></RequestHeader></SOAP-ENV:Header><SO
AP-ENV:Body xmlns:ns1="https://adwords.google.com/api/adwords/cm/v201008"><ns1:m
utate xmlns:ns1="https://adwords.google.com/api/adwords/cm/v201008"><ns1:operati
ons><ns1:operator>ADD</ns1:operator><ns1:operand><ns1:name>Interplanetary Cruise
#201138165440485000</ns1:name><ns1:status>PAUSED</ns1:status><ns1:endDate>20120
101</ns1:endDate><ns1:budget xsi:type="ns1:Budget"><ns1:period>DAILY</ns1:period
><ns1:amount xsi:type="ns1:Money"><ns1:microAmount>50000000</ns1:microAmount></n
s1:amount><ns1:deliveryMethod>STANDARD</ns1:deliveryMethod></ns1:budget><ns1:bid
dingStrategy xsi:type="ns1:ManualCPC"></ns1:biddingStrategy></ns1:operand></ns1:
operations></ns1:mutate></SOAP-ENV:Body></SOAP-ENV:Envelope>
My code is below:
from adspygoogle.adwords.AdWordsClient import AdWordsClient
from adspygoogle.common import Utils
client = AdWordsClient(path = '../../installers/adwords_api_python_13.2.0.tar/adwords_api_python_13.2.0')
campaign_service = client.GetCampaignService('https://adwords-sandbox.google.com', 'v201008')
operations = [{
'operator': 'ADD',
'operand': {
'name': 'Interplanetary Cruise #%s' % Utils.GetUniqueName(),
'status': 'PAUSED',
'biddingStrategy': {
'type': 'ManualCPC'
},
'endDate': '20120101',
'budget': {
'period': 'DAILY',
'amount': {
'microAmount': '50000000'
},
'deliveryMethod': 'STANDARD'
}
}
}]
campaigns = campaign_service.Mutate(operations)[0]
# Display results.
for campaign in campaigns['value']:
print ('Campaign with name \'%s\' and id \'%s\' was added.'
% (campaign['name'], campaign['id']))
print
print ('Usage: %s units, %s operations' % (client.GetUnits(),
client.GetOperations()))
Note that the tutorial code at http://code.google.com/apis/adwords/docs/tutorial.html doesn't even slightly work, there's no such thing as aw_api.Client in the current version of the client. But the above is mostly copied from the tutorial.
The examples provided with the client result in the same error.
I could try an older client library if the current one is simply broken, although if it makes any difference, I will need one that supports TargetingIdeaService.
[Edit: oh, sorry, I forgot to say that I actually edited the AdWords client code. In Utils.GetDataFromCsvFile, I removed .replace(':','|') from the line that constructs the file url passed to urllib.urlopen.
This is what makes me suspect that my version of Python might not be supported, since Windows Python 2.6 didn't seem able to open the url file:///c|/foo/bar/baz/]

Have you tried with ZSI-2.0? This issue report looks similar to http://code.google.com/p/google-api-adwords-python-lib/issues/detail?id=141. Please append your info to the bug report, if you continue to see this issue.
As for tutorial code being out of date, good catch! I'll have it fixed shortly. Please track at http://code.google.com/p/google-api-adwords-python-lib/issues/detail?id=152.