I recently enclosed a function call inside a try-catch block to catch all URL Fetch exceptions but as per logs below it did not get caught.
from google.appengine.api import urlfetch_errors
::
::
try:
gitkit_user = gitkit_instance.VerifyGitkitToken (self.request.cookies['gtoken'])
except ( urlfetch_errors.DeadlineExceededError, urlfetch_errors.ConnectionClosedError, urlfetch_errors.DNSLookupFailedError, urlfetch_errors.DownloadError, urlfetch_errors.Error, urlfetch_errors.InternalTransientError, urlfetch_errors.InvalidMethodError, urlfetch_errors.InvalidURLError, urlfetch_errors.MalformedReplyError, urlfetch_errors.PayloadTooLargeError, urlfetch_errors.ResponseTooLargeError, urlfetch_errors.SSLCertificateError, urlfetch_errors.TooManyRedirectsError ):
logging.error ("Exception occurred while executing gitkit_instance.VerifyGitkitToken (). \nDetails: "+traceback.format_exc ())
Error log including trace
The API call urlfetch.Fetch() took too long to respond and was cancelled.
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 1057, in add_context_wrapper
return synctaskletfunc(*args, **kwds)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 1038, in synctasklet_wrapper
return taskletfunc(*args, **kwds).get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 1011, in tasklet_wrapper
result = func(*args, **kwds)
File "/base/data/home/apps/s~myapp/1.389163610985502634/main_v3.py", line 29956, in get
gitkit_user = gitkit_instance.VerifyGitkitToken (self.request.cookies['gtoken'])
File "/base/data/home/apps/s~myapp/1.389163610985502634/gitkitclient.py", line 216, in VerifyGitkitToken
certs = self.rpc_helper.GetPublicCert()
File "/base/data/home/apps/s~myapp/1.389163610985502634/rpchelper.py", line 159, in GetPublicCert
resp, content = self.http.request(cert_url, headers=headers)
File "/base/data/home/apps/s~myapp/1.389163610985502634/httplib2/__init__.py", line 1570, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/base/data/home/apps/s~myapp/1.389163610985502634/httplib2/__init__.py", line 1317, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/base/data/home/apps/s~myapp/1.389163610985502634/httplib2/__init__.py", line 1286, in _conn_request
response = conn.getresponse()
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/gae_override/httplib.py", line 522, in getresponse
**extra_kwargs)
File "/base/data/home/apps/s~myapp/1.389163610985502634/httplib2/__init__.py", line 1089, in fixed_fetch
validate_certificate=validate_certificate)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 271, in fetch
return rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 378, in _get_fetch_result
rpc.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
self.__rpc.CheckSuccess()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 134, in CheckSuccess
raise self.exception
DeadlineExceededError: The API call urlfetch.Fetch() took too long to respond and was cancelled.
I am using Google App Engine with Python.
What did I do wrong ?
This might not entirely answer your question, but why not do something like this:
try:
gitkit_user = gitkit_instance.VerifyGitkitToken (self.request.cookies['gtoken'])
except Exception as e:
logging.error ("Exception occurred while executing gitkit_instance.VerifyGitkitToken (). \nDetails: " + e)
The error will catch and the type of error will be printed (e).
Related
Just to give a brief introduction, I am using elastic beanstalk from Amazon to deploy my python (Falcon) application. I have two environments of elastic beanstalk (one being the main app server and the other one a worker environment). The worker environment receives HTTP requests for async tasks and then executes them. Most of the tasks are either push notifications (mobile devices) or sending emails. The strange part is that my tasks are failing with errors that I am unable to determine the cause for (this works fine for my local environment). Some of the tracebacks look like this:
self.send_message(message=message)
File "/var/app/current/application/services/push.py", line 114, in send_message
response = messaging.send_multicast(message)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/firebase_admin/messaging.py", line 167, in send_multicast
return _get_messaging_service(app).send_all(messages, dry_run)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/firebase_admin/messaging.py", line 390, in send_all
batch.execute()
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/http.py", line 1540, in execute
self._execute(http, self._order, self._requests)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/http.py", line 1470, in _execute
self._batch_uri, method="POST", body=body, headers=headers
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/google_auth_httplib2.py", line 201, in request
uri, method, body=body, headers=request_headers, **kwargs)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1994, in request
cachekey,
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
conn, request_uri, method, body, headers
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1589, in _conn_request
response = conn.getresponse()
File "/usr/lib64/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/usr/lib64/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.7/ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.7/ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
File "/var/app/current/application/services/push.py", line 152, in execute_push
self.send_message(message=message)
File "/var/app/current/application/services/push.py", line 114, in send_message
response = messaging.send_multicast(message)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/firebase_admin/messaging.py", line 167, in send_multicast
return _get_messaging_service(app).send_all(messages, dry_run)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/firebase_admin/messaging.py", line 390, in send_all
batch.execute()
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/http.py", line 1540, in execute
self._execute(http, self._order, self._requests)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/http.py", line 1470, in _execute
self._batch_uri, method="POST", body=body, headers=headers
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/google_auth_httplib2.py", line 201, in request
uri, method, body=body, headers=request_headers, **kwargs)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1994, in request
cachekey,
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
conn, request_uri, method, body, headers
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1589, in _conn_request
response = conn.getresponse()
File "/usr/lib64/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/usr/lib64/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.7/ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.7/ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: UNKNOWN_STATE] unknown state (_ssl.c:2570)
File "/var/app/current/application/services/push.py", line 114, in send_message
response = messaging.send_multicast(message)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/firebase_admin/messaging.py", line 167, in send_multicast
return _get_messaging_service(app).send_all(messages, dry_run)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/firebase_admin/messaging.py", line 390, in send_all
batch.execute()
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/http.py", line 1540, in execute
self._execute(http, self._order, self._requests)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/googleapiclient/http.py", line 1470, in _execute
self._batch_uri, method="POST", body=body, headers=headers
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/google_auth_httplib2.py", line 201, in request
uri, method, body=body, headers=request_headers, **kwargs)
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1994, in request
cachekey,
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
conn, request_uri, method, body, headers
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/httplib2/__init__.py", line 1618, in _conn_request
content = response.read()
File "/usr/lib64/python3.7/http/client.py", line 468, in read
return self._readall_chunked()
File "/usr/lib64/python3.7/http/client.py", line 575, in _readall_chunked
chunk_left = self._get_chunk_left()
File "/usr/lib64/python3.7/http/client.py", line 563, in _get_chunk_left
self._read_and_discard_trailer()
File "/usr/lib64/python3.7/http/client.py", line 536, in _read_and_discard_trailer
line = self.fp.readline(_MAXLINE + 1)
AttributeError: 'NoneType' object has no attribute 'readline'
Some other exceptions are as follows:
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2570)
ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2570)
ssl.SSLError: [SSL: BLOCK_CIPHER_PAD_IS_WRONG] block cipher pad is wrong (_ssl.c:2570)
The code looks like this:
message = messaging.MulticastMessage(
tokens=tokens,
data=data,
notification=messaging.Notification(
title=title,
body=body
)
)
response = messaging.send_multicast(message)
This is quite strange for me since the tasks are failing and as a result, retrying. But somehow I am receiving multiple notifications. That means the notifications are being sent but the tasks are getting failed. Please help me out in understanding what could be the problem. Does it have something to do with the system date time?
Do you know why I have this error ?
timeout: The read operation timed out
How to except it please ? From which package can I retrieve the tiemout exception ?
File "C:\Users\root\Desktop\bot\sheetapi.py", line 58, in
update
spreadsheetId=SAMPLE_SPREADSHEET_ID, range=SAMPLE_RANGE_NAME, valueInputOption='USER_ENTERED', body=body).execute()
File
"C:\Users\root\Anaconda3\lib\site-packages\googleapiclient_helpers.py",
line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File
"C:\Users\root\Anaconda3\lib\site-packages\googleapiclient\http.py",
line 892, in execute
headers=self.headers,
File
"C:\Users\root\Anaconda3\lib\site-packages\googleapiclient\http.py",
line 195, in _retry_request
raise exception
File
"C:\Users\root\Anaconda3\lib\site-packages\googleapiclient\http.py",
line 172, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File
"C:\Users\root\Anaconda3\lib\site-packages\google_auth_httplib2.py",
line 198, in request
uri, method, body=body, headers=request_headers, **kwargs)
File
"C:\Users\root\Anaconda3\lib\site-packages\httplib2__init__.py", line
1514, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File
"C:\Users\root\Anaconda3\lib\site-packages\httplib2__init__.py", line
1264, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File
"C:\Users\root\Anaconda3\lib\site-packages\httplib2__init__.py", line
1217, in _conn_request
response = conn.getresponse()
File "C:\Users\root\Anaconda3\lib\http\client.py", line 1331, in
getresponse
response.begin()
File "C:\Users\root\Anaconda3\lib\http\client.py", line 297, in
begin
version, status, reason = self._read_status()
File "C:\Users\root\Anaconda3\lib\http\client.py", line 258, in
_read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\root\Anaconda3\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "C:\Users\root\Anaconda3\lib\ssl.py", line 1009, in recv_into
return self.read(nbytes, buffer)
File "C:\Users\root\Anaconda3\lib\ssl.py", line 871, in read
return self._sslobj.read(len, buffer)
File "C:\Users\root\Anaconda3\lib\ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
timeout: The read operation timed out
We have a kubernetes cron job on GCP that submits several copies of the same Python dataflow job, each in its own container. Whenever we need a new copy of the job, we just add it to the spec->jobTemplate->spec->template->spec->containers part of the cron job yaml and adjust the dataflow job parameters as needed. This usually works fine, but the latest copy we tried to add does not work. The existing copies are all still working as expected. The job seems to fail on submission to GCP, and the error message is not very helpful:
Traceback (most recent call last):
File "/app/job.py", line 117, in <module>
newness.pipeline.run_dataflow(sys.argv)
File "/app/newness/pipeline.py", line 480, in run_dataflow
result = pipe.run()
File "/usr/local/lib/python2.7/dist-packages/apache_beam/pipeline.py", line 403, in run
self.to_runner_api(), self.runner, self._options).run(False)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/pipeline.py", line 416, in run
return self.runner.run_pipeline(self)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 389, in run_pipeline
self.dataflow_client.create_job(self.job), self)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 184, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 504, in create_job
return self.submit_job_description(job)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 184, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 551, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 578, in Create
config, request, global_params=global_params)
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpError: <exception str() failed>
The job does not appear in the dataflow console at all.
The previous lines of the container logs look like:
2019-10-13T03:57:47.725542287Z Successfully downloaded apache-beam
2019-10-13T03:58:17.125601280Z INFO:root:Staging SDK sources from PyPI to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/dataflow_python_sdk.tar
2019-10-13T03:58:17.324843623Z INFO:root:Starting GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/dataflow_python_sdk.tar...
2019-10-13T03:58:24.825657227Z INFO:root:Completed GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/dataflow_python_sdk.tar
2019-10-13T03:58:25.225646529Z INFO:root:Downloading binary distribtution of the SDK from PyPi
2019-10-13T03:58:25.225716554Z INFO:root:Executing command: ['/usr/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/tmpk5TfMS', 'apache-beam==2.8.0', '--no-deps', '--only-binary', ':all:', '--python-version', '27', '--implementation', 'cp', '--abi', 'cp27mu', '--platform', 'manylinux1_x86_64']
2019-10-13T03:59:33.926186906Z Collecting apache-beam==2.8.0
2019-10-13T03:59:52.125678183Z Using cached https://files.pythonhosted.org/packages/0f/63/ea5453ba656d060936acf41d2ec057f23aafd69649e2129ac66fdda67d48/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-13T04:00:11.525435891Z Saved /tmp/tmpk5TfMS/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-13T04:00:12.025054706Z Successfully downloaded apache-beam
2019-10-13T04:00:26.726190542Z INFO:root:Staging binary distribution of the SDK from PyPI to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-13T04:00:26.825618945Z INFO:root:Starting GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl...
2019-10-13T04:00:33.725522899Z INFO:root:Completed GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-13T04:06:14.525017097Z Traceback (most recent call last):
...
Why is this job failing to submit? Are there any other logs we can look at to see the cause of this failure?
(Most of our dataflow jobs are written in Java, where the error messages are usually more helpful.)
UPDATE: Running job locally (Windows) with apache-beam 2.16 has the same issue but more logging detail:
...
INFO:root:Starting GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1571606418.971000/apache_beam-2.16.0-cp27-cp27mu-manylinux1_x86_64.whl...
INFO:root:Completed GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1571606418.971000/apache_beam-2.16.0-cp27-cp27mu-manylinux1_x86_64.whl in 3 seconds.
WARNING:root:Discarding unparseable args: ['job.py', '--days_history=30']
WARNING:root:Discarding unparseable args: ['job.py', '--days_history=30']
WARNING:root:Retry with exponential backoff: waiting for 2.64795143823 seconds before retrying submit_job_description because we caught exception: error: [Errno 10053] An established connection was aborted by the software in your host machine
Traceback for above exception (most recent call last):
File "C:\Python27\lib\site-packages\apache_beam\utils\retry.py", line 206, in wrapper
return fun(*args, **kwargs)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\apiclient.py", line 593, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\clients\dataflow\dataflow_v1b3_client.py", line 657, in Create
config, request, global_params=global_params)
File "C:\Python27\lib\site-packages\apitools\base\py\base_api.py", line 729, in _RunMethod
http, http_request, **opts)
File "C:\Python27\lib\site-packages\apitools\base\py\http_wrapper.py", line 346, in MakeRequest
check_response_func=check_response_func)
File "C:\Python27\lib\site-packages\apitools\base\py\http_wrapper.py", line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File "C:\Python27\lib\site-packages\oauth2client\transport.py", line 169, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\oauth2client\transport.py", line 169, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1694, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1434, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1390, in _conn_request
response = conn.getresponse()
File "C:\Python27\lib\httplib.py", line 1121, in getresponse
response.begin()
File "C:\Python27\lib\httplib.py", line 438, in begin
version, status, reason = self._read_status()
File "C:\Python27\lib\httplib.py", line 394, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\lib\socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "C:\Python27\lib\ssl.py", line 754, in recv
return self.read(buflen)
File "C:\Python27\lib\ssl.py", line 641, in read
v = self._sslobj.read(len)
... retries 4 times total ...
Traceback (most recent call last):
File "job.py", line 117, in <module>
newness.pipeline.run_dataflow(sys.argv)
File "C:\Users\LeeW\Desktop\newness\newness\pipeline.py", line 480, in run_dataflow
result = pipe.run()
File "C:\Python27\lib\site-packages\apache_beam\pipeline.py", line 407, in run
self._options).run(False)
File "C:\Python27\lib\site-packages\apache_beam\pipeline.py", line 420, in run
return self.runner.run_pipeline(self, self._options)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\dataflow_runner.py", line 485, in run_pipeline
self.dataflow_client.create_job(self.job), self)
File "C:\Python27\lib\site-packages\apache_beam\utils\retry.py", line 206, in wrapper
return fun(*args, **kwargs)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\apiclient.py", line 546, in create_job
return self.submit_job_description(job)
File "C:\Python27\lib\site-packages\apache_beam\utils\retry.py", line 219, in wrapper
raise_with_traceback(exn, exn_traceback)
File "C:\Python27\lib\site-packages\apache_beam\utils\retry.py", line 206, in wrapper
return fun(*args, **kwargs)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\apiclient.py", line 593, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\clients\dataflow\dataflow_v1b3_client.py", line 657, in Create
config, request, global_params=global_params)
File "C:\Python27\lib\site-packages\apitools\base\py\base_api.py", line 729, in _RunMethod
http, http_request, **opts)
File "C:\Python27\lib\site-packages\apitools\base\py\http_wrapper.py", line 346, in MakeRequest
check_response_func=check_response_func)
File "C:\Python27\lib\site-packages\apitools\base\py\http_wrapper.py", line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File "C:\Python27\lib\site-packages\oauth2client\transport.py", line 169, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\oauth2client\transport.py", line 169, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1694, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1434, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1390, in _conn_request
response = conn.getresponse()
File "C:\Python27\lib\httplib.py", line 1121, in getresponse
response.begin()
File "C:\Python27\lib\httplib.py", line 438, in begin
version, status, reason = self._read_status()
File "C:\Python27\lib\httplib.py", line 394, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\lib\socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "C:\Python27\lib\ssl.py", line 754, in recv
return self.read(buflen)
File "C:\Python27\lib\ssl.py", line 641, in read
v = self._sslobj.read(len)
socket.error: [Errno 10053] An established connection was aborted by the software in your host machine
Which version of Beam Python SDK are you using?
Basically what I am doing is submitting a file to my python server and I need the python server then to forward that same file to another API (third party OCR). I am trying to use the request library, but it doesn't seem to like it. It tells me, currently, this error:
AttributeError: 'set' object has no attribute 'read'
So here is the code that basically runs it. Any help or points would be greatly appreciated!
Note: self.request.post['file'] is a fieldstorage datatype.
import requests
files = {'file': {self.request.POST['fileName'], self.request.POST['file'].file, self.request.POST['type']}}
r = requests.post(url, files=files)
Edit: Here is the full Traceback
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "main.py", line 91, in post
resp = self.save_ocr()
File "main.py", line 143, in save_ocr
r = requests.post(url, files=files)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
prep = self.prepare_request(req)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/lib/python2.7/dist-packages/requests/models.py", line 290, in prepare
self.prepare_body(data, files)
File "/usr/lib/python2.7/dist-packages/requests/models.py", line 427, in prepare_body
(body, content_type) = self._encode_files(files, data)
File "/usr/lib/python2.7/dist-packages/requests/models.py", line 140, in _encode_files
rf = RequestField(name=k, data=fp.read(),
AttributeError: 'set' object has no attribute 'read'
I have a problem, I want to access the data which are stocked in my google cloud storage but I have an error and I don't know where it comes from.
Here is my code :
DECORATOR = oauth2decorator_from_clientsecrets(
CLIENT_SECRETS,
scope=[
'https://www.googleapis.com/auth/devstorage.read_only',
],
message=MISSING_CLIENT_SECRETS_MESSAGE)
http = DECORATOR.http()
service_cloud = build("storage", "v1beta1")
list_response = service_cloud.objects().list(bucket="directory_structure").execute(http=http)
params = {'directory_list':list_response['items']}
return self.render_template('directoryChoice.html', **params)
and the error message I receive is:
Traceback (most recent call last):
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__ rv = self.router.dispatch(request, response)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher return route.handler_adapter(request, response)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__ return handler.dispatch()
File "/base/data/home/apps/s~jba-gae-boilerplate/dev.366111306063368728/boilerplate/lib/basehandler.py", line 162, in dispatch webapp2.RequestHandler.dispatch(self)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch return self.handle_exception(e, self.app.debug)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch return method(*args, **kwargs)
File "/base/data/home/apps/s~jba-gae-boilerplate/dev.366111306063368728/web/lib/oauth2client/appengine.py", line 469, in check_oauth return method(request_handler, *args, **kwargs)
File "/base/data/home/apps/s~jba-gae-boilerplate/dev.366111306063368728/web/handlers.py", line 190, in get list_response = service_cloud.objects().list(bucket="jba_directory").execute(http=http)
File "/base/data/home/apps/s~jba-gae-boilerplate/dev.366111306063368728/web/lib/oauth2client/util.py", line 120, in positional_wrapper return wrapped(*args, **kwargs)
File "/base/data/home/apps/s~jba-gae-boilerplate/dev.366111306063368728/web/lib/apiclient/http.py", line 678, in execute raise HttpError(resp, content, uri=self.uri)
HttpError:
One thing which looks a bit odd:
In your code you're listing the directory_structure bucket, but the traceback references jba_directory. There is a mis-match of some sort. Perhaps in a wrapping decorator or other code above the snippet you included here?