I wrote a code to upload (create and update) a file to google drive,
in Windows 10 with python 3.9 it work, but in windows 2008 server with python 3.8 it give me an error.
just to remember 3.8 is the max version that supports windows 2008
if I try to list from gdrive it work, the problem is just to create or update the file.
just to remember 3.8 is the last python version that supports windows 2008.
I suspect its related with windows 2008 and ssl maybe!?!?
the error is this:
C:\backupmgr>python drive.py
Traceback (most recent call last):
File "drive.py", line 112, in <module>
envia_zip('sexta.7z')
File "drive.py", line 104, in envia_zip
file = service.files().create(body=file_metadata, media_body=media).execute(
)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\site-p
ackages\googleapiclient\_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\site-p
ackages\googleapiclient\http.py", line 923, in execute
resp, content = _retry_request(
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\site-p
ackages\googleapiclient\http.py", line 222, in _retry_request
raise exception
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\site-p
ackages\googleapiclient\http.py", line 191, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\site-p
ackages\google_auth_httplib2.py", line 218, in request
response, content = self.http.request(
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\site-p
ackages\httplib2\__init__.py", line 1720, in request
(response, content) = self._request(
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\site-p
ackages\httplib2\__init__.py", line 1440, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, he
aders)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\site-p
ackages\httplib2\__init__.py", line 1363, in _conn_request
conn.request(method, request_uri, body, headers)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\http\c
lient.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\http\c
lient.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\http\c
lient.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\http\c
lient.py", line 1046, in _send_output
self.send(chunk)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\http\c
lient.py", line 968, in send
self.sock.sendall(data)
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\ssl.py
", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Users\Administrador\AppData\Local\Programs\Python\Python38\lib\ssl.py
", line 1173, in send
return self._sslobj.write(data)
socket.timeout: The write operation timed out
Well it works now, as #DaImTo poited to the issue #632 in the google api github, it is not a problem with the api. The problem is that the socket core module has low default timeout. The pc with windows server 2008 that I was using was very slow and was hiting this default timeout, so I just had rise the default timeout by inserting the code in the beginin of the script:
import socket
socket.setdefaulttimeout(600)
Related
I'm trying to run a Dataflow job that used to rune fine but now I get a timeout exception.
The command that I run:
python -m dataflow --input_topic projects/xxx/subscriptions/yyy --events_table_name events --postgres_user postgres --postgres_password xxx --postgres_db test-db --postgres_host xx.xx.xx.xx --region us-central1 --temp_location gs://temp_test_356/tmp/ --project xxx --runner DataflowRunner --setup_file ./setup.py
The exception:
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
WARNING:apache_beam.utils.retry:Retry with exponential backoff: waiting for 4.58774193007135 seconds before retrying _gcs_file_copy because we caught exception: socket.timeout: The write operation timed out
Traceback for above exception (most recent call last):
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/apache_beam/utils/retry.py", line 253, in wrapper
return fun(*args, **kwargs)
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 565, in _gcs_file_copy
self.stage_file(to_folder, to_name, f, total_size=total_size)
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 641, in stage_file
response = self._storage_client.objects.Insert(request, upload=upload)
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py", line 1152, in Insert
return self._RunMethod(
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/apitools/base/py/base_api.py", line 728, in _RunMethod
http_response = http_wrapper.MakeRequest(
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/apitools/base/py/http_wrapper.py", line 348, in MakeRequest
return _MakeRequestNoRetry(
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/apitools/base/py/http_wrapper.py", line 397, in _MakeRequestNoRetry
info, content = http.request(
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/oauth2client/transport.py", line 173, in new_request
resp, content = request(orig_request_method, uri, method, body,
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/oauth2client/transport.py", line 280, in request
return http_callable(uri, method=method, body=body, headers=headers,
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/oauth2client/transport.py", line 173, in new_request
resp, content = request(orig_request_method, uri, method, body,
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/oauth2client/transport.py", line 280, in request
return http_callable(uri, method=method, body=body, headers=headers,
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/httplib2/__init__.py", line 1708, in request
(response, content) = self._request(
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/httplib2/__init__.py", line 1424, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/home/schnitzel/Visual Code Projects/python_scripts/QuestRewardConsumer/env/lib/python3.8/site-packages/httplib2/__init__.py", line 1347, in _conn_request
conn.request(method, request_uri, body, headers)
File "/usr/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1050, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 972, in send
self.sock.sendall(data)
File "/usr/lib/python3.8/ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "/usr/lib/python3.8/ssl.py", line 1173, in send
return self._sslobj.write(data)
Like I said, this job used to run (it still runs fine local). I drained the previous job and then wanted to start it again but without success. I've spent countless hours on this issue. The last time that I had this problem I had to create a new virtual machine just to get my jobs running but now I want to get to the bottom of this. Things that I've tried:
Set GOOGLE_APPLICATION_CREDENTIALS with service account that is Owner.
Configure gcloud. Remove gcloud.
Use neither of above two and use the automatic login mechanism when you run the script.
Change machine type.
Change job name.
Change region.
Use python, use python3.
We have a kubernetes cron job on GCP that submits several copies of the same Python dataflow job, each in its own container. Whenever we need a new copy of the job, we just add it to the spec->jobTemplate->spec->template->spec->containers part of the cron job yaml and adjust the dataflow job parameters as needed. This usually works fine, but the latest copy we tried to add does not work. The existing copies are all still working as expected. The job seems to fail on submission to GCP, and the error message is not very helpful:
Traceback (most recent call last):
File "/app/job.py", line 117, in <module>
newness.pipeline.run_dataflow(sys.argv)
File "/app/newness/pipeline.py", line 480, in run_dataflow
result = pipe.run()
File "/usr/local/lib/python2.7/dist-packages/apache_beam/pipeline.py", line 403, in run
self.to_runner_api(), self.runner, self._options).run(False)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/pipeline.py", line 416, in run
return self.runner.run_pipeline(self)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 389, in run_pipeline
self.dataflow_client.create_job(self.job), self)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 184, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 504, in create_job
return self.submit_job_description(job)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 184, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 551, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 578, in Create
config, request, global_params=global_params)
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpError: <exception str() failed>
The job does not appear in the dataflow console at all.
The previous lines of the container logs look like:
2019-10-13T03:57:47.725542287Z Successfully downloaded apache-beam
2019-10-13T03:58:17.125601280Z INFO:root:Staging SDK sources from PyPI to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/dataflow_python_sdk.tar
2019-10-13T03:58:17.324843623Z INFO:root:Starting GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/dataflow_python_sdk.tar...
2019-10-13T03:58:24.825657227Z INFO:root:Completed GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/dataflow_python_sdk.tar
2019-10-13T03:58:25.225646529Z INFO:root:Downloading binary distribtution of the SDK from PyPi
2019-10-13T03:58:25.225716554Z INFO:root:Executing command: ['/usr/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/tmpk5TfMS', 'apache-beam==2.8.0', '--no-deps', '--only-binary', ':all:', '--python-version', '27', '--implementation', 'cp', '--abi', 'cp27mu', '--platform', 'manylinux1_x86_64']
2019-10-13T03:59:33.926186906Z Collecting apache-beam==2.8.0
2019-10-13T03:59:52.125678183Z Using cached https://files.pythonhosted.org/packages/0f/63/ea5453ba656d060936acf41d2ec057f23aafd69649e2129ac66fdda67d48/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-13T04:00:11.525435891Z Saved /tmp/tmpk5TfMS/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-13T04:00:12.025054706Z Successfully downloaded apache-beam
2019-10-13T04:00:26.726190542Z INFO:root:Staging binary distribution of the SDK from PyPI to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-13T04:00:26.825618945Z INFO:root:Starting GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl...
2019-10-13T04:00:33.725522899Z INFO:root:Completed GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1570936519.827087/apache_beam-2.8.0-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-13T04:06:14.525017097Z Traceback (most recent call last):
...
Why is this job failing to submit? Are there any other logs we can look at to see the cause of this failure?
(Most of our dataflow jobs are written in Java, where the error messages are usually more helpful.)
UPDATE: Running job locally (Windows) with apache-beam 2.16 has the same issue but more logging detail:
...
INFO:root:Starting GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1571606418.971000/apache_beam-2.16.0-cp27-cp27mu-manylinux1_x86_64.whl...
INFO:root:Completed GCS upload to gs://gcs-bucket-name/staging/newness-boosting-c2898.1571606418.971000/apache_beam-2.16.0-cp27-cp27mu-manylinux1_x86_64.whl in 3 seconds.
WARNING:root:Discarding unparseable args: ['job.py', '--days_history=30']
WARNING:root:Discarding unparseable args: ['job.py', '--days_history=30']
WARNING:root:Retry with exponential backoff: waiting for 2.64795143823 seconds before retrying submit_job_description because we caught exception: error: [Errno 10053] An established connection was aborted by the software in your host machine
Traceback for above exception (most recent call last):
File "C:\Python27\lib\site-packages\apache_beam\utils\retry.py", line 206, in wrapper
return fun(*args, **kwargs)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\apiclient.py", line 593, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\clients\dataflow\dataflow_v1b3_client.py", line 657, in Create
config, request, global_params=global_params)
File "C:\Python27\lib\site-packages\apitools\base\py\base_api.py", line 729, in _RunMethod
http, http_request, **opts)
File "C:\Python27\lib\site-packages\apitools\base\py\http_wrapper.py", line 346, in MakeRequest
check_response_func=check_response_func)
File "C:\Python27\lib\site-packages\apitools\base\py\http_wrapper.py", line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File "C:\Python27\lib\site-packages\oauth2client\transport.py", line 169, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\oauth2client\transport.py", line 169, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1694, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1434, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1390, in _conn_request
response = conn.getresponse()
File "C:\Python27\lib\httplib.py", line 1121, in getresponse
response.begin()
File "C:\Python27\lib\httplib.py", line 438, in begin
version, status, reason = self._read_status()
File "C:\Python27\lib\httplib.py", line 394, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\lib\socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "C:\Python27\lib\ssl.py", line 754, in recv
return self.read(buflen)
File "C:\Python27\lib\ssl.py", line 641, in read
v = self._sslobj.read(len)
... retries 4 times total ...
Traceback (most recent call last):
File "job.py", line 117, in <module>
newness.pipeline.run_dataflow(sys.argv)
File "C:\Users\LeeW\Desktop\newness\newness\pipeline.py", line 480, in run_dataflow
result = pipe.run()
File "C:\Python27\lib\site-packages\apache_beam\pipeline.py", line 407, in run
self._options).run(False)
File "C:\Python27\lib\site-packages\apache_beam\pipeline.py", line 420, in run
return self.runner.run_pipeline(self, self._options)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\dataflow_runner.py", line 485, in run_pipeline
self.dataflow_client.create_job(self.job), self)
File "C:\Python27\lib\site-packages\apache_beam\utils\retry.py", line 206, in wrapper
return fun(*args, **kwargs)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\apiclient.py", line 546, in create_job
return self.submit_job_description(job)
File "C:\Python27\lib\site-packages\apache_beam\utils\retry.py", line 219, in wrapper
raise_with_traceback(exn, exn_traceback)
File "C:\Python27\lib\site-packages\apache_beam\utils\retry.py", line 206, in wrapper
return fun(*args, **kwargs)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\apiclient.py", line 593, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "C:\Python27\lib\site-packages\apache_beam\runners\dataflow\internal\clients\dataflow\dataflow_v1b3_client.py", line 657, in Create
config, request, global_params=global_params)
File "C:\Python27\lib\site-packages\apitools\base\py\base_api.py", line 729, in _RunMethod
http, http_request, **opts)
File "C:\Python27\lib\site-packages\apitools\base\py\http_wrapper.py", line 346, in MakeRequest
check_response_func=check_response_func)
File "C:\Python27\lib\site-packages\apitools\base\py\http_wrapper.py", line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File "C:\Python27\lib\site-packages\oauth2client\transport.py", line 169, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\oauth2client\transport.py", line 169, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1694, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1434, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2\__init__.py", line 1390, in _conn_request
response = conn.getresponse()
File "C:\Python27\lib\httplib.py", line 1121, in getresponse
response.begin()
File "C:\Python27\lib\httplib.py", line 438, in begin
version, status, reason = self._read_status()
File "C:\Python27\lib\httplib.py", line 394, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\lib\socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "C:\Python27\lib\ssl.py", line 754, in recv
return self.read(buflen)
File "C:\Python27\lib\ssl.py", line 641, in read
v = self._sslobj.read(len)
socket.error: [Errno 10053] An established connection was aborted by the software in your host machine
Which version of Beam Python SDK are you using?
I'm using Gmail API to send an email with attachments in Python 3.
I'm trying the same code as google developers as shown below:
https://developers.google.com/gmail/api/guides/sending
The problem is that when attachments are 4.2KB or 2.6MB, the code works Well; but when attachments are 3.0MB or 9.6MB or bigger, an Error occurs:
Traceback (most recent call last):
File "quickstart2.py", line 184, in <module>
main()
File "quickstart2.py", line 170, in main
send_message(service, "me", message)
File "quickstart2.py", line 147, in send_message
message = (service.users().messages().send(userId=user_id, body=message).execute())
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/googleapiclient/http.py", line 837, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/googleapiclient/http.py", line 176, in _retry_request
raise exception
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/googleapiclient/http.py", line 163, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
redirections, connection_type)
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
connection_type=connection_type)
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/httplib2/__init__.py", line 1322, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/httplib2/__init__.py", line 1072, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/home/yizhu/anaconda3/lib/python3.6/site-packages/httplib2/__init__.py", line 996, in _conn_request
conn.request(method, request_uri, body, headers)
File "/home/yizhu/anaconda3/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/yizhu/anaconda3/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/yizhu/anaconda3/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/yizhu/anaconda3/lib/python3.6/http/client.py", line 1065, in _send_output
self.send(chunk)
File "/home/yizhu/anaconda3/lib/python3.6/http/client.py", line 986, in send
self.sock.sendall(data)
File "/home/yizhu/anaconda3/lib/python3.6/ssl.py", line 972, in sendall
v = self.send(byte_view[count:])
File "/home/yizhu/anaconda3/lib/python3.6/ssl.py", line 941, in send
return self._sslobj.write(data)
File "/home/yizhu/anaconda3/lib/python3.6/ssl.py", line 642, in write
return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe
What's the problem here?
Thanks
It seems an exception is raised at _retry_request.
I haven't encountered this error myself, but there's a thread on github discussing about the same error.
https://github.com/google/google-api-python-client/issues/218
Try using httplib2shim, it seems oauth2client is still not replaced by google-auth.
Another suggestion I found was to use the MEDIA /upload option for files over 10 MB. Docs for how to use /upload: https://developers.google.com/gmail/api/v1/reference/users/messages/send
I am trying to make a GET request over TOR which is listening on 127.0.0.1:9050
I've installed request socks: pip install -U requests[socks]
import requests
tor_proxy = {'http': 'socks5://127.0.0.1:9050', 'https': 'socks5://127.0.0.1:9050'}
r = requests.get("http://www.google.com", proxies=tor_proxy, timeout=20)
Here is the error from when I run the code
Traceback (most recent call last):
r = requests.get("http://www.google.com", proxies=tor_proxy, timeout=20)
File "C:\Python27\lib\site-packages\requests\api.py", line 71, in get
return request('get', url, params=params, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 585, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 403, in send
timeout=timeout
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 578, in urlopen
chunked=chunked)
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 362, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Python27\lib\httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 1097, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 1053, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 897, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 859, in send
self.connect()
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connection.py", line 167, in connect
conn = self._new_conn()
File "C:\Python27\lib\site-packages\requests\packages\urllib3\contrib\socks.py", line 81, in _new_conn
**extra_kw
File "C:\Python27\lib\site-packages\socks.py", line 195, in create_connection
sock.connect((remote_host, remote_port))
File "C:\Python27\lib\site-packages\socks.py", line 747, in connect
negotiate(self, dest_addr, dest_port)
File "C:\Python27\lib\site-packages\socks.py", line 419, in _negotiate_SOCKS5
CONNECT, dest_addr)
File "C:\Python27\lib\site-packages\socks.py", line 482, in _SOCKS5_request
resolved = self._write_SOCKS5_address(dst, writer)
File "C:\Python27\lib\site-packages\socks.py", line 517, in _write_SOCKS5_address
addr_bytes = socket.inet_pton(family, host)
AttributeError: 'module' object has no attribute 'inet_pton'
Jun 11 13:13:55.000 [notice] Tried for 120 seconds to get a connection to [scrubbed]:0. Giving up. (waiting for socks info)
Thanks
The problem is with the library itself, in particular PySocks socks.py, inet_pton is only available on unix:
socket.inet_pton(address_family, ip_string)
Convert an IP address from its family-specific string format to a packed, binary format. inet_pton() is useful when a library or network protocol calls for an object of type struct in_addr (similar to inet_aton()) or struct in6_addr.
Supported values for address_family are currently AF_INET and AF_INET6. If the IP address string ip_string is invalid, socket.error will be raised. Note that exactly what is valid depends on both the value of address_family and the underlying implementation of inet_pton().
Availability: Unix (maybe not all platforms).
One workaround is to install win_inet_pton and import it in your script, the method will automatically be added to the socket lib.
I have a python 3.5 app that makes calls to Google Cloud Storage using the python sdk.
Every once in a while, for 10-30 minutes, all calls to the API fail with BrokenPipeError or ssl.SSLError errors. After some time, they just start working again, I have not noticed a pattern as to why.
Is this a known issue? Is it specific to the python sdk or is this a real performance regression on the side of google?
It should also be noted that these errors will emanate from the same code running on my local machine as well as from a GCE machine.
The trace for BrokenPipe:
Traceback (most recent call last):
File "oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "googleapiclient/http.py", line 722, in execute
body=self.body, headers=self.headers)
File "oauth2client/client.py", line 596, in new_request
redirections, connection_type)
File "httplib2/__init__.py", line 1314, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "httplib2/__init__.py", line 1064, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "httplib2/__init__.py", line 988, in _conn_request
conn.request(method, request_uri, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1083, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1128, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1079, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 911, in _send_output
self.send(msg)
File "/usr/lib/python3.5/http/client.py", line 885, in send
self.sock.sendall(data)
File "/usr/lib/python3.5/ssl.py", line 886, in sendall
v = self.send(data[count:])
File "/usr/lib/python3.5/ssl.py", line 856, in send
return self._sslobj.write(data)
File "/usr/lib/python3.5/ssl.py", line 581, in write
return self._sslobj.write(data)
BrokenPipeError[Errno 32] Broken pipe
The trace for ssl.SSLError:
File "oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "googleapiclient/http.py", line 722, in execute
body=self.body, headers=self.headers)
File "oauth2client/client.py", line 596, in new_request
redirections, connection_type)
File "httplib2/__init__.py", line 1314, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "httplib2/__init__.py", line 1064, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "httplib2/__init__.py", line 1017, in _conn_request
response = conn.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1174, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 282, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 243, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.5/ssl.py", line 924, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.5/ssl.py", line 786, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.5/ssl.py", line 570, in read
v = self._sslobj.read(len, buffer)
ssl.SSLError[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1974)
Definitely looks like an intermittent issue on Googles side.
The broken pipe issue relates to httpclient2 being unable to reconnect an existing connection to their API, this is the error that has the greatest impact on our services. We also on few occasions received "503 Backend Error".
Our "solution" was to basically allow the connections to close themselves by releasing the client once done and creating a new one for the next request.
Bare in mind though that our requests are very sparse, services using Cloud Storage as primary storage probably wants to keep the connections open for as long as possible.