I need to iterate all emails in a folder, eventually exporting to CSV format. Here's the important bits of code:
credentials = ServiceAccount(user, password)
...
qs = account.inbox.all()
qs.page_size = 100
qs.order_by('-datetime_received')
emails = qs.iterator()
for i, email in enumerate(emails):
print(i, email.subject, email.sender, email.datetime_received)
When I run, I get an error every time, but not at the same point in the process.
Sometimes I get one page back (i == 99), sometimes I get three pages back (i==299) or five pages back (i==499). Five pages are the most I've gotten back. No matter where the error occurs, it's this error. FWIW, the error does happen at page breaks.
Traceback (most recent call last):
File "outlook.py", line 81, in <module>
for i, email in enumerate(emails):
File "C:\dev\programs\anaconda3\lib\site-packages\exchangelib\queryset.py", line 390, in _as_items
for i in iterable:
File "C:\dev\programs\anaconda3\lib\site-packages\exchangelib\account.py", line 628, in fetch
shape=IdOnly,
File "C:\dev\programs\anaconda3\lib\site-packages\exchangelib\services.py", line 602, in _pool_requests
for elem in r.get():
File "C:\dev\programs\anaconda3\lib\multiprocessing\pool.py", line 644, in get
raise self._value
File "C:\dev\programs\anaconda3\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\dev\programs\anaconda3\lib\site-packages\exchangelib\services.py", line 591, in <lambda>
lambda c: self._get_elements(payload=payload_func(c, **kwargs)),
File "C:\dev\programs\anaconda3\lib\site-packages\exchangelib\services.py", line 90, in _get_elements
response = self._get_response_xml(payload=payload)
File "C:\dev\programs\anaconda3\lib\site-packages\exchangelib\services.py", line 174, in _get_response_xml
res = self._get_soap_payload(soap_response=soap_response_payload)
File "C:\dev\programs\anaconda3\lib\site-packages\exchangelib\services.py", line 229, in _get_soap_payload
raise SOAPError('Unknown SOAP response: %s' % xml_to_str(body))
exchangelib.errors.SOAPError: Unknown SOAP response: <ns0:Body xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" />
Is this the correct approach to walking through all emails? What does this error mean and how do I fix it? Thanks!
UPDATE:
Following is the captured output. We don't see anything that looks like a 'backoff' message. Does this help you identify any issues?
<< This run had page size set to 10 >>
DEBUG:exchangelib.util:Session 5122 thread 9556: Useful response from https://our.host.name/EWS/Exchange.asmx
DEBUG:exchangelib.protocol:Server our.host.name: Releasing session 5122
DEBUG:exchangelib.services:Trying API version Exchange2010_SP2 for account myaccount#myhost.com
DEBUG:exchangelib.services:FindItem: Got page with next offset 360 (last_page False)
DEBUG:exchangelib.services:Starting GetItem._get_elements worker 36 for 10 items
DEBUG:exchangelib.services:GetItem._get_elements result 30 is ready early
<< 10 of these DEBUG lines come out >>
DEBUG:exchangelib.services:Getting item ('AAMkADJjMDIwNmQwLWQ2NzYtNDk5OC04MjgxLT
MzZTdhMTk4YWRiNQBGAAAAAABGQ9HU4LA/RY8Mwwr5Jp5OBwDOJO2TCOmBSZeM4xH+g3x5AAAAiE7BAA
DOJO2TCOmBSZeM4xH+g3x5AAAe2nAfAAA=', 'CQAAABYAAADOJO2TCOmBSZeM4xH+g3x5AAAe2wDA')
DEBUG:exchangelib.services:Getting item ('AAMkADJjMDIwNmQwLWQ2NzYtNDk5OC04MjgxLT
MzZTdhMTk4YWRiNQBGAAAAAABGQ9HU4LA/RY8Mwwr5Jp5OBwDOJO2TCOmBSZeM4xH+g3x5AAAAiE7BAA
DOJO2TCOmBSZeM4xH+g3x5AAAe2nAeAAA=', 'CQAAABYAAADOJO2TCOmBSZeM4xH+g3x5AAAe2wC+')
...
<<Seemed to process the above page successfully, then starts w/ errors >>
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 502, in read_chunked
DEBUG:exchangelib.protocol:Server our.host.name: Waiting for session
DEBUG:exchangelib.protocol:Server our.host.name: Got session 5122
DEBUG:exchangelib.util:Session 5122 thread 6408: retry 0 timeout 120000
POST'ing to https://our.host.name/EWS/Exchange.asmx after 1000s wait self._update_chunk_length()
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 448, in _update_chunk_length
line = self._fp.fp.readline()
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\socket.py", line 575, in readinto
return self._sock.recv_into(b)
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\ssl.py", line 929, in recv_into
return self.read(nbytes, buffer)
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\ssl.py", line 791, in read
return self._sslobj.read(len, buffer)
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\ssl.py", line 575, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\models.py", line 676, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 353, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 530, in read_chunked
self._original_response.close()
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 250, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ("Connection broken:
ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)",
ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "email_test.py", line 27, in <module>
for i, email in enumerate(emails):
File "%WORKSPACE%\nlp\exchangelib\exchangelib\queryset.py", line 390, in _as_items
for i in iterable:
File "%WORKSPACE%\nlp\exchangelib\exchangelib\account.py", line 628, in fetch
shape=IdOnly,
File "%WORKSPACE%\nlp\exchangelib\exchangelib\services.py", line 596, in _pool_requests
for elem in r.get():
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 608, in get
raise self._value
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "%WORKSPACE%\nlp\exchangelib\exchangelib\services.py", line 585, in <lambda>
lambda c: self._get_elements(payload=payload_func(c, **kwargs)),
File "%WORKSPACE%\nlp\exchangelib\exchangelib\services.py", line 89, in _get_elements
response = self._get_response_xml(payload=payload)
File "%WORKSPACE%\nlp\exchangelib\exchangelib\services.py", line 163, in _get_response_xml
allow_redirects=False)
File "%WORKSPACE%\nlp\exchangelib\exchangelib\util.py", line 547, in post_ratelimited
_raise_response_errors(r, protocol, log_msg, log_vals) # Always raises an exception
File "%WORKSPACE%\nlp\exchangelib\exchangelib\util.py", line 603, in _raise_response_errors
raise r.headers['TimeoutException']
File "%WORKSPACE%\nlp\exchangelib\exchangelib\util.py", line 494, in post_ratelimited
r = session.post(url=url, headers=headers, data=data, allow_redirects=False, timeout=protocol.TIMEOUT)
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\sessions.py", line 522, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\sessions.py", line 628, in send
r.content
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\models.py", line 755, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "%USERPROFILE%\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\models.py", line 679, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken:
ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)",
ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
Related
I'm trying to make automated crypto monitoring program with python and using ccxt library.
(python version : 3.11)
After running this program for 6 to 7 hours, the following error occurred and the program was terminated.
Traceback (most recent call last):
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\util\retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\packages\six.py", line 769, in reraise
raise value.with_traceback(tb)
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\ccxt\base\exchange.py", line 599, in fetch
response = self.session.request(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\adapters.py", line 547, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\Users\uid\Python\PJT_Crypto\Binance_Crypto_realtime.py", line 312, in <module>
Tracking_Crypto_Start_Start('15m', f, f2)
File "c:\Users\uid\Python\PJT_Crypto\Binance_Crypto_realtime.py", line 125, in Tracking_Crypto_Start_Start
get_data = bollinger_trade_logic(exchange, interval)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\uid\Python\PJT_Crypto\Binance_Crypto_realtime.py", line 29, in bollinger_trade_logic
symbol_df = get_data_frame(obj_binance, interval)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\uid\Python\PJT_Crypto\Binance_Crypto_realtime.py", line 19, in get_data_frame
p = obj_binance.fetch_ohlcv(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\ccxt\binance.py", line 2709, in fetch_ohlcv
response = getattr(self, method)(self.extend(request, params))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\ccxt\base\exchange.py", line 505, in inner
return entry(_self, **inner_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\ccxt\binance.py", line 6200, in request
response = self.fetch2(path, api, method, params, headers, body, config, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\ccxt\base\exchange.py", line 2830, in fetch2
return self.fetch(request['url'], request['method'], request['headers'], request['body'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uid\AppData\Local\Programs\Python\Python311\Lib\site-packages\ccxt\base\exchange.py", line 652, in fetch
raise NetworkError(details) from e
ccxt.base.errors.NetworkError: binance GET https://fapi.binance.com/fapi/v1/klines?interval=15m&limit=20&symbol=XRPUSDT
I think the Binance server forcibly rejected the request because I made too many requests.
Main code like this:
def get_data_frame(obj_binance, interval):
p = obj_binance.fetch_ohlcv(
'XRP/USDT', interval, limit=20)
df = pd.DataFrame(p, columns=['date', 'open',
'high', 'low', 'close', 'vol'])
return df
def Start_Monitoring(Param1,Param2):
try:
get_data = get_data_frame(Param1,Param2)
.....
...
time.sleep(1)
except ccxt.RequestTimeout as e:
print(e)
exchange = binanceperps(param) # This line for re-connect & request again. But never catched e untill now
There were many similar articles on the Internet, but I couldn't find a case that fits my case.
Please give me some advice.
Thanks
BR
I tried to add catch/exception code in while loop. But It couldn't capture the error.
The error what I post was occurred in python library not ccxt library.
I have a "connection aborted" error when calling the query() function of the bigquery python API. I have a loop which calls one sql with API at a time. The weird thing is that the error happened randomly. If 10 queries are run, this time it may have 8 queries succeeded, 2 failed due to the "connection aborted" error. Next time, it may have 6 succeeded and 4 failed. I have run each sql individually through "bq query" command and BigQuery Console. The SQLs have no problem.
I run the program on Google Chromebook. Python version is 3.7.3 and the google-cloud-bigquery version is 2.31.0. The code snippet to call the API is as follows:
client = bigquery.Client(project=cfg.project_id)
job_config = bigquery.QueryJobConfig(use_query_cache = False)
if cfg.test_mode.lower() == "true":
job_config.dry_run = True
query_job = client.query(sql,
job_config = job_config
)
query_job.result()
df = query_job.to_dataframe()
The log of the error message is as follows:
2022-01-11 11:15:06,262.262 ERROR data_validator - run_a_testcase: Exception happens when processing query: DHCP#Null_relay_ip. Exception: Deadline of 600.0s exceeded while calling functools.partial(functools.partial(<bound method JSONConnection.api_request of <google.cloud.bigquery._http.Connection object at 0x7d05e67bbba8>>, method='POST', path='/projects/ml-mps-app-ctp-anlys-d-ce71/jobs', data={'jobReference': {'jobId': '82e30ee3-380e-4481-98f4-67b473c623d0', 'projectId': 'ml-mps-app-ctp-anlys-d-ce71'}, 'configuration': {'query': {'useQueryCache': False, 'useLegacySql': False, 'query': "select count(*) from res_ctapprocessed_us_d.dhcpdemo where last_activity BETWEEN cast('2021-10-12 16:00:00.000' as DATETIME) AND cast('2021-10-12 16:59:59.999' AS DATETIME) and (relay_ip is null)"}, 'dryRun': True}}, timeout=None)), last exception: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')),
Traceback (most recent call last):
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1352, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 279, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/s013949/py373/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/s013949/py373/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1352, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 279, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/s013949/py373/lib/python3.7/site-packages/google/api_core/retry.py", line 189, in retry_target
return target()
File "/home/s013949/py373/lib/python3.7/site-packages/google/cloud/_http.py", line 480, in api_request
timeout=timeout,
File "/home/s013949/py373/lib/python3.7/site-packages/google/cloud/_http.py", line 338, in _make_request
method, url, headers, data, target_object, timeout=timeout
File "/home/s013949/py373/lib/python3.7/site-packages/google/cloud/_http.py", line 376, in _do_request
url=url, method=method, headers=headers, data=data, timeout=timeout
File "/home/s013949/py373/lib/python3.7/site-packages/google/auth/transport/requests.py", line 486, in request
**kwargs
File "/home/s013949/py373/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/s013949/py373/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/s013949/py373/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "data_validator.py", line 332, in run_a_testcase
job_config = job_config
File "/home/s013949/py373/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 3391, in query
future = do_query()
File "/home/s013949/py373/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 3368, in do_query
query_job._begin(retry=retry, timeout=timeout)
File "/home/s013949/py373/lib/python3.7/site-packages/google/cloud/bigquery/job/query.py", line 1249, in _begin
super(QueryJob, self)._begin(client=client, retry=retry, timeout=timeout)
File "/home/s013949/py373/lib/python3.7/site-packages/google/cloud/bigquery/job/base.py", line 517, in _begin
timeout=timeout,
File "/home/s013949/py373/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
return call()
File "/home/s013949/py373/lib/python3.7/site-packages/google/api_core/retry.py", line 291, in retry_wrapped_func
on_error=on_error,
File "/home/s013949/py373/lib/python3.7/site-packages/google/api_core/retry.py", line 211, in retry_target
last_exc,
File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 600.0s exceeded while calling functools.partial(functools.partial(<bound method JSONConnection.api_request of <google.cloud.bigquery._http.Connection object at 0x7d05e67bbba8>>, method='POST', path='/projects/ml-mps-app-ctp-anlys-d-ce71/jobs', data={'jobReference': {'jobId': '82e30ee3-380e-4481-98f4-67b473c623d0', 'projectId': 'ml-mps-app-ctp-anlys-d-ce71'}, 'configuration': {'query': {'useQueryCache': False, 'useLegacySql': False, 'query': "select count(*) from res_ctapprocessed_us_d.dhcpdemo where last_activity BETWEEN cast('2021-10-12 16:00:00.000' as DATETIME) AND cast('2021-10-12 16:59:59.999' AS DATETIME) and (relay_ip is null)"}, 'dryRun': True}}, timeout=None)), last exception: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Any comments are highly appreciated. Thanks a lot for your help.
The problem solved by changing the "mtu" setting of the network interface on Chromebook. The command is "sudo ifconfig eth0 mtu 1200". Thanks.
my Code runs on my local network(using VBOX) but when im trying to run it on another network im getting this Error:
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
my code is:
import docker
client = docker.DockerClient(base_url ='ssh://george#192.168.56.103')
print(client.images.list())
the whole Error message is:
C:\Users\George\PycharmProjects\docker_project\venv\Scripts\python.exe C:/Users/George/PycharmProjects/docker_project/docker_test.py
Traceback (most recent call last):
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\George\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1332, in getresponse
response.begin()
File "C:\Users\George\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\George\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\util\retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\George\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1332, in getresponse
response.begin()
File "C:\Users\George\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\George\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\docker\api\client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\docker\api\daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\docker\utils\decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\docker\api\client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\requests\sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/George/PycharmProjects/docker_project/docker_test.py", line 25, in <module>
client = docker.DockerClient(base_url ='ssh://george#192.168.56.103')
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\docker\client.py", line 45, in __init__
self.api = APIClient(*args, **kwargs)
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\docker\api\client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "C:\Users\George\PycharmProjects\docker_project\venv\lib\site-packages\docker\api\client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
what could be the reason for that?
the problem was not with the connection itself. i tried to run the script localy on the remote server:
client = docker.from_env()
i had a permission denied exception.
the user must have sudo permission for the docker on the remote server to be able to use the Docker SDK. here more infos how to do that:
https://docs.docker.com/engine/install/linux-postinstall/
whenever it made the script works fine.
My scrapy application is outputting this long chain of exceptions and I am failing to see what the issue is and the last one has me especially confused.
Before I explain why here is the chain:
2020-11-04 17:38:58,394:ERROR:Error while obtaining start requests
Traceback (most recent call last):
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\http\client.py", line 1347, in getresponse
response.begin()
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\http\client.py", line 307, in begin
version, status, reason = self._read_status()
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\http\client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\util\retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\http\client.py", line 1347, in getresponse
response.begin()
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\http\client.py", line 307, in begin
version, status, reason = self._read_status()
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\http\client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\shadow_useragent\core.py", line 35, in _update
r = requests.get(url=self.URL)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\scrapy\core\engine.py", line 129, in _next_request
request = next(slot.start_requests)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\scrapy_splash\middleware.py", line 167, in process_start_requests
for req in start_requests:
File "C:\Users\lguarro\Documents\Work\SearchEngine_Pure\SearchEngine_Pure\spiders\SearchEngine.py", line 36, in start_requests
user_agent = self.ua.random_nomobile
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\shadow_useragent\core.py", line 120, in random_nomobile
return self.pickrandom(exclude_mobile=True)
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\shadow_useragent\core.py", line 83, in pickrandom
self.update()
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\shadow_useragent\core.py", line 59, in update
self._update()
File "C:\Users\lguarro\Anaconda3\envs\virtual_workspace\lib\site-packages\shadow_useragent\core.py", line 38, in _update
self.logger.error(r.content.decode('utf-8'))
UnboundLocalError: local variable 'r' referenced before assignment
Now the last exception is complaining about some
UnboundLocalError: local variable 'r' referenced before assignment
The only code which is mine in that trace is the SearchEngine.py file which doesn't even have a variable 'r' thus leaving me very confused. Here is the implementation of start_requests from which the error occurs:
def start_requests(self):
user_agent = self.ua.random_nomobile # Exception raised here
rec = self.mh.FindIdleOneWithNoURLs()
if rec:
self.logger.info("Starting url scrape for company, %s using user agent: %s", rec["Company"], user_agent)
script = self.template.substitute(useragent=user_agent, searchquery=rec["Company"])
yield SplashRequest(url=self.url, callback=self.parse, endpoint="execute",
args={
'lua_source': script
},
meta={'RecID': rec["_id"], 'Company': rec["Company"]},
errback = self.logerror
)
It is complaining about the first line in that function for which I see no problem.
In case it is relevant, I will also add that my script seemed to be running fine just yesterday but today I had to reset my Docker configuration (that the splash container is running in) and since then I haven't been able to run my script smoothly.
I found out what was causing the issue! In fact there was no error on part, instead it is a bug inside the shadow-useragent library.
The library periodically makes an API request to fetch a list of the most used user agents and the server corresponding to this API is down and the authors of shadow-useragent were not properly handling the exception.
Fortunately shadow-useragent does cache the list of user agents that it was most recently able to receive. So my solution was to edit the shadow-useragent code to bypass the update function entirely and to use the cached list (inside the data.pk file) beyond its scheduled update. If anyone else runs into this issue, this is a temporary solution you can use until that server gets up and running again.. hopefully soon!
Recently, as I have tried to connect to make requests with the eBay finding SDK with python, I have been getting Connection errors. The error print does not state what is wrong.
This is the python script I wrote to make API requests.
from ebaysdk.finding import Connection as find_connect
import xmltodict
APP_ID = 'ActualID was removed-PRD-For privacy'
# keywords = input("Enter search keywords(e.g 'white board'): ")
api = find_connect(appid=APP_ID, config_file=None, siteid="EBAY-ENCA")
request = {
'keywords': "Iphone x 64gb",
'itemFilter': [
{'name': 'Condition', 'value': 'Used'},
{'name': 'currency', 'value': 'CAD'},
{'name': 'minPrice', 'value': 100.0}
],
'paginationInput': {
'entriesPerPage': 100,
'pageNumber': 10
},
}
resp = api.execute('findItemsByKeywords', request).dict() # Change from XML to dictionary
When I run the script, I get this error
Traceback (most recent call last):
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1322, in getresponse
response.begin()
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 264, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 669, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\util\retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1322, in getresponse
response.begin()
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 264, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 669, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/Lase.Adebayo/PycharmProjects/eBay Data Collector/eBayCaller/ebay scripts/ebayxml.py", line 24, in <module>
resp = api.execute('findItemsByKeywords', request).dict() # Change from XML to dictionary
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\ebaysdk\connection.py", line 123, in execute
self.execute_request()
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\ebaysdk\connection.py", line 189, in execute_request
self.response = self.session.send(self.request,
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\Lase.Adebayo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
This same code worked about two weeks ago.
It seems that the python sdk devs are yet to change code from HTTP to HTTPS. You can use a YAML config file to specify https requests.
svcs.ebay.com:
appid: YOUR_APP_ID
https: 1
YAML configuration is needed if you want to make calls over https for version 2.1.5 of the SDK or earlier. 2.2.0+ will force all calls over https since eBay has moved completely to https.
https://github.com/timotheus/ebaysdk-python/wiki/YAML-Configuration
If you don't want to use YAML, you can add a parameter to the api_call:
For instance, here is the code that I use when not using a YAML in Python:
api = finding(https=True, appid='your_app_id', config_file = None)
OR you could set https=1