Py4JJavaError. Can't manipulate an Spark Dataframe - python

Evening!
I'm with the following error in my code and can't understand exactly what should i do to solve it:
File "C:\Spark\python\pyspark\sql\dataframe.py", line 804, in count
return int(self._jdf.count())
File "C:\Users\ariel\AppData\Roaming\Python\Python39\site-packages\py4j\java_gateway.py", line 1321, in __call__
return_value = get_return_value(
File "C:\Spark\python\pyspark\sql\utils.py", line 190, in deco
return f(*a, **kw)
File "C:\Users\ariel\AppData\Roaming\Python\Python39\site-packages\py4j\protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: <unprintable Py4JJavaError object>
During handling of the above exception, another exception occurred:
response = connection.send_command(command)
File "C:\Users\ariel\AppData\Roaming\Python\Python39\site-packages\py4j\clientserver.py", line 539, in send_command
raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
ConnectionRefusedError: [WinError 10061] Nenhuma conexão pôde ser feita porque a máquina de destino as recusou ativamente
My inputs are less then 10g, and even turning my default spark config to use 10g it is not working.
I already reinstalled spark and Py4J and is not working.
This error occurs when i was just trying to do a count on a Spark Dataframe after aggregation!I happens too when i was trying to convert the Spark dataframe to a Pandas Dataframe.

Related

exchangelib not work with some subject filters

I am having the following problem to get the unread emails.
When the subject is "REALIZADAS" it works fine and brings me the emails quickly, but when the subject is "G.09.xlsx" it takes a long time and fails to deliver this error
'An existing connection was forced to break by the remote host', None, 10054, None) ", ConnectionResetError (10054, 'An existing connection was forced to break by the remote host'
This is the untranslated error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Rocketbot\Desktop\Rocketbot\\modules\libs\exchangelib\services.py", line 89, in _get_elements
response = self._get_response_xml(payload=payload)
File "C:\Users\Rocketbot\Desktop\Rocketbot\\modules\libs\exchangelib\services.py", line 162, in _get_response_xml
stream=self.streaming,
File "C:\Users\Rocketbot\Desktop\Rocketbot\\modules\libs\exchangelib\util.py", line 677, in post_ratelimited
_raise_response_errors(r, protocol, log_msg, log_vals) # Always raises an exception
File "C:\Users\Rocketbot\Desktop\Rocketbot\\modules\libs\exchangelib\util.py", line 749, in _raise_response_errors
raise response.headers['TimeoutException']
File "C:\Users\Rocketbot\Desktop\Rocketbot\\modules\libs\exchangelib\util.py", line 631, in post_ratelimited
stream=stream)
File "site-packages\requests\sessions.py", line 578, in post
File "site-packages\requests\sessions.py", line 530, in request
File "site-packages\requests\sessions.py", line 683, in send
File "site-packages\requests\models.py", line 829, in content
File "site-packages\requests\models.py", line 754, in generate
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(10054, 'Se ha forzado la interrupción de una conexión existente por el host remoto', None, 10054, None)", ConnectionResetError(10054, 'Se ha forzado la interrupción de una conexión existente por el host remoto', None, 10054, None))
This is the code
id_ = []
mails_filters = a.inbox.filter(subject=filter_)
for m in mails_filters:
if not m.is_read:
id_.append(m.id)
I tried with
.filter (subject__contains = filter_)
.filter (Q ("subject:" + filter_))
and it only worked for me when the filter was .filter (subject__contains = "09"), if I use "G" or "xlsx" it doesn't work.
I do not know what it could be. Maybe it's the time it takes to search? Can I modify that time so that it waits longer without getting the error?

catching "socket.timeout The read operation timed out" in python

I'm making calls to an API and am receiving the following timeout error:
socket.timeout The read operation timed out
which traces back from
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/http/client.py", line 1198, in getresponse
response.begin()
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/socket.py", line 576, in readinto
return self._sock.recv_into(b)
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/ssl.py", line 937, in recv_into
return self.read(nbytes, buffer)
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/ssl.py", line 799, in read
return self._sslobj.read(len, buffer)
File "/Users/someuser/anaconda/envs/python3/lib/python3.5/ssl.py", line 583, in read
v = self._sslobj.read(len, buffer)
socket.timeout The read operation timed out
How can I catch this error from the top of the traceback?
This is a duplicate question, however the correct approach is to catch socket.timeout by using
import socket
try:
...
except socket.timeout:
...
The error states the exception that was thrown explicitly and you can rely on that. Even though the calling method is the ssl lib, the error seems to be related to the actual socket connection.
Google search led to me https://community.home-assistant.io/t/difficulty-catching-an-exception/12955/6
It seems like SSL throws ssl.SSLError when it times out. You could try something like:
import ssl
def getresponse():
...
try:
response.begin
except ssl.SSLError as err:
handle_error(err)

Django Closed Socket Error with Repeated AJAX Calls [WinError 10053] An established connection was aborted by the software in your host machine

I'm trying to implement a live search through the jquery plugin Select2 in my Django 1.11.4 project. Running Python 3.6. When I type in the textbox, the server doesn't seem to be able to handle the number of requests and it closes, followed by these series of errors.
I've spent a couple hours following a couple threads on SO SO(more) , Google, etc but none have a working solution. I thought it was a python version issue, but I've upgraded from 2.7 to 3.6 and still have it. Any help would be great thanks!
This the view that's being called by the ajax call:
def directory_search(request):
# Query text from search bar
query = request.GET.get('q', None)
if query:
# Searches the DB for matching value
customers = Customer.objects.filter(
Q(name__icontains= query)|
Q(contract_account__icontains= query)|
Q(address__icontains= query)|
Q(address__icontains= query)|
Q(email__icontains= query)
).distinct()
# Turns queryset into dict
customers = customers.values("contract_account","name")
# Turns dict into list
customers_list = list(customers)
return JsonResponse(customers_list, safe=False)
else:
return JsonResponse(data={'success': False,'errors': 'No mathing items found'})
This is the js/ajax call for the Select2 plugin:
$(".customer-search-bar").select2({
ajax:{
dataType: 'json',
type: 'GET',
url:"{% url 'portal:directory_search' %}",
data: function (params) {
var queryParameters = {
q: params.term
}
return queryParameters;
},
processResults: function (data) {
return {
results: $.map(data, function (item) {
return {
text: item.name,
id: item.contract_account
}
})
};
}
},
escapeMarkup: function (markup) { return markup; },
minimumInputLength: 1,
templateResult: formatRepo,
templateSelection: formatRepoSelection,
language: { errorLoading:function(){ return "Searching..." }}
});
Errors: (I just broke them apart to make it more legible, but they occurred in the presented order)
1.
Traceback (most recent call last):
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 138, in run
self.finish_response()
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 180, in finish_response
self.write(data)
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 274, in write
self.send_headers()
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 332, in send_headers
self.send_preamble()
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 255, in send_preamble
('Date: %s\r\n' % format_date_time(time.time())).encode('iso-8859-1')
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 453, in _write
result = self.stdout.write(data)
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\socketserver.py", line 775, in write
self._sock.sendall(b)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
2.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 141, in run
self.handle_error()
File "C:\Users\leep\PythonStuff\virtual_environments\rpp_3.6\lib\site-packages\django\core\servers\basehttp.py", line 88, in handle_error
super(ServerHandler, self).handle_error()
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 368, in handle_error
self.finish_response()
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 180, in finish_response
self.write(data)
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 274, in write
self.send_headers()
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 331, in send_headers
if not self.origin_server or self.client_is_modern():
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 344, in client_is_modern
return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable
3.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\socketserver.py", line 639, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\socketserver.py", line 696, in __init__
self.handle()
File "C:\Users\leep\PythonStuff\virtual_environments\rpp_3.6\lib\site-packages\django\core\servers\basehttp.py", line 155, in handle
handler.run(self.server.get_app())
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\handlers.py", line 144, in run
self.close()
File "C:\Users\leep\AppData\Local\Programs\Python\Python36-32\lib\wsgiref\simple_server.py", line 35, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
It appears that its a python 2.7 known bug, maybe it remains on python 3.6:
https://bugs.python.org/issue14574
I'm having a similar issue in django, also running python 3.6
TypeError: 'NoneType' object is not subscriptable followed by AttributeError: 'NoneType' object has no attribute 'split'
Edit
It seems to be a error on chrome. It didn't come to me, 'cause I was using Opera, but opera uses chromium as well. In internet explorer the error didn't show.
https://code.djangoproject.com/ticket/21227#no1
This is the bug tracker

Python: Why do is the traceback getting printed?

I have a function. The function get started in some more threads. I tried to print a own error message. But its not important what I do I get still the traceback printed. My function:
def getSuggestengineResultForThree(suggestengine, seed, dynamoDBLocation):
results[seed][suggestengine] = getsuggestsforsearchengine(seed, suggestengine)
for keyword_result in results[seed][suggestengine]:
o = 0
while True:
try:
allKeywords.put_item(
Item={
'keyword': keyword_result
}
)
break
except ProvisionedThroughputExceededException as pe:
if (o > 9):
addtoerrortable(keyword_result)
print('ProvisionedThroughputExceededException 10 times in getSuggestengineResultForThree for allKeywords')
break
sleep(1)
o = o + 1
print("ProvisionedThroughputExceededException in getSugestengineResult")
But I get for every Thread an output like this:
Exception in thread Thread-562:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/Users/iTom/ownCloud/Documents/Workspace/PyCharm/Keywords/TesterWithDB.py", line 362, in getSuggestengineResultForThree
'keyword': keyword_result
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/boto3/resources/factory.py", line 518, in do_action
response = action(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/client.py", line 252, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/client.py", line 542, in _make_api_call
raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ProvisionedThroughputExceededException) when calling the PutItem operation: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API
Can someone help me to get my own print instead of the traceback? :)
This answer is a bit late for your question but here it is in case anyone is searching for this answer.
The exception that boto3 throws is a botocore.exceptions.ClientError as Neil has answered. However you should check response error code for 'ProvisionedThroughputExceededException' because the ClientError could be for another issue.
from botocore.exceptions import ClientError
except ClientError as e:
if e.response['Error']['Code'] != 'ProvisionedThroughputExceededException':
raise
# do something else with 'e'
I am using Python 2.7 which may or may not make a difference. The exception that I receive suggests that boto3 is already doing retries (up to 9 times) which is different from your exception:
An error occurred (ProvisionedThroughputExceededException) when calling the PutItem operation (reached max retries: 9): The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
It's possible that ProvisionedThroughputExceededException is not actually the error. Try:
except botocore.exceptions.ClientError as pe:
instead.
If that doesn't work, figure out what line the error is occurring on and put the except statement there.

Python socket ConnectionResetError: [Errno 54] Connection reset by peer vs socket.error: [Errno 104] Connection reset by peer

I'm having trouble debugging my code because I cannot understand the socket error being raised.
Here is the traceback.
Traceback (most recent call last):
File "clickpression.py", line 517, in <module> presser.main()
File "clickpression.py", line 391, in main
File "clickpression.py", line 121, in clickpress self.refresh_proxies(country=country)
File "clickpression.py", line 458, in refresh_proxies self.proxies = self.get_proxies(country=country)
File "helpers.py", line 72, in wrapper return func(*args, **kwargs)
File "clickpression.py", line 264, in get_proxies self.settings.SUPER_PROXY).read().decode('utf-8')
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 161, in urlopen return opener.open(url, data, timeout)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 463, in open response = self._open(req, data)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 481, in _open '_open', req)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 441, in _call_chain result = func(*args)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1210, in http_open return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1185, in do_open r = h.getresponse()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1171, in getresponse response.begin()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351, in begin version, status, reason = self._read_status()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 374, in readinto return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
According to the errno library Errno 54 is errno.EXFULL which in the python 3 documentation is explained as exchange full.
To my understanding the Connection reset by peer is Errno 104 i.e errno.ECONNRESET.
So what does errno.EXFULL mean? and why does socket raise the error with a connection reset by peer description instead of exchange full. And or how are the two errors errno.EXFULL and errno.ECONNRESET related?
PS: I read that the errno 54 might be related to http proxy (I'm using a proxy in my code). If so, how?
According to the errno library Errno 54 is errno.EXFULL
Did you determine that by examining errno.errorcode[54]? Anyway - this errno library might be at fault. You could verify the meaning of an error code on your system by looking into errno.h, e. g. with the help of gcc:
gcc -xc -imacros errno.h -Wp,-P -E <(echo ECONNRESET)
Also, the Python documentation says:
To translate a numeric error code to an error message, use
os.strerror().
It may well be that error number 54 is ECONNRESET on your system, and that os.strerror(54) will attest that.
Now that you have verified that os.strerror(54) returns 'Exchange full', I am puzzled why the error number 54 and the error string Connection reset by peer do not match. If that happens on a system with strace or something similar, I would further check which error is returned by the operating system through use of strace -e network on the affected process.
Regarding your question about EXFULL: Its meaning seems somewhat system dependent; e. g. on Linux, EXFULL is returned from only a handful places in the kernel, the only network-related place being in br_if.c concerning network bridges, when no available bridge port number is found (other places are in USB and SCSI drivers).
I tried to use python to crew coin market on OKEX.com using WebSocket,cause the url is an outer address,i used a vpn service provided by us,but it still can work. here is the code an traceback.
from ws4py.client.threadedclient import WebSocketClient
class DummyClient(WebSocketClient):
def opened(self):
# self.send("{'event': 'addChannel', 'channel': 'ok_sub_futureusd_btc_ticker_this_week'}") #发送请求数据格式
# self.send("www.baidu.com")
self.send("{'event':'addChannel','channel':'ok_sub_spot_bch_btc_ticker'}")
def closed(self, code, reason=None):
print("Closed down", code, reason)
#服务器返回消息
def received_message(self, m):
print("recv:", m)
if __name__ == '__main__':
try:
# 服务器连接地址wss://real.okex.com:10440/websocket/okexapi
# ws = DummyClient('wss://real.okcoin.cn:10440/websocket/okcoinapi', protocols=['chat'])
ws = DummyClient('wss://real.okex.com:10440/websocket/okexapi', protocols=['chat'])
ws.connect()
#ws.send("my test...")
ws.run_forever()
except KeyboardInterrupt:
ws.close()
You can try this code to your project:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
if it not work,make sure the server open TLSv1 support.

Categories

Resources