Develop Reference

How to add retry for celery backend connection?

I am using celery 5.0.1 and using CELERY_BACKEND_URL as redis://:password#redisinstance1:6379/0. It works fine, but when there is a Redis instance loose connection, it breaks out tasks with an error.
Exception: Error while reading from socket: (104, 'Connection reset by peer')
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 198, in _read_from_socket
data = recv(self._sock, socket_read_size)
File "/usr/local/lib/python3.7/dist-packages/redis/_compat.py", line 72, in recv
return sock.recv(*args, **kwargs)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 477, in trace_task
uuid, retval, task_request, publish_result,
File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 154, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 439, in store_result
request=request, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 855, in _store_result
current_meta = self._get_task_meta_for(task_id)
File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 873, in _get_task_meta_for
meta = self.get(self.get_key_for_task(task_id))
File "/usr/local/lib/python3.7/dist-packages/celery/backends/redis.py", line 346, in get
return self.client.get(key)
File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 1606, in get
return self.execute_command('GET', name)
File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 739, in read_response
response = self._parser.read_response()
File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 324, in read_response
raw = self._buffer.readline()
File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 256, in readline
self._read_from_socket()
File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 223, in _read_from_socket
(ex.args,))
redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer')
Celery worker: None
Celery task id: 244b56af-7c96-56cf-a01a-9256cfd98ade
Celery retry attempt: 0
Task args: []
Task kwargs: {'address': 'ipadd', 'uid': 'uid', 'hexID': 'hexID', 'taskID': '244b56af-7c96-56cf-a01a-9256cfd98ade'}
When I run the second tasks, it works fine, there is some glitch in the connection for a short period of time.
Can I set something by which, when celery tries to update the results to Redis, if it returns an error, it will retry after 2-5 seconds?
I know how to set retry in the task, but this does not task failure. My tasks work fine and it returns the data, but celery is losing connection while updating to the backend.

To deal with connection timeouts you can have the following in your Celery configuration:
app.conf.broker_transport_options = {
'retry_policy': {
'timeout': 5.0
}
}
app.conf.result_backend_transport_options = {
'retry_policy': {
'timeout': 5.0
}
}
There are few other Redis backend settings that you may want to consider having in your configuration, like the redis_retry_on_timeout for an example.

Error trying to connect Celery through SQS using STS

I'm trying to use Celery with SQS as broker. In order to use the SQS from my container I need to assume a role and for that I'm using STS. My code looks like this:
role_info = {
'RoleArn': 'arn:aws:iam::xxxxxxx:role/my-role-execution',
'RoleSessionName': 'roleExecution'
}
sts_client = boto3.client('sts', region_name='eu-central-1')
credentials = sts_client.assume_role(**role_info)
aws_access_key_id = credentials["Credentials"]['AccessKeyId']
aws_secret_access_key = credentials["Credentials"]['SecretAccessKey']
aws_session_token = credentials["Credentials"]["SessionToken"]
os.environ["AWS_ACCESS_KEY_ID"] = aws_access_key_id
os.environ["AWS_SECRET_ACCESS_KEY"] = aws_secret_access_key
os.environ["AWS_DEFAULT_REGION"] = 'eu-central-1'
os.environ["AWS_SESSION_TOKEN"] = aws_session_token
broker = "sqs://"
backend = 'redis://redis-service:6379/0'
celery = Celery('tasks', broker=broker, backend=backend)
celery.conf["task_default_queue"] = 'my-queue'
celery.conf["broker_transport_options"] = {
'region': 'eu-central-1',
'predefined_queues': {
'my-queue': {
'url': 'https://sqs.eu-central-1.amazonaws.com/xxxxxxx/my-queue'
}
}
}
In the same file I have the following task:
#celery.task(name='my-queue.my_task')
def my_task(content) -> int:
print("hello")
return 0
When I execute the following code I get an error:
[2020-09-24 10:38:03,602: CRITICAL/MainProcess] Unrecoverable error: ClientError('An error occurred (AccessDenied) when calling the ListQueues operation: Access to the resource https://eu-central-1.queue.amazonaws.com/ is denied.',)
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 921, in create_channel
return self._avail_channels.pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 208, in start
self.blueprint.start(self)
File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/connection.py", line 23, in start
c.connection = c.connect()
File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 405, in connect
conn = self.connection_for_read(heartbeat=self.amqheartbeat)
File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 412, in connection_for_read
self.app.connection_for_read(heartbeat=heartbeat))
File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 439, in ensure_connected
callback=maybe_shutdown,
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 422, in ensure_connection
callback, timeout=timeout)
File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 341, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 275, in connect
return self.connection
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 823, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 778, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 941, in establish_connection
self._avail_channels.append(self.create_channel(self))
File "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 923, in create_channel
channel = self.Channel(connection)
File "/usr/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 100, in __init__
self._update_queue_cache(self.queue_name_prefix)
File "/usr/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 105, in _update_queue_cache
resp = self.sqs.list_queues(QueueNamePrefix=queue_name_prefix)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 337, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 656, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListQueues operation: Access to the resource https://eu-central-1.queue.amazonaws.com/ is denied.
If I use boto3 directly without Celery, I'm able to connect to the queue and retrieve data without this error. I don't know why Celery/Kombu try to list queues when I specify the predefined_queues configuration, tha is used to avoid these behavior (from docs):
If you want Celery to use a set of predefined queues in AWS, and to never attempt to list SQS queues, nor attempt to create or delete them, pass a map of queue names to URLs using the predefined_queue_urls setting
Source here
Anyone know what happens? How I should modify my code in order to make it work?. Seems that Celery is not using the credentials at all.
The versions I'm using:
celery==4.4.7
boto3==1.14.54
kombu==4.5.0
Thanks!
PS: I created and issue in Github to track if this can be a library error or not...

I solved the problem updating dependencies to the latest versions:
celery==5.0.0
boto3==1.14.54
kombu==5.0.2
pycurl==7.43.0.6

I was able to get celery==4.4.7 and kombu==4.6.11 working by setting the following configuration option:
celery.conf["task_create_missing_queues"] = False

Changing Permissions on Python Modules so I Don't Need to "sudo" My Python Script Calls

Is there a way to reconfigure all my Python3 modules from which I call certain utilities (i.e., urlopen) so that I no longer need to preface my Python3 script calls with "sudo", without having to rebuild my Ubuntu VM?
Example, with my script code as follows:
import socks
import socket
from urllib.request import urlopen
from time import sleep
from bs4 import BeautifulSoup
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
url_name1 = "http://www.google.com"
print("url name is : " + url_name1)
print("About to open the web page")
sleep(5)
webpage = urlopen(url_name1)
print("Web page opened successfully")
sleep(5)
html = webpage.read().decode("utf-8")
soup = BeautifulSoup(html, "html.parser")
print("HTML extracted")
sleep(5)
Without prefacing my command with "sudo", the output looks like this:
$ python3 sample_script2.py
url name is : http://www.google.com
About to open the web page
1599238298 WARNING torsocks[29740]: [connect] Connection to a local address are denied since it might be a TCP DNS query to a local DNS server. Rejecting it for safety reasons. (in tsocks_connect() at connect.c:193)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/socks.py", line 832, in connect
super(socksocket, self).connect(proxy_addr)
PermissionError: [Errno 1] Operation not permitted
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/urllib/request.py", line 1326, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/lib/python3.8/http/client.py", line 1240, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1286, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1235, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1006, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 946, in send
self.connect()
File "/usr/lib/python3.8/http/client.py", line 917, in connect
self.sock = self._create_connection(
File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
File "/usr/lib/python3/dist-packages/socks.py", line 100, in wrapper
return function(*args, **kwargs)
File "/usr/lib/python3/dist-packages/socks.py", line 844, in connect
raise ProxyConnectionError(msg, error)
socks.ProxyConnectionError: Error connecting to SOCKS5 proxy 127.0.0.1:9050: [Errno 1] Operation not permitted
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "sample_script2.py", line 14, in <module>
webpage = urlopen(url_name1)
File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 1355, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/lib/python3.8/urllib/request.py", line 1329, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error Error connecting to SOCKS5 proxy 127.0.0.1:9050: [Errno 1] Operation not permitted>
$
Adding "sudo" to the command yields the following:
jbottiger#ubuntu:~/DarkWeb$ sudo python3 sample_script2.py
[sudo] password for jbottiger:
url name is : http://www.google.com
About to open the web page
Web page opened successfully
HTML extracted
Printing soup object text
Google(function(){window.google={kEI:'uHBSX4DxFqWd5wKA1KSAAw',kEXPI:'0,202162,1151585,5662,730,224,5105,206,3204,10,1226,364,1499,612,91,114,383,246,5,1354,648,3451,315,3,66,308,676,90,41,153,864,117,44,407,415,205,138,511,258,1119056,1197771,329496,13677,4855,32691,15248,861,28690,9188,8384,1326,3532,1362,9290,3028,4735,5,2649,8384,1808,4998,7933,5295,2054,920,873,4192,6430,7432,7095,4517,2778,919,2277,8,2796,1593,1279,2212,532,147,1103,842,515,1139,1,278,104,4258,312,1137,2,2063,606,2023,1733,43,521,1947,2229,93,328,1284,16,2927,2247,1819,1780,3227,2845,7,2903,2696,469,6286,4455,641,602,1847,3685,1742,4929,108,1456,1951,908,2,941,715,1899,2397,2650,4820,1704,473,1098,3,346,230,1835,4,4620,149,189,3313,743,1745,2220,32,4072,1661,4,498,1030,2304,1236,271,874,405,1860,2393,1791,52,2377,464,459,1201,354,4067,153,882,1316,3,610,1498,1172,1426,69,644,1,1388,386,196,2811,935,818,690,1542,1639,533,2,425,862,1019,189,56,264,198,25,887,564,464,217,8,431,30,130,340,832,2287,181,223,1314,23,1102,655,990,52,535,1239,1257,254,1209,35,591,379,850,437,2,16,6,86,197,22,689,6,632,146,411,108,1,958,360,115,2,93,200,1189,157,1938,792,80,4,26,500,37,891,820,765,286,63,299,60,696,86,1,353,290,52,56,3,403,11,89,685,78,1,217,513,92,383,617,363,1393,5765060,8800593,1323,549,333,444,1,2,80,1,900,896,1,9,2,2551,1,748,141,795,10,553,1,4265,1,1,2,1017,9,305,3299,248,283,527,32,1,10,2,3,1,6,1,14,9,1,2,2,4,4,12,6,10,8,2,35,12,2,1,23959867,53,2704777',kBL:'QdLX'};google.sn='webhp';google.kHL='ru';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var c;a&&(!a.getAttribute||!(c=a.getAttribute("eid")));)a=a.parentNode;return c||google.kEI};google.getLEI=function(a){for(var c=null;a&&(!a.getAttribute||!(c=a.getAttribute("leid")));)a=a.parentNode;return c};google.ml=function(){return null};google.time=function(){return Date.now()};google.log=function(a,c,b,d,g){if(b=google.logUrl(a,c,b,d,g)){a=new Image;var e=google.lc,f=google.li;e[f]=a;a.onerror=a.onload=a.onabort=function(){delete e[f]};google.vel&&google.vel.lu&&google.vel.lu(b);a.src=b;google.li=f+1}};google.logUrl=function(a,c,b,d,g){var e="",f=google.ls||"";b||-1!=c.search("&ei=")||(e="&ei="+google.getEI(d),-1==c.search("&lei=")&&(d=google.getLEI(d))&&(e+="&lei="+d));d="";!b&&google.cshid&&-1==c.search("&cshid=")&&"slh"!=a&&(d="&cshid="+google.cshid);b=b||"/"+(g||"gen_204")+"?atyp=i&ct="+a+"&cad="+c+e+f+"&zx="+google.time()+d;/^http:/i.test(b)&&"https:"==window.location.protocol&&(google.ml(Error("a"),!1,{src:b,glmm:1}),b="");return b};}).call(this);(function(){google.y={};google.x=function(a,b){if(a)var c=a.id;else{do c=Math.random();while(google.y[c])}google.y[c]=[a,b];return!1};google.lm=[];google.plm=function(a){google.lm.push.apply(google.lm,a)};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};}).call(this);google.f={};(function(){
document.documentElement.addEventListener("submit",function(b){var a;if(a=b.target){var c=a.getAttribute("data-submitfalse");a="1"==c||"q"==c&&!a.elements.q.value?!0:!1}else a=!1;a&&(b.preventDefault(),b.stopPropagation())},!0);document.documentElement.addEventListener("click",function(b){var a;a:{for(a=b.target;a&&a!=document.documentElement;a=a.parentElement)if("A"==a.tagName){a="1"==a.getAttribute("data-nohref");break a}a=!1}a&&b.preventDefault()},!0);}).call(this);
var a=window.location,b=a.href.indexOf("#");if(0<=b){var c=a.href.substring(b+1);/(^|&)q=/.test(c)&&-1==c.indexOf("#")&&a.replace("/search?"+c.replace(/(^|&)fp=[^&]*/g,"")+"&cad=h")};#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}#media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}
body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-left:4px}input{font-family:inherit}body{background:#fff;color:#000}a{color:#11c;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;border-color:#ccc #999 #999 #ccc;height:30px}.lsbb{display:block}#fll a{display:inline-block;margin:0 12px}.lsb{background:url(/images/nav_logo229.png) 0 -261px repeat-x;border:none;color:#000;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#ccc}.lst:focus{outline:none}.tiah{width:458px}(function(){var src='/images/nav_logo229.png';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}
if (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}
}
})();Поиск Картинки Карты Play YouTube Новости Почта Диск Ещё »История веб-поиска | Настройки | Войти (function(){var id='tsuid1';document.getElementById(id).onclick = function(){var s = document.createElement('script');s.src = this.getAttribute('data-script-url');(document.getElementById('xjsc')||document.body).appendChild(s);};})();(function(){var id='tsuid2';document.getElementById(id).onclick = function(){if (this.form.q.value){this.checked = 1;if (this.form.iflsig)this.form.iflsig.disabled = false;}
else top.location='/doodles/';};})();Расширенный поиск(function(){var a,b="1";if(document&&document.getElementById)if("undefined"!=typeof XMLHttpRequest)b="2";else if("undefined"!=typeof ActiveXObject){var c,d,e=["MSXML2.XMLHTTP.6.0","MSXML2.XMLHTTP.3.0","MSXML2.XMLHTTP","Microsoft.XMLHTTP"];for(c=0;d=e[c++];)try{new ActiveXObject(d),b="2"}catch(h){}}a=b;if("2"==a&&-1==location.search.indexOf("&gbv=2")){var f=google.gbvu,g=document.getElementById("gbv");g&&(g.value=a);f&&window.setTimeout(function(){location.href=f},0)};}).call(this);Рекламные программыРешения для бизнесаВсё о GoogleGoogle.ru© 2020 - Конфиденциальность - Условия(function(){window.google.cdo={height:0,width:0};(function(){var a=window.innerWidth,b=window.innerHeight;if(!a||!b){var c=window.document,d="CSS1Compat"==c.compatMode?c.documentElement:c.body;a=d.clientWidth;b=d.clientHeight}a&&b&&(a!=google.cdo.width||b!=google.cdo.height)&&google.log("","","/client_204?&atyp=i&biw="+a+"&bih="+b+"&ei="+google.kEI);}).call(this);})();(function(){var u='/xjs/_/js/k\x3dxjs.hp.en.6FZeP6lo3MI.O/m\x3dsb_he,d/am\x3dAJ5gcw/d\x3d1/rs\x3dACT90oG6N5VH73PFnXBwBd2MrAZnJY6t4Q';
setTimeout(function(){var b=document;var a="SCRIPT";"application/xhtml+xml"===b.contentType&&(a=a.toLowerCase());a=b.createElement(a);a.src=u;google.timers&&google.timers.load&&google.tick&&google.tick("load","xjsls");document.body.appendChild(a)},0);})();(function(){window.google.xjsu='/xjs/_/js/k\x3dxjs.hp.en.6FZeP6lo3MI.O/m\x3dsb_he,d/am\x3dAJ5gcw/d\x3d1/rs\x3dACT90oG6N5VH73PFnXBwBd2MrAZnJY6t4Q';})();function _DumpException(e){throw e;}
function _F_installCss(c){}
(function(){google.jl={dw:false,em:[],emw:false,lls:'default',pdt:0,snet:true,uwp:true};})();(function(){var pmc='{\x22d\x22:{},\x22sb_he\x22:{\x22agen\x22:true,\x22cgen\x22:true,\x22client\x22:\x22heirloom-hp\x22,\x22dh\x22:true,\x22dhqt\x22:true,\x22ds\x22:\x22\x22,\x22ffql\x22:\x22ru\x22,\x22fl\x22:true,\x22host\x22:\x22google.com\x22,\x22isbh\x22:28,\x22jsonp\x22:true,\x22msgs\x22:{\x22cibl\x22:\x22Удалить поисковый запрос\x22,\x22dym\x22:\x22Возможно, вы имели в виду:\x22,\x22lcky\x22:\x22Мне повезёт!\x22,\x22lml\x22:\x22Подробнее...\x22,\x22oskt\x22:\x22Экранная клавиатура\x22,\x22psrc\x22:\x22Этот запрос был удален из вашей \\u003Ca href\x3d\\\x22/history\\\x22\\u003Eистории веб-поиска\\u003C/a\\u003E\x22,\x22psrl\x22:\x22Удалить\x22,\x22sbit\x22:\x22Поиск по картинке\x22,\x22srch\x22:\x22Поиск в Google\x22},\x22ovr\x22:{},\x22pq\x22:\x22\x22,\x22refpd\x22:true,\x22rfs\x22:[],\x22sbpl\x22:16,\x22sbpr\x22:16,\x22scd\x22:10,\x22stok\x22:\x22WKTHIsN6ufJvVLrcm5Yf_IkFoE0\x22,\x22uhde\x22:false}}';google.pmc=JSON.parse(pmc);})();
jbottiger#ubuntu:~/DarkWeb$

I posed this question to my professor who recommended that I preface my python3 command with "torsocks" after enabling torsocks on my Ubuntu VM (must have torsocks installed and configured prior to running the script).
After that, remove the following two statements from the script.
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
Now when I enter: "torsocks python3 <script_name>.py", I do not receive these errors anymore, including when trying to open a dark-web page.
According to my professor, Dr. Terrence O'Connor, PhD (Florida Institute of Technology), both my original approach of specifying a proxy (i.e., tor) in my script and using torsocks to tunnel traffic of a specific command (i.e., "python3" in my case) are viable methods of connecting to the ToR network via the proxy service om my Ubuntu VM. It appears as if the second method recommended by Dr. O'Connor worked better than the first one.

Youtube Data API V3 returning ssl.SSLWantWriteError: The operation did not complete (write) error

I asked almost the same question earlier, but failed to get a solution to it due to a new error that appeared while trying to solve that, which overshadowed the initial one. The problem is that when I try to upload a youtube video using the code below, I get the error in the title (full trace below) and am unable to do so. I am clueless as to what is causing the error and would appreciate any and all help.
def upload(beatName, mainArtist, keywords, oneLiner):
CLIENT_SECRET_FILE = 'client_secret.json'
API_NAME = 'youtube'
API_VERSION = 'v3'
SCOPES = ['https://www.googleapis.com/auth/youtube']
service = Create_Service(CLIENT_SECRET_FILE, API_NAME, API_VERSION, SCOPES)
upload_date_time = DT.datetime(2020, 9, 3, 12, 30, 0).isoformat() + '.000Z'
title = "OTOROSHI - "test"
description = 'Test'
request_body = {
'snippet': {
'categoryI': 10,
'title': title,
'description': description,
'tags': keywords
},
'status': {
'privacyStatus': 'private',
'publishAt': upload_date_time,
'selfDeclaredMadeForKids': False,
},
'notifySubscribers': True
}
mediaFile = MediaFileUpload('E:\Everything.mp4')
response_upload = service.videos().insert(
part = 'snippet,status',
body = request_body,
media_body = mediaFile
).execute()
client_secret.json-youtube-v3-(['https://www.googleapis.com/auth/youtube'],)
['https://www.googleapis.com/auth/youtube']
youtube service created successfully
[INFO ] [Base ] Leaving application in progress...
Traceback (most recent call last):
File "C:/Users/ricsi/Desktop/Automatic Uploader/main.py", line 34, in <module>
AutomaticUploader().run()
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\app.py", line 950, in run
runTouchApp()
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\base.py", line 573, in runTouchApp
EventLoop.mainloop()
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\base.py", line 347, in mainloop
self.idle()
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\base.py", line 391, in idle
self.dispatch_input()
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\base.py", line 342, in dispatch_input
post_dispatch_input(*pop(0))
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\base.py", line 248, in post_dispatch_input
listener.dispatch('on_motion', etype, me)
File "kivy\_event.pyx", line 709, in kivy._event.EventDispatcher.dispatch
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\core\window\__init__.py", line 1412, in on_motion
self.dispatch('on_touch_down', me)
File "kivy\_event.pyx", line 709, in kivy._event.EventDispatcher.dispatch
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\core\window\__init__.py", line 1428, in on_touch_down
if w.dispatch('on_touch_down', touch):
File "kivy\_event.pyx", line 709, in kivy._event.EventDispatcher.dispatch
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\uix\widget.py", line 545, in on_touch_down
if child.dispatch('on_touch_down', touch):
File "kivy\_event.pyx", line 709, in kivy._event.EventDispatcher.dispatch
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\uix\widget.py", line 545, in on_touch_down
if child.dispatch('on_touch_down', touch):
File "kivy\_event.pyx", line 709, in kivy._event.EventDispatcher.dispatch
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\uix\behaviors\button.py", line 151, in on_touch_down
self.dispatch('on_press')
File "kivy\_event.pyx", line 705, in kivy._event.EventDispatcher.dispatch
File "kivy\_event.pyx", line 1248, in kivy._event.EventObservers.dispatch
File "kivy\_event.pyx", line 1132, in kivy._event.EventObservers._dispatch
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\kivy\lang\builder.py", line 57, in custom_callback
exec(__kvlang__.co_value, idmap)
File "C:\Users\ricsi\Desktop\Automatic Uploader\automaticuploader.kv", line 53, in <module>
on_press: root.upload()
File "C:/Users/ricsi/Desktop/Automatic Uploader/main.py", line 27, in upload
upload(beatName, mainArtist, keywords, oneLiner)
File "C:\Users\ricsi\Desktop\Automatic Uploader\youtube.py", line 228, in upload
response_upload = service.videos().insert(
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\googleapiclient\_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\googleapiclient\http.py", line 892, in execute
resp, content = _retry_request(
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\googleapiclient\http.py", line 204, in _retry_request
raise exception
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\googleapiclient\http.py", line 177, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\google_auth_httplib2.py", line 200, in request
response, content = self.http.request(
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\httplib2\__init__.py", line 1985, in request
(response, content) = self._request(
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\httplib2\__init__.py", line 1650, in _request
(response, content) = self._conn_request(
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\site-packages\httplib2\__init__.py", line 1558, in _conn_request
conn.request(method, request_uri, body, headers)
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\http\client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\http\client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\http\client.py", line 1049, in _send_output
self.send(chunk)
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\http\client.py", line 971, in send
self.sock.sendall(data)
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Users\ricsi\Anaconda3\envs\Automatic Uploader\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
ssl.SSLWantWriteError: The operation did not complete (write) (_ssl.c:2472)
Process finished with exit code 1

Synopsis
The TLS/SSL errors are due to issues that may occur at one of the following levels:
The level of Client Application;
The level of Google's API Client Library and of its corresponding requisites (see them listed below);
The level of the Standard Python's SSL implementation (TLS/SSL wrapper for socket objects);
The level of the Operating System itself.
Addressing the OP's issues above, my recommendation is as follows:
Tackling SSL errors at the level of the client application, if experiencing the kind of errors seen above, instead of uploading video files in one go, do use the Resumable Upload Protocol.
The employment of YouTube Data API's resumable uploading feature can be done quite easily, for both Python v2 and v3, as to be seen below.
Do note that using this feature will make the app be more resilient to network-related errors.
Details
Before attempting to use the Python script quoted below, one should make sure that all of the libraries (i.e. Python packages) listed below are up-to-date. (Just issue pip install --upgrade on each of the named packages.)
Google API Client Library for Python
google-api-python-client
Google Authentication Library for Python: oauthlib integration
google-auth-oauthlib
Google Authentication Library for Python: httplib2 transport
google-auth-httplib2
Google Authentication Library for Python
google-auth
Prior to modifying your app, do test that the resumable uploading feature of the API (that is the Resumable Uploading Protocol) does indeed work without error (thus without SSL errors too).
First thing to do is to download the time-tested public Google script upload_video.py. (This script has an official documentation too. To obtain a self-explanatory usage information page from the script just issue it with the command line option --help.)
If you're running Python v3, then you'll have to patch upload_video.py as to be seen further below. In any case, for testing the resumable upload feature, simply issue at a command line prompt a command like:
python upload_video.py --file your_video_file.mp4 --title '...' --description '...' --privacyStatus private
or, by case:
python3 upload_video.py --file your_video_file.mp4 --title '...' --description '...' --privacyStatus private
Note that your client secrets file -- client_secret.json -- has to be present in the directory that hosts upload_video.py.
Upon running successfully upload_video.py, do check the newly uploaded video on (and eventually remove it from) your own YouTube Studio page at:
https://studio.youtube.com/channel/YOUR_CHANNEL_ID/videos/upload.
If everything worked OK -- no SSL errors --, then you may proceed refactoring your own code such that it to include the Python code in upload_video.py according to your needs.
Patching upload_video.py
As currently provided by Google, the script upload_video.py is coded in the Python v2 language; for to make it work with Python v3, do apply to it the following changes:
First line of this script: replace python with python3 (this is not really necessary on a Windows machine);
Replace import httplib with import http.client;
Replace all occurrences of httplib. with http.client. (do note the dots there);
Replace all print EXPRESSION with print(EXPRESSION);
Replace all except IDENTIFIER, e with except IDENTIFIER as e.
Or, otherwise, do apply the following .patch file to the script file:
--- upload_video.py
+++ upload_video.py
## -1,7 +1,7 ##
-#!/usr/bin/python
+#!/usr/bin/python3
import argparse
-import httplib
+import http.client
import httplib2
import os
import random
## -23,10 +23,10 ##
MAX_RETRIES = 10
# Always retry when these exceptions are raised.
-RETRIABLE_EXCEPTIONS = (httplib2.HttpLib2Error, IOError, httplib.NotConnected,
- httplib.IncompleteRead, httplib.ImproperConnectionState,
- httplib.CannotSendRequest, httplib.CannotSendHeader,
- httplib.ResponseNotReady, httplib.BadStatusLine)
+RETRIABLE_EXCEPTIONS = (httplib2.HttpLib2Error, IOError, http.client.NotConnected,
+ http.client.IncompleteRead, http.client.ImproperConnectionState,
+ http.client.CannotSendRequest, http.client.CannotSendHeader,
+ http.client.ResponseNotReady, http.client.BadStatusLine)
# Always retry when an apiclient.errors.HttpError with one of these status
# codes is raised.
## -104,31 +104,31 ##
retry = 0
while response is None:
try:
- print 'Uploading file...'
+ print('Uploading file...')
status, response = request.next_chunk()
if response is not None:
if 'id' in response:
- print 'Video id "%s" was successfully uploaded.' % response['id']
+ print('Video id "%s" was successfully uploaded.' % response['id'])
else:
exit('The upload failed with an unexpected response: %s' % response)
- except HttpError, e:
+ except HttpError as e:
if e.resp.status in RETRIABLE_STATUS_CODES:
error = 'A retriable HTTP error %d occurred:\n%s' % (e.resp.status,
e.content)
else:
raise
- except RETRIABLE_EXCEPTIONS, e:
+ except RETRIABLE_EXCEPTIONS as e:
error = 'A retriable error occurred: %s' % e
if error is not None:
- print error
+ print(error)
retry += 1
if retry > MAX_RETRIES:
exit('No longer attempting to retry.')
max_sleep = 2 ** retry
sleep_seconds = random.random() * max_sleep
- print 'Sleeping %f seconds and then retrying...' % sleep_seconds
+ print('Sleeping %f seconds and then retrying...' % sleep_seconds)
time.sleep(sleep_seconds)
if __name__ == '__main__':
## -150,5 +150,5 ##
try:
initialize_upload(youtube, args)
- except HttpError, e:
- print 'An HTTP error %d occurred:\n%s' % (e.resp.status, e.content)
+ except HttpError as e:
+ print('An HTTP error %d occurred:\n%s' % (e.resp.status, e.content))
The .patch file above could well be generated by Python's own tool 2to3. This tool is also able to modify upload_video.py in place. (For having it installed simply issue pip install 2to3.)

Python: pip spitting out long errors when trying to install packages [duplicate]

coincidentally, I run pip search django command and I got time out error. even specifing a high value of timeout
Below the logs:
D:\PERFILES\rmaceissoft\virtualenvs\fancy_budget\Scripts>pip search django --timeout=300
Exception:
Traceback (most recent call last):
File "D:\PERFILES\Marquez\rmaceissoft\Workspace\virtualenvs\fancy_budget\lib\s
ite-packages\pip-1.1-py2.7.egg\pip\basecommand.py", line 104, in main
status = self.run(options, args)
File "D:\PERFILES\Marquez\rmaceissoft\Workspace\virtualenvs\fancy_budget\lib\s
ite-packages\pip-1.1-py2.7.egg\pip\commands\search.py", line 34, in run
pypi_hits = self.search(query, index_url)
File "D:\PERFILES\Marquez\rmaceissoft\Workspace\virtualenvs\fancy_budget\lib\s
ite-packages\pip-1.1-py2.7.egg\pip\commands\search.py", line 48, in search
hits = pypi.search({'name': query, 'summary': query}, 'or')
File "C:\Python27\Lib\xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "C:\Python27\Lib\xmlrpclib.py", line 1575, in __request
verbose=self.__verbose
File "C:\Python27\Lib\xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "C:\Python27\Lib\xmlrpclib.py", line 1297, in single_request
return self.parse_response(response)
File "C:\Python27\Lib\xmlrpclib.py", line 1462, in parse_response
data = stream.read(1024)
File "C:\Python27\Lib\httplib.py", line 541, in read
return self._read_chunked(amt)
File "C:\Python27\Lib\httplib.py", line 574, in _read_chunked
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\Lib\socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
Storing complete log in C:\Users\reiner\AppData\Roaming\pip\pip.log
however, another search command finish without problems:
pip search django-registration
Is that a bug of pip due to the big amount of packages name that contains "django"?
Note: speed internet connection = 2 Mbits

the --timeout option doesn't seem to work properly.
I can install django properly by using either:
pip --default-timeout=60 install django
or
export PIP_DEFAULT_TIMEOUT=60
pip install django
Note: using pip version 1.2.1 on RHEL 6.3
Source: DjangoDay2012-Brescia.pdf, page 11

The pypi is probably overloaded. Just enable mirror fallback and caching in pip. Maybe tune the timeout a bit. Add these in ~/.pip/pip.conf:
[global]
default-timeout = 60
download-cache = ~/.pip/cache
[install]
use-mirrors = true

There is too short default timeout set for pip by default. You should really set this environment variable PIP_DEFAULT_TIMEOUT to at least 60 (1 minute)
Source: http://www.pip-installer.org/en/latest/configuration.html