I'm trying to figure out how to send email asynchroniously so User don't have to wait until email is sent.
I've installer celery and django-celery-email and added django-celery-email into my settings.py - djcelery_email.
I've also changed database backend:
EMAIL_BACKEND = 'djcelery_email.backends.CeleryEmailBackend'
Now I've tried to do an action which sends email but it freezes and no email has been sent.
Do you know what to do?
Here is a traceback from cmd:
Internal Server Error: /ajax/reservation/confirm/
Traceback (most recent call last):
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\django\core\handlers\base.py", line 149, in get_response
response = self.process_exception_by_middleware(e, request)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\django\core\handlers\base.py", line 147, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\django\views\decorators\csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\dolava_app\views.py", line 68, in reservation_confirm
reservation.save()
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\dolava_app\models.py", line 150, in save
notifications.AdminNotifications.new_pair(self, paired_reservation)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\dolava_app\notifications.py", line 79, in new_pair
send_message_to_admin(subject,message)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\dolava_app\notifications.py", line 8, in send_message_to_admin
mail.send()
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\django\core\mail\message.py", line 292, in send
return self.get_connection(fail_silently).send_messages([self])
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\djcelery_email\backends.py", line 17, in send_messages
result_tasks.append(send_emails.delay(chunk, self.init_kwargs))
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\celery\app\task.py", line 453, in delay
return self.apply_async(args, kwargs)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\celery\app\task.py", line 565, in apply_async
**dict(self._get_exec_options(), **options)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\celery\app\base.py", line 354, in send_task
reply_to=reply_to or self.oid, **options
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\celery\app\amqp.py", line 310, in publish_task
**kwargs
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\kombu\messaging.py", line 172, in publish
routing_key, mandatory, immediate, exchange, declare)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\kombu\connection.py", line 457, in _ensured
interval_max)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\kombu\connection.py", line 369, in ensure_connection
interval_start, interval_step, interval_max, callback)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\kombu\utils\__init__.py", line 246, in retry_over_time
return fun(*args, **kwargs)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\kombu\connection.py", line 237, in connect
return self.connection
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\kombu\connection.py", line 742, in connection
self._connection = self._establish_connection()
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\kombu\connection.py", line 697, in _establish_connection
conn = self.transport.establish_connection()
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\kombu\transport\pyamqp.py", line 116, in establish_connection
conn = self.Connection(**opts)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\amqp\connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\amqp\connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\amqp\transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "C:\Users\Milano\PycharmProjects\FutileStudio\dolava\venv\lib\site-packages\amqp\transport.py", line 95, in __init__
raise socket.error(last_err)
error: [Errno 10061] No connection could be made because the target machine actively refused it
Problem
From https://github.com/pmclanahan/django-celery-email
By default django-celery-email will use Django's builtin SMTP email
backend for the actual sending of the mail.
This means you need to run an SMTP server locally to accept the email. That is why you are getting a socket error - cause you don't have any SMTP server set to receive the email.
Possible Solutions - Local Development
A. Start an SMTP server and use your own client
Option 1
Run a combined SMTP server and client.
Mailcatcher is great for this.
It even comes with django setup instructions!
Set it up then fire it up with:
$ mailcatcher -fv
Note: mailcatcher requires gem (ruby package manager) to install.
So you will have to install that first.
Its probably worth the effort, as mailcatcher works quite nicely and
gives you html and text representations of the email in a browser tab.
For docker users you can get a mailcatcher docker image
Option 2
Run your own SMTP server and a separate mail client.
e.g. Use a simple python SMTP server
From the command line
python -m smtpd -n -c DebuggingServer localhost:1025
in your django settings:
EMAIL_HOST = 'localhost'
EMAIL_HOST_USER = None
EMAIL_HOST_PASSWORD = None
EMAIL_PORT = 1025
EMAIL_USE_TLS = False
Then you will need to setup a client to get those emails.
B. Don't use SMTP as a backend in your settings
Instead print to the console:
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
Problem with this is ugly formatting.
HTML will appear as text which makes the email very hard to read. You can output this to a file by manually copying and pasting it into a .html file and opening it in a browser, but that will get boring very fast.
Solution - Continuous Integration Server
Use this for making email accessible to the django test client, so you can check that emails got sent, check the content of email programatically etc.
EMAIL_BACKEND = 'django.core.mail.backends.locmem.EmailBackend'
Solution - Production Server
You will need to use a SMTP server again.
Either you can run your own or use Sendgrid, Mailgun, Mandrill or something similar.
Related
I tried to get all the active/scheduled/reserved tasks in redis:
from celery.task.control import inspect
inspect_obj = inspect()
inspect_obj.active()
inspect_obj.scheduled()
inspect_obj.reserved()
But was greeted with a list of errors as follows:
My virtual environment ==> HubblerAPI.
Iam using this from the ec2 console
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/celery/app/control.py", line 81, in active
return self._request('dump_active', safe=safe)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/celery/app/control.py", line 71, in _request
timeout=self.timeout, reply=True,
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/celery/app/control.py", line 316, in broadcast
limit, callback, channel=channel,
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/pidbox.py", line 283, in _broadcast
chan = channel or self.connection.default_channel
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/connection.py", line 771, in default_channel
self.connection
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/connection.py", line 756, in connection
self._connection = self._establish_connection()
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/connection.py", line 711, in _establish_connection
conn = self.transport.establish_connection()
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/transport/pyamqp.py", line 116, in establish_connection
conn = self.Connection(**opts)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
**OSError: [Errno 111] Connection refused**
My celery config file is as follows:
BROKER_TRANSPORT = 'redis'
BROKER_TRANSPORT_OPTIONS = {
'queue_name_prefix': 'dev-',
'wait_time_seconds': 10,
'polling_interval': 30,
# The polling interval decides the number of seconds to sleep
between unsuccessful polls
'visibility_timeout': 3600 * 5,
# If a task is not acknowledged within the visibility_timeout, the
task will be redelivered to another worker and executed.
}
CELERY_MESSAGES_DB = 6
BROKER_URL = "redis://%s:%s/%s" % (AWS_REDIS_ENDPOINT, AWS_REDIS_PORT,
CELERY_MESSAGES_DB)
What am i doing wrong here as the error log suggests that its not using the redis broker.
Looks like your python code doesn't recognize your configs since it is attempting to use RabbitMQ's ampq protocol instead of the configured broker.
I suggest the following
https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html
Your configs look similar to Django configs for Celery yet it doesn't seem you are using Celery with Django.
https://docs.celeryq.dev/en/latest/django/first-steps-with-django.html
The issue is using "BROKER_URL" instead of "CELERY_BROKER_URL" in settings.py. Celery wasn't finding the URL and was defaulting to the rabbitmq port instead of the redis port.
I have a RabbitMQ 3.6.1 server on Ubuntu 14.04 running properly. I tried to configure an SSL listener according to official documentation. No problems during the startup.
However when trying to establish a connection, I get the following error on Python/pika side (full transcript below):
pika.exceptions.AuthenticationError: EXTERNAL
What does EXTERNAL mean here? How to debug / get further details of the error?
Course of actions (to test I used a Vagrant box and a local connection):
RabbitMQ starts SSL Listener on port 5671 (per /var/log/rabbitmq/rabbit#rabbitmq-server.log):
started SSL Listener on [::]:5671
I execute the pika.BlockingConnection on the client side.
On the server side I can see an incoming connection:
=INFO REPORT==== 17-Apr-2016::17:07:15 ===
accepting AMQP connection <0.2788.0> (127.0.0.1:48404 -> 127.0.0.1:5671)
Client fails with:
pika.exceptions.AuthenticationError: EXTERNAL
Server timeouts:
=ERROR REPORT==== 17-Apr-2016::17:07:25 ===
closing AMQP connection <0.2788.0> (127.0.0.1:48404 -> 127.0.0.1:5671):
{handshake_timeout,frame_header}
Full transcript of the client side:
>>> import pika, ssl
>>> from pika.credentials import ExternalCredentials
>>> ssl_options = ({"ca_certs": "/etc/rabbitmq/certs/testca/cacert.pem",
... "certfile": "/etc/rabbitmq/certs/client/cert.pem",
... "keyfile": "/etc/rabbitmq/certs/client/key.pem",
... "cert_reqs": ssl.CERT_REQUIRED,
... "server_side": False})
>>> host = "localhost"
>>> connection = pika.BlockingConnection(
... pika.ConnectionParameters(
... host, 5671, credentials=ExternalCredentials(),
... ssl=True, ssl_options=ssl_options))
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 339, in __init__
self._process_io_for_connection_setup()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 374, in _process_io_for_connection_setup
self._open_error_result.is_ready)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 410, in _flush_output
self._impl.ioloop.poll()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/select_connection.py", line 602, in poll
self._process_fd_events(fd_event_map, write_only)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events
handler(fileno, events, write_only=write_only)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 364, in _handle_events
self._handle_read()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 415, in _handle_read
self._on_data_available(data)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1347, in _on_data_available
self._process_frame(frame_value)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1414, in _process_frame
if self._process_callbacks(frame_value):
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1384, in _process_callbacks
frame_value) # Args
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 60, in wrapper
return function(*tuple(args), **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 92, in wrapper
return function(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 236, in process
callback(*args, **keywords)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1298, in _on_connection_start
self._send_connection_start_ok(*self._get_credentials(method_frame))
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1077, in _get_credentials
raise exceptions.AuthenticationError(self.params.credentials.TYPE)
pika.exceptions.AuthenticationError: EXTERNAL
>>>
The Python / pika code in the question is correct.
The error:
pika.exceptions.AuthenticationError: EXTERNAL
is reported when client certificate authorisation is not enabled on the RabbitMQ server side. The word EXTERNAL in the error refers to the authentication mechanism as described here.
To enable:
rabbitmq-plugins enable rabbitmq_auth_mechanism_ssl
I am running Neo4j 2.2.1 in ubuntu Amazon EC2 instance. When I am trying to connect through python using py2neo-2.0.7, I am getting following error :
py2neo.packages.httpstream.http.SocketError: Operation not permitted
I am able to access the web-interface through http://52.10.**.***:7474/browser/
CODE :-
from py2neo import Graph, watch, Node, Relationship
url_graph_conn = "https://neo4j:password#52.10.**.***:7474/db/data/"
print url_graph_conn
my_conn = Graph(url_graph_conn)
babynames = my_conn.find("BabyName")
for babyname in babynames:
print 2
Error message :-
https://neo4j:password#52.10.**.***:7474/db/data/
Traceback (most recent call last):
File "C:\Users\rharoon002\eclipse_workspace\peace\peace\core\graphconnection.py", line 39, in <module>
for babyname in babynames:
File "C:\Python27\lib\site-packages\py2neo\core.py", line 770, in find
response = self.cypher.post(statement, parameters)
File "C:\Python27\lib\site-packages\py2neo\core.py", line 667, in cypher
metadata = self.resource.metadata
File "C:\Python27\lib\site-packages\py2neo\core.py", line 213, in metadata
self.get()
File "C:\Python27\lib\site-packages\py2neo\core.py", line 258, in get
response = self.__base.get(headers=headers, redirect_limit=redirect_limit, **kwargs)
File "C:\Python27\lib\site-packages\py2neo\packages\httpstream\http.py", line 966, in get
return self.__get_or_head("GET", if_modified_since, headers, redirect_limit, **kwargs)
File "C:\Python27\lib\site-packages\py2neo\packages\httpstream\http.py", line 943, in __get_or_head
return rq.submit(redirect_limit=redirect_limit, **kwargs)
File "C:\Python27\lib\site-packages\py2neo\packages\httpstream\http.py", line 433, in submit
http, rs = submit(self.method, uri, self.body, self.headers)
File "C:\Python27\lib\site-packages\py2neo\packages\httpstream\http.py", line 362, in submit
raise SocketError(code, description, host_port=uri.host_port)
py2neo.packages.httpstream.http.SocketError: Operation not permitted
You are trying to access neo4j via https on the standard port for http (7474):
url_graph_conn = "https://neo4j:password#52.10.**.***:7474/db/data/"
The standard port for a https connection is 7473. Try:
url_graph_conn = "https://neo4j:password#52.10.**.***:7473/db/data/"
And make sure you can access the web interface via https:
https://52.10.**.***:7473/browser/
You can change/see the port settings in your neo4j-server.properties file.
Ugh, I have tried as many ways is I have been able to find online thus far, but to no avail. My primary objective is to send email from a django based web application on a private LAN. I don't care how it happens...smtp or win32com and outlook...anything, as long is it works.
Using django default settings and this code in a django shell:
from django.core.mail import send_mail
send_mail('subject','message','myemail#email.com',['myemail#email.com'])
I get this error:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "C:\Python27_32\lib\site-packages\django\core\mail\__init__.py", line 50, in send_mail
connection=connection).send()
File "C:\Python27_32\lib\site-packages\django\core\mail\message.py", line 274, in send
return self.get_connection(fail_silently).send_messages([self])
File "C:\Python27_32\lib\site-packages\django\core\mail\backends\smtp.py", line 87, in send_messages
new_conn_created = self.open()
File "C:\Python27_32\lib\site-packages\django\core\mail\backends\smtp.py", line 48, in open
local_hostname=DNS_NAME.get_fqdn())
File "C:\Python27_32\lib\smtplib.py", line 250, in __init__
(code, msg) = self.connect(host, port)
File "C:\Python27_32\lib\smtplib.py", line 311, in connect
(code, msg) = self.getreply()
File "C:\Python27_32\lib\smtplib.py", line 359, in getreply
+ str(e))
SMTPServerDisconected: Connection unexpectedly closed: [Errno 10057] A request to send or
receive data was disallowed because the socket is not connected and (when sending on a
datagram socket using a sendto call) no address was supplied
After digging into the code, I inserted a print statement in smtplib showing the address as ('localhost', 25). So I don't know why it says no address was supplied. That's when I discovered django-smtp-ssl.py and after installing added the following to my settings:
EMAIL_USE_TLS = True #also tried 1
EMAIL_BACKEND = 'django_smtp_ssl.SSLEmailBackend'
EMAIL_HOST = 'localhost'
EMAIL_HOST_USER = 'myemail#email.com'
EMAIL_HOST_PASSWORD = 'mypassword'
EMAIL_PORT = 465
Then executing the same code, I now get:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "C:\Python27_32\lib\site-packages\django\core\mail\__init__.py", line 50, in send_mail
connection=connection).send()
File "C:\Python27_32\lib\site-packages\django\core\mail\message.py", line 274, in send
return self.get_connection(fail_silently).send_messages([self])
File "C:\Python27_32\lib\site-packages\django\core\mail\backends\smtp.py", line 87, in send_messages
new_conn_created = self.open()
File "C:\Python27_32\lib\site-packages\django_smtp_ssl.py", line 12, in open
local_hostname=DNS_NAME.getfqdn())
File "C:\Python27_32\lib\smtplib.py", line 777, in __init__
SMTP.__init__(self, host, port, local_hostname, timeout)
File "C:\Python27_32\lib\smtplib.py", line 250, in __init__
(code, msg) = self.connect(host, port)
File "C:\Python27_32\lib\smtplib.py", line 311, in connect
(code, msg) = self.getreply()
File "C:\Python27_32\lib\smtplib.py", line 355, in getreply
line = self.file.readline()
File "C:\Python27_32\lib\smtplib.py", line 186, in readline
chr = self.sslobj.read(1)
File "C:\Python27_32\lib\ssl.py", line 160, in read
return self._sslobj.read(len)
AttributeError: 'NoneType' object has no attribute 'read'
I have additionally tried sending mail via Outlook and win32com in a production environment using Apache. I have received a couple different exceptions trying this, such as: 'Server execution failed and 'Call was rejected by callee'. Although it works in a django shell (of course).
Has anyone seen these any of these errors and found resolution? Thank you for any help!
And it's the workaround, nothing short of shoddy, but it'll get the job done until a better solution presents itself.
First I created a custom command that can be called from manage.py as it is not executed from the Apache server (which further tells me that the system does not recognize Apache as a valid user for automatic email generation, certainly there's a way to tell the system "it's okay").
Following that, I created a task in Windows Task Scheduler to run every 30 min (which will suffice for the objective). Action:
Start a program C:\python27_32\python.exe C:\www\myproject\manage.py send_email
myapp/management/commands/send_email.py
from django.core.management.base import BaseCommand
from myapp.util.gen_email import AutoEmail #custom code to generate email via win32com
from myapp.models import MyModel
class Command(BaseCommand):
def __init__(self, *args, **kwargs):
super(Command, self).__init__(*args, **kwargs)
items = MyModel.objects.filter(status__in=['status_a','status_b'])
if items:
self.send_proc_email()
def send_proc_email(self):
subject = 'There are new items to process'
to = 'someemail#email.com'
message = ''
email = AutoEmail(subject, message, to)
def handle(self, *args, **kwargs):
pass
Far from optimal, I know, but hey if it works...
I am trying to cache MySQL queries in my Cherrypy server.
I could not figure out how to solve the error when I was installing pylibmc, so I decided to use Redis-py.
Here I am trying a very simple example.
import redis
cache = redis.StrictRedis(host='localhost', port=8080, db=0)
...
...
cache.set('0', '1') # I also tested with other string keys, but failed with same error
and it's throwing the following error!
[05/May/2014:13:11:13] HTTP Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
response.body = self.handler()
File "/Library/Python/2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/Library/Python/2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
return self.callable(*self.args, **self.kwargs)
File "server.py", line 92, in submit_data
cache.set(str(idx), '1')#res)
File "/Library/Python/2.7/site-packages/redis/client.py", line 897, in set
return self.execute_command('SET', *pieces)
File "/Library/Python/2.7/site-packages/redis/client.py", line 461, in execute_command
return self.parse_response(connection, command_name, **options)
File "/Library/Python/2.7/site-packages/redis/client.py", line 471, in parse_response
response = connection.read_response()
File "/Library/Python/2.7/site-packages/redis/connection.py", line 339, in read_response
response = self._parser.read_response()
File "/Library/Python/2.7/site-packages/redis/connection.py", line 118, in read_response
(str(byte), str(response)))
InvalidResponse: Protocol Error: H, TTP/1.1 400 Bad Request
I could not figure out what was wrong, and my website runs without a problem on localhost at port 8080 when I am not using Redis.
Your webserver is running on port 8080, not your Redis server. Your Redis server is most likely running on port 6379, unless you changed your config for some reason. Right now you are trying to run Redis queries against your webserver, and that is not going to work. Make sure you are connecting to the correct Redis server address and port and then try again.