I have a tasks.py that contains a subclass of Task.
According to the docs the base class is instantiated only once per tasks.
But this is only true for same tasks method. Calling a different task creates a new instance. So I can't access sessions via get_sessions created with create_session. How may I have only a single instance that is shared between different tasks?
class AuthentificationTask(Task):
connections = {}
def login(self, user, password, server):
if not user in self.connections:
self.connections = {user: ServerConnection(verbose=True)}
# from celery.contrib import rdb
# rdb.set_trace()
self.connections[user].login(user=user, password=password, server=server)
#task(bind=True, max_retries=1, queue='test', base=AuthentificationTask)
def create_session(self, user, password, server):
self.login(user, password, server)
#task(bind=True, max_retries=1, queue='test', base=AuthentificationTask)
def get_sessions(self, user, password, server):
return self.connections[user].sessions
Set the task_cls arg for your Celery application like this:
class AuthentificationTask(Task):
def example(self):
logger.info('AuthentificationTask.example() method was called')
#celery.task(bind=True)
def test_my_task(self):
# call AuthentificationTask.example
self.example()
app = celery.Celery(
__name__,
broker='redis://localhost:6379/0',
task_cls=AuthentificationTask,
# other args
)
In this case will be use your custom class for all tasks as default.
Seems this was an issue on my site caused by reinitialising self.connections each time.
self.connections = {user: ServerConnection(verbose=True)}
In further tests base was instantiated only once for all (different) tasks. Thanks #Danila Ganchar for suggesting an alternative approach. I will give it a try!
You're on the right track by making connections a class variable on AuthentificationTask. That makes it available as a property on the class itself (i.e. as AuthentificationTask.connections). When you reference self.connections in the login method, I believe Python is looking for an instance variable connections, not the class variable of the same name. For the desired behavior, replace self.connections (in both login and get_sessions) with AuthentificationTask.connections.
Related
I've got a Django app and a message queue and I want to be able to switch between queue services easily (SQS or RabbitMQ for example).
So I set up a BaseQueue "interface":
class BaseQueue(ABC):
#abstractmethod
def send_message(self, queue_name, message, message_attributes=None):
pass
And two concrete classes that inherit from BaseQueue:
class SqsQueue(BaseQueue):
def send_message(self, queue_name, message, message_attributes=None):
# code to send message to SQS
class RabbitMqQueue(BaseQueue):
def send_message(self, queue_name, message, message_attributes=None):
# code to send message to RabbitMQ
Then in settings.py I've got a value pointing to the implementation the app should use:
QUEUE_SERVICE_CLS = "queues.sqs_queue.SqsQueue"
Because it's a Django app it's in settings.py, but this value could be coming from anywhere. It just says where the class is.
Then I've got a QueueFactory whose job is to return the queue service to use:
class QueueFactory:
#staticmethod
def default():
return import_string(settings.QUEUE_SERVICE_CLS)()
The factory imports the class and instantiates it.
I would then use it like so:
QueueFactory.default().send_message(queue_name, message)
It works, but I was wondering if there's a more Python way to do it? Like with some magic methods?
I have a Python class where the constructor creates a MySQL database connection as follows:
class MySQL:
def __init__(self):
self.client = self.get_client()
def get_client():
client = pymysql.connect(**mysql_credentials)
return client
The problem with this implementation is that the connection never ends. So I want to modify the class to create the database connection in the __enter__ method and close the connection on __exit__ method as follows:
class MySQL:
def __enter__(self):
self.client = self.get_client()
def __exit__(self, exc_type, exc_val, exc_tb):
self.client.close()
def get_client():
client = pymysql.connect(**mysql_credentials)
return client
def execute_query(self, sql_query: str):
with self.client.cursor() as cursor:
cursor.execute(sql_query)
Now the question. How can instantiate MySQL class with __enter__ and __exit__ inside the constructor of another class?
Can't be do like this because it calls MySQL __init__ method and it will not open the connection:
class AnotherClass:
def __init__(self):
self.mysql_cli = MySQL()
def run_etl(self):
self.mysql_cli.execute_query('''SELECT VERSION();''')
Any suggestions?
Many thanks!
There are three different ways for a class to properly use a context manager inside its own code.
One option is for the new class to be a context manager itself, so its users can do the managing of the connection lifetimes through it. This is pretty straight forward, just call the __enter__ and __exit__ methods of your contained context manager from your own versions of those methods. That might look like this:
class AnotherClass:
def __init__(self):
self.db = MySQL()
def __enter__(self):
self.db.__enter__()
return self
def __exit__(self, *exc_args):
return self.db.__exit__(*exc_args)
def do_stuff(self):
# do stuff with the database
The burden of managing the connection is delegated to the user of the class, who can use with statements on their AnotherClass instance:
with AnotherClass() as another:
another.do_stuff()
This approach can get tedious though if you have lots of layers of aggregation, and need all the intermediate containers to become context managers just because they contain one at some much lower level.
Another approach is to make separate database connections for each operation that needs to use the database for something. This lets you use with statements for the connection management, but may require that you do a lot more connecting and reconnecting than you'd like:
class AnotherClass:
def __init__(self):
self.db = MySQL()
def do_stuff(self):
with self.db:
# do the actual stuff here, using the database
With this design, the user of AnotherClass doesn't need to use the context manager protocol because the database connections are only alive during the runtime of the methods you call.
another = AnotherClass()
another.do_stuff() # creates and destroys the db connection internally
The third option is to say that AnotherClass should not be involved with the connection management of the MySQL class it uses. That's the responsibility of some other part of the code, and we don't even need to know about it here. For this approach, you probably want the MySQL instance to be created elsewhere (wherever it is being managed) and have it be passed in as an argument to the AnotherClass constructor:
class AnotherClass:
def __init__(self, db):
self.db = db
def do_stuff(self):
# do stuff with the database
The caller would do something like this:
db = MySQL()
with db:
another = AnotherClass(db)
another.do_stuff()
This has the advantage that the db object passed in to AnotherClass can be whatever type you want, as long as it has the APIs that AnotherClass expects. If you need to change your database from MySQL to Postgres, you don't need to change AnotherClass, only the higher level code.
Im trying to follow this Celery Based Background Tasks to create a celery settings for a simple application.
In my task.py
from celery import Celery
def make_celery(app):
celery = Celery(app.import_name, backend=app.config['CELERY_RESULT_BACKEND'],
broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
This method works in the app.py of main flask application.
from flask import Flask
flask_app = Flask(__name__)
flask_app.config.update(
CELERY_BROKER_URL='redis://localhost:6379',
CELERY_RESULT_BACKEND='redis://localhost:6379'
)
celery = make_celery(flask_app)
#celery.task()
def add_together(a, b):
return a + b
My use case is I want to create another module helpers.py where I
can define a collections of asynchronous classes. To separate
celery based methods and make it modular.
What I did is call the task.py module to other module helpers.py in order to create a class AsyncMail to handle email action background work.
from task import make_celery
class AsyncMail(object):
def __init__(self, app):
"""
:param app: An instance of a flask application.
"""
self.celery = make_celery(app)
def send(self, msg):
print(msg)
Now how can I access self.celery attribute to be a decorator for any method of the class?
#celery.task()
def send(self, msg):
print(msg)
If it impossible, what other alternative steps in order to achieved this problem?
You can't do what you're trying to do. At the time the class is being defined, there is no self, much less self.celery, to call, so you can't use #self.celery. Even if you had some kind of time machine, there could be 38 different AsyncMail instances created, and which one's self.celery would you want here?
Before getting into how you could do what you want, are you sure you want to? Do you actually want each AsyncMail object to have it own separate Celery? Normally you only have one per app, which is why normally this doesn't come up.
If you really wanted to, you could give each instance decorated methods after you have an object to decorate them with. But it's going to be ugly.
def __init__(self, app):
self.celery = make_celery(app)
# We need to get the function off the class, not the bound method off self
send = type(self).send
# Then we decorate it manually—this is all #self.celery.task does
send = self.celery.task(send)
# Then we manually bind it as a method
send = send.__get__(self)
# And now we can store it as an instance attribute, shadowing the class's
self.send = send
Or, if you prefer to put it all together in one line:
self.send = self.celery.task(type(self).send).__get__(self)
For Python 2, the "function off the class" is actually an unbound method, and IIRC you have to call __get__(self, type(self)) to turn it into a bound method at the end, but otherwise it should all be the same.
I have a Flask, SQLAlchemy webapp which uses a single mysql server. I want to expand the database setup to have a read-only slave server such that I can spread the reads between both master and slave while continuing to write to the master db server.
I have looked at few options and I believe I can't do this with plain SQLAlchemy. Instead I'm planning to create 2 database handles in my webapp, one each for master and slave db servers. Then using a simple random value use either the master/slave db handle for "SELECT" operations.
However, I'm not sure if this is the right way to go with using SQLAlchemy. Any suggestion/tips on how to pull this off?
I have an example of how to do this on my blog at http://techspot.zzzeek.org/2012/01/11/django-style-database-routers-in-sqlalchemy/ . Basically you can enhance the Session so that it chooses from master or slave on a query-by-query basis. One potential glitch with that approach is that if you have one transaction that calls six queries, you might end up using both slaves in one request....but there we're just trying to imitate Django's feature :)
A slightly less magic approach that also establishes the scope of usage more explicitly I've used is a decorator on view callables (whatever they're called in Flask), like this:
#with_slave
def my_view(...):
# ...
with_slave would do something like this, assuming you have a Session and some engines set up:
master = create_engine("some DB")
slave = create_engine("some other DB")
Session = scoped_session(sessionmaker(bind=master))
def with_slave(fn):
def go(*arg, **kw):
s = Session(bind=slave)
return fn(*arg, **kw)
return go
The idea is that calling Session(bind=slave) invokes the registry to get at the actual Session object for the current thread, creating it if it doesn't exist - however since we're passing an argument, scoped_session will assert that the Session we're making here is definitely brand new.
You point it at the "slave" for all subsequent SQL. Then, when the request is over, you'd ensure that your Flask app is calling Session.remove() to clear out the registry for that thread. When the registry is next used on the same thread, it will be a new Session bound back to the "master".
Or a variant, you want to use the "slave" just for that call, this is "safer" in that it restores any existing bind back to the Session:
def with_slave(fn):
def go(*arg, **kw):
s = Session()
oldbind = s.bind
s.bind = slave
try:
return fn(*arg, **kw)
finally:
s.bind = oldbind
return go
For each of these decorators you can reverse things, have the Session be bound to a "slave" where the decorator puts it on "master" for write operations. If you wanted a random slave in that case, if Flask had some kind of "request begin" event you could set it up at that point.
Or, we can try another way. Such as we can declare two different class with all the instance attributes the same but the __bind__ class attribute is different. Thus we can use rw class to do read/write and r class to do read only. :)
I think this way is more easy and reliable. :)
We declare two db models because we can have tables in two different db with the same names. This way we can also bypass the 'extend_existing' error when two models with the same __tablename__.
Here is an example:
app = Flask(__name__)
app.config['SQLALCHEMY_BINDS'] = {'rw': 'rw', 'r': 'r'}
db = SQLAlchemy(app)
db.Model_RW = db.make_declarative_base()
class A(db.Model):
__tablename__ = 'common'
__bind_key__ = 'r'
class A(db.Model_RW):
__tablename__ = 'common'
__bind_key__ = 'rw'
Maybe this answer is too late! I use a slave_session to query the slave DB
class RoutingSession(SignallingSession):
def __init__(self, db, bind_name=None, autocommit=False, autoflush=True, **options):
self.app = db.get_app()
if bind_name:
bind = options.pop('bind', None)
else:
bind = options.pop('bind', None) or db.engine
self._bind_name = bind_name
SessionBase.__init__(
self, autocommit=autocommit, autoflush=autoflush,
bind=bind, binds=None, **options
)
def get_bind(self, mapper=None, clause=None):
if self._bind_name is not None:
state = get_state(self.app)
return state.db.get_engine(self.app, bind=self._bind_name)
else:
if mapper is not None:
try:
persist_selectable = mapper.persist_selectable
except AttributeError:
persist_selectable = mapper.mapped_table
info = getattr(persist_selectable, 'info', {})
bind_key = info.get('bind_key')
if bind_key is not None:
state = get_state(self.app)
return state.db.get_engine(self.app, bind=bind_key)
return SessionBase.get_bind(self, mapper, clause)
class RouteSQLAlchemy(SQLAlchemy):
def __init__(self, *args, **kwargs):
SQLAlchemy.__init__(self, *args, **kwargs)
self.slave_session = self.create_scoped_session({'bind_name':
'slave'})
def create_session(self, options):
return orm.sessionmaker(class_=RoutingSession,db=self,**options)
db = RouteSQLAlchemy(metadata=metadata, query_class=orm.Query)
I'm building a web crawler with Python. I created a parent class to save the user and the password, that I'd like to be inputed via keyboard.
The parent class looks like this:
class ParentCrawler(object):
def __init__(self):
"""Saves the user and the password"""
self.user = input("Email: ")
self.password = getpass.getpass("Password: ")
Then I created a subclass of that parent class with the idea of running parallel instances of it to make the crawling faster. But everytime I create a new object of the child class, I'm asked to input user and pass again, like in the pic below, and that's not what I want.
When a child object is created...
I know I could just hard code my user and pass into the parent class constructor method, but I'd like to know how to input them manually every time the program runned.
Th __init__ method of a class will be run every time you create a new instance. Since this values are needed just once, and you don't need different values for them for each instance, it makes little sense for their values to be requested inside the class initialiser, or other method.
Moreover, if your classes have nothing to do with user interaction on the terminal, there is no reason to hardcode this user interaction in the class code - if you make modifications to your program that will use the same class, and get this information from a configuration file, or from a POSTed web form, for example, you won't be able to use these classes in this way.
There is nothing wrong to pass the credentials as mandatory values when instantiating a class. To continue development and use of your program using it interactivelly at the terminal, you can create a simple function that will request these input data, and return them -
NUM_WORKERS = 4
def get_credentials():
user = input("Email: ")
password = getpass.getpass("Password: ")
return user, password
def main():
workers = []
user, password = get_credentials()
for i in range(NUM_WORKERS):
worker = Crawler(user, password)
workers.append(worker)
worker.start()
...
class Crawler:
def __init__(self, user, password):
...