Develop Reference

Provide object methods into the namespace of a python environment

I am trying to provide wrappers for short-cutting every-day commands. Python environments are very useful to do that.
Is it possible to provide all methods of an object to the local namespace within a new environment?
class my_object:
def method_a():
...
class my_environment:
...
def __enter__(self):
some_object = my_object()
# something like `from some_object import *` ??
return(some_object)
...
with my_environment() as some_object:
# standard syntax:
some_object.method_a()
# shortcut:
method_a() # how to make this possible?

It will be rather complex, and IMHO will not be worth it. The problem is that in Python, local variables are local to a function and not to a bloc. So what you are asking for would require that:
__enter__ declares nonlocal variables for all of the methods from some_object and saves their previous value if any
__exit__ restore the previous values if any of those variables, or deletes them if they did not previously existed
Possible but not really Pythonic IMHO (the reason why I have not proposed any code...). After all, inside a method Python requires the object to be explicitely passed, and requires it to be prepended to any internal method call or attribute access. So my advice is to stick to the standard syntax here...

What you are looking for is class hierarchy. On the way, please be careful with the conventions for class names.
class MyObject:
def method_a():
...
class MyEnvironment(MyObject):
...
def __enter__(self):
return self
...
with MyEnvironment() as some_object:
# standard syntax:
some_object.method_a()
The shortcut you are looking doesn't make much sense because the method_a() was defined as a method, therefore it should be called together with the instance.
Maybe #staticmethod can serve your case better.
class MyEnvironment:
#staticmethod
def method_a():
...
MyEnvironment.method_a()

Resource Aquisition Is Initialization, in Python

I am new to Python. I come from C++.
In some code reviews, I've had several peers wanting me to move things from init and del to a start and stop method. Most of them time, this goes against the RAII that was beaten into my head with decades of C++.
https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization
Is RAII not a thing in Python?
Shouldn't it be?
After all, we can throw exceptions and we'd want to release resources when we do, no?
If it isn't. Can someone give some insight as to why things are done differently? Is there a language feature that I don't understand?
if I have:
class Poop:
def __init__:
# Get some Windows Resource
def __del__:
#Release some Windows Resource
def foo():
poop = Poop()
raise Exception("Poop happens")
The Windows Resource is released, right?

RAII works in C++ because destruction is deterministic.
In garbage collected languages like Python, your object could theoretically never be destroyed, even if you call del on it.
Anyway, the idiomatic way to handle resources in Python is not with RAII, nor with start/stop, but with context managers.
The simplest example is with a file object:
with open('this_file.txt') as f:
# ... do stuff with f ...
# ... back to code that doesn't touch f ...
The with statement is, more or less, a try-finally block that creates a resource and ensures that the resource is cleaned up when the block ends; something like this:
try:
f = open('this_file.txt')
# ... do stuff with f ...
finally:
f.close()
# ... back to code that doesn't touch f ...
I don't know Java, but I believe that the JVM also uses garbage collection, and similarly try-finally is an idiom for resource management in Java.
Anyway, the with statement takes a context manager, which is an instance of a class defining the __enter__ and __exit__ methods (see the docs).
For completeness, there may be cases where you want a context manager, but don't want to define a whole class just for that. In that case, contextlib may help.
A worked example; say you have a resource:
class Resource:
def method(self):
pass
get_resource = Resource
release_resource = lambda x: None
A RAII-like class might look something like this:
class RAIILike:
def __init__(self):
self.resource = get_resource()
def __del__(self):
release_resource(self.resource)
def do_complex_thing(self):
# do something complex with resource
pass
raii_thingy = RAIILike()
And you would use the resource like this:
raii_thingy.resource.method()
On the other hand, a context managed resource could look like this...
class ContextManagedResource:
def __enter__(self):
self._resource = get_resource()
return self._resource
def __exit__(self, exc_type, exc_value, traceback):
if exc_type is not None:
# handle exception here
pass
else:
pass
release_resource(self._resource)
return True
...and be used like this:
with ContextManagedResource() as res:
res.method()
Once the with block ends, the resource will be automatically released, regardless of whether the object that obtained it has been garbage collected.

Your own reference to wikipedia says:
Perl, Python (in the CPython implementation), and PHP manage
object lifetime by reference counting, which makes it possible to use
RAII. Objects that are no longer referenced are immediately destroyed
or finalized and released, so a destructor or finalizer can release
the resource at that time. However, it is not always idiomatic in such
languages, and is specifically discouraged in Python (in favor of
context managers and finalizers from the weakref package).

You can do RAII in python, or get pretty close. However, unlike C++ where you do the work in the constuctor and destructor, in python you need to use the dunder functions of enter and exit. This post has a excellent write up of how to write the functions and how they will behave in the presence of exceptions: https://preshing.com/20110920/the-python-with-statement-by-example/

Use context manager as a function

I'd like to make a function which would also act as context manager if called with with statement. Example usage would be:
# Use as function
set_active_language("en")
# Use as context manager
with set_active_language("en"):
...
This is very similar to how the standard function open is used.
Here's the solution I came up with:
active_language = None # global variable to store active language
class set_active_language(object):
def __init__(self, language):
global active_language
self.previous_language = active_language
active_language = language
def __enter__(self):
pass
def __exit__(self, *args):
global active_language
active_language = self.previous_language
This code is not thread-safe, but this is not related to the problem.
What I don't like about this solution is that class constructor pretends to be a simple function and is used only for its side effects.
Is there a better way to do this?
Note that I haven't tested this solution.
Update: the reason why I don't want to split function and context manager into separate entities is is naming. The function and the context manager do the same thing, basically, so it seems reasonable to use one name for both. Naming the context processor would be problematic if I wanted to keep it separate. What should it be? active_language? This name may (and will) collide with variable name. override_active_language might work, though.

Technically no, you cannot do this. But you can fake it well enough that people (who didn't overthink it) wouldn't notice.
def set_active_language(language):
global active_language
previous_language = active_language
active_language = language
class ActiveScope(object):
def __enter__(self):
pass
def __exit__(self, *args):
global active_language
active_language = previous_language
return ActiveScope()
When used as a function the ActiveScope class is just a slightly wasteful no-op.

Hopefully someone will prove me wrong, but I think the answer is no: there is no other way. And also, another short-coming of the method you chose is that it might misbehave when used along with other context managers in a with a, b, c: statement. The intended side-effect of the CM is executed on object construction, and not in the __enter__ method as would be expected.
To be able to do what you want, you would have to know from inside the class constructor whether it was initialized as a context manager in a with statement, or simply called as a function. As far as I can tell, there is no way to gather that, not even with the inspect module.

How do I design this procedural code as class based (object oriented)?

I'm a beginner-intermediate self taught Python developer,
In most of the projects I completed, I can see the following procedure repeats. I don't have any outside home code experiences, I think the below code is not so professional as it is not reusable, and seems like it is not fitting all together in a container, but loosely coupled functions on different modules.
def get_query():
# returns the query string
pass
def make_request(query):
# makes and returns the request with query
pass
def make_api_call(request):
# calls the api and returns response
pass
def process_response(response):
# process the response and returns the details
pass
def populate_database(details):
# populates the database with the details and returns the status of population
pass
def log_status(status):
# logs the status so that developer knows whats happening
pass
query = get_query()
request = make_request(query)
response = make_api_call(request)
details = process_response(response)
status = populate_database(details)
log_status(status)
How do I design this procedure as a class based design?

If I understand correctly, you want these group of functions to be reused. Good approach to this would be create Abstract base class with these methods as shown below:
from abc import ABCMeta
class Generic(object):
__metaclass__ = ABCMeta
def get_query(self):
# returns the query string
pass
def make_request(self, query):
# makes and returns the request with query
pass
def make_api_call(self, request):
# calls the api and returns response
pass
def process_response(self, response):
# process the response and returns the details
pass
def populate_database(self, details):
# populates the database with the details and returns the status of population
pass
def log_status(self, status):
# logs the status so that developer knows whats happening
pass
Now whenever you need to use any of these methods in your project, inherit your class from this abstract class.
class SampleUsage(Generic):
def process_data(self):
# In any of your methods you can call these generic functions
self.get_query()
And then you can create object to actually get results which you want.
obj = SampleUsage()
obj.process_data()

You may have several classes here. To name a few, Query, Request, Response, Database, Logger
Some of your functions may map as follows:
make_query -> Query.make() constructor or class method
make_request -> Request.make(query) constructor or class method
make_api_call -> Request.make_api_call()
process_response -> Response.process()
populate_database -> Database.populate()
log_status -> Logger.status Consider using logging module
You have to think about your application and design it as an interaction of cooperating objects. This is just a starting point in order for you to be partition the functionality of the application between the classes.
Some of these Classes may be Singletons, meaning they are instantiated only once at the beginning of the application and accessed everywhere else. Database and Logger fit that role.
Here is some skeleton definitions:
class Query(object):
#classmethod
def make(cls, *args, **kwargs):
pass
class Request(object):
#classmethod
def make(cls, query):
pass
def make_api_call(self, *args, **kwargs):
# possibly return Response
pass
class Response(object):
def process_response(self):
pass
class Database(object):
_the_db = None
#classmethod
def get_db(cls):
# Simple man's singleton
if not cls._the_db:
cls._the_db = Database()
return cls._the_db
def populate(self):
pass
class Logger(object):
def log(self):
# consider using logging module
pass

I think what lacks in your question is the sense of purpose. You don't switch a perfectly fine procedural code to object-oriented code without a reason. Depending on the reason, there are several ways to do it. As this problem is quite a common one, there are some common techniques that are known to work well for some common reasons.
So, let's assume you encapsulate the main procedure in an object. What are your needs?
Allow re-using the procedure, possibly overriding some parts? See below the template method pattern.
Allow dynamically altering the behavior of the procedure at runtime depending on external factors? Look into the Strategy pattern.
Allow dynamically altering the behavior of the procedure at runtime depending on internal factors? For example, if some request may switch the procedure into "maintenance mode"? Look into the State pattern.
I'll just describe the template method pattern, which looks the closest to Marty's concerns. I cut down the example to 3 steps so it's easier to explain, but I made you a fully working example gist.
The template method
You want to provide a way to re-use the procedure, while allowing to override some well-defined parts? Let's create an empty, fill-in-the-blanks-style template:
class BaseRequestProcesor(object):
def get_query(self):
raise NotImplementedError()
def process_query(self, query):
raise NotImplementedError()
def log_status(self, status):
raise NotImplementedError()
def process(self): # main procedure
query = self.get_query()
status = self.process_query(query)
self.log_status(status)
__call__ = process # allow "calling" the requestprocessor
We have our basic template. Let's create some template fillers:
class DemoQueryReader(object):
def get_query(self):
return 'this is a query'
class HelloQueryProcessor(object):
def process_query(self, query):
return 'Hello World, {}!'.format(query)
class StdoutLogProcessor(object):
def log_status(self, status):
print(status)
Now build a full request processor from the bits we want. This is where the pieces comes together:
class DemonstrationProcessor(DemonQueryReader, HelloQueryProcessor, StdoutLogProcessor, BaseRequestProcessor):
pass
Demonstrating in the console:
>>> from marty_example import DemonstrationProcessor
>>> processor = DemonstrationProcessor()
>>> processor()
Hello World, this is a query!
This is the most pedantic example you can build. You could supply default implementations when that makes sense (doing nothing, mostly). And you can group together overrides, should that make sense.
The point is, you made your process a template, allowing easy override of chosen details, while still being in control of the overall workflow. This is a form of inversion of control.

You can also save a Python file with the class name, or you can create external modules with some functions, organizing them into the modules depending on what they do. Some modules will only contain one function; others will contain a lot.

How do I correctly clean up a Python object?

class Package:
def __init__(self):
self.files = []
# ...
def __del__(self):
for file in self.files:
os.unlink(file)
__del__(self) above fails with an AttributeError exception. I understand Python doesn't guarantee the existence of "global variables" (member data in this context?) when __del__() is invoked. If that is the case and this is the reason for the exception, how do I make sure the object destructs properly?

I'd recommend using Python's with statement for managing resources that need to be cleaned up. The problem with using an explicit close() statement is that you have to worry about people forgetting to call it at all or forgetting to place it in a finally block to prevent a resource leak when an exception occurs.
To use the with statement, create a class with the following methods:
def __enter__(self)
def __exit__(self, exc_type, exc_value, traceback)
In your example above, you'd use
class Package:
def __init__(self):
self.files = []
def __enter__(self):
return self
# ...
def __exit__(self, exc_type, exc_value, traceback):
for file in self.files:
os.unlink(file)
Then, when someone wanted to use your class, they'd do the following:
with Package() as package_obj:
# use package_obj
The variable package_obj will be an instance of type Package (it's the value returned by the __enter__ method). Its __exit__ method will automatically be called, regardless of whether or not an exception occurs.
You could even take this approach a step further. In the example above, someone could still instantiate Package using its constructor without using the with clause. You don't want that to happen. You can fix this by creating a PackageResource class that defines the __enter__ and __exit__ methods. Then, the Package class would be defined strictly inside the __enter__ method and returned. That way, the caller never could instantiate the Package class without using a with statement:
class PackageResource:
def __enter__(self):
class Package:
...
self.package_obj = Package()
return self.package_obj
def __exit__(self, exc_type, exc_value, traceback):
self.package_obj.cleanup()
You'd use this as follows:
with PackageResource() as package_obj:
# use package_obj

The standard way is to use atexit.register:
# package.py
import atexit
import os
class Package:
def __init__(self):
self.files = []
atexit.register(self.cleanup)
def cleanup(self):
print("Running cleanup...")
for file in self.files:
print("Unlinking file: {}".format(file))
# os.unlink(file)
But you should keep in mind that this will persist all created instances of Package until Python is terminated.
Demo using the code above saved as package.py:
$ python
>>> from package import *
>>> p = Package()
>>> q = Package()
>>> q.files = ['a', 'b', 'c']
>>> quit()
Running cleanup...
Unlinking file: a
Unlinking file: b
Unlinking file: c
Running cleanup...

A better alternative is to use weakref.finalize. See the examples at Finalizer Objects and Comparing finalizers with __del__() methods.

As an appendix to Clint's answer, you can simplify PackageResource using contextlib.contextmanager:
#contextlib.contextmanager
def packageResource():
class Package:
...
package = Package()
yield package
package.cleanup()
Alternatively, though probably not as Pythonic, you can override Package.__new__:
class Package(object):
def __new__(cls, *args, **kwargs):
#contextlib.contextmanager
def packageResource():
# adapt arguments if superclass takes some!
package = super(Package, cls).__new__(cls)
package.__init__(*args, **kwargs)
yield package
package.cleanup()
def __init__(self, *args, **kwargs):
...
and simply use with Package(...) as package.
To get things shorter, name your cleanup function close and use contextlib.closing, in which case you can either use the unmodified Package class via with contextlib.closing(Package(...)) or override its __new__ to the simpler
class Package(object):
def __new__(cls, *args, **kwargs):
package = super(Package, cls).__new__(cls)
package.__init__(*args, **kwargs)
return contextlib.closing(package)
And this constructor is inherited, so you can simply inherit, e.g.
class SubPackage(Package):
def close(self):
pass

Here is a minimal working skeleton:
class SkeletonFixture:
def __init__(self):
pass
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
pass
def method(self):
pass
with SkeletonFixture() as fixture:
fixture.method()
Important: return self
If you're like me, and overlook the return self part (of Clint Miller's correct answer), you will be staring at this nonsense:
Traceback (most recent call last):
File "tests/simplestpossible.py", line 17, in <module>
fixture.method()
AttributeError: 'NoneType' object has no attribute 'method'
Hope it helps the next person.

I don't think that it's possible for instance members to be removed before __del__ is called. My guess would be that the reason for your particular AttributeError is somewhere else (maybe you mistakenly remove self.file elsewhere).
However, as the others pointed out, you should avoid using __del__. The main reason for this is that instances with __del__ will not be garbage collected (they will only be freed when their refcount reaches 0). Therefore, if your instances are involved in circular references, they will live in memory for as long as the application run. (I may be mistaken about all this though, I'd have to read the gc docs again, but I'm rather sure it works like this).

I think the problem could be in __init__ if there is more code than shown?
__del__ will be called even when __init__ has not been executed properly or threw an exception.
Source

Just wrap your destructor with a try/except statement and it will not throw an exception if your globals are already disposed of.
Edit
Try this:
from weakref import proxy
class MyList(list): pass
class Package:
def __init__(self):
self.__del__.im_func.files = MyList([1,2,3,4])
self.files = proxy(self.__del__.im_func.files)
def __del__(self):
print self.__del__.im_func.files
It will stuff the file list in the del function that is guaranteed to exist at the time of call. The weakref proxy is to prevent Python, or yourself from deleting the self.files variable somehow (if it is deleted, then it will not affect the original file list). If it is not the case that this is being deleted even though there are more references to the variable, then you can remove the proxy encapsulation.

It seems that the idiomatic way to do this is to provide a close() method (or similar), and call it explicitely.

A good idea is to combine both approaches.
To implement a context manager for explicit life-cycle handling. As well as handle cleanup in case the user forgets it or it is not convenient to use a with statement. This is best done by weakref.finalize.
This is how many libraries actually do it. And depending on the severity, you could issue a warning.
It is guaranteed to be called exactly once, so it is safe to call it at any time before.
import os
from typing import List
import weakref
class Package:
def __init__(self):
self.files = []
self._finalizer = weakref.finalize(self, self._cleanup_files, self.files)
#staticmethod
def _cleanup_files(files: List):
for file in files:
os.unlink(file)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
self._finalizer()
weakref.finalize returns a callable finalizer object which will be called when obj is garbage collected. Unlike an ordinary weak reference, a finalizer will always survive until the reference object is collected, greatly simplifying lifecycle management."
Unlike atexit.register the object is not held in memory until the interpreter is shut down.
And unlike object.__del__, weakref.finalize is guaranteed to be called at interpreter shutdown. So it is much more safe.

atexit.register is the standard way as has already been mentioned in ostrakach's answer.
However, it must be noted that the order in which objects might get deleted cannot be relied upon as shown in example below.
import atexit
class A(object):
def __init__(self, val):
self.val = val
atexit.register(self.hello)
def hello(self):
print(self.val)
def hello2():
a = A(10)
hello2()
a = A(20)
Here, order seems legitimate in terms of reverse of the order in which objects were created as program gives output as :
20
10
However when, in a larger program, python's garbage collection kicks in object which is out of it's lifetime would get destructed first.