I'm making a program that detects python files and reads their classes and functions for then to be called. How do I list the functions from another python file without importing it? and how do I call them?
Structure of the program:
Index.py (main py file)
/Content
/Modules
Modules.py
ChickenModule.py (module I want to get functions from)
Modules.py list all the python files from /Modules and stores them on a list.
Index.py calls the functions in Modules.py (Just to keep things less messy)
ChickenModule.py has a Class named ChickenStuff that prints "I'm a chicken" whenever it's self.(something) called.
The goal of the program is for the user to be able to put .py files in modules and when running index.py the functions and classes of said .py files will be listed.
There's a built in library called importlib that might be what you need. I'm assuming that you want classes listed. You'll need inspect for this.
import os
import os.path as path
import importlib
import inspect
# get a list of all files in Content/Modules
filelist = os.listdir("Content/Modules")
# import every file ending in .py
for fname in filelist:
if path.splitext(fname)[1] == 'py':
my_module = importlib.import_module(path.splitext(fname)[0]) # load the module
for _, obj in inspect.getmembers(my_module): # iterate through members
if isinstance(obj, type): # check if members is a class
print(obj)
Related
In my project I have a generic class named Interface and many subclasses which inherit from Interface:
/interface_folder/
interface.py
interface_a.py
interface_b.py
interface_c.py
...
/test/
test.py
The Interface class has a method which lists all the available methods of all its subclasses : their names, docstring, input and output arguments. I did this using the inspect module on Interface.__subclasses__().
In test.py, if I want the Interface method using __subclasses__() to work, I need to import interface_a.py, interface_b.py, interface_c.py, etc.
I would like to automate these imports, so that I can add as many interface_x.py as I want in the folder without having to think of adding every time "from interface_folder.interface_d import Interface_d" at the beginning of test.py.
Is this possible?
If not, is there a solution to force python to build the __subclasses__()?
Here is an idea:
Create a new subfolder, say /interface_folder/child_interfrace_folder/.
Move all interface_x.py in the new subfolder.
Import all files of the new subdirectory in interface.py. To do this:
Create an __init__.py file in the new subdirectory and open it in your editor.
In the new __init__.py file, list all interface_x.py files and put them in a variable named __all__. Example:
From here:
# __init__.py
from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]
You can now use from child_interfrace_folder import * in your interface.py file.
I have a module that includes signal processors in a subdirectory. I'd like to have things set up so that the main file (above that subdirectory) can import all python files in the subdirectory, and call a known function in them. This is so that the input signal can be run through each signal processor in that directory. Is there a clean way to do this in Python?
Assuming the files tree is:
main.py
processors/
processors/proc1.py
processors/proc1.py
and each proc module has a method:
def process():
# Do something
Then you can use the following code from main.py:
import pkgutil
import processors
if __name__ == '__main__':
modules = []
for loader, name, is_pkg in pkgutil.walk_packages(processors.__path__):
modules.append(loader.find_module(name).load_module(name))
# Later in the code when processing is required:
for module in modules:
module.process()
This code lacks input checks, e.g. each module has process() method.
I am new to writing functions and modules, and I need some help creating a function within my new module that will make my currently repetitive process of loading data much more efficient.
I would like this function to reside in a larger overall module with other function that I can keep stored on my home directory and not have to copy into my working directory every time I want to call one of my function.
The data that I have is just some JSON Twitter data from the streaming API, and I would like to use a function to load the data (list of dicts) into a list that I can access after the function runs by using something like data = my_module.my_function('file.json').
I've created a folder in my home directory for my python modules and I have two files in that directory: __init__.py and my_module.py.
I've also went ahead and added the python module folder to sys.path by using sys.path.append('C:\python')
Within python module folder, the file __init__.py has nothing in it, it's just an empty file.
Do I need to put anything in the __init__.py file?
my_module.py has the following code:
import json
def my_function(parameter1):
tweets = []
for line in open(parameter1):
try:
tweets.append(json.loads(line))
except:
pass
I would like to cal the function as such:
import my_module
data = my_module.my_function('tweets.json')
What else do I need to do to create this function to make loading my data more efficient?
To import the module from the package, for example:
import my_package.my_module as my_module
would do what you want. In this case it's fine to leave the init.py empty, and the module will be found just by it being in the package folder "my_package". There are many alternatives on how to define a package/module structure and how to import them, I encourage you to read up, as otherwise you will get confused at some point.
I would like this function to reside in a larger overall module with
other function that I can keep stored on my home directory and not
have to copy into my working directory every time I want to call one
of my function.
For this to work, you need to create a .pth file in C:\Python\site-packages\ (or wherever you have installed Python). This is a simple text file, and inside it you'd put the path to your module:
C:/Users/You/Some/Path/
Call it custom.pth and make sure its in the site-packages directory, otherwise it won't work.
I've also went ahead and added the python module folder to sys.path by
using sys.path.append('C:\python')
You don't need to do this. The site-packages directory is checked by default for modules.
In order to use your function as you intend, you need to make sure:
The file is called my_module.py
It is in the directory you added to the .pth file as explained earlier
That's it, nothing else needs to be done.
As for your code itself the first thing is you are missing a return statement.
If each line in the file is a json object, then use this:
from __future__ import with_statement
import json
def my_function(parameter1):
tweets = []
with open(parameter1) as inf:
for line in inf:
try:
tweets.append(json.loads(line))
except:
pass
return tweets
If the enter file is a json object, then you can do this:
def my_function(parameter1):
tweets = None
with open(parameter1) as inf:
tweets = json.load(inf)
return tweets
Let me explain the use case...
In a simple python web application framework designed for Google App Engine, I'd like to have my models loaded automatically from a 'models' directory, so all that's needed to add a model to the application is place a file user.py (for example), which contains a class called 'User', in the 'models/' directory.
Being GAE, I can't read from the file system so I can't just read the filenames that way, but it seems to me that I must be able to 'import * from models' (or some equivalent), and retrieve at least a list of module names that were loaded, so I can subject them to further processing logic.
To be clear, I want this to be done WITHOUT having to maintain a separate list of these module names for the application to read from.
You can read from the filesystem in GAE just fine; you just can't write to the filesystem.
from models import * will only import modules listed in __all__ in models/__init__.py; there's no automatic way to import all modules in a package if they're not declared to be part of the package. You just need to read the directory (which you can do) and __import__() everything in it.
As explained in the Python tutorial, you cannot load all .py files from a directory unless you list them manually in the list named __all__ in the file __init__.py. One of the reasons why this is impossible is that it would not work well on case-insensitive file systems -- Python would not know in which case the module names should be used.
Let me start by saying that I'm not familiar with Google App Engine, but the following code demonstrates how to import all python files from a directory. In this case, I am importing files from my 'example' directory, which contains one file, 'dynamic_file.py'.
import os
import imp
import glob
def extract_module_names(python_files):
module_names = []
for py_file in python_files:
module_name = (os.path.basename(py_file))[:-3]
module_names.append(module_name)
return module_names
def load_modules(modules, py_files):
module_count = len(modules)
for i in range(0, module_count):
globals()[modules[i]] = imp.load_source(modules[i], py_files[i])
if __name__ == "__main__":
python_files = glob.glob('example/*.py')
module_names = extract_module_names(python_files)
load_modules(module_names, python_files)
dynamic_file.my_func()
Also, if you wish to iterate over these modules, you could modify the load_modules function to return a list of the loaded module objects by appending the 'imp.load_source(..)' call to a list.
Hope it helps.
In a big application I am working, several people import same modules differently e.g.
import x
or
from y import x
the side effects of that is x is imported twice and may introduce very subtle bugs, if someone is relying on global attributes
e.g. suppose I have a package mypakcage with three file mymodule.py, main.py and init.py
mymodule.py contents
l = []
class A(object): pass
main.py contents
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
it prints
updated list [1]
lets check []
updated list [1, 1]
lets check again []
because now there are two lists in two different modules, similarly class A is different
To me it looks serious enough because classes itself will be treated differently
e.g. below code prints False
def create():
from mypackage import mymodule
return mymodule.A()
def check(a):
import mymodule
return isinstance(a, mymodule.A)
print check(create())
Question:
Is there any way to avoid this? except enforcing that module should be imported one way onyl. Can't this be handled by python import mechanism, I have seen several bugs related to this in django code and elsewhere too.
Each module namespace is imported only once. Issue is, you're importing them differently. On the first you're importing from the global package, and on the second you're doing a local, non-packaged import. Python sees modules as different. The first import is internally cached as mypackage.mymodule and the second one as mymodule only.
A way to solve this is to always use absolute imports. That is, always give your module absolute import paths from the top-level package onwards:
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
from mypackage import mymodule
return mymodule.l
Remember that your entry point (the file you run, main.py) also should be outside the package. When you want the entry point code to be inside the package, usually you use a run a small script instead. Example:
runme.py, outside the package:
from mypackage.main import main
main()
And in main.py you add:
def main():
# your code
I find this document by Jp Calderone to be a great tip on how to (not) structure your python project. Following it you won't have issues. Pay attention to the bin folder - it is outside the package. I'll reproduce the entire text here:
Filesystem structure of a Python project
Do:
name the directory something
related to your project. For example,
if your project is named "Twisted",
name the top-level directory for its
source files Twisted. When you do
releases, you should include a version
number suffix: Twisted-2.5.
create a directory Twisted/bin and
put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects.
If your project
is expressable as a single Python
source file, then put it into the
directory and name it something
related to your project. For example,
Twisted/twisted.py. If you need
multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and
place your source files in it. For
example,
Twisted/twisted/internet.py.
put
your unit tests in a sub-package of
your package (note - this means that
the single Python source file option
above was a trick - you always need at
least one other file for your unit
tests). For example,
Twisted/twisted/test/. Of course,
make it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory
called src or lib. This makes it
hard to run without installing.
put
your tests outside of your Python
package. This makes it hard to run the
tests against an installed version.
create a package that only has a
__init__.py and then put all your
code into __init__.py. Just make a
module instead of a package, it's
simpler.
try to come up with
magical hacks to make Python able to
import your module or package without
having the user add the directory
containing it to their import path
(either via PYTHONPATH or some other
mechanism). You will not correctly
handle all cases and users will get
angry at you when your software
doesn't work in their environment.
I can only replicate this if main.py is the file you are actually running. In that case you will get the current directory of main.py on the sys path. But you apparently also have a system path set so that mypackage can be imported.
Python will in that situation not realize that mymodule and mypackage.mymodule is the same module, and you get this effect. This change illustrates this:
def add(x):
from mypackage import mymodule
print "mypackage.mymodule path", mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
print "mymodule path", mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
$ export PYTHONPATH=.
$ python mypackage/main.py
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mymodule' from '/tmp/mypackage/mymodule.pyc'>
But add another mainfile, in the currect directory:
realmain.py:
from mypackage import main
and the result is different:
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
So I suspect that you have your main python file within the package. And in that case the solution is to not do that. :-)