I place my frequently used modules in a subdirectory lib/ and expect to load all modules into main.py by: (refer to Python: how to import from all modules in dir?)
from lib import *
but encounter the issue TypeError: 'module' object is not callable. More specifically, in main.py:
#!/usr/bin/env python
from lib import * # cause: TypeError: 'module' object is not callable
#from lib.DominatingSets import * # it works
dominatingSets = DominatingSets()
The full exception traceback:
$ python main.py
Traceback (most recent call last):
File "main.py", line 6, in <module>
dominatingSets = DominatingSets()
TypeError: 'module' object is not callable
The directories in a tree-like format.
$ tree -P '*.py' .
.
├── __init__.py
├── lib
│ ├── AnalyzeGraph.py
│ ├── AutoVivification.py
│ ├── DominatingSets.py
│ ├── __init__.py
│ ├── Output.py
│ ├── PlotGraph.py
│ ├── ProcessDatasets.py
│ └── ReadGTFS.py
├── main.py
The contents of lib/__init__.py are as follows. (refer to Loading all modules in a folder in Python)
from os.path import dirname, basename, isfile
import glob
modules = glob.glob(dirname(__file__)+"/*.py")
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not basename(f).startswith('__')] # exclude __init__.py
This confusion happened, in part, because your module names are the same as the names of the classes you want to load from them. (At least, that's what makes it more confusing.) Your code does correctly load the modules that your classes are in. However, it doesn't load the classes out of those modules, and this is what you actually wanted to do.
Because your class DominatingSets is in the module lib.DominatingSets, its full path from root is lib.DominatingSets.DominatingSets.
from lib import *
in your case will do the same thing as
from lib import DominatingSets
from lib import AnalyzeGraph
# ...
However,
from lib import DominatingSets
is equivalent to
import lib.DominatingSets
DominatingSets = lib.DominatingSets
but lib.DominatingSets is a module (lib/DominatingSets.py), not the class you want.
from lib.DominatingSets import DominatingSets
is equivalent to
import lib.DominatingSets
DominatingSets = lib.DominatingSets.DominatingSets
which is why it works: this is the class you want imported into the name DominatingSets.
If you want to have from lib import * import all the classes in the submodules, you need to import these classes into the lib module. For example, in lib/__init__.py:
from DominatingSets import *
from AnalyzeGraph import *
# ...
While you're making changes, I'd suggest (as others have) using normal Python naming conventions, and have your module names in lowercase: change DominatingSets.py to dominatingsets.py. Then this code would become
from dominatingsets import *
from analyzegraph import *
# ...
Looking at your Traceback, I think your problem might lie here:
Firstly, lets look at an example:
import datetime
d = datetime(2005, 23, 12)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable
Basically, we've just imported the entire datetime module, and we're trying to call it like a class object within a module. Let's now do:
k = datetime.datetime(2005, 12, 22)
print k
2005-12-22 00:00:00
No problems this time, as we are referencing the datetime object type within the datetime module
If we do:
from datetime import datetime
datetime
<type 'datetime.datetime'>
Again we reach the desired object, as we are importing the datetime class within the datetime module.
Also, using *
from datetime import *
d = datetime(2005, 3, 12)
will also work, as you are importing all the classes within the datetime module.
Your code saying:
from lib import * # This imports all MODULES within lib, not the classes
#from lib.DominatingSets import * # it works because you import the classes within the DominatingSets Module
You could either use from lib.DominatingSets import DominatingSets which should solve your problem, or if you stick to from lib import *, change your code to dominatingsets = DominatingSets.DominatingSets()
Hope this helps!
I learnt a lot from the accepted answer here ... but I still had problems with what to put in the lib/__init__.py file, if this directory lib is not actually included in PYTHONPATH.
I found that in addition to adding the parent directory of lib in the caller file, i.e.
sys.path.append( '.../parent_dir_of_lib' )
I either 1) had to do this in addition in the caller file:
sys.path.append( '.../parent_dir_of_lib/lib' )
Or 2) had to make the lib directory "self-loading", by putting this in its __init__.py:
import sys
from pathlib import Path
parent_dir_str = str( Path(__file__).resolve().parent )
sys.path.append( parent_dir_str )
from analyse_graph import *
from auto_vivification import *
...
Related
I have tried to organize my Python (Python version 3.9.2) project better and have created following file structure (I am using VS Code on Macbook Air M2). When I try to call a function in the "function file structure" I get the following error message:
NameError: name 'getAssetHoldings' is not defined
# File structure:
# main.py
my_functions
_init_.py
stock_functions.py
utility_functions.py
# Content of files:
#main.py:
import my_functions
import pandas as pd
import seaborn as sns
# load transaction file
filepath_transact = os.path.abspath(f"/Users/Sandbox/Data/transactions.csv")
df_sq = pd.read_csv(filepath_transact,encoding = "ISO-8859-1",sep=';')
# get asset holdings (function resides in stock_functions.py)
df_assets = getAssetHoldings(df_sq)
print('Assets:', '\n',df_assets,'\n')
#_init.py:
from .stock_functions import *
from .utility_functions import *
print('hello')
#stock_functions.py:
import pandas as pd
import yfinance as yf
import pandas_market_calendars as mcal
def getAssetHoldings(df):
df_numberAssets_k = df_transactions[df_transactions['Transaktionen'].isin(['Buy'])]
df_numberAssets_k = df_numberAssets_k[['Symbol','Quantity']]
df_numberAssets_v = df_transactions[df_transactions['Transaktionen'].isin(['Sell'])]
df_numberAssets_v = df_numberAssets_k[['Symbol','Quantity']]
....
return df_current_assets
I tried to restart the kernel and import the libraries and functions several times.
To close this question, I write here the solutions proposed in the comments.
To be more clear I re-write the main structure of the question:
│
├─ main.py
│
└─ my_functions/
├─ _init_.py
├─ stock_functions.py
└─ utility_functions.py
As #Roberto stated, the problem was that the __init__.py file needs the double underscore. And inside the main.py file, the function has to ba called through the dot notation my_functions.getAssetHoldings(...) since the module is imported with the line import my_functions.
I have a Python project in which I have the following folder structure:
> root
> download_module
> __init__.py
> downloadProcess.py
> sharedFunctions.py
> someHelper.py
> useSharedFunction.py
The download_module/__init__.py has the following code:
from .sharedFunctions import stringArgumentToDate
from .downloadProcess import downloadProcessMethod
The sharedFunctions.py file contains the following function:
def stringArgumentToDate(arg):
dateformat = "%m/%d/%Y"
date = None
if arg.isnumeric():
date = datetime.fromtimestamp(int(arg))
if date == None:
date = datetime.strptime(arg, dateformat)
return date
Then on the useSharedFunction.py I try to import the shared function and use it like this.
from download_module import stringArgumentToDate
from download_module import downloadProcessMethod
def main():
arg = '03/14/2022'
dateArg = stringArgumentToDate(arg)
if __name__ == '__main__':
main()
When I try to run this by using python3 useSharedFunction.py I got the following error:
Traceback (most recent call last):
File "useSharedFunction.py", line 4, in <module>
from download_module import stringArgumentToDate
File "/Users/jacobo/Documents/project/download_module/__init__.py", line 2, in <module>
from .download_module import downloadAndProcessMethod
File "/Users/jacobo/Documents/project/download_module/downloadProcess.py", line 10, in <module>
from sharedFunctions import stringArgumentToDate, otherFunction
ModuleNotFoundError: No module named 'sharedFunctions'
I do believe the error is in downloadProcess since at the beggining of the file we got this import:
from sharedFunctions import stringArgumentToDate, otherFunction
from someHelper import Helper
Which refers to sibling files.
However I'm unsure what will be a proper fix to allow to run the downloadProcess.py main independently but also, being able to call it one of its method from a root or any other file out of the module.
Consider this structure:
┬ module
| ├ __init__.py
| ├ importing_submodule.py
| └ some_submodule.py
├ __main__.py
├ some_submodule.py
└ module_in_parent_dir.py
with content:
__main__.py
import module
/module/__init__.py
from . import importing_submodule
/module/importing_submodule.py
from some_submodule import SomeClass
/module/some_submodule.py
print("you imported from module")
class SomeClass:
pass
/some_submodule.py
print("you imported from root")
class SomeClass:
pass
/module_in_parent_dir.py
class SomeOtherClass:
pass
How sibling import works
(skip this section if you know already)
Now lets run __main__.py and it will say "you imported from root".
But if we change code a bit..
/module/importing_submodule.py
from module.some_submodule import SomeClass
It now says "You imported from module" as we wanted, probably with scary red line in IDE saying "Unresolved reference" if you didn't config working directory in IDE.
How this happen is simple: script root(Current working directory) is decided by main script(first script that's running), and python uses namespaces.
Python's import system uses 2 import method, and for convenience let's call it absolute import and relative import.
Absolute import: Import from dir listed in sys.path and current working directory
Relative import: Import relative to the very script that called import
And what decide the behavior is whether we use . at start of module name or not.
Since we imported by from some_submodule without preceeding dot, python take it as 'Absolute import'(the term we decided earlier).
And then when we also specified module name like from module.some_submodule python looks for module in path list or in current working directory.
Of course, this is never a good idea; script root can change via calls like os.chdir() then submodules inside module folder may get lost.
Therefore, the best practices for sibling import is using relative import inside module folder.
/module/importing_submodule.py
from .some_submodule import SomeClass
Making script that work in both way
To make submodule import it's siblings when running as main script, yet still work as submodule when imported by other script, then use try - except and look for ImportError.
For importing_submodule.py as an example:
/module/importing_submodule.py
try:
from .some_submodule import SomeClass
except ImportError:
# attempted relative import with no known parent package
# because this is running as main script, there's no parent package.
from some_submodule import SomeClass
Importing modules from parent directory is a bit more tricky.
Since submodule is now main script, relative import to parent level directory doesn't work.
So we need to add the parent directory to sys.path, when the script is running as main script.
/module/importing_submodule.py
try:
from .some_submodule import SomeClass
except ImportError:
# attempted relative import with no known parent package
# because this is running as main script, there's no parent package.
from some_submodule import SomeClass
# now since we don't have parent package, we just append the path.
from sys import path
import pathlib
path.append(pathlib.Path(__file__).parent.parent.as_posix())
print("Importing module_in_parent_dir from sys.path")
else:
print("Importing module_in_parent_dir from working directory")
# Now either case we have parent directory of `module_in_parent_dir`
# in working dir or path, we can import it
# might need to suppress false IDE warning this case.
# noinspection PyUnresolvedReferences
from module_in_parent_dir import SomeOtherClass
Output:
"C:\Program Files\Python310\python.exe" .../module/importing_module.py
you imported from module
Importing module_in_parent_dir from sys.path
Process finished with exit code 0
"C:\Program Files\Python310\python.exe" .../__main__.py
you imported from module
Importing module_in_parent_dir from working directory
Process finished with exit code 0
I am running a python project like this:
project
Test.py
COMMON.py
SYSTEM.py
PTEST1
Hello.py
when run the code "Test.py" it will show NameError, I am not sure why?
But if I replaced the "from SYSTEM import *" with "from COMMON import *" in Test.py and PTEST1/Hello.py, it works as expect.
#Test.py is like this:
from SYSTEM import *
myvalue.Hello.printf()
# COMMON.py is like this:
myvalue = lambda: None
from PTEST1.Hello import Hello
myvalue.Hello = Hello
# SYSTEM.py is like this:
from COMMON import *
#PTEST1/Hello.py
from SYSTEM import *
class Hello():
#staticmethod
def printf():
print("Hello1")
print(vars(myvalue))
I expect there is no "NameError" by not changing import code. BTW, my python is 3.6+
Good practice is to give filenames in lowercase.
It looks like you are creating a Python project under project/. Any directory needs to have a file __init__.py in every directory in order for it to be discovered in Python.
You then need to refer to modules by their full name (not relative naming).
So the directory structure should be:
project/
__init__.py
test.py
common.py
system.py
ptest1/
__init__.py
hello.py
Every time you refer to file you should give the full path.
# import everything from hello.py
from project.ptest1.hello import *
# import everything from common.py
from project.common import *
# import everything from system.py
from project.system import *
I've been trying to import some python classes which are defined in a child directory. The directory structure is as follows:
workspace/
__init__.py
main.py
checker/
__init__.py
baseChecker.py
gChecker.py
The baseChecker.py looks similar to:
import urllib
class BaseChecker(object):
# SOME METHODS HERE
The gChecker.py file:
import baseChecker # should import baseChecker.py
class GChecker(BaseChecker): # gives a TypeError: Error when calling the metaclass bases
# SOME METHODS WHICH USE URLLIB
And finally the main.py file:
import ?????
gChecker = GChecker()
gChecker.someStuff() # which uses urllib
My intention is to be able to run main.py file and call instantiate the classes under the checker/ directory. But I would like to avoid importing urllib from each file (if it is possible).
Note that both the __init__.py are empty files.
I have already tried calling from checker.gChecker import GChecker in main.py but a ImportError: No module named checker.gChecker shows.
In the posted code, in gChecker.py, you need to do
from baseChecker import BaseChecker
instead of import baseChecker
Otherwise you get
NameError: name 'BaseChecker' is not defined
Also with the mentioned folders structure you don't need checker module to be in the PYTHONPATH in order to be visible by main.py
Then in main.y you can do:
from checker import gChecker.GChecker
I have the following structure for my views directory.
views
|--__init__.py
|--a_management.py
|--b_management.py
|--c_management.py
|--d_management.py
|--e_management.py
and __init__.py starts with the following:
from a_management import *
from b_management import *
from c_management import *
from d_management import *
from e_management import *
.........
...etc...
.........
In each of the files (a_management.py,b_management.py...) I need to write the same code importing modules:
import sys,os
from django.shortcuts import render, redirect
from django.http import HttpResponseRedirect
.........
...etc...
.........
Is it possible to perform all the module imports only in __init__.py?
When I tried, it still seems like the imports are not found and I get errors like: NameError: global name 'sys' is not defined
If each management module needs access to sys, then they all must import it. No way around that. (And really, if they all need it, then they all should import it. It's not a bad thing.)
You could save a bit of typing by having __init__ import sys, os and whtever else is needed, and then each management module can do from __init__ import *, thus "inheriting" all the imported modules from __init__.
Well, except you can't do it this way, because __init__ already imports stuff from the management modules, so the above suggestion would result in circular imports, which are a no-no.
I don't know the specifics of your application, but I have to believe that there's a better way to organize your modules to avoid so much repeated importing, and especially so much use of import *. Generally you want to use that as little as possible.
What about using a shared script to do all the system imports?
BTW, I agree that import * is not the greatest of idea. It makes sense in my use of importer, but I am not so sure in your general setup. Also, you need to careful about circular imports.
So, my answer is specifically only geared towards I need to write the same code importing modules:, not towards whether your setup as a whole makes sense.
Proof of concept, importer is what you really care about.:
├── pack
│ ├── __init__.py
│ ├── importer.py
│ ├── mgmt_1.py
│ ├── mgmt_2.py
└── test.py
test.py
import pack
pack.foo_1()
pack.foo_2()
init.py
from mgmt_1 import *
from mgmt_2 import *
mgmt_1.py
from .importer import *
print "sys", sys
print "os", os
def foo_1():
print "foo_1"
mgmt_2.py:
from .importer import *
print "sys", sys
print "os", os
def foo_2():
print "foo_2", dir(sys)[0:5]
importer.py
import sys
import os
output:
sys <module 'sys' (built-in)>
os <module 'os' from '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc'>
sys <module 'sys' (built-in)>
os <module 'os' from '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc'>
foo_1
foo_2:sys ['__displayhook__', '__doc__', '__excepthook__', '__name__', '__package__']