My flask application is one big lists of files right now and I am trying to divide it up into two primary folders, app and spiders so that it looks like this:
- app
-- __init__.py
-- app.py
-- models.py
- spiders
-- __init__.py
-- scrape.py
Procfile
requirements.txt
The problem I am having is when trying to import my models into the scrape.py file.
When I try to import it absolutely with:
from app.models import Rate, Hotel, Location I get the error 'no module named app'.
When I try to import relatively:
from ..app.models import Rate, Hotel, Location I get the error SystemError: Parent module '' not loaded, cannot perform relative import.
What can I do to fix this?
This should work:
import sys
from os.path import dirname, abspath, sep
sys.path.append(abspath(dirname(__file__) + sep + ".."))
import app.models as models
Note: This is really an ugly solution as it messes up your import block in the file. I suggest to add a main code file in the root folder like main.py and import spiders and app modules in there. You should not spread your flow controllers among files in sub folders
Related
I've the following structure in my simple python project:
MainFolder
|
├───authentication
│ └───apikey.py
| └───tokengenerator.py
├───Functions
│ └───generatedata.py
The tokengenerator.py module produces Token variables and I need to call it in generatedata.pymodule and I used the following code line for this purpose:
from authentication.tokengenerator import Token
but it returns the error below:
Exception has occurred: ModuleNotFoundError
No module named 'authentication'
Would you please advise ?
from this article, you can add the path below the Functions folder to the searchpath for modules by adding .. (combined with the scriptpath)
import os
import sys
script_dir = os.path.dirname( __file__ )
mymodule_dir = os.path.join( script_dir, '..')
sys.path.append( mymodule_dir )
from authentication.tokengenerator import Token
token = Token()
Consider adding an empty __init__.py file in the authentication and Functions folder.
Additionally, you may need to use:
import sys
import os
sys.path.append(os.path.abspath(os.getcwd()))
before importing the modules.
I have created a package with the following file structure:
- package
- __init__.py
- load.py
- train.py
- test.py
My __init__.py file is simply an import of classes for these files:
from package.load import Load
from package.train import Train
from package.test import Test
Most of the time, I want to load all three, however on occasion I only want to load one of these classes specifically. For example in an ad hoc script (outside of the package) I want to be able to call only the Load class like so:
from package import Load
While all of the above works in this design, I have an issue where dependencies from train/test are also loaded when I import Load like the above. How can I setup the __init__.py file such that I can make the same import call without getting the dependency to load from train/test?
Additional explanation:
Why I am doing this: I have an issue where I want some people to be able to use the Load class, which only uses base python, however the Train/Test files include specialized dependencies which users of just the Load class wont want to utilize or even install.
Here's a way to do something very close what you want. Instead of unconditionally importing all the package's classes in your __init__.py, you can define a function in it to explicitly import any of the ones desired (or all of them if none are specified).
__init__.py:
from pathlib import Path
import sys
print(f'In {Path(__file__).name}')
package_name = Path(__file__).parent.name
package_prefix = package_name + '.'
class_to_module_map = {'Load': 'load', 'Train': 'train', 'Test': 'test'}
def import_classes(*class_names):
namespace = sys._getframe(1).f_globals # Caller's globals.
if not class_names:
class_names = class_to_module_map.keys() # Import them all.
for class_name in class_names:
module = class_to_module_map[class_name]
temp = __import__(package_prefix+module, globals(), locals(), [class_name])
namespace[class_name] = getattr(temp, class_name) # Add to caller's namespace.
For testing purposes, here's what I put in the load.py script:
(I also put something similar in the other two modules in order to verify whether or not they were getting imported.)
load.py:
from pathlib import Path
print(f'In {Path(__file__).name}')
class Load: pass
And finally here's a example of using it to only import the Load class:
ad_hoc.py:
from my_package import import_classes
#from my_package import Load
import_classes('Load')
test = Load()
print(test)
Along with the output produced:
In __init__.py
In load.py
<my_package.load.Load object at 0x001FE4A8>
Inside a folder oranges\, this is our __init__.py file:
__all__ = []
from pathlib import Path
from importlib import import_module
from sys import modules
package = modules[__name__]
initfile = Path(__file__)
for entry in initfile.parent.iterdir():
is_file = entry.is_file()
is_pyfile = entry.name.endswith('.py')
is_not_initpy = (entry != initfile)
if is_file and is_pyfile and is_not_initpy:
module_name = entry.name.removesuffix('.py')
module_path = __name__ + '.' + module_name
module = import_module(module_path)
setattr(package, module_name, module)
__all__.append(module_name)
When we do from oranges import *, the code inside oranges\__init__.py cycles through the *.py files inside oranges\ (except __init__.py), and for each .py file does the following:
imports the .py file as a module into the variable module using Python's importlib.import_module
sets the module as a variable within the __init__.py file (or, more correctly, within the oranges package) using Python's setattr
finally, appends the module to the __all__ list
I've tried too many solutions that are provided here on SO but wasn't able to make this work.
here is directory structure.
|project
|------->folder1
|-------------->file1.py #file1.py contains class File1
|-------------->file2.py #file2.py contains class File2
|------->folder2
|-------------->another_file1.py #another_file1.py contains class SomeClass1
|-------------->another_file2.py #another_file2.py contains class SomeClass2
|------->Runner
|-------------->logic.py
I need to create instance of all the classes here in logic.py
I created __init__.py in all folders, and one in the main project.
__init__.py # folder1
from . import file1
from . import file2
__init__.py # folder2
from . import another_file1
from . import another_file2
__init__.py # project
from . import folder1
from . import folder2
I'm unable to import any of stuff inside Runner.
I get following errors.
No module named file1.
No module named file2.
SystemError: Parent module '' not loaded, cannot perform relative import
However I can make it to work by this tweak.
import sys
sys.path.append('../')
in my files inside Runner directory. But I don't want to do this in my every file.
P.S. using Python 3.5
Anyone coming across the same issue, can do something like this.
1) create new project e.g: StreamingApplication.
|root --StreamingApplication
|-------> utils
|-------------->helper.py
|-------------->storage_handler.py
|------->config
|-------------->config.py
|-------------->hook.py
|------->app
|-------------->application.py
So, you can also create __init__.py file if you want to mark folder as python package.
add project root path to PYTHONPATH system variable.
in ~/.bashrc or /etc/profile
export PYTHONPATH=/path/to/StreamingApplication
source file with source ~/.bashrc or source /etc/profile
then in application.py you can do something like this.
from utils.helper import write_file #write_file is function.
from utils.storage_handler import StorageHandler # StorageHandler is class
import config as conf
class App:
def __init__(self):
self.executors = conf.executors # access executors variable from config
All set.
There are multiple threads about getting the current Python's script directory, for example:
import os
dir = os.path.dirname(os.path.abspath(__file__))
The question is, what should I do if I want to add this function into some utility file, while I want the returned value to be the directory of the calling file, in this case, the file executed by the user?
What would happen here?
// utils.py
def get_script_dir():
import os
return os.path.dirname(os.path.abspath(__file__))
// main_app.py
from utils import get_script_dir
print(get_script_dir())
// user's shell
python c:\path\to\somewhere\main_app.py
Will it print the directory of main_app.py or the directory of utils.py? What is an elegant solution for placing the function in the utils file while getting the directory of the file that was actually executed by the user?
Please try following and let me know whether it's exactly what you want:
# utils.py
def get_script_dir():
import sys
import os
return os.path.dirname(sys.modules['__main__'].__file__)
# main_app.py
from utils import get_script_dir
print(get_script_dir())
Supposing that your project directory tree is like the following:
Python/
main.py
modules/
utils.py
__init__.py
Being Python/modules/utils.py:
import os
get_script_dir = lambda file: os.path.dirname(os.path.abspath(file))
and Python/main.py:
from modules import utils
import os
script_dir = utils.get_script_dir(__file__)
print("[i] Currently executing file {} located at {}".format(os.path.basename(__file__), script_dir))
Executing Python/main.py is going to output something like:
[i] Currently executing file main.py located at C:\Users\BlackVirusScript\Desktop\Python
I have this problem and do not know how to solve it efficiently.
I've this file structure
THE NUMBER OF FOLDERS NOR THE NAMES ARE GIVEN, IT'S ALL UNKNOWN
app/
__init__.py
modules/
__init__.py
ModuleA/
__init__.py
file.py
otherfile.py
config.ini
ModuleB/
__init__.py
file.py
otherfile.py
config.ini
ModuleC/
__init__.py
file.py
otherfile.py
config.ini
**arbitrary number of modules with same structure*
As you can notiche, app is the main package of my app, but I need an efficient way to import the mods folder and its' content
* My actual solution *
from app import modules ad mods
def load_modules_from_packages(self, pkg = mods):
pkgname = pkg.__name__
pkgpath = dirname(pkg.__file__)
for loader,name,ispkg in pkgutil.walk_packages(pkg.__path__, pkgname+'.'):
if ispkg is True:
__import__(name,globals(),locals(),[],0)
elif ispkg is False:
__import__(name,globals(),locals(),[],0)
This works since pkgutil iterate the structure with the dot notation for names, so import works well.
But now I want load infos in the config file if I am in one of the somemodule folder(the one with own init.py and config.ini
I want to do this to recreate the structure of module package and output it in a JSON rapresentation for another thing
* my other solution does not works*
def load_modules_from_packages(directory)
dir_path = dirname(directory.__file__)
dir_name = directory.__name__
for filename in glob.glob(dir_path + '/**/*.ini', recursive=True):
plugin = {}
plugin['name'] = filename.split('/')[-2]
plugin['path'] = dirname(filename)
plugin['config_file'] = filename
for pyname in glob.glob(dirname(filename)+ '/**/*.py', recursive=True):
importlib.import_module(pyname)
I cant use the solution posted in this thread
How to import a module given the full path?
since I do not know the module name, ad pointed without solutions in the comment.
spec = importlib.util.spec_from_file_location('what.ever', 'foo.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
I know 'foo.py' but I cant figure out 'what.ever' like pkgutil.walk_package does.
In fact the modules imported in this way have the package and name entry wrong. With this approach I cant figure out where I am in the file structure to create modules dictionary and the relative modules (for the JSON output)
Any help?