I have 2 identical files:
a.py and b.py .
they both contain the line
from mypackage.utils import common
I also have a package:
mypackage/
__init__.py
mymodule.py
utils/
__init__.py
common.py
myutils1.py
b.py
data_classes/
__init__.py
command_file.py
myclass.py
I preliminarly do a
pip install mypackage
Then I run:
python a.py
which succeeds
and then I run
python b.py (note that b.py is a file within mypackage)
b.py fails with no module named utils
Any ideas?
(note: I am fairly new to stackoverflow, this is not really related to
Python modules import fails , although somehow a followup question, so if I am doing something wrong by asking on a new question please let me know :)
edit:
I install mypackage (and specifically the file b within mypackage) in:
$HOME\github\mypackage\mypackage\utils\b.py
I copy b.py to a.py in:
$HOME\github\mypackage
this works
I copy b.py to a.py in:
$HOME\github\
this doesn't work
so this somewhat explains this... although I still don't know how to fix it!
edit:
I edited setup.py from the following line:
packages=['mypackage'],
to
packages=['mypackage','mypackage.data_classes','mypackage.utils']
this seems to fix my problem: now everything works.
is there any good reason why I should not do this?
i have found a couple of solutions to this:
import common
or
import sys
sys.path.append("..\\..") # relative path to mypackage, check the slash if you're on linux
from mypackage.utils import common
some questions:
why are you moving b.py?
have you installed your package?
The issue is that Python doesn't know where to find your package in the first place. In the first explain, it looks at the current directory and is able to understand that it is a module, thanks to your __init__.py file. So it is able to search for your utils.
In the 2nd example, there is no __init__.py file in the current directory and so Python doesn't know where to search for the modules.
The PYTHONPATH variable tells Python the directories/modules to search for accessed methods/packages.
So in your case, if you can add your current directory to the PYTHONPATH like export PYTHONPATH=$(pwd) while in "$HOME\github\", that should help resolve your issue.
P.S: The OP wants to use the installed package instead of the local module. This solution is applicable when your local modules are not visible for the Python interpreter. For more details, check the comments below.
I edited setup.py from the following line:
packages=['mypackage'],
to
packages=['mypackage','mypackage.data_classes','mypackage.utils']
this seems to fix my problem: now everything works.
is there any good reason why I should not do this?
Related
I'm trying to run a simple project in python but I have always the same problem and it drives me crazy every single time.
My project structure looks like the following:
└── myproject
├── mypackage
├── __init__.py
├── a.py
├── b.py
With vscode I open up the myproject folder. When I click on run in the a.py file which looks like
# a.py
from mypackage import b
.
.
.
I always get ModuleNotFoundError: No module named 'mypackage'. What is confusing to me is that vscode itself sees it correctly and shows the text colored (means it detects the module correctly).
Edit:
I want to use myproject as a package so just import b won't help.
The solution I found is actually relatively easy. Just type python3 -m mypackage.a into the command line in the top myproject directory. Worked for me.
I know this has nothing to do with vscode but I was just confused why vscode seems to accept the import but the python interpreter didn't.
Since you've probably didn't install myproject (or mypackage, if it is top level Python package), Python cannot find it on the path. You have two options:
Install the package.
Use relative import import b. However, see here for implications.
I recommend to avoid approaches like adding mypackage path to system path (see here). Maybe this documentation/tutorial can help you to understand better, what happened and why the import you used is not working.
However, I do not know, how VS Code searches for files in project, so I cannot tell you, why it marks it like this...
according to what you mentioned, both file 'a.py' and 'b.py' are in the same directory and if both files are in the same director, you don't have to actually mention the name of the directory in which they are present.
To import the file 'b.py' in 'a.py', you can just simply type the code shown below to import 'b.py':
import b
This should work.
This is a follow up question for a question I asked not long ago. The answer is correct and works great, but when I tried to apply it to the main package it didn't work.
Lets say I have the following files structure:
a/
->b/
->c/
-> __init__.py
-> script1.py
-> script2.py
->d/
-> __init__.py
-> script3.py
(The __init__ files are just like in the answer I linked above).
And in script3.py I import script1.py like so: from b.c import script1.
It works when I run it in Pycharm, but when I clone the repository in Colab (all this code is in a GitHub repository) I get the error: ModuleNotFoundError: No module named 'b'
Which makes sense because my package is not in the sys.path variable.
And adding the folder a to the sys.path manually, after I cloned the repository, helps but it is not a real solution because I can't always do it (e.g. in unit-tests).
So my question is, how can I fix it¿ Adding the __init__ file to folder b didn't help.
Notes:
Folder a is the project's folder, so it is not part of the path.
My GitHub repository is private, so I am not sure if making it installable (by adding setup.py and all that) would be helpful, but its my first time so I am not sure.
In Colab I import script3 like so from a.b.d import script3. In this case I must specify folder a because, again, b is not in sys.path.
As I mentioned above, I can manually fix it in Colab but it doesn't really solves the issue, because if I want to run unit-tests for example it won't work.
You can backwardly add folder b to your path and then import only from c folder, like this:
from os.path import realpath, dirname
import sys
sys.path.append(dirname(dirname(realpath(__file__))))
from c import script1
custom/
scripts/
__init__.py
file1.py
utils/
__init__.py
utilFile1.py
utilFile2.py
utilFile3.py
I have the above file structure I'm struggling trying to figure out how to import modules in python. I have gone through stackoverflow posts that have the same question but I still couldn't figure out why I can't get it to work.
I am trying to import these modules in maya and when I run
from utils import utilFile1 I get
ImportError: cannot import name utilFile1.
Running from custom.scripts.utils import utilFile1 gives me this error ImportError: no module named custom.scripts.utils. However if I run import file1 it imports without any errors
I have appended custom/scripts to sys.path and when that didn't work, tried appending custom/scripts/utils as well, but that didn't work either. Based off some of the posts on stackoverflow, I saw some people suggesting to run "python -m" but I'm not sure if I should run this or where to execute it from.
I'm really at a loss as to why I can't get it to work at all.
you have make it a python package by following steps,
in your root dir create setup.py,
in your case
custom/
setup.py
scripts/
...
...
in setup.py
from setuptools import setup, find_packages
setup(
name='your_package_name',
description='bla bla bla',
version='0.0.1-dev',
install_requires=[
],
)
and then hit
pip install -e .
for more info refer to this docs
Please update the directory structure as below.
Python might not be considering custom as a module as it doesn't have __init__.py file,
custom/
__init__.py
scripts/
__init__.py
file1.py
utils/
__init__.py
utilFile1.py
utilFile2.py
utilFile3.py
Yash's method works, I believe that's what companies use to setup their environments. An alternative way is to add your scripts path as a PYTHONPATH in your maya.env file instead of using sys.path.append. I'm not sure why there's a difference or if it's coz I'm trying this on windows, but oddly enough it worked in my case.
I make a lot of little python packages to help with this that or the other. As such, I have a directory, say /packages where I like to keep all of my handy tools, and I put that on the python path $export PYTHONPATH=/packages.
Now suppose that I put an __init__.py into /packages/my_package. I can $python -c "import my_package" successfully. However, if I do the typical thing with my directory structure (since I still have docs and whatnot) as /packages/my_package/my_package/hello_world.py, then having an __init__.py in /packages/my_package/my_package is insufficient to be able to import some_code.py. I can add an __init__.py into my root /package/my_package, but then I have to import my_package.my_package.hello_world, which conflicts with the normal "installed" way of importing. How do I set up my root-level __init__.py so that imports "skip" the intervening "code" directory, and import my_package.some_code works as expected?
tl;dr
Given this directory structure:
packages/
my_package/
__init__.py
my_package/
__init__.py
hello_world.py
with the packages directory in my PYTHONPATH. How do I compose my __init__.pys so that import my_package.hello_world works?
You need to write import in every init.py file. You can try the below code, it might need some modifications as per your need. But overall this would be the format.
packages/
my_package/
__init__.py
**from .my_package import main**
my_package/
__init__.py
**from .hello_world import main**
hello_world.py
def main():
#Do something
Given the deduplicated directory structure:
packages/
my_package1/
__init__.py
my_package2/
__init__.py
hello_world.py
It is sufficient to add the following in the __init__.py file of my_package1:
from .my_package2 import hello_world
That's a relative import of the hello_world module in my_package2 into the parent package my_package1.
This solves my particular use-case, in a non-python way. Instead of putting /packages on the PYTHONPATH, I just wrote a script that puts all of its subdirectories on to the PYTHONPATH. This allows me to use all of the modules in the "normal" way, without having to install them, and without having the weird root __init__.py. It does not answer my question, however, and I feel like there must be a nice pythonic way to handle it.
Specifically, I put this at the end of my ~/.bashrc
LOCAL_PYTHON_PACKAGES="[my_package_directory]"
for d in $LOCAL_PYTHON_PACKAGES/*/ ; do
export PYTHONPATH=$PYTHONPATH:$d
done
Of course, this only works for packages for which the actual source code is one directory down. It would be better to handle this in a package-specific format.
I'm trying to keep a data science project well-organized so I've created a directory inside my src directory called utils that contains a file called helpers.py, which contains some helper functions that will be used in many scripts. What is the best practice for how I should import func_name from src/utils/helpers.py into a file in a totally different directory, such as src/processing/clean_data.py?
I see answers to this question, and I've implemented a solution that works, but this feels ugly:
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))
Am I doing this right? Do I need to add this to every script that wants to import func_name, like train_model.py?
My current project folder structure:
myproject
/notebooks
notebook.ipynb
/src
/processing
clean_data.py
/utils
helpers.py
/models
train_model.py
__init__.py
Example files:
# clean_data.py
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))
from src.utils.helpers import func_name
func_name()
# helpers.py
def func_name():
print('I'm a helper function.')
The correct way to do it is to use __init__.py, setup.py and the setuptools Python package:
myPackage/
myPackage/
__init__.py
setup.py
This link has all the steps.
First of all, let me describe you the differences between a Python module & a Python package so that both of us are on the same page. ✌
A module is a single .py file (or files) that are imported under one import and used. ✔
import aModuleName
# Here 'aModuleName' is just a regular .py file.
Whereas, a package is a collection of modules in directories that give a package hierarchy. A package contains a distinct __init__.py file. ✔
from aPackageName import aModuleName
# Here 'aPackageName` is a folder with a `__init__.py` file
# and 'aModuleName', which is just a regular .py file.
Therefore, when we have a project directory named proj-dir of the following structure ⤵
proj-dir
--|--__init__.py
--package1
--|--__init__.py
--|--module1.py
--package2
--|--__init__.py
--|--module2.py
🔎 Notice that I've also added an empty __init__.py into the proj-dir itself which makes it a package too.
👍 Now, if you want to import any python object from module2 of package2 into module1 of package1, then the import statement in the file module1.py would be
from package2.module2 import object2
# if you were to import the entire module2 then,
from package2 import module2
I hope this simple explanation clarifies your doubts on Python imports' mechanism and solves the problem. If not then do comment here. 😊
First of all let me clarify you that importing an entire module, if you are going to use a part of it, then is not a good idea. Instead of that you can use from to import specific function under a library/package. By doing this, you make your program efficient in terms of memory and performance.
To know more refer these:
'import module' or 'from module import'
difference between import and from
Net let us look into the solution.
Before starting off with the solution, let me clarify you the use of __init__.py file. It just tells the python interpreter that the *.py files present there are importable which means they are modules and are/maybe a part of a package.
So, If you have N no of sub directories you have to put __init__.py file in all those sub directories such that they can also be imported. Inside __init__.py file you can also add some additional information like which path should be included, default functions,variables,scope,..etc. To know about these just google about __init__.py file or take some python library and go through the same __init__.py file to know about it. (Here lies the solution)
More Info:
modules
Be pythonic
So as stated by #Sushant Chaudhary your project structure should be like
proj-dir
--|--__init__.py
--package1
--|--__init__.py
--|--module1.py
--package2
--|--__init__.py
--|--module2.py
So now, If I put __init__.py file under my directory like above, Will
it be importable and work fine?
yes and no.
Yes :
If you are importing the modules within that project/package directory.
for example in your case
you are importing package1.module1 in pakage2.module2 as from package1 import module1.
Here you have to import the base dir inside the sub modules, Why? the project will run fine if you are running the module from the same place. i.e: inside package2 as python module2.py, But will throw ModuleNotFoundError If you run the module from some other directory. i.e: any other path except under package2 for example under proj-dir as python package2/module2.py. This is what happening in your case. You are running the module from project-dir.
So How to fix this?
1- You have to append basedir path to system path in module2.py as
from sys import path
dir_path = "/absolute/path/to/proj-dir"
sys.path.insert(0, dir_path)
So that module2 will be able to find package1 (and module1 inside it).
2- You have to add all the sub module paths in __init__.py file under proj-dir.
For example:
#__init__.py under lxml
# this is a package
def get_include():
"""
Returns a list of header include paths (for lxml itself, libxml2
and libxslt) needed to compile C code against lxml if it was built
with statically linked libraries.
"""
import os
lxml_path = __path__[0]
include_path = os.path.join(lxml_path, 'includes')
includes = [include_path, lxml_path]
for name in os.listdir(include_path):
path = os.path.join(include_path, name)
if os.path.isdir(path):
includes.append(path)
return includes
This is the __init__.py file of lxml (a python library for parsing html,xml data). You can refer any __init__.py file under any python libraries having sub modules.ex (os,sys). Here I've mentioned lxml because I thought it will be easy for you to understand. You can even check __init__.py file under other libraries/packages. Each will have it's own way of defining the path for submodules.
No
If you are trying to import modules outside the directory. Then you have to export the module path such that other modules can find them into environment variables. This can be done directly by appending absolute path of the base dir to PYTHONPATH or to PATH.
To know more:
PATH variables in OS
PYTHONPATH variable
So to solve your problem, include the paths to all the sub modules in __init__.py file under proj-dir and add the /absolute/path/to/proj-dir either to PYTHONPATH or PATH.
Hope the answer explains you about usage of __init__.py and solves your problem.
On Linux, you can just add the path to the parent folder of your src directory to ~/.local/lib/python3.6/site-packages/my_modules.pth. See
Using .pth files. You can then import modules in src from anywhere on your system.
NB1: Replace python3.6 by any version of Python you want to use.
NB2: If you use Python2.7 (don't know for other versions), you will need to create __init__.py (empty) files in src/ and src/utils.
NB3: Any name.pth file is ok for my_modules.pth.
Yes, you can only import code from installed packages or from files in you working directory or subdirectories.
the way I see it, your problem would be solved if you would have your module or package installed, like an yother package one installs and then imports (numpy, xml, json etc.)
I also have a package I constantly use in all my projects, ulitilies, and I know it's a pain with the importing.
here is a description on how to How to package a python application to make it pip-installable:
https://marthall.github.io/blog/how-to-package-a-python-app/
Navigate to your python installation folder
Navigate to lib
Navigate to site-packages
Make a new file called any_thing_you_want.pth
Type .../src/utils/helpers.py inside that file with your favorite text editor
Note: the ellipsis before scr/utils/helpers.py will look something like: C:/Users/blahblahblah/python_folders/scr... <- YOU DO NEED THIS!
This is a cheap way out but it keeps code clean, and is the least complicated. The downside is, for every folder your modules are in, example.pth will need them. Upside: works with Windows all the way up to Windows 10