So, I was having the simple error,
"No module named "data_utils"
when trying to import it into a python program. So I thought it must not have downloaded and spent like 20 mins trying to ensure a proper download. Turns out it was fine all along and the data_utils.py file is in the utils folder.
I'm really stuck because I see it right there, yet it simply won't import. I looked for a .bin after the __init__.py files but it seems like they are fine. Any help would be greatly appreciated. Thank you!
Deducing from your comment the answer would be:
The files have to be in the same directory /and some sub-directory for an import like import data_utils to work.
There are some way to get around this but to start try to keep it simple at first.
for example:
given a directory structure like this:
| --main.py
| --data_utils.py
| --train.py
and suppose you have a function remove_punctuation in data_utils:
you could use import that with:
from data_utils import remove_punctuation
or you could import all the functions (and or classes in data_utils) with:
from data_utils import *
or you can import data_utils with
import data_utils
# use remove_punctuation
data_utils.remove punctuation
The directory structure could also be:
|--main.py
|--bar
|--foo
|--src
|--train.py
and you could import remove_punctuation using
the same semantics as above qualifying the directory using dot notation:
from bar.foo import foo
EDIT: would module import like scipy have to be in the same directory?
Short answer
no
Long answer:
When you install scipy and numpy etc (packages you install from pip or using the sudo) they add themselves (their location) to the PYTHONPATH so you don't have to have them in the same directory as your project code.
Modules that you want to use globally must be added to your PYTHONPATH. Python searches for modules (roughly) in the directory and sub-directory of the file it is executing and in the PYTHONPATH.
if you want global import of your own modules
I.e, if you want to use data_utils.py everywhere you could do assuming you are using bash on linux and assuming you have data_utils.py in a directory named data-utils:
add an __init__.py to data-utils, so your directory structure would look like this:
|--data-utils
|--__init__.py
|--data_utils.py
then add this line to your ~/.bashrc:
export PYTYHONPATH=$PYTHONPATH:/path/to/data-utils
data-utils (and by extension data_utils.py) is permanently added to the PYTHONPATH and can be imported by any project code.
Related
This question already has answers here:
Relative imports for the billionth time
(12 answers)
Closed last year.
So basicly I want to acces a created module from a folder on the parent directory from the folder I am
Currently I'm at Twitter.py and I want to access /utils/magic_eden.py
On the __init__.py file I've got:
from .magic_eden import MagicEden
from .open_sea import OpenSea
from .tools import Tools
Now inside the Twitter.py file im trying to import these classes of the module by doing:
from utils import MagicEden
But im getting ModuleNotFoundError: No module named 'utils'.
I've tried so many sort of things but none worked. What should I do?
(btw if I execute the __init__.py file I get ImportError: attempted relative import with no known parent package)
From what I see, it seems that utils is a utility package logically separate from the code in Twitter.py. Given this, what you want is to cause your utils package to be on your Python search path (sys.path) so that you can import that package as a separate entity (no relative paths). If you don't want to configure your environment to place utils on the Python search path, you can do it in your Twitter.py file. Here's how that looks:
import os
import sys
here = os.path.dirname(__file__)
sys.path.append(os.path.join(here, '..'))
import utils
utils.MagicEden.hello()
utils.OpenSea.hello()
utils.Tools.hello()
The first line of code computes the full path to the directory containing Twitter.py. The second line then computes the path to the parent directory of that directory and adds that path to sys.path, the Python search path. After doing this, import utils will work, giving you access to that package and everything imported in that package's __init__.py file.
I created three small files for magic_eden.py, open_sea.py, and tools.py, each containing something that looks like this:
class MagicEden:
#staticmethod
def hello():
print("Hello from MagicEden!")
I can then run Twitter.py, and I get the following result with no additional configuration:
Hello from MagicEden!
Hello from OpenSea!
Hello from Tools!
There's nothing wrong with using the above solution during early development. But you will likely at some point want to remove the code that is hacking sys.path and instead install your module in a more official way. There's a way to do this from the start so that you never have to change code on either side when you want to install your module in the official way...
What I do in situations like this is create a setup.py file in my package that lets me build it as an installable package. There are many docs and tutorials on the net that explain how to do this. Here's just one of them: https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html
Once you've done this, what you can do is install your package in "development mode" (pip install -e <package file>). What this does is install the package so that your system can find it (adds it to sys.path as a package), but installs links to the original sources rather than installing copies of them. In this mode, when you make changes to that package's source files, the changes take immediate effect in the installed module. So now you have the best of both worlds. Your package is installed in the official way such that you don't need to do anything special in the code that imports that package (no need to modify sys.path), and yet you can still make changes directly to the sources for that package and not have to keep reinstalling the package to see those changes take affect.
When you're done messing with a package in this way, you can reinstall it in the regular mode (pip install <package file>) and isolate changes to the sources for that package from uses of the package elsewhere on your system. At this point, you need to rebuild and reinstall the package to see any changes you've made to its sources.
You're attempting to import from a "sibling" folder.
To do this, you'll need to add __init__.py to your Twitter/ and parent src/ folders.
Then you'll need to import the path as Twitter.py doesn't know about any parent structure in this setup. You can then import from the utils module which is in the path.
import sys
sys.path.append('..')
from utils.magic_eden import MagicEden
assuming MagicEden is a class in your magic_eden.py file.
I have the following directory structure:
base_folder
methods_folder
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
class_methods.py imports the other files in the same directory, like this:
from method_1 import method_1
from method_2 import method_2
.
.
.
from method_n import method_n
(obs: these methods files has a method with its own file names inside them)
If I run class_methods.py by myself, no problem. But if I try to run top_class.py, which imports class_methods.py, I get the error no module named method_1
So, when executing top_class.py, it is not seeing the files in methods_folder/. Why?
the correct import inside top_class.py would be from methods_folder.method_n import method_n. This is because you are treating methods_folder as a package. If you are running a version of Python that is before 3.3 you must also unclude __init__.py file inside the methods_folder in order to turn it into a package.
Files only have direct access to things they import. Say we have a.py which imports b.py, and b.py imports c.py. When running within a.py functions in b.py that use c.py, this will work fine, because a has access to b, and b has access to c. This does not mean, however, that the imports chain (as in C++) and you can use functions from c in a. You will get an error, because a can only see the contents of b, which it imported.
So if you want to use all your method_i.py files from within top_class.py, you need to import them directly in the same file.
Edit: You also have some other issues. To import other files in a subfolder, you would need to, inside top_class.py, call import methods_folder.method_i. To import something in the same directory, just use import method_i. Since you have a method of the same name in each file, what you have works fine in class_methods.py You also need to create an empty file called __init__.py in any folder containing python files you intend to import to/from which lets python know it's allowed to look there.
You can create an importable package in python in one of two ways. The first way is what you are doing: you create a file called my_package.py and import it with import my_package. This is commonly used for simpler packages that don't need to be further broken up. For this to work, your .py file has to be on the PYTHONPATH which is an environment variable that tells python where to look for packages. If not defined, there are some default places that python will use to look for packages. One of these default locations is the current working directory, which is why your first set of imports works.
In theory you should be able to run the second piece of code from the same location (python ../top_class.py) and use the same import style, but I assume you are changing directories to run that file. This means your files are no longer in the current working directory and no longer findable by python.
One way to get your code to work using the existing style would be to define PYTHONPATH with the location of your methodX.py files. You will typically add to the python search path like this:
PYTHONPATH=$PYTHONPATH:./methods_folder python top_class.py
This tells python to look in methods_folder, in addition to the standard places, when you try to import something. Playing with the PYTHONPATH gets a little annoying after a while, so I actually prefer the next approach.
A second way to create a package is by creating a folder with an __init__.py file inside. This tells python that you want it to treat that directory as a package. This is the preferred style for more complicated pieces of code that might benefit from organization across multiple files. For your example, you could organize your code in the following way:
base_folder
methods_folder
__init__.py
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
And then your import in top_class.py would look like this:
from methods_folder.method1 import method1
from methods_folder.method2 import method2
from methods_folder.method3 import method3
This has the effect of creating a top level methods_folder package with modules method1, method2, etc. Because methods_folder is in the same directory as the one you are running top_class.py from, python picks that up as a package using the default PYTHONPATH and lets you import from within it.
I assume you are running them from their respective directories? Unless they are installed in your Python path (I'm going to assume they are not), then Python will by default look for imports in your current directory. So when you run class_methods.py from its directory, then it can find methods_1.py in order to satisfy from methods_1 import method_1. But when you execute top_class.py, it looks for methods_1.py or methods_1/__init__.py, neither of which it fines from that directory.
Assuming Python 3, you would need to use relative imports in class_methods.py.
# class_methods.py
from .methods_1 import method_1
from .methods_2 import method_2
This will let you run it from top_class.py. Unfortunately, you can't use relative imports when running a script in the same package, so you wouldn't be able to run class_methods.py directly.
Another option in that case is to keep the absolute imports in class_methods.py and add the methods folder to the path in top_class.py.
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'methods'))
import class_methods
from methods_1 import method
Of course editing sys.path is fine for small scripts and standalone things. But if this grows past being a script, you'll need to be careful about doing it, and you'll probably just want to come up with another solution. Likely, the best thing is to create a package that you install (you can still do this from your source directory while you are developing) and then you can import the same way from anywhere.
The recommended way of running a python script is using the -m switch from the parent of your root package - so in your case:
$ cd base_folder
$ python -m top_class
Python will automatically add the base_folder to its sys.path and you don't need to do any sys.path/PYTHOPATH hacks that are just this - hacks that bloat the code with boilerplate and will blow when least expected.
Now to run the class_methods the correct way is also
$ cd base_folder
$ python -m methods_folder.class_methods
but then the imports in class_methods should be modified to either absolute:
from methods_folder.method1 import method1
...
or relative:
from .method1 import method1
...
I understand there are many SO questions on relative imports. I will document the extent I have tried the solutions therein, but I am still unable to solve my problem.
I have the following directory structure. It's not mine by design but I'm game for modifying things as necessary (forked repo).
exp
main_Exp.py
kaffe
__init__.py
tensorflow
__init__.py
network_shape.py
ResNet
__init__.py (*)
ThreeDMM_shape.py
To run the model in this repo, I am to use
python main_Exp.py input_file.csv
Inside main_Exp.py:
sys.path.append('./kaffe')
sys.path.append('./ResNet')
from ThreeDMM_shape import ResNet_101 as resnet101_shape
from ThreeDMM_expr import ResNet_101 as resnet101_expr
Inside ResNet/ThreeDMM_shape.py:
sys.path.append('/home/usc/Desktop/Research/FG18/ExpNet_Code_Release/kaffe/tensorflow')
from network_shape import Network_Shape
Ok, so obviously I need to change this hard-coded absolute path. I'd like to do it the right way and not use my own specific path that I happened to install these files to.
So I try
from ..kaffee.tensorflow import Network_Shape
>>> ValueError: Attempted relative import in non-package
(1) I added __init__.py file in the ResNet folder (shown with the (*))
(2) I tried running the file as a module: python -m main_Exp input_file.csv
(3) I also tried adding __init__.py to the top level folder (exp), though I believe doing so is nonsense.
(4) Given that the first import was happening using the kaffe path that was appended to sys.path, I tried changing import to from .tensorflow.network_shape import Network_Shape
Same error after all steps.
So I'm not understanding the rules around relative imports and how to reference files in a sane way. I would really appreciate a pointer that helps me understand how to do this, and how to think about such imports in general!
exp is indeed not a package and won't be made to a package even if you add an init file to it, cause that won't magically add it to the paths python looks for packages. If you do add the __init__.py you can then run as python -m exp.main_Exp input_file.csv (from ../exp). This would make python recognize exp as a package and kaffe/ResNet as subpackages. You would need to change imports to from ResNet.ThreeDMM_shape import ResNet_101 as resnet101_shape etc.
Edit in response to comment:
Running from the parent dir using the m switch is the recommended way of running the script see for instance https://stackoverflow.com/a/23540051/281545 (that's for python 3 however it should still apply). If you want to avoid it (it would break hardcoded relative paths for one) you should add exp to sys path (once maybe is enough) then change the imports to absolute ones as in:
# main_Exp.py
sys.path.append(os.path.abspath(os.path.dirname(__file__))) # the exp folder
from ResNet.ThreeDMM_shape import ResNet_101 as resnet101_shape
from ResNet.ThreeDMM_expr import ResNet_101 as resnet101_expr
# ResNet/ThreeDMM_shape.py
from kaffee.tensorflow import Network_Shape
I have trouble importing package.
My file structure is like this:
filelib/
__init__.py
converters/
__init__.py
cmp2locus.py
modelmaker/
__init__.py
command_file.py
In module command_file.py I have a class named CommandFile which i want to call in the cmp2locus.py module.
I have tried the following in cmp2locus.py module:
import filelib.modelmaker.command_file
import modelmaker.command_file
from filelib.modelmaker.command_file import CommandFile
All these options return ImportError: No modules named ...
Appreciate any hint on solving this. I do not understand why this import does not work.
To perform these imports you have 3 options, I'll list them in the order I'd prefer. (For all of these options I will be assuming python 3)
Relative imports
Your file structure looks like a proper package file structure so this should work however anyone else trying this option should note that it requires you to be in a package; this won't work for some random script.
You'll also need to run the script doing the importing from outside the package, for example by importing it and running it from there rather than just running the cmp2locus.py script directly
Then you'll need to change your imports to be relative by using ..
So:
import filelib.modelmaker.command_file
becomes
from ..modelmaker import command_file
The .. refers to the parent folder (like the hidden file in file systems).
Also note you have to use the from import syntax because names starting with .. aren't valid identifiers in python. However you can of course import it as whatever you'd like using from import as.
See also the PEP
Absolute imports
If you place your package in site-packages (the directories returned by site.getsitepackages()) you will be able to use the format of imports that you were trying to use in the question. Note that this requires any users of your package to install it there too so this isn't ideal (although they probably would, relying on it is bad).
Modifying the python path
As Meera answered you can also directly modify the python path by using sys.
I dislike this option personally as it feels very 'hacky' but I've been told it can be useful as it gives you precise control of what you can import.
To import from another folder, you have to append that path of the folder to sys.path:
import sys
sys.path.append('path/filelib/modelmaker')
import command_file
I have some code in a project which I'd like to reuse in another project. What do I need to do (in both folders) so that I can do this?
The directory structure is something like:
Foo
Project1
file1.py
file2.py
Bar
Project2
fileX.py
fileY.py
I want to use functions from file1.py and file2.py in fileX.py and fileY.py.
Ideally both projects will be an installable python package, replete with __init__.py and setup.py. They could then be installed with python setup.py install or similar.
If that is not possible, don't use execfile()! Manipulate the PYTHONPATH to add Foo so that import Project1.file1 works.
For example, from Project2/fileX.py:
from os import path
import sys
sys.path.append(path.abspath('../Foo'))
from Project1.file1 import something
However, the real answer is to make each a discrete installable package.
There's a lot going on here. you should read about python packages and module management http://docs.python.org/2/tutorial/modules.html#packages but the basic idea is that fileX needs to know where file1 and file2 are in order to use them.
To turn a folder into a package, it just needs to contain an __init__.py file. What I would suggest you do is (in a terminal)
$ touch Foo/__init__.py
$ touch Foo/Project1/__init__.py
(assuming you're using unix/linux).
Then somehow, fileX needs to know where the Foo package is. You can call sys.path.append(PATH) where PATH is the location of Foo.
finally inside fileX.py you'd have
import sys
sys.path.append(PATH) #replace PATH with the path to Foo
from Foo.Project1 import file1
#use its functions
file1.function_name(argument)
if you really want to just say function_name without the preceeding file1. you can import all of its functions by saying from Foo.Project1.file1 import * however please note that from module import * is highly frowned upon as it mixes names and make code less readable and understandable
You may want to make a module out of it and then import whatever you need from it.
You take the code you want to use in both projects, and you put it into a module, which you extract into a third separate project. That project you make into a package, which you can work on separately. You then release version of it, and reuse them in your other projects.
It is important that you have versions that you "release" so that you can keep track of which version of the module each project uses. Otherwise you will end up in a situation where an old project stops working because you have made incompatible changes to the common module.
If it's generically usable not only for you but for others, consider uploading it to PyPI.
I think you can add Foo to the current Python path using os.path.join() or os.path.append() and do from import Project1.file1 import function_name.