Develop Reference

Why importing in the imported modules works differently?

If I have
a.py
b.py
In b.py I can import a
But if I have
c.py
m
a.py
b.py
and in c.py do import m.b, suddenly in b.py I get ModuleNotFoundError: No module named 'a'
What's the problem? I don't see how the second case is any different from the first
So... the modules are searched in the directory of the module that was started initially. I just don't understand the reasoning.
I'm not asking how to fix the problem. But rather asking why there's a problem in the first place...
(tested on Python 3.8.8)

It's based what file you run python.
If you run with python c.py
file c.py use:
from m import a, b
file b.py also use:
from m import a
If you go python b.py you just need import a to use in b.py
so depend what file you run it's will be the parent for relative import.

Let's start with why your first one works.
Assuming you have a file structure such that:
c.py
m - |
|
__init__.py
a.py
b.py
Let's say your current directory is within the m folder and you run the command.
python b.py
If within your b.py you place something like:
import a
import sys
print(sys.path)
it will work just fine. But let's take a look at the output from print(sys.path).
On windows, this is will look something like ['C:\\path\\to\\myprogram\\m', '<path to python installation>', etc.]
The important thing is that the m folder is on your sys.path and that is how import a gets resolved.
However if we go up one level and run:
python c.py
you should immediately notice that m is no longer on your sys.path and instead is replaced by 'C:\\path\\to\\myprogram'.
That is why it fails. Python automatically includes your current working directory in sys.path and changing out of it means it no longer knows where to look for the import.
This is an example of an absolute import. You can manipulate where python looks for these imports by modifying sys.path to include the path of the file you want to import.
sys.path.append('C:\\path\\to\\myprogram\\m')
But there's a better way to do it. If m is a package or sub-package (includes an __init__.py) you can use a relative import.
from . import a
However, there is a small caveat to this.
You can only use relative imports when the file using them is being run as a module or package.
So running them directly as a top level script
python b.py
will produce
ImportError: attempted relative import with no known parent package
Python luckily has the ability to run a script as a module built in.
if you cd one level up so as to encapsulate your m package, you can run
python -m m.b
and have it run just fine.
Additionally, since b.py is being treated as a module in c.py because of the import, the relative import will work in this case as well.
python c.py

Path appended but python does not find module

I have the following structure:
~/git/
~/git/folder1
~/git/folder2
in ~/git/folder1 I have main.py, which imports doing the following:
import folder2.future_data as future_data
which throws the following error:
import folder2.future_data as f_d
ImportError: No module named folder2.future_data
Despite my $PATH containing
user#mac-upload:~$ echo $PATH
/home/user/anaconda2/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/user/git/folder2
Why am I unable to import from folder2 despite it being in my path?
Am I missing something?

Try putting an empty __init__.py file in each directory (~/git, ~/git/folder1, and ~/git/folder2). Then do export PYTHONPATH=${HOME}/git:$PYTHONPATH (assuming bash shell).
This will also allow you to just set your PYTHONPATH once at the top level and be done with it. If you add more directories (modules) that you need to import, you can just keep adding __init__.py files to your structure (instead of having to constantly modify your PYTHONPATH every time your file/directory structure changes).

You can explicitly added the path inside the main.py script before you doing import
import sys
sys.path.append(r'~/git/folder2')
import future_data

Why can I import successfully without __init__.py?

What exactly is the use of __init__.py? Yes, I know this file makes a directory into an importable package. However, consider the following example:
project/
foo/
__init__.py
a.py
bar/
b.py
If I want to import a into b, I have to add following statement:
sys.path.append('/path_to_foo')
import foo.a
This will run successfully with or without __init__.py. However, if there is not an sys.path.append statement, a "no module" error will occur, with or without __init__.py. This makes it seem lik eonly the system path matters, and that __init__.py does not have any effect.
Why would this import work without __init__.py?

__init__.py has nothing to do with whether Python can find your package. You've run your code in such a way that your package isn't on the search path by default, but if you had run it differently or configured your PYTHONPATH differently, the sys.path.append would have been unnecessary.
__init__.py used to be necessary to create a package, and in most cases, you should still provide it. Since Python 3.3, though, a folder without an __init__.py can be considered part of an implicit namespace package, a feature for splitting a package across multiple directories.
During import processing, the import machinery will continue to
iterate over each directory in the parent path as it does in Python
3.2. While looking for a module or package named "foo", for each directory in the parent path:
If <directory>/foo/__init__.py is found, a regular package is imported and returned.
If not, but <directory>/foo.{py,pyc,so,pyd} is found, a module is imported and returned. The exact list of extension varies by platform
and whether the -O flag is specified. The list here is
representative.
If not, but <directory>/foo is found and is a directory, it is recorded and the scan continues with the next directory in the parent
path.
Otherwise the scan continues with the next directory in the parent path.
If the scan completes without returning a module or package, and at
least one directory was recorded, then a namespace package is created.

If you really want to avoid __init__.py for some reason, you don't sys.path. Rather, create a module object and set its __path__ to a list of directories.

if I want to import a into b, I have to add following statement:
No! You'd just say: import foo.a. All this is provided you run the entire package at once using python -m main.module where main.module is the entry point to your entire application. It imports all other modules, and the modules that import more modules will try to look for them from the root of this project. For instance, foo.bar.c will import as foo.bar.b
Then it seems that only the system path matters and init.py does not have any effect.
You need to modify sys.path only when you are importing modules from locations that are not in your project, or the places where python looks for libraries. __init__.py not only makes a folder look like a package, it also does a few more things like "export" objects to outside world (__all__)

When you import something it has to either:
Retrieve an already loaded module or
Load the module that was imported
When you do import foo and python finds a folder called foo in a folder on your sys.path then it will look in that folder for an __init__.py to be considered the top level module.
(Note that if the package is not on your sys.path then you would need to append it's location to be able to import it.)
If that is not present it will look for a __init__.pyc version possibly in the __pycache__ folder, if that is also missing then that folder foo is not considered a loadable python package. If no other options for foo are found then an ImportError is raised.
If you try deleting the __init__.pyc file as well you will see that the the initializer script for a package is indeed necessary.

How do I resolve sys.path differences in Python and IronPython

I am running into pathing issues with some scripts that I wrote to test parsers I've written. It would appear that Python (27 and 3) both do not act like IronPython when it comes to using the current working directory as part of sys.path. As a result my inner package pathings do not work.
This is my package layout.
MyPackage
__init__.py
_test
__init__.py
common.py
parsers
__init__.py
_test
my_test.py
I am attempting to call the scripts from within the MyPackage directory.
Command Line Statements:
python ./parsers/_test/my_test.py
ipy ./parsers/_test/my_test.py
The import statement located in my my_test.py file is.
from _test.common import TestClass
In the python scenario I get ONLY the MyPackage/parsers/_test directory appended to sys.path so as a result MyPackage cannot be found. In the IronPython scenario both the MyPackage/parsers/_test directory AND MyPackage/ is in sys.path. I could get the same bad reference if I called the test from within the _test directory (it would have no idea about MyPackage). Even if i consolidated the test files into the _test directory I would still have this pathing issue.
I should note, that I just tested and if i did something like
import sys
import os
sys.path.append(os.getcwd())
It will load correctly. But I would have to do this for every single file.
Is there anyway to fix this kind of pathing issue without doing the following.
A. Appending a relative pathing for sys.path in every file.
B. Appending PATH to include MyPackage (I do not want my package as part of the python "global" system pathing)
Thanks alot for any tips!

Two options spring to mind:
Use the environment variable PYTHONPATH and include .
Add it to your sys.path at the beginning of your program
import sys.path
sys.path.append('.')
If you need to import a relative path dynamically you can always do something like
import somemodule
import sys
dirname = os.path.dirname(os.path.abspath(somemodule.__file__))
sys.path.append(dirname)

Import a module from a relative path

How do I import a Python module given its relative path?
For example, if dirFoo contains Foo.py and dirBar, and dirBar contains Bar.py, how do I import Bar.py into Foo.py?
Here's a visual representation:
dirFoo\
Foo.py
dirBar\
Bar.py
Foo wishes to include Bar, but restructuring the folder hierarchy is not an option.

Assuming that both your directories are real Python packages (do have the __init__.py file inside them), here is a safe solution for inclusion of modules relatively to the location of the script.
I assume that you want to do this, because you need to include a set of modules with your script. I use this in production in several products and works in many special scenarios like: scripts called from another directory or executed with python execute instead of opening a new interpreter.
import os, sys, inspect
# realpath() will make your script run, even if you symlink it :)
cmd_folder = os.path.realpath(os.path.abspath(os.path.split(inspect.getfile( inspect.currentframe() ))[0]))
if cmd_folder not in sys.path:
sys.path.insert(0, cmd_folder)
# Use this if you want to include modules from a subfolder
cmd_subfolder = os.path.realpath(os.path.abspath(os.path.join(os.path.split(inspect.getfile( inspect.currentframe() ))[0],"subfolder")))
if cmd_subfolder not in sys.path:
sys.path.insert(0, cmd_subfolder)
# Info:
# cmd_folder = os.path.dirname(os.path.abspath(__file__)) # DO NOT USE __file__ !!!
# __file__ fails if the script is called in different ways on Windows.
# __file__ fails if someone does os.chdir() before.
# sys.argv[0] also fails, because it doesn't not always contains the path.
As a bonus, this approach does let you force Python to use your module instead of the ones installed on the system.
Warning! I don't really know what is happening when current module is inside an egg file. It probably fails too.

Be sure that dirBar has the __init__.py file -- this makes a directory into a Python package.

You could also add the subdirectory to your Python path so that it imports as a normal script.
import sys
sys.path.insert(0, <path to dirFoo>)
import Bar

import os
import sys
lib_path = os.path.abspath(os.path.join(__file__, '..', '..', '..', 'lib'))
sys.path.append(lib_path)
import mymodule

Just do simple things to import the .py file from a different folder.
Let's say you have a directory like:
lib/abc.py
Then just keep an empty file in lib folder as named
__init__.py
And then use
from lib.abc import <Your Module name>
Keep the __init__.py file in every folder of the hierarchy of the import module.

If you structure your project this way:
src\
__init__.py
main.py
dirFoo\
__init__.py
Foo.py
dirBar\
__init__.py
Bar.py
Then from Foo.py you should be able to do:
import dirFoo.Foo
Or:
from dirFoo.Foo import FooObject
Per Tom's comment, this does require that the src folder is accessible either via site_packages or your search path. Also, as he mentions, __init__.py is implicitly imported when you first import a module in that package/directory. Typically __init__.py is simply an empty file.

The easiest method is to use sys.path.append().
However, you may be also interested in the imp module.
It provides access to internal import functions.
# mod_name is the filename without the .py/.pyc extention
py_mod = imp.load_source(mod_name,filename_path) # Loads .py file
py_mod = imp.load_compiled(mod_name,filename_path) # Loads .pyc file
This can be used to load modules dynamically when you don't know a module's name.
I've used this in the past to create a plugin type interface to an application, where the user would write a script with application specific functions, and just drop thier script in a specific directory.
Also, these functions may be useful:
imp.find_module(name[, path])
imp.load_module(name, file, pathname, description)

This is the relevant PEP:
http://www.python.org/dev/peps/pep-0328/
In particular, presuming dirFoo is a directory up from dirBar...
In dirFoo\Foo.py:
from ..dirBar import Bar

The easiest way without any modification to your script is to set PYTHONPATH environment variable. Because sys.path is initialized from these locations:
The directory containing the input script (or the current
directory).
PYTHONPATH (a list of directory names, with the same
syntax as the shell variable PATH).
The installation-dependent default.
Just run:
export PYTHONPATH=/absolute/path/to/your/module
You sys.path will contains above path, as show below:
print sys.path
['', '/absolute/path/to/your/module', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/ubuntuone-client', '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel', '/usr/lib/python2.7/dist-packages/ubuntuone-couch', '/usr/lib/python2.7/dist-packages/ubuntuone-installer', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol']

In my opinion the best choice is to put __ init __.py in the folder and call the file with
from dirBar.Bar import *
It is not recommended to use sys.path.append() because something might gone wrong if you use the same file name as the existing python package. I haven't test that but that will be ambiguous.

The quick-and-dirty way for Linux users
If you are just tinkering around and don't care about deployment issues, you can use a symbolic link (assuming your filesystem supports it) to make the module or package directly visible in the folder of the requesting module.
ln -s (path)/module_name.py
or
ln -s (path)/package_name
Note: A "module" is any file with a .py extension and a "package" is any folder that contains the file __init__.py (which can be an empty file). From a usage standpoint, modules and packages are identical -- both expose their contained "definitions and statements" as requested via the import command.
See: http://docs.python.org/2/tutorial/modules.html

from .dirBar import Bar
instead of:
from dirBar import Bar
just in case there could be another dirBar installed and confuse a foo.py reader.

For this case to import Bar.py into Foo.py, first I'd turn these folders into Python packages like so:
dirFoo\
__init__.py
Foo.py
dirBar\
__init__.py
Bar.py
Then I would do it like this in Foo.py:
from .dirBar import Bar
If I wanted the namespacing to look like Bar.whatever, or
from . import dirBar
If I wanted the namespacing dirBar.Bar.whatever. This second case is useful if you have more modules under the dirBar package.

Add an __init__.py file:
dirFoo\
Foo.py
dirBar\
__init__.py
Bar.py
Then add this code to the start of Foo.py:
import sys
sys.path.append('dirBar')
import Bar

Relative sys.path example:
# /lib/my_module.py
# /src/test.py
if __name__ == '__main__' and __package__ is None:
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../lib')))
import my_module
Based on this answer.

Well, as you mention, usually you want to have access to a folder with your modules relative to where your main script is run, so you just import them.
Solution:
I have the script in D:/Books/MyBooks.py and some modules (like oldies.py). I need to import from subdirectory D:/Books/includes:
import sys,site
site.addsitedir(sys.path[0] + '\\includes')
print (sys.path) # Just verify it is there
import oldies
Place a print('done') in oldies.py, so you verify everything is going OK. This way always works because by the Python definition sys.path as initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.
If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.

Another solution would be to install the py-require package and then use the following in Foo.py
import require
Bar = require('./dirBar/Bar')

Simply you can use: from Desktop.filename import something
Example:
given that the file is name test.py in directory
Users/user/Desktop , and will import everthing.
the code:
from Desktop.test import *
But make sure you make an empty file called "__init__.py" in that directory

Here's a way to import a file from one level above, using the relative path.
Basically, just move the working directory up a level (or any relative location), add that to your path, then move the working directory back where it started.
#to import from one level above:
cwd = os.getcwd()
os.chdir("..")
below_path = os.getcwd()
sys.path.append(below_path)
os.chdir(cwd)

I'm not experienced about python, so if there is any wrong in my words, just tell me. If your file hierarchy arranged like this:
project\
module_1.py
module_2.py
module_1.py defines a function called func_1(), module_2.py:
from module_1 import func_1
def func_2():
func_1()
if __name__ == '__main__':
func_2()
and you run python module_2.py in cmd, it will do run what func_1() defines. That's usually how we import same hierarchy files. But when you write from .module_1 import func_1 in module_2.py, python interpreter will say No module named '__main__.module_1'; '__main__' is not a package. So to fix this, we just keep the change we just make, and move both of the module to a package, and make a third module as a caller to run module_2.py.
project\
package_1\
module_1.py
module_2.py
main.py
main.py:
from package_1.module_2 import func_2
def func_3():
func_2()
if __name__ == '__main__':
func_3()
But the reason we add a . before module_1 in module_2.py is that if we don't do that and run main.py, python interpreter will say No module named 'module_1', that's a little tricky, module_1.py is right beside module_2.py. Now I let func_1() in module_1.py do something:
def func_1():
print(__name__)
that __name__ records who calls func_1. Now we keep the . before module_1 , run main.py, it will print package_1.module_1, not module_1. It indicates that the one who calls func_1() is at the same hierarchy as main.py, the . imply that module_1 is at the same hierarchy as module_2.py itself. So if there isn't a dot, main.py will recognize module_1 at the same hierarchy as itself, it can recognize package_1, but not what "under" it.
Now let's make it a bit complicated. You have a config.ini and a module defines a function to read it at the same hierarchy as 'main.py'.
project\
package_1\
module_1.py
module_2.py
config.py
config.ini
main.py
And for some unavoidable reason, you have to call it with module_2.py, so it has to import from upper hierarchy.module_2.py:
import ..config
pass
Two dots means import from upper hierarchy (three dots access upper than upper,and so on). Now we run main.py, the interpreter will say:ValueError:attempted relative import beyond top-level package. The "top-level package" at here is main.py. Just because config.py is beside main.py, they are at same hierarchy, config.py isn't "under" main.py, or it isn't "leaded" by main.py, so it is beyond main.py. To fix this, the simplest way is:
project\
package_1\
module_1.py
module_2.py
config.py
config.ini
main.py
I think that is coincide with the principle of arrange project file hierarchy, you should arrange modules with different function in different folders, and just leave a top caller in the outside, and you can import how ever you want.

This also works, and is much simpler than anything with the sys module:
with open("C:/yourpath/foobar.py") as f:
eval(f.read())

Call me overly cautious, but I like to make mine more portable because it's unsafe to assume that files will always be in the same place on every computer. Personally I have the code look up the file path first. I use Linux so mine would look like this:
import os, sys
from subprocess import Popen, PIPE
try:
path = Popen("find / -name 'file' -type f", shell=True, stdout=PIPE).stdout.read().splitlines()[0]
if not sys.path.__contains__(path):
sys.path.append(path)
except IndexError:
raise RuntimeError("You must have FILE to run this program!")
That is of course unless you plan to package these together. But if that's the case you don't really need two separate files anyway.