Overtime I have built up a collection of utility functions for various things.
I would like to put them all in package, with a bit more structure than just a single file containing all the functions.
Some of these functions are written assuming certain packages have been imported e.g. I have several numpy and pandas utility functions that assume something like import numpy as np
Obviously I will not use this hypothetical package like from <pkg> import * but I do not want to hinder performance either.
So if I have a numpy utility function, should I add this to every function
# mypkg.np.utils
import sys
def np_util_fn(...):
if 'np' not in sys.modules: import numpy as np
# rest of func
or
# mypkg.np.utils
import sys
if 'np' not in sys.modules: import numpy as np
def np_util_fn(...):
# rest of func
which is more performant if I use a different part of this package? e.g. from pkg.other.utils import fn
Ok, let's analyze your issue. Assume you have a file module.py:
print("Module got imported")
and a file test.py with:
import module
import module
. If you now execute test.py you will get
Module got imported
. Please note that this line is not outputted two times. This means that python already checks whether a module was already imported (before reimporting it). So your check if 'np' not in sys.modules: import numpy as np is not needed. This check only delays things as it may result in a double check.
In case you want to reimport a module you need reload(module). So if you have
import module
import module
reload(module)
in code.py you will see the line Module got imported two times.
This means that
import numpy as np
is sufficient. There is no need to check whether it already got imported via:
if 'np' not in sys.modules: import numpy as np
It depends whether it is advantageous to do import numpy as np at the very beginning of your script or in a function. If the function is executed multiple times, it is advantageous to do so only at the very beginning. Otherwise you are rechecking whether 'np' is not in sys.modules all the time. In contrast if you can argue that your function is not called to often / is not necessarily executed in your program (e.g. because it depends on user input) then it may be advantageous (seen from the "point vu" of speed) to import this module in a function only.
I normally don't use any import statements in functions as I always have the feeling that they blow up the function body and thus reduce readability.
I want to write a Python module that automatically imports all the good stuff for me (about 50 other modules) so I don't have to copy and past them every time I start a new script. I attempted this by defining the following method in my module, soon to realize when I import my module and call this method, the imports take place locally.
def auto_import():
import os
import sys
# plus 50 other modules...
How can I accomplish this automation using modular programming? (I am using Python 3.6. on Ubuntu.)
You don't need a function to do that, you can simply make a file like commonimports.py which looks like this:
import os
import numpy as np
import sys
#and so on...
And add this import statement in other files
from commonimports import *
And you'll have all the modules ready to use within that namespace
Just make the name of your imported modules global:
def auto_import():
import os
import sys
global os, sys
This is not necessary to use this method if you def auto_import() then every time you have to use a autoimport function whenever you want to use those module.
I'm cleaning up a project that was refactored into smaller .py files. I noticed that a lot of modules are being imported again and again in various files. Some statements are in files that import another which has the same import statement used by the importing file. For example:
main.py
import alt
print (os.getcwd())
alt.py
import os
The print(os.getcwd()) throws a NameError: name 'os' is not defined. Shouldn't os be part of sys.modules when the import statement is executed in alt.py?
Is it possible to use a module imported by another module that was imported first?
They are available in the following namespace:
import alt
print (alt.os.getcwd())
To answer your immediate question, it is possible but not recommended.
Imports adjust the namespace in which they are made. This means that alt has an attribute os that can be accessed in main as
print(alt.os.getcwd())
This is not the recommended way, however, since it makes it less clear which actual os module you are using. You should do import os directly in main. This will not create a separate module object, so do not worry about cluttering your memory with duplicate modules.
The first time an import is encountered during runtime, it is added to the dictionary sys.modules, keyed by the fully qualified name of the module. Future import statements will look in sys.modules for an existing reference before doing any actual work.
In your case, import alt will create a module referenced by sys.modules['alt'] as well as by the name alt in main. The statement import os in alt will be run next. It will create a module referenced by sys.modules['os'] and alt.os. If you were to add a line import os in main after import alt, it will not create and load another module object. Instead, the name os will be bound to the same object pointed to by sys.modules['os'].
The following three versions of main will all call the same getcwd function:
Direct import (recommended):
import alt
import os
print(os.getcwd())
Use the reference in alt (harder to read/trace):
import alt
print(alt.os.getcwd())
Using sys.modules (really not recommended for production unless you know what you are doing):
import alt # necessary to trigger the actual import
import sys
print(sys.modules['os'].getcwd())
All imports load a full module. This even applies to imports of the form from os import getcwd. The module sys.modules['os'] is still created. The only difference is that the importing namespace will only have access to the name getcwd, not os. In your case, if alt contained from os import getcwd instead of import os, the three access methods would change as follows:
Unchanged.
print(alt.getcwd()) since alt.os no longer exists.
Unchanged.
You are importing os only in the in the submodul level of alt. os is to say so only available by access through alt.os. A way around this would be to import all from alt as from alt import *, but this is not what you should do...
As a general rule of thumb, you should re-import your directly called-to modules on the top level of the module that you currently process.
thus at main.py:
import os
and everything is fine
You write it this way
__author__ = 'kerberos'
__date__ = '2017/10/25 20:48 '
import alt
print(alt.os.getcwd())
This is the result:
C:\Users\Administrator\Desktop\11
I have a couple of import statements which I want to use in every module in my project:
import logging
if DEBUG:
import ipdb
from pprint import pprint
I don't want to paste the exact same code in 20 modules so I tried to group the import statements in a single module, auxilliary_modules.py. But when I put import auxilliary_modules in my modules, logging, ipdb and pprint don't seem to be imported:
NameError: name 'logging' is not defined
Is there a way to follow the DRY principle here?
In your case you would need to do:
from auxilliary_modules import *
That would import all the attributes in the auxilliary_modules module into the local namespace.
Otherwise you would need do do this:
import auxilliary_modules
and instead of referencing "logging" (for example), reference:
auxilliary_modules.logging
When importing modules in Python, what is the difference between this:
from module import a, b, c, d
and this
from module import a
from module import b
from module import c
from module import d
To me it makes sense always to condense code and use the first example, but I've been seeing some code samples out there dong the second. Is there any difference at all or is it all in the preference of the programmer?
There is no difference at all. They both function exactly the same.
However, from a stylistic perspective, one might be more preferable than the other. And on that note, the PEP-8 for imports says that you should compress from module import name1, name2 onto a single line and leave import module1 on multiple lines:
Yes: import os
import sys
No: import sys, os
Ok: from subprocess import Popen, PIPE
In response to #teewuane's comment (repeated here in case the comment gets deleted):
#inspectorG4dget What if you have to import several functions from one
module and it ends up making that line longer than 80 char? I know
that the 80 char thing is "when it makes the code more readable" but I
am still wondering if there is a more tidy way to do this. And I don't
want to do from foo import * even though I am basically importing
everything.
The issue here is that doing something like the following could exceed the 80 char limit:
from module import func1, func2, func3, func4, func5
To this, I have two responses (I don't see PEP8 being overly clear about this):
Break it up into two imports:
from module import func1, func2, func3
from module import func4, func5
Doing this has the disadvantage that if module is removed from the codebase or otherwise refactored, then both import lines will need to be deleted. This could prove to be painful
Split the line:
To mitigate the above concern, it may be wiser to do
from module import func1, func2, func3, \
func4, func5
This would result in an error if the second line is not deleted along with the first, while still maintaining the singular import statement
To add to some of the questions raised from inspectorG4dget's answer, you can also use tuples to do multi-line imports when folder structures start getting deeply nested or you have modules with obtuse names.
from some.module.submodule.that_has_long_names import (
first_item,
second_item,
more_imported_items_with_really_enormously_long_names_that_might_be_too_descriptive,
that_would_certainly_not_fit,
on_one_line,
)
This also works, though I'm not a fan of this style:
from module import (a_ton, of, modules, that_seem, to_keep, needing,
to_be, added, to_the_list, of_required_items)
I would suggest not to follow PEP-8 blindly. When you have about half screen worth of imports, things start becoming uncomfortable and PEP-8 is then in conflicts with PEP-20 readability guidelines.
My preference is,
Put all built-in imports on one line such as sys, os, time etc.
For other imports, use one line per package (not module)
Above gives you good balance because the reader can still quickly glance the dependencies while achieving reasonable compactness.
For example,
My Preference
# one line per package
import os, json, time, sys, math
import numpy as np
import torch, torch.nn as nn, torch.autograd, torch.nn.functional as F
from torchvision models, transforms
PEP-8 Recommandation
# one line per module or from ... import statement
import os
import json
import time
import sys
import math
import numpy as np
import torch
from torch import nn as nn, autograd, nn.functional as F
from torchvision import models, transforms
A concern not mentioned by other answers is git merge conflicts.
Let's say you start with this import statement:
import os
If you change this line to import os, sys in one branch and import json, os in another branch, you will get this conflict when you attempt to merge them:
<<<<<<< HEAD
import os, sys
=======
import json, os
>>>>>>> branch
But if you add import sys and import json on separate lines, you get a nice merge commit with no conflicts:
--- a/foo.py
+++ b/foo.py
### -1,2 -1,2 +1,3 ###
+ import json
import os
+import sys
You will still get a conflict if the two imports were added at the same location, as git doesn't know which order they should appear in. So if you had imported time instead of json, for example:
import os
<<<<<<< HEAD
import sys
=======
import time
>>>>>>> branch
Still, it can be worth sticking with this style for the occasions where it does avoid merge conflicts.
Imports should usually be on separate lines as per PEP 8 guidelines.
# Wrong Use
import os, sys
# Correct Use
import os
import sys
For more import based PEP 8 violations and fixes please check this out https://ayush-raj-blogs.hashnode.dev/making-clean-pr-for-open-source-contributors-pep-8-style.
Both are same.
Use from module import a, b, c, d.
If you want to import only one part of a module, use:
from module import a
If u want to import multiple codes from same module, use:
from module import a,b,c,d
No need to write all in separate lines when both are same.