For example:
if I write
import math
I am not good in programming. For a program to run it has to load the code to its memory and convert to 0s and 1s which a computer can understand?
So, will it load the entire math module when a reference to any function in that module is made in my program, or will it expand the module only once in the program? If it expands only once, I assume, the computer will load entire python file and all the modules it imports completely in memory? won't that cause a memory running out of space issue if I import too many native python code from the library?
Is that the reason some people say it is always good to import exact function in your program instead of wild cards?
Will it load the entire math module when a reference to any function
in that module is made in my program, or will it expand the module
only once in the program?
The math module will be loaded into memory once per Python program (or interpreter).
If it expands only once, I assume, the computer will load entire
python file and all the modules it imports completely in memory
Yes, in normal circumstances.
Won't that cause a memory running out of space issue if I import too
many native python code from the library?
No, not typically. Python modules would not put a dent in the memory of modern computers.
Is that the reason some people say it is always good to import exact
function in your program instead of wild cards?
No, the entire module will be loaded regardless if you use just one function in it. This is because that one function can rely on any other code in the module.
The reason it is advised to import specific functions is more of a best practice or recommendation to be explicit about what you are using from the module.
Also, the module may contain function names in it that are the same as ones you define yourself or are even in another imported module so you want to be careful not to import a bunch of names, especially if you are not going to use them.
Importing a python module means to load the namespace of what is available in that python module, into memory. Specifically, writing "import " tells python to look for a module with that name on your python path (typically a folder), and if it finds such an object, to then run that objects __init__.py file. This file is typically blank, meaning that python should simply load what is available in the module by reading through the files, but this file can also be customized.
Python automatically tracks what is loaded and doesn't re-load already loaded items. Hence, writing
import time
import itertools
import itertools as it
doesn't reload time if itertools also uses the time module and doesn't reload itertools if you rename it to it. In general, this is done very quickly; in fact, if the code is compiled (some modules are installed already compiled) it can be as fast as literally copying the bytes into memory (down to ns of time regardless of module size). Importing is one of the fastest commands you can do in python and is rarely the source of any speed issues. Copying and loading bytes into ram can be done millions of times a second and costs the computer nothing. It is when the computer has to perform calculations and compute that there is a concern.
Something like:
from itertools import combinations
does not load itertools any faster or slower than if you simply loaded the whole module (there are exceptions to this where some modules are so big that they are broken into sub-packages and so loading at the highest level doesn't load anything at all, for example, scipy, you have to specify the sub-package you want to load).
The reason it is recommended to not run from itertools import * is because this loads your namespace with dozens of functions which no one can track. Suddenly, you use the function combinations and no one that reads your code has any idea which module it came from. Hence, it is considered bad practice to * import. Ironically, some programming languages like C do nothing by * importing and it really is impossible to track where variables have come from.
Related
Question
If I have import statements nested in an if/else block, am I increasing efficiency? I know some languages do "one passes" over code for import and syntax issues. I'm just not sure how in depth Python goes into this.
My Hypothesis
Because Python is interpreted and not compiled, by nesting the import statements within the else block, those libraries will not be imported until that line is reached, thus saving system resources unless otherwise needed.
Scenario
I have written a script that will be used by both the more computer literate and those are lesser so. My department is very comfortable with running scripts from the command line with arguments so I have set it up to take arguments for what it needs and, if it does not find the arguments it was expecting, it will launch a GUI with headings, buttons, and more verbose instructions. However, this means that I am importing libraries that are only being used in the event that the arguments were not provided.
Additional Information
The GUI is very, very basic (A half dozen text fields and possibly fewer buttons) so I am not concerned with just creating and spawning a custom GUI class in which the necessary libraries would be imported. If this gets more complicated, I'll consider it in the future or even push to change to web interface.
My script fully functions as I would expect it to. The question is simply about resource consumption.
import statements are executed as they're encountered in normal execution, so if the conditional prevents that line from being executed, the import doesn't occur, and you'll have avoided unnecessary work.
That said, if the module is going to be imported in some other way (say, unconditionally imported module B depends on A, and you're conditionally importing A), the savings are trivial; after the first import of a module, subsequent imports just get a new reference to the same cached module; the import machinery has to do some complicated stuff to handle import hooks and the like first, but in the common case, it's still fairly cheap (sub-microsecond when importing an already cached module).
The only way this will save you anything is if the module in question would not be imported in any way otherwise, in which case you avoid the work of loading it and the memory used by the loaded module.
If I were to create a module that was called for example imp_mod.py and inside it contained all (subjectively used) relevant modules that I frequently used.
Would importing this module into my main program allow me access to the imports contained inside imp_mod.py?
If so, what disadvantages would this bring?
I guess a major advantage would be a reduction of time spent importing even though its only a couple of seconds saved...
Yes, it would allow you to access them. If you place these imports in imp_mod.py:
from os import listdir
from collections import defaultdict
from copy import deepcopy
Then, you could do this in another file, say, myfile.py:
import imp_mod
imp_mod.listdir
imp_mod.defaultdict
imp_mod.deepcopy
You're wrong about reduction of importing time, as what happens is the opposite. Python will need to import imp_mod and then import the other modules afterwards, while the first import would not be needed if you were importing these modules in myfile.py itself. If you do the same imports in another file, they will already be in cache, so virtually no time is spent in the next import.
The real disadvantage here is less readability. Whoever looks at imp_mod.listdir, for example, will ask himself what the heck is this method and why it has the same name as that os module's method. When he had to open imp_mod.py just to find out that it's the same method, well, he probably wouldn't be happy. I wouldn't.
As lucasnadalutti mentioned, you can access them by importing your module.
In terms of advantages, it can make your main program care less about where the imports are coming from if the imp_mod handles all imports, however, as your program gets more complex and starts to include more namespaces, this approach can get more messy. You can start to handle a bit of this by using __init__.py within directories to handle imports to do a similar thing, but as things get more complex, personally, I feel it add a little more complexity. I'd rather just know where a module came from to look it up.
Does importing a specific function from a module is a faster process than importing the whole module?
That is, is from module import x debugs faster than import module?
I would say there is little or no peformance difference, as importing a module for the first time will execute the entire module - all classes, variables and functions are all built, regardless of the actually symbol you need.
The 2nd time you import the module in the same program that will be much quicker, as the module is not reloaded, and all existing definitions are used.
No, it shouldn't be faster, and that shouldn't matter anyway: importing things is not usually considered a performance-critical operation, so you can expect it to be fairly slow compared to other things you can do in Python. If you require importing to be very fast, probably something is wrong with your design.
The whole module has to compile before you can import the specific function.
Instead, it's just a difference in namespace. (ie. you call module_x.function_y vs just calling function_y)
I'm running a website using Django, and I import ipdb at the beginning of almost all of my scripts to make debugging easier. However, most of the time I never use the functions from the module (only when I'm debugging).
Just wondering, will this decrease my performance? It's just that when I want to create a breakpoint I prefer to write:
ipdb.set_trace()
as opposed to:
import ipdb; ipdb.set_trace()
But I've seen the second example done in several places, which makes me wonder if it's more efficient...
I just don't know how importing python modules relates to efficiency (assuming you're not using the module methods within your script).
As #wRAR mentioned, Loading a module may imply executing any amounts of code which can take any amount of time. On the other hand, the module will only be loaded once and any subsequent attempt to import will find the module present in os.sys.modules and reference to that.
In a Django environment in debuging mode, modules are removed from Django's AppCache and actually re-imported only when they are changed, which you will probably not do with ipdb, so in your case it should not be an issue.
However, in cases it would be an issue, there are some ways around it. Suppose you have a custom module that you use to load anyway, you can add a function to it that imports ipdb only when you require it:
# much used module: mymodule
def set_trace():
import ipdb
ipdb.set_trace()
in the module you want to use ipdb.set_trace:
import mymodule
mymodule.set_trace()
or, on top of your module, use the cross-module __debug__ variable:
if __debug__:
from ipdp import set_trace
else:
def set_trace(): return
Short answer: Not usually
Long answer:
It will take time to load the module. This may be noticeable if you are loading python off a network drive or other slow source. But if running directly off a hard drive you'll never notice.
As #wRar points out, importing a module can execute any amount of code. You can have whatever code you want executed at module startup. However, most modules avoid executing unreasonable amounts of code during startup. So that itself probably isn't a huge cause.
However, importing very large modules especially those that also result in importing a large number of c modules will take time.
So importing will take time, but only once per module imported. If you import modules at the top of your modules (as opposed to in functions) it only applies to startup time anyways. Basically, you aren't going to get much optimisation mileage out of avoiding importing modules.
Importing a module but not using it decreases (the system) performance:
It takes time to import the module
Imported modules use up memory
While the first point makes your program slower to start, the second point might make ALL your programs slower, depending on the total amount of memory you have on your system.
I'm just wondering, I often have really long python files and imports tend to stack quite quickly.
PEP8 says that the imports should always be written at the beginning of the file.
Do all the imported libraries get imported when calling a function coded in the file? Or do only the necessary libraries get called?
Does it make sense to worry about this? Is there no reason to import libraries within the functions or classes that need them?
Every time Python hits an import statement, it checks to see if that module has already been imported, and if not, imports it. So the imports at the top of your file will happen as soon as your file is run or imported by another module.
There is some overhead to this, so it's generally best to keep imports at the top of your file so that cost gets taken care of up front.
The best place for imports is at the top of your file. That documents the dependencies in one place and makes errors from their absence appear earlier. The import itself actually occurs at the time of the import statement, but this seldom matters much.
It is not typical that you have anything to gain by not importing a library until you are in a function or method that needs it. (There is never anything to gain by doing so inside the body of a class.) It is rare that you want optional dependencies and even rarer that this is the right technique to get them, though. Perhaps you can share a compelling use case?
Does it make sense to worry about
this?
No
There no reason to import libraries within the functions or classes that need them.
It's just slow because the import statement has to check to see if it's been imported once, and realize that it has been imported.
If you put this in a function that's called frequently, you can waste some time with all the import checking.
Imports happen when the module that contains the imports gets executed or imported, not when the functions are called.
Ordinarily, I wouldn't worry about it. If you are encountering slowdowns, you might profile to see if your problem is related to this. If it is, you can check to see if your module can divided up into smaller modules.
But if all the files are getting used by the same program, you'll just end up importing everything anyway.
If a function inside a module is the only one to import a given other module (say you have a function sending tweets, only if some configuration option is on), then it makes sense to import that specific module in the function.
Unless I see some profiling data proving otherwise, my guess is that the overhead of an import statement in a function is completely negligible.