When searching for a solution, it's common to come across several methods. I often use the solution that most closely aligns with syntax I'm familiar with. But sometimes the far-and-away most upvoted solution involves importing a module new to me, as in this thread.
I'm already importing various modules in large script that will be looping 50K times. Does importing additional modules affect processing time, or otherwise affect the script's efficiency? Do I need to worry about the size of the module being called? Seeking guidance on whether, generally, it's worth the extra time/effort to find solutions using methods contained in modules I'm already using.
Every bytecode in Python affects performance. However, unless that code is on a critical path and repeated a high number of times, the effect is so small as to not matter.
Using import consists of two distinct steps: loading the module (done just once), and binding names (where the imported name is added to your namespace to refer to something loaded by the module, or the module object itself). Binding names is almost costless. Because loading a module happens just once, it won't affect your performance.
Focus instead on what the module functionality can do to help you solve your problem efficiently.
Related
When module A imports B, and B imports A, I get error:
ImportError: cannot import name from partially initialized module (most likely due to a circular import).
However, if I merge contents of both modules into one, no error happens. Is there a reason why python does not treat circularly imported files as one big one?
I understand, that ambiguity arises: "Which to run first?", but that can be easily achieved by slightly adjusting syntax of imports and specifying the priority/order of import. Something like:
from foo import A priority 2
from foo import B priority 1
Please provide counter-arguments, if you see any.
I think the main reason that answers your question lays in that Python is an interpreted language. Your script does not run, it is read by the interpreter line by line subsequently (if a different, specific, pattern is not explicitly coded), and the interpreter executes the code you ask for.
For many reasons, one of which is performance, the interpreter cannot (or, better, is not designed to) inspect your whole script to search for patterns. This is completely different from what happens in compiled languages (e.g. C/C++), where the compiler spends much time looking at your code, building up an idea of what you want to do overall, and then (and only then) generates a byte code optimized to do the job you asked for (the optimisation parameters are up to you).
Coming back to Python, imagine when script A is importing script B, and at a certain line of script B, script A (that is still being imported... Recall? The interpreter proceeds line by line) is requested to be imported. It's easy to understand that I cannot proceed with the importing of script A since it relies on B and B is not yet fully imported.
When asking why something has been designed that way, please, do not thing to the most banal example, but to the most complex (or weirdest) one.
Some technics like TYPE_CHECKING can be used to avoid circular error while declaring types from a partially imported module. That's fine, since it's a specific, narrow case, where you're explicitly stating that you're not going to use any object from module A in module B before the former has been completely imported, but you just need to declare in advance their type being.
Hope I've been clear, though I'm aware my answer is not rigorous. Suggestion from those who are more expert than me in programming theory are warmly adviced.
Your example demonstrates the simplest case of circular imports. Solving the general case is vastly more complicated, possibly intractable.
Now consider import chains with multiple loops and loops on loops.
Question
If I have import statements nested in an if/else block, am I increasing efficiency? I know some languages do "one passes" over code for import and syntax issues. I'm just not sure how in depth Python goes into this.
My Hypothesis
Because Python is interpreted and not compiled, by nesting the import statements within the else block, those libraries will not be imported until that line is reached, thus saving system resources unless otherwise needed.
Scenario
I have written a script that will be used by both the more computer literate and those are lesser so. My department is very comfortable with running scripts from the command line with arguments so I have set it up to take arguments for what it needs and, if it does not find the arguments it was expecting, it will launch a GUI with headings, buttons, and more verbose instructions. However, this means that I am importing libraries that are only being used in the event that the arguments were not provided.
Additional Information
The GUI is very, very basic (A half dozen text fields and possibly fewer buttons) so I am not concerned with just creating and spawning a custom GUI class in which the necessary libraries would be imported. If this gets more complicated, I'll consider it in the future or even push to change to web interface.
My script fully functions as I would expect it to. The question is simply about resource consumption.
import statements are executed as they're encountered in normal execution, so if the conditional prevents that line from being executed, the import doesn't occur, and you'll have avoided unnecessary work.
That said, if the module is going to be imported in some other way (say, unconditionally imported module B depends on A, and you're conditionally importing A), the savings are trivial; after the first import of a module, subsequent imports just get a new reference to the same cached module; the import machinery has to do some complicated stuff to handle import hooks and the like first, but in the common case, it's still fairly cheap (sub-microsecond when importing an already cached module).
The only way this will save you anything is if the module in question would not be imported in any way otherwise, in which case you avoid the work of loading it and the memory used by the loaded module.
The memory cost obviously depends on exactly how large a module is, but I'm only looking for a general answer: Is it generally expensive or cheap to import a module in Python? If I have a few tens of small scripts that potentially stay in memory for the whole duration of the application, how much will that hog the memory?
It sounds like you aren't worried about time cost (good; that would be silly, since modules are only imported once) but memory cost. I put it to you: if you need all the functionality in these modules, then how exactly do you plan to avoid having them all in memory? Might as well just import things in the most logical way.
That said, in Python import is a statement and not some kind of preprocessor directive (or similar), and so you can delay the import of a module until you actually need its contents, by simply arranging for the statement to run at the appropriate time. This can be a meaningful optimization in some particularly large or complex projects, or at least make tricky things possible. Remember, the Python compiler doesn't try to check if there is a foo when you refer to foo.bar; every name is looked up at run-time. Duck-typing comes into play here; the foo.bar code doesn't care whether foo is the name of a module, class or object. (That's basically because modules and classes are objects, but I digress...)
I have around 80 lines of a function in a file. I need the same functionality in another file so I am currently importing the other file for the function.
My question is that in terms of running time on a machine which technique would be better :- importing the complete file and running the function or copying the function as it is and run it from same package.
I know it won't matter in a large sense but I want to learn it in the sense that if we are making a large project is it better to import a complete file in Python or just add the function in the current namespace.....
Importing is how you're supposed to do it. That's why it's possible. Performance is a complicated question, but in general it really doesn't matter. People who really, really need performance, and can't be satisfied by just fixing the basic algorithm, are not using Python in the first place. :) (At least not for the tiny part of the project where the performance really matters. ;) )
Importing is good cause it helps you manage stuff easily. What if you needed the same function again? Instead of making changes at multiple places, there is just one centralized location - your module.
In case the function is small and you won't need it anywhere else, put it in the file itself.
If it is complex and would require to be used again, separate it and put it inside a module.
Performance should not be your concern here. It should hardly matter. And even if it does, ask yourself - does it matter to you?
Copy/Paste cannot be better. Importing affects load-time performance, not run-time (if you import it at the top-level).
The whole point of importing is to allow code reuse and organization.
Remember too that you can do either
import MyModule
to get the whole file or
from MyModule import MyFunction
for when you only need to reference that one part of the module.
If the two modules are unrelated except for that common function, you may wish to consider extracting that function (and maybe other things that are related to that function) into a third module.
I'm just wondering, I often have really long python files and imports tend to stack quite quickly.
PEP8 says that the imports should always be written at the beginning of the file.
Do all the imported libraries get imported when calling a function coded in the file? Or do only the necessary libraries get called?
Does it make sense to worry about this? Is there no reason to import libraries within the functions or classes that need them?
Every time Python hits an import statement, it checks to see if that module has already been imported, and if not, imports it. So the imports at the top of your file will happen as soon as your file is run or imported by another module.
There is some overhead to this, so it's generally best to keep imports at the top of your file so that cost gets taken care of up front.
The best place for imports is at the top of your file. That documents the dependencies in one place and makes errors from their absence appear earlier. The import itself actually occurs at the time of the import statement, but this seldom matters much.
It is not typical that you have anything to gain by not importing a library until you are in a function or method that needs it. (There is never anything to gain by doing so inside the body of a class.) It is rare that you want optional dependencies and even rarer that this is the right technique to get them, though. Perhaps you can share a compelling use case?
Does it make sense to worry about
this?
No
There no reason to import libraries within the functions or classes that need them.
It's just slow because the import statement has to check to see if it's been imported once, and realize that it has been imported.
If you put this in a function that's called frequently, you can waste some time with all the import checking.
Imports happen when the module that contains the imports gets executed or imported, not when the functions are called.
Ordinarily, I wouldn't worry about it. If you are encountering slowdowns, you might profile to see if your problem is related to this. If it is, you can check to see if your module can divided up into smaller modules.
But if all the files are getting used by the same program, you'll just end up importing everything anyway.
If a function inside a module is the only one to import a given other module (say you have a function sending tweets, only if some configuration option is on), then it makes sense to import that specific module in the function.
Unless I see some profiling data proving otherwise, my guess is that the overhead of an import statement in a function is completely negligible.