I want to split a large python module i wrote into multiple files within a directory, where each file is a function that may or may not have dependencies with other functions within the module. Here's a simple example of what i came up with:
First, here's a self contained .py module
#[/pie.py]
def getpi():
return pi()
def pi():
return 3.1416
Obviously, this works fine when importing and calling either function. So now i split it in different files with an init.py file to wrap it all up:
#[/pie/__init__.py]
from getpi import *
from pi import *
__all__=['getpi','pi']
#[/pie/getpi.py]
def getpi():
return pi()
#[/pie/pi.py]
def pi():
return 3.1416
Because getpi() has a dependency with pi(), calling it as currently structured raises an exception:
>>> import pie
>>> pie.getpi()
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
pie.getpi()
File "C:\python\pie\getpi.py", line 2, in getpi
return pi()
NameError: global name 'pi' is not defined
And so to fix this issue, my current solution is to write init.py like so:
#[/pie/__init__.py]
import os as _os
__all__ = []
for _f in _os.listdir(__path__[0]):
if not _f == '__init__.py' and _f.endswith('.py'):
execfile('%s\\%s'%(__path__[0],_f))
__all__.append(_os.path.splitext(_f)[0])
So now it works fine:
>>> import pie
>>> pie.getpi()
3.1416
So now everything works as if everything was contained in a single .py file. init.py can contain all the high level imports (numpy, os, sys, glob...) that all the individual functions need.
Structuring a module this way feels "right" to me. New functions are loaded automatically at the next import (no need to append init.py each time). It lets me see at a glance which functions are meant to be used just by looking at what's within a directory, plus it keeps everything nicely sorted alphabetically.
The only negative i can see at this time is that only init.py gets byte-compiled and not any of the sub .py files. But loading speed hasn't been an issue so i don't mind. Also, i do realize this might cause an issue with packaging, but it's also something i don't mind because our scripts get distributed via our own revision control system.
Is this an acceptable way of structuring a python module? And if not, what would be the correct way to achieve what i've done properly.
The "correct" way would be to import the necessary modules where they are needed:
# pi.py
def pi(): return 3.1417
# getpi.py
from .pi import pi
def getpi(): return pi()
# __init__.py
from .pi import *
from .getpi import *
Make sure you don't have cyclic dependencies. These are bad in any case, but you can avoid them by abstracting up to the necessary level.
Related
I'm getting this error
Traceback (most recent call last):
File "/Users/alex/dev/runswift/utils/sim2014/simulator.py", line 3, in <module>
from world import World
File "/Users/alex/dev/runswift/utils/sim2014/world.py", line 2, in <module>
from entities.field import Field
File "/Users/alex/dev/runswift/utils/sim2014/entities/field.py", line 2, in <module>
from entities.goal import Goal
File "/Users/alex/dev/runswift/utils/sim2014/entities/goal.py", line 2, in <module>
from entities.post import Post
File "/Users/alex/dev/runswift/utils/sim2014/entities/post.py", line 4, in <module>
from physics import PostBody
File "/Users/alex/dev/runswift/utils/sim2014/physics.py", line 21, in <module>
from entities.post import Post
ImportError: cannot import name Post
and you can see that I use the same import statement further up and it works. Is there some unwritten rule about circular importing? How do I use the same class further down the call stack?
See also What happens when using mutual or circular (cyclic) imports in Python? for a general overview of what is allowed and what causes a problem WRT circular imports. See What can I do about "ImportError: Cannot import name X" or "AttributeError: ... (most likely due to a circular import)"? for techniques for resolving and avoiding circular dependencies.
I think the answer by jpmc26, while by no means wrong, comes down too heavily on circular imports. They can work just fine, if you set them up correctly.
The easiest way to do so is to use import my_module syntax, rather than from my_module import some_object. The former will almost always work, even if my_module included imports us back. The latter only works if my_object is already defined in my_module, which in a circular import may not be the case.
To be specific to your case: Try changing entities/post.py to do import physics and then refer to physics.PostBody rather than just PostBody directly. Similarly, change physics.py to do import entities.post and then use entities.post.Post rather than just Post.
When you import a module (or a member of it) for the first time, the code inside the module is executed sequentially like any other code; e.g., it is not treated any differently that the body of a function. An import is just a command like any other (assignment, a function call, def, class). Assuming your imports occur at the top of the script, then here's what's happening:
When you try to import World from world, the world script gets executed.
The world script imports Field, which causes the entities.field script to get executed.
This process continues until you reach the entities.post script because you tried to import Post
The entities.post script causes physics module to be executed because it tries to import PostBody
Finally, physics tries to import Post from entities.post
I'm not sure whether the entities.post module exists in memory yet, but it really doesn't matter. Either the module is not in memory, or the module doesn't yet have a Post member because it hasn't finished executing to define Post
Either way, an error occurs because Post is not there to be imported
So no, it's not "working further up in the call stack". This is a stack trace of where the error occurred, which means it errored out trying to import Post in that class. You shouldn't use circular imports. At best, it has negligible benefit (typically, no benefit), and it causes problems like this. It burdens any developer maintaining it, forcing them to walk on egg shells to avoid breaking it. Refactor your module organization.
To understand circular dependencies, you need to remember that Python is essentially a scripting language. Execution of statements outside methods occurs at compile time. Import statements are executed just like method calls, and to understand them you should think about them like method calls.
When you do an import, what happens depends on whether the file you are importing already exists in the module table. If it does, Python uses whatever is currently in the symbol table. If not, Python begins reading the module file, compiling/executing/importing whatever it finds there. Symbols referenced at compile time are found or not, depending on whether they have been seen, or are yet to be seen by the compiler.
Imagine you have two source files:
File X.py
def X1:
return "x1"
from Y import Y2
def X2:
return "x2"
File Y.py
def Y1:
return "y1"
from X import X1
def Y2:
return "y2"
Now suppose you compile file X.py. The compiler begins by defining the method X1, and then hits the import statement in X.py. This causes the compiler to pause compilation of X.py and begin compiling Y.py. Shortly thereafter the compiler hits the import statement in Y.py. Since X.py is already in the module table, Python uses the existing incomplete X.py symbol table to satisfy any references requested. Any symbols appearing before the import statement in X.py are now in the symbol table, but any symbols after are not. Since X1 now appears before the import statement, it is successfully imported. Python then resumes compiling Y.py. In doing so it defines Y2 and finishes compiling Y.py. It then resumes compilation of X.py, and finds Y2 in the Y.py symbol table. Compilation eventually completes w/o error.
Something very different happens if you attempt to compile Y.py from the command line. While compiling Y.py, the compiler hits the import statement before it defines Y2. Then it starts compiling X.py. Soon it hits the import statement in X.py that requires Y2. But Y2 is undefined, so the compile fails.
Please note that if you modify X.py to import Y1, the compile will always succeed, no matter which file you compile. However if you modify file Y.py to import symbol X2, neither file will compile.
Any time when module X, or any module imported by X might import the current module, do NOT use:
from X import Y
Any time you think there may be a circular import you should also avoid compile time references to variables in other modules. Consider the innocent looking code:
import X
z = X.Y
Suppose module X imports this module before this module imports X. Further suppose Y is defined in X after the import statement. Then Y will not be defined when this module is imported, and you will get a compile error. If this module imports Y first, you can get away with it. But when one of your co-workers innocently changes the order of definitions in a third module, the code will break.
In some cases you can resolve circular dependencies by moving an import statement down below symbol definitions needed by other modules. In the examples above, definitions before the import statement never fail. Definitions after the import statement sometimes fail, depending on the order of compilation. You can even put import statements at the end of a file, so long as none of the imported symbols are needed at compile time.
Note that moving import statements down in a module obscures what you are doing. Compensate for this with a comment at the top of your module something like the following:
#import X (actual import moved down to avoid circular dependency)
In general this is a bad practice, but sometimes it is difficult to avoid.
For those of you who, like me, come to this issue from Django, you should know that the docs provide a solution:
https://docs.djangoproject.com/en/1.10/ref/models/fields/#foreignkey
"...To refer to models defined in another application, you can explicitly specify a model with the full application label. For example, if the Manufacturer model above is defined in another application called production, you’d need to use:
class Car(models.Model):
manufacturer = models.ForeignKey(
'production.Manufacturer',
on_delete=models.CASCADE,
)
This sort of reference can be useful when resolving circular import dependencies between two applications...."
I was able to import the module within the function (only) that would require the objects from this module:
def my_func():
import Foo
foo_instance = Foo()
If you run into this issue in a fairly complex app it can be cumbersome to refactor all your imports. PyCharm offers a quickfix for this that will automatically change all usage of the imported symbols as well.
I was using the following:
from module import Foo
foo_instance = Foo()
but to get rid of circular reference I did the following and it worked:
import module.foo
foo_instance = foo.Foo()
According to this answer we can import another module's object in the block( like function/ method and etc), without circular import error occurring, for example for import Simple object of another.py module, you can use this:
def get_simple_obj():
from another import Simple
return Simple
class Example(get_simple_obj()):
pass
class NotCircularImportError:
pass
In this situation, another.py module can easily import NotCircularImportError, without any problem.
just check your file name see if it is not the same as library you are importing.
Eg - sympy.py
import sympy as sym
I am currently writing automation scripts for a proprietary Windows desktop application at work using WinAppDriver with Python. Our application has the user upload a handful of files, does some behind the scenes calculating based on the files uploaded and then spits out results. I have automation in place that uploads these files using the UI and am not having any issues with this specifically. The process to do this is as follows:
Click the ‘Choose File’ button. Browse to file location in pop up window
Click in ‘File Name’ field and input the direct path to the file. Click OK (This is being done with the Python Keyboard library)
Repeat previous steps for all necessary files
Click ‘Go’
To tidy up my scripts, I have set the file paths to variables instead of using their direct paths in my code. Then I just call the variable name for the file I need.
E.g. file_to_upload_1: str = r”C:\Users\user\...\filename.txt
I have created a separate filePaths.py where all these file paths set to variables are stored so adding/modifying them in the future is easy and all in one place.
The issue that I am running into with all of this is when I import this .py that contains my file paths set to variables. Right now, I am doing from filePaths import * for simplicity sake. This is generally frowned upon and VS Code throws warnings at me advising I have imported unused imports. I went ahead and set my variables to separate classes and then tried to import them in the following way: from filePaths import dataset_1 When I do this I get the follow error: Undefined variable “variable_name” and my tests fail to run. It seems like I can only get this all to work if I import everything and I would like to avoid doing that if possible. All my scripts are in the same directory. What am I missing here?
Sample of code:
from filePaths import * <-- THIS WORKS!
# from filePaths import class_1 <-- THIS DOES NOT
#Open App
desired_caps = {}
desired_caps["app"] = "C:\\Users\\Public\\Desktop\\Application_Being_Tested.lnk"
driver = webdriver.Remote("http://127.0.0.1:4723", desired_caps)
#Login
driver.find_element_by_accessibility_id("Username").send_keys("tester")
driver.find_element_by_accessibility_id("UserPassword").send_keys("password")
driver.find_element_by_accessibility_id("btnLogin").click()
###Upload Files###
#First File To Upload
driver.find_element_by_accessibility_id("ChooseFile").click()
time.sleep(.1)
driver.find_element_by_accessibility_id("FileName").click()
keyboard.write(filePaths_variable)
keyboard.press_and_release('enter')
You have three options:
Import everything using the wildcard (i.e. from filePaths import *)
Import select objects (i.e. from filePaths import object1, object2, object3 #...)
Use dot notation (i.e. import filePaths then filePaths.object1 #etc)
Some options may be considered better programming style than others.
The reason the wildcard works is because it is the same as option 2 from above if you had listed all created objects within filePaths on you import statement. In general, you should either selectively import only the methods and objects you need, or just import the script and use dot notation to selectively use methods and objects as needed.
The following example code shows how to use dot notation.
file 1:
# objects_to_import.py
bob = 127
string = 'my string'
def foo():
print('bar')
def bar():
print('foo')
def print_var(var):
print(var)
file 2:
# main.py in the same directory as objects_to_import.py
import objects_to_import
print(objects_to_import.bob)
objects_to_import.print_var(objects_to_import.bob)
objects_to_import.foo()
objects_to_import.bar()
try:
print(string)
except NameError:
print("You didn't import that variable or use correct notation!")
Then, running main.py outputs:
"""
127
127
bar
foo
You didn't import that variable or use correct notation!
"""
The results are identical if main.py instead read:
from objects_to_import import bob, foo, bar, print_var
print(bob)
print_var(bob)
foo()
bar()
try:
print(string)
except NameError:
print("You didn't import that variable or use correct notation!")
Note the if we add the following code to both versions of main.py:
if('bob' in globals()):
print('Bob is in your globals!')
else:
print("Can't find bob in your globals")
We find that bob is in your globals' space when explicitly imported, but is not present when using dot notation with the general non-explicit import statement. There therefore might be pragmatic reasons to choose one import method over the other (e.g. if you program is long and complex and you would like to more easily manage potential name collisions, you should use dot notation).
Alright I've come up with a solution!
I have my filePaths.py module with class_1 in there containing a set of certain variables: var_1, var_2, etc. respectively...
In my script that wants these variables, I'm bringing the module in like so:
import filePaths
path = filePaths.class_1
When I call one of the variables in class_1 instead of just var_1 I call path.var_1 and it comes in with no issues. Thank you everyone for helping out with this!
My file is named "foo.py". It has only two lines.
import random
print(random.randint(10))
The error is...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/random.py", line 45, in <module>
from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
File "math.py", line 2, in <module>
from random import randint
ImportError: cannot import name randint
I am on MacOS 10.14.6
I note that I did not call random.randint() in this script, even
though it showed up in the error text.
I ran the script with $python
My /usr/bin/python is linked to python 2.7
I've tried this with python 3 as well, with the same error.
EDIT:
My script was originally named "math.py", but I changed it in response to another solution that pointed out the name conflict with the math.py library (even though my script was not importing that library). Even after my script name change, I'm still seeing --File "math.py"-- errors. Even after I'm no longer using random.randint(), I'm still seeing that function referenced in my errors.
I've tried deleting random.pyc and math.pyc to purge the artifacts of previous executions. But these do not see to eliminate the remnants of earlier errors.
Read the traceback:
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/random.py"
Python tries to do something inside the standard library random module...
from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
in particular, it tries to import the standard library math module...
File "math.py", line 2, in <module>
but it gets yours instead (notice there's no path on the filename this time; it's just math.py in the current directory); i.e. the script you started from. Python detects the circular import and fails:
ImportError: cannot import name randint
Actually using randint doesn't matter, because this is a problem with the actual import of the module.
This happens because Python is configured by default (using sys.path, which is a list of paths to try, in order) to try to import scripts from the current working directory before looking anywhere else. It's convenient when you just want to write a few source files in the same folder and have them work with each other, but it causes these problems.
The expected solution is to just rename your file. Unfortunately there isn't an obvious list of names to avoid, although you could peek at your installation folder to be sure (or just check the online library reference, though that's not quite so direct).
I guess you could also modify sys.path:
import sys
sys.path.remove('') # the empty string in this list is for the current directory
sys.path.append('') # put it back, at the end this time
import random # now Python will look for modules in the standard library first,
# and only in the current folder as a last resort.
However, this is an ugly hack. It might break something else (and it can't save you if you have a local sys.py).
I have a fairly extensive coding background in multiple environments, but, am new to Python. I thought I had figured out a way to import functions dynamically from an external '.py' file, but am unsure if it is the best way. I found problems with using importlib.import_module() and importlib.__import__ based on my goal. Is there another way to accomplish what I am doing here? Effectively, I am wanting to have the same result as I would get when using from x import y where x and y are variables. I thought I would be able to use eval('from '+x+' import '+y) but this throws a syntax error.
I was hoping to accomplish this by contriving a dictionary with files as the keys (i.e., a file named 'file1.py' would create a key of 'file1') and a list of the desired functions as a list associated with its relative key. This can easily be built either literally, or by reading a path for file names and then using the dir() function to get a list of the functions in each individual file (among many other ways). Next, I hoped to simply use nested for loops to walk the dictionary keys and their associated key value lists and used eval('from '+key+' import '+currentListItem). Unfortunately, this throws a syntax error on the execution of the generated 'from...import...' statement. See below for example code. My problems with importlib (and getattr) is that I am unable to maintain the 'abstraction' provided by this method as I have to define a 'handle' in order to use importlib (i.e., handle = getattr(...) or handle = importlib.import_module(key) meaning that I basically have to hard code a 'handle name' for the given module being imported and thus may just as well hard code the 'from file_name import function' statements).
# simplistic example of what I was thinking....
# FILE file.py contains the code...
def asub(p1, p2 = None, p3 = None):
print(p1, p2 if p2 else 'p2 defaulted', p3 if p3 else 'p3 defaulted')
return
# FILE b.py contains the code...
#!/usr/local/bin/python3
subs = {'file':['asub']}
for key in subs:
for subrt in subs[key]:
print("==>", key, subrt)
eval('from '+key+' import '+subrt)
# on execution of b.py (i.e., ```python3 b.py``` I get the following...
Traceback (most recent call last):
File "b.py", line 9, in <module>
eval('from '+key+' import '+subrt)
File "<string>", line 1
from file import asub
^
SyntaxError: invalid syntax
NOTE: I understand that this may not be the greatest thing for importing modules/functions as it tends to 'hide' the information that you would need for documentation and usage of the imported functions.
However, I am sure that there will be instances where this may be useful or where similar methodology may be useful in a case other than importing.
As I said, I am new to Python, so, am looking for feedback/guidance on this in a (cringe) 'Pythonic' sense.
One other thing: I get that eval() can open the door for code insertion but for the above specific use, and given that the files containing the functions are well locked down, I think it should be plenty safe (?)...
Thanks in advance for any feedback
Try this:
from importlib.machinery import SourceFileLoader
subs = {'file':['asub']}
for key in subs:
# import module by file path
# since path is relative, it will import against working directory
mod = SourceFileLoader("", "%s.py" % key).load_module()
for subrt in subs[key]:
print("==>", key, subrt)
# retrieve object from module by name subrt
obj = getattr(mod, subrt)
# this should call file.asub and it does
obj('1', '2')
# this add local variable with the same name
locals()[subrt] = obj
# so now we can call asub as if it were done
# from file import asub
asub('3', '4')
This code prints:
==> file asub
1 2 p3 defaulted
3 4 p3 defaulted
This works on my python3 and requires python 3.3 at least. This is built on top of question How to import a module given the full path?. locals()[subrt] = obj puts imported object into local variables directory, which allows calling to it as if it were done from file import asub. If i understood correctly this is what you wanted to achieve?
FYI, Final form based on accepted answer provided by Radoslaw. NOTE: I also tested with importlib.import_module(key) as well and that worked the same way except that you may have to use sys.path.insert(...) to make sure your module can be located.
import os, sys, re
from importlib.machinery import SourceFileLoader
# Build list of 'things' to be imported
# (presumably via a lookup of some sort)
# I hard coded it here as a simple dictionary element for testing/dev.
# Notice that a fully qualified file name is used so that the target
# subroutine/function can come from any specified/discovered file
# Another option may be to use only a list of file names here and then # use 'dir()' to get the functions from the file and then choose from
# the the list returned. This is illustrated below.
subs = {'/some/fully/qualified/path/file.txt':['asub']}
# import the files
for key in subs:
mod = SourceFileLoader("", key).load_module()
# Example of how to 'extract' the function names from the module for
# filtering and loading of explicit modules when using 'subs' as a
# list vs. a dictionary. Here I simply excluded anything beginning
# and ending with double '_' and then processed those 'functions'.
# for d in dir(mod):
# if not re.match(r'^__.+?__$', d):
# locals()[subrt] = getattr(mod, subrt)
# otherwise, if 'subs' is used as a dictionary as
# described above, just walk the list of functions
# built as part of the 'subs' dictionary
for subrt in subs[key]:
locals()[subrt] = getattr(mod, subrt)
# done
asub('parm1', 'p2_parm')
quit()
An interesting exercise. Thanks much for the help Radoslaw.
I have split a project that i write from 1 file to several files
The thing is, that i have around 50 classes that I created and are called from the main file, I don't want to rewrite all those class references and add the module name before each class.
I tried to make all those classes accessible through 1 package (Tokens)
So I have
Main.py
Tokens /
__init__.py
Gen.py
BuilltIns.py
The Idea was to populate the package namespace with all the classes, and then import the package inside Main.py
__init__.py:
from Gen import *
from BuilltIns import *
Main.py:
from Tokens import *
When I ran __init__ it works perfectly, and dir() reveals that all class names are imported to package namepace.
However, when i run Main.py, I get the error message:
Traceback (most recent call last):
File "../Main.py", line 1, in <module>
from Tokens import *
File "..\Tokens\__init__.py", line 1, in <module>
from Gen import *
ModuleNotFoundError: No module named 'Gen'
How should I extract the 60+ classes from Main.py to other modules without rewriting all the calls for those classes?
You should use from Tokens.Gen import ... since Tokens is the package where the Gen module resides. The import path should be relative to the main script, unless you've modified the sys.path to specify additional directories to be searched during imports.
Alternatively, in Tokens/__init__.py you can do from .Gen import * (note the . before Gen). This denotes a relative import. What happens when you run the main script is that the current working directory is added to the paths to be searched during imports, so when from Gen import ... is encountered only the default locations are searched (which doesn't include the Tokens directory). By using a relative import you tell Python where it can find that module relative to the current one.
Note that you can define your classes in __all__ in order to constrain what will be imported during from ... import *. This way you don't leak other names in your main script's namespace.