Hello generous SO'ers,
This is a somewhat complicated question, but hopefully relevant to the more general use of global objects from a child-module.
I am using some commercial software that provides a python library for interfacing with their application through TCP. (I can't post the code for their library I don't think.)
I am having an issue with calling an object from a child module, that I think is more generally related to global variables or some such. Basically, the object's state is as expected when the child-module is in the same directory as all the other modules (including the module that creates the object).
But when I move the offending child module into a subfolder, it can still access the object but the state appears to have been altered, and the object's connection to the commercial app doesn't work anymore.
Following some advice from this question on global vars, I have organized my module's files as so:
scriptfile.py
pyFIMM/
__init__.py # imports all the other files
__globals.py # creates the connection object used in most other modules
__pyfimm.py # main module functions, such as pyFIMM.connect()
__Waveguide.py # there are many of these files with various classes and functions
(...other files...)
PhotonDesignLib/
__init__.py # imports all files in this folder
pdPythonLib.py # the commercial library
proprietary/
__init__.py # imports all files in this folder
University.py # <-- The offending child-module with issues
pyFIMM/__init__.py imports the sub-files like so:
from __globals import * # import global vars & create FimmWave connection object `fimm`
from __pyfimm import * # import the main module
from __Waveguide import *.
(...import the other files...)
from proprietary import * # imports the subfolder containing `University.py`
The __init__.py's in the subfolders "PhotonDesignLib" & "proprietary" both cause all files in the subfolders to imported, so, for example, scriptfile.py would access my proprietary files as so: import pyFIMM.proprietary.University. This is accomplished via this hint, coded as follows in proprietary/__init__.py:
import os, glob
__all__ = [ os.path.basename(f)[:-3] for f in glob.glob(os.path.dirname(__file__)+"/*.py")]
(Numerous coders from a few different institutions will have their own proprietary code, so we can share the base code but keep our proprietary files/functions to ourselves this way, without having to change any base code/import statements. I now realize that, for the more static PhotonDesignLib folder, this is overkill.)
The file __globals.py creates the object I need to use to communicate with their commercial app, with this code (this is all the code in this file):
import PhotonDesignLib.pdPythonLib as pd # the commercial lib/object
global fimm
fimm = pd.pdApp() # <- - this is the offending global object
All of my sub-modules contain a from __global import * statement, and are able to access the object fimm without specifically declaring it as a global var, without any issue.
So I run scriptfile.py, which has an import statement like from pyFIMM import *.
Most importantly, scriptfile.py initiates the TCP connection made to the application via fimm.connect() right at the top, before issuing any commands that require the communication, and all the other modules call fimm.Exec(<commands for app>) in various routines, which has been working swimmingly well - the fimm object has so-far been accessible to all modules, and keeps it's connection state without issue.
The issue I am running into is that the file proprietary/University.py can only successfully use the fimm object when it's placed in the pyFIMM root-level directory (ie. the same folder as __globals.py etc.). But when University.py is imported from within the proprietary sub-folder, it gives me an "application not initialized" error when I use the fimm object, as if the object had been overwritten or re-initialized or something. The object still exists, it just isn't maintaining it's connection state when called by this sub-module. (I've checked that it's not reinitialized in another module.)
If, after the script fails in proprietary/University.py, I use the console to send a command eg. pyFimm.fimm.Exec(<command to app>), it communicates just fine!
I set proprietary/University.py to print a dir(fimm) as a test right at the beginning, which works fine and looks like the fimm object exists as expected, but a subsequent call in the same file to fimm.Exec() indicates that the object's state is not correct, returning the "application not initialized" error.
This almost looks like there are two fimm objects - one that the main python console (and pyFIMM modules) see, which works great, and another that proprietary/University.py sees which doesn't know that we called fimm.connect() already. Again, if I put University.py in the main module folder "pyFIMM" it works fine - the fimm.Exec() calls operate as expected!
FYI proprietary/University.py imports the __globals.py file as so:
import sys, os, inspect
ScriptDir = inspect.currentframe().f_code.co_filename # get path to this module file
(ParentDir , tail) = os.path.split(ScriptDir) # split off top-level directory from path
(ParentDir , tail) = os.path.split(ParentDir) # split off top-level directory from path
sys.path.append(ParentDir) # add ParentDir to the python search path
from __globals import * # import global vars & FimmWave connection object
global fimm # This line makes no difference, was just trying it.
(FYI, Somewhere on SO it was stated that inspect was better than __file__, hence the code above.)
Why do you think having the sub-module in a sub-folder causes the object to lose it's state?
I suspect the issue is either the way I instruct University.py to import __globals.py or the "import all files in this folder" method I used in proprietary/__init__.py. But I have little idea how to fix it!
Thank you for looking at this question, and thanks in advance for your insightful comments.
Related
I am currently writing automation scripts for a proprietary Windows desktop application at work using WinAppDriver with Python. Our application has the user upload a handful of files, does some behind the scenes calculating based on the files uploaded and then spits out results. I have automation in place that uploads these files using the UI and am not having any issues with this specifically. The process to do this is as follows:
Click the ‘Choose File’ button. Browse to file location in pop up window
Click in ‘File Name’ field and input the direct path to the file. Click OK (This is being done with the Python Keyboard library)
Repeat previous steps for all necessary files
Click ‘Go’
To tidy up my scripts, I have set the file paths to variables instead of using their direct paths in my code. Then I just call the variable name for the file I need.
E.g. file_to_upload_1: str = r”C:\Users\user\...\filename.txt
I have created a separate filePaths.py where all these file paths set to variables are stored so adding/modifying them in the future is easy and all in one place.
The issue that I am running into with all of this is when I import this .py that contains my file paths set to variables. Right now, I am doing from filePaths import * for simplicity sake. This is generally frowned upon and VS Code throws warnings at me advising I have imported unused imports. I went ahead and set my variables to separate classes and then tried to import them in the following way: from filePaths import dataset_1 When I do this I get the follow error: Undefined variable “variable_name” and my tests fail to run. It seems like I can only get this all to work if I import everything and I would like to avoid doing that if possible. All my scripts are in the same directory. What am I missing here?
Sample of code:
from filePaths import * <-- THIS WORKS!
# from filePaths import class_1 <-- THIS DOES NOT
#Open App
desired_caps = {}
desired_caps["app"] = "C:\\Users\\Public\\Desktop\\Application_Being_Tested.lnk"
driver = webdriver.Remote("http://127.0.0.1:4723", desired_caps)
#Login
driver.find_element_by_accessibility_id("Username").send_keys("tester")
driver.find_element_by_accessibility_id("UserPassword").send_keys("password")
driver.find_element_by_accessibility_id("btnLogin").click()
###Upload Files###
#First File To Upload
driver.find_element_by_accessibility_id("ChooseFile").click()
time.sleep(.1)
driver.find_element_by_accessibility_id("FileName").click()
keyboard.write(filePaths_variable)
keyboard.press_and_release('enter')
You have three options:
Import everything using the wildcard (i.e. from filePaths import *)
Import select objects (i.e. from filePaths import object1, object2, object3 #...)
Use dot notation (i.e. import filePaths then filePaths.object1 #etc)
Some options may be considered better programming style than others.
The reason the wildcard works is because it is the same as option 2 from above if you had listed all created objects within filePaths on you import statement. In general, you should either selectively import only the methods and objects you need, or just import the script and use dot notation to selectively use methods and objects as needed.
The following example code shows how to use dot notation.
file 1:
# objects_to_import.py
bob = 127
string = 'my string'
def foo():
print('bar')
def bar():
print('foo')
def print_var(var):
print(var)
file 2:
# main.py in the same directory as objects_to_import.py
import objects_to_import
print(objects_to_import.bob)
objects_to_import.print_var(objects_to_import.bob)
objects_to_import.foo()
objects_to_import.bar()
try:
print(string)
except NameError:
print("You didn't import that variable or use correct notation!")
Then, running main.py outputs:
"""
127
127
bar
foo
You didn't import that variable or use correct notation!
"""
The results are identical if main.py instead read:
from objects_to_import import bob, foo, bar, print_var
print(bob)
print_var(bob)
foo()
bar()
try:
print(string)
except NameError:
print("You didn't import that variable or use correct notation!")
Note the if we add the following code to both versions of main.py:
if('bob' in globals()):
print('Bob is in your globals!')
else:
print("Can't find bob in your globals")
We find that bob is in your globals' space when explicitly imported, but is not present when using dot notation with the general non-explicit import statement. There therefore might be pragmatic reasons to choose one import method over the other (e.g. if you program is long and complex and you would like to more easily manage potential name collisions, you should use dot notation).
Alright I've come up with a solution!
I have my filePaths.py module with class_1 in there containing a set of certain variables: var_1, var_2, etc. respectively...
In my script that wants these variables, I'm bringing the module in like so:
import filePaths
path = filePaths.class_1
When I call one of the variables in class_1 instead of just var_1 I call path.var_1 and it comes in with no issues. Thank you everyone for helping out with this!
When writing throwaway scripts it's often needed to load a configuration file, image, or some such thing from the same directory as the script. Preferably this should continue to work correctly regardless of the directory the script is executed from, so we may not want to simply rely on the current working directory.
Something like this works fine if defined within the same file you're using it from:
from os.path import abspath, dirname, join
def prepend_script_directory(s):
here = dirname(abspath(__file__))
return join(here, s)
It's not desirable to copy-paste or rewrite this same function into every module, but there's a problem: if you move it into a separate library, and import as a function, __file__ is now referencing some other module and the results are incorrect.
We could perhaps use this instead, but it seems like the sys.argv may not be reliable either.
def prepend_script_directory(s):
here = dirname(abspath(sys.argv[0]))
return join(here, s)
How to write prepend_script_directory robustly and correctly?
I would personally just os.chdir into the script's directory whenever I execute it. It is just:
import os
os.chdir(os.path.split(__file__)[0])
However if you did want to refactor this thing into a library, you are in essence wanting a function that is aware of its caller's state. You thus have to make it
prepend_script_directory(__file__, blah)
If you just wanted to write
prepend_script_directory(blah)
you'd have to do cpython-specific tricks with stack frames:
import inspect
def getCallerModule():
# gets globals of module called from, and prints out __file__ global
print(inspect.currentframe().f_back.f_globals['__file__'])
I think the reason it doesn't smell right is that $PYTHONPATH (or sys.path) is the proper general mechanism to use.
You want pkg_resources
import pkg_resources
foo_fname = pkg_resources.resource_filename(__name__, "foo.txt")
I've got a Python script.
I've had several functions in this script which I decided to move to a 'package' folder beside the main script.
In this folder, I created a *.py file where I put all my functions.
I've placed an empty init.py near this file within the 'package' folder.
When starting the code of my main script with:
from package_folder.my_functions import *
the script works well when calling every functions from that file.
But when trying to import it directly:
import package_folder.my_functions
it doesn't seems to work as well as with the above technique.
The cause seems to be the fact that in the file wellmy_functions.py, I have a function that needs an other one, declared previously in that file.
I had this obscure error on that function that needs an other one:
TypeError: 'NoneType' object is not callable
Is this permissible and if not, how to manage this case?
It's generally not a good idea to use from module import *. Wildcard importing leads to namespace pollution; you imported more names than you need and if you accidentally refer to an imported name you may not get the NameError you wanted.
Also, if a future version of the library added additional names, you could end up masking other names, leading to strange bugs still:
Example
from my_mod1 import func1
from my_mod2 import *
If you upgrade my_mod2 and it now includes a my_mod2.func1 it'll replace the my_mod1.func1 import in the 1st line.
I am importing a lot of different scripts, so at the top of my file it gets cluttered with import statements, i.e.:
from somewhere.fileA import ...
from somewhere.fileB import ...
from somewhere.fileC import ...
...
Is there a way to move all of these somewhere else and then all I have to do is import that file instead so it's just one clean import?
I strongly advise against what you want to do. You are doing the global include file mistake again. Although only one module is importing all your modules (as opposed to all modules importing the global one), the remaining point is that if there's a valid reason for all those modules to be collected under a common name, fine. If there's no reason, then they should be kept as separate includes. The reason is documentation. If I open your file, and see only one import, I don't get any information about what is imported and where it comes from. If on the other hand, I have the list of imports, I know at a glance what is needed and what not.
Also, there's another important error I assume you are doing. When you say
from somewhere.fileA import ...
from somewhere.fileB import ...
from somewhere.fileC import ...
I assume you are importing, for example, a class, like this
from somewhere.fileA import MyClass
this is wrong. This alternative solution is much better
from somewhere import fileA
<later>
a=fileA.MyClass()
Why? two reasons: first, namespacing. If you have two modules having a class named MyClass, you would have a clash. Second, documentation. Suppose you use the first option, and I find in your code the following line
a=MyClass()
now I have no idea where this MyClass comes from, and I will have to grep around all your files in order to find it. Having it qualified with the module name allows me to immediately understand where it comes from, and immediately find, via a /search, where stuff coming from the fileA module is used in your program.
Final note: when you say "fileA" you are doing a mistake. There are modules (or packages), not files. Modules map to files, and packages map to directories, but they may also map to egg files, and you may even create a module having no file at all. This is naming of concepts, and it's a lateral issue.
Of course there is; just create a file called myimports.py in the same directory where your main file is and put your imports there. Then you can simply use from myimports import * in your main script.
Let me explain the use case...
In a simple python web application framework designed for Google App Engine, I'd like to have my models loaded automatically from a 'models' directory, so all that's needed to add a model to the application is place a file user.py (for example), which contains a class called 'User', in the 'models/' directory.
Being GAE, I can't read from the file system so I can't just read the filenames that way, but it seems to me that I must be able to 'import * from models' (or some equivalent), and retrieve at least a list of module names that were loaded, so I can subject them to further processing logic.
To be clear, I want this to be done WITHOUT having to maintain a separate list of these module names for the application to read from.
You can read from the filesystem in GAE just fine; you just can't write to the filesystem.
from models import * will only import modules listed in __all__ in models/__init__.py; there's no automatic way to import all modules in a package if they're not declared to be part of the package. You just need to read the directory (which you can do) and __import__() everything in it.
As explained in the Python tutorial, you cannot load all .py files from a directory unless you list them manually in the list named __all__ in the file __init__.py. One of the reasons why this is impossible is that it would not work well on case-insensitive file systems -- Python would not know in which case the module names should be used.
Let me start by saying that I'm not familiar with Google App Engine, but the following code demonstrates how to import all python files from a directory. In this case, I am importing files from my 'example' directory, which contains one file, 'dynamic_file.py'.
import os
import imp
import glob
def extract_module_names(python_files):
module_names = []
for py_file in python_files:
module_name = (os.path.basename(py_file))[:-3]
module_names.append(module_name)
return module_names
def load_modules(modules, py_files):
module_count = len(modules)
for i in range(0, module_count):
globals()[modules[i]] = imp.load_source(modules[i], py_files[i])
if __name__ == "__main__":
python_files = glob.glob('example/*.py')
module_names = extract_module_names(python_files)
load_modules(module_names, python_files)
dynamic_file.my_func()
Also, if you wish to iterate over these modules, you could modify the load_modules function to return a list of the loaded module objects by appending the 'imp.load_source(..)' call to a list.
Hope it helps.