The function below works just fine. But if I remove all py files (and leave the pycs intact) then I get an error:
To explain what I mean by 'intact' here is more or less what I did:
1. write a bunch of py files and stick them in a friendly directory structure
2. test code code. It works
3. compile all py files to get pyc files
4. delete py files
5. test code. It fails
The function:
def get_module_name_and_line():
"""
return the name of the module from which the method calling this method was called.
"""
import inspect
lStack = inspect.stack()
oStk = lStack[2]
oMod = inspect.getmodule(oStk[0])
oInfo = inspect.getframeinfo(oStk[0])
sName = oMod.__name__ #<<<<<<<<<<<<<<<<<< ERROR HERE
iLine = oInfo.lineno
return sName,iLine
The error:
AttributeError: 'NoneType' object has no attribute '__name__'
So oMod is None in this error. If the py files are around then oMod is never None.
The question:
Why does inspect only return a module if py files are intact? How can I make this function work without py files.
Full Traceback:
Original exception was:
Traceback (most recent call last):
File "/home/criticalid/programs/damn.py", line 630, in <module>
File "/home/criticalid/programs/golly/class_foo.py", line 121, in moo
File "/home/criticalid/programs/golly/class_foo.py", line 151, in get_module_name_and_line
AttributeError: 'NoneType' object has no attribute '__name__'
This works for me. It assumes that all modules are in packages within the current working directory. And it doesn't return the __main__ module, rather its file name.
I'm sure there is a better solution but this solves my problems.
def get_module_name_and_line():
"""
return the name of the module from which the method calling this method was called.
"""
def get_name_from_path(sPath):
import os
sCWD = os.getcwd()
lCWD = list(os.path.split(sCWD))
lPath = list(os.path.split(sPath))
lPath[-1] = '.'.join(lPath[-1].split('.')[:-1]) #remove file extension
lRet = [s for s in lPath[len(lCWD)-1:]]
return '.'.join(lRet)
import inspect
lStack = inspect.stack()
oStk = lStack[2]
iLine = inspect.getlineno(oStk[0])
sName = get_name_from_path(inspect.getfile(oStk[0]))
return sName,iLine
Related
I'm running a code and it gives me an error I can't solve !
how can I add the missing attribute?
the relevant part of the code :
ALL_FILES = provider.getDataFiles('indoor3d_sem_seg_hdf5_data/all_files.txt') #line 63
room_filelist = [line.rstrip() for line in open('indoor3d_sem_seg_hdf5_data/room_filelist.txt')]
The error:
Traceback (most recent call last):
File "E:\Research\Codes\pointnet\pointnet-master\sem_seg\train.py", line 63, in <module>
ALL_FILES = provider.getDataFiles('indoor3d_sem_seg_hdf5_data/all_files.txt')
AttributeError: module 'provider' has no attribute 'getDataFiles'
First, check if you have import provider in your code, you can also do from model import *
I found out that you are using pointnet. So I search the source code and I found this method is:
def getDataFiles(list_filename):
return [line.rstrip() for line in open(list_filename)]
You can search your library for this method. It might not be in the provider.py
You could just added this method to your code. But the best idea is to search for it.
For you case, the provider.py should be at \pointnet\pointnet-master\, and there is also a train.py at that location.
Problem solved ! All I had to do is to copy the provider.py file into the sem.seg.py file which I used. It appears it couldn't find it in the previous file.
I hope the following question is not too long. But otherwise I cannot explain by problem and what I want:
Learned from How to use importlib to import modules from arbitrary sources? (my question of yesterday)
I have written a specfic loader for a new file type (.xxx).
(In fact the xxx is an encrypted version of a pyc to protect code from being stolen).
I would like just to add an import hook for the new file type "xxx" without affecting the other types (.py, .pyc, .pyd) in any way.
Now, the loader is ModuleLoader, inheriting from mportlib.machinery.SourcelessFileLoader.
Using sys.path_hooks the loader shall be added as a hook:
myFinder = importlib.machinery.FileFinder
loader_details = (ModuleLoader, ['.xxx'])
sys.path_hooks.append(myFinder.path_hook(loader_details))
Note: This is activated once by calling modloader.activateLoader()
Upon loading a module named test (which is a test.xxx) I get:
>>> import modloader
>>> modloader.activateLoader()
>>> import test
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'test'
>>>
However, when I delete content of sys.path_hooks before adding the hook:
sys.path_hooks = []
sys.path.insert(0, '.') # current directory
sys.path_hooks.append(myFinder.path_hook(loader_details))
it works:
>>> modloader.activateLoader()
>>> import test
using xxx class
in xxxLoader exec_module
in xxxLoader get_code: .\test.xxx
ANALYZING ...
GENERATE CODE OBJECT ...
2 0 LOAD_CONST 0
3 LOAD_CONST 1 ('foo2')
6 MAKE_FUNCTION 0
9 STORE_NAME 0 (foo2)
12 LOAD_CONST 2 (None)
15 RETURN_VALUE
>>>>>> test
<module 'test' from '.\\test.xxx'>
The module is imported correctly after conversion of the files content to a code object.
However I cannot load the same module from a package: import pack.test
Note: __init__.py is of course as an empty file in pack directory.
>>> import pack.test
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 2218, in _find_and_load_unlocked
AttributeError: 'module' object has no attribute '__path__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'pack.test'; 'pack' is not a package
>>>
Not enough, I cannot load plain *.py modules from that package anymore: I get the same error as above:
>>> import pack.testpy
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 2218, in _find_and_load_unlocked
AttributeError: 'module' object has no attribute '__path__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'pack.testpy'; 'pack' is not a package
>>>
For my understanding sys.path_hooks is traversed until the last entry is tried. So why is the first variant (without deleting sys.path_hooks) not recognizing the new extension "xxx" and the second variant (deleting sys.path_hooks) do?
It looks like the machinery is throwing an exception rather than traversing further to the next entry, when an entry of sys.path_hooks is not able to recognize "xxx".
And why is the second version working for py, pyc and xxx modules in the current directory, but not working in the package pack? I would expect that py and pyc is not even working in the current dir, because sys.path_hooks contains only a hook for "xxx"...
The short answer is that the default PathFinder in sys.meta_path isn't meant to have new file extensions and importers added in the same paths it already supports. But there's still hope!
Quick Breakdown
sys.path_hooks is consumed by the importlib._bootstrap_external.PathFinder class.
When an import happens, each entry in sys.meta_path is asked to find a matching spec for the requested module. The PathFinder in particular will then take the contents of sys.path and pass it to the factory functions in sys.path_hooks. Each factory function has a chance to either raise an ImportError (basically the factory saying "nope, I don't support this path entry") or return a finder instance for that path. The first successfully returned finder is then cached in sys.path_importer_cache. From then on PathFinder will only ask those cached finder instances if they can provide the requested module.
If you look at the contents of sys.path_importer_cache, you'll see all of the directory entries from sys.path have been mapped to FileFinder instances. Non-directory entries (zip files, etc) will be mapped to other finders.
Thus, if you append a new factory created via FileFinder.path_hook to sys.path_hooks, your factory will only be invoked if the previous FileFinder hook didn't accept the path. This is unlikely, since FileFinder will work on any existing directory.
Alternatively, if you insert your new factory to sys.path_hooks ahead of the existing factories, the default hook will only be used if your new factory doesn't accept the path. And again, since FileFinder is so liberal with what it will accept, this would lead to only your loader being used, as you've already observed.
Making it Work
So you can either try to adjust that existing factory to also support your file extension and importer (which is difficult as the importers and extension string tuples are held in a closure), or do what I ended up doing, which is add a new meta path finder.
So eg. from my own project,
import sys
from importlib.abc import FileLoader
from importlib.machinery import FileFinder, PathFinder
from os import getcwd
from os.path import basename
from sibilant.module import prep_module, exec_module
SOURCE_SUFFIXES = [".lspy", ".sibilant"]
_path_importer_cache = {}
_path_hooks = []
class SibilantPathFinder(PathFinder):
"""
An overridden PathFinder which will hunt for sibilant files in
sys.path. Uses storage in this module to avoid conflicts with the
original PathFinder
"""
#classmethod
def invalidate_caches(cls):
for finder in _path_importer_cache.values():
if hasattr(finder, 'invalidate_caches'):
finder.invalidate_caches()
#classmethod
def _path_hooks(cls, path):
for hook in _path_hooks:
try:
return hook(path)
except ImportError:
continue
else:
return None
#classmethod
def _path_importer_cache(cls, path):
if path == '':
try:
path = getcwd()
except FileNotFoundError:
# Don't cache the failure as the cwd can easily change to
# a valid directory later on.
return None
try:
finder = _path_importer_cache[path]
except KeyError:
finder = cls._path_hooks(path)
_path_importer_cache[path] = finder
return finder
class SibilantSourceFileLoader(FileLoader):
def create_module(self, spec):
return None
def get_source(self, fullname):
return self.get_data(self.get_filename(fullname)).decode("utf8")
def exec_module(self, module):
name = module.__name__
source = self.get_source(name)
filename = basename(self.get_filename(name))
prep_module(module)
exec_module(module, source, filename=filename)
def _get_lspy_file_loader():
return (SibilantSourceFileLoader, SOURCE_SUFFIXES)
def _get_lspy_path_hook():
return FileFinder.path_hook(_get_lspy_file_loader())
def _install():
done = False
def install():
nonlocal done
if not done:
_path_hooks.append(_get_lspy_path_hook())
sys.meta_path.append(SibilantPathFinder)
done = True
return install
_install = _install()
_install()
The SibilantPathFinder overrides PathFinder and replaces only those methods which reference sys.path_hook and sys.path_importer_cache with similar implementations which instead look in a _path_hook and _path_importer_cache which are local to this module.
During import, the existing PathFinder will try to find a matching module. If it cannot, then my injected SibilantPathFinder will re-traverse the sys.path and try to find a match with one of my own file extensions.
Figuring More Out
I ended up delving into the source for the _bootstrap_external module
https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py
The _install function and the PathFinder.find_spec method are the best starting points to seeing why things work the way they do.
#obriencj's analysis of the situation is correct. But I came up with a different solution to this problem that doesn't require putting anything in sys.meta_path. Instead, it installs a special hook in sys.path_hooks that acts almost as a sort of middle-ware between the PathFinder in sys.meta_path, and the hooks in sys.path_hooks where, rather than just using the first hook that says "I can handle this path!" it tries all matching hooks in order, until it finds one that actually returns a useful ModuleSpec from its find_spec method:
#PathEntryFinder.register
class MetaFileFinder:
"""
A 'middleware', if you will, between the PathFinder sys.meta_path hook,
and sys.path_hooks hooks--particularly FileFinder.
The hook returned by FileFinder.path_hook is rather 'promiscuous' in that
it will handle *any* directory. So if one wants to insert another
FileFinder.path_hook into sys.path_hooks, that will totally take over
importing for any directory, and previous path hooks will be ignored.
This class provides its own sys.path_hooks hook as follows: If inserted
on sys.path_hooks (it should be inserted early so that it can supersede
anything else). Its find_spec method then calls each hook on
sys.path_hooks after itself and, for each hook that can handle the given
sys.path entry, it calls the hook to create a finder, and calls that
finder's find_spec. So each sys.path_hooks entry is tried until a spec is
found or all finders are exhausted.
"""
class hook:
"""
Use this little internal class rather than a function with a closure
or a classmethod or anything like that so that it's easier to
identify our hook and skip over it while processing sys.path_hooks.
"""
def __init__(self, basepath=None):
self.basepath = os.path.abspath(basepath)
def __call__(self, path):
if not os.path.isdir(path):
raise ImportError('only directories are supported', path=path)
elif not self.handles(path):
raise ImportError(
'only directories under {} are supported'.format(
self.basepath), path=path)
return MetaFileFinder(path)
def handles(self, path):
"""
Return whether this hook will handle the given path, depending on
what its basepath is.
"""
path = os.path.abspath(path)
return (self.basepath is None or
os.path.commonpath([self.basepath, path]) == self.basepath)
def __init__(self, path):
self.path = path
self._finder_cache = {}
def __repr__(self):
return '{}({!r})'.format(self.__class__.__name__, self.path)
def find_spec(self, fullname, target=None):
if not sys.path_hooks:
return None
last = len(sys.path_hooks) - 1
for idx, hook in enumerate(sys.path_hooks):
if isinstance(hook, self.__class__.hook):
continue
finder = None
try:
if hook in self._finder_cache:
finder = self._finder_cache[hook]
if finder is None:
# We've tried this finder before and got an ImportError
continue
except TypeError:
# The hook is unhashable
pass
if finder is None:
try:
finder = hook(self.path)
except ImportError:
pass
try:
self._finder_cache[hook] = finder
except TypeError:
# The hook is unhashable for some reason so we don't bother
# caching it
pass
if finder is not None:
spec = finder.find_spec(fullname, target)
if (spec is not None and
(spec.loader is not None or idx == last)):
# If no __init__.<suffix> was found by any Finder,
# we may be importing a namespace package (which
# FileFinder.find_spec returns in this case). But we
# only want to return the namespace ModuleSpec if we've
# exhausted every other finder first.
return spec
# Module spec not found through any of the finders
return None
def invalidate_caches(self):
for finder in self._finder_cache.values():
finder.invalidate_caches()
#classmethod
def install(cls, basepath=None):
"""
Install the MetaFileFinder in the front sys.path_hooks, so that
it can support any existing sys.path_hooks and any that might
be appended later.
If given, only support paths under and including basepath. In this
case it's not necessary to invalidate the entire
sys.path_importer_cache, but only any existing entries under basepath.
"""
if basepath is not None:
basepath = os.path.abspath(basepath)
hook = cls.hook(basepath)
sys.path_hooks.insert(0, hook)
if basepath is None:
sys.path_importer_cache.clear()
else:
for path in list(sys.path_importer_cache):
if hook.handles(path):
del sys.path_importer_cache[path]
This is still, depressing, far more complication than should be necessary. I feel like on Python 2, before the import system rewrite, it was much simpler to do this since less of the support for the built-in module types (.py, etc.) was built on top of the import hooks themselves, so it was harder to break importing normal modules by adding hooks to import new modules types. I'm going to start a discussion on python-ideas to see if there's any way we can't improve this situation.
I came up with yet an alternative tweak. I won't say it is beautiful as it does a closure on an already existing one, but at least short :)
It adds loaders to the default FileLoader objects through a new hook. The original path_hook_for_FileFinder is wrapped in a closure and the loaders are injected into the FileFinder objects returned by the original hook.
After the new hook added the path_importer_cache is cleared as that is already filled with the original FileFinder objects. Those could also be updated dynamically, but I did not bother for now.
Disclaimer: not extensively tested yet. It does what I need in the easiest possible way I know, but the import system is complicated enough to produce funny side-effects for a tweak like this.
import sys
import importlib.machinery
def extend_path_hook_for_FileFinder(*loader_details):
orig_hook, orig_pos = None, None
for i, hook in enumerate(sys.path_hooks):
if hook.__name__ == 'path_hook_for_FileFinder':
orig_hook, orig_pos = hook, i
break
sys.path_hooks.remove(orig_hook)
def extended_path_hook_for_FileFinder(path):
orig_finder = orig_hook(path)
loaders = []
for loader, suffixes in loader_details:
loaders.extend((suffix, loader) for suffix in suffixes)
orig_finder._loaders.extend(loaders)
return orig_finder
sys.path_hooks.insert(orig_pos, extended_path_hook_for_FileFinder)
MY_SUFFIXES = ['.pymy']
class MySourceFileLoader(importlib.machinery.SourceFileLoader):
pass
loader_detail = (MySourceFileLoader, MY_SUFFIXES)
extend_path_hook_for_FileFinder(loader_detail)
# empty cache as it is already filled with simple FileFinder
# objects for the most common path elements
sys.path_importer_cache.clear()
sys.path_importer_cache.invalidate_caches()
I am very new to Python. Reading up on new material for class, I had to copy and paste a few examples to see how things work. So, I copied and pasted code from this website ( https://www.e-education.psu.edu/geog485/node/54 ) under the topic 'Looping in GIS Models'.
Here's mine:
import arcpy
try:
arcpy.env.workspace = "C:\Users\dan and kathryn\Desktop\school\programming\Lesson1"
fcList = arcpy.ListFeatureClasses()
for featureClass in fcList:
arcpy.CopyFeatures_management (featureClass, "C:\Users\dan and kathryn\Desktop\school\programming\Lesson 2\PracticeData/" + featureClass)
except:
print "Script failed to complete"
print arcpy.GetMessages(2)
Here's the error msg:
Traceback (most recent call last):
File "C:\Python27\ArcGIS10.1\Lib\site-packages\pythonWin\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 325, in RunScript
exec codeObject in __main__.__dict__
File "C:\Users\dan and kathryn\Desktop\loops_gis.py", line 1, in <module>
import arcpy
File "C:\Users\dan and kathryn\Desktop\arcpy.py", line 5, in <module>
desc = arcpy.Describe(featureClass)
AttributeError: 'module' object has no attribute 'Describe'
Contacted my teacher and he wrote code and sent it to me but everything matches. All of the backslashes, indentations, everything except for folder locations. Not sure what is going on but he suspects this problem runs beyond what I've written in my code.
I've already had to alter the site-packages folder within pythonWin and it now contains:
# .pth file for the PyWin32 extensions
win32
win32\lib
Pythonwin
C:\Program Files (x86)\ArcGIS\Desktop10.1\bin
C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy
C:\Program Files (x86)\ArcGIS\Desktop10.1\ArcToolbox\Scripts
Not sure what's going on or why 'Describe' is even popping up in the problems
Whenever writing path string, use r"path with backslash". It will take care of the backslash and forward slash problem. Use 'os' module when, the paths needed to be joined.
heres what i had the change (all in the backslashes, I had no idea):
arcpy.env.workspace = "C:/Users/dan and kathryn/Desktop/school/programming/Lesson1"
AND
arcpy.CopyFeatures_management (featureClass, "C:\Users\dan and kathryn\Desktop\school\programming\Lesson 2\PracticeData/" + featureClass)
i have written two modules in python "Sample.py" and "GenericFunctions.py" from "Sample.py", iam calling a function
present in "GenericFunctions.py" but i am not able to call that function getting error as "AttributeError: 'module' object has no attribute 'fn_ElseLog'"
Code in Sample.py:
import GenericFunctions
def sampleee():
g_TCaseID="SD1233"
g_TCDescription="Login"
g_TestData="hi"
g_Result="Fail"
g_Remarks="testing"
g_TargetEnvironment="S1"
g_TargetSystem="Legacy"
g_TargetRegion="EU"
x = GenericFunctions.fn_ElseLog(g_TCaseID, g_TCDescription, g_TestData, g_Result, g_Remarks)
sampleee()
Code in GenericFunctions.py:
def fn_ElseLog(g_TCaseID, g_TCDescription, g_TestData, g_Result, g_Remarks):
print "entered in ElseLog Function"
Output= fn_Output(g_TCaseID, g_TCDescription, g_TestData, g_Result , g_Remarks)
print ("Testcase"+"'"+g_TCDescription+"'"+"execution completed"+"'"+g_TargetEnvironment+"-"+g_TargetRegion)
def fn_Output(p_TCaseID, p_TCDescription, p_TestData, p_Result , p_Remarks):
OutputSheet=""
OutputSheet="\Test"+"_"+g_TargetEnvironment+"_"+g_TargetSystem+"_"+g_TargetRegion+".xlsx"
OutputPath=r"C:\Users\u304080\Desktop\PlayAround\Shakedowns\OutputResultFiles"
#objExcel1 = win32.gencache.EnsureDispatch('Excel.Application')
Outputfile=os.path.exists(OutputPath+OutputSheet)
if Outputfile==True :
print('Output file is present')
else:
print('Output file is not present')
return
objExceloutput = win32.gencache.EnsureDispatch('Excel.Application')
#excel.DisplayAlerts = False
objoutputworkbook = objExceloutput.Workbooks.Open(OutputPath+OutputSheet)
objSheetOutput = objoutputworkbook.Sheets(1)
OutputRowCount =objSheetOutput.UsedRange.Rows.Count
print "OutputRowcount" , OutputRowCount
objSheetOutput.Cells(OutputRowCount+1,1).Value=p_TCaseID
objSheetOutput.Cells(OutputRowCount+1,2).Value=p_TCDescription
objSheetOutput.Cells(OutputRowCount+1,3).Value=p_TestData
objSheetOutput.Cells(OutputRowCount+1,4).Value=p_Result
objSheetOutput.Cells(OutputRowCount+1,4).Font.Bold = True
if p_Result=="Pass":
objSheetOutput.Cells(OutputRowCount+1,1).Font.ColorIndex = 10
else:
objSheetOutput.Cells(OutputRowCount+1,1).Font.ColorIndex = 3
objoutputworkbook.SaveAs(OutputPath)
objSheetOutput=None
objoutputworkbook=None
objExceloutput.Quit()
objExceloutput=None
Can you guys show me a solution for this?
Try:
delete all pyc files in your directory. (to ensure it is not in cached)
print(dir(GenericFunctions)) and print(GenericFunctions.__file__) in Sample.py, to see if you are importing the file you think you are importing.
see if there are other files called GenericFunctions somewhere on your PYTHONPATH (echo $PYTHONPATH).
present in "GenericFunctions.py" but i am not able to call that
function getting error as "AttributeError: 'module' object has
no attribute 'fn_ElseLog'"
As presented, your code looks correct.
Check to make sure both files are in the same directory, then either restart your python or do a reload(GenericFunctions) to make sure you don't have an out-of-date version in the sys.modules cache.
I have been stuck with this error for a couple of hours now. Not sure what is wrong. Below is the piece of code
NameError: global name 'GetText' is not defined
class BaseScreen(object):
def GetTextFromScreen(self, x, y, a, b, noofrows = 0):
count = 0
message = ""
while (count < noofrows):
line = Region(self.screen.x + x, self.screen.y + y + (count * 20), a, b)
message = message + "\n" + line.text()
count += 1
return message
class HomeScreen(BaseScreen):
def GetSearchResults(self):
if self.screen.exists("Noitemsfound.png"):
return 'No Items Found'
else:
return self.GetTextFromScreen(36, 274, 680, 20, 16)
class HomeTests(unittest.TestCase):
def test_001S(self):
Home = HomeScreen()
Home.ResetSearchCriteria()
Home.Search("0009", "Key")
self.assertTrue("0009" in Home.GetSearchResults(), "Key was not returned")
Basescreen class has all the reusable methods applicable across different screens.
Homescreen inherits Basescreen.
In HomeTests test case class, the last step is to Home.GetSearchResults() which in turn calls a base class method and the error.
Note:
I have another screenclass and testcaseclass doing the same which works without issues.
I have checked all the importing statements and is ok
'GetText' in the error message is the name of method initially after which i changed it to GetTextFromScreen
Error message is still pointing to a line 88 in code which is not there any more. Module import/reloading issue?
Try clearing out your *.pyc files (or __pycache__ if using 3+).
You asked:
Error message is still pointing to a line 88 in code which is not there any more. Module import/reloading issue?
Yes. The traceback (error messages) will show the current (newest saved) file, even if you haven't run it yet. You must reload/reimport to get the new file.
The discrepancy comes from the fact that traceback printouts read from the script file (scriptname.py) saved on your drive. However, the program is run either from the module saved in memory, or sometimes from the .pyc file. If you fix an error by changing your script, and save it to your drive, then the same error will still occur if you don't reload it.
If you're running interactively for testing, you can use the reload function:
>>> import mymodule
>>> mymodule.somefunction()
Traceback (most recent call last):
File "mymodule.py", line 3, in somefunction
Here is a broken line
OhNoError: Problem with your file
Now, you fix the error and save mymodule.py, return to your interactive session, but you still get the error, but the traceback shows the fixed line
>>> mymodule.somefunction()
Traceback (most recent call last):
File "mymodule.py", line 3, in somefunction
Here is the fixed line
OhNoError: Problem with your file
So you have to reload the module:
>>> reload(mymodule)
<module 'mymodule' from '/path/to/mymodule.py'>
>>> mymodule.somefunction()
Success!