The python docs state:
A complete Python program is executed in a minimally initialized environment: all built-in and standard modules are available, but none have been initialized, except for sys (various system services), builtins (built-in functions, exceptions and None) and __main__.
This would suggest that only those three modules should be listed as loaded modules with the following code snippet:
import sys
print(sys.modules.keys())
However, running the code snippet using CPython v3.10 (with -S option) returns the following on my PC:
dict_keys(['sys', 'builtins', '_frozen_importlib', '_imp', '_thread', '_warnings', '_weakref', '_io', 'marshal', 'nt', 'winreg', '_frozen_importlib_external', 'time', 'zipimport', '_codecs', 'codecs', 'encodings.aliases', 'encodings', 'encodings.utf_8', 'encodings.cp1252', '_signal', '_abc', 'abc', 'io', '__main__'])
Why are there 22 extra modules loaded at runtime as compared to the "minimally initialized environment" mentioned in the docs?
I am updating my understanding of CPython's extra loaded modules with my own answer below.
What I have found so far:
The majority of the extra modules are active to provide the import keyword functionality, and for text encodings.
During interpreter initialisation, the import functionality is provided by importing _frozen_importlib and _imp (during pycore_interp_init, within init_importlib)
_setup() in importlib's _bootstrap.py then imports _thread, _warnings, and _weakref because they are builtin modules that are explicitly imported during bootstrap, and hence not really extra modules.
Interpreter initialisation then imports _frozen_importlib_external (during init_interp_main, within init_importlib_external)
importlib's _bootstrap_external.py imports four new module dependencies: _io, marshal, nt, and winreg packages. If not on Windows, posix gets imported rather than nt and winreg.
_io is imported because it is a builtin module explicitly imported during bootstrap, and hence not really an extra module.
marshal is imported because it is used to load/dump bytecode from/to .pyc files.
nt/posix is imported because it is used for operating system functions such as reading the current working directory.
winreg is imported because it is used to find modules declared in the windows registry.
As part of importing _frozen_importlib_external, the interpreter initialization then imports zipimport, presumably to allow for opening zip-format python archives
As part of importing zipimport, the only new dependency is the time module which is imported. The only use is time.mktime() to "convert the date/time values found in the Zip archive to a value that's compatible with the time stamp stored in .pyc files"
After _frozen_importlib_external (and thus after import keyword functionality is sorted), the interpreter initialization then imports encodings, presumably for decoding source text.
encodings.aliases is imported because it provides a dictionary of names to map to known encodings.
codecs is imported as it is a dependency of encodings
_codecs is imported, presumably because it is the C version of codecs?
encodings.utf_8 is then imported, presumably because it is the default encoding.
Because we are on Windows, encodings.cp1252 is also imported (encodings.latin_1 is imported instead if on Linux).
The interpreter initialization then imports _signal, presumably for the interpreter to deal with signal handling.
io is then fully imported, presumably to open source files?
abc is then imported as it is a dependency on io?
_abc is then imported, presumably because it is the C version of codecs?
(On Linux, readline is also imported)
And thus 22 extra 'modules' are loaded when using CPython.
This question already has answers here:
Can't import dll module in Python
(6 answers)
Closed 11 months ago.
I am trying to upgrade a library to Python 3.10. Up to now, I have been using 3.7.6.
In my library, I am using DLLs. In the package __init__.py, I add the DLL path to the PATH variable. Since I want to have the code compatible with 32 and 64 bit systems, I check the bitness at runtime and add the appropriate DLL folder to the PATH:
import os
import sys
# Add libs to PATH environment variable such that DLLs can be loaded.
# Load either x64 or x86 depending on whether we are running a 32/64
# bit system.
package_directory = os.path.dirname(os.path.abspath(__file__))
if sys.maxsize > 2**32:
path_dir = os.path.join(package_directory, 'libs', 'x64')
else:
path_dir = os.path.join(package_directory, 'libs', 'x86')
os.environ['PATH'] += os.pathsep + path_dir
The corrresponding folder structure is
package_root
|- libs
|- x64
|- libbristolpolled.dll
...
|- x86
|- libbristolpolled.dll
...
Then in a sub-package, I am using:
from ctypes import CDLL
dll = "libbristolpolled.dll"
_lib_bristlp = CDLL(dll)
This has worked fine in 3.7.6 but fails in 3.10.3 with the following error message:
File "C:\...\lib\site-packages\optoMD\sensor\optical\bristol\dll_api.py", line 37, in <module>
_lib_bristlp = CDLL(dll) # The correct DLL
File "C:\Program Files\Python310\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'libbristolpolled.dll' (or one of its dependencies). Try using the full path with constructor syntax.
If I instead use the absolute path, as the error message suggests, it works:
from ctypes import CDLL
dll = r"C:\...\libs\x64\libbristolpolled.dll"
_lib_bristlp = CDLL(dll)
What is the reason the old method doesn't work anymore, or how can I make it work again? I'd prefer not to have absolute paths in my code and I liked the fact that I handle the DLL path once in the init file and afterwards I just need to import the DLL using the file name.
This particular features (by adding the path to find a .dll to PATH environment variable) has been removed in Python 3.8. As per the notice, the new function os.add_dll_directory is provided; quoting the documentation for the reason for the change:
New in version 3.8: Previous versions of CPython would resolve DLLs using the default behavior for the current process. This led to inconsistencies, such as only sometimes searching PATH or the current working directory, and OS functions such as AddDllDirectory having no effect.
The more portable solution going forward is to attempt to use that function, alternatively just generate the absolute path as you have current done using os.path.dirname and os.path.join on the dll variable - i.e. to maintain compatibility with prior Python versions; also this assumes those .dll files do not have external dependencies, if there are, this thread has a significantly more detail regarding the issues at hand (even with absolute paths - though given that it did appear to work for your program and library, this should not an issue for your use case).
I am trying to link to my own C library from Cython, following the directions I've found on the web, including this answer:
Using Cython To Link Python To A Shared Library
I am running IPython through Spyder.
My setup.py looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy as np
setup(
ext_modules = cythonize(
[Extension("*",["*.pyx"],
libraries =["MyLib"],
extra_compile_args = ["-fopenmp","-O3"],
extra_link_args=["-L/path/to/lib"])
]),
include_dirs = [np.get_include()],
)
The file libMyLib.so is in /path/to/lib and it compiles fine.
I have a Python script in my IPython profile startup folder that does this
try:
os.environ["LD_LIBRARY_PATH"] += ":/path/to/lib"
except KeyError:
os.environ["LD_LIBRARY_PATH"] = "/path/to/lib"
I can confirm that this is running, because if I type os.environ["LD_LIBRARY_PATH"] into the IPython interpreter, it returns /path/to/lib
But when I try to load the Cython module (i.e. import mycythonmodule) I get:
ImportError: libMyLib.so: cannot open shared object file: No such file or directory
I've also tried putting libMyLib.so in other places to see if cython would find it:
In the directory where Python is running
On the Python path
In the same folder as the cython module
But it still doesn't find the shared library. The only way I can get it to find the library is by dropping it in /usr/lib, but I don't want it there, I want to be able to set the library path.
Am I missing something?
I'm self-answering, in case anyone else runs into the same problem. Looks like the answers are here:
Set LD_LIBRARY_PATH before importing in python
Changing LD_LIBRARY_PATH at runtime for ctypes
According to these answers (and my experience), the linker reads LD_LIBRARY_PATH when python is launched, so changing it from within python doesn't have any useful effect, at least not the effect I was hoping for. The only solution is to either wrap python in a shell script that sets LD_LIBRARY_PATH, or else drop the shared object somewhere on the linker search path.
Kind of a pain, but it is what it is.
I have fixed it by change setup.py.
I have a C++ dynamic shared library called "libtmsmdreader.so". and a header file named "TmsMdReader.hpp"
I wrapper C++ shared library to cython library called "tmsmdreader-pythonxxxxxx.so"
from setuptools import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
name="tmsmdreader",
ext_modules=cythonize([
Extension(
name="tmsmdreader",
language="c++",
sources=["TmsMdReaderApi.pyx"],
libraries=["tmsmdreader"],
library_dirs=["."],
include_dirs=["."],
extra_compile_args=["-std=c++14"],
compiler_directives={'language_level': 3},
runtime_library_dirs=["."])
]))
library_dirs=["."] and runtime_library_dirs=["."] can fixed LD_LIBRARY_PATH if libtmsmdreader.so in python scripy directory
I have found most python modules in python source directory, under Python/Lib or Python/Modules ,but where is the sys (import sys) module ? I didn't find it .
The Answer
I find it here: ./Python/sysmodule.c
If you're on Linux or Mac OS X, and in doubt, just try find . -name 'sysmodule.c' in the Python directory.
Other Stuff
The way I found it was by searching for the string "platform" throughout the Python directory (using TextMate), as I've used e.g. sys.platform before from the sys module... something similar can be done with grep and xargs.
Another suggestion could be : for i in ./**/*.c ; do grep -H platform $i ; done
This will loop through all *.c files present in anywhere up the file tree you're currently located at, and search the file for "platform". The -H flag will ensure we get a filename path so we can trace the matches back to the files they are found in.
import sys
help(sys)
Then you will see something like the following:
Help on built-in module sys:
NAME
sys
FILE
(built-in)
In my working environment, sys is built into python itself.
It's in Python/Python/sysmodule.c.
How do I learn where the source file for a given Python module is installed? Is the method different on Windows than on Linux?
I'm trying to look for the source of the datetime module in particular, but I'm interested in a more general answer as well.
For a pure python module you can find the source by looking at themodule.__file__.
The datetime module, however, is written in C, and therefore datetime.__file__ points to a .so file (there is no datetime.__file__ on Windows), and therefore, you can't see the source.
If you download a python source tarball and extract it, the modules' code can be found in the Modules subdirectory.
For example, if you want to find the datetime code for python 2.6, you can look at
Python-2.6/Modules/datetimemodule.c
You can also find the latest version of this file on github on the web at
https://github.com/python/cpython/blob/main/Modules/_datetimemodule.c
Running python -v from the command line should tell you what is being imported and from where. This works for me on Windows and Mac OS X.
C:\>python -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# C:\Python24\lib\site.pyc has bad mtime
import site # from C:\Python24\lib\site.py
# wrote C:\Python24\lib\site.pyc
# C:\Python24\lib\os.pyc has bad mtime
import os # from C:\Python24\lib\os.py
# wrote C:\Python24\lib\os.pyc
import nt # builtin
# C:\Python24\lib\ntpath.pyc has bad mtime
...
I'm not sure what those bad mtime's are on my install!
I realize this answer is 4 years late, but the existing answers are misleading people.
The right way to do this is never __file__, or trying to walk through sys.path and search for yourself, etc. (unless you need to be backward compatible beyond 2.1).
It's the inspect module—in particular, getfile or getsourcefile.
Unless you want to learn and implement the rules (which are documented, but painful, for CPython 2.x, and not documented at all for other implementations, or 3.x) for mapping .pyc to .py files; dealing with .zip archives, eggs, and module packages; trying different ways to get the path to .so/.pyd files that don't support __file__; figuring out what Jython/IronPython/PyPy do; etc. In which case, go for it.
Meanwhile, every Python version's source from 2.0+ is available online at http://hg.python.org/cpython/file/X.Y/ (e.g., 2.7 or 3.3). So, once you discover that inspect.getfile(datetime) is a .so or .pyd file like /usr/local/lib/python2.7/lib-dynload/datetime.so, you can look it up inside the Modules directory. Strictly speaking, there's no way to be sure of which file defines which module, but nearly all of them are either foo.c or foomodule.c, so it shouldn't be hard to guess that datetimemodule.c is what you want.
If you're using pip to install your modules, just pip show $module the location is returned.
The sys.path list contains the list of directories which will be searched for modules at runtime:
python -v
>>> import sys
>>> sys.path
['', '/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', ... ]
from the standard library try imp.find_module
>>> import imp
>>> imp.find_module('fontTools')
(None, 'C:\\Python27\\lib\\site-packages\\FontTools\\fontTools', ('', '', 5))
>>> imp.find_module('datetime')
(None, 'datetime', ('', '', 6))
datetime is a builtin module, so there is no (Python) source file.
For modules coming from .py (or .pyc) files, you can use mymodule.__file__, e.g.
> import random
> random.__file__
'C:\\Python25\\lib\\random.pyc'
Here's a one-liner to get the filename for a module, suitable for shell aliasing:
echo 'import sys; t=__import__(sys.argv[1],fromlist=[\".\"]); print(t.__file__)' | python -
Set up as an alias:
alias getpmpath="echo 'import sys; t=__import__(sys.argv[1],fromlist=[\".\"]); print(t.__file__)' | python - "
To use:
$ getpmpath twisted
/usr/lib64/python2.6/site-packages/twisted/__init__.pyc
$ getpmpath twisted.web
/usr/lib64/python2.6/site-packages/twisted/web/__init__.pyc
In the python interpreter you could import the particular module and then type help(module). This gives details such as Name, File, Module Docs, Description et al.
Ex:
import os
help(os)
Help on module os:
NAME
os - OS routines for Mac, NT, or Posix depending on what system we're on.
FILE
/usr/lib/python2.6/os.py
MODULE DOCS
http://docs.python.org/library/os
DESCRIPTION
This exports:
- all functions from posix, nt, os2, or ce, e.g. unlink, stat, etc.
- os.path is one of the modules posixpath, or ntpath
- os.name is 'posix', 'nt', 'os2', 'ce' or 'riscos'
et al
On windows you can find the location of the python module as shown below:i.e find rest_framework module
New in Python 3.2, you can now use e.g. code_info() from the dis module:
http://docs.python.org/dev/whatsnew/3.2.html#dis
Check out this nifty "cdp" command to cd to the directory containing the source for the indicated Python module:
cdp () {
cd "$(python -c "import os.path as _, ${1}; \
print _.dirname(_.realpath(${1}.__file__[:-1]))"
)"
}
Just updating the answer in case anyone needs it now, I'm at Python 3.9 and using Pip to manage packages. Just use pip show, e.g.:
pip show numpy
It will give you all the details with the location of where pip is storing all your other packages.
On Ubuntu 12.04, for example numpy package for python2, can be found at:
/usr/lib/python2.7/dist-packages/numpy
Of course, this is not generic answer
Another way to check if you have multiple python versions installed, from the terminal.
$ python3 -m pip show pyperclip
Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
$ python -m pip show pyperclip
Location: /Users/umeshvuyyuru/Library/Python/2.7/lib/python/site-packages
Not all python modules are written in python. Datetime happens to be one of them that is not, and (on linux) is datetime.so.
You would have to download the source code to the python standard library to get at it.
For those who prefer a GUI solution: if you're using a gui such as Spyder (part of the Anaconda installation) you can just right-click the module name (such as "csv" in "import csv") and select "go to definition" - this will open the file, but also on the top you can see the exact file location ("C:....csv.py")
If you are not using interpreter then you can run the code below:
import site
print (site.getsitepackages())
Output:
['C:\\Users\\<your username>\\AppData\\Local\\Programs\\Python\\Python37', 'C:\\Users\\<your username>\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages']
The second element in Array will be your package location. In this case:
C:\Users\<your username>\AppData\Local\Programs\Python\Python37\lib\site-packages
In an IDE like Spyder, import the module and then run the module individually.
enter image description here
as written above
in python just use help(module)
ie
import fractions
help(fractions)
if your module, in the example fractions, is installed then it will tell you location and info about it, if its not installed it says module not available
if its not available it doesn't come by default with python in which case you can check where you found it for download info