So when I try to find the path from the cmd with where azuredatastudio, I get the path. When I go in Python and do print(os.environ), I get many defined paths, but not this from the upper command in cmd.
How to get in this example azuredatastudio path from Python and where is it stored?
The WHERE command is roughly equivalent to the UNIX 'which' command. By default, the search is done in the current directory and in the PATH.
Source: https://ss64.com/nt/where.html
So you'll have to explicitly look at the paths in the PATH environment variable: os.environ['PATH']. You'll find an implementation in this question here for example: Test if executable exists in Python?
Also, you should be able to just run the command from Python:
from subprocess import check_output
path = check_output(["where", "azuredatastudio"])
print(path)
A easy way to do this is:
import os
os.system("where azuredatastudio")
or if you want to save it in a variable.
import subprocess
process = subprocess.Popen("where azuredatastudio",stdout=subprocess.PIPE)
print(process.stdout.readline())
Directory structure:
Application/
proto/
payload.proto
lab_account.proto
public_trail.proto
protocompiled/
payload_pb2.py
lab_account_pb2.py
Contents of the payload.proto
syntax = "proto3";
import "lab_account.proto";
import "public_trail.proto";
if I compile my payload.proto file, with command .
Application⟫ protoc --proto_path=./proto --python_out=./protocompiled payload.proto
The compiled payload_pb2.py doest have the required imports. It has wrong import statement like this.
import lab_account_pb2 as lab__account__pb2
import public_trail_pb2 as public__trail__pb2
instead of this;
import protocompiled.lab_account_pb2 as lab__account__pb2
import protocompiled.public_trail_pb2 as public__trail__pb2
Also referred, https://github.com/protocolbuffers/protobuf/issues/1491 but couldnt solve the issue.
There is an open issue 5374 in protobuf github regarding this problem.
Until it is resolved, I use the following workaround: after protoc run a sed script (works at least for GNU sed) which will add relative imports.
protoc $PATH/*.proto --python_out=$PROTOC_OUTDIR
sed -i $PROTOC_OUTDIR/*_pb2.py -e 's/^import [^ ]*_pb2/from . \0/'
But this works only when all files are in the same directory. More complex script will be required to handle files spread over multiple directories.
I would like to check if all modules imported by a script are installed before I actually run the script, because the script is quite complex and is usually running for many hours. Also, it may import different modules depending on the options passed to it, so just running it once may not check everything. So, I wouldn't like to run this script on a new system for few hours only to see it failing before completion because of a missing module.
Apparently, pyflakes and pychecker are not helpful here, correct me if I'm wrong. I can do something like this:
$ python -c "$(cat *.py|grep import|sed 's/^\s\+//g'|tr '\n' ';')"
but it's not very robust, it will break if the word 'import' appears in a string, for example.
So, how can I do this task properly?
You could use ModuleFinder from the standard lib modulefinder
Using the example from the docs
from modulefinder import ModuleFinder
finder = ModuleFinder()
finder.run_script('bacon.py')
print 'Loaded modules:'
for name, mod in finder.modules.iteritems():
print '%s: ' % name,
print ','.join(mod.globalnames.keys()[:3])
print '-'*50
print 'Modules not imported:'
print '\n'.join(finder.badmodules.iterkeys())
You could write a test.py that just contains all the possible imports for example:
import these
import are
import some
import modules
Run it and if there are any problems python will let you know
I'm running the following MapReduce on AWS Elastic MapReduce:
./elastic-mapreduce --create --stream --name CLI_FLOW_LARGE --mapper
s3://classify.mysite.com/mapper.py --reducer
s3://classify.mysite.com/reducer.py --input
s3n://classify.mysite.com/s3_list.txt --output
s3://classify.mysite.com/dat_output4/ --cache
s3n://classify.mysite.com/classifier.py#classifier.py --cache-archive
s3n://classify.mysite.com/policies.tar.gz#policies --bootstrap-action
s3://classify.mysite.com/bootstrap.sh --enable-debugging
--master-instance-type m1.large --slave-instance-type m1.large --instance-type m1.large
For some reason the cacheFile classifier.py is not being cached, it would seem. I get this error when the reducer.py tries to import it:
File "/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201204290242_0001/attempt_201204290242_0001_r_000000_0/work/./reducer.py", line 12, in <module>
from classifier import text_from_html, train_classifiers
ImportError: No module named classifier
classifier.py is most definitely present at s3n://classify.mysite.com/classifier.py. For what it's worth, the policies archive seems to load in just fine.
I don't know how to fix this problem in EC2, but I've seen it before with Python in traditional Hadoop deployments. Hopefully the lesson translates over.
What we need to do is add the directory reduce.py is in to the python path, because presumably classifier.py is in there too. For whatever reason, this place is not in the python path, so it is failing to find classifier.
import sys
import os.path
# add the directory where reducer.py is to the python path
sys.path.append(os.path.dirname(__file__))
# __file__ is the location of reduce.py, along with "reduce.py"
# dirname strips the file name and only gives the directory
# sys.path is the python path where it looks for modules
from classifier import text_from_html, train_classifiers
The reason why your code might work locally is because of the current working directory in which you are running it from. Hadoop might not be running it from the same place you are in terms of the current working directory.
orangeoctopus deserves credit for this from his comment. Had to append the working directory system path:
sys.path.append('./')
Also, I recommend anyone who has similar issues to me to read this great article on using Distributed Cache on AWS:
https://forums.aws.amazon.com/message.jspa?messageID=152538
How do I learn where the source file for a given Python module is installed? Is the method different on Windows than on Linux?
I'm trying to look for the source of the datetime module in particular, but I'm interested in a more general answer as well.
For a pure python module you can find the source by looking at themodule.__file__.
The datetime module, however, is written in C, and therefore datetime.__file__ points to a .so file (there is no datetime.__file__ on Windows), and therefore, you can't see the source.
If you download a python source tarball and extract it, the modules' code can be found in the Modules subdirectory.
For example, if you want to find the datetime code for python 2.6, you can look at
Python-2.6/Modules/datetimemodule.c
You can also find the latest version of this file on github on the web at
https://github.com/python/cpython/blob/main/Modules/_datetimemodule.c
Running python -v from the command line should tell you what is being imported and from where. This works for me on Windows and Mac OS X.
C:\>python -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# C:\Python24\lib\site.pyc has bad mtime
import site # from C:\Python24\lib\site.py
# wrote C:\Python24\lib\site.pyc
# C:\Python24\lib\os.pyc has bad mtime
import os # from C:\Python24\lib\os.py
# wrote C:\Python24\lib\os.pyc
import nt # builtin
# C:\Python24\lib\ntpath.pyc has bad mtime
...
I'm not sure what those bad mtime's are on my install!
I realize this answer is 4 years late, but the existing answers are misleading people.
The right way to do this is never __file__, or trying to walk through sys.path and search for yourself, etc. (unless you need to be backward compatible beyond 2.1).
It's the inspect module—in particular, getfile or getsourcefile.
Unless you want to learn and implement the rules (which are documented, but painful, for CPython 2.x, and not documented at all for other implementations, or 3.x) for mapping .pyc to .py files; dealing with .zip archives, eggs, and module packages; trying different ways to get the path to .so/.pyd files that don't support __file__; figuring out what Jython/IronPython/PyPy do; etc. In which case, go for it.
Meanwhile, every Python version's source from 2.0+ is available online at http://hg.python.org/cpython/file/X.Y/ (e.g., 2.7 or 3.3). So, once you discover that inspect.getfile(datetime) is a .so or .pyd file like /usr/local/lib/python2.7/lib-dynload/datetime.so, you can look it up inside the Modules directory. Strictly speaking, there's no way to be sure of which file defines which module, but nearly all of them are either foo.c or foomodule.c, so it shouldn't be hard to guess that datetimemodule.c is what you want.
If you're using pip to install your modules, just pip show $module the location is returned.
The sys.path list contains the list of directories which will be searched for modules at runtime:
python -v
>>> import sys
>>> sys.path
['', '/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', ... ]
from the standard library try imp.find_module
>>> import imp
>>> imp.find_module('fontTools')
(None, 'C:\\Python27\\lib\\site-packages\\FontTools\\fontTools', ('', '', 5))
>>> imp.find_module('datetime')
(None, 'datetime', ('', '', 6))
datetime is a builtin module, so there is no (Python) source file.
For modules coming from .py (or .pyc) files, you can use mymodule.__file__, e.g.
> import random
> random.__file__
'C:\\Python25\\lib\\random.pyc'
Here's a one-liner to get the filename for a module, suitable for shell aliasing:
echo 'import sys; t=__import__(sys.argv[1],fromlist=[\".\"]); print(t.__file__)' | python -
Set up as an alias:
alias getpmpath="echo 'import sys; t=__import__(sys.argv[1],fromlist=[\".\"]); print(t.__file__)' | python - "
To use:
$ getpmpath twisted
/usr/lib64/python2.6/site-packages/twisted/__init__.pyc
$ getpmpath twisted.web
/usr/lib64/python2.6/site-packages/twisted/web/__init__.pyc
In the python interpreter you could import the particular module and then type help(module). This gives details such as Name, File, Module Docs, Description et al.
Ex:
import os
help(os)
Help on module os:
NAME
os - OS routines for Mac, NT, or Posix depending on what system we're on.
FILE
/usr/lib/python2.6/os.py
MODULE DOCS
http://docs.python.org/library/os
DESCRIPTION
This exports:
- all functions from posix, nt, os2, or ce, e.g. unlink, stat, etc.
- os.path is one of the modules posixpath, or ntpath
- os.name is 'posix', 'nt', 'os2', 'ce' or 'riscos'
et al
On windows you can find the location of the python module as shown below:i.e find rest_framework module
New in Python 3.2, you can now use e.g. code_info() from the dis module:
http://docs.python.org/dev/whatsnew/3.2.html#dis
Check out this nifty "cdp" command to cd to the directory containing the source for the indicated Python module:
cdp () {
cd "$(python -c "import os.path as _, ${1}; \
print _.dirname(_.realpath(${1}.__file__[:-1]))"
)"
}
Just updating the answer in case anyone needs it now, I'm at Python 3.9 and using Pip to manage packages. Just use pip show, e.g.:
pip show numpy
It will give you all the details with the location of where pip is storing all your other packages.
On Ubuntu 12.04, for example numpy package for python2, can be found at:
/usr/lib/python2.7/dist-packages/numpy
Of course, this is not generic answer
Another way to check if you have multiple python versions installed, from the terminal.
$ python3 -m pip show pyperclip
Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
$ python -m pip show pyperclip
Location: /Users/umeshvuyyuru/Library/Python/2.7/lib/python/site-packages
Not all python modules are written in python. Datetime happens to be one of them that is not, and (on linux) is datetime.so.
You would have to download the source code to the python standard library to get at it.
For those who prefer a GUI solution: if you're using a gui such as Spyder (part of the Anaconda installation) you can just right-click the module name (such as "csv" in "import csv") and select "go to definition" - this will open the file, but also on the top you can see the exact file location ("C:....csv.py")
If you are not using interpreter then you can run the code below:
import site
print (site.getsitepackages())
Output:
['C:\\Users\\<your username>\\AppData\\Local\\Programs\\Python\\Python37', 'C:\\Users\\<your username>\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages']
The second element in Array will be your package location. In this case:
C:\Users\<your username>\AppData\Local\Programs\Python\Python37\lib\site-packages
In an IDE like Spyder, import the module and then run the module individually.
enter image description here
as written above
in python just use help(module)
ie
import fractions
help(fractions)
if your module, in the example fractions, is installed then it will tell you location and info about it, if its not installed it says module not available
if its not available it doesn't come by default with python in which case you can check where you found it for download info