line_profiler is a great Python package to find performance bottlenecks. The only complication when using it is that we have to specify each profiled module one by one on the command line:
%lprun -m toppkg.pkg1.module11 -m toppkg.pkg1.module12 ... -m toppkg.pkgN.moduleNK my_entry_point()
This can be a tedious task in a project of tens of thousands of line.
I wonder if anyone knows an automated approach, where we can specify all sub-packages and modules below a particular package like:
%mylprun -p toppkg my_entry_point()
Related
Usually, python libraries such as numpy or matplotlib are used with "import numpy" in a .py script, then we call the functions we imported.
However some libraries such as "pip" or "anaconda" are not imported in a script, but rather they are command-based: called from a terminal using arguments and options such as "pip install [options] [whatever]".
Today I installed a library and it took me a while to realise it was command-based, and wouldn't work if I just ran the script "main.py" it contained.
My question is: what is the fundamental difference between those 2 kind of libraries? How do I learn to make command-based libraries?
Basically I just want to understand them, but don't even know where to begin my research.
I want to run a library like "library ...".
There is a major misunderstanding here. pip is both a command-line script and a library.
The script resides in bin/pip, for example /usr/bin/pip (on w32 it's Scripts\pip).
The library resides somewhere in sys.path, perhaps in site-packages/pip, for example /usr/lib/python3.8/site-packages/pip/.
And of course the command-line script is written in Python and it imports the library pip. This main.py is converted to script pip by setuptools.
If you want to do the same — create a command-line script that imports your library — you have to go the same route. Create a library and a script or an entry points.
Let's say I have a standard python directory structure of python package like here and consider I need to add a function to the package. More specifically, I want to do it with trial-and-error, by running a test code. What is the correct work-flow for this?
I currently do the following:
do sudo python setup.py install anytime I made a change in the package
source ~/.bashrc
open a python interpreter,
run the test code.
But apparently this flow takes a lot of time to just check the modification via test code. And I feel that I'm doing something wrong, and the better ways exist.
I would comment, but alas I do not have enough reputation.
Use pip install -e path/to/package. This will install it "editably" and locally, so any change to the package code is effective immediately on your system. This way you can change and test code on the fly.
Further answers: https://stackoverflow.com/a/23075617/12164878
I found out that I can discover and run unit tests under my directory tree by doing this:
python3 -m test
The above works, but the documented method to discover and run all tests finds hundreds more, including a new one that was not found by the previous method:
python3 -m unittest
What exactly is -m test and why can't I find documentation on it after a quick search, except the following page which seems to be about CPython?
https://devguide.python.org/runtests/
The test package is intended to test the Python API itself. According to the documentation:
Note: The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.
The link to this documentation appears in the TOC under Development Tools. While it is not entirely surprising that the python3 -m test command discovers and runs tests, it is not really designed to discover and run the tests that you write for your own code.
I have a single Python file which is supposed to take in a bunch of inputs during the command.
For eg: python script.py "string_1" "string_2"
I also have a bunch of dependencies including pandas, datetime and even Python3.
I want to package all this code in a manner that anyone can install the package along with the dependencies as well (in a directory or so) and then just call the script/module : in the above manner. Without having to actually go into a Python interpreter.
I tried using the python-packaging resource, but with that I would need to go into the interpreter, right ?
I found a good article today that explains quite well the procedure: https://medium.com/dreamcatcher-its-blog/making-an-stand-alone-executable-from-a-python-script-using-pyinstaller-d1df9170e263
pyinstaller --onefile <script.py> is the tl;dr on linux. On windows you need also py32exe
If you can rely on a base install of python being present already.
Then it's worth looking at Python's zipapp module introduced in Python3.5 https://docs.python.org/3/library/zipapp.html#creating-standalone-applications-with-zipapp For background info PEP441 https://www.python.org/dev/peps/pep-0441/
Also there is a project called Shiv which adds some extra abilities to the zipapp module bundled in python3.5
https://shiv.readthedocs.io/en/latest/
Have a look at pex (https://pex.readthedocs.io/en/stable/). It wraps up your python scripts, files, dependencies, etc into a single executable. You still need the python interpreter installed, but it includes everything else.
For my simple C library I have a simple Python wrapper using ctypes. I am now writing the Makefile for my library. In it under the install target I have:
install -m 644 mylib.py $(shell python3 -m site | grep $(PREFIX).*packages | cut -d \' -f2)/
This is something I contrived myself. I don't want to use distutils or all those multiple Python installation "solutions" because they are too convoluted for my simple requirements: Once the .so file is installed, all that is really needed for my library to be accessible from Python is for the .py interface to be in the correct import-able location. So I'm figuring a simple copy should do.
Does anyone foresee any problem with what I am doing? If so, is there a better way to do this?
Thanks!
By using setup-tools and uploading your source files to pipy.org, anyone on the world can install your project by simply typing pip install <yourmodule>.
Moreover, larger Python projects which happen to want to use your library will just have to list its name in a single line, either in a requirements.txt file, or on setup.py
if your project is not open source and you don't intend it to be accessible by "the world", the distutils solutions - no quotes needed - allow your project to be fetched from a private git repository, or even a prevate Python "Cheese Shop".
If you absolutely don't care about using the right way to do things, you might as well just drop a plain English line on your README telling the user to copy your ".py" file to any folder he likes - installation won't be as expected by Python users anyway.
Otherwise, just have a minimalist setup.py file and have a make line that goes python setup.py install - it certainly won't hurt (your library will still be short of usable on larger projects that uses several packages - all those projects need to be pip-installable )
The minimalist setup.py file is a two liner and can be found here:
https://docs.python.org/2/distutils/examples.html#pure-python-distribution-by-module - certainly far from "convoluted".