Develop Reference

Running a module workaround python "Scalene"

I am trying to debug a memory leak in a module using Scalene.
Unfortunately, it appears that I can only run scalene script.py while I need to be able to specify the module to correctly run the application with python -m mymodule, which I can't seem to do with scalene.
Is there a way to overcome this? Thank you in advance

cf Scalene's documentation :
scalene your_prog.py # full profile (prints to console)
python3 -m scalene your_prog.py # equivalent alternative
You can use the second form with Scalene.

You can use runpy.run_module() to create a wrapper around your module, which you can then profile!
wrapper.py might contain:
from runpy import run_module
run_module('your_module_name', run_name='__main__')
and then you can run scalene wrapper.py!
The run_name argument is needed in order to "trick" the if __name__ == '__main__' clause into executing, if you have one.

How do I debug a built-in Python command, package or module?

I would like to debug some of the basic packages that come with the Python install and/or are built-in packages, including pip and venv.
The desire comes from an error message of file permissions (unable to access a file with an "unprintable file name") some of my team is getting running these commands - see this question for details.
Question
How do you debug the Python source code when trying to catch issues in the main python executable, or when directly running a base python module (see following examples for pip and venv)?
$ python -m pip install --upgrade
$ python -m venv .venv
If it matters, my environment is VSCode, where I am happily able to engage the debugger on any custom script I have written, using the built-in debugger that interacts (I assume) with the main Microsoft Python extension.

You will need to set "justMyCode": false in your launch.json for the debugger to trace into third-party code.

Start by looking at the source code for those modules; the -m switch looks for a package or module to import first. If it's a package, then Python imports the __main__ module in that package and runs it as the main script. If it is a module, the module itself is imported and run as __main__.
Usually the code is structured such that a function is called you can import directly too. You can then just write a bit of code that imports the same function and calls it the same way the __main__ module would. From there on out it is trivial to run this under a debugger.
E.g. pip is a package, so python -m pip will import pip.__main__ and run that as a script. This then triggers:
from pip._internal.cli.main import main as _main # isort:skip # noqa
if __name__ == '__main__':
sys.exit(_main())
to be run. You can do the same in VSCode; import pip._internal.cli.main.main and call it.
You can find the source code for these modules by just importing them and printing out the resulting object:
python -c "import pip; print(pip)"
The representation of a module, if loaded from disk, will include it's filename. If the filename ends in /__init__.py it's a package, so you can also double-check that the __main__.py file exists:
python -c "import pip.__main_; print(pip.__main__)"
You can do the same for the venv module. This one is part of the Python standard library, so the documentation actually links directly to the source code, and the venv.__main__ module just imports venv.main() and calls it.

python gives ImportError: No module named "" when triggered in a perl program

In my perl program which runs the python script
I have provided the PYTHONPATH env param with the path for the lib and i have run the python script. I am getting
ImportError: No module named "....."
my $script = "/path/pythonscript.py";
$ENV{'PYTHONPATH'} = "/path/lib";
system("python $script");
Whereas when i run the same python script on command line in the same directory where the script executes in my perl program, it is working.
Can anyone give me some pointers on why this is happening.

Try printing the contents of sys.path and compare the difference e.g. change your python script to
import sys
print(sys.path)
Most likely there is a difference here and this is causing the module to not be found.

I once had a similar problem. I solved it by creating an executable script (chmod) and making that script run instead of the python script. The script simply contained a cd to the directory and a python3 program. py

Unit testing in Python

This is my first project in Python and I have just learnt the unittest framework. The test module runs well when I do python test_module.py but when I want to execute a certain class or a method as said in the documentation using:
python -m unittest test_module.TestClass.test_method # or even just test_module
I get the following error:
AttributeError: 'module' object has no attribute 'test_module'
The directory where I run the command contains graphm_test.py (I also tried to change the name to test_graphm.py), with class graphm_test(unittest.TestCase): and methods all starting with test_* and here is the command I run on the terminal:
python -m unittest test_graphm.py
I could not find a similar problem to this anywhere, it would be great to know the reason behind the error and how to run a certain class inside the module or a certain method

There are 2 issues: you are not calling the correct module name, and you are using the extension .py.
So you need to be in the folder with your graphm_test.py file, and run:
python -m unittest graphm_test

executing standalone fabric script by calling it by its name, without the .py extension

I have a fabric script called fwp.py that I run without calling it throug fab by using:
if __name__ == '__main__':
# imports for standalone mode only
import sys
import fabric.main
fabric.main.main(fabfile_locations=[__file__])
The thing is then have to call the script by calling fwp.py. I'd like to rename it as fwp to be able to call it as fwp. But doing that would result in
Fatal error: Couldn't find any fabfiles!
Is there a way to make Python/Fabric import this file, despite the lack of a ".py" extension?
To reiterate and clarify:
I'm not using the "fab" utility (e.g. as fab task task:parameter); just calling my script as fwp.py task task:parameter, and would like to be able to call it as fwp task task:parameter.
Update
It's not a duplicate of this question. The question is not "How to run a stand-alone fabric script?", but "How to do so, while having a script without a .py" extension.

EDIT: Original answer corrected
The fabric.main.main() function automatically adds .py to the end of supplied fabfile locations (see https://github.com/fabric/fabric/blob/master/fabric/main.py#L93). Unfortunately that function also uses Python's import machinery to load the file so it has to look like a module or package. Without reimplementing much of the fabric.main module I don't think it will be possible. You could try monkey-patching both fabric.main.find_fabfiles and fabric.main.load_fabfiles to make it work.
Origininal answer (wrong)
I can get this to work unaltered on a freshly installed fabric package. The following will execute with a filename fwp and executable permission on version 1.10.1, Python2.7. I would just try upgrading fabric.
#!/usr/bin/env python
from fabric.api import *
import fabric.main
def do():
local('echo "Hello World"')
if __name__ == '__main__':
fabric.main.main(fabfile_locations=[__file__])
Output:
$ ./fwp do
Hello World
Done