I am developing a Python application, and using Conda to distribute it and manage dependencies.
I have successfully run conda-build to wrap up my code into a .tar.bz2 file. I created a new conda environment, and tested installing the package. Everything works, and I am able to run my code this way.
What I am wondering is, how do I structure my package and/or Python code, so that I can run my script by simply typing with my conda environment active, rather than navigating to the folder where I developed the code?
For example, in this conda cheatsheet: https://conda.io/docs/_downloads/conda-cheatsheet.pdf, the fourth line from the top, "Run a package after install, example Spyder" the command to run is simply "spyder".
My Python program is launched from a single .py file, launch.py, but how do I let conda know which script to call? When I do now, it simply says "command not found".
This section in the metadata specification seems relevant: https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#entry-point
I tried setting that to "entry: python launch.py" in my meta.yaml, rebuilding and reinstalling, but that did not work and I am not sure where to go from here.
Any help would be tremendously appreciated, thank you.
Edit: After more digging, I may have found the answer, it seems I want to create an entry point as described here: https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins Still trying to put it all together though.
Let's say your package structure is something like
my_package/
├── my_package/
│ ├── __init__.py
| ├── __main__.py
├── conda.recipe/
│ ├── meta.yaml
├── setup.py
The conda recipe in meta.yaml doesn't need to go in a conda.recipe folder, it's just convenient if you want to store the recipe with the rest of the code. Now, in __main__.py, you have some code that you want to call (this code could be in any module, __main__.py happens to be a convenient example because code in there is also executed if you run python -m my_package). The code in __main__.py might be like
import sys
# Import whatever else from your package or otherwise to run the code
def main(argv):
# Process command line arguments, etc
if __name__ == '__main__':
sys.exit(main())
Now, in your meta.yaml, you'll have something like:
package:
name: my_package
version: '1.0.0'
source:
path: ..
build:
number: 0
script: python setup.py install --single-version-externally-managed --record=record.txt
entry_points:
- my_package_script = my_package.__main__:main
requirements:
- ...
The key line is my_package_script = my_package.__main__:main. The syntax here is the same as for setup.py entry points:
script name = package name.module name:function name
So, in our example, we're using the main function from the __main__.py module in the my_package package.
For a complete example, you can take a look at my package here: https://github.com/bryanwweber/CanSen
Related
I'm working on a project with this folder structure using pytest for tests
project
├── example
│ ├── __init__.py
│ └── example.py
├── setup.py
└── tests
└── test_1.py
I'm trying to import example.py from test_1.py but every way I've tried hasn't worked.
I've tried using sys.path.append('../) then import example and just using from ... import example but neither of these methods has worked. sys.path.append('../') didn't let me import example and from ... import example errored with ImportError: attempted relative import with no known parent package. I've also tried using python setup.py develop and it let me import example, but none of the methods are imported.
This is the code that is in test_1.py. It currently imports example but it has no content.
import sys
import os
sys.path.append("../example/")
import example
example.test()
Contents of example.py
class test:
def __init__(self):
self.a = 1
self.b = 2
The setup.py is the template from here
I think your problem is that the path resolution isn't functioning as you expect. In your example.py file do this:
import sys
import os
dir_path = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.realpath(f"{dir_path}/../example"))
import example
example.test()
Then, when in your root directory run python tests/test_1.py. This should execute test() from the example directory.
The realpath() function is the key here, as it will resolve the actual system path. If that isn't resolved it assumes whatever you're handing it is from the root (i.e. /example, which is wrong).
Use of sys.path.append results in flake8 reporting E402 module level import not at the top of the file. There are various hacks to this around which essentially depend on disabling E402; directly or by deception. There are at least two other ways.
Add __init__.py to the tests directory so that pytest loads the tests from within the parent dir and natively finds the include directory.
pip install -e . from within the parent and have the package loaded in a developer environment.
For the sake of consistency I started with uninstalling every package (so pip freeze returns nothing) and creating virtual environments for each of the approaches. I omitted the setup.py file favoring a minimal pyproject.toml file. Then pip install pytest and pip install flake8 in both. For packages, the second approach then shows an additional example package, i.e.
(.venv) PS C:\xxxx\Stackoverflow\proj_hatch> pip freeze
attrs==22.1.0
colorama==0.4.5
# Editable install with no version control (example==0.0.1)
-e c:\xxxx\stackoverflow\proj_hatch
flake8==5.0.4
iniconfig==1.1.1
mccabe==0.7.0
packaging==21.3
pluggy==1.0.0
py==1.11.0
pycodestyle==2.9.1
pyflakes==2.5.0
pyparsing==3.0.9
pytest==7.1.3
tomli==2.0.1
The net outcome is that both approaches have a working pytest.
(.venv) PS C:\xxxx\Stackoverflow\proj_hatch> pytest
================ test session starts =========================
platform win32 -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
rootdir: C:\xxxx\Stackoverflow\proj_hatch
collected 1 item
tests\test_1.py . [100%]
=================== 1 passed in 0.01s ========================
(.venv) PS C:\xxxx\Stackoverflow\proj_hatch>
In both versions set example/__init__.py as below. This has an additional benefit of allowing the structure (which function in which file) to change later while keeping the import for other applications stable.
from example.example import test
__all__ = [
"test",
]
In both versions the test is:
import example
def test_1():
ex1 = example.Test()
assert ex1.b == 2
With the first approach it is necessary to add .\tests\__init__.py so that pytest loads the test from within the project directory and is able to find the example directory. I expect this has some implications for how you write tests.
With the second approach I started off using hatch however it seemed to work with little more than moving the example directory to src\example and by creating a pyproject.toml file. I've a suspicion there is some magic in the later versions of python that help. I am using 3.10.8 for this exercise.
[project]
name = "example"
version = "0.0.1"
Example:
sys.path.insert(0, "path_to_your_project")
I'm trying to run a simple project in python but I have always the same problem and it drives me crazy every single time.
My project structure looks like the following:
└── myproject
├── mypackage
├── __init__.py
├── a.py
├── b.py
With vscode I open up the myproject folder. When I click on run in the a.py file which looks like
# a.py
from mypackage import b
.
.
.
I always get ModuleNotFoundError: No module named 'mypackage'. What is confusing to me is that vscode itself sees it correctly and shows the text colored (means it detects the module correctly).
Edit:
I want to use myproject as a package so just import b won't help.
The solution I found is actually relatively easy. Just type python3 -m mypackage.a into the command line in the top myproject directory. Worked for me.
I know this has nothing to do with vscode but I was just confused why vscode seems to accept the import but the python interpreter didn't.
Since you've probably didn't install myproject (or mypackage, if it is top level Python package), Python cannot find it on the path. You have two options:
Install the package.
Use relative import import b. However, see here for implications.
I recommend to avoid approaches like adding mypackage path to system path (see here). Maybe this documentation/tutorial can help you to understand better, what happened and why the import you used is not working.
However, I do not know, how VS Code searches for files in project, so I cannot tell you, why it marks it like this...
according to what you mentioned, both file 'a.py' and 'b.py' are in the same directory and if both files are in the same director, you don't have to actually mention the name of the directory in which they are present.
To import the file 'b.py' in 'a.py', you can just simply type the code shown below to import 'b.py':
import b
This should work.
I want to import business code from my test folder.
I installed my code using python setup.py install, which copy the code into the anaconda site-package folder. But I want to test my code in my dev. directory, as I don't want to constantly install my code to test it after any small change.
I am using Anaconda and Spyder IDE.
Here is my dir structure:
dev
└── myproject
├── setup.py
├── myproject
│ ├── __init__.py
│ └── myproject.py
└── test
├── __init__.py
└── test_importprojecthere.py
I took it from here: Running unittest with typical test directory structure
For now, I'm simply trying to import a function.
# dev: projectfile.py
def hello():
print('Hello world!')
Here is where I call it from.
# dev: test_importprojecthere.py
from myproject.projectfile import hello # Use installed package.
hello()
more information:
Python relative paths for unit tests
Import a module from a relative path Not ideal!
relative path not working even with __init__.py
When you are creating packages, you do not want to install them in your site-packages.
Do not use
python setup.py install
Instead, use
python setup.py develop or...
pip install -e .
Those commands install your setup using symlinks. Instead of creating a copy, it links to your actual project. So when you modify it, it's being modified instantly everywhere.
Thanks to that, you will not have to update/install or do anything to run your unit tests.
-- "But now, every time I modify my package, everything using it might break. I don't want to break my production code!"
Do not worry! This is the reason why we use virtual environments. When you create a package, make sure you create a virtual environment, that you might name, by example "myproject_dev".
In the dev. env. use python setup.py develop so that you can develop on it.
In the prod. env. use python setup.py install, so that you use a copy, that will not suddenly change. You won't modify it, because in prod, you consume your package, you don't develop on it! Keeps things simple an safe.
I have an R script that can be executed in a terminal with Rscript app/myapp.R. The R script is stored in an R package project so that I can benefit from the documentation, check and unit tests tools.
Just for the sake of user-friendlyness, I was wondering if there is a method to mimic python's behaviour of a __main__.py inside a module.
EDIT 6/1/2020 : I actually need other users, that are not accustumed to R, to use the script. So once the package is installed, finding the full path of the script is not really an option for them.
In a python project, when I have the following package structure :
mypackage
├── mymodule
│ ├── __init__.py
│ └── __main__.py
└── setup.py
I can do python -m mymodule from any folder in the terminal provided the package was indeed installed in my python library. The command will execute mypackage/mymodule/__main__.py.
I'd like to have the same behaviour for an R package. Let's assume that the R package mypackage is already installed in my R user's library. I would like to be able to run mypackage/app/myapp.R from anywhere in the terminal (actually, I'd like others to be able to install the package and run the app without having to clone the repo).
I know that I can do
Rscript app/myapp.R
but that will only work if I cd into path/to/mypackage. I'd like to be able to do something like below from anywhere in the terminal provided the package is installed in the R user's library.
Rscript -m myapp
demo() seems to be made for interactive sessions and I need a non-interactive session.
As it turns out there was a really simple solution using R command and not Rscript. Just wrap up the script within a function from the package, then use :
R -e "mypackage::myfunc()"
The following is my directory structure.
ankur
├── ankur1
│ ├── __init__.py
│ └── util.py
├── ankur2
│ └── main.py
└── __init__.py
In main.py, I am importing the following.
import ankur.ankur1.util
When I execute the code in Windows, it works perfectly fine. But in Linux, I get the following error.
ImportError: No module named ankur.ankur1.util
I also read the official python doc on Modules and Packages.
Your package structure is OK. Your import statement is OK. The only thing missing is for the package to be visible in sys.path, a list of locations where import statements can be resolved.
Usually we do this by "installing" the package locally with pip, which copies your code into site-packages†. This directory is one of the entries in sys.path, so when your code is installed in site-packages, the import statements can now be resolved as usual.
However, to install your code you'll need an installer (setup.py script) or a build system (pyproject.toml file) defined for the package. Your project doesn't appear to have any installer or build system, so you'll need to create one (see the Python Packaging User Guide for details about that) and then install the package with pip. If you don't want to learn Python packaging just yet, you'll need to find another way around.
It is possible to modify sys.path directly in main.py, which is subsequently enabling the statement import ankur.ankur1.util to be resolved. This is hacky and I recommend against that. It would add the restriction that executing main.py is the only entry point to the rest of the package, and so any other code wanting to import ankur will first need to know the path to main.py on the filesystem. That's a messy approach and should be avoided.
Another way around is to use the environment - there is an environment variable PYTHONPATH which can be used to augment the default search path for module files. In your shell:
export PYTHONPATH=/path/to/parent # linux/macOS
SET PYTHONPATH=C:/path/to/parent # Windows
Where parent is the directory containing ankur subdirectory.
† The exact location of site-packages depends on your OS/platform, but you can check with import sysconfig; sysconfig.get_paths()["purelib"]