Develop Reference

Python executable script ModuleNotFound

I am trying to better understand importing modules. I read about how to do this from here https://stackoverflow.com/a/14132912/14179793 and I can get it to work using solution 1. There is an additional variable that I need to figure out though.
This is a dummy project I am testing with:
.
├── a_package
│   ├── __init__.py
│   └── lib_a.py
├── b_package
│   ├── __init__.py
│   └── test_script.py
├── main.py
└── src
└── src_lib
└── src_lib.py
With this setup I can do:
python -m b_package.test_script
this is lib a function
This is src_lib_function.
test_script.py:
from a_package.lib_a import lib_a_function
from src.src_lib.src_lib import src_lib_function
if __name__ == '__main__':
lib_a_function()
src_lib_function()
pass
The goal is to make b_package/test_script.py executable without using python test_script ie ./test_script
However, adding the shebang at the top #!/usr/bin/env python causes an import error:
$ ./b_package/test_script.py
Traceback (most recent call last):
File "./b_package/test_script.py", line 2, in <module>
from a_package.lib_a import lib_a_function
ModuleNotFoundError: No module named 'a_package'
I assume it is because python is not loading it as a module based off the above mentioned question but I am not sure how to resolve this.

I ended up using setuptools as suggested by kosciej16 to achieve the desired results.
New project structure:
.
├── a_package
│   ├── __init__.py
│   └── lib_a.py
├── b_package
│   ├── __init__.py
│   └── test_script.py
├── main.py
├── pyproject.toml
├── setup.cfg
└── src
├── __init__.py
└── src_lib
├── __init__.py
└── src_lib.py
setup.cfg:
[metadata]
name = cli_test
version = 0.0.1
[options]
packages = find:
[options.entry_points]
console_scripts =
test_script = b_package.test_script:main
This allows the user to clone the repo and run pip install . from the top level then they can execute the script by just typing test_script

ModuleNotFoundError with package installed from github

I installed a package in my anaconda environment by entering the following line:
pip3 install -e git+https://github.com/gauravmm/jupyter-testing.git#egg=jupyter-testing
I keep getting ModuleNotFoundError: No module named 'testing' on line: from testing.testing import test. I have no idea why this is happening, and believe it has something to do with the way my directory structure is set up.
My directory tree looks like this:
├── hw1_get_started.ipynb
├── requirements.txt
└── src
└── jupyter-testing
├── jupyter_testing.egg-info
│   ├── dependency_links.txt
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
├── LICENSE
├── README.md
├── setup.py
└── testing
├── __init__.py
└── testing.py
I am trying to use this module : https://github.com/gauravmm/jupyter-testing.git#egg=jupyter-testing to do some testing in an online class.
I appreciate any help and explanation as to what I am doing wrong! :)

pyImporterror running subfolder python scripts from the parent folder

I am trying to run from the directory folder:
$ python subdirectoryTwo/file.py command (Python 2.7).
Folders structure:
-directory
-subdirectoryOne
__init.py__
config.py
-subdirectoryTwo
__init.py__
file.py
My file.py has:
from subdirectoryOne.config import config
However I am getting an error:
file.pyImportError: No module named subdirectoryOne.config`
(I guess it still looks in the directory folder)

There few thins you need to change.
(public)landpacks-MacBook-Pro:qx frank$ tree
.
├── __init__.py
├── __init__.pyc
├── a
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── config.py
│   └── config.pyc
└── b
├── __init__.py
└── test.py
Create a __init__.py with with your subdirectoryOne and subdirectoryTwo like here I used a and b. and then add few codes at the begin of your file.py. And I name it as test.py here. the code is:
import sys
sys.path.append("..")
from project.a.config import myconf
print(myconf)
You can see I import it by project.a.config instead of a.config. because you run your code under the project.
UPDATE
My a/config.py just simple with:
(public)landpacks-MacBook-Pro:qx frank$ cat a/config.py
myconf='127.0.0.1'

One of the solutions (not the optimum one) is to set PYTHONPATH to your directory:
$ export PYTHONPATH='/absolute/path/to/directory'

Python - Packaging Alembic Migrations with Setuptools

What is the right way to package Alembic migration files in a Setuptools setup.py file? Everything is in my repo root as alembic/.
This is a Python application, not a library.
My desired installation flow is that someone can pip install the wheel that is my application. They would then be able to initialize the application database by running something like <app> alembic upgrade --sqlalchemy.url=<db_url>. Upgrades would then require a pip install -U, after which they can run the Alembic command again.
Is this unorthodox?
If not, how would I accomplish this? Certainly a console_scripts entry_points. But beyond that?

I am not sure this is the right way but I did it this way:
First, you can add sort of custom options to alembic using the -x option and you can find details explained in this great answer. This allows you to specify the db_url at runtime and make it override the value in the config.ini.
Then I managed to package alembic and my migrations by moving the alembic.ini file and the alembic directory from my project root to my top-level python package:
<project root>
├── src
│   └── <top-level package dir>
│      ├── alembic
│      │   ├── env.py
│      │   ├── README
│      │   ├── script.py.mako
│      │   └── versions
│      │   ├── 58c8dcd5fbdc_revision_1.py
│      │   └── ec385b47da23_revision_2.py
│      ├── alembic.ini
│      ├── __init__.py
│      └── <other files and dirs>
└── <other files and dirs>
This allows to use the setuptools package_data directive inside my setup.py:
setup(
name=<package_name>,
package_dir={'': 'src'},
packages=find_packages(where='src'),
package_data={
'<top-level package dir>': ['alembic.ini', 'alembic/*', 'alembic/**/*'],
},
[...]
)
A this point, the alembic config and revisions are correctly packaged but the alembic.ini settings have to be tweaked to reflect the new directory tree. It can be done using the %(here)s param which contains the absolute path of the directory containing the alembic.ini file:
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = %(here)s/alembic
[...]
# version location specification; this defaults
# to alembic/versions. When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions
version_locations = %(here)s/alembic/versions
[...]
Finally, you have to call alembic with the -c option which allows to provide the path of the config file:
alembic -c <path to alembic.ini> ...

One way to do this which keeps the main alembic folder along the main package folder is to treat the alembic folder as it's own package to be installed along side your main package.
To do this you must rename it (it can't be called alembic, as it will be a top level package, so needs a unique name - I've used migrations), and add a __init__.py file in the alembic folder and the versions folder.
Running the migrations on deployment requires knowing the path to the installed package - a simple way to do this is to provide a console scripts that applies the migrations.
So the project structure looks like this:
<project root>
├── setup.py
├── mypackage
│ └── <project source files...>
│
├── migrations
│ ├── __init__.py
│ ├── alembic.ini
│ ├── apply.py
│ ├── env.py
│ ├── README
│ ├── script.py.mako
│ └── versions
│ ├── __init__.py
│ ├── 58c8dcd5fbdc_revision_1.py
│ └── ec385b47da23_revision_2.py
│
└── <other files and dirs>
And setup.py:
from setuptools import find_packages
from setuptools import setup
setup(
name='mypackage',
packages=find_packages(exclude=('tests',)),
package_data={'migrations': ['alembic.ini']},
entry_points={
'console_scripts': ['apply-migrations=migrations.apply:main'],
},
install_requires=[
"SQLAlchemy==1.3.0",
"alembic==1.0.10",
# ...
]
)
And finally migrations/apply.py:
# Python script that will apply the migrations up to head
import alembic.config
import os
here = os.path.dirname(os.path.abspath(__file__))
alembic_args = [
'-c', os.path.join(here, 'alembic.ini'),
'upgrade', 'head'
]
def main():
alembic.config.main(argv=alembic_args)
Now after installing your wheel, you will have a command apply-migrations which you can invoke directly. Note the version I've implemented here doesn't have any arguments - though if you wanted to pass eg. --sqlalchemy.url you could add it in alembic_args.
Personally I prefer to set the url in migrations/env.py. For example if you had an environment variable called SQLACLHEMYURL you could add this in migrations/env.py:
import os
config.set_main_options(os.getenv('SQLALCHEMYURL'))
Then you can invoke:
SQLALCHEMYURL=... apply-migrations
On deploment.

Why is alembic not autogenerating?

I am learning flask . Using sqlalchemy for orm and alembic for migrations
Going through and following:
http://alembic.readthedocs.org/en/latest/autogenerate.html
whenever i pass the command " alembic revision --autogenerate -m 'name' " , this error pops up . whatever i do , i configured the config.py file but i think maybe i am configuring the env.py file the wrong way . Or something is wrong because i followed every step of the tutorial.
File "alembic/env.py", line 20, in <module>
from myapp.mymodel import Base
ImportError: No module named app.models
folder directory:
project/
app/
models.py
views.py
__init__.py
alembic/
versions
env.py
config.py

When you run the alembic command, your app package is not in Python's module path. So it can't be imported. The easiest way to solve this is to use an extension such as Flask-Migrate or Flask-Alembic to handle setting up the migration environment for you. Both these extensions require you to use Flask-SQLAlchemy as well.
If you don't want to use an extension, the quick and dirty way is to just force the directory containing your app package to be on the path. In env.py, before importing Base, add
import os, sys
sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
A better solution would be to properly set up your project with a setup.py file and install your package in editable mode: pip install -e .. Then your package would be on the path the "right" way, as if it were actually installed.

You should use export PYTHONPATH='.'

you need to add the absolute path to the root directory where your alembic folder lives.
Eg:
If your current folder structure is:
.
├── app
│   ├── alembic
│   │   ├── env.py
│   │   ├── __pycache__
│   │   ├── README
│   │   ├── script.py.mako
│   │   └── versions
│   ├── alembic.ini
│   ├── constants
│   ├── core
│   ├── enums
│   ├── models
│   ├── repository
│   ├── routes
│   ├── __init__.py
│   ├── main.py
and your app has absolute path of:
/home/username/dev/app
then run the following line to add the path in your ~/.bashrc or ~/.zshrc for local development and update the env variables.
echo "export $PYTHONPATH=$PYTHONPATH:/home/username/dev/app" >> ~/.bashrc
source ~/.bashrc
or
echo "export $PYTHONPATH=$PYTHONPATH:/home/username/dev/app" >> ~/.zshrc
source ~/.zshrc