Develop Reference

Python executable script ModuleNotFound

I am trying to better understand importing modules. I read about how to do this from here https://stackoverflow.com/a/14132912/14179793 and I can get it to work using solution 1. There is an additional variable that I need to figure out though.
This is a dummy project I am testing with:
.
├── a_package
│   ├── __init__.py
│   └── lib_a.py
├── b_package
│   ├── __init__.py
│   └── test_script.py
├── main.py
└── src
└── src_lib
└── src_lib.py
With this setup I can do:
python -m b_package.test_script
this is lib a function
This is src_lib_function.
test_script.py:
from a_package.lib_a import lib_a_function
from src.src_lib.src_lib import src_lib_function
if __name__ == '__main__':
lib_a_function()
src_lib_function()
pass
The goal is to make b_package/test_script.py executable without using python test_script ie ./test_script
However, adding the shebang at the top #!/usr/bin/env python causes an import error:
$ ./b_package/test_script.py
Traceback (most recent call last):
File "./b_package/test_script.py", line 2, in <module>
from a_package.lib_a import lib_a_function
ModuleNotFoundError: No module named 'a_package'
I assume it is because python is not loading it as a module based off the above mentioned question but I am not sure how to resolve this.

I ended up using setuptools as suggested by kosciej16 to achieve the desired results.
New project structure:
.
├── a_package
│   ├── __init__.py
│   └── lib_a.py
├── b_package
│   ├── __init__.py
│   └── test_script.py
├── main.py
├── pyproject.toml
├── setup.cfg
└── src
├── __init__.py
└── src_lib
├── __init__.py
└── src_lib.py
setup.cfg:
[metadata]
name = cli_test
version = 0.0.1
[options]
packages = find:
[options.entry_points]
console_scripts =
test_script = b_package.test_script:main
This allows the user to clone the repo and run pip install . from the top level then they can execute the script by just typing test_script

Trouble organizing Python library for import to behave in expected manner

I've had a lot of trouble figuring out a key point about how the import mechanism works, and how this relates to organizing packages.
Suppose I've written two or more unrelated, reusable libraries. (I'll use "library" informally as a collection of code and resources, including tests and possibly data, as opposed to a "package" in the formal Python sense.) Here are two imaginary libraries in a parent directory called "my_libraries":
my_libraries/
├── audio_studio
│   ├── src
│   │   ├── distortion.py
│   │   ├── filter.py
│   │   └── reverb.py
│   └── test
│   └── test_audio.py
└── picasso_graphics
├── src
│   ├── brushes.py
│   ├── colors.py
│   └── easel.py
└── test
└── test_picasso.py
I'm hoping to accomplish all three of the following, all of which seem to me to be normal practice or expectation:
1. MAIN LIBRARY CODE IN SUBDIRECTORY
For neatness of library organization, I want to put the library's core code in a subdirectory such as "src" rather than at the top-level directory. (My point here isn't to debate whether "src" in particular is a good naming approach; I've read multiple pages pro and con. Some people appear to prefer the form foo/foo, but I think I'd have the same problem I'm describing with that too.)
2. ADD TO $PYTHONPATH JUST ONCE
I'd like to be able to add "my_libraries" to $PYTHONPATH or sys.path just once. If I add a new library to "my_libraries", it's automatically discoverable by my scripts.
3. NORMAL-LOOKING import STATEMENTS
I'd like to be able import from these libraries into other projects in a normal-looking way, without mentioning the "src" directory:
import picasso_graphics.brushes
OR
from picasso_graphics import brushes
HOW TO DO THIS?
Despite much reading and experimentation, I haven't been able to find a solution which satisfies all three of these criteria. The closes I've gotten is to create a picasso_graphics/__init__.py file containing the following:
base_dir = os.path.dirname(__file__)
src_dir = os.path.join(base_dir, "src")
sys.path.insert(0, src_dir)
This almost does what I want, but I have to break up the imports into two statements, so that the __init__.py file executes with the first import:
import picasso_graphics
import brushes
Am I making a wrong assumption here about what's possible? Is there a solution which satisfies all three of these criteria?

What you want, Sean, is most likely what is called a namespace project; Use a tool called pyscaffold to help with writing the boilerplate. Each project should have a setup.cfg with all your project dependencies. Once you have that, create a virtual environment for your project, then install each with ... (inside the environment).
pip install -e audio-studio
pip install -e picasso-graphics
Installing your project into your virtual environment will cause your imports to behave as you want -- between projects.
This is a bit of overhead to get started, I know, but these are skills you want to have sooner or later. Setup.cfg, virtual environments, and pip install -e is a magical pattern that just makes things work where other approaches will drive you mad.
Below is a simple example project I created using pyscaffold. Notice there is a package below src and that src does not have an init.py. That is a decision made by the pyscaffold folks to help ease import confusion - you should likely adopt it.
my_libraries/
├── audio-studio
│   ├── AUTHORS.rst
│   ├── CHANGELOG.rst
│   ├── LICENSE.txt
│   ├── README.rst
│   ├── requirements.txt
│   ├── setup.cfg
│   ├── setup.py
│   ├── src
│   │   └── audio_studio
│   │   ├── __init__.py
│   │   └── skeleton.py
│   └── tests
│   ├── conftest.py
│   └── test_skeleton.py
└── picasso-graphics
├── AUTHORS.rst
├── CHANGELOG.rst
├── LICENSE.txt
├── README.rst
├── requirements.txt
├── setup.cfg
├── setup.py
├── src
│   └── picasso_graphics
│   ├── __init__.py
│   └── skeleton.py
└── tests
├── conftest.py
└── test_skeleton.py

What import system should I use when I want to run my application both 'from source' and from installing with setuptools?

Consider this application:
.
├── LICENSE
├── MANIFEST.in
├── program
│   ├── apple.py
│   ├── __init__.py
│   ├── __main__.py
│   ├── nonfruit.py
│   ├── pear.py
│   ├── strawberry.py
│   └── vegetables
│   ├── carrot.py
│   ├── __init__.py
│   └── lettuce.py
├── README.md
├── setup.cfg
└── setup.py
__main__.py is the file that users should run to use my program. I am distributing my program via PyPI and so I want to be able to install it via pip as well. Because of that, I created a setup.py file with an entry point:
entry_points = {
'console_scripts': ['pg=program.__main__:main']}
The problem I'm facing is that there are several imports in my program, and these result in the situation that my program does run 'locally' (by executing python ./__main__.py, but not from installation (by running pg). Or, depending on the way I import it, the other way around.
__main__.py imports nonfruit.py:
from nonfruit import Nonfruit
nonfruit.py imports vegetables/carrot.py:
import vegetables.carrot
ca = vegetables.carrot.Carrot()
I would like to hear some advice in structuring my project regarding imports, so that it runs both locally and from setuptools installation. For example, should I use absolute imports or relative imports? And should I use from X import Y or import X.Y?

I found a solution on Jan-Philip Gehrcke's website.
The instructions are written for use with Python 3, but I applied it to Python 2.7 with success. The problem I was having originated from my directory becoming a package. Jan advices to create one file to run it from source (bootstrap-runner.py) and one file to run it from installation (bootstrap/__main__.py). Furthermore, he advices to use explicit relative imports:
from .X import Y
This will probably be a good guideline in the next applications I'm writing.

Installation of MySQL-python in shared hosting

I'm using Hostgator as a testing environment and I had a problem installing MySQL-python, after using:
pip install MySQL-python
Next error raises:
unable to execute gcc: Permission denied
enter code here`error: command 'gcc' failed with exit status 1
I ask technical support about this and they answer me:
This script requires a compiler, which shared accounts do not have
access to. You would need to upload any Python scripts that you want
to use as a precompiled script. You should be able to compile it
elsewhere and then upload to the account to use it.
This is my first project using Python and I have not idea how to do this.
Thanks
======
UPDATE
As André proposed, What I did was using my linux I created two virtual environments (using virtualenv) one with and one without MySQL-python installed.
Checking the file structure, missing files were:
.
├── MySQLdb
│   ├── connections.py
│   ├── connections.pyc
│   ├── constants
│   │   ├── CLIENT.py
│   │   ├── CLIENT.pyc
│   │   ├── CR.py
│   │   ├── CR.pyc
│   │   ├── ER.py
│   │   ├── ER.pyc
│   │   ├── FIELD_TYPE.py
│   │   ├── FIELD_TYPE.pyc
│   │   ├── FLAG.py
│   │   ├── FLAG.pyc
│   │   ├── __init__.py
│   │   ├── __init__.pyc
│   │   ├── REFRESH.py
│   │   └── REFRESH.pyc
│   ├── converters.py
│   ├── converters.pyc
│   ├── cursors.py
│   ├── cursors.pyc
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── release.py
│   ├── release.pyc
│   ├── times.py
│   └── times.pyc
├── _mysql_exceptions.py
├── _mysql_exceptions.pyc
├── MySQL_python-1.2.5-py2.7.egg-info
│   ├── dependency_links.txt
│   ├── installed-files.txt
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
└── _mysql.so
So I copied those files to:
/venv/lib/python2.7/site-packages/
Were /venv/ is the folder of the virtual enviroment created in the hosting.
Thanks again

There is a really simple solution to this. If the root user on the shared hosting has MySQLdb python module installed, then you can create a user specific virtual environment by using the --sytem-site-package flag. This creates a virtual environment with all the modules of the root python installed onto the local venv.
virtualenv --sytem-site-package
You can look up: Make virtualenv inherit specific packages from your global site-packages

You don't have permission to compile things using gcc. You will have to install MySQL-python in another place and then move the files back onto your server.
See py_compile for compiling python scripts

If they mean precompiled I am assuming to make an executable, if your HostGator account is using Windows then you can use py2exe and create an executable. Py2exe makes it so you can run your script on other computers without having to install python.
First create a setup.py which tells what script and all its dependencies, and then run python setup.py py2exe and it will create two folders. You will just need the dist folder with the executable located in there.
There are many nice tutorials on how to do this, good luck!

Python - No module named foo

I'm new to python and I am trying to write a simple program for an led cube. First I need to execute one of the other sample programs provided, but I'm getting the error "No module named cube_interface" when I try and run a python program.
Now before you mark this question as duplicate, and refer me to the 7,776 similar questions stack overflow, let me tell disclose that I've already tried implementing the solutions provided on 14 other SO questions, and followed 3 blogs covering how to fix this error, with no luck.
OS: Mac OsX 10.8.1
Python: 2.7.5
Repo: https://github.com/chadharrington/all_spark_cube
The example program is src/tetris.py
The module is src/all_spark_cube_client
├── Makefile
└── src
├── LICENSE.txt
├── all_spark_cube_client
│   ├── __init__.py
│   └── __init__.pyc
├── all_spark_cube_client.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   ├── not-zip-safe
│   ├── requires.txt
│   └── top_level.txt
├── build
│   ├── bdist.macosx-10.9-intel
│   └── lib
│   └── all_spark_cube_client
│   └── __init__.py
├── client_demo.py
├── colors.py
├── dist
│   └── all_spark_cube_client-0.1-py2.7.egg
├── load_test.py
├── setup.py
├── supervisord.conf
├── supervisord_init_script.debian
├── supervisord_init_script.redhat
└── tetris.py
When I try and run the tetris.py program, I get an error "No module named.."
python tetris.py
Traceback (most recent call last):
File "tetris.py", line 5, in <module>
from all_spark_cube_client import CubeClient
File "/Users/sowen/Code/all_spark_cube/software/clients/python_client/src/all_spark_cube_client/__init__.py", line 8, in <module>
from cube_interface import CubeInterface
ImportError: No module named cube_interface
Many of the answers suggest adding pwd . /Library/Python ect.. to the PYTHONPATH. I've tried implementing all of them independently, and even combined all suggestions into one frakenstein path as shown, with no luck.
cd ~/Code/all_spark_cube/software/clients/python_client
export PYTHONPATH=$PATH:$PYTHONPATH:`pwd`:`pwd`/src:.:/Library/Python/2.7/site-packages/
I've tried building the python module
$sudo python setup.py build
running build
running build_py
file all_spark_cube_client.py (for module all_spark_cube_client) not found
file all_spark_cube_client.py (for module all_spark_cube_client) not found
I've tried installing the python module (It doesn't give any errors, but I still am unable to run tetris.py)
cd ~/Code/all_spark_cube/software/clients/python_client/src/
sudo python setup.py install
How can I execute the tetris.py program?
Additional Resources
http://docs.python.org/2/using/mac.html
http://www.confusedcoders.com/random/python-module-importerror-no-module-named-pocketsphinx

You forgot to build cube_interface:
See: https://github.com/chadharrington/all_spark_cube/tree/master/software/thrift
You need to build this with the gen_py tool.
i.e: (I assume:):
cd /path/to/all_spark_cube/software/thrift/
make
cp cube_interface.py /path/to/python/site-packages