Develop Reference

Dynamically create and run tests in Github Actions

I have a git project with directory structure similar to this:
.
├── .github/
│ └── workflows/
│ └── test.yml
└── my_packages/
├── package_1/
│ ├── tests/
│ │ └── test_package.py
│ ├── package_logic.py
│ ├── configurations.yml
│ └── requirements.py
├── package_2/
│ ├── tests/
│ │ └── test_package.py
│ ├── package_logic.py
│ ├── configurations.yml
│ └── requirements.py
├── ...
└── package_n/
├── tests/
│ └── test_package.py
├── package_logic.py
├── configurations.yml
└── requirements.py
There are n packages in this repository and every package is independent one from another with its own requirements.py, logic, and tests.
There is also test.yml file, which is currently not implemented, that needs to create a testing environment for every package separately and run its test (pytest). Another requirement is that every package defines its python version in the configurations.yml file in which the module test should run. And finally, the test should run only for modules which code was changed.
How can this be implemented in Github Actions? And is it possible to run every module in a separate job (or just in parallel)?

Accessing Folder Structure for Spark Submit zipped PyFile

I have a folder structure that looks like this:
├── repo
│ ├── src
│ └── main.py
│ ├── __init__.py
│ ├── utils
│ ├── __init__.py
│ └── helpers.py
│ └── predict
│ ├── __init__.py
│ └── predict.py
│
I am submitting a spark job with the --file src/predict/predict.py and a py-files src.zip (the spark submit command is built from main.py) of the entire source directory.
In predict, I want to reference a method in helpers.py. I do this by attempting an import: from src.utils.helpers import helper_func
However this does not work: ModuleNotFoundError: No module named 'src'. The imports work locally, but I am trying to understand how this works in the EMR. Ideally I dont want to have to change the importing from local to EMR. I just want them to be the same.

Run single folder of pytest tests in vscode

I have a sort of a "micro-service" Python repo with a setup similar to the following:
sample
├── bar
│ ├── src
│ │ └── main.py
│ └── tests
│ └── test_main.py
├── foo
│ ├── src
│ │ └── main.py
│ └── tests
│ └── test_main.py
└── shared
├── src
│ └── main.py
└── tests
└── test_main.py
In vscode I only have the option of running all tests in foo,bar,shared or running individual test methods in the subfolders. What I want to do is be able to quickly run just the foo/tests/.
Is there some way I can configure pytest/Python to do this? I don't want to split each top level folder into its own workspace because I regularly jump back and for between them and don't want to have multiple windows per workspace open.

You should be able to run the command pytest foo/tests/ in the terminal according to the
pytest documentation

Python/Django "Import Error" in project directory

I've been having troubles with this and cannot find the solution... I've browsed all SO questions about this but nothing worked in my case... So here's the problem:
I clone a project from git repository
Set up a virtualenv called env in project base dir with proper Python version(2.7) & install all reqs succesfully
Here's where it gets fun... I navigate to the folder that has my manage.py and do a python manage.py makemigrations which results in
from foo.bar.data import CoolioClass
ImportError: No module named foo.bar.data
My directories look like this (just for illustration):
project_root/
├──env/
├──django/
│ ├── app
│ │ └── models.py (from foo.bar.data import CoolioClass)
│ ├── django
│ │ └── settings.py
│ └── manage.py
└──foo/
├── bar
│ ├── __init__.py
│ ├── data.py
│ └── test.py
├── baz
│ ├── __init__.py
│ ├── data.py
│ └── test.py
└── __init__.py
When I print sys.path in python shell it results in:
/home/johnny/project_root/env/lib/python2.7
/home/johnny/project_root/env/lib/python2.7/plat-x86_64-linux-gnu
/home/johnny/project_root/env/lib/python2.7/lib-tk
/home/johnny/project_root/env/lib/python2.7/lib-old
/home/johnny/project_root/env/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/home/johnny/project_root/env/local/lib/python2.7/site-packages
/home/johnny/project_root/env/lib/python2.7/site-packages

Python module import issue subdir

I have the following directory structure in my Python3 project:
├── README.md
├── requirements.txt
├── schemas
│ ├── collector.sql
│ └── controller.sql
├── src
│ ├── controller.db
│ ├── controller.py
│ ├── measurement_agent.py
├── tests
│ ├── regression.py
│ ├── test-invalid-mac.py
│ ├── test-invalid-url.py
│ ├── test-register-ma-json.py
│ ├── test-register-ma-return-code.py
│ ├── test-send-capabilities-return-code.py
│ ├── test-valid-mac.py
│ └── test-valid-url.py
└── todo
In my tests folder I have some regression tests which are ran to check the consistency of the code from src/measurement_agent.py. The problem now is that I do not want to add to my path manually the measurement_agent.py to make an import from it. I would want to know if there is any trick how to tell Python to look in my root for the import I am trying to use.
Currently I am doing:
import os.path
ma = os.path.abspath('..') + '/src'
sys.path.append(ma)
from measurement_agent import check_hardware_address
and would want to have something just like
from measurement_agent import check_hardware_address
without using any os.path tricks.
Any suggestions?

Relative imports
Make sure there is an __init__.py in all folders including the top-most (the parent)
Use a relative import, like this:
from ..src import measurement_agent
Now to run your code, cd up to the parent of your parent directory and then
python -m parent.test.regression