I've been asked to take a look at some tests in our project. The component tests all run individually no problem, but when run together we get hundreds of errors. Given the scale of the problem I'm no longer actively looking into it, but I've become really frustrated trying to figure out why it's happening.
A section of the project layout is below. There are modules dealing with various aspects of the project defined in the 'lambdas' folder, with a 'common' folder that contains utilities used by other files. I'm specifically looking at some tests in the 'episodes/api' directory.
.
└── lambdas
├── access
├── ...
├── common
│ ├── dal
│ └── utils
│ │ ├── api_utils.py
│ │ ├── audit_utils.py
│ │ └── date_utils.py
│ └── ...
└── episodes
└── api
├── accept_participant
│ ├── component_tests
│ │ ├── __init__.py
│ │ └── accept_participant_test.py
│ ├── unit_tests
│ │ ├── __init__.py
│ │ └── accept_participant_test.py
│ ├── accept_participant.py
│ └── __init__.py
└── reject_participant
│ ├── component_tests
│ │ ├── __init__.py
│ │ └── reject_participant_test.py
│ ├── unit_tests
│ │ ├── __init__.py
│ │ └── reject_participant_test.py
│ ├── reject_participant.py
│ └── __init__.py
accept_participant.py imports two methods from common.utils.audit_utils, one of which uses a boto3.client to communicate with SQS. The component test accept_participant_test mocks out the boto3.client call, which all works fine when run on its own. It also works fine when run in conjunction with the other component test, as long as the unit tests are not physically present. If the unit tests are run, or even if they're present but not selected for run (using a pytest.mark), the accept_participant_test component test fails.
The unit test causing the problem is the reject_participant_test. It imports common.utils.audit_utils in order to check the value of an enum in it. When this line is commented out, the component test passes ok.
I'm probably missing a gotcha here somewhere, but why does this one import cause a problem? The unit test isn't even being run. I'm thinking I might need to change the boto3.client patch, but I've tried all sorts of different variations and it either tells me it's not a package or doesn't exist in the namespace.
Related
I have a git project with directory structure similar to this:
.
├── .github/
│ └── workflows/
│ └── test.yml
└── my_packages/
├── package_1/
│ ├── tests/
│ │ └── test_package.py
│ ├── package_logic.py
│ ├── configurations.yml
│ └── requirements.py
├── package_2/
│ ├── tests/
│ │ └── test_package.py
│ ├── package_logic.py
│ ├── configurations.yml
│ └── requirements.py
├── ...
└── package_n/
├── tests/
│ └── test_package.py
├── package_logic.py
├── configurations.yml
└── requirements.py
There are n packages in this repository and every package is independent one from another with its own requirements.py, logic, and tests.
There is also test.yml file, which is currently not implemented, that needs to create a testing environment for every package separately and run its test (pytest). Another requirement is that every package defines its python version in the configurations.yml file in which the module test should run. And finally, the test should run only for modules which code was changed.
How can this be implemented in Github Actions? And is it possible to run every module in a separate job (or just in parallel)?
I have a folder structure that looks like this:
├── repo
│ ├── src
│ └── main.py
│ ├── __init__.py
│ ├── utils
│ ├── __init__.py
│ └── helpers.py
│ └── predict
│ ├── __init__.py
│ └── predict.py
│
I am submitting a spark job with the --file src/predict/predict.py and a py-files src.zip (the spark submit command is built from main.py) of the entire source directory.
In predict, I want to reference a method in helpers.py. I do this by attempting an import: from src.utils.helpers import helper_func
However this does not work: ModuleNotFoundError: No module named 'src'. The imports work locally, but I am trying to understand how this works in the EMR. Ideally I dont want to have to change the importing from local to EMR. I just want them to be the same.
I have a sort of a "micro-service" Python repo with a setup similar to the following:
sample
├── bar
│ ├── src
│ │ └── main.py
│ └── tests
│ └── test_main.py
├── foo
│ ├── src
│ │ └── main.py
│ └── tests
│ └── test_main.py
└── shared
├── src
│ └── main.py
└── tests
└── test_main.py
In vscode I only have the option of running all tests in foo,bar,shared or running individual test methods in the subfolders. What I want to do is be able to quickly run just the foo/tests/.
Is there some way I can configure pytest/Python to do this? I don't want to split each top level folder into its own workspace because I regularly jump back and for between them and don't want to have multiple windows per workspace open.
You should be able to run the command pytest foo/tests/ in the terminal according to the
pytest documentation
I am currently writing a library in Python. I am trying to use a class that I have defined in a module but I am unable to use it in my main.py. Inside Selector and SeasonSelector I have 2 classes defined with the same name of the file. I get the following error:
No name 'Selector' in module 'Selectors'
main.py:
import pyf1
testSeasonSelector = pyf1.Selectors.SeasonSelector()
testSelector.loadData()
pyf1/__init__.py:
from Selectors.Selector import Selector
from Selectors.SeasonSelector import SeasonSelector
Directory
├── main.py
└── pyf1
├── Selector.pyc
├── Selectors
│ ├── SeasonSelector.py
│ ├── SeasonSelector.pyc
│ ├── Selector.py
│ ├── Selector.pyc
│ └── __init__.pyc
├── __init__.py
├── __init__.pyc
├── __pycache__
│ └── __init__.cpython-38.pyc
└── data
Your directory should look like this
├── main.py
└── pyf1
├── selector.pyc
├── selectors
| ├── __init__.py
│ ├── season_selector.py
│ ├── season_Selector.pyc
│ ├── selector.py
│ ├── selector.pyc
│ └── __init__.pyc
├── __init__.py
├── __init__.pyc
├── __pycache__
│ └── __init__.cpython-38.pyc
└── data
Your directory is not a certified package without init.py as such you cannot make import statements. I also took the liberty of correcting how you named your modules to fit good python naming convention\practices. With this you should be able to import from your built package without issues
I managed to find a solution.
The issue was I wasn't using the correct syntax. Once I added the missing __init__() files I used the import statement Import .parent_file from ParentClass Which then allowed me to pass functionality up the chain how I wanted.
Inside the Selectors directory I could then use from .selector import __Selector to use in SeasonSelector
Furthermore - I hadn't included the PyF1 in my sys.path, therefore python wasn't able to scan the directories. This included with the syntax above allowed me to do what I wanted.
I have the following directory structure in my Python3 project:
├── README.md
├── requirements.txt
├── schemas
│ ├── collector.sql
│ └── controller.sql
├── src
│ ├── controller.db
│ ├── controller.py
│ ├── measurement_agent.py
├── tests
│ ├── regression.py
│ ├── test-invalid-mac.py
│ ├── test-invalid-url.py
│ ├── test-register-ma-json.py
│ ├── test-register-ma-return-code.py
│ ├── test-send-capabilities-return-code.py
│ ├── test-valid-mac.py
│ └── test-valid-url.py
└── todo
In my tests folder I have some regression tests which are ran to check the consistency of the code from src/measurement_agent.py. The problem now is that I do not want to add to my path manually the measurement_agent.py to make an import from it. I would want to know if there is any trick how to tell Python to look in my root for the import I am trying to use.
Currently I am doing:
import os.path
ma = os.path.abspath('..') + '/src'
sys.path.append(ma)
from measurement_agent import check_hardware_address
and would want to have something just like
from measurement_agent import check_hardware_address
without using any os.path tricks.
Any suggestions?
Relative imports
Make sure there is an __init__.py in all folders including the top-most (the parent)
Use a relative import, like this:
from ..src import measurement_agent
Now to run your code, cd up to the parent of your parent directory and then
python -m parent.test.regression