Develop Reference

Python on Gitlab has ModuleNotFound, But Not When I Run Locally

The following snapshot shows the file structure:
When I run on Gitlab CI, here is what I am seeing:
Why is this error occurring when Gitlab runs it but not when I run locally?
Here is my .gitlab-ci.yml file.
Note that this had been working before.
I recently made win_perf_counters a Git submodule instead of being an actual subdirectory. (Again, it works locally.)
test:
before_script:
- python -V
- pip install virtualenv
- virtualenv venv
- .\venv\Scripts\activate.ps1
- refreshenv
script:
- python -V
- echo "******* installing pip ***********"
- python -m pip install --upgrade pip
- echo "******* installing locust ********"
- python -m pip install locust
- locust -V
- python -m pip install multipledispatch
- python -m pip install pycryptodome
- python -m pip install pandas
- python -m pip install wmi
- python -m pip install pywin32
- python -m pip install influxdb_client
- set LOAD_TEST_CONF=load_test.conf
- echo "**** about to run locust ******"
- locust -f ./src/main.py --host $TARGET_HOST -u $USERS -t $TEST_DURATION -r $RAMPUP -s 1800 --headless --csv=./LoadTestsData_VPOS --csv-full-history --html=./LoadTestsReport_VPOS.html --stream-file ./data/stream_jsons/streams_vpos.json --database=csv
- Start-Sleep -s $SLEEP_TIME
variables:
LOAD_TEST_CONF: load_test.conf
PYTHON_VERSION: 3.8.0
TARGET_HOST: http://10.10.10.184:9000
tags:
- win2019
artifacts:
paths:
- ./LoadTests*
- public
only:
- schedules
after_script:
- ls src -r
- mkdir .public
- cp -r ./LoadTests* .public
- cp metrics.csv .public -ErrorAction SilentlyContinue
- mv .public public
When I tried with changing the Gitlab CI file to use requirements.txt:

Probably the python libraries you are using in your local environment are not the same you are using in gitlab. Run a pip list or pip freeze in your local machine and see which versions do you have there. Then pip install those in your gitlab script. A good practice is to have a requirements.txt or a setup.py file with specific versions rather than pulling the latest versions every time.

Probably the module you are developing doesn't have the __init__.py file and thus it cannot be found when imported from the external.

Circleci does not work with my config.yml file when trying to run 'pytest --html=pytest_report.html', producing the error 'no such option: --html'

What I tried:
placing 'pip install --user -r requirements.txt' in the second run command
placing 'pip install pytest' in the second run command along with 'pip install pytest-html'
both followed by,
pytest --html=pytest_report.html
I am new to CircleCI and using pytest as well
Here is the steps portion of the config.yml file
version: 2.1
jobs:
run_tests:
docker:
- image: circleci/python:3.9.1
steps:
- checkout
- run:
name: Install Python Dependencies
command:
echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV && source $BASH_ENV
pip install --user -r requirements.txt
- run:
name: Run Unit Tests
command:
pip install --user -r requirements.txt
pytest --html=pytest_report.html
- store_test_results:
path: test-reports
- store_artifacts:
path: test-reports
workflows:
build_test:
jobs:
- run_tests

--html is not one of the builtin options for pytest -- it likely comes from a plugin
I believe you're looking for pytest-html -- make sure that's listed in your requirements
it's also possible / likely that the pip install --user is installing another copy of pytest into the image which'll only be available at ~/.local/bin/pytest instead of whatever pytest comes with the circle ci image
disclaimer, I'm one of the pytest core devs

Pytest does not find pyYAML in Travis CI tests that have PyPI and conda package dependencies

I am trying to set up automatic testing for my Python package using Travis CI. My Python package depends on Iris as well as other packages such as PyYAML, numpy, etc. It also depends on a PyPI package (ScriptEngine). Now, I would like to set up a Travis CI environment using conda (to install Iris) and pip (to install the PyPI package as well as checking the requirements for PyYAML and numpy). I would then like to install my package using pip install ..
To test if this works, I have written one simple Pytest test that imports PyYAML.
I am currently trying to do this using this .travis.yml file:
language: python
python:
- "3.6"
- "3.7"
- "3.8"
# command to install dependencies
install:
- sudo apt-get update
- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -p $HOME/miniconda
- source "$HOME/miniconda/etc/profile.d/conda.sh"
- hash -r
- conda config --set always_yes yes --set changeps1 no
- conda update -q conda
# Useful for debugging any issues with conda
- conda info -a
- conda env create -f tests/test-environment.yml python=$TRAVIS_PYTHON_VERSION
- conda activate test-environment
- conda install pip
- conda install -c conda-forge iris
- pip install -r requirements.txt
- pip install .
# command to run tests
script: pytest
Note: This is the first time for me to really work with Travis CI. This script is a mixture of examples from the Conda docs as well as the Travis CI docs.
Pytest then fails to import PyYAML (although it gets installed because of the requirements.txt as well as the Iris dependencies):
Here is the confirmation from the logs that it got installed:
Requirement already satisfied: pyYAML>=5.1 in /home/travis/miniconda/envs/test-environment/lib/python3.8/site-packages (from ece-4-monitoring==0.1.0) (5.3.1)
And this is the Error from Pytest:
$ pytest
============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.3.1, py-1.7.0, pluggy-0.8.0
rootdir: /home/travis/build/valentinaschueller/ece-4-monitoring, inifile:
collected 1 item / 1 errors
==================================== ERRORS ====================================
_________________ ERROR collecting tests/test_file_handling.py _________________
ImportError while importing test module '/home/travis/build/valentinaschueller/sciptengine-tasks-ecearth/tests/test_file_handling.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_file_handling.py:3: in <module>
import helpers.file_handling as file_handling
helpers/file_handling.py:1: in <module>
import yaml
E ModuleNotFoundError: No module named 'yaml'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.12 seconds ============================
The command "pytest" exited with 2.
If I try this exact setup using a conda virtual environment locally on my computer, I do not get this problem. Why does this not work on the Travis CI virtual machine?

As cel suggested in their comment: I could fix the problem by explicitly requiring pytest in the requirements.txt.
It is not necessary to activate test-environment in the script part. However, a very helpful tip was to use echo $(which pytest) && pytest instead of just pytest to check if pytest is installed at /home/travis/miniconda/envs/test-environment/bin/pytest.

Python project can't find modules inside of Docker

I'm trying to run a python project inside of docker using the following Dockerfile for machine learning purposes:
FROM python:3
RUN apt-get update \
&& apt-get install -yq --no-install-recommends \
python3 \
python3-pip
RUN pip3 install --upgrade pip==9.0.3 \
&& pip3 install setuptools
# for flask web server
EXPOSE 8081
# set working directory
ADD . /app
WORKDIR /app
# install required libraries
COPY requirements.txt ./
RUN pip3 install -r requirements.txt
# This is the runtime command for the container
CMD python3 app.py
And here is my requirements file:
flask
scikit-learn[alldeps]
pandas
textblob
numpy
matplotlib[alldeps]
But when i try to import textblob and pandas, i get a no module named 'X' error in my docker cmd.
| warnings.warn(msg, category=FutureWarning)
| Traceback (most recent call last):
| File "app/app.py", line 12, in <module>
| from textblob import Textblob
| ImportError: No module named 'textblob'
exited with code 1
Folder structure
machinelearning:
backend:
app.py
Dockerfile
requirements.txt
frontend:
... (frontend works fine.)
docker-compose.yml
Does anyone know the solution to this problem?
(I'm fairly new to Docker, so I might just be missing something crucial.)

This worked for me
FROM python:3
RUN apt-get update
RUN apt-get install -y --no-install-recommends
# for flask web server
EXPOSE 8081
# set working directory
WORKDIR /app
# install required libraries
COPY requirements.txt .
RUN pip install -r requirements.txt
# copy source code into working directory
COPY . /app
# This is the runtime command for the container
CMD python3 app.py

On Linux, whenever you have the message:
ImportError: No module named 'XYZ'`
check whether you can install it or its dependencies with apt-get, example here that does not work for textblob, though, but may help with other modules:
(This does not work; it is an example what often helps, but not here)
# Python3:
sudo apt-get install python3-textblob
# Python2:
sudo apt-get install python-textblob
See Python error "ImportError: No module named" or How to solve cannot import name 'abort' from 'werkzeug.exceptions' error while importing Flask.
In the case of "textblob", this does not work for python2.7, and I did not test it on python3 but it will likely not work either, but in such cases, one should give it a try.
And just guessing is not needed, search through the apt cache with a RegEx. Then:
$ apt-cache search "python.*blob"
libapache-directory-jdbm-java - ApacheDS JDBM Implementation
python-git-doc - Python library to interact with Git repositories - docs
python-swagger-spec-validator-doc - Validation of Swagger specifications (Documentation)
python3-azure-storage - Microsoft Azure Storage Library for Python 3.x
python3-bdsf - Python Blob Detection and Source Finder
python3-binwalk - Python3 library for analyzing binary blobs and executable code
python3-discogs-client - Python module to access the Discogs API
python3-git - Python library to interact with Git repositories - Python 3.x
python3-mnemonic - Implementation of Bitcoin BIP-0039 (Python 3)
python3-nosehtmloutput - plugin to produce test results in html - Python 3.x
python3-swagger-spec-validator - Validation of Swagger specifications (Python3 version)
python3-types-toml - Typing stubs for toml
python3-types-typed-ast - Typing stubs for typed-ast
would be needed to check whether there are some python packages for "textblob" out there.

Multiple python versions with anaconda and travis

I have a python project that makes use of libraries that needs to be built. Given that I use anaconda. I want to create a plan for travis that would let me test against multiple python versions and I am not able to do it. Here is what I have:
I want to test it against multiple python versions (e.g. 2.7, 3.5, 3.6)
I have requirements.yml file which looks like following:
channels:
- kne
dependencies:
- numpy
- pytest
- numpy
- scipy
- matplotlib
- seaborn
- pybox2d
- pip:
- gym
- codecov
- pytest
- pytest-cov
My .travis.yml contains:
language: python
# sudo false implies containerized builds
sudo: false
python:
- 3.5
- 3.4
before_install:
# Here we download miniconda and install the dependencies
- export MINICONDA=$HOME/miniconda
- export PATH="$MINICONDA/bin:$PATH"
- hash -r
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -f -p $MINICONDA
- conda config --set always_yes yes
- conda update conda
- conda info -a
- echo "Python version var"
- echo $TRAVIS_PYTHON_VERSION
- conda env create -n testenv -f environment.yml python=$TRAVIS_PYTHON_VERSION
- source activate testenv
install:
- python setup.py install
script:
- python --version
- python -m pytest --cov=.
- codecov
If I put python version into environment.yml it works fine but I can't use multiple python versions. For me, it seems to if -f is provided, it ignores any additional packages listed for conda env create.
Also, adding - conda install -n testenv python=$TRAVIS_PYTHON_VERSION after env creation does not work.
UnsatisfiableError: The following specifications were found to be in conflict:
- functools32 -> python 2.7.*
- python 3.5*
Use "conda info <package>" to see the dependencies for each package.
What am I supposed to do, in order make it work?
// If you would like to see more details, it is available here: https://travis-ci.org/mbednarski/Chiron/jobs/220644726

You can use sed to modify the python dependency in environment.yml file before creating the conda environment.
Include python in the environment.yml:
channels:
- kne
dependencies:
- python=3.6
- numpy
- pytest
- numpy
- scipy
- matplotlib
- seaborn
- pybox2d
- pip:
- gym
- codecov
- pytest
- pytest-cov
And then modify your .travis.yml:
before_install:
# Here we download miniconda and install the dependencies
- export MINICONDA=$HOME/miniconda
- export PATH="$MINICONDA/bin:$PATH"
- hash -r
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -f -p $MINICONDA
- conda config --set always_yes yes
- conda update conda
- conda info -a
- echo "Python version var"
- echo $TRAVIS_PYTHON_VERSION
# Edit the environment.yml file for the target Python version
- sed -i -E 's/(python=)(.*)/\1'$TRAVIS_PYTHON_VERSION'/' ./environment.yml
- conda env create -n testenv -f environment.yml
- source activate testenv
The sed regex will replace the text python=3.6 with the equivalent for the target python version.
BTW: I see in your repository you've worked around this by using multiple environment.yml files. This seems reasonable, even necessary for certain dependencies, but perhaps tedious to maintain for many Python versions.