I can't get Gst (or seemingly any Python module) to install inside Gitlab CI
Here is my .gitlab-ci.yml file
image: "python:3.7"
before_script:
- apt-get -qq update
- apt-get -qq install -y python3-dev python3-pip libgirepository1.0-dev
- python3 --version
- python3 -m pip install --upgrade pip
- pip3 install -r requirements.txt
test:
script:
- pip3 install flake8 # you can also use tox
- flake8 --max-line-length=254 --extend-ignore=F401,E402 matinee.py
run:
script:
- ./matinee.py -h
But no matter what I tried the closest I got to was:
$ ./matinee.py -h
Traceback (most recent call last):
File "./matinee.py", line 6, in <module>
gi.require_version('Gst', '1.0')
File "/usr/local/lib/python3.7/site-packages/gi/__init__.py", line 126, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gst not available
Before that it complained that gi didn't exist (?) so basically it seems like python modules are not installable this way.
It's been a week now, and frankly I'm giving up on setting up Gitlab CI for python ; I have nodejs projects with tons of deps passing installing without a glitch ; This post is my last hail Mary attempt :|
Naturally, everything works perfectly locally (Ubuntu 22)
My requirements file
# cat requirements.txt
PyGObject==3.42.1
What I tried
image: "python:3.7"
image: "python:latest"
image: "ubuntu:latest"
venv
And everything on this page. Please, help me understand why I can't install Python modules in Gitlab CI.
Related
My file structure is as follows:
Dockerfile
.gitlab-ci.yml
Here is my Dockerfile:
FROM python:3
RUN apt-get update && apt-get install make
RUN apt-get install -y python3-pip
RUN pip3 install --upgrade pip
RUN pip3 install pygdbmi
RUN pip3 install pyyaml
RUN pip3 install Path
And here is my .gitlab-ci.yml file:
test-job:
stage: test
image: runners:test-harness
script:
- cd test-harness
# - pip3 install pygdbmi
# - pip3 install pyyaml
- python3 main.py
artifacts:
untracked: false
when: on_success
expire_in: "30 days"
paths:
- test-harness/script.log
For some reason the pip3 install in the Dockerfile doesn't seem to be working as I get the error:
python3 main.py
Traceback (most recent call last):
File "/builds/username/test-harness/main.py", line 6, in <module>
from pygdbmi.gdbcontroller import GdbController
ModuleNotFoundError: No module named 'pygdbmi'
When I uncomment the two commented lines in .gitlab-ci.yml:
# - pip3 install pygdbmi
# - pip3 install pyyaml
It works fine but ideally, I want those 2 packages to be installed in the Dockerfile not the .gitlab-ci.yml pipeline stage
I've tried changing the WORKDIR as well as USER and it doesn't seem to have any effect.
Any ideas/solutions?
I'm really bad with python. This is on a CentOS7 vm
Problem:
When trying to use awscli in a python virtual environment, I get an error:
(python3ve) [user#ncwv-jlnxnode01 ~]$ aws --version
Traceback (most recent call last):
File "/home/user/venv/python3ve/bin/aws", line 27, in
sys.exit(main())
File "/home/user/venv/python3ve/bin/aws", line 23, in main
return awscli.clidriver.main()
File "/home/user/venv/python3ve/lib64/python3.6/site-packages/awscli/clidriver.py", line 69, in main
driver = create_clidriver()
File "/home/user/venv/python3ve/lib64/python3.6/site-packages/awscli/clidriver.py", line 79, in create_clidriver
event_hooks=session.get_component('event_emitter'))
File "/home/user/venv/python3ve/lib64/python3.6/site-packages/awscli/plugin.py", line 44, in load_plugins
modules = _import_plugins(plugin_mapping)
File "/home/user/venv/python3ve/lib64/python3.6/site-packages/awscli/plugin.py", line 61, in _import_plugins
module = __import__(path, fromlist=[module])
ModuleNotFoundError: No module named '/root/'
ultimately i'm trying to put together a step by step method in an ansible playbook for not only installing awscli, but also awscli-plugin-endpoint, so i'd prefer to install this through pip instead of the centos repos and instead of just downloading the binaries.
Installation Steps:
remove python3 and everything python3 related on the system.
~$ rm -rf ~/venv/python3ve/
~$ sudo yum remove -y python3
~$ sudo yum autoremove -y
~$ sudo find / -name "python3*" > ~/file
~$ sudo xargs rm -r ~/file (missing the arrow because stackoverflow formatting is freaking out with it)
install
~$ sudo yum install -y python3
~$ /usr/bin/python3 -m venv ~/venv/python3ve
~$ source ~/venv/python3ve/bin/activate
~$ ~/venv/python3ve/bin/python3 -m pip install --upgrade pip
~$ ~/venv/python3ve/bin/python3 -m pip install --upgrade awscli
~$ which aws
~/venv/python3ve/bin/aws
~$ aws --version
---output is in the problem description above---
suggestions?
ultimately found that the error was stemming from my ~/.aws/config which I wasnt removing when I reinstalled. that had a reference to the plugin not yet installed and also the old site-packages path (pre venv)
cli_legacy_plugin_path=/root/.local/lib/python3.6/site-packages/
endpoint = awscli_plugin_endpoint
Once I removed those, it worked fine again.
~$ aws --version
aws-cli/1.24.10 Python/3.6.8 Linux/3.10.0-957.el7.x86_64 botocore/1.26.10
The error was referencing /root/ because of how _import_plugins within /awscli/plugin.py splits the path based on . if present
module = path.rsplit('.', 1)
I found other stackoverflow questions and answers for this error, but none of them worked for me. My python package problem:
I tried to install packages with these commands so I could run tests:
$ pip3 install virtualenv
$ virtualenv -p $(which python3.9) venv
$ source venv/bin/activate
$ pip install --upgrade pip # stop local environments from messing things up
$ pip install -r path/to/requirements.txt
I tried to run my tests:
$ python -m unittest discover
I got this error:
======================================================================
ERROR: some_file (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: some_file
Traceback (most recent call last):
File "/usr/local/opt/python#3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/local/opt/python#3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
__import__(name)
File "/Users/path/some_file.py", line 3, in <module>
from .other_file import some_function, SomeClass
ImportError: attempted relative import with no known parent package
----------------------------------------------------------------------
As the error said, I was trying to use a relative path in my file:
from .other_file import some_function, SomeClass
I don't know about other versions of python, but I expected this to be possible in python3.
I needed to add this line to my installation commands:
$ pip install --editable .
That is:
$ pip3 install virtualenv
$ virtualenv -p $(which python3.9) venv
$ source venv/bin/activate
$ pip install --upgrade pip # stop local environments from messing things up
$ pip install -r path/to/requirements.txt
$ pip install --editable .
I don't yet understand why it works. I'll update this answer if I find out more.
I'm trying to run a python project inside of docker using the following Dockerfile for machine learning purposes:
FROM python:3
RUN apt-get update \
&& apt-get install -yq --no-install-recommends \
python3 \
python3-pip
RUN pip3 install --upgrade pip==9.0.3 \
&& pip3 install setuptools
# for flask web server
EXPOSE 8081
# set working directory
ADD . /app
WORKDIR /app
# install required libraries
COPY requirements.txt ./
RUN pip3 install -r requirements.txt
# This is the runtime command for the container
CMD python3 app.py
And here is my requirements file:
flask
scikit-learn[alldeps]
pandas
textblob
numpy
matplotlib[alldeps]
But when i try to import textblob and pandas, i get a no module named 'X' error in my docker cmd.
| warnings.warn(msg, category=FutureWarning)
| Traceback (most recent call last):
| File "app/app.py", line 12, in <module>
| from textblob import Textblob
| ImportError: No module named 'textblob'
exited with code 1
Folder structure
machinelearning:
backend:
app.py
Dockerfile
requirements.txt
frontend:
... (frontend works fine.)
docker-compose.yml
Does anyone know the solution to this problem?
(I'm fairly new to Docker, so I might just be missing something crucial.)
This worked for me
FROM python:3
RUN apt-get update
RUN apt-get install -y --no-install-recommends
# for flask web server
EXPOSE 8081
# set working directory
WORKDIR /app
# install required libraries
COPY requirements.txt .
RUN pip install -r requirements.txt
# copy source code into working directory
COPY . /app
# This is the runtime command for the container
CMD python3 app.py
On Linux, whenever you have the message:
ImportError: No module named 'XYZ'`
check whether you can install it or its dependencies with apt-get, example here that does not work for textblob, though, but may help with other modules:
(This does not work; it is an example what often helps, but not here)
# Python3:
sudo apt-get install python3-textblob
# Python2:
sudo apt-get install python-textblob
See Python error "ImportError: No module named" or How to solve cannot import name 'abort' from 'werkzeug.exceptions' error while importing Flask.
In the case of "textblob", this does not work for python2.7, and I did not test it on python3 but it will likely not work either, but in such cases, one should give it a try.
And just guessing is not needed, search through the apt cache with a RegEx. Then:
$ apt-cache search "python.*blob"
libapache-directory-jdbm-java - ApacheDS JDBM Implementation
python-git-doc - Python library to interact with Git repositories - docs
python-swagger-spec-validator-doc - Validation of Swagger specifications (Documentation)
python3-azure-storage - Microsoft Azure Storage Library for Python 3.x
python3-bdsf - Python Blob Detection and Source Finder
python3-binwalk - Python3 library for analyzing binary blobs and executable code
python3-discogs-client - Python module to access the Discogs API
python3-git - Python library to interact with Git repositories - Python 3.x
python3-mnemonic - Implementation of Bitcoin BIP-0039 (Python 3)
python3-nosehtmloutput - plugin to produce test results in html - Python 3.x
python3-swagger-spec-validator - Validation of Swagger specifications (Python3 version)
python3-types-toml - Typing stubs for toml
python3-types-typed-ast - Typing stubs for typed-ast
would be needed to check whether there are some python packages for "textblob" out there.
I'm trying to setup Circle-CI for the first time for my application. It's a python 3.7.0 based app with a few tests. The app builds just fine, but fails when running the test job. Locally the tests work fine, so I assume I'm missing some Circle-CI configuration?
This is my yaml:
version: 2.0
jobs:
build:
docker:
- image: circleci/python:3.7.0
steps:
- checkout
- run:
name: "Run tests"
command: python -m unittest
This is the error:
======================================================================
ERROR: tests.test_auth (unittest.loader._FailedTest)
ImportError: Failed to import test module: tests.test_auth
Traceback (most recent call last):
File "/usr/local/lib/python3.7/unittest/loader.py", line 434, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/local/lib/python3.7/unittest/loader.py", line 375, in _get_module_from_name
import(name)
File "/home/circleci/project/tests/test_auth.py", line 5, in
from werkzeug.datastructures import MultiDict
ModuleNotFoundError: No module named 'werkzeug'
What am I missing?
EDIT:
I have added now pip install -r requirements.txt but I get now:
Could not install packages due to an EnvironmentError: Errno 13] Permission denied: '/usr/local/lib/python3.7/site-packages/MarkupSafe-1.1.1.dist-info'
EDIT:
In addition to the answer, here is complete yaml configuration working:
version: 2.0
jobs:
build:
docker:
- image: circleci/python:3.7.0
steps:
- checkout
- run:
name: "Install dependencies"
command: |
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install --no-cache-dir -r requirements.txt
- run:
name: "Run tests"
command: |
. venv/bin/activate
python -m unittest
It simply means that a dependency 'werkzeug' is not installed. You might need to install additional packages which are required separately.
Consider adding the dependency installations to the Dockerfile something like below
RUN pip install --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
If you get permission denied issues, then your tests are started with a user who have no privileges to manage python. But its unlikely to be so.