I'm new to python. I have a maven project which uses the org.jolokia maven-docker-plugin to create a docker image that consumes a python library.
Currently the container uses a pip install to install the python library.
I have forked the python library and made some changes, and now I would like my docker container to consume MY version of the python library. How can I do this?
What I have tried:
Copied my changed python file to overwrite the folder located in /usr/local/lib/python2.7/dist-packages/ which was generated after pip install (via mount directory into container).
Created tar of entire python project, added it into image using fileSets, ran pip install /maven/mypythonversion.tar.gz.
Any help much appreciated!
The way to do this is to get your python project into the image and run install on the setup.py. This method assumes your image already has the python interpreter and the relevant dependencies installed for your project to run.
Copy your python project code into a folder in your maven project i.e. one called input.
Use fileSets in the assembly of the image to assemble the python source code into the image. <directory> should point to the input folder containing your python source code:
<assembly>
<mode>tar</mode>
<inline>
<fileSets>
<fileSet>
<directory>C:/.../input</directory>
<outputDirectory>/output</outputDirectory>
<includes>
<include>**/*</include>
</includes>
</fileSet>
</fileSets>
</inline>
</assembly>
Then you want a script to run python setup.py install from the output directory (/maven/output/mypythonproject).
I.e. use a runCmd:
<runCmds>
<run>
cd /maven/output/mypythonproject \
python setup.py install
</run>
</runCmds>
This installs the python module and puts egg file in the /usr/local/lib/python27/dist-packages folder, which will be found by your python interpreter.
Related
I am trying to build a minimal docker image, capable of nothing more but running the Python interpreter. During development I start from alpine, the final image will be build FROM scratch. I am working on a Linux Mint machine.
I am able to compile the Python compiler and install it into my current working directory like this:
cd Python-3.8.7
./configure --prefix=$PWD/../python
make install
However, I did not find out how to tweak the --prefix settings correctly so that the created shebangs will later work in the docker container.
First line of ./python/pip3 contains the absolute path of my host and reads
#!/home/orion/minimal_py_image/Python-3.8.7/../python/bin/python3.8
But it should read
#!/python/bin/python3.8
because /python is the location under which the Python interpreter will be found in the docker image.
How can I trick the make install script so that the final destination will be /python/bin?
I would like to keep the build contained in the current directory i.e. not using the folder /python on the host where I do the compilation.
Additional Information
Probably not directly relevant for this question but as reference: Here is the Dockerfile I am trying to get working:
FROM alpine
COPY python /python
COPY lib64/* /lib64/
ENV LD_LIBRARY_PATH=/usr/lib64/:/lib64/:/python/lib
ENV PATH="${PATH}:/python/bin"
I am capable of running Python already with docker run -it mini python3 -c "print('hello from python')" but pip is not working yet due-to the wrong shebang.
A common convention in Autoconf-based build systems is to support a Make variable DESTDIR. When you run make install, if DESTDIR is set, it actually installs into the configured directory under DESTDIR, but still built with the original path. You can then create an archive of the target directory, or in a Docker context, use that directory as the build context.
cd Python-3.8.7
# Use the final install target as the --prefix
./configure --prefix=/python
# Installs into e.g. ../python/python/bin/python3.8
make install DESTDIR=../python
cd ../python
tar cvzf ../python.tar.gz .
You can see this variable referenced in the Python Makefile in many places.
I have a python project and I am using pipenv to handle deps.
I need to create a zip file that includes the source code and all the dependencies code as well. I need this zip file for uploading it to AWS Lambda.
When working with pipenv, it downloads the dependency libraries somewhere in the computer, but for packaging/distribution of the project I need all the necessary code to be contained in the same place (a zip file).
Is there a way to run pipenv and set it to install dependencies at a specific path? If not, does someone knows where those dependencies are located in my machine?
Thanks
This has worked for me:
#!/bin/bash
# this is b/c pipenv stores the virtual env in a different
# directory so we need to get the path to it
SITE_PACKAGES=$(pipenv --venv)/lib/python3.6/site-packages
echo "Library Location: $SITE_PACKAGES"
DIR=$(pwd)
# Make sure pipenv is good to go
echo "Do fresh install to make sure everything is there"
pipenv install
cd $SITE_PACKAGES
zip -r9 $DIR/package.zip *
cd $DIR
zip -g package.zip posts.py
I've specifically tried it with numpy and it works correctly. It includes the .so files as well which is great because everything is self contained.
I currently have an executable file that is running Python code inside a zipfile following this: https://blogs.gnome.org/jamesh/2012/05/21/python-zip-files/
The nice thing about this is that I release a single file containing the app. The problems arise in the dependencies. I have attempted to install files using pip in custom locations and when I embed them in the zip I always have import issues or issues that end up depending on host packages.
I then started looking into virtual environments as a way to ensure package dependencies. However, it seems that the typical workflow on the target machine is to source the activation script and run the code within the virtualenv. What I would like to do is have a single file containing a Python script and all its dependencies and for the user to just execute the file. Is this possible given that the Python interpreter is actually packaged with the virtualenv? Is it possible to invoke the Python interpreter from within the zip file? What is the recommended approach for this from a Python point of view?
You can create a bash script that creates the virtual env and runs the python scripts aswell.
!#/bin/bash
virtualenv .venv
.venv/bin/pip install <python packages>
.venv/bin/python script
When I installed Python Markdown, I noticed that it added files to a build/docs/extensions folder. Where can I find this folder? I've searched through my machine, but came up empty.
As part of the build process, Python-Markdown also builds the docs (from Markdown text files into HTML files). However, as the generated files are written to the build directory, they are deleted in the cleanup step after install (the build dir is deleted). The docs are hosted here, but if you would like a local copy, you can build them yourself.
First you need a copy if the source files either from PyPI or GitHub. then from within the top directory, run the following command:
python setup.py build_docs
The docs will be written to build/docs. As we didn't complete the install process, the files haven't been deleted yet.
The complete steps to download and build might look this (on Linux using the current release of Python-Markdown):
wget http://pypi.python.org/packages/source/M/Markdown/Markdown-2.5.2.tar.gz
tar xvzf Markdown-2.5.2.tar.gz
cd markdown-2.5.2/
python setup.py build_docs
cd build/docs
From that point you can move/copy the files to wherever you would like them, or open them in your browser of choice to view them.
I have created my program using virtual env. It is working in my project folder fine. Now i need to take this program and release it to the production environment that is supposed to be accessible by everybody.So this program should be runnable as is or it might be incorporated into other programs as a step. How am i supposed to deploy it? Zip the whole project folder? Is it possible to do without requiring clients to copy it and then unzip and run? Or the only way is to create a commonly accessible script that automates unzipping of the thing and configuring virtual env and then running it or there is a smarter way?
More complicated scenario is when it supposed to be used as library. How to deploy it so others could specify it as their dependency and pick it up? Seems like the only way is to create your own PyPi-like local repository - is that correct?
Thanks!
So here is what i have found:
If we have a project A as API:
create a folder where you will store the wheels (~/wheelhouse)
using pip config specify this folder as one to find links in http://www.pip-installer.org/en/latest/configuration.html
i have:
[global]
[install]
no-index = yes
find-links = /home/users/me/wheelhouse
Make sure the wheel package is installed.
In your project create setup.py file that will allow for the wheel creation and execute
python setup.py bdist_wheel
copy the generated wheel to the wheelhouse so it has:
~/wheelhouse/projectA-0.1-py33-none-any.whl
Now we want to create a project that uses that projectA API - project B
we are creating a separate folder for this project and then create a virtual environment for it.
mkdir projectB; cd projectB
virtualenv projectB_env
source projectB_env/bin/activate
pip install projectA
Now if you run python console in this folder you will be able to import the classes from the projectA! One problem solved!
Now you have finished the development of projectB and you need to run it.
For that purpose I'd recommend to use Pex (twitter.common.python) library. Pex now supports (v0.5.1) wheels lookup as dependencies. I'm feeding it the content of requirements.txt file to resolve dependencies. So as the result you will get the executable lightweight archived virtualenv that will have everything necessary for the project to run.
This should get you started:
http://docs.python.org/2/distutils/
http://guide.python-distribute.org/
http://pythonhosted.org/setuptools/