How can I read files within a python module while executing in docker? I have a python module which I import in my code. normally in order to fetch the relative path of the module one can do <<module_name>>.__path__. however this approach does not work in docker but works locally.
Is there a common way in which I can read the files from the module in docker as well as in local?
There are a few things to consider:
Where is python installed in the container and what version? Does
that match your dev environment?
Look at your dockerfile - what is your working directory? Did you set one? Perhaps, you are running your python code from one directory, but trying to import a module from another.
Is your PYTHONPATH set in your container?
Have you installed the modules in the container that you're attempting to use? Perhaps with a requirements.txt file or manually? If so, are you executing your python code with the same python version/path that you installed the modules with?
Are you using a virtual environment? Has it been sourced?
What user is your container running as? Does it have access to the python modules? You may need to chown the site-packages path or run as a different user or root.
Related
So I don't use virtual env often. In this particular case I need to have a desktop app for a client where they can input a CSV and then I use pandas to modify columns and such and return it to them. For the desktop app I'm using electron, and I already have been able to send and return data between the two processes it's using. To work on this I've been using this tutorial:
https://efficientcoder.net/connect-python-3-electron-nodejs-build-desktop-apps/
So I set up my virtualenv but when I ran the process it said that any module I had installed in the virtualenv was unable to be found. I ran a check and it's using the python from my system not the env from the parent directory. How can I fix this?
calc.py
import sys
print(sys.path)
>>> Outputs: ['C:\\Users\\*Me*\\Projects\\electron-poc\\py', 'C:\\Python310\\python310.zip', 'C:\\Python310\\DLLs', 'C:\\Python310\\lib', 'C:\\Python310', 'C:\\Python310\\lib\\site-packages']
I have the following problem:
I have to run some test/diagnostic Python script on a Windows system. Due to explicit requirement, the system has no default system-wide Python instance, but there are two different Python instances installed, used locally by applications running on the system. However, both these instances lack some basic modules my script uses (like logging, urllib, configparser etc.).
I want to run %PYTHONPATH%\python.exe myscript.py where %PYTHONPATH% points to one of the installed Python instances, but install the required additional modules "somewhere" outside %PYTHONPATH% (preferrably, in the same directory where my script is installed) so that my script can use them.
As my script is a test tool, it should not modify the OS or installed software, so the Python installation under %PYTHONPATH% should not be changed in any way.
It is also expected that the installation can be fully automated, ie. the best way to install would be just have the modules in the same .zip file with my script which is unpacked onto the target path.
It is also important that the system has no Internet access, so I have to download required files on another machine and copy them to the target system.
Can you guide me how to do it?
I found an answer myself - it is quite simple:
obtain the zip file containing standard modules from the appropriate Python version distribution (in my case it was the file python38.zip, it is inside the main zip file downloadable from Python site)
Unpack the contents of this file to c:\mydir\Python38\site-packages, where c:\mydir is the directory containing my script
set the environment variable PYTHONUSERBASE=c:\mydir before running my script
Now I can run the script and it finds all "missing" standard modules in c:\mydir\Python38\site-packages.
I think what youy are seeking for is a python virtual environment.
( internet needed )
Check :
https://docs.python.org/fr/3/library/venv.html
Then for the installation, you can creat a .exe file containing all dependencies.
(no internet needed)
Check:
https://www.pyinstaller.org/
My team is enjoying using python to solve problems for our business. We write many small independent scripty applications.
However, we have to have a central windows box that runs these along with legacy applications.
Our challenge is going through a build and deploy process.
We want to have Bamboo check the script out of git, install requirements and run tests, then if all is green, just deploy to our production box.
We'd like libraries to be isolated from script to script so we don't have dependency issues.
We've tried to get virtualenvs to be portable but that seems a no go.
Pex looked promising, but it doesn't work on windows.
Ideally you'd see a folder like so:
AppOne
/Script.py
/Libs
/bar.egg
/foo.egg
AppTwo
/Script2.py
/Libs
/fnord.egg
/fleebly.py
Are we thinking about this wrong? What's the pythonic way to distribute scripts within an enterprise?
You may be able to do that with a neat if relatively unknown feature that was sneaked into Python 2.6 without much ado: executing zip files as Python applications. It got a bit (just a bit) more of publicity after PEP 441 (which is the one PEX is inspired in), although I think most people is still unaware of it. The idea is that you create a zip file (the recommeded extension is .pyz or .pyzw for windowed applications, but that's obviously not important) with all the code and modules that you want and then you simply run it with Python. The interpreter will add the contents of the zip file to sys.path and look for a top level module named __main__ and run it. Python 3.5 even introduced the convenience module zipapp to create such packaged applications, but there is really no magic in it and you may as well create it by hand or script.
In your case, I guess Bamboo could do the check out, dependency install and tests in virtualenvs and then package the application along with the environment libraries. It's not a one-click solution but it may do the trick without additional tools.
TL:DR:
Use Docker
A short story long:
You can use docker to create an independent image for every script that you want to deploy.
You can install a python image (slim is the lightest) as a base environment for each script or a group of scripts/applications and use it like a "virtualenv" in which you can install all your dependencies for that script.
There is also an integration for Bamboo and Docker which you may find useful.
Here is the Docker documentation for reference.
You can test each script individually in a separated container and if it passes then you can use the same container to deploy it in your main server.
It is not exactly what you are asking, but you can use this solution in every platform (Windows, Linux, etc.), you can deploy all your scripts to the enterprise server (or anywhere for that matter) and use them across your company.
Disclaimer: This is not THE solution, it is a solution that I am aware of which applies to the time of this answer (2017)
Another possibility is pyinstaller. It creates an executable that can be deployed. Python is not even required to be installed on the deployed production box. It is harder to debug problems that occur only on the deployed box. You also can't modify the scripts on the deployed box which depending on your trust of the owners of the machine is either a positive or negative. See http://www.pyinstaller.org/
As I understand it, you want to create self-contained application directories on a build server, then copy them over to a production server and run scripts directly from them. In particular, you want all dependencies (your own and external packages) installed within a Libs subdirectory in each application directory. Here's a fairly robust way to do that:
Create the top-level application directory (AppOne) and the Libs subdirectory inside it.
Use pip install --ignore-installed --target=Libs package_name to install dependencies into the Libs subdirectory.
Copy your own packages and modules into the Libs subdirectory (or install them there with pip).
Copy Script.py into the top-level directory.
Include code at the top of Script.py to add the Libs directory to sys.path:
import os, sys
app_path = os.path.dirname(__file__)
lib_path = os.path.abspath(os.path.join(app_path, 'Libs'))
sys.path.insert(0, lib_path)
This will make packages like Libs\bar.egg and modules like Libs\fleebly.py available to your script via import bar or import fleebly. Without code like this, there is no way for your script to find those packages and modules.
If you want to streamline this part of your script, there are a couple of options: (1) Put these lines in a separate fix_path.py module in the top-level directory and just call import fix_path at the start of your script. (2) Create a Libs\__init__.py file with the line sys.path.insert(0, os.path.dirname(__file__)), and then call import Libs from your script. After that, Libs\x can be imported via import x. This is neat, but it's a nonstandard use of the package and path mechanisms (it uses Libs as both a library directory and a package), so it could create some confusion about how importing works.
Once these directories and files are in place, you can copy this whole structure over to any Windows system with Python installed, and then run it using cd AppOne; python Script.py or python AppOne\Script.py. If you name your top-level script __main__.py instead of Script.py, then you can run your app just by executing python AppOne.
Further, as #jdehesa pointed out, if your script is named __main__.py, you can compress the contents of the AppOne directory (but not the AppOne directory itself) into a file called AppOne.zip, and then copy that to your production server and run it by calling python AppOne.zip. (On Python 3.5 or later, you can also create the zip file via python -m zipapp AppOne if your script is called __main__.py. You may also be able to use python -m zipapp AppOne -m Script if your script is called Script.py. See https://docs.python.org/3/library/zipapp.html.)
This kind of thing can be easily dealt with python setup.py
Sample setup.py
from setuptools import setup
setup(
name=name_for_distribution,
version=version_number,
py_modules=[pythonfiles],
install_requires=[
python packages that need to be installed
]
)
Create a virtual environment , activate it and run :
python setup.py install
I feel this is the most pythonic way to distribute and package your project.
Reading links:
https://pythonhosted.org/an_example_pypi_project/setuptools.html
https://docs.python.org/2/distutils/setupscript.html
I currently have an executable file that is running Python code inside a zipfile following this: https://blogs.gnome.org/jamesh/2012/05/21/python-zip-files/
The nice thing about this is that I release a single file containing the app. The problems arise in the dependencies. I have attempted to install files using pip in custom locations and when I embed them in the zip I always have import issues or issues that end up depending on host packages.
I then started looking into virtual environments as a way to ensure package dependencies. However, it seems that the typical workflow on the target machine is to source the activation script and run the code within the virtualenv. What I would like to do is have a single file containing a Python script and all its dependencies and for the user to just execute the file. Is this possible given that the Python interpreter is actually packaged with the virtualenv? Is it possible to invoke the Python interpreter from within the zip file? What is the recommended approach for this from a Python point of view?
You can create a bash script that creates the virtual env and runs the python scripts aswell.
!#/bin/bash
virtualenv .venv
.venv/bin/pip install <python packages>
.venv/bin/python script
I am running web2py on a Windows machine.
I'm working on an application, but it keeps erroring because it says the module I'm trying to use isn't installed. It is however installed in my local python install.
How can I install modules so that web2py can recognize them?
web2py recognize any module you have in your local Python installation, unless you have a module with the same name under /modules folder of your application.
If you are on windows I do not recommend the use of .exe version of web2py (this version is only for studies) and it has a self contained isolated Python interpreter.
Make sure you are using source version of web2py and Python 2.5+ on your windows.
web2py should import any module from your Python path, also you can drop modules in app/modules folder ], then web2py will check there first when import something.
If you are using the Windows binary version (i.e., web2py.exe), note that it includes its own Python interpreter, which means it will not use your installed version of Python and will therefore not see any of your installed modules. You can put Python modules in the /web2py/site-packages folder (which is created the first time you run the binary version), but the better approach is probably just to run the source code version of web2py. It's just as easy -- simply download and unzip the source code package, and instead of clicking on web2py.exe, you click on web2py.py (or at a command prompt, cd to the web2py directory and enter python web2py.py).
What about add your local module path into sys.path variable?
sys.path.apend('/path/to/your/module/directory')
By the way, which module is not found by web2py