I've got a cloud function I deployed a while ago. It's running fine, but some of its dependent libraries were updated, and I didn't specify == in the requirements.txt, so now when I try to deploy again pip can't resolve dependencies. I'd like to know which specific versions my working, deployed version is using, but I can't just do a pip freeze of the environment as far as I know.
Is there a way to see which versions of libraries the function's environment is using?
I would suggest using the pip list as it has the option to display outdated packages using the --outdated (-o) flag.
You can check this documentation for pip list for additional information and flags that would be useful to your project.
I am still unaware how to get this information directly from Google Cloud Platform. I think it may not be surfaced after deploy. But a coworker had a workaround if you've deployed from a CI pipeline: Go back and look in that pipeline's logs to see which packages got installed upon deploy. It's printed. This didn't quite save me, because I'd deployed my function manually from a terminal, but it got me closer, because I could see which versions were being used around that time.
Related
I have read through a couple of dozens of writeups on how to equip a modern python project to automate linting, testing, coverage, type checking etc (and eventually deploy it on cloud servers. Not interested in the latter part yet.)
I am thinking about using conda as my environment manager. This would give me the advantage of being able to install non python packages if needed and also by creating the project environment with a specified python version I believe it would replace pyenv.
Within the newly conda created environment I would use poetry to manage dependencies and to initialize its TOML and add other needed python packages which would be installed by pip from PyPI.
Did I get the above right?
To complete the "building" phase I am also looking into using pytest and pytest-cov for code coverage, mypy/pydantic for type checking and black for formatting.
All of this should work both on my local development machine but then when I push on GitHub trigger an Action and perform the same checks, so that any contributor will go through them. Currently managed to do it for pylint, pytest and coverage for very simple projects without requirements.
Does all of this make sense? Am I missing some important component/step? For example I'm trying to understand if tox would help me in this workflow by automating testing on different python versions, but haven't come to grips integrating this flood on (new to me) concepts. Thanks
I have jupyter-notebook running on my own Mac with the caylsto-processing library plugged in so I can run processing scripts in a notebook in a browser tab. But I am trying to be able to run this all in binder, so that I can share my processing scripts with students during class. I created a Github repository and have it linked to a binder, and the binder builds and launches, but the only kernel available is python 3.
I have read that I can include a bunch of configuration files, but I'm new to these and I don't see any examples that bring in the calysto-processing kernel, so I'm unsure on how to proceed.
Screenshot of my binder with the jupyter-notebook with a processing script - but when you click on kernels, the only kernel it shows is python:
Any help would be appreciated.
Very good question. Ayman suggestion is good.
I've just installed calysto_processing and noticed 3 things are necessary:
installing the calysto_processing package via pip,
running install on the calysto_processing package.
installing Processing.
First point should be easy with requirements.txt.
I'm unsure what the best option is for the second step (maybe a custom setup.py ?).
Step 3 feels the trickiest.
Installing Processing currently isn't supported with apt-get so Dockerfile might be way forward (even through mybinder recommend that only as a last resort).
Let's assume a Dockerfile would contain all the steps to manually download/install processing (and I'm not super experienced with Docker at the moment btw), it will need to be executed which will require a windowing system to render the Processing window.
I don't know how well that plays with Docker, sounds like it's getting into virtual machine territory.
That being said, looking at the source code right here:
Processing is used only to validate the sketch, and pull syntax errors to display them otherwise.
ProcessingJS is used to actually render the processing code in a <canvas/> element within the Jupyter Notebook
I'm not sure what the easiest way to run the current calysto_processing in mybinder as is.
My pragmatic (even hacky if you will) suggestion is to:
fork the project and remove the processing-java dependency (which means might loose error checking)
install the cloned/tweaked version via pip/requirements.txt (pip can install a package from a github repo)
Update I have tried the above: you can run test kernel here
The source is here and the module is installed from this fork which simply comments out the processing-java part.
In terms of the mybinder configuration it boils down to:
create a binder folder in the repo containing the notebook
add requirements.txt which points to the tweaked version of calysto_processing stripped off the processing-java dependency: git+https://github.com/orgicus/calysto_processing.git#hotfix/PJS-only-test
add postBuild file which runs install on the calysto_processing module: python -m calysto_processing install --user
Notes
With this workaround java error checking is gone
Although Processing syntax is used it's execute as javascript and rendered in <canvas/> using ProcessingJS: this means no processing-java libraries, no threads or other java specific features,(buggy or no 3D),etc. just basic Processing drawing sketches
It might be worth looking at replacing ProcessingJS with p5.js and checking out other JS notebooks ? (e.g. Observable or IJavascript)
When I build my python project, tox is taking a really long time to set up the environment. I've traced the slowness to the fact that when it's trying to up the environment in .tox, it's trying to access an old pypi repo that my company has since decommissioned in favor of a new one (which works fine). Most stuff I do tries to access this old repo many times (~100), waiting a second or two to timeout on each connection.
I've done everything I can think of to cleanse my machine of links to the old repository, but it's still lingering somewhere. I've tried:
Uninstalling & re-installing both tox & pip
Blowing away ~/.pip
grepping through my project folders for the url of the old repo
grepping through in /Library/Frameworks/ for the url of the old repo (this is on OSX Yosemite 10.10.5)
No dice on any of them.
Where else should I look? Is there some obscure nook or cranny I haven't thought of? Where does pip look for repos by default if you don't explicitly specify one?
Can it be tox settings? This and this parameters seem like they could be relevant. If you don't keep your tox files in the same place as the project itself, you could've missed it while grepping.
Found it through some painstaking println debugging & reading through the source on github. It was in ~/Library/Application Support/pip.
For anyone finding this in the future, pip has some default directories it looks in.
I'm integrating a java project by using maven. This project should be pushed to a server(linux) with limited access(cannot use pip). I've added all the dependencies for my java component adding them to the pom.xml, but In part of my code I've used client/server approach to call a python script, which requires Pandas and Sklearn. Unfortunately, I've memory issue and cannot copy the entire directory of these libraries into server.
I'm afraid if maven could help me to download the python dependencies or if there is another efficient way of adding python dependencies into repository. I've done some research but couldn't find any helpful way to address that. I'm a beginner in python and I'd be happy if you could help me address that.
If limited access (Assuming you can download but cannot install) is your issue you can download Anaconda and it doesn't need to be installed with root access. It install everything to your home directory and creates a virtual environment for you. That way you can use pip too. Just make sure you call your python script with your anaconda virtual environment python.
/home/USER/anaconda2/envs/ml/bin/python script_name.py
I'm tryin' to find a way to install a python package with its docs.
I have to use this on machines that have no connection to the internet and so online help is not a solution to me. Similar questions already posted here are telling that this is not possible. Do you see any way to make this easier as I'm currently doing this:
downloading the source archive
extracting the docs folder
running sphinx
launching the index file from a browser (firefox et al.)
Any ideas?
P.S. I'm very new to Python, so may be I'm missing something... And I'm using Windows (virtual) machines...
Edit:
I'm talking about two possible ways to install a package:
installing the package via easy_install (or any other to me unknown way) on a machine while I'm online, and then copying the changes to my installation to the target machine
downloading the source package (containing sphinx compatible docs) and installing the package on the target machine off-line
But in any case I do not know a way to install the package in a way that the supplied documentations are installed alltogether with module!
You might know that there exists a folder for the docs: <python-folder>/Doc which will contain only python278.chm after installation of Python 2.78 on Windows. So, I expect that this folder will also contain the docs for a newly installed package. This will avoid looking at docs for a different package version on the internet as well as my specific machine setup problems.
Most packages I'm currently using are supplied with documentation generated with sphinx, and their source package contains all the files necessary to generate the docs offline.
So what I'm looking for is some cli argument for a package installer like it's common for unix/linux based package managers. I did expect something like:
easy_install a_package --with-html-docs.
Here are some scenarios:
packages have documentation included within the zip/tar
packages have a -docs to download/install seperately
packages that have buildable documentation
packages that only have online documentation
packages with no documentation other than internal.
packages with no documentation anywhere.
The sneaky trick that you can use for options 1 & 3 is to download the package as a tar or zip and then use easy-install archive_name on the target machine this will install the package from the zip or tar file including (I believe) any documentation. You will find that there are dependencies that are unmet in some packages - those should give an error on the easy install mentioning what is missing - you will need to get those and use the same trick.
A couple of things that are very handy - virtual-env will let you have a library free version of python running so you can get the requirements and pip -d <dir> which will download without installing storing your packages in dir.
You should be able to use the same trick for option 2.
With packages that only have on-line documentation you could look to see if there is a downloadable version or could scrape the web pages and use a tool like pandoc to convert to something useful.
In the 5 scenario I would suggest raising a ticket on the package stating that lack of accessible documentation makes it virtually unusable and running sphinx on it.
In scenario 6 I suggest raising the ticket but missing out virtually and avoiding the use of that package on the basis that if it has no documentation it probably has a lot of other problems as well - if you are a package author feeling slandered reading this then you should be feeling ashamed instead.
Mirror/Cache PyPi
Another possibly is to have a linux box, or VM, initially outside of your firewall, running a cached or mirroring service e.g. pipyserver, install the required packages through it to populate the cache and then move it, (or its cache to another pip server), inside the firewall and you can then use pip with the documented settings to do all your installs inside the firewall. See also the answer here.