Develop Reference

Install Python to Self-hosted Windows build agent

I have installed Windows agent and I need to be able run Python scripts. I know I need to install Python, but I have no idea how.
I added Python files from standard installation to
$AGENT_TOOLSDIRECTORY/
Python/
3.8.2/
x64/
{tool files}
x64.complete
Restarted agent, but what now? How to put it to Capabilities?
What I'm missing?
EDIT:
I need to run this YAML task
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.x'
addToPath: true
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt
displayName: 'Install dependencies'
- task: BatchScript#1
displayName: 'Run script make.bat'
inputs:
filename: make.bat
arguments: html

I have set up a self-hosted agent on a Windows 10 laptop, (for which I have admin access), and I'm running Azure DevOps Express 2020.
I found, downloaded and installed an agent according to the instructions at Download and configure the agent. I used vsts-agent-win-x64-2.170.1.zip and set this up to run as a service, (I guess anyone running it manually needs to double check that it's runnning at show time). I also ran the install command as admin in powershell!
To install a Python version I need to download the appropriate installer from the ftp site at Python.org, eg. for 3.7.9 I've used python-3.7.9-amd64.exe.
I then run this from the command line (CMD run as administrator) without UI with:
python-3.7.9-amd64.exe /quiet InstallAllUsers=0 TargetDir=$AGENT_TOOLSDIRECTORY\Python\3.7.9\x64 Include_launcher=0
(other options for install available in python docs)
Once this is complete, (and it runs in background so will take longer than the initial command), you need to create an empty {platform}.complete file (as described here), in my case this is x64.complete.
This then worked! I did restart the server for this first version, but I've added other python versions since without needing to. My pipeline task was simply:
steps:
- task: UsePythonVersion#0
displayName: 'Use Python $(python.version)'
inputs:
versionSpec: '$(python.version)'
(with a variable python.version set us as a list of versions 3.7.9, 3.8.8)
One key element for me was the file structure, where the documentation says {tool files} this means the python.exe file and other common dirs such as Lib and Scripts. I initially installed these in a sub-dir which didn't work. So it should look like this:
$AGENT_TOOLSDIRECTORY/
Python/
3.7.9/
x64/
Doc/
Lib/
Scripts/
python.exe
...etc...
x64.complete
To be honest I'm mostly relieved that this worked without too much trouble. I gave up trying to get Artifacts to work on-prem. In my limited experience all of this is much easier, and better, on the cloud version. Haven't yet persuaded my employer to take that leap however...

For this issue, in order to use the python version installed in your on-premise machine. You either need to point to the python.exe physical path in cmd task. Or add the python.exe path to environment variable path manually in powershell task. For example:
To use local python in powershell task:
$env:Path += ";c:\{local path to}\Python\Python38\; c:\{local path to}\Python\Python38\Scripts\"
python -V
To use python in CMD task:
c:\{local path to}\Python\Python38\python.exe -V
c:\{local path to}\Python\Python38\Scripts\pip.exe install
So, I think to run python script with private agent, just make sure python is installed locally, then point to python.exe path. You can refer to this case for details.

I added these 4 Tasks before being able to execute python on my pipeline with a vs2017-win2016 agent:
Use Python 3.x
steps:
- task: UsePythonVersion#0
displayName: 'Use Python 3.x'
Use Pip Authenticate
steps:
- task: PipAuthenticate#1
displayName: 'Pip Authenticate'
Use Commandline task
steps:
- script: |
python -m pip install --upgrade pip setuptools wheel
failOnStderr: true
displayName: 'install pip for setup of python framework'
Use Commandline task
steps:
- script: 'pip install -r _python-test-harness/requirements.txt'
failOnStderr: true
displayName: 'install python framework project''s specific requirements'
Hope that helps

How to eliminate Error Code Signing Qt Framework - unsealed contents present in the root directory of an embedded framework

I am attempting to codesign and notarize a MacOS .app bundle generated by Pyinstaller. The application is developed using Python3, PyQt bindings, and the Qt toolkit.
Here are the versions:
PyInstaller 3.5
Python version: 3.7.4 (v3.7.4:e09359112e, Jul 8 2019, 14:54:52)
[Clang 6.0 (clang-600.0.57)]
Qt version: 5.13.0
PyQt version: 5.13.1
sip version: 4.19.19
The app itself runs great on the MAC, the problems start when attempting to code sign the app so it can be notarized. The issue I'm illustrating here has to do with the Qt Frameworks. If anyone can provide a set of steps that will successfully sign one of the Qt Frameworks in the pyinstaller generated .app bundle I'd really appreciate it.
Here is my process...
The .app bundle is created by Pyinstaller and runs fine.
The first step is to fix the folder structure created by Pyinstaller. It doesn't conform exactly to what Apple expects, but to be fair, what Qt installs doesn't either. Run this script to fix the QtCore framework in the .app bundle
Echo Fix the QtCore framework folders
qtfolder="/Users/xxx/Qt/5.13.1/clang_64/lib"
basefolder="/Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework"
cp -R $qtfolder/QtCore.framework/Versions/5/Resources $basefolder/Versions/5
ln -s $basefolder/Versions/5 $basefolder
mv $basefolder/5 $basefolder/Versions/Current
ln -s $basefolder/Versions/Current/Resources $basefolder
ln -s $basefolder/Versions/Current/QtCore $basefolder
This produces the following folder structure and files which I believe conform to what Apple expects. this is based on numerous blog posts etc.
QtCore# -> /Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework/Versions/Current/QtCore
Resources# -> /Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework/Versions/Current/Resources
Versions/
./Versions:
5/
Current# -> /Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework/Versions/5
./Versions/5:
QtCore*
Resources/
./Versions/5/Resources:
Info.plist
Now I start signing the files. There are many posts on what to sign and how to sign all of which seem outdated and contradictory. The best current advice I think is to sign everything. I've tried many combinations and they've all ended up with the same error.
sudo codesign --verbose --force --sign "Developer ID Application: (xxx)" /Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework/versions/5/resources/Info.plist
sudo codesign --verbose --force --sign "Developer ID Application: (xxx)" /Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework/versions/5/QtCore
sudo codesign --verbose --force --sign "Developer ID Application: (xxx)" /Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework/versions/5/_CodeSignature/CodeResources
sudo codesign --verbose --force --sign "Developer ID Application: (xxx)" /Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework/versions/5
At this point every file and folder in the framework that can be signed has been signed successfully. I don't sign the symlinks as I think that just resigns the underlying files. The final step is to sign the framework itself.
sudo codesign --verbose --force --sign "Developer ID Application: (xxx)" /Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework
this produces the following error message:
/Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework: replacing existing signature
/Users/xxx/Documents/PYINSTALLERTEST/dist/MyApp.app/Contents/MacOS/PyQt5/Qt/lib/QtCore.framework: **unsealed contents present in the root directory of an embedded framework**
Any ideas on what I'm doing wrong?

Python 3 flask install wkhtmltopdf on heroku

I have a problem to install the wkhtmltopdf binary on my heroku python app (flask).
A year ago (python 2) I already had an issue, but I was able to solve it by first adding the wkhtmltopdf-pack to the requirements and installing it on heroku and then setting the config var to WKHTMLTOPDF_BINARY=wkhtmltopdf-pack. Here is my old thread
Problem now:
I am trying to use the same approach for python 3, but no version of the wkhtmltopdf-pack works, every push gets rejected and I cant install it.
I tried these versions in the requirements:
wkhtmltopdf-pack==0.12.5
wkhtmltopdf-pack==0.12.4
wkhtmltopdf-pack==0.12.3
wkhtmltopdf-pack==0.12.3.0.post1
wkhtmltopdf-pack==0.12.2.4
I get these errors:
No matching distribution
or
error: can't copy 'bin/wkhtmltopdf-pack': doesn't exist or not a regular file
and I remember once it told me there was a SyntaxError and it could not decode something.
Alternative approach:
It seems it is also possible to use a buildpack, so I tried adding a buildpack:
heroku buildpacks:add https://github.com/dscout/wkhtmltopdf-buildpack.git
I see that the buildpack has been added, but there was no installation and there is also no config var for wkhtmltopdf. I dont understand how to trigger the installation, in all documantations for buildpacks its written "add the buildpack and you are ready to go".
Trying to create a PDF gives me a server error here:
OSError: No wkhtmltopdf executable found: "b''"
EDIT:
I managed to install the buildpack:
The push was successful, but no config var has been created and I have no clue what the path to the binary is.
EDIT
I was able to find the files through heroku bash:
app bin dev etc lib lib64 lost+found proc sbin sys tmp usr var
/ $ cd app
~ $ cd vendor
~/vendor $ dir
wkhtmltox
~/vendor $ cd wkhtmltox
~/vendor/wkhtmltox $ dir
lib
~/vendor/wkhtmltox $ cd lib
~/vendor/wkhtmltox/lib $ dir
libwkhtmltox.so libwkhtmltox.so.0 libwkhtmltox.so.0.12 libwkhtmltox.so.0.12.3
~/vendor/wkhtmltox/lib $ exit
Now I tried to all these files but all give an error:
OSError: wkhtmltopdf exited with non-zero code -11. error
Here is how I set the path:
# WKHTMLTOPDF config
if 'DYNO' in os.environ:
print ('loading wkhtmltopdf path on heroku')
MYDIR = os.path.dirname(__file__)
WKHTMLTOPDF_CMD = os.path.join(MYDIR + "/vendor/wkhtmltox/lib/", "libwkhtmltox.so")
else:
print ('loading wkhtmltopdf path on localhost')
MYDIR = os.path.dirname(__file__)
WKHTMLTOPDF_CMD = os.path.join(MYDIR + "/static/executables/bin/", "wkhtmltopdf.exe")

The best approach to get installed wkhtmltopdf on Heroku by getting the binary of wkhtmltopdf for python 3 instead of wkhtmltopdf-pack and you can achieve this by using pydf.
You can install it simply using pip like:
pip install python-pdf
or for Python 2:
pip install python-pdf==0.30.0
Unlike the buildpack based approach pydf installs with the wkhtmltopdf binary included making it very easy to use, and this is the right approach for Heroku.
But if you still want to stick with build-pack wkhtmltopdf, here's another solution you can give it a try:
Via: CLI Installation
$ heroku create --buildpack https://github.com/homelight/wkhtmltox-buildpack.git
Or Manually:
Add the following line to your .buildpacks file
https://github.com/homelight/wkhtmltox-buildpack.git
Please note that this buildpack is only compatible with the cedar-14 stack. You can use heroku stack:set cedar-14 to set the correct stack.

I was able to solve the problem on my own, following my first approach.
I found an other wkhtmltopdf-pack on pypi and added it to my requirements.txt:
wkhtmltopdf-pack-ng==0.12.3.0
Heroku was able to install this pack.
After that I added the config var for wkhtmltopdf:
heroku config:set WKHTMLTOPDF_BINARY=wkhtmltopdf-pack
The installation is now complete. I need to use the correct path now on my app:
if 'DYNO' in os.environ:
print ('loading wkhtmltopdf path on heroku')
WKHTMLTOPDF_CMD = subprocess.Popen(
['which', os.environ.get('WKHTMLTOPDF_BINARY', 'wkhtmltopdf-pack')], # Note we default to 'wkhtmltopdf' as the binary name
stdout=subprocess.PIPE).communicate()[0].strip()
else:
print ('loading wkhtmltopdf path on localhost')
MYDIR = os.path.dirname(__file__)
WKHTMLTOPDF_CMD = os.path.join(MYDIR + "/static/executables/bin/", "wkhtmltopdf.exe")
Thats it.

error: cannot locate an Oracle software installation

I'm working on Plone.
PRELUDE
I've installed:
oracle-instantclient12.1-basic-12.1.0.1.0-1.x86_64.rpm
oracle-instantclient12.1-devel-12.1.0.1.0-1.x86_64.rpm
oracle-instantclient12.1-sqlplus-12.1.0.1.0-1.x86_64.rpm
and also cx_Oracle.
I've tested the installations and it's all ok: db connection successfully.
echo $ORACLE_HOME
/usr/lib/oracle/12.1/client64
echo $TNS_ADMIN
/usr/lib/oracle/12.1/client64/admin
echo $LD_LIBRARY_PATH
/usr/lib/oracle/12.1/client64/lib
THE PROBLEM
I've edited buildout.cfg as follows:
[...]
eggs =
Plone
Pillow
collective.documentviewer
Products.OpenXml
Products.AROfficeTransforms
tus
wildcard.foldercontents==2.0a7
**cx_Oracle**
[...]
I receive this error:
Unused options for buildout: 'environment-vars'.
Installing instance.
Getting distribution for 'cx-Oracle'.
error: cannot locate an Oracle software installation
An error occurred when trying to install cx-Oracle 5.1.3. Look above this message for any errors that were output by easy_install.
While:
Installing instance.
Getting distribution for 'cx-Oracle'.
Error: Couldn't install: cx-Oracle 5.1.3
I have no idea how to solve this.
"cannot locate an Oracle software installation" How to fix this?

Got the same problem, background is:
echo $ORACLE_HOME
/usr/lib/oracle/12.1/client64
But:
sudo env | grep ORACLE_HOME
yields nothing.
The solution:
sudo visudo
Then add the line :
Defaults env_keep += "ORACLE_HOME"
As found here

You must be sure that the right envvars are setted for the user that run the Plone instance.
The best way is to add those vars in the buildout configuration::
[buildout]
...
[instance]
...
environment-vars =
...
LD_LIBRARY_PATH /usr/lib/oracle/10.2.0.3/client64/lib
ORACLE_HOME /usr/lib/oracle/10.2.0.3/client64
(This is what I have on a CentOS installation)

Django-dilla "Unknown command: 'dilla'"

I'm unable to make django-dilla work with my django1.4 project. I've installed django-dilla through pip and I can import it properly from shell.
>>import dilla
>>dilla.__file__
'/Users/misterte/.envs/python2.7-Django1.4/lib/python2.7/site-packages/django_dilla-0.2beta-py2.7.egg/dilla/__init__.py'
I've added it to my installed apps just before south, and ran my syncdb command.
INSTALLED_APPS = (
[...]
'dilla',
'south',
)
But when I try to call it, it won't work.
$python manage.py dilla --cycles=30
Unknown command: 'dilla'
Type 'manage.py help' for usage.
$python manage.py run_dilla --cycles=30
Unknown command: 'run_dilla'
Type 'manage.py help' for usage.
Then, of course, no sub commands are present under the [dilla] app when running help.
$python manage.py help | grep dilla
# emptiness :(
Any clues? Does dilla work in the django1.4 layout?
Thanks!
A.

So I found the problem.
For some reason pip is not installing dilla under site packages. I can import dilla and the egg is present under my site-packages folder, but further inspection showed that there were no packages under dilla.
Solution: copy dilla into you folder manually. I downloaded it from github.
Also, I recommend you copy it to a 'vendor' subfolder in your project. This way you avoid probably having to do the same hack in your production server. Then you import app as 'vendor.dilla' in your installed apps.
A.