I'm maintaining a project that is using Bazel 0.5.4, and it needs to run on an environment where python does not exist. I'd like to point the application to using python3.
This issue is similar to Bazel 0.26.1 use Python3 on py_test, but the version of rules_python mentioned in that ticket is so new that it's not compatible with bazel 0.5.4
GOAL: Issue python commands with /usr/bin/python3, not python
As-is, the application fails when it's unable to find python
bazel build completes without an issue, but bazel test continues to look for python, and fail.
bazel test ... --test_output=all --python_top=//sandbox_app:python-3.8.10
...
/usr/bin/env: 'python': No such file or directory
================================================================================
INFO: Elapsed time: 0.658s, Critical Path: 0.10s
INFO: Build completed, 1 test FAILED, 2 total actions
//sandbox_app:sandbox_test FAILED in 0.1s
This didn't work
The following configuration worked for a similar user, described here:
https://groups.google.com/g/bazel-discuss/c/nVQ48R94S_8
But even after a bazel clean, bazel still continues to invoke python when running the tests.
WORKSPACE
git_repository(
name = "io_bazel_rules_go",
remote = "https://github.com/bazelbuild/rules_go.git",
tag = "0.5.4",
)
BUILD.bazel
package(default_visibility = ["//visibility:public"])
py_test(
name = "sandbox_test",
srcs = ["sandbox_test.py"],
default_python_version = "PY3",
srcs_version = "PY3",
)
py_runtime(
name = "python-3.8.10",
files = [],
interpreter_path = "/usr/bin/python3",
)
Version details:
$ bazel info release
release 0.5.4
It would minimize disruption to the application if this can be done without upgrading bazel.
sandbox_test.py
#!/usr/bin/python3
import unittest
class SandboxTest(unittest.TestCase):
def testRunSandbox(self):
# Valid for Python 3, syntax error for python2
print(print("Hello, Python!"))
if __name__ == '__main__':
unittest.main()
Possibly related issues
https://github.com/bazelbuild/bazel/issues/4815
https://github.com/bazelbuild/bazel/issues/200
Rules spec:
https://bazel.build/reference/be/python
Similar issues:
How do I select the runtime in bazel for Python and pip?
Bazel, python3 and python interpreter options
Getting Bazel to run using Python3 (syntax error)
Related
I am trying to build a GTK application in MacOS (Monterey, v. 12.4) that includes both C GTK components and Python GTK components. I am following the instructions from both here and here. I had minimal issues with the first part (although for some reason I got an error where jhbuild said cargo did not exist when building librsvg during the call to jhbuild build meta-gtk-osx-gtk3, despite .new_local/bin being at the front of PATH). The instructions there were simply:
sh gtk-osx-setup.sh
alias jhbuild="PATH=.new_local/bin:$PATH jhbuild"
jhbuild bootstrap-gtk-osx
jhbuild build meta-gtk-osx-bootstrap meta-gtk-osx-gtk3
In any case, the issue that I am having now is installing either set of bindings for Python. When I attempt to build either, jhbuild states both:
jhbuild#Cytocyberneticss-Mac-mini ~ % jhbuild build meta-gtk-osx-python-gtk3
Loading .env environment variables...
jhbuild build: A module called ''meta-gtk-osx-python-gtk3'' could not be found.
Usage: run_jhbuild.py [ -f config ] command [ options ... ]
jhbuild#Cytocyberneticss-Mac-mini ~ % jhbuild build meta-gtk-osx-python
Loading .env environment variables...
jhbuild build: A module called ''meta-gtk-osx-python'' could not be found.
Usage: run_jhbuild.py [ -f config ] command [ options ... ]
I do not have home-brew or MacPorts installed so neither of those could be getting in the way. I really am at a loss as to what the problems is here, when the other builds went fine. Any pointers would be greatly appreciated. Let me know if you need any more information about my setup.
As per the package maintainer:
I need to rewrite that Python wiki page, it's thoroughly out of date.
meta-gtk-osx-python-gtk3 got changed to meta-gtk-osx-python2-gtk3, and current versions of gtk-osx don't support gtk2. What's more, python2 is obsolete and its use is deprecated; you should use meta-gtk-osx-python3-gtk3. I haven't yet made a meta-gtk-osx-python3-gtk4 but you can easily do so in your own moduleset if your application is ready for it.
So simply use either:
jhbuild build meta-gtk-osx-python3-gtk3
jhbuild build meta-gtk-osx-python2-gtk3
This question will be irrelevant though as soon as the Wiki is updated...
I'm using PythonKit in my Swift project for MacOS. At the moment I'm using Python 2.7 but from MacOs 12.3 it isn't no more supported so I'm trying to migrate to Python 3 but it doesn't work.
func applicationDidFinishLaunching(_ notification: Notification) {
if #available(OSX 12, *) {
PythonLibrary.useVersion(3)
}
else {
PythonLibrary.useVersion(2)
}
let sys = Python.import("sys")
print("Python \(sys.version_info.major).\(sys.version_info.minor)")
print("Python Version: \(sys.version)")
print("Python Encoding: \(sys.getdefaultencoding().upper())")
sys.path.append(Bundle.main.resourceURL!.absoluteURL.path)
let checker = Python.import("checkLibrary")
_ = Array(checker.check())
}
This is the error message:
PythonKit/PythonLibrary.swift:46: Fatal error: Python library not found. Set the PYTHON_LIBRARY environment variable with the path to a Python library.
The code fail on MacOs 12 on line 9th line (let sys = Python.import("sys")), so I can't interact so sys in any way.
I've already tried to disable sandbox and Hardened Runtime but is seems useless.
I was having the same issue.
where python3
which python3
type -a python3
I could clearly see that Python3 was present using any of the above commands from terminal. Python3 wasnt something that I directly installed, (probably something I added during an install of XCode) but I could see it located at "/usr/bin/python3"
I had already removed the sandbox and disabled the hardened runtime.
Adding the environment variable to XCode did not work and Google ultimately told me to do what had already been done.
Finally, I decided to just perform a fresh install, following the blog as guidance.
https://www.dataquest.io/blog/installing-python-on-mac/#installing-python-mac
https://www.python.org/downloads/macos/ (direct URL for Python download)
After installing, everything worked as expected.
PythonLibrary.useVersion(3)
PythonLibrary.useLibrary(at: "/usr/local/bin/python3")
I could use either of the above methods, without adding the environment variable to the XCode scheme.
Hopefully that helps
when running bazel test Bazel seems to default to Python 2 even when --python-version flag is specified
bazel test //... --python_version PY3
INFO: From Testing //test:py-unit-tests:
==================== Test output for //test:py-unit-tests:
/usr/bin/env: 'python': No such file or directory
This is my BUILD file
py_test(
name = "py-unit-tests",
srcs = glob(["unit/**/*.py"]),
deps = [
],
main = "unit/unit_test_runner.py",
timeout = "short",
)
And the test file
import sys
import unittest
class TestGeneration(unittest.TestCase):
def test_base(self):
pass
def test_urdf(self):
self.assertEqual("hello", 'test')
if __name__ == '__main__':
unittest.main()
Bazel version: 3.3.1
Other notable things:
My system has both py2 and py3 installed
Py3 is located at /usr/bin/python3
Py2 is located at /usr/bin/python2
There is no /usr/bin/python
As described in https://github.com/bazelbuild/bazel/issues/11554, the issue is that your Bazel python stub has a #!/usr/bin/env python shebang. If python is not on the PATH like in your case, the stub will fail.
Previously the only solution would be to add a symlink from python to python3, e.g. by installing
sudo apt install python-is-python3
However since https://github.com/bazelbuild/bazel/commit/763dd0ce6e1644bf895231432f616427a11d385a has landed that stub shebang has become configurable. You can now define your own py_runtime (https://docs.bazel.build/versions/main/be/python.html#py_runtime)
Alternatively, since https://github.com/bazelbuild/bazel/commit/2945ef5072f659878dfd88b421c7b80aa4fb6c80 the default shebang has also become #!/usr/bin/env python3
These two commits are only available from Bazel 5.0.0 though, so for now you might have to build Bazel yourself to get them.
I am trying to run poetry install command, but I get the following error:
[EnvCommandError]
Command ['pip', 'install', '-e', '<PROJECT_FOLDER_PATH>'] errored with
the following return code 1, and output:
Obtaining file:///<PROJECT_FOLDER_PATH>
ERROR: Package '<subfolder>' requires a different Python: 3.6.8 not in '>=3.7,<4.0'
Where my project directory containing .toml file is marked as <PROJECT_FOLDER> (PROJECT_FOLDER_PATH is, correspondingly, it's full path), and it contains <subfolder>.
Part of my toml file:
[tool.poetry]
name = "<PROJECT_FOLDER>"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
[tool.poetry.dependencies]
python = "^3.7"
It seems that poetry tries to install the project itself as a dependency, but for some reason that I don't understand it seems the conflicting Python version. I temporarily solved it by setting python = "^3.6", but now the issue is back, as I need some package which only accepts python = "^3.7".
I am writing a python application which dependents on Scrapy module. It works fine locally but failed when I run it from aws lambda test console. My python project has a requirements.txt file with below dependency:
scrapy==1.6.0
I packaged all dependencies by following this link: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html. And also, I put my source code *.py at the root level of in the zip file. My package script can be found https://github.com/zhaoyi0113/quote-datalake/blob/master/bin/deploy.sh.
It basically does two things, first run command pip install -r requirements.txt -t dist to download all dependencies to dist directory. second, copy app python source code to dist directory.
The deployment is done via terraform and below is the configuration file.
provider "aws" {
profile = "default"
region = "ap-southeast-2"
}
variable "runtime" {
default = "python3.6"
}
data "archive_file" "zipit" {
type = "zip"
source_dir = "crawler/dist"
output_path = "crawler/dist/deploy.zip"
}
resource "aws_lambda_function" "test_lambda" {
filename = "crawler/dist/deploy.zip"
function_name = "quote-crawler"
role = "arn:aws:iam::773592622512:role/LambdaRole"
handler = "handler.handler"
source_code_hash = "${data.archive_file.zipit.output_base64sha256}"
runtime = "${var.runtime}"
}
It zip the directory and upload the file to lambda.
I found I get the runtime error in lambda Unable to import module 'handler': cannot import name 'etree' when there is a statement import scrapy. I didn't use etree in my code so I believe there is something used by scrapy.
My source code can be found at https://github.com/zhaoyi0113/quote-datalake/tree/master/crawler. There are only two simple python files.
It works fine if I run them locally. The error only appears in lambda. Is there a different way to package scrapy to lambda?
Based on the communication with Tim, the issue is caused by incompatible library versions between local and lambda.
The easiest way to resolve this issue is to use the docker image lambci/lambda to build a package with the command:
$ docker run -v $(pwd):/outputs -it --rm lambci/lambda:build-python3.6 pip install scrapy -t /outputs/
You need to provide the entire dependency tree, scrapy also has a set of dependencies (and they may also have dependencies).
The easiest way to download all the required dependencies is to use pip
$ pip -t packages/ install scrapy
This will download scrapy and all its dependencies into the folder packages.
Scrapy has lxml and pyOpenSSL as dependencies that include compiled components. Unless they are statically compiled they will likely require that the c-libraries they require are also installed on the lambda VM.
From the lxml documentation it requires:
libxml2 version 2.9.2 or later.
libxslt version 1.1.27 or later.
We recommend libxslt 1.1.28 or later.
Maybe try adding installation of these to your deploy script. You should be able to use (I'm making a guess at the package names) yum -y install libxml2 libxslt
Another good idea is to test your scripts on an Amazon Linux EC2 instance as this is close to the environment that Lambda executes in.