The library I'm working on generates python files according to an executable (which turns ANTLRv4 .g4 files into python files), and I have the following install step:
import os
import subprocess
from setuptools import setup
from setuptools.command.install import install
class AntlrInstallCommand(install):
def run(self):
output_dir = compile_grammar()
print(f"Compiled ANTLRv4 grammar in {output_dir}")
install.run(self)
def compile_grammar():
parser_dir = os.path.join(os.path.dirname(__file__), "my_project/grammar")
subprocess.check_output(
["antlr", "MyGrammar.g4", "-Dlanguage=Python3", "-visitor", "-o", "gen"],
cwd=parser_dir,
)
# The result is created in the subfolder `gen`
return os.path.join(parser_dir, "gen")
setup(
...
install_requires=[
"setuptools",
"antlr4-python3-runtime==4.9.2",
...
],
cmdclass={"install": AntlrInstallCommand},
license="MIT",
python_requires=">=3.6",
)
Which works great if I'm pip install'ing the project on a machine that has antlr installed (since I'm calling it via subprocess).
Ideally, attempting to do this on a machine that doesn't have antlr installed would first install the executable(with the correct version) in either a system directory like /usr/bin, or whatever relevant python bin directory we're working in, but right now it errors out with the following message(which is expected):
running install
error: [Errno 2] No such file or directory: 'antlr'
----------------------------------------
ERROR: Failed building wheel for my_project
I see a couple of solutions each with slight caveats:
sympy uses ANTLR, but it requires the user to install antlr first. See here
setuptools-antlr allows me to download an antlr jar as a giant blob in a python package, and then I can invoke it here. However, the version doesn't match mine (which is 4.9.2).
java2python precompiles the files for me and writes them into the github repo. However, these files are extremely large and are very hard to read as they're autogenerated. If I slightly modify the grammar and don't modify the parser it would also lead to unexpected bugs. As a result, I would like to hide this complexity from the repository as it's tangential to development.
If I can get the right version of the antlr binary and be able to invoke it at install time, that would be optimal. Otherwise I'm okay with picking one of these alternatives. Any suggestions for either case would be appreciated.
Related
I'm on MacOSX (12.0.1) and with Python 3.9. I want to create a simple python package for personal use. Upon creating the package using python setup.py install, almost everything works: I can import the package when using python, etc. However, I've tried to follow every tutorial online to create an associated executable script. I.e., a command that I can execute from the shell that contains some functionality from the package I made. However, nothing has worked.
My setup.py code:
from setuptools import setup
setup(name='my_package',
version='1.0.0',
description='heeheehoohoo',
author='Me',
author_email='me#me',
url='me.com',
packages=['my_package'],
entry_points={
'console_scripts': ['mypkg=my_package:run']},
install_requires=['cairosvg',
'selenium',
'PyPDF2',
],
include_package_data=True,
zip_safe=False
)
And under my_package/__init__.py I have:
from . mine import main
def run():
import argparse
parser = argparse.ArgumentParser(prog = 'eeeeeee', description = 'eeeeee')
parser.add_argument('eeeeee', help = 'eeeeeee')
args = parser.parse_args()
print(f'eeeee ...')
main(args.eeeeeee)
print(f'Success!')
Everything gets installed, yet for some reason when I try to execute $ mypkg, I get zsh: command not found: mypkg. From python, I can import the function and directly try to execute run(). And strangest of all, each tutorial I have seen that has done anything like this can execute the commands without a problem once they'd executed python setup.py install.
Thank you!
Setting pip to the respective version of python and using pip install . instead of python setup.py install did the trick. However, it's still strange that python setup.py install does not work...
Given a package random.whl containing hello.py:
print("Hello World!")
Is there a way to create a setup.py, setup.cfg or pyproject.toml, that when executed, will install the package in such a way that hello.py will be executed every time Python is started?
pip install random.whl
python unrelated.py # First prints "Hello World", then continues on.
I know it's possible to hook on readline.py that Python automatically loads, but is there a different and less "hacky" way to achieve it?
Some impossible ways that I thought of:
Running a post-install script on a .whl distribution (post-install is only avaiable on sdist).
Modifying PYTHONSTARTUP env variable or copying files.
Changing the import machinery.
While being a security risk, a method achieving it is good for implementing debuggers or auditing tools without requiring a change in either pre-compiled or post-compiled Python code, or used for penetration testing in side-channel attacks.
So far, using sitecustomize.py and publishing an sdist with a custom install command was the most reliable, and worked in virtual environments unlike usercustomize.py or .pth files.
Relevant setup.py code:
import sys
import os
import setuptools
import sysconfig
from setuptools.command.install import install
class PreInstall(install):
def run(self):
site_packages_dir = sysconfig.get_path("purelib")
sitecustomize_path = os.path.join(site_packages_dir, "sitecustomize.py")
if os.path.exists(sitecustomize_path):
raise FileExistsError("Site customize file already exists. "
"Please remove it before installing.")
install.run(self)
setuptools.setup(
name='...',
version='0.0.1',
py_modules=["sitecustomize"],
cmdclass={'install': PreInstall,},
)
It still doesn't work with .whl distributions as it might overwrite an existing sitecustomize without being able to check.
I am using cython to cross-compile external python module. I am using python3.6 on the host and python3.5 on the target. Also I am compiling on x86_64 for target aarch64.
My setup.py looks like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import builder_config
import os
os.environ["PATH"] = builder_config.PATH
os.environ["CC"] = builder_config.COMPILER
os.environ["LDSHARED"] = builder_config.COMPILER + " -lpython3.5m -shared"
os.environ["CFLAGS"] = builder_config.CFLAGS
os.environ["LDFLAGS"] = builder_config.LDFLAGS
os.environ["ARCH"] = "aarch64"
setup(
ext_modules = cythonize((Extension("my_ext", ["file1.pyx", "file2.pyx", "file3.pyx", "file4.pyx", "file5.pyx"]))),
)
When I run python3.6 setup.py build_ext -i I get a file named: my_ext.cpython-36m-x86_64-linux-gnu.so
My problem is that on the target the library will not be loaded unless the name is changed to:
my_ext.cpython-35m-aarch64-linux-gnu.so
How can I change the generated filename?
As stated in the comments, what you are trying to achieve is unsafe.
You can work around the architecture tag with the environment variable _PYTHON_HOST_PLATFORM (e.g. you can change it in your sitecustomize.py). But, if the modules are actually incompatible (and they most likely are), you will only get core dumps later on.
I don't think you can work around the major Python version.
In order to come back to safer grounds, I would try to rely on portable solutions. For example, it doesn't look official, but we can find some articles on the web about Conda and aarch64 (e.g. you can look for 'Archiconda'). One more time, you wouldn't be able to simply copy the conda environments from one machine to another, but, you can freeze these environments (via a 'conda export') and build similar ones on the target machine.
An option is to upgrade the target interpreter to v3.6 if that's possible for you.
Another option is to install v3.5 on the machine you're using to build with that interpreter. It's pretty uncomplicated to get several different versions of the python interpreter installed on the same machine. I don't know your specifics so I can't provide any links but I'm sure a quick search will get you what you need.
I am trying to install Wireshark. There is a dependency of atk package. I downloaded atk-2.32.0 from net. atk-2.32.0 could only be install by meson.
Below are the steps I followed for installation.
Downloaded atk
Download meson
Moved to atk directory.
Used command
../meson-0.50.0/meson.py _build .
>>However it is give this error:
>>"meson.build:97:0: ERROR: Subproject directory not found and glib.wrap
>>file not found."
Please help to resolve this issue.
To add a Meson subproject, you need to create a .wrap file and put it in the subprojects directory of the main Meson project.
If the subprojects directory doesn't exist in your project directory, you need to create it. Then create a file in the subprojects directory called glib.wrap with the following contents:
[wrap-git]
url = https://gitlab.gnome.org/GNOME/glib.git
[provide]
dependency_names = glib-2.0, gobject-2.0, gio-2.0
This extremely basic .wrap file should work, but if you're shipping a project (or a cloned repository requires it) you may prefer specifying a given version of GLib by downloading a versioned .wrap file from Meson's WrapDB (scroll down to glib).
I'm extremely new to Meson subprojects, but it seems like the .wrap file should have been included in the cloned repository in the first place? Also on Linux at least GLib should "just work", or at least it should be available with a compiler.find_library() call in your meson.build like so:
deps = [
meson.get_compiler('c').find_library('glib'),
...
]
executable(
meson.project_name(),
dependencies: deps,
...
)
Please sound off in the comments if these don't quite work. I'm posting an answer here because this is the top DuckDuckGo search result for this error message, but in my case I was trying to add a different subproject to a project of my own, so it would have been quite a bit more work for me to test this particular example.
When I make a python package, I often include some non-python files such as the requirements.txt, configuration files, documentations or small data files. Using the following script, it is easy for any user to pip install the package directly from github:
from setuptools import setup, find_packages
import os
current_folder = os.path.dirname(os.path.abspath(__file__))
version = '0.0.0.0.0.0' # year.month.day.hour.minute.second
with open(os.path.join(current_folder,'VERSION')) as version_file:
version = version_file.read().strip()
setup(name='package_name',
version=version,
description='My Little Python Package',
url='https://github.com/github_user_name/package_name',
author='Me',
author_email='me#email.com',
license='MIT',
include_package_data=True,
packages=find_packages(exclude=['docs', 'tests']),
package_data={'': ['package_name/resources/*', 'package_name/conf.yml']},
install_requires=[
'numpy',
'scipy',
'PyMySQL',
'PyYAML'
],
zip_safe=False)
Then, if I do: pip install git+https://github.com/github_user_name/package_name, the package is installed with the necessary files. Sadly, it is not possible to upload files that are more than 50 Mb on GitHub. Let say I want to provide a 1 Gb file with the package (example: a dataset of 1 million small texts). What is the best practice to do so ? Some suggestions:
Give an additional link on the GitHub's readme as a prerequisites.
Add a line in the setup.py file that downloads the data (ideally it stores the data in the package directory, not sure if possible).
Create an explicit script in the package that fetches the data.
Other ?
The only example I know is nltk, which uses the 3rd otpion.
You should look Into Git Large file storage. It will create pointers of the large files to another server so you can exceed the 50 MB size limit.
From their Getting Started guide:
# This part is only done once
git lfs install
git lfs track "*.psd" # Or your file extension
git add .gitattributes
# Normal workflow begins
git add file.psd
git commit -m "Add design file"
git push origin master