Develop Reference

Poetry add dependency that uses cython

I have a project which needs to depend on the latest commit of pysam, because I'm working in python 3.11.
This means building the package from source, so I do the following:
poetry add git+https://github.com/pysam-developers/pysam
However, I get an error which I think boils down to poetry not including cython in the build environment:
Unable to determine package info for path: /Users/agreen/Library/Caches/pypoetry/virtualenvs/rnacentral-pipeline-GU-1IkEM-py3.11/src/pysam
Fallback egg_info generation failed.
Command ['/var/folders/sg/3858brmd79z4rz781g0q__940000gp/T/tmpw8auvhsm/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output:
# pysam: no cython available - using pre-compiled C
Traceback (most recent call last):
File "/Users/agreen/Library/Caches/pypoetry/virtualenvs/rnacentral-pipeline-GU-1IkEM-py3.11/src/pysam/setup.py", line 345, in <module>
raise ValueError(
ValueError: no cython installed, but can not find pysam/libchtslib.c.Make sure that cython is installed when building from the repository
Cython is definitely installed, its in the pyproject.toml, and I can call it from the poetry shell, or import it in a python started in the poetry virtualenv. However, If I use the python from the command poetry is running, then indeed cython is not available.
I think I'm missing some configuration of the build, or some extra option to poetry add. The documentation isn't particularly clear about this use of cython - as far as I can tell it's all about using cython in the package I'm writing, which is not quite what I want.

Cython is a build dependency of pysam, but apparently pysam does not have a pyproject.toml and thus does not declare its build dependencies (Cython and maybe others). So this is a dead end.
If I were you I would build a wheel of pysam myself and tell Poetry to use this wheel until pysam releases wheels for Python 3.11 on PyPI themselves. Or I would use Python 3.10.
It seems like it is being worked on: https://github.com/pysam-developers/pysam/pull/1168

Failed building wheel for pyarrow in pypy3 interpreter

I am installing streamlit with pypy3 as interpreter in pycharm and stuck at this ERROR: Failed building wheel for pyarrow
I tried every solutions found on the web related with pyarrow, but seems like all solutions posted are for python as interpreter and not for pypy. If you guys have any solution, please let me know.
Command : pip install streamlit
Operating system : ubuntu 22.04
pypy3 version : 7.3.9
Traceback (most recent call last):
File "/home/lungsang/Desktop/streamlit/.env/bin/cmake", line 5, in <module>
from cmake import cmake
ModuleNotFoundError: No module named 'cmake'
error: command '/home/lungsang/Desktop/streamlit/.env/bin/cmake' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
I have installed cmake before running the command.

You need to pypy3 -m pip install cmake. With that, I am not sure this will work. Even if it does, pyarrow is a c++-based project, so the PyPy python JIT will have a hard time finding opportunities to speed up python code. Additionally, the need to emulate the CPython interactions between python and C mean the code may actually run slower under PyPy than under CPython, if you can get it to run at all.

Seems like currently pyarrow can't be use with pypy.
https://github.com/apache/arrow/issues/2089

Pip, pip3, and conflicting python distributions

I am having trouble installing the python packages PyTransit and ktransit to Anaconda 3 (on Linux). It seems like lots of others have had similar errors, but none of their solutions have worked for me. There's a bunch of stuff going on:
pip install ktransit gives:
/usr/bin/ld: /home/name/anaconda3/lib/libgfortran.a(pause.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/home/name/anaconda3/lib/libgfortran.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
I would try to use ./configure with --enable-shared, but there's no config file. I tried adding extra_compile_args = ["-fPIC"], extra_link_args=["-fPIC"] to setup.py (as in https://github.com/h5py/h5py/issues/851), but that did nothing.
pip install pytransit gives different errors:
#warning "Using deprecated NumPy API, disable it by " \
Warning: src/gimenez.f90:35: Illegal preprocessor directive
Warning: src/gimenez.f90:36: Illegal preprocessor directive
Warning: src/gimenez.f90:37: Illegal preprocessor directive
src/gimenez.f90:271.33:
real(8), dimension(CHUNK_SIZE, npol) :: d
1
Error: Symbol 'chunk_size' at (1) has no IMPLICIT type
Here's where it gets interesting. Using pip3 (or sudo pip) allows ktransit to install successfully, but does nothing for pytransit. Pip and pip3 seem to be tied to different places:
which pip
/home/name/anaconda3/bin/pip
which pip3
/usr/local/bin/pip3
which python
/home/name/anaconda3/bin/python
Lastly, When I try to import ktransit in a jupyter notebook through anaconda, it fails, but when I import ktransit in the Jupyter QtConsole, it succeeds, but seems to contain itself and not provide actual functionality. What I mean is that I can't reference anything within the module, and:
Help on package ktransit:
NAME
ktransit
PACKAGE CONTENTS
ktransit (package)
setup
FILE
(built-in)
I'm thinking that whatever version of python is associated with /usr/bin is somehow messing with me...or maybe there's something wrong with my gcc or libgfortran? I have gcc versions 4.8.5 and 6 (gcc --version gives 4.8.5, but -update-alternatives says there's only one version, 6).
Thanks for taking the time to read this far. I would be very grateful for any suggestions or ideas!

Figured it out -- I uninstalled Anaconda, made sure my python was pointing in the right direction, and then used sudo python -m pip install. Thank you to Nagev for pushing me to get rid of Anaconda!

How should I execute the Bazel Build Command in compiling Tensorflow?

I've been trying to install Tensorflow for a few weeks now and I keep getting a lot of errors with the simple installations so I think that it would be best for me to install Tensorflow from source. I'm following the instructions on the Tensorflow website exactly, and my ./configure is mostly all default so I can see if it works before I make modifications:
./configure
Please specify the location of python. [Default is /usr/bin/python]: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] n
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N] n
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA support will be enabled for TensorFlow
Found possible Python library paths:
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
Please input the desired Python library path to use. Default is [/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages]
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
Do you wish to build TensorFlow with OpenCL support? [y/N] n
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] n
No CUDA support will be enabled for TensorFlow
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Configuration finished
(This is not the first time I've edited the configuration)
After this, I execute the following bazel build command straight from the Tensorflow.org website instructions for installing from source :
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
(In the future, I'm going to add some additional flags to account for the fact that I've been getting CPU instruction errors about SSE, AVX, etc.)
When I execute that bazel command, I get an extremely long wait time and a list of errors that piles up:
r08ErCk:tensorflow kendrick$ bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
WARNING: /Users/kendrick/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': Use SavedModel Builder instead.
WARNING: /Users/kendrick/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': Use SavedModel instead.
INFO: Found 1 target...
INFO: From Compiling external/protobuf/src/google/protobuf/compiler/js/embed.cc [for host]:
external/protobuf/src/google/protobuf/compiler/js/embed.cc:37:12: warning: unused variable 'output_file' [-Wunused-const-variable]
const char output_file[] = "well_known_types_embed.cc";
^
1 warning generated.
INFO: From Compiling external/protobuf/python/google/protobuf/pyext/message_factory.cc:
external/protobuf/python/google/protobuf/pyext/message_factory.cc:78:28: warning: ISO C++11 does not allow conversion from string literal to 'char *' [-Wwritable-strings]
static char* kwlist[] = {"pool", 0};
^
external/protobuf/python/google/protobuf/pyext/message_factory.cc:222:6: warning: ISO C++11 does not allow conversion from string literal to 'char *' [-Wwritable-strings]
{"pool", (getter)GetPool, NULL, "DescriptorPool"},
^
external/protobuf/python/google/protobuf/pyext/message_factory.cc:222:37: warning: ISO C++11 does not allow conversion from string literal to 'char *' [-Wwritable-strings]
{"pool", (getter)GetPool, NULL, "DescriptorPool"},
^
3 warnings generated.
This is only a small portion of all the errors that looked similar to this that piled up. Even after all of the error messages, the command never returns and I just get the blinking cursor on an empty line.
Can someone please provide me with some exact instructions on what I should enter into terminal to avoid these errors? I've been following stack advice for weeks but continue to get errors.
MAC OS Sierra (MacBook Air)
What should I enter into terminal? (specifically)
Everything that I've done up to this point has been almost exactly what is told to do on the Tensorflow.org website instructions.

I installed for the first time using http://queirozf.com/entries/installing-cuda-tk-and-tensorflow-on-a-clean-ubuntu-16-04-install and not only was it a very simple process, but working with tf is really easy.. just source <name_of_virtual_environment>/bin/activate and then run python/python3 through that.
Bear in mind that the walkthrough in the link is for gpu tensorflow, however using the cpu tensorflow download for your mac instead, with with this virtual environment process should work just fine.

Since you do not have a GPU, do have SSE and AVX, and are on a mac sierra - the instructions found on google will NOT work with 1.3. i am befuddled on why they do not provide an exact script to do this. Regardless, here is the answer to your question http://www.josephmiguel.com/building-tensorflow-1-3-from-source-on-mac-osx-sierra-macbook-pro-i7-with-sse-and-avx/
/*
do each of these steps independently
will take around 1hr to complete all the steps regardless of machine type
*/
one time install
install anaconda3 pkg # manually download this and install the package
conda update conda
conda create -n dl python=3.6 anaconda
source activate dl
cd /
brew install bazel
pip install six numpy wheel
pip install –upgrade https://storage.googleapis.com/tensorflow/mac/cpu/protobuf-3.1.0-cp35-none-macosx_10_11_x86_64.whl
sudo -i
cd /
rm -rf tensorflow # if rerunning the script
cd /
git clone https://github.com/tensorflow/tensorflow
Step 1
cd /tensorflow
git checkout r1.3 -f
cd /
chmod -R 777 tensorflow
cd /tensorflow
./configure # accept all default settings
Step 2
// https://stackoverflow.com/questions/41293077/how-to-compile-tensorflow-with-sse4-2-and-avx-instructions
bazel build –config=opt –copt=-mavx –copt=-mavx2 –copt=-mfma //tensorflow/tools/pip_package:build_pip_package
Step 3
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install /tmp/tensorflow_pkg/tensorflow-1.0.1-cp36-cp36m-macosx_10_7_x86_64.whl
Step 4
cd ~
ipython
Step 5
import tensorflow as tf
hello = tf.constant(‘Hello, TensorFlow!’)
sess = tf.Session()
print(sess.run(hello))
Step 6
pip uninstall /tmp/tensorflow_pkg/tensorflow-1.0.1-cp36-cp36m-macosx_10_7_x86_64.whl

Python/PyQuery: Unable to find vcvarsall.bat?

I have Python 2.7 and I was trying to use PyQuery, so for a test I just typed "import PyQuery" and I got an error:
Traceback (most recent call last):
File "C:\Users\Jacob\Documents\dupes.py", line 1, in <module>
import pyquery
File "C:\Python27\lib\site-packages\pyquery-1.2.1-py2.7.egg\pyquery\__init__.py", line 12, in <module>
from .pyquery import PyQuery
File "C:\Python27\lib\site-packages\pyquery-1.2.1-py2.7.egg\pyquery\pyquery.py", line 8, in <module>
from lxml import etree
ImportError: No module named lxml
So I went to the command prompt and tried to install lxml, but I got this:
Building lxml version 2.3.5.
Building without Cython.
ERROR: 'xslt-config' is not recognized as an internal or external command,
operable program or batch file.
** make sure the development packages of libxml2 and libxslt are installed **
Using build configuration of libxslt
error: Setup script exited with error: Unable to find vcvarsall.bat
I don't really understand what's wrong or what I should do...can someone help?
Thanks.
EDIT:
In response to the comment, I used easy install...

From installation instructions of lxml:
easy_install --allow-hosts=lxml.de,*.python.org lxml
On MS Windows, the above will install the binary builds that we provide. If there is no binary build of the latest release yet, please search PyPI for the last release that has them and pass that version to easy_install like this:
easy_install --allow-hosts=lxml.de,*.python.org lxml==2.2.2
[edit]
Ok, 2.2.2 was en example. I went and looked for you, try:
easy_install --allow-hosts=lxml.de,*.python.org lxml==2.3

Unless your Windows environment is properly set (proper Visual Studio version for your Python version and all that), you should download binary eggs and install them. I know setuptools (and possibly distribute) support installing binary executable packages on Windows (the distutils-based executables only - the ones with the blue background and old dot-matrix Python logo; sorry, but I haven't done Python on Windows in a couple months). Pip doesn't (probably what you are/were using).
But to answer your question, the batch file vcvarsall.bat is used to set the environment variables necessary for building Visual Studio projects/solutions (and generally using any Visual Studio tools) from the command line. It's not in your PATH by default, and since pip is trying to use it to build lxml properly, it fails.
My advice: unless you know how to use Visual Studio command line tools, you're much better off using binary packages on Windows.