Develop Reference

Pandas development environment: pytest does not see changes after building edited .pyx file

Q: Why is pytest not seeing changes when I edit a .pyx file and build? What step am I missing?
I'm using Visuals Studio Code with remote containers as described at the end of this page.
If I add changes to pandas/_libs/tslibs/offsets.pyx, and then run
(pandas-dev) root#60017c489843:/workspaces/pandas# python setup.py build_ext -j 4
Compiling pandas/_libs/tslibs/offsets.pyx because it changed.
[1/1] Cythonizing pandas/_libs/tslibs/offsets.pyx
/opt/conda/envs/pandas-dev/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*. warnings.warn(msg, _BetaConfiguration)
# ... more output here without errors ...
my unit-test fails because it does not test against my updated version of offsets.pxy. He points to a line (see below) where the error exists only in the old version of the file.
pandas/tests/tseries/offsets/test_offsets.py ........x......................................................F
.... more output here ...
E TypeError: __init__() got an unexpected keyword argument 'milliseconds'
pandas/_libs/tslibs/offsets.pyx:325: TypeError
Whatever change I add to cdef _determine_offset and build, pytest does not see the edits, therefore I assume I'm missing a compilation step somewhere.
Reproducible example
clone my pandas fork: git clone git#github.com:markopacak/pandas.git
git checkout bug-dateoffset-milliseconds
In your dev-environment (docker container or VS Code remote container) run:
conda activate pandas-dev
python setup.py build_ext -j 4
pytest pandas/tests/tseries/offsets/test_offsets.py::TestDateOffset
Assumes you have set-up a dev environment for pandas, ideally using remote-containers on VS Code like I did.
(pandas-dev) root#60017c489843:/workspaces/pandas# python --version
Python 3.8.15

I'm pretty sure you need to install once the extensions are built (otherwise where are the built extension and how python/pytest should know where to look?). This is how my workflow looked some time ago (not sure it still applies but should be close enough):
python setup.py build_ext --inplace -j 4
python -m pip install -e . --no-build-isolation --no-use-pep517
...
pytest pandas/tests/xxxx/yyyy.py
Installing in development mode (-e) is the most convenient option in my opinion for development.

ta-lib replit python install problem, ERROR: No matching distribution found for talib-binary

I use it on my windows machine by downloading its binary. I also use it in Heroku from its herokus build pack. I don't know what operating system replit use. But I try every possible commed like.
!pip install ta-lib
!pip install talib-binary
It's not working with replit. I thought it work like google co-lab but its not the same.
can anyone use TA-LIB with replit. if so. How you install it?

Getting TA-Lib work on Replit
(by installing it from sources)
Create a new replit with Nix toolset with a Python template.
In main.py write:
import talib
print (talib.__ta_version__)
This will be our test case. If ta-lib is installed the python main.py (executed in Shell) will return something like:
$ python main.py
b'0.6.0-dev (Jan 1 1980 00:00:00)'
We need to prepare a tools for building TA-Lib sources. There is a replit.nix file in your project's root folder (in my case it was ~/BrownDutifulLinux). Every time you execute a command like cmake the Nix reports that:
cmake: command not installed. Multiple versions of this command were found in Nix.
Select one to run (or press Ctrl-C to cancel):
cmake.out
cmakeCurses.out
cmakeWithGui.out
cmakeMinimal.out
cmake_2_8.out
If you select cmake.out it will add a record about it into the replit.nix file. And next time you call cmake, it will know which cmake version to launch. Perhaps you may manually edit replit.nix file... But if you're going to add such commands in a my way, note that you must execute them in Shell in your project root folder as replit.nix file is located in it. Otherwise Nix won't remember your choice.
After all my replit.nix file (you may see its content with cat replit.nix) content was:
{ pkgs }: {
deps = [
pkgs.libtool
pkgs.automake
pkgs.autoconf
pkgs.cmake
pkgs.python38Full
];
env = {
PYTHON_LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [
# Needed for pandas / numpy
pkgs.stdenv.cc.cc.lib
pkgs.zlib
# Needed for pygame
pkgs.glib
# Needed for matplotlib
pkgs.xorg.libX11
];
PYTHONBIN = "${pkgs.python38Full}/bin/python3.8";
LANG = "en_US.UTF-8";
};
}
Which means I executed libtool, autoconf, automake and cmake in Shell. I always choose a generic suggestion from Nix, without a specific version. Note: some commands may report errors as we executing them in a wrong way just to add to a replit.nix.
3.
Once build tools are set up we need to get and build TA-Lib C library sources. To do that execute in Shell:
git clone https://github.com/TA-Lib/ta-lib.git
then
cd ta-lib/
libtoolize
autoreconf --install
./configure
If configure script is completed without any problems, build the library with:
make -j4
It will end up with some compilation errors, but they are related to some additional tools which are used to add new TA-Lib indicators and build at the end, but not the library itself. The library will be successfully compiled and you should be able to see it with:
$ ls ./src/.libs/
libta_lib.a libta_lib.lai libta_lib.so.0
libta_lib.la libta_lib.so libta_lib.so.0.0.0
Now we have our C library built, but we can't install it to a system default folders. So we have to use the library as is from the folders where it was build. All we need is just one more additional preparation:
mkdir ./include/ta-lib
cp ./include/*.h ./include/ta-lib/
This will copy a library headers to a subfolder, as they are designed to be used from a such subfolder (which they don't have due to impossibility to perform the installation step).
4.
Now we have TA-Lib C library built and prepared to be used locally from its build folders. All we need after that - is to compile the Python wrapper for it. But Python wrapper will look for a library only in system default folders, so we need to instruct it where our library is.
To do this, execute pwd and remember the absolute path to your project's root folder. In my case it was:
/home/runner/FormalPleasedOffice
Then adjust the paths (there are two) in a following command to lead to your project path:
TA_INCLUDE_PATH=/home/runner/FormalPleasedOffice/ta-lib/include/ TA_LIBRARY_PATH=/home/runner/FormalPleasedOffice/ta-lib/src/.libs/ pip install ta-lib
This is one line command, not a two commands.If the paths would be shorter it would look like:
TA_INCLUDE_PATH=/path1/ TA_LIBRARY_PATH=/path2/ pip install ta-lib.
After execution of this command the wrapper will be installed with two additional paths where it will look for a library and its header files.
That's actually all.
An alternative way would be to clone the wrapper sources, edit its setup.py and install wrapper manually. Just for the record this would be:
cd ~/Your_project
git clone https://github.com/mrjbq7/ta-lib.git ta-lib-wrapper
cd ta-lib-wrapper
Here edit the setup.py. Find the lines include_dirs = [ and library_dirs = [ and append your paths to these lists. Then you just need to:
python setup.py build
pip install .
Note the dot at the end.
5.
Go to the project's folder and try our python script:
$python main.py
b'0.6.0-dev (Jan 1 1980 00:00:00)'
Bingo!

The #truf answer is correct.
after you add the
pkgs.libtool
pkgs.automake
pkgs.autoconf
pkgs.cmake
in the replit.nix dippendancies.
git clone https://github.com/TA-Lib/ta-lib.git
cd ta-lib/
libtoolize
autoreconf --install
./configure
make -j4
mkdir ./include/ta-lib
cp ./include/*.h ./include/ta-lib/
TA_INCLUDE_PATH=/home/runner/FormalPleasedOffice/ta-lib/include/ TA_LIBRARY_PATH=/home/runner/FormalPleasedOffice/ta-lib/src/.libs/ pip install ta-lib
Note : FormalPleasedOffice should be your project name
Done.
Here is the youtube video :
https://www.youtube.com/watch?v=u20y-nUMo5I

cd command in python

I am new to python and
I am trying to test syntaxnet in python using this github repo.
and in the section "how to run the parser" is as following :
git clone https://github.com/spoddutur/syntaxnet.git
cd <syntaxnet-git-clone-directory>
python main.py
That's it!! It prints syntaxnet dependency parser output for given input english sentence
through some research, I understood that first one indicates that I need to install the syntaxnet package in my cmd so I did and the package was successfully installed.
but I don't understand how to perform the second one
what does cd do, and where and how should I use it?
also in main.py,
from my_parser_eval import SyntaxNetProcess
from my_parser_eval import _write_input
from my_parser_eval import _read_output
from my_parser_eval import pretty_print
eclipse cannot find those imports, even after I created a package/module named my_parser_eval that includes all necessary codes from the reference

The CD command means Change Directory.
Once you have finished cloning Syntaxnet Github Repository, you should enter its directory. That's why you have to enter CD command.
But have in mind that CD takes one parameter - the directory you want to enter.
In order to solve your problem you must write cd syntaxnet resulting in:
git clone https://github.com/spoddutur/syntaxnet.git
cd syntaxnet
python main.py
First command:
Second command

How does pylint use easy_install at runtime?

Sorry this is a long question. See the sentence in bold at the bottom for the TL;DR version.
I've spent many hours trying to track down a problem where pylint sometimes doesn't report all the errors in a module. Note that it does find some errors (e.g. long lines), just not all of them (e.g. missing docstrings).
I'm running pylint 1.7.2 on Ubuntu 16.04. (The version available from apt was 1.5.2 but installing via pip gives 1.7.2.)
We typically run pylint from tox, with a tox.ini that looks something like this (this is a cut-down version):
[tox]
envlist = py35
[testenv]
setenv =
MODULE_NAME=our_module
ignore_errors = True
deps =
-r../requirements.txt
whitelist_externals = bash
commands =
pip install --editable=file:///{toxinidir}/../our_other_module
pip install -e .
bash -c \'set -o pipefail; pylint --rcfile=../linting/pylint.cfg our_module | tee pylint.log\'
Amongst other things, the ../requirements.txt file contains a line for pylint==1.7.2.
The behaviour is like this:
[wrong] When the line that imports our_other_module is present, pylint appears to complete successfully and not report any warnings, even though there are errors in the our_module code that it should pick up.
[correct] When that line is commented out, pylint generates the expected warnings.
As part of tracking this down I took two copies of the .tox folder with and without the module import, naming them .tox-no-errors-reported and .tox-with-errors-reported respectively.
So now, even without sourcing their respective tox virtualenvs, I can do the following:
$ .tox-no-errors-reported/py35/bin/pylint --rcfile=../linting/pylint.cfg our_module -- reports no linting warnings
$ .tox-with-errors-reported/py35/bin/pylint --rcfile=../linting/pylint.cfg our_module -- reports the expected linting warnings
(where I just changed the pylint script's #! line in each case to reference the python3.5 inside that specific .tox directory instead of the unrenamed .tox)
By diffing .tox-no-errors-reported and .tox-with-errors-reported, I've found that they are very similar. But I can make the "no errors" version start to report errors by removing the path to our_other_module from .tox-no-errors-reported/py35/lib/python3.5/site-packages/easy-install.pth.
So my question is why is pylint using easy_install at runtime, and what is it picking up from our other component that is causing it to fail to report some errors.
As I understand it, pylint has dependencies on astroid and logilab-common, but including these in the requirements.txt doesn't make any difference.

One possible reason for the surprising pylint behavior is the --editable option.
it creates a special .egg-link file in the deployment directory, that links to your project’s source code. And, ..., it will also update the easy-install.pth file to include your project’s source code
The pth file will then affect the sys.path which affects the module import logic of astroid and it is deeply buried in the call stack of pylint.expand_files via pylint.utils.expand_modules. Also pylint identifies the module part and function names in the AST using astroid.modutils.get_module_part.
To test the theory, you can try calling some of the affected astroid functions manually:
import sys, astroid
print(sys.path)
print(astroid.modutils.get_module_part('your_package.sub_package.module'))
astroid.modutils.file_from_modpath(['your_package', 'sub_package', 'module'])

Why does nosetests say --with-coverage is not an option?

I am using nosetests to automatically discover and run my unittests. I would also like to have it generate coverage reports.
When I run nosetests with the following command everything works just fine
nosetests .
I looked up online that to generate the coverage, nosetests has a command line argument --with-coverage. I also double checked that this command exists using nosetests --help. However, whenever I run the following command I get the following output
nosetests --with-coverage .
Usage: nosetests [options]
nosetests: error: no such option: --with-coverage
I double checked that the coverage plugin is installed by running
nosetests --plugins
coverage shows up in the list along with a bunch of other plugins.
I also know I have coverage installed because I can manually run the coverage data collection using something along the lines of:
coverage run test.py
Am I misusing the --with-coverage option? Or is there something else I am missing?
Thanks in advance.

I never got the command line options working. I did what Janne Karila suggested and created a setup.cfg file in my projects main directory. Once I had that file I could just run nosetests with no arguments and everything would run.
One problem I had when trying to create my document was that I couldn't figure out what parameters were allowed in the config. But it turns out that any of the commands listed here https://nose.readthedocs.org/en/latest/usage.html#options can be used. Just leave off the double dashes before the command.
For reference my current config file is
[nosetests]
verbosity=1
detailed-errors=1
with-coverage=1
cover-erase=1
cover-package=application
cover-html=1
cover-html-dir=htmlcov
where=tests
This config file says to use coverage, to erase the previous run's coverage, to only report on the files in the application package, and to output an html report to the htmlcov directory.
Hopefully this will help someone else in the future.

Your syntax is correct. It maybe an issue with your environment, double check your python environment and where your have nose and coverage installed. As a sanity check, you can quickly setup a new virtualenv, install nose, and run the command with the coverage option.

As of nose 1.3.7 - the most recent version available on Pypy - that command doesn't exist:
https://github.com/nose-devs/nose/blob/release_1.3.7/nose/plugins/cover.py
It looks like the documentation is being generated from the master branch of the project that does have those options available:
https://github.com/nose-devs/nose/blob/master/nose/plugins/cover.py
What you can do is install nose from the master branch like this:
pip install git+https://github.com/nose-devs/nose#master --upgrade
It will say it just installed 1.3.7 but that's only because the version hasn't been bumped in the project setup.py file yet: https://github.com/nose-devs/nose/blob/master/setup.py#L4
Remember you have just installed an unreleased version of nose, there may be other bugs.