Develop Reference

Problems with pip und Command line

I am trying to create a Python pip package. This works also well. I can successfully upload and download the package and use it in the Python code. What I can't do is to use the Python package via the command line. In another StackOverflow post I found the link to a tutorial. I tried to follow it. Obviously I made a mistake. Can you guys help me please ?
Installation of the package via pip
here you can see that the installation worked. Unfortunately, not the whole script fit on the image.
Pip does not find the package.
Unfortunately, I can't embed the images directly, so I'll just embed them as links.
I have created a simple Python package. It represents here only an example. Here you can see the structure of the folder
Riffecs
| .gitignore
| .pylintrc
| LICENSE
| README.md
| requirements.txt
| setup.py
|
|
\---riffecs
__init__.py
__main__.py
Here are the basic files shown.
main.py
from . import hello_world
if __name__ == '__main__':
hello_world()
and init.py
def hello_world():
print("Hello world")
In the following you can see the "setup.py". I am of the opinion that I have followed the instructions. But obviously I made a mistake somewhere. Can you please help me to correct this mistake.
import io
import os
import setuptools
def read_description():
url = "README.md"
""" Read and Return the description """
return io.open(os.path.join(os.path.dirname(__file__), url), encoding="utf-8").read()
def def_requirements():
""" Check PIP Requirements """
with open('requirements.txt', encoding='utf-8') as file_content:
pip_lines = file_content.read().splitlines()
return pip_lines
setuptools.setup(
name="riffecs",
version='0.0.3',
description='test',
entry_points={'console_scripts': ['hello-world=riffecs:hello_world',]},
long_description=read_description(),
long_description_content_type="text/markdown",
license="MIT",
keywords="test - riffecs",
url="https://github.com/Riffecs/riffecs",
packages=["riffecs"],
install_requires=def_requirements(),
python_requires=">=3.6",
classifiers=[
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
],
)

In your setup.py file you have this line...
entry_points={'console_scripts': ['hello-world=riffecs:hello_world',]},
This is the entry point to calling you package via command line. This configuration is asking the entry point to be hello-world, which I tried and it runs fine.
In your image however you run riffecx which is not configured as an entrypoint to the package.
If you wanted the entrypoint to be riffecx. change the line to:
entry_points={'console_scripts': ['riffecx=riffecs:hello_world']},
Hope this helped.

ModuleNotFoundError after making a setup.py and running "pip install -e ."

I am trying to import the classes SqlQuery, SqlPost, and SqlPage to the file Query.py from the file "database.py" with the path "C:\Users\myName\dir1\2020\Indeed-Scraper\database\database.py". My first attempt at this was to use relative imports: from ...database.database import SqlQuery which I presumed would work since the "database" folder is up two levels from Query.py. Not so. That returned ValueError: attempted relative import beyond top-level package.
I posted about this on StackOverflow and a helpful user suggested this fix: "make a setup.py for your project. Then run pip install -e".
Well I tried that. I now have a setup.py folder in the root directory of my project that looks like this:
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="indeedscraper",
version="0.1.1",
author="myName",
author_email="myName#provider.com",
description="A tool used to scrape & display data from Indeed",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/myUsername/myProject",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.6',
)
And I ran pip install -e . in the root directory of my project. Now PyCharm autocompletes "from in" to "from indeedscraper", so something changed. But from indeedscraper.database.database import SqlQuery still gives me:
Traceback (most recent call last):
File "C:/Users/myName/dir1/2020/Indeed-Scraper/scraper/classes/Query.py", line 17, in <module>
from indeedscraper.database.database import SqlQuery
ModuleNotFoundError: No module named 'indeedscraper'
Any suggestions on how to proceed? I'm kind of stuck here learning to make a package and import from that package, because the alternative is to copy ~50 lines of code from my "database.py" file, which is ugly.
Edit: I do have __init__.py files in my top level dir, "database.py", and the directory housing the file I am trying to import into. Like this:
project_name/
__init__.py
database/
__init__.py
database.py
scraper/
__init__.py # I do have this file here, sorry, didn't realize it was important
classes/
__init__.py
Query.py
Edit2: I realize the problem may be stemming from the fact that I am using Anaconda. I import in Query.py right, but the first 2 lines of my output in PyCharm read: "C:\Users\myname\Anaconda3\envs\august_2019_django_project\python.exe C:/Users/myname/dir1/2020/Indeed-Scraper/scraper/classes/Query.py"
Could this be it?

PyPI - package contains no files

I have followed this tutorial to upload a package to PyPI. My package is now uploaded to test instance of the index: https://test.pypi.org/project/ilya-ezplots/#files. However when I install it with pip, I still can't import it.
>>> import ilya_ezplots
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ilya_ezplots'
I have inspected the site-packages of my python interpreter, and found only ilya_ezplots-0.0.2.dist-info directory. For other packages there would be two directories, e.g. h5py-2.9.0.dist-info and h5py.
My setup.py:
import setuptools
setuptools.setup(
name="ilya_ezplots",
version="0.0.2",
packages=['plots'],
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
)
Upload script:
#!/usr/bin/env bash
rm -r dist
python setup.py sdist bdist_wheel
python -m twine upload -r testpypi dist/* -u ikamensh
File structure:
./
./plots/__init__.py
./setup.py
./upload.sh

Let's me point you to my previous answer: https://stackoverflow.com/a/54599368/7976758
ilya_ezplots is the name of your distribution but it's not importable name. The importable name after installation is plots. So:
import plots

As #phd answered, you should import: import plots.
or you should change you directory structure to:
./ilya_ezplots/__init__.py
And then import ilya_ezplots can do the trick.
The name of the import should match the parent directory.
Refer this post to maintain the correct structure and for good practices.
The good practice is maintain some namespace or same directory name as your package name, so that the user won't get confused.
if your project name is xyz then you should maintain your directory name as xyz. So, the use can do import xyz to import your package and won't get confused with the other package. For eg. if you use yyy as your directory name (and your package name is xyz) then users might get confused and install package yyy instead.
One real life example:
package beautifulsoup needs import bs4 to import it and hence they maintain bs4 package (dummy package) so that no one can create the package with the name bs4 as that might become a security risk.

PyBuilder can't find src module when running from virtualenv?

Here's my build.py:
import os
import shutil
import subprocess
from pybuilder.core import use_plugin, init, task, Author
use_plugin('python.core')
use_plugin('python.unittest')
use_plugin('python.coverage')
use_plugin('python.distutils')
use_plugin('python.install_dependencies')
use_plugin('pypi:elasticsearch')
authors = [Author('eagle', 'email')]
version = '1.0'
description = 'Pipeline thing test'
requires_python = '>=3.0'
default_task = ['install_dependencies', 'publish', 'setup', 'analyze']
#init
def initialize(project):
project.build_depends_on('elasticsearch')
project.set_property('dir_source_unittest_python','tests')
project.set_property('dir_source_main_python','src')
project.set_property('unittest_module_glob','test_*')
project.set_property('coverage_branch_partial_threshold_warn ',80)
project.set_property('coverage_threshold_warn',80)
project.set_property('coverage_break_build',False)
project.set_property('distutils_use_setuptools',True)
project.set_property('distutils_classifiers',['Development Status :: 3 - Alpha', 'Intended Audience :: Developers', 'Topic :: Software Development :: Pipeline', 'License :: OSI Approved :: MIT License', 'Programming Language :: Python :: 3.6'])
#task
def setup():
subprocess.check_output(['python3.6','target/dist/pythonutilities-1.0/setup.py', 'install'])
It's specifically looking for the test directory under projectDir/tests
as specified in project.set_property('dir_source_unittest_python','tests')
And it does look there and find the correct test file:
File "/Users/adc1blz/Desktop/Work/pythonutilities/tests/test_elastic_controller.py", line 4, in <module>
But I still receive the following error:
ImportError: Failed to import test module: test_elastic_controller
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
module = __import__(module_name)
File "/Users/adc1blz/Desktop/Work/pythonutilities/tests/test_elastic_controller.py", line 4, in <module>
from src.elastic_controller import ElasticController
ModuleNotFoundError: No module named 'src'
I've tried using the default project structure by deleting
project.set_property('dir_source_unittest_python','tests')
project.set_property('dir_source_main_python','src')
and then moving around my test and src directory, but still I get the same error. The specific commands that I'm running that cause this are:
> python3.6 -m virtualenv env
> source env/bin/activate
> pip3.6 install pybuilder
> pyb -X
Why can't PyBuilder find the src module while in a virtual environment? Please, this is has been so frustrating.
Chris

PIP-installed global script throws exception, local copy works flawlessly

I've just registered my new package with PIP
python setup.py register
python setup.py sdist upload
And I wanted to install it on other machine globally (i.e. no virtualenv) using 'PIP':
sudo pip install standardiser
This all went fine, since I have one file registered as a script:
setup(
...
scripts=['standardiser/bin/standardiser.py'],
)
'standariser.py' is now available as CLI command system wide. But if I execute it, I'm getting:
mnowotka#candela:~/Documents/ci/curation_interface/trunk/src$ standardiser.py
Traceback (most recent call last):
File "/usr/local/bin/standardiser.py", line 32, in <module>
from standardiser import standardise, SDF
File "/usr/local/bin/standardiser.py", line 32, in <module>
from standardiser import standardise, SDF
ImportError: cannot import name standardise
I get the same when I explicitly call python:
python /usr/local/bin/standardiser.py
But if I copy this to some local folder:
sudo cp python /usr/local/bin/standardiser.py bla.py
And run it from there:
mnowotka#candela:~$ python bla.py
usage: bla.py [-h] [-V] [-r] infile
bla.py: error: too few arguments
I don't have any ImportErors. What I'm doing wrong? Can you help me?
My setyp.py:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
__author__ = 'mnowotka'
import sys
try:
from setuptools import setup
except ImportError:
from ez_setup import use_setuptools
use_setuptools()
from setuptools import setup
setup(
name='standardiser',
version='0.1.4',
author='Francis Atkinson',
author_email='francis#ebi.ac.uk',
description='Provides a simple way of standardising molecules as a prelude to e.g. molecular modelling exercises.',
url='https://www.ebi.ac.uk/chembldb/index.php/ws',
license='Apache License, Version 2.0',
scripts=['standardiser/bin/standardiser.py'],
packages=['standardiser'],
long_description=open('ReadMe.txt').read(),
package_data={
'standardiser': ['bin/*', 'data/*', 'docs/*', 'knime/*', 'test/*',],
},
classifiers=['Development Status :: 2 - Pre-Alpha',
'Environment :: Console',
'Intended Audience :: Science/Research',
'License :: OSI Approved :: Apache Software License',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Topic :: Scientific/Engineering :: Chemistry'],
zip_safe=False,
)

You are asking for a script to be installed named standardiser.py; that script (at least, the one you've uploaded to the cheese shop) contains the line:
from standardiser import standardise, SDF
But thats an ambigious import; the script you're executing, /usr/local/bin/standardise.py appears in sys.path, because the main script is located there. it's importing itself!
you should be using the console_scripts feature of setuptools anyway.
alter your script file from
#! /guess/path/to python
from standardise import import *
do_things()
do_more_things()
to
from __future__ import absolute_import
from standardise import import *
def main():
do_things()
do_more_things()
if __name__ == '__main__':
main()
Which is to say;
get rid of the shebang; you never need it in python!
use the absolute_import feature to get the module named foo.bar.foo to be able to import foo instead of only foo.bar.foo (you can still import foo.bar.foo as either from foo.bar import foo or import .foo). __future__ imports must appear first in the source file, before any other non-comment lines (including other imports)
most importantly; wrap import time side effects in a function, and only invoke those side effects if this is the "main script"
then change your setup.py around from
setup(
scripts=['standardiser/bin/standardiser.py'],
...)
to
setup(
entry_points={
'console_scripts': [
'standardiser=standardiser.bin.standardiser:main']},
...)
which is to say:
use an entry point; setuptools knows how to get your installed package on sys.path correctly in this case, and it knows how to connect to the proper python interpreter, the one that was used to run setup.py. This matters when there are multiple versions of python, or when running in a virtualenv. this is why you never need a shebang.
remove the '.py' from the installed executable name. this shouldn't ever be there for scripts that are to be on your executable path.