In the gradle/JVM world there is the awesome plugin: https://github.com/diffplug/spotless-changelog
Is there any similar public solution available I am currently missing when building pip or conda packages?
This: Automating Python package release process refers to using poetry, but is not nearly as fully-fledged as spotless-changelog and seems to require some manual setup/scripts.
I know that there is a python package for validating the keepachangelog format https://pypi.org/project/keepachangelog/ available - but do not see any integration like spotless-changelog.
Are manual steps required to chain it up? Or am I overlooking a great & already existing tool for such an automated process similar to spotless-changelog?
https://zestreleaser.readthedocs.io/en/latest/ so far looks like the closest match
I'm integrating a java project by using maven. This project should be pushed to a server(linux) with limited access(cannot use pip). I've added all the dependencies for my java component adding them to the pom.xml, but In part of my code I've used client/server approach to call a python script, which requires Pandas and Sklearn. Unfortunately, I've memory issue and cannot copy the entire directory of these libraries into server.
I'm afraid if maven could help me to download the python dependencies or if there is another efficient way of adding python dependencies into repository. I've done some research but couldn't find any helpful way to address that. I'm a beginner in python and I'd be happy if you could help me address that.
If limited access (Assuming you can download but cannot install) is your issue you can download Anaconda and it doesn't need to be installed with root access. It install everything to your home directory and creates a virtual environment for you. That way you can use pip too. Just make sure you call your python script with your anaconda virtual environment python.
/home/USER/anaconda2/envs/ml/bin/python script_name.py
having problems seeing full intellisense (code completion) options in PyCharm.
working with python 3.4 on Windows.
the suggests are partially working:
import boto3
s = boto3.Session() (boto3. will bring up list of methods/params of object boto3)
ec2 = s.resource('ec2') (resource is a suggested method!)
ec2. <<<< this brings up nothing. For some reason PyCharm cant detect that ec2 object would have
while I can work off documentation alone, intellisense is just such a nice feature to have!
ive had similar problems getting it to complete lxml syntax but I thought that was because I had to install lxml directly as a binary (too many hoops to jump through on windows to install it via pip)
Anyone else encounter similar problems?
While we are here,
I see a lot of different libraries around using awscli with python: boto, boto3, troposphere etc. What are some advantages of using one over the other. Amazon states that boto3 is the prefered method over boto but for my usage of starting/stopping ec2 instances could be easily done with older boto.
I was frustrated with the same issue. So I decided to parse boto3 documentation and generate wrapper classes from the documentation. Here is the link to project
https://github.com/gehadshaat/pyboto3
To install it
pip install pyboto3
To use it
import boto3
s3 = boto3.client('s3')
""" :type : pyboto3.s3 """
# s3. -> will give you autocomplete for s3 methods in pycharm
Make sure that you first:
Install pyboto3 -> pip install pyboto3 | pip3.x install pyboto3
Check your interpreter settings and verify that you see pyboto3 on the list
Do a File -> Invalidate Caches/Restart
After Pycharm restarts you should see intellisense working in your favor and all of the available methods for the service (in the case above s3) you are trying to use available to you!
This is happening because all of the methods on the boto3 clients and resource objects are generated at runtime based on a JSON file that describes what operations the service supports. Pycharm would have to have specific knowledge about this process in order to auto complete method names.
For your second question, boto3 is the official AWS SDK for Python. One of the main advantages of boto3 is that because of this JSON model driven process that describes the AWS APIs, most new service features only require a simple model update. This means API updates happen in a quick, consistent, and reliable manner.
But if you're using boto in existing code and it's working for you, feel free to keep using it. You can always install boto3 along side boto if you need to pull in new functionality.
The room's getting a little crowded here, but I also have created a boto3 typing solution (GitHub link), boto3_type_annotations. I took the pyboto3 approach and parsed the docstrings of service objects and then programmatically wrote class definitions for them and annotated arguments and return types with the typing module. Unlike pyboto3 I created everything including service resources, paginators, and waiters. There's also an option where I left the docstrings in, so PyCharm's quick documentation will work. But fair warning, that package is really big.
# Without docs
pip install boto3_type_annotations
# Or with docs
pip install boto3_type_annotations_with_docs
Probably not an official method, but I did find something that works.
In PyCharm, open python console (tools/python console). The console will also have variable list on the right side. If you initialize a resource object on the console, it will have its sub objects listed in variable object tree. Some limited intellisense as well.
The way I started doing it, is writing code right into the interpreter using variable watch window as a cheat sheet. Once code is written, I copy/paste it into the actual script file. Clunky...
I love boto3, but I was also frustrated that every time I want to make a simple ad-hoc request I have to open boto3 documentation. So I wrote autoboto:
https://pypi.org/project/autoboto/
It doesn't just auto-complete. It also returns dataclasses which means that you don't have to look up the names of attributes of the returned objects. PyCharm will tell you what is available.
At the moment, it's also probably very slow because of all the generic serialisation and deserialisation.
While autocomplete solutions for boto3 are being discussed I'm surprised that nobody mentioned botostubs yet. Works on any IDE and is automatically kept up-to-date.
This works for me. If you are using python3
python3 -m pip install boto3-stubs
python3 -m pip install 'boto3-stubs[essential]'
Make sure that you:
Install pyboto3 -> pip install pyboto3 | pip3.x install pyboto3
Check your interpreter settings and verify that you see pyboto3 on the list
Do a File -> Invalidate Caches/Restart
After Pycharm restarts you should see intellisense working in your favor and all of the available methods for the service you are trying to use available to you!
boto3-stubs
Type annotations for boto3 1.16.62 compatible with VSCode, PyCharm, Emacs, Sublime Text, mypy, pyright and other tools.
Is there a way to obtain statistics on how many times a package (every package) has been installed using pip-install or any other package manager?
This question came about when trying to determine the popularity of some python packages I was looking into using for a personal project - I am sure developers would find it useful to be able to obtain such statistics.
You can view the download statistics from the official PyPI data. This data is stored in BigQuery (see https://bigquery.cloud.google.com/dataset/the-psf:pypi)
But I developed a page to facilitate this information, take a look at http://pepy.tech/ :-)
You can get this statistics via vanity:
https://pypi.python.org/pypi/vanity
I am trying to define a process for migrating django projects from my development server to my production server using git, and it's driving me crazy that distutils installs python modules system-wide. I've read the documentation but unless I'm missing something it seems to be mostly about how to change the installation directory. I need to be able to use different versions of the same module in different projects running on the same server, and deploy projects from git without having to download and install dependencies.
tl;dr: I need to know how to install python modules, using distutils, into my project's source tree for version control without compromising other projects using different versions of the same module.
I'm new to python, so I apologize in advance if this is common knowledge.
Besides the already mentioned virtualenv which is a good option but has the potential drawback of requiring a third-party module, Distutils itself has options to install modules into arbitrary locations. In particular, there is the home scheme which allows you to "build and maintain a personal stash of Python modules". It's described in the Python documentation set here.
Perhaps you are looking for virtualenv. It will allow you to install packages into a separate virtual Python "root".
for completeness sake, virtualenvwrapper makes every day work with virtualenv a lot quicker and simpler once you are working on multiple projects and/or on multiple development platforms at the same time.
If you are looking something akin to npm or yarn of the JavaScript world or composer of the PHP world, then you may want to look at pipenv (not to be confused with pip). Here's a guide to get you started.
Alternatively there is also Poetry, which some people say is even better, but I haven't used it yet.