Develop Reference

Jupyter packages

I am trying to import certain packages as I am working with Jupyter notebook files, and most of the packages seem to be missing, even though I have installed them. For example, when I do the command: from bs4 import BeautifulSoup or import requests
I get the error saying ModuleNotFoundError: No module named 'bs4' for the first one and a similar one for importing requests as well. I have tried pip install requests and pip install bs4, but same issue persists. I have installed them on:
"(base) aminnazemzadeh#amins-MacBook-Pro ~ % " which seems to be my home directory, and I also have anaconda3 installed alongside python3. What is the issue that I cannot import these modules.
I am using visual studio if it makes any difference
Once I add :
!pip install requests
!pip install bs4
I get:
/Users/aminnazemzadeh/.zshenv:.:1: no such file or directory: /Users/aminnazemzadeh/.cargo/env
Requirement already satisfied: requests in /Users/aminnazemzadeh/opt/anaconda3/lib/python3.9/site-packages (2.28.1)
Requirement already satisfied: charset-normalizer<3,>=2 in /Users/aminnazemzadeh/opt/anaconda3/lib/python3.9/site-packages (from requests) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/aminnazemzadeh/opt/anaconda3/lib/python3.9/site-packages (from requests) (1.26.11)
Requirement already satisfied: idna<4,>=2.5 in /Users/aminnazemzadeh/opt/anaconda3/lib/python3.9/site-packages (from requests) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in /Users/aminnazemzadeh/opt/anaconda3/lib/python3.9/site-packages (from requests) (2022.9.24)
/Users/aminnazemzadeh/.zshenv:.:1: no such file or directory: /Users/aminnazemzadeh/.cargo/env
Requirement already satisfied: bs4 in /Users/aminnazemzadeh/opt/anaconda3/lib/python3.9/site-packages (0.0.1)
Requirement already satisfied: beautifulsoup4 in /Users/aminnazemzadeh/opt/anaconda3/lib/python3.9/site-packages (from bs4) (4.11.1)
Requirement already satisfied: soupsieve>1.2 in /Users/aminnazemzadeh/opt/anaconda3/lib/python3.9/site-packages (from beautifulsoup4->bs4) (2.3.1)
followed by this warning:
ModuleNotFoundError Traceback (most recent call last)
Cell In[7], line 4
2 get_ipython().system('pip install bs4')
3 from urllib.request import urlopen
----> 4 from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'
Thanks

probably you're installing the packages on an environment other than the one vs code is using. you can try installing the packages directly from your jupyter notebook by running the following code in a notebook cell. the current best practice to be running installs in the notebook is using the magic commands %pip or %conda:
%pip install requests beautifulsoup4
# or
%conda install requests beautifulsoup4
this should install the packages in the same environment that the notebook is running on.
note that you may need to restart the kernel to use the affected packages.
sources:
Jupyter Discourse Forum - Location of libraries or extensions installed in JupyterLab
Jupyter Discourse Forum - Why users can install modules from pip but not from conda?
Installing Beautiful Soup
ps: thanks #wayne for the comments regarding the current best practices for installing on the current running environment.

If you're using conda, you should install via conda whenever possible. When you install via pip, conda loses some of its ability to manage dependency versions.
Try creating a new conda environment, install the needed packages via conda, then set the kernel to your new environment in vscode. Dedicate conda environments to specific projects. It is okay to have a default/generic environment for playing around but not for any significant work as you can easily create errors in your other work if a dependency changes to an incompatible version.
Conda cheet sheet for reference if you need it: https://docs.conda.io/projects/conda/en/4.6.0/_downloads/52a95608c49671267e40c689e0bc00ca/conda-cheatsheet.pdf
You will need the Jupyter extension in vscode if you do not already have it installed.
You will also need to install the corresponding jupyter package in your conda environment.

pandas version is not updated after installing a new version on databricks

I am trying to solve a problem of pandas when I run python3.7 code on databricks.
The error is:
ImportError: cannot import name 'roperator' from 'pandas.core.ops' (/databricks/python/lib/python3.7/site-packages/pandas/core/ops.py)
the pandas version:
pd.__version__
0.24.2
I run
from pandas.core.ops import roperator
well on my laptop with
pandas 0.25.1
So, I tried to upgrade pandas on databricks.
%sh pip uninstall -y pandas
Successfully uninstalled pandas-1.1.2
%sh pip install pandas==0.25.1
Collecting pandas==0.25.1
Downloading pandas-0.25.1-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Requirement already satisfied: python-dateutil>=2.6.1 in /databricks/conda/envs/databricks-ml/lib/python3.7/site-packages (from pandas==0.25.1) (2.8.0)
Requirement already satisfied: numpy>=1.13.3 in /databricks/conda/envs/databricks-ml/lib/python3.7/site-packages (from pandas==0.25.1) (1.16.2)
Requirement already satisfied: pytz>=2017.2 in /databricks/conda/envs/databricks-ml/lib/python3.7/site-packages (from pandas==0.25.1) (2018.9)
Requirement already satisfied: six>=1.5 in /databricks/conda/envs/databricks-ml/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas==0.25.1) (1.12.0)
Installing collected packages: pandas
ERROR: After October 2020 you may experience errors when installing or updating packages.
This is because pip will change the way that it resolves dependency conflicts.
We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
mlflow 1.8.0 requires alembic, which is not installed.
mlflow 1.8.0 requires prometheus-flask-exporter, which is not installed.
mlflow 1.8.0 requires sqlalchemy<=1.3.13, which is not installed.
sklearn-pandas 2.0.1 requires numpy>=1.18.1, but you'll have numpy 1.16.2 which is incompatible.
sklearn-pandas 2.0.1 requires pandas>=1.0.5, but you'll have pandas 0.25.1 which is incompatible.
sklearn-pandas 2.0.1 requires scikit-learn>=0.23.0, but you'll have scikit-learn 0.20.3 which is incompatible.
sklearn-pandas 2.0.1 requires scipy>=1.4.1, but you'll have scipy 1.2.1 which is incompatible.
Successfully installed pandas-0.25.1
When I run:
import pandas as pd
pd.__version__
it is still:
0.24.2
Did I missed something ?
thanks

It's really recommended to install libraries via cluster initialization script. The %sh command is executed only on the driver node, but not on the executor nodes. And it also doesn't affect Python instance that is already running.
The correct solution will be to use dbutils.library commands, like this:
dbutils.library.installPyPI("pandas", "1.0.1")
dbutils.library.restartPython()
this will install library to all places, but it will require restarting of the Python to pickup new libraries.
Also, although it's possible to specify only package name, it's recommended to specify version explicitly, as some of the library version may not be compatible with runtime. Also, consider usage of the newer runtimes where library versions are already updated - check the release notes for runtimes to figure out the library versions installed out of the box.
For newer Databricks runtimes you can use new magic commands: %pip and %conda to install dependencies. See the documentation for more details.

keep getting python ModuleNotFoundError: No module named

I am working on a raspberry pi python project and every time I import a package I get the same error ModuleNotFoundError: No module named ''. for example I am trying to add the ambient_api package
I followed the steps in their set up:
pip install ambient_api
Added the imports at the top of my file:
from ambient_api.ambientapi import AmbientAPI
api = AmbientAPI()
but I am getting the error:
ModuleNotFoundError: No module named 'ambient_api'
This is happening with all the imports I am trying to add and I cant figure out what I am missing.
when I was looking on google I came across __init__.py might be a solution but I am not sure how this works?
edit:
here is what was output when I installed:
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: ambient_api in /home/pi/.local/lib/python2.7/site-packages (1.5.2)
Requirement already satisfied: requests in /usr/lib/python2.7/dist-packages (from ambient_api) (2.21.0)
Requirement already satisfied: urllib3 in /usr/lib/python2.7/dist-packages (from ambient_api) (1.24.1)
version in my terminal:
pi#raspberrypi:~/Raspberry-Pi-Greenhouse $ python --version
Python 2.7.16
but it looks like the version in the shell in the ide when I run the program is saying:
Python 3.7.3 (/usr/bin/python3)

As explained by #JaFizz in Pytorch is installed but do not working ubuntu 18.04 it solved by giving an alias and installing correctly to desired python version (there were two pythons installed on the same machine)
First it was necessary to specify the location of the python, example:
alias python=/usr/local/bin/python3.6
Then install it:
python pip install <package name>

Am I importing a package from conda wrong in my python script?

Heyo! I know this may look like a duplicate question of this question, but there is a little difference. I want to import the whole package, not just one or a groups of functions. When I import packages from Anaconda, and try to import them into a python script, I get ErrorModuleNotFound. I might be importing this wrong, so here is the information:
I am using Python 3.6 and conda 4.3.21.
Anaconda packages can be found in this directory: C:\ProgramData\Miniconda3
What I put into the Anaconda Prompt: pip install wikipedia (It installed successfully...)
Directory of my Python Script: C:\Users\aglin\Desktop\test.py
Conda env list: root * C:\ProgramData\Miniconda3
Path of Python.exe C:\Users\aglin\AppData\Local\Programs\Python\Python36-32\python.exe and C:\ProgramData\Miniconda3\python.exe
What my Python Script contains:
import wikipedia
print (wikipedia.summary("Windows OS"))
win = wikipedia.page("New York")
win.content
Error I'm getting:
================== RESTART: C:\Users\aglin\Desktop\test.py ==================
Traceback (most recent call last):
File "C:\Users\aglin\Desktop\test.py", line 1, in <module>
import wikipedia
ModuleNotFoundError: No module named 'wikipedia'
My question: How can I import the package wikipedia without getting this error? If it helps, "Wikipedia API for Python's Documentation" can be found here. Also, I'm pretty new to Python, so if I need to do anything specific or look at documentation, please tell me. Thanks! :D

Here are the usual steps to create an environment in conda, putting a version of Python into it, and importing a library into it afterwards.
The steps are:
conda create --name Py36 python=3.6
activate Py36
pip install wikipedia
You will notice that, at the end of the display, it's possible to open the Python interpreter and import wikipedia.
Microsoft Windows [Version 10.0.16299.125]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\scratch>conda create --name Py36 python=3.6
Fetching package metadata .............
Solving package specifications: .
Package plan for installation in environment C:\ProgramData\Miniconda3\envs\Py36:
The following NEW packages will be INSTALLED:
certifi: 2017.11.5-py36hb8ac631_0
pip: 9.0.1-py36h226ae91_4
python: 3.6.3-h3b118a2_4
setuptools: 36.5.0-py36h65f9e6e_0
vc: 14-h2379b0c_2
vs2015_runtime: 14.0.25123-hd4c4e62_2
wheel: 0.30.0-py36h6c3ec14_1
wincertstore: 0.2-py36h7fe50ca_0
Proceed ([y]/n)? y
#
# To activate this environment, use:
# > activate Py36
#
# To deactivate an active environment, use:
# > deactivate
#
# * for power-users using bash, you must source
#
C:\scratch>activate Py36
(Py36) C:\scratch>pip install wikipedia
Collecting wikipedia
Collecting beautifulsoup4 (from wikipedia)
Using cached beautifulsoup4-4.6.0-py3-none-any.whl
Collecting requests<3.0.0,>=2.0.0 (from wikipedia)
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests<3.0.0,>=2.0.0->wikipedia)
Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests<3.0.0,>=2.0.0->wikipedia)
Using cached idna-2.6-py2.py3-none-any.whl
Requirement already satisfied: certifi>=2017.4.17 in c:\programdata\miniconda3\envs\py36\lib\site-packages (from requests<3.0.0,>=2.0.0->wikipedia)
Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0.0,>=2.0.0->wikipedia)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: beautifulsoup4, urllib3, idna, chardet, requests, wikipedia
Successfully installed beautifulsoup4-4.6.0 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22 wikipedia-1.4.0
(Py36) C:\scratch>python
Python 3.6.3 |Anaconda, Inc.| (default, Nov 8 2017, 15:10:56) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import wikipedia
>>> ^Z
(Py36) C:\scratch>

Specifying which python version to pip install to

I'm having troubles when trying to download and run packages within python, using pip install. Often pip will tell me that I have the package downloaded, but when I try to use the function it gives me an error and says 'no such file or directory'. It seems to me that I am using 2.7.10 and that is where pip is trying to install to, but I can't get why this isn't working
Peters-MBP:~ pete$ python import plotly
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file 'import': [Errno 2] No such file or directory
Peters-MBP:~ pete$ pip install plotly
Requirement already satisfied (use --upgrade to upgrade): plotly in ./Library/Python/2.7/lib/python/site-packages
Collecting pytz (from plotly)
Using cached pytz-2016.3-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): requests in ./Library/Python/2.7/lib/python/site-packages (from plotly)
Collecting six (from plotly)
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pytz, six
Successfully installed pytz-2016.3 six-1.10.0
You are using pip version 8.1.0, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Peters-MBP:~ pete$ python --version
Python 2.7.10
Peters-MBP:~ pete$ python
Python 2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Try to upgrade 'pip install --upgrade pip'
Then try to upgrade also plotly and check it again.
Please post if the plotly module is visible in the Python27 installation folder.
In addition you should go inside the "python interpreter" to import a module.
Just try to
> python
import plotly

You can’t just import a module on the command line like that(!!!):
$ python import plotly
Instead, do this:
First start the interactive Python interpreter:
$ python
Then, at the Python prompt, import your installed module:
>>> import plotly
Or, of course, put the import statement in a script and execute that script.
Edit: If you try to run python install plotly at the shell prompt, as you did, the Python interpreter will look for a script called install in the current directory, in order to execute it. Since no such script exists, it gives the error can't open file 'import': [Errno 2] No such file or directory.