Sagemaker default python environments hosted in my work environment have outdated pandas, and therefore must have their conda environment updated. However, this is incredibly slow (15-30 mins), and I would like to find a faster way to get a working environment
I update with the following:
!conda update pandas fsspec --yes
Which gives the following output, with the key problem being an inconsistent starting environment (How?) as shown by
failed with repodata from current_repodata.json, will retry with next repodata source. Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done
output:
Collecting package metadata (current_repodata.json): done
Solving environment: /
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- defaults/linux-64::pandas==1.0.1=py36h0573a6f_0
- defaults/noarch::jupyterlab==1.2.6=pyhf63ae98_0
- defaults/linux-64::scikit-learn==0.22.1=py36hd81dba3_0
- defaults/linux-64::python-language-server==0.31.7=py36_0
- defaults/linux-64::bkcharts==0.2=py36_0
- defaults/linux-64::nb_conda==2.2.1=py36_0
- defaults/noarch::numpydoc==0.9.2=py_0
- defaults/linux-64::pytest-arraydiff==0.3=py36h39e3cac_0
- defaults/linux-64::bottleneck==1.3.2=py36heb32a55_0
- defaults/linux-64::pywavelets==1.1.1=py36h7b6447c_0
- defaults/noarch::pytest-astropy==0.8.0=py_0
- defaults/linux-64::numexpr==2.7.1=py36h423224d_0
- defaults/noarch::anaconda-project==0.8.4=py_0
- defaults/noarch::boto3==1.9.162=py_0
- defaults/linux-64::s3transfer==0.2.1=py36_0
- defaults/linux-64::nbconvert==5.6.1=py36_0
- defaults/linux-64::h5py==2.10.0=py36h7918eee_0
- defaults/linux-64::bokeh==1.4.0=py36_0
- defaults/noarch::jupyterlab_server==1.0.6=py_0
- defaults/linux-64::numpy-base==1.18.1=py36hde5b4d6_1
- defaults/noarch::botocore==1.12.189=py_0
- defaults/linux-64::jupyter==1.0.0=py36_7
- defaults/linux-64::astropy==4.0=py36h7b6447c_0
- defaults/linux-64::patsy==0.5.1=py36_0
- defaults/linux-64::scikit-image==0.16.2=py36h0573a6f_0
- defaults/linux-64::matplotlib-base==3.1.3=py36hef1b27d_0
- defaults/linux-64::imageio==2.6.1=py36_0
- defaults/linux-64::pytables==3.6.1=py36h71ec239_0
- defaults/linux-64::nb_conda_kernels==2.2.4=py36_0
- defaults/linux-64::mkl_fft==1.0.15=py36ha843d7b_0
- defaults/linux-64::statsmodels==0.11.0=py36h7b6447c_0
- defaults/linux-64::spyder==4.0.1=py36_0
- defaults/noarch::seaborn==0.10.0=py_0
- defaults/linux-64::requests==2.22.0=py36_1
- defaults/linux-64::numba==0.48.0=py36h0573a6f_0
- defaults/linux-64::scipy==1.4.1=py36h0b6359f_0
- defaults/noarch::pytest-doctestplus==0.5.0=py_0
- defaults/linux-64::mkl_random==1.1.0=py36hd6b4f25_0
- defaults/noarch::dask==2.11.0=py_0
- defaults/noarch::ipywidgets==7.5.1=py_0
- defaults/linux-64::widgetsnbextension==3.5.1=py36_0
- defaults/noarch::s3fs==0.4.2=py_0
- defaults/linux-64::notebook==6.0.3=py36_0
- defaults/linux-64::matplotlib==3.1.3=py36_0
- defaults/linux-64::anaconda-client==1.7.2=py36_0
- defaults/linux-64::numpy==1.18.1=py36h4f9e942_0
failed with repodata from current_repodata.json, will retry with next repodata source.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: |
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- defaults/noarch::jupyterlab==1.2.6=pyhf63ae98_0
- defaults/linux-64::python-language-server==0.31.7=py36_0
- defaults/linux-64::nb_conda==2.2.1=py36_0
- defaults/noarch::numpydoc==0.9.2=py_0
- defaults/noarch::anaconda-project==0.8.4=py_0
- defaults/noarch::boto3==1.9.162=py_0
- defaults/linux-64::s3transfer==0.2.1=py36_0
- defaults/linux-64::nbconvert==5.6.1=py36_0
- defaults/linux-64::bokeh==1.4.0=py36_0
- defaults/noarch::jupyterlab_server==1.0.6=py_0
- defaults/noarch::botocore==1.12.189=py_0
- defaults/linux-64::jupyter==1.0.0=py36_7
- defaults/linux-64::scikit-image==0.16.2=py36h0573a6f_0
- defaults/linux-64::imageio==2.6.1=py36_0
- defaults/linux-64::nb_conda_kernels==2.2.4=py36_0
- defaults/linux-64::spyder==4.0.1=py36_0
- defaults/linux-64::requests==2.22.0=py36_1
- defaults/noarch::dask==2.11.0=py_0
- defaults/noarch::ipywidgets==7.5.1=py_0
- defaults/linux-64::widgetsnbextension==3.5.1=py36_0
- defaults/noarch::s3fs==0.4.2=py_0
- defaults/linux-64::notebook==6.0.3=py36_0
- defaults/linux-64::anaconda-client==1.7.2=py36_0
done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.4
latest version: 4.9.2
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: /home/ec2-user/anaconda3/envs/python3
added / updated specs:
- fsspec
- pandas
- s3fs
The following packages will be downloaded:
package | build
---------------------------|-----------------
astroid-2.4.2 | py36h9f0ad1d_1 297 KB conda-forge
certifi-2020.12.5 | py36h5fab9bb_1 143 KB conda-forge
docutils-0.16 | py36h5fab9bb_3 738 KB conda-forge
pandas-1.1.4 | py36hd87012b_0 10.5 MB conda-forge
pillow-7.1.2 | py36hb39fc2d_0 604 KB
pylint-2.6.0 | py36h9f0ad1d_1 446 KB conda-forge
sphinx-3.4.3 | pyhd8ed1ab_0 1.5 MB conda-forge
toml-0.10.2 | pyhd8ed1ab_0 18 KB conda-forge
urllib3-1.25.11 | py_0 93 KB conda-forge
------------------------------------------------------------
Total: 14.3 MB
The following NEW packages will be INSTALLED:
astroid conda-forge/linux-64::astroid-2.4.2-py36h9f0ad1d_1
bleach conda-forge/noarch::bleach-3.2.1-pyh9f0ad1d_0
brotlipy conda-forge/linux-64::brotlipy-0.7.0-py36he6145b8_1001
docutils conda-forge/linux-64::docutils-0.16-py36h5fab9bb_3
pillow pkgs/main/linux-64::pillow-7.1.2-py36hb39fc2d_0
pylint conda-forge/linux-64::pylint-2.6.0-py36h9f0ad1d_1
sphinx conda-forge/noarch::sphinx-3.4.3-pyhd8ed1ab_0
toml conda-forge/noarch::toml-0.10.2-pyhd8ed1ab_0
urllib3 conda-forge/noarch::urllib3-1.25.11-py_0
The following packages will be UPDATED:
ca-certificates 2020.11.8-ha878542_0 --> 2020.12.5-ha878542_0
certifi 2020.11.8-py36h5fab9bb_0 --> 2020.12.5-py36h5fab9bb_1
fsspec pkgs/main::fsspec-0.6.2-py_0 --> conda-forge::fsspec-0.8.5-pyhd8ed1ab_0
pandas pkgs/main::pandas-1.0.1-py36h0573a6f_0 --> conda-forge::pandas-1.1.4-py36hd87012b_0
Downloading and Extracting Packages
pillow-7.1.2 | 604 KB | ##################################### | 100%
astroid-2.4.2 | 297 KB | ##################################### | 100%
pylint-2.6.0 | 446 KB | ##################################### | 100%
sphinx-3.4.3 | 1.5 MB | ##################################### | 100%
pandas-1.1.4 | 10.5 MB | ##################################### | 100%
docutils-0.16 | 738 KB | ##################################### | 100%
urllib3-1.25.11 | 93 KB | ##################################### | 100%
certifi-2020.12.5 | 143 KB | ##################################### | 100%
toml-0.10.2 | 18 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Happy to take any suggestions for how to get a python notebook up in sagemaker as quickly as possible with modern packages.
Other attempted solutions:
a fast pip install -U doesn't work due to dependency issues -- the
local environment in the notebook will try to point pandas to
outdated fsspec and it will crash
Following AWS documentation for adding my conda requests to the startup script doesn't work because there is a timeout on the startup script (10 mins I think?) so a 15+ minute conda update process just ensures the sagemaker instance cannot start
The reason for this issue is because conda does a dependencies checks. It tries to find the version of the package which is compatible with all packages while pip install the required package and it's dependencies which might result with inconsistency. [1]
There are two workarounds for this issue,
Creating a custom environment with the required packages and create a kernel to be used from Sagemaker notebook.
Using the --no-deps option pip install pandas==<version> --no-deps. You might need to use the -U option.
To recap, I would suggest either creating a custom environment or using pip to install the package and all it's dependencies with the option --no-deps. You might need to try both approaches while the notebook is running and then apply to the lifecycle configurations script.
Related
During package installation on my Windows, there was an error, during which conda (most probably) deleted itself. Now, conda command is not present on my Windows.
The main error is:
ERROR conda.core.link:_execute(733):
An error occurred while installing package 'defaults::conda-22.11.0-py39haa95532_1'
Question. How can I restore conda command without (deleting and) reinstalling all Anaconda?
I do not want t lose all installed packages, environments, etc. and reinstall everything. Is there a solution to this?
Details:
The result of OCR tool (I could not just copy-paste):
C:\WINDOWS\system32> conda install -c conda-forge textblob
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: C:\Users\user\anaconda3
added / updated specs:
- textblob
The following packages will be downloaded:
package | build
----------------------------|---------------
conda-22.11.0 | py39haa9SS32_1 932 KB
openssl-l.l.ls | hcfcfb64_1 5.1 MB conda-forge
ruamel.yaml-0.16.12 | py39h2bbfflb_3 173 KB
ruamel.yaml.clib-0.2.7 | py39ha55989b_0 111 KB conda-forge
------------------------------------------------------
Total: 6.3 MB
The following NEW packages will be INSTALLED:
ruamel.yaml pkgs/main/win-64::ruamel.yaml-0.16.12-py39h2bbfflb_3 None
ruamel.yaml.clib conda-forge/win-64::ruamel.yaml.clib-0.2.7-py39ha55989b_0 None
textblob conda-forge/noarch::textblob-0.15.3-py_0 None
The following packages will be UPDATED:
conda conda-forge::conda-22.9.0-py39hcbf530- --> pkgs/main::conda-22.11.0-py39haa95532_l None
openssl l.l.ls-hcfcfb64_0 --> 1.1.ls-hcfcfb64_l None
Proceed ([y]/n)? y
Downloading and Extracting Packages
openssl-l.l.ls | 5.1 MB |
ruamel.yaml-0.16.12 | 173 KB
conda-22.11.0 | 932 KB
ruamel.yaml.clib-0.2 | 111 KB |
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
ERROR conda.core.link:_execute(733): An error occurred while installing package 'defaults::conda-22.11.0-py39haa95532_1'.
Rolling back transaction: done
CondaError: Cannot link a source that does not exist.
C:\Users\user\anaconda3\.condatmp\0d55d647-3842-4S31-a301-4bedl75b9998
Running 'conda clean --packages' may resolve your problem.
()
[Errno 2] No such file or directory: ’C:\\Users\\user\\anaconda3\\conda-meta\\openssl-l.l.ls-hcfcfb64_0.json’
[Errno 2] No such file or directory: ’C:\\Users\\user\\anaconda3\\conda-meta\\conda-22.9.0-py39hcbf5309_2.json’
The batch file cannot be found.
The batch file cannot be found.
I want to install Scrapy on Windows Server 2019, running in a Docker container (please see here and here for the history of my installation).
On my local Windows 10 machine I can run my Scrapy commands like so in Windows PowerShell (after simply starting Docker Desktop):
scrapy crawl myscraper -o allobjects.json in folder C:\scrapy\my1stscraper\
For Windows Server as recommended here I first installed Anaconda following these steps: https://docs.scrapy.org/en/latest/intro/install.html.
I then opened the Anaconda prompt and typed conda install -c conda-forge scrapy in D:\Programs
(base) PS D:\Programs> dir
Directory: D:\Programs
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 4/22/2021 10:52 AM Anaconda3
-a---- 4/22/2021 11:20 AM 0 conda
(base) PS D:\Programs> conda install -c conda-forge scrapy
Collecting package metadata (current_repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.9.2
latest version: 4.10.1
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: D:\Programs\Anaconda3
added / updated specs:
- scrapy
The following packages will be downloaded:
package | build
---------------------------|-----------------
automat-20.2.0 | py_0 30 KB conda-forge
conda-4.10.1 | py38haa244fe_0 3.1 MB conda-forge
constantly-15.1.0 | py_0 9 KB conda-forge
cssselect-1.1.0 | py_0 18 KB conda-forge
hyperlink-21.0.0 | pyhd3deb0d_0 71 KB conda-forge
incremental-17.5.0 | py_0 14 KB conda-forge
itemadapter-0.2.0 | pyhd8ed1ab_0 12 KB conda-forge
parsel-1.6.0 | py_0 15 KB conda-forge
pyasn1-0.4.8 | py_0 53 KB conda-forge
pyasn1-modules-0.2.7 | py_0 60 KB conda-forge
pydispatcher-2.0.5 | py_1 12 KB conda-forge
pyhamcrest-2.0.2 | py_0 29 KB conda-forge
python_abi-3.8 | 1_cp38 4 KB conda-forge
queuelib-1.6.1 | pyhd8ed1ab_0 14 KB conda-forge
scrapy-2.4.1 | py38haa95532_0 372 KB
service_identity-18.1.0 | py_0 12 KB conda-forge
twisted-21.2.0 | py38h294d835_0 5.1 MB conda-forge
twisted-iocpsupport-1.0.1 | py38h294d835_0 49 KB conda-forge
w3lib-1.22.0 | pyh9f0ad1d_0 21 KB conda-forge
------------------------------------------------------------
Total: 9.0 MB
The following NEW packages will be INSTALLED:
automat conda-forge/noarch::automat-20.2.0-py_0
constantly conda-forge/noarch::constantly-15.1.0-py_0
cssselect conda-forge/noarch::cssselect-1.1.0-py_0
hyperlink conda-forge/noarch::hyperlink-21.0.0-pyhd3deb0d_0
incremental conda-forge/noarch::incremental-17.5.0-py_0
itemadapter conda-forge/noarch::itemadapter-0.2.0-pyhd8ed1ab_0
parsel conda-forge/noarch::parsel-1.6.0-py_0
pyasn1 conda-forge/noarch::pyasn1-0.4.8-py_0
pyasn1-modules conda-forge/noarch::pyasn1-modules-0.2.7-py_0
pydispatcher conda-forge/noarch::pydispatcher-2.0.5-py_1
pyhamcrest conda-forge/noarch::pyhamcrest-2.0.2-py_0
python_abi conda-forge/win-64::python_abi-3.8-1_cp38
queuelib conda-forge/noarch::queuelib-1.6.1-pyhd8ed1ab_0
scrapy pkgs/main/win-64::scrapy-2.4.1-py38haa95532_0
service_identity conda-forge/noarch::service_identity-18.1.0-py_0
twisted conda-forge/win-64::twisted-21.2.0-py38h294d835_0
twisted-iocpsuppo~ conda-forge/win-64::twisted-iocpsupport-1.0.1-py38h294d835_0
w3lib conda-forge/noarch::w3lib-1.22.0-pyh9f0ad1d_0
The following packages will be UPDATED:
conda pkgs/main::conda-4.9.2-py38haa95532_0 --> conda-forge::conda-4.10.1-py38haa244fe_0
Proceed ([y]/n)? y
Downloading and Extracting Packages
constantly-15.1.0 | 9 KB | ############################################################################ | 100%
itemadapter-0.2.0 | 12 KB | ############################################################################ | 100%
twisted-21.2.0 | 5.1 MB | ############################################################################ | 100%
pydispatcher-2.0.5 | 12 KB | ############################################################################ | 100%
queuelib-1.6.1 | 14 KB | ############################################################################ | 100%
service_identity-18. | 12 KB | ############################################################################ | 100%
pyhamcrest-2.0.2 | 29 KB | ############################################################################ | 100%
cssselect-1.1.0 | 18 KB | ############################################################################ | 100%
automat-20.2.0 | 30 KB | ############################################################################ | 100%
pyasn1-0.4.8 | 53 KB | ############################################################################ | 100%
twisted-iocpsupport- | 49 KB | ############################################################################ | 100%
python_abi-3.8 | 4 KB | ############################################################################ | 100%
hyperlink-21.0.0 | 71 KB | ############################################################################ | 100%
conda-4.10.1 | 3.1 MB | ############################################################################ | 100%
scrapy-2.4.1 | 372 KB | ############################################################################ | 100%
incremental-17.5.0 | 14 KB | ############################################################################ | 100%
w3lib-1.22.0 | 21 KB | ############################################################################ | 100%
pyasn1-modules-0.2.7 | 60 KB | ############################################################################ | 100%
parsel-1.6.0 | 15 KB | ############################################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(base) PS D:\Programs>
In PowerShell on my VPS I then tried to run scrapy via D:\Programs\Anaconda3\Scripts\scrapy.exe
I want to run the spider I have stored in folder D:\scrapy\my1stscraper, see:
The Docker Engine service is running as a Windows Service (presuming I don't need to explicitly start a container when running my scrapy command, if I do, I would not know how):
I tried starting my scraper like so D:\Programs\Anaconda3\Scripts\scrapy.exe crawl D:\scrapy\my1stscraper\spiders\my1stscraper -o allobjects.json, resulting in errors:
Traceback (most recent call last):
File "D:\Programs\Anaconda3\Scripts\scrapy-script.py", line 6, in <module>
from scrapy.cmdline import execute
File "D:\Programs\Anaconda3\lib\site-packages\scrapy\__init__.py", line 12, in <module>
from scrapy.spiders import Spider
File "D:\Programs\Anaconda3\lib\site-packages\scrapy\spiders\__init__.py", line 11, in <module>
from scrapy.http import Request
File "D:\Programs\Anaconda3\lib\site-packages\scrapy\http\__init__.py", line 11, in <module>
from scrapy.http.request.form import FormRequest
File "D:\Programs\Anaconda3\lib\site-packages\scrapy\http\request\form.py", line 10, in <module>
import lxml.html
File "D:\Programs\Anaconda3\lib\site-packages\lxml\html\__init__.py", line 53, in <module>
from .. import etree
ImportError: DLL load failed while importing etree: The specified module could not be found.
I checked here:
from lxml import etree ImportError: DLL load failed: The specified module could not be found
This talks about pip, which I did not use, but to be sure I did install the C++ build tools:
I still get the same error. How can I run my Scrapy crawler in the Docker container?
UPDATE 1
My VPS is my only environment so not sure how to test in a virtual environment.
What I did now:
Uninstall Anacondo
Install Miniconda with Python 3.8 (https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe), did not add to path and used miniconda as systems' python 3.8
Looking at your recommendations:
Get steps to manually install the app on Windows Server - ideally test in a virtualised environment so you can reset it cleanly
When you say app, what do you mean? Scrapy? Conda?
Convert all steps to a fully automatic powershell script (e.g. for conda, need to download the installer via wget, execute the installer etc.
I now installed Conda on the host OS, since I thought that would allow me to have the least amount of overhead. Or would you install it in the image directly and if so, how do I not have to install it each time?
Lastly, just to check to be sure, I want to run multiple Scrapy scrapers, but I want to do this with as little overhead as possible.
I should just repeat the RUN command in the SAME docker container for each scraper I want to execute, correct?
UPDATE 2
whomami indeed returns user manager\containeradministrator
scrapy benchmark returns
Scrapy 2.4.1 - no active project
Unknown command: benchmark
Use "scrapy" to see available commands
I have the scrapy project I want to run in folder D:\scrapy\my1stscraper, how can I run that project, since D:\ drive is not available within my container?
UPDATE 3
A few months later when we discussed this, when I now run your proposed the Dockerfile it breaks and I now get this output:
PS D:\Programs> docker build . -t scrapy
Sending build context to Docker daemon 1.644GB
Step 1/9 : FROM mcr.microsoft.com/windows/servercore:ltsc2019
---> d1724c2d9a84
Step 2/9 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
---> Running in 5f79f1bf9b62
Removing intermediate container 5f79f1bf9b62
---> 8bb2a477eaca
Step 3/9 : RUN setx /M PATH $('C:\Users\ContainerAdministrator\miniconda3\Library\bin;C:\Users\ContainerAdministrator\miniconda3\Scripts;C:\Users\ContainerAdministrator\miniconda3;' + $Env:PATH)
---> Running in f3869c4f64d5
SUCCESS: Specified value was saved.
Removing intermediate container f3869c4f64d5
---> 82a2fa969a88
Step 4/9 : RUN Invoke-WebRequest "https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe" -OutFile miniconda3.exe -UseBasicParsing; Start-Process -FilePath 'miniconda3.exe' -Wait -ArgumentList '/S', '/D=C:\Users\ContainerAdministrator\miniconda3'; Remove-Item .\miniconda3.exe; conda install -y -c conda-forge scrapy;
---> Running in 3eb8b7bfe878
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with the existing python installation in your environment:
Specifications:
- scrapy -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|3.4.*']
Your python: python=3.9
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
Not sure if I'm reading this correctly but it seems as if Scrapy does not support Python 3.9, except that here I see "Scrapy requires Python 3.6+" https://docs.scrapy.org/en/latest/intro/install.html
Do you know what's causing this issue? I also checked here but no answer yet either.
To run a containerised app, it must be installed in a container image first - you don't want to install any software on the host machine.
For linux there are off-the-shelf container images for everything which is probably what your docker desktop environment was using; I see 1051 results on docker hub search for scrapy but none of them are windows containers.
The full process of creating a windows container from scratch for an app is:
Get steps to manually install the app (scrapy and its dependencies) on Windows Server - ideally test in a virtualised environment so you can reset it cleanly
Convert all steps to a fully automatic powershell script (e.g. for conda, need to download the installer via wget, execute the installer etc.
Optionaly, test the powershell steps in an interactive container
docker run -it --isolation=process mcr.microsoft.com/windows/servercore:ltsc2019 powershell
This runs a windows container and gives you a shell to verify that your install script works
When you exit the shell the container is stopped
Create a Dockerfile
Use mcr.microsoft.com/windows/servercore:ltsc2019 as the base image via FROM
Use the RUN command for each line of your powershell script
I tried installing scrapy on an existing windows Dockerfile that used conda / python 3.6, it threw error SettingsFrame has no attribute 'ENABLE_CONNECT_PROTOCOL' at a similar stage.
However I tried again with miniconda and python 3.8, and was able to get scrapy running, here's the dockerfile:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN setx /M PATH $('C:\Users\ContainerAdministrator\miniconda3\Library\bin;C:\Users\ContainerAdministrator\miniconda3\Scripts;C:\Users\ContainerAdministrator\miniconda3;' + $Env:PATH)
RUN Invoke-WebRequest "https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Windows-x86_64.exe" -OutFile miniconda3.exe -UseBasicParsing; \
Start-Process -FilePath 'miniconda3.exe' -Wait -ArgumentList '/S', '/D=C:\Users\ContainerAdministrator\miniconda3'; \
Remove-Item .\miniconda3.exe; \
conda install -y -c conda-forge scrapy;
Build it with docker build . -t scrapy and run with docker run -it scrapy.
To verify you are running a shell inside the container run whoami - should return user manager\containeradministrator.
Then, scrapy benchmark command should be able to run and dump some stats.
The container will stop when you close the shell.
Windows 10
conda 4.9.2 (via miniconda)
I installed a single package that did not require any other dependencies to be installed anew or upgraded. Once I realised that I had installed an unsuitable version of the package, I went to remove it, and this is the screen I was presented with:
(pydata) PS C:\Users\Navneeth> conda remove xlrd
Collecting package metadata (repodata.json): done
Solving environment: |
Warning: 2 possible package resolutions (only showing differing packages):
- defaults/win-64::libtiff-4.1.0-h56a325e_1, defaults/win-64::zstd-1.4.9-h19a0ad4_0
- defaults/win-64::libtiff-4.2.0-hd0e1b90_0, defaults/win-64::zstd-1.4.5-h04227a9done
## Package Plan ##
environment location: C:\Users\Navneeth\Miniconda3\envs\pydata
removed specs:
- xlrd
The following packages will be downloaded:
package | build
---------------------------|-----------------
decorator-5.0.3 | pyhd3eb1b0_0 12 KB
importlib-metadata-3.7.3 | py38haa95532_1 31 KB
importlib_metadata-3.7.3 | hd3eb1b0_1 11 KB
ipython-7.22.0 | py38hd4e2768_0 998 KB
jupyter_client-6.1.12 | pyhd3eb1b0_0 88 KB
libtiff-4.1.0 | h56a325e_1 739 KB
nbformat-5.1.3 | pyhd3eb1b0_0 44 KB
notebook-6.3.0 | py38haa95532_0 4.4 MB
pandoc-2.12 | haa95532_0 13.2 MB
parso-0.8.2 | pyhd3eb1b0_0 69 KB
pillow-8.2.0 | py38h4fa10fc_0 671 KB
prometheus_client-0.10.0 | pyhd3eb1b0_0 46 KB
prompt-toolkit-3.0.17 | pyh06a4308_0 256 KB
terminado-0.9.4 | py38haa95532_0 26 KB
zipp-3.4.1 | pyhd3eb1b0_0 15 KB
zstd-1.4.9 | h19a0ad4_0 478 KB
------------------------------------------------------------
Total: 21.0 MB
The following packages will be REMOVED:
xlrd-2.0.1-pyhd3eb1b0_0
The following packages will be UPDATED:
decorator 4.4.2-pyhd3eb1b0_0 --> 5.0.3-pyhd3eb1b0_0
importlib-metadata pkgs/main/noarch::importlib-metadata-~ --> pkgs/main/win-64::importlib-metadata-3.7.3-py38haa95532_1
importlib_metadata 2.0.0-1 --> 3.7.3-hd3eb1b0_1
ipython 7.21.0-py38hd4e2768_0 --> 7.22.0-py38hd4e2768_0
jupyter_client 6.1.7-py_0 --> 6.1.12-pyhd3eb1b0_0
nbformat 5.1.2-pyhd3eb1b0_1 --> 5.1.3-pyhd3eb1b0_0
notebook 6.2.0-py38haa95532_0 --> 6.3.0-py38haa95532_0
pandoc 2.11-h9490d1a_0 --> 2.12-haa95532_0
parso 0.8.1-pyhd3eb1b0_0 --> 0.8.2-pyhd3eb1b0_0
pillow 8.1.2-py38h4fa10fc_0 --> 8.2.0-py38h4fa10fc_0
prometheus_client 0.9.0-pyhd3eb1b0_0 --> 0.10.0-pyhd3eb1b0_0
prompt-toolkit 3.0.8-py_0 --> 3.0.17-pyh06a4308_0
sqlite 3.33.0-h2a8f88b_0 --> 3.35.3-h2bbff1b_0
terminado 0.9.2-py38haa95532_0 --> 0.9.4-py38haa95532_0
zipp 3.4.0-pyhd3eb1b0_0 --> 3.4.1-pyhd3eb1b0_0
zstd 1.4.5-h04227a9_0 --> 1.4.9-h19a0ad4_0
The following packages will be DOWNGRADED:
libtiff 4.2.0-he0120a3_0 --> 4.1.0-h56a325e_1
Proceed ([y]/n)?
Why does conda want to update or downgrade all these other packages when the opposite wasn't done when I installed xlrd? Is there a way that I can safely remove the just xlrd. (I hear using --force is risky.)
Asymmetry
Conda re-solves when removing. When installing, Conda first attempts a frozen solve, which amounts to keeping all installed packages fixed and just searching for a version of the requested package(s) that are compatible. In this specific case, xlrd (v2.1.0) is a noarch with only a python>=3.6 constraint. So this installs in this frozen solve pass.
The constraint xlrd will also be added to the explicit specifications.1
When removing, Conda will first remove the constraint, and then re-solves the environment with the new set of explicit specifications. It is in this solve that Conda identifies that newer versions of packages and then proposes updating then.
So, the asymmetry is that the frozen solve explicitly avoids checking for any new packages, but the removal will trigger such a check. There is not currently a way to avoid this without bypassing dependency checking.
Mamba
Actually, mamba, a compiled (fast!) drop-in replacement for conda, will remove only the specified package if it doesn't have anything depending on it. That is its default behavior in my testing.
Addendum: Still Some Unexplained Behavior
I replicated your experience by first creating an environment with two specs:
name: foo
channels:
- conda-forge
dependencies:
- python=3.8.0
- pip=20
To simulate this being an old environment, I went into the envs/foo/conda-meta/history and changed2 the line
# update specs: ['pip=20', 'python=3.8.0']
to
# update specs: ['python=3.8']
Subsequently running conda install xlrd does as expected. Then conda remove xlrd gives a somewhat odd result:
## Package Plan ##
environment location: /opt/conda/envs/foo
removed specs:
- xlrd
The following packages will be downloaded:
package | build
---------------------------|-----------------
pip-21.1.1 | pyhd8ed1ab_0 1.1 MB conda-forge
------------------------------------------------------------
Total: 1.1 MB
The following packages will be REMOVED:
xlrd-2.0.1-pyhd8ed1ab_3
The following packages will be UPDATED:
pip 20.3.4-pyhd8ed1ab_0 --> 21.1.1-pyhd8ed1ab_0
Proceed ([y]/n)?
This effectively replicates OP result, however, the additional oddity here is that the python package is not suggested to be updated, even though I had intentionally loosened its constraint from 3.8.0 to 3.8. It appears that only packages not in the explicit specifications are subject to updating during package removal.
[1] The explicit specifications are the internally maintained records that Conda keeps of every constraint a user has explicitly specified. One can view the current explicit specifications of an environment with conda env export --from-history. The raw internal records can be found at yourenv/conda-meta/history.
[2] Not a recommended practice!
Following the release of Matplotlib to the 3.1.2 version I am having issues updating my package version.
I tried:
conda install -c conda-forge matplotlib=3.1.2 in Jupyter notebook (Without success, the code kept running for 20 mins before I interrupted); in the Anaconda prompt with the following failed result:
(base) C:\Users\Adrien>conda install -c conda-forge matplotlib=3.1.2
Collecting package metadata (current_repodata.json): done Solving
environment: failed with initial frozen solve. Retrying with flexible
solve. Collecting package metadata (repodata.json): done Solving
environment: failed with initial frozen solve. Retrying with flexible
solve. Solving environment: | Found conflicts! Looking for
incompatible packages. This can take several minutes. Press CTRL-C to
abort. failed
conda update matplotlib with the following result:
(base) C:\Users\Adrien>conda update matplotlib Collecting package
metadata (current_repodata.json): done Solving environment: /
Updating matplotlib is constricted by
anaconda -> requires matplotlib==3.1.1=py37hc8f65d3_0
If you are sure you want an update of your package either try conda
update --all or install a specific version of the package you want
using conda install <pkg>=<version>
done
Package Plan
environment location: C:\Users\Adrien\Anaconda3
added / updated specs:
- matplotlib
The following packages will be downloaded:
package | build
---------------------------|-----------------
backports.functools_lru_cache-1.6.1| py_0 11 KB
conda-4.8.3 | py37_0 2.8 MB
future-0.18.2 | py37_0 656 KB
------------------------------------------------------------
Total: 3.5 MB
The following packages will be UPDATED:
backports.functoo~ 1.5-py_2 -->
1.6.1-py_0 conda 4.8.2-py37_0 --> 4.8.3-py37_0 future 0.17.1-py37_0 --> 0.18.2-py37_0
Proceed ([y]/n)? y
Downloading and Extracting Packages conda-4.8.3 | 2.8 MB |
################################################################## | 100% future-0.18.2 | 656 KB |
################################################################## | 100% backports.functools_ | 11 KB |
################################################################## | 100% Preparing transaction: done Verifying transaction: done
Executing transaction: done
(base) C:\Users\Adrien>import matplotlib 'import' is not recognized as
an internal or external command, operable program or batch file.
And after restarting the system and Jupyter as you guess:
import matplotlib
print('matplotlib: {}'.format(matplotlib.__version__))
matplotlib: 3.1.1
Any idea on what could be the next step ?
Many thanks in advance
Either you do:
conda update --all
or you try:
conda install matplotlib=3.1.2
I'd like to use PyTorch in a Python program. The instructions for installing it require conda. After installing Conda I ran:
>conda install -c pytorch pytorch (as instructed on the PyTorch [page][1])
It looked promising -- until the end.
Solving environment: done
## Package Plan ##
environment location: C:\ProgramData\Miniconda3
added / updated specs:
- pytorch
The following packages will be downloaded:
package | build
---------------------------|-----------------
icc_rt-2017.0.4 | h97af966_0 8.0 MB
vs2015_runtime-15.5.2 | 3 2.2 MB
pytorch-0.4.0 |py36_cuda80_cudnn7he774522_1 529.2 MB pytorch
mkl-2018.0.3 | 1 178.1 MB
numpy-1.14.5 | py36h9fa60d3_4 35 KB
intel-openmp-2018.0.3 | 0 1.7 MB
numpy-base-1.14.5 | py36h5c71026_4 3.8 MB
vc-14.1 | h0510ff6_3 5 KB
blas-1.0 | mkl 6 KB
conda-4.5.8 | py36_0 1.0 MB
mkl_fft-1.0.2 | py36hb217b18_0 113 KB
mkl_random-1.0.1 | py36h77b88f5_1 268 KB
------------------------------------------------------------
Total: 724.4 MB
The following NEW packages will be INSTALLED:
blas: 1.0-mkl
icc_rt: 2017.0.4-h97af966_0
intel-openmp: 2018.0.3-0
mkl: 2018.0.3-1
mkl_fft: 1.0.2-py36hb217b18_0
mkl_random: 1.0.1-py36h77b88f5_1
numpy: 1.14.5-py36h9fa60d3_4
numpy-base: 1.14.5-py36h5c71026_4
pytorch: 0.4.0-py36_cuda80_cudnn7he774522_1 pytorch
The following packages will be UPDATED:
conda: 4.5.4-py36_0 --> 4.5.8-py36_0
vc: 14-h0510ff6_3 --> 14.1-h0510ff6_3
vs2015_runtime: 14.0.25123-3 --> 15.5.2-3
Proceed ([y]/n)? y
Downloading and Extracting Packages
icc_rt-2017.0.4 | 8.0 MB | ############################################################################## | 100%
vs2015_runtime-15.5. | 2.2 MB | ############################################################################## | 100%
pytorch-0.4.0 | 529.2 MB | ############################################################################# | 100%
mkl-2018.0.3 | 178.1 MB | ############################################################################# | 100%
numpy-1.14.5 | 35 KB | ############################################################################## | 100%
intel-openmp-2018.0. | 1.7 MB | ############################################################################## | 100%
numpy-base-1.14.5 | 3.8 MB | ############################################################################## | 100%
vc-14.1 | 5 KB | ############################################################################## | 100%
blas-1.0 | 6 KB | ############################################################################## | 100%
conda-4.5.8 | 1.0 MB | ############################################################################## | 100%
mkl_fft-1.0.2 | 113 KB | ############################################################################## | 100%
mkl_random-1.0.1 | 268 KB | ############################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
But then this.
Executing transaction: failed
ERROR conda.core.link:_execute(502): An error occurred while uninstalling package 'defaults::conda-4.5.4-py36_0'.
PermissionError(13, 'Access is denied')
Attempting to roll back.
Rolling back transaction: done
PermissionError(13, 'Access is denied')
Apparently it was at least partly installed because PyCharm was able to see it. But when I asked PyCharm to install it in an environment, I got this error.
RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).
Please follow the instructions at http://pytorch.org/ to install with miniconda instead.
It suggests an alternative way to install PyTorch. So I tried that.
>conda install pytorch torchvision -c pytorch
Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
- torchvision
Current channels:
- https://conda.anaconda.org/pytorch/win-64
- https://conda.anaconda.org/pytorch/noarch
- https://repo.anaconda.com/pkgs/main/win-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/free/win-64
- https://repo.anaconda.com/pkgs/free/noarch
- https://repo.anaconda.com/pkgs/r/win-64
- https://repo.anaconda.com/pkgs/r/noarch
- https://repo.anaconda.com/pkgs/pro/win-64
- https://repo.anaconda.com/pkgs/pro/noarch
- https://repo.anaconda.com/pkgs/msys2/win-64
- https://repo.anaconda.com/pkgs/msys2/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
But when I do that and search for PyTorch, I eventually find myself back at the original instructions.
When I search for Torchvision, no Windows versions are listed.
Try the following steps in Windows:
Create a virtual environment using the command :
conda create -n py_env python=3.5
source activate py_env
conda install pytorch-cpu -c pytorch
pip install torchvision
Note: You can use any name instead of py_env
Thanks
What is your platform?
For your first installation method, the error message says that you don't have the permission. I encountered that error before on a Linux system. The reason was that Anaconda was installed by another user. I configured the path to point python to that installation so that I could run python without installing my own copy of Anaconda. However, it didn't permit me installing new packages and I got the same error message.
Solution: I installed my own copy of Anaconda and everything worked.
just run:
pip install torch torchvision
An alternative way to install PyTorch is the following steps:
conda create -n pytorch_env python=3
source activate pytorch_env
conda install pytorch-cpu torchvision -c pytorch
Go to python shell and import using the command
import torch
Open the terminal in administrative mode and if you are in linux try
sudo pip install "your package name"