I have two machine that one does not have internet access. I want to install modules with anaconda and copy them to offline computer from the other computer that has internet access.
I tried looking for dependencies and install tar. files manually one by one and sent them to the offline machine but it is very time-consuming.
What is the easiest way? Does miniconda helpful ??
P.S: I forgot to mention that I am using anaconda in both machines. So I guess I need to create an env., install packages then export it for offline computer. Are there any other way to install number of packages to offline comp. from a copy <dir> in the online computer ??
Edit: I tried conda install --file C:\Users\myName\Desktop\OfflineInstall\packagelist.txt --channel file://C:\Users\myName\Desktop\OfflineInstall\pkgs2 but offline machine still tried to connect internet. I also used --no-deps
Edit2: For those who stuck on the same problem, I solved using conda install --file C:\Users\myName\Desktop\OfflineInstall\packagelist.txt --channel file:///C:\Users\myName\Desktop\OfflineInstall\pkgs2 --override-channels The tricky way is the file:/// prefix. You need to put ///. Also remember to put --override-channels flag to prevent connection to default channels.
It sounds like Conda-pack is what you are looking for.
Installing:
$ conda install conda-pack
On the source machine:
# Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
On the target machine:
# Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env
# Use python without activating or fixing the prefixes. Most python
# libraries will work fine, but things that require prefix cleanups
# will fail.
$ ./my_env/bin/python
# Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate
# Run python from in the environment
(my_env) $ python
# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of python is already installed on the machine.
(my_env) $ conda-unpack
The caveat being that conda-pack will take the whole environment.
Had this problem the other day, very simple implementation.
First make a .txt file which contains all your python libraries. Now you can just pass this .txt file to whatever machine you want the solution to be installed under and issue the following command :
pip install -r packages.txt
Where "packages" is the name of your .txt file. Hope this helps!
Edit using Conda :
while read requirement; do conda install --yes $requirement; done < requirements.txt
Related
I've tried everything I've seen on SO to get this to work, but so far everything fails. Using macOS Big Sur 11.6, bash in Terminal (not zsh).
I'm trying to create a setup file and execute with sh setup.sh that will setup the env, install python, and then activate it. Nothing fancy. Doing it manually works fine, but once I put it in a shell script, it won't work. I'm running this script from inside an empty project folder.
Current script:
conda create -n MASTER python=3.8.5 -y
conda activate MASTER
Yeah, it's that simple to start with. I commented out the other pip installs until this works properly.
I tried running: bash -i setup.sh but it still does not activate. I get no errors but I'm still stuck in (base).
I tried using source: source /opt/anaconda3/etc/profile.d/conda.sh at beginning of script and/or before activate, still doesn't work. No errors again, but stuck in (base).
I tried using: eval $(conda shell.bash hook) at the start of script and before I try to activate the env, but it fails. This time I get the error:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
But if I run conda init bash (in Terminal or in the script itself), it outputs:
no change /opt/anaconda3/condabin/conda
no change /opt/anaconda3/bin/conda
no change /opt/anaconda3/bin/conda-env
no change /opt/anaconda3/bin/activate
no change /opt/anaconda3/bin/deactivate
no change /opt/anaconda3/etc/profile.d/conda.sh
no change /opt/anaconda3/etc/fish/conf.d/conda.fish
no change /opt/anaconda3/shell/condabin/Conda.psm1
no change /opt/anaconda3/shell/condabin/conda-hook.ps1
no change /opt/anaconda3/lib/python3.8/site-packages/xontrib/conda.xsh
no change /opt/anaconda3/etc/profile.d/conda.csh
no change /Users/liquidRock/.bash_profile
No action taken.
I tried doing /opt/anaconda3/bin/conda activate MASTER which also prompts me to do conda init bash.
Even tried adding #!/bin/bash to the top of the file just in case, but no dice.
Thanks to #fravadona for the simplest of solutions.
Simply executing the script with source instead of sh. 🤦🏻‍♂️
Final setup.sh script (with my preliminary pip installs):
# env & python
conda create -n MASTER python=3.8.5 -y
conda activate MASTER
# pip installs
pip install cmake
pip install --upgrade pip setuptools wheel
pip install opencv-python==4.2.0.32
pip install argparse
pip install datetime
pip install colorama
pip install python-dotenv
pip install python-dotenv[cli]
Executed thusly:
$ source setup.sh
Anaconda creates the env, installs python and dependencies, activates the env, then pip installs the additional dependencies.
Still not sure why it won't work by adding other things to the shell script, but this is still a great, simple solution. And yes, I am a novice with this stuff.
I have ssh access to a GPU cluster where I am taking over research from someone else. I wish to take the anaconda envs from /rhome/someoneelse/anaconda3/envs and move them to my installation of anaconda on /rhome/me/anaconda3. I have tried to directly copy the environment to the new directory but that didn't work. What is the way to copy the environment and have my anaconda installation work?
Use conda-pack
Install conda-pack
conda install -c conda-forge conda-pack
or
pip install conda-pack
Zipping other persons environment
While in the other machine and inside the other persons environment pack the environment and save to a tar file:
conda pack -n other_persons_environment -o environment_unpacked.tar.gz
Transfer the file environment_unpacked.tar.gz to your machine using scp or another tool.
Now to unzip the environment
mkdir -p new_directory_for_your_environment
tar -xzf environment_unpacked.tar.gz -C new_directory_for_your_environment
Unpacking
source new_directory_for_your_environment/bin/activate
python
conda-unpack
To your comment about running conda , you can see which conda is running by this command: which conda (to see the path from where conda is executing.)
Note - I misunderstood the question i.e. I though that the environment was on a different machine. It should be a lot easier if it is on the same machine. You would need to use the same steps with conda-pack and just omit the scp (other tool) file transfer part.
I've got a Python virtual environment that is intended to provide an environment for many people.
It is to be stored in SVN in order to ensure that the environment is "frozen" for various releases.
I have attempted to make it "dynamic" in the sense that the environment activation script is made to not contain a hardcoded path and I have attempted to make it relocatable because it is to be used by many people at many different directories.
I have archived the environment because storing it as a directory in the repository would mean that updates to the environment would result in SVN change summaries that would be almost unreadable.
In have split the archive across multiple files in order to respect SVN file size restrictions.
Is this approach robust? Are there likely to be problems when multiple people use it at multiple directories? Is archiving in the way I am doing it a good idea? Are there better approaches to any of the ideas I've expressed?
The specific procedure I'm using is something like the following:
ssh "${USER}"#sern.ch
setupSALTAEnvironment
echo "create Python virtual environment"
virtual_environment_name="virtual_environment"
virtualenv "${virtual_environment_name}"
echo "make activation dynamic"
activate_filename="virtual_environment/bin/activate"
temporary_filename="/tmp/"$(date "+%Y-%m-%dT%H%MZ" --utc)""
cat > "${temporary_filename}" << "EOF"
directory_bin="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
directory_env="$(dirname "${directory_bin}")"
VIRTUAL_ENV="${directory_env}"
EOF
sed -i "/^VIRTUAL_ENV.*/ {
r ${temporary_filename}
d
}" "${activate_filename}"
rm "${temporary_filename}"
echo "activate Python virtual environment"
source "${virtual_environment_name}"/bin/activate
IFS= read -d '' text << "EOF"
import sys
reload(sys)
sys.setdefaultencoding("utf8")
EOF
echo "${text}" > "${virtual_environment_name}"/lib/python2.7/site-packages/sitecustomize.py
echo "install software in Python virtual environment"
pip install docopt
pip install dataset
pip install numpy
pip install matplotlib
pip install pyfiglet
pip install datavision
pip install nodemaster
pip install propyte
pip install git+https://github.com/wdbm/pyprel.git
pip install git+https://github.com/wdbm/pyrecon.git
pip install shijian
pip install technicolor
#pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl
#pip install git+git://github.com/google/skflow.git
echo "make Python virtual environment relocatable"
virtualenv --relocatable virtual_environment
echo "archive Python virtual environment with respect to SVN limitations"
tar -czf - virtual_environment/ | split --bytes=50MB - virtual_environment.tgz.
rm -rf virtual_environment
I've cloned a codebase from Heroku onto a new comp, when I try to run it none of the Python libraries that I've installed are present. After I run which pip I see that my path is /usr/local/bin/pip.
(1) How do I change the path so all the libraries install into my virtual env and (2) how can I install everything from my requirements.txt instead of individually install libraries.
(venv)admins-MacBook-Air:lhv-talenttracker surajkapoor$ which pip
/usr/local/bin/pip
Try looking at your venv/bin/activate file and see if the VIRTUAL_ENV matches your current path. If it doesn't match, change it to match your path and activate again.
$ cat activate |grep VIRTUAL_ENV=
VIRTUAL_ENV="/does/this/path/match?"
I am attempting to install Portia, a python app from Github: https://github.com/scrapinghub/portia
I use the following steps at the command line:
set up new virtualenv 'portia' in Mac terminal
git clone https://github.com/scrapinghub/portia.git
follow readme instructions:
cd slyd
pip install -r requirements.txt
run Portia
cd slyd
twistd -n slyd
But every time I attempt the last step to run the program, I get the following error:
ImportError: No module named scrapy
Any idea why this error is occurring? All previous steps seem to install correctly. Is it an error earlier in my install process?
Thanks!
I don't have the rep to upvote Alagappan's answer but he's correct. Also, if you're as inexperienced as I am, you may need further clarity on this.
You have to create, activate and navigate into the virtualenv before installing anything (including cloning portia from github). Here's the whole thing working from start to finish:
1: cd to wherever you’d like to store your project...
and Install virtualenv:
$ pip install virtualenv
2: Create the virtual environment. (I called mine “portia” but this can be anything.):
$ virtualenv portia
3: Activate the virtual environment you created (change the path to reflect the name you used here if not “portia”.):
$ source portia/bin/activate
At this point your terminal should have display the virtualenv name in parenthesis before the standard directory path prompt:
 (name-of-virtualenv) [your-machine]:[current-directory]: [user]$
...and if you list the files within your pwd you’ll see the name of you virtualenv there.
4: cd into your virtualenv (“portia” for me):
$ cd portia
5: Now you can clone portia from github into your virtualenv...
$ git clone https://github.com/scrapinghub/portia
6: cd into the cloned portia/slyd...
$ cd portia/slyd
7/8: pip install twisted and Scrapy...
$ pip install twisted
$ pip install Scrapy
You’re virtualenv should still be activated and you should still be in [virtualenv-name]/portia/slyd
9: Install the requirements.txt:
$ pip install -r requirements.txt
10: Run slyd:
$ twistd -n slyd
--- No more scrapy error! ---
Another Installation Method For Portia: Using Vagrant
Here is the method that made me install Portia with ease. Works with Mac, Windows and Linux. With a few commands and clicks, you'll get a fully functional web scraper.
Things Needed:
VirtualBox
Vagrant
Clone the repo for Portia or download the zip file.
Additional Steps To Take:
Install VirtualBox.
Install Vagrant
Open your terminal and navigate to where you cloned the Portia repo or where you've extracted it (in case of a zip file).
Then make a command vagrant up - This will download and setup a VirtualBox Guest VM for you + will install all the necessary requirements for Portia and will install Portia from start to finished.
After the above process, you may now open your browser and navigate to
http://the-virtualbox-ip:8000/static/main.html
And you're setup.
It's quite simple, you just need to install the python module scrapy in the same way that the Twitter API requires setuptools
pip install scrapy
I suppose the issue you are facing is because of the virtualenv. Once you setup a new virtual environment you need to run the activate script in order to start using it. In your case you'll have to run the following command:
$ source portia/bin/activate
On successful activation, your prompt will look like:
(portia) $
Can you check if you activated your virtual environment before you installed the packages using pip? I believe doing so will fix your issue.