Develop Reference

Why can't I download a dataset with the Gensim download API

When I do the below:
>>> import gensim.downloader as api
>>> model = api.load("glove-twitter-25") # load glove vectors
the gensim.downloader API throws the below error:
[Errno 2] No such file or directory:
'/Users/vtim/gensim-data/information.json'.
What am I doing wrong?

I had the same problem and I solved it in these steps. I am using mac, pycharm, and virtualenv. I don't have too much python experience but this is how I did it:
1.1 You have to create a folder named 'gensim-data' with directory '/Users/vtim/gensim-data'. This can be done by running command 'mkdir gensim-data' in your terminal (the same place where you can use pip install commands).
1.2 Then you have to add the folder to your project as a content root (so that the code can access it). From Pycharm go from the main application menu (next to Apple logo with mac) Pycharm -> Preferences and there Project -> Project Structure and from there on the right menu choose 'Add content root'. Find the gensim-data folder that you just made and add it.
1.3 Now you should see the 'gensim-data' folder in your project folder where, for example, venv (virtualenv) is also if you are using it. Now create a file to the 'gensim-data' folder named as 'information.json'. Then copy the code found from this link to the 'information.json' file: https://github.com/RaRe-Technologies/gensim-data/blob/master/list.json
(The problem that you have is that gensim.downloader api may not have access to write documents to the specific directory or it can not read them. In my case it couldn't do either.)
If your code is still not working, you should do the next step:
2.1 In my case I had also a problem that the api could not access files the right files from internet. This problem is solved here: https://stackoverflow.com/a/42098127/14075343 . So find the folder/application named Python 3.8 (if you are using 3.8 version) from your computer, open it and double click 'Install Certificates.command'. Or you can try to run from terminal 'open /Applications/Python\ 3.8/Install\ Certificates.command'
Now the code should work. If it still doesn't you can try to run these codes. I am not sure if it makes a difference but I run these on the way I found the solution:
sudo python3 -m pip install --upgrade gensim
sudo -H pip install virtualenv
sudo chown -R $USERNAME /Users/$USERNAME/Library/Caches/pip

I had both the issues 'information.json' related as well as the certificate one and was able to resolve it by following the steps above. As a tip you can also try testing it in command line by doing
python3 -m gensim.downloader -i word2vec-google-news-300
replace word2vec-google-news-300 with the dataset that you want to download in
https://github.com/RaRe-Technologies/gensim-data/blob/master/list.json

pip in venv on Windows 10: 'Fatal error in launcher: Unable to create process using '"c:\users\..." ' encountered during flask tutorial

I'm going through the Flask tutorial, and I encounter a problem where I can't use pip (and thus anything else really) a venv in Powershell. The same flask app runs correctly in an Ubuntu terminal with Python 3.6.6 in WSL.
The problem seems to depend on the directory, which makes me think it's somehow related to file path length; I enabled long file paths in the windows Group Editor but this hasn't fixed the problem. In the below steps my venv directory is c:\users\rwgpu\google-drive\code\flask-tutorial\winEnvflaskr\ and I experience the error, but everything works correctly in the directory C:\python\test\testVenv. (in each case I'm running the commands in the directory one up from the listed above, flask-tutorial and test respectively)
Minimal steps to reproduce are:
Completely fresh Python 3.7 installation.
Change directory to app folder.
py -m venv winEnvFlaskr
./winEnvFlaskr/Scripts/activate
pip list
The ultimate goal is to then
pip install FLask
and run the Flask tutorial app. Again, all steps work correctly in Bash on Ubuntu in WSL (running its own Python) and in a different windows directory with the same Python 3.7.
After pip list I get the error:
Fatal error in launcher: Unable to create process using '"c:\users\rwgpu\google-drive\code\flask-tutorial\winenvflaskr\scripts\python.exe" "C:\Users\rwgpu\Google-Drive\Code\flask-tutorial\winEnvFlaskr\Scripts\pip.exe" list'
and if I try
python -m pip list
I get nothing; the terminal hangs for a second and returns with no output. If I run
py -m pip install -U pip
in the bugged venv it will try to install and report success. It will do this again, and will never report "requirement already satisfied" which would be correct.

I had the same problem on Windows with running flask in command line from venv(for example, "(venv)...\flask run"). I resolved the problem with changing path in flask.exe code(open "your_venv\Scripts\flask.exe" with notepad or etc.): in my way I correct the 436-th line at the end; you need to put there "your_absolute_path_to_venv\Scripts\python.exe" instead of the path indicated there.
Good luck!

Well, I still don't know why this was happening -- it persisted through deleting and recreating the venv (obviously), but when I deleted the entire folder, that somehow cleared it up. I just copied the code into a new directory in the same parent and everything seems to be working ¯\_(ツ)_/¯
If anyone knows what would cause this in my case I still welcome input.

Copy Python App to a new Machine?

I have a Python app running on windows that has imports for the following packages:
requests
json
psycopg2
I copy the entire project (I used Pycharms to write the app and import the packages) to a new machine and expected it would work. The new machine is also windows and I'm trying to run my script from the command line (i.e. no Pycharm on the new machine).
Instead, I get an error saying "ModuleNotFoundError: No module named 'requests'"
If I look at the project, I have the directories:
venv
Lib
site-packages
requests
What am I missing/doing wrong?

You have a couple of options here but first the problem. You are exporting your code base to a new machine without the required Modules installed on that machine and/or within your Python project's environment. After you have python installed on your new machine, you need to be sure to point your PyCharm Project to the proper environment.
File > Default Preferences > Project Interpreter
The window that appears on the right will contain a drop down menu labeled Project Interpreter. If you click on the drop down, it should reveal a list of the available Python environments on your machine.
Based on your description of your site-packages directory I would assume you do not have your interpreter pointed the proper environment on your new machine. With that said, you would be better served creating a new virtual python environment on your machine and installing each relevant dependency within that environment.
Take a look at this post here for your first best option on re-creating your old python environment on your new machine.
EDIT: I apologize for not reading the question more thoroughly before answering the questions. If this is running on a Windows machine you will need to double check the environment path python is using. It is very easy to install python at a different PATH than the command line environment is checking on a Windows box. If for example your PATH is pointing to a different version of Python and PIP is installing packages somewhere else this issue can occur. Double check your System PATH for python and which version the command line is running.

On the new machine you must source venv/bin/activate so your path environment variables are set properly. In particular, which python should say venv/bin/python rather than /usr/bin/python. Also, take care to conda env update or pip install -r requirements.txt so you'll have suitable venv libraries on the new machine.

Error checking for subterfuge update on Ubuntu

I am trying to run subterfuge on ubuntu. I downloaded and installed the latest software but when I try to run it I get this Message
Subterfuge courtesy of r00t0v3rr1d3 & 0sm0s1z
Checking for updates. You can disable this feature through the settings page.
sh: 1: /usr/share/subterfuge/manage.py: not found

I'm going to go out on a limb and guess that the file /usr/share/subterfuge/manage.py doesn't exist. This is likely because you installed subterfuge elsewhere on your machine.
Their source code has a number of hard-coded paths (for whatever reason) that require a specific installation directory.
I would copy your installation over to the directory in which they expect it to be
sudo cp -r /old/installation_dir/subterfuge/* /usr/share/subterfuge

Copy complete virtualenv to another pc

I have a virtualenv located at /home/user/virtualenvs/Environment. Now I need this environment at another PC. So I installed virtualenv-clone and used it to clone /Environment. Then I copied it to the other PC via USB. I can activate it with source activate, but when I try to start the python interpreter with sudo ./Environment/bin/python I get
./bin/python: 1: ./bin/python: Syntax Error: "(" unexpected
Executing it without sudo gives me an error telling me that there is an error in the binaries format.
But how can this be? I just copied it. Or is there a better way to do this? I can not just use pip freeze because there are some packages in /Environment/lib/python2.7/site-packages/ which I wrote myself and I need to copy them, too. As I understand it pip freeze just creates a list of packages which pip then downloads and installs.

Do the following steps on the source machine:
workon [environment_name]
pip freeze > requirements.txt
copy requirements.txt to other PC
On the other PC:
create a virtual environment using mkvirtualenv [environment_name]
workon [environment_name]
pip install -r requirements.txt
You should be done.
Other Resources:
How to Copy/Clone a Virtual Environment from Server to Local Machine

Pip Freeze Not Applicable For You?
Scenario: you have libraries installed on your current system that are very hard to migrate using pip freeze and am talking really hard, because you have to download and install the wheels manually such as gdal, fiona, rasterio, and then even doing so still causes the project to crash because possibly they were installed in the wrong order or the dependencies were wrong and so on.
This is the experience I had when I was brought on board a project.
For such a case, when you finally get the environment right you basically don't want to go through the same hell again when you move your project to a new machine. Which I did, multiple times. Until finally I found a solution.
Now, disclaimer before I move on:
I don't advocate for this method as the best, but it was the best for my case at the time.
I also cannot guarantee it will work when switching between different OSes as I have only tried it between Windows machine. In fact I don't expect it to work when you move from Windows to other OSs as the structure of the virtualenv folder from Unix-based OS is different from that of Windows.
Finally, the best way to do all of this is to use Docker. My plan is to eventually do so. I have just never used Docker for a non-web-app project before and I needed a quick fix as my computer broke down and the project could not be delayed. I will update this thread when I can once I apply Docker to the project.
THE HACK
So this is what I did:
Install the same base Python on your new machine. If you have 3.9 on the old, install 3.9 on the new one and so on. Keep note of where the executable can be located, usually something like C:\Users\User\Appdata\Local\Programs\Python\PythonXX
Compress your virtual env folder, copy it into the project directory
inside your new machine. Extract all files there
Using text editor of your choice, or preferably IDE, use the 'Search
in all files' feature to look for all occurrences of references to
your old machine paths: C:\Users*your-old-username*
Replace these with your new references. For my case I had to do it in
the following files inside the virtual env folder: pyvenv.cfg, Scripts/activate, Scripts/activate.bat, Scripts/activate.fish and Scripts/activate.nu.
And that's it!
Good luck everyone.

I think what occurs is that you just copy the symbolic links in the source file to the target machine as binary files(no longer links). You should copy it using rsync -l to copy to keep those links.

Usually I use virtualenv to create a new environment, then I go to the environment where I want to copy from, copy all the folders and paste it into the environment folder I just created, but most importantly when asking if you want to replace the Destination files, choose to skip these files. This way you keep your settings.
At least for me, this has worked very well.
I hope it works for you too.

I share my experience.
Suppose another PC does not install Python
Python version: 3.7.3
Platform: Platform: Windows 10, 7 (64bit)
The following is working for me.
Step:
download Windows embeddable zip file
download get-pip.py (because the embeddable zip file does not provide pip)
[Optional] install tkinter, see this article: Python embeddable zip: install Tkinter
Choose a packaging method (I use NSIS: https://nsis.sourceforge.io/Download)
folder artictures:
- main.nsi
- InstallData
- contains: Step1 & Step2
- YourSitePackages # I mean that packages you do not intend to publish to PyPI.
- LICENSE
- MANIFEST.in
- README.rst
- ...
- requirements.txt
- setup.py
The abbreviated content about main.nsi is as follows:
!define InstallDirPath "$PROGRAMFILES\ENV_PYTHON37_X64"
!define EnvScriptsPath "${InstallDirPath}\Scripts"
...
CreateDirectory "${InstallDirPath}" # Make sure the directory exists before the writing of Uninstaller. Otherwise, it may not write correctly!
SetOutPath "${InstallDirPath}"
SetOverwrite on
File /nonfatal /r "InstallData\*.*"
SetOutPath "${InstallDirPath}\temp"
SetOverwrite on
File /nonfatal /r "YourSitePackages\*.*"
nsExec::ExecToStack '"${InstallDirPath}\python.exe" "${InstallDirPath}\get-pip.py"' # install pip
nsExec::ExecToStack '"${InstallDirPath}\Scripts\pip.exe" install "${InstallDirPath}\temp\."' # install you library. same as: `pip install .`
RMDir /r "${InstallDirPath}\temp" # remove source folder.
...
/*
Push ${EnvScriptsPath} # Be Careful about the length of the HKLM.Path. it is recommended to write it to the HKCU.Path, it is difficult for the user path to exceed the length limit
Call AddToPath # https://nsis.sourceforge.io/Path_Manipulation
*/
hope someone will benefit from this.