I'm trying to follow the fine tuning guide for Openai here.
I ran:
pip install --upgrade openai
Which install without any errors.
But even after restarting my terminal, i still get
zsh: command not found: openai
Here is the output of echo $PATH:
/bin:/usr/bin:/usr/local/bin:/Users/nickrose/Downloads/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
Here is the output of which python:
/usr/bin/python
Any tips for how to fix this? I'm on MacOS Big Sur 11.6.
Basically pip installs the packages under its related python directory, in a directory called site-packages (most likely, I'm not a python expert tbh). This is not included in the path you provided. First, ask pip to show the location to the package:
pip show openai
The output would be something like this:
Name: openai
Version: 0.22.0
Summary: Python client library for the OpenAI API
Home-page: https://github.com/openai/openai-python
Author: OpenAI
Author-email: support#openai.com
License:
Location: /Users/<USER>/DIR/TO/SOME/PYTHON/site-packages
Requires: numpy, openpyxl, pandas, pandas-stubs, requests, tqdm
Required-by:
So your package will be available in
/Users/<USER>/DIR/TO/SOME/PYTHON/site-packages/openai
Either add /Users/<USER>/DIR/TO/SOME/PYTHON/site-packages/ to your path, or use the complete address to your package, or try to access it using your python:
python -m openai # -m stands for module
To get more information about the -m flag, run python --help.
Update
So as you mentioned in the comments, you get permission denied after you add the directory to your package. This actually means that the package exists, but it's not permitted by your OS to execute. This is the thing you have to do, locate your package, and then:
sudo chmod +x /PATH/TO/script
And the reason you're getting command not found after you use sudo directly with the package, is that you update your path variable in zsh, but when you use sudo, superuser uses sh instead of zsh.
This doesn't answer the question directly but specifies an alternative if you only want to prepare the data set and create the new model for finetunning. It doesn't matter which system you have.
After a lot of struggle I decided it was not worth the hassel to run the cli on my specific machine because of so many different configurations and the mess. My end goal was just to create a model and upload it to OpenAI.
So if someone else stumbles on this post, just use Google Colab. I have also shared one of mine with steps to follow in here.
In case the links don't work in the future I'll list the steps here below as well:
(Step 1)
Set your API key (The already added api key is fake so please replace it with your own):
%env OPENAI_API_KEY=sk-Kz8Weh1234ddgYBmsdfinsdf7ndsfg55532432
(Step 2)
Install the openai package with pip like the following:
!pip install -Uq openai
(Step 3)
Import the openai package like the following:
import openai
(Step 4)
Make sure to upload the promptdata.csv file in the Google Colab folders.
The way to do it is:
On the right side you'll see a Hamburger Menu icon click on it.
You'll see the "Table of Contents"
Click on the last folder icon on the top. If you hover on the icon it says "Files".
Now you'll see a folder called "sample_data".
Click on the three dots menu for "sample_data" and then select "upload".
You should be able to upload your csv file
It is not mandatory to upload a csv file. You can also upload any type of TSV, XLSX, JSON or JSONL file as listed by the OpenAI documentation here. But it will always be converted to JSONL file after runnning the below command.
Once you're done uploading the file you can run the below command to prepare your data set which will return you a new JSONL file at the same location where the original file was with all the corrections the tool provides.
!openai tools fine_tunes.prepare_data -f "/content/sample_data/promptdata.csv"
(Step 5)
Run the below command once again after the corrections and it will most likely say "No remediations found".
!openai tools fine_tunes.prepare_data -f "/content/sample_data/promptdata_prepared.jsonl"
(Step 6)
Finally run the below command using the file promptdata_prepared.jsonl and create a model.
!openai api fine_tunes.create -t "/content/sample_data/promptdata_prepared.jsonl"
(Step 7)
Once the model is created note the name of the "Uploaded model"
So what happens is that after installing the package there are no actual executables available. That's why you get the error message when you try to execute for example:
openai --help
What i managed to find is that the actual parsing of the commands is done in
/Users/<USER>/DIR_TO_PYTHON/site-packages/openai/_openai_scripts.py
That's just a python script which by default is not executable, so you have to make a workaround of which I find the easiest is creating an executable which basically calls it with the given arguments. Below are the steps which I've done to make it work on "macOS Monterey 12.0.1"
Locate the "openai" package which should be in
/Users/<USER>/DIR_TO_PYTHON/site-packages/
Make sure you are in the "openai" package folder and run
sudo vim /bin/openai
That should create a new file, put in the following command and make sure the path to the file is correct
python3 /Users/<USER>/DIR_TO_PYTHON/site-packages/openai/_openai_scripts.py $#
$# is for the params that you pass when you call the executable
After saving the file the next step is making it executable which is done with
chmod +x /bin/openai
Last step is adding it to the PATH which is done by adding the file path in /etc/paths and after restarting the terminal, you should have fully working openai command globally
I was facing similar issue. It might due to global python in your machine is not maching with the pip installation path and it might be installing in some other python folder like in 3.9 and you have 3.10 python version globally set in your Mac.
First install fresh python using homebrew
brew install python
It will install the latest python into your machine. Then try to install openai again using
pip3 install openai
OR using pip (you can try installing using both and see which works as per your system config)
pip install openai
Now
ENJOY a cup of coffee ;)
Related
When I do the below:
>>> import gensim.downloader as api
>>> model = api.load("glove-twitter-25") # load glove vectors
the gensim.downloader API throws the below error:
[Errno 2] No such file or directory:
'/Users/vtim/gensim-data/information.json'.
What am I doing wrong?
I had the same problem and I solved it in these steps. I am using mac, pycharm, and virtualenv. I don't have too much python experience but this is how I did it:
1.1 You have to create a folder named 'gensim-data' with directory '/Users/vtim/gensim-data'. This can be done by running command 'mkdir gensim-data' in your terminal (the same place where you can use pip install commands).
1.2 Then you have to add the folder to your project as a content root (so that the code can access it). From Pycharm go from the main application menu (next to Apple logo with mac) Pycharm -> Preferences and there Project -> Project Structure and from there on the right menu choose 'Add content root'. Find the gensim-data folder that you just made and add it.
1.3 Now you should see the 'gensim-data' folder in your project folder where, for example, venv (virtualenv) is also if you are using it. Now create a file to the 'gensim-data' folder named as 'information.json'. Then copy the code found from this link to the 'information.json' file: https://github.com/RaRe-Technologies/gensim-data/blob/master/list.json
(The problem that you have is that gensim.downloader api may not have access to write documents to the specific directory or it can not read them. In my case it couldn't do either.)
If your code is still not working, you should do the next step:
2.1 In my case I had also a problem that the api could not access files the right files from internet. This problem is solved here: https://stackoverflow.com/a/42098127/14075343 . So find the folder/application named Python 3.8 (if you are using 3.8 version) from your computer, open it and double click 'Install Certificates.command'. Or you can try to run from terminal 'open /Applications/Python\ 3.8/Install\ Certificates.command'
Now the code should work. If it still doesn't you can try to run these codes. I am not sure if it makes a difference but I run these on the way I found the solution:
sudo python3 -m pip install --upgrade gensim
sudo -H pip install virtualenv
sudo chown -R $USERNAME /Users/$USERNAME/Library/Caches/pip
I had both the issues 'information.json' related as well as the certificate one and was able to resolve it by following the steps above. As a tip you can also try testing it in command line by doing
python3 -m gensim.downloader -i word2vec-google-news-300
replace word2vec-google-news-300 with the dataset that you want to download in
https://github.com/RaRe-Technologies/gensim-data/blob/master/list.json
I'm trying to run python3 script in command prompt Windows(cmd) but facing some issue if python script contains xlwings package. I'm using xlwings package to read and write the information that I needed in excel file. I had go through some research but all of the solution is pointed to run python from excel (meaning call python script in vba) and I don't want to do that. Here is the sample of the python code for testing purpose.
import xlwings as xw
bookName= r'C:\Users\Desktop\Python_Projects\Test.xlsm'
wb= xw.Book(bookName)#specified full name of excel file
sht = wb.sheets('Sheet1')
app= xw.apps.active #need to kill excel apps
sht.api.Cells(1,1).Value="test"
sht.api.Cells(1,1).Font.Bold = True
wb.save(bookName)
wb.close()
app.kill() #or del app
I'm trying to run the script and hit this issue:
I'm running the code from my IDE Pycharm, no issue and able to run. Note: I don't have admin right permission in my Windows.
Addition:
I had try to search and find the post similar with my environment(using Pycharm-not run via terminal, setup as a virtual environment) but different issue facing. I'm not very understand the answer in this post. Thus, not sure whether the answer is fixing my current issue or not. If the solution is same, hope that someone can describe it further details. Here is the link:
Python script works in PyCharm but throws path errors in windows cmd
Moving comments down to answer...
Sounds like PyCharm had been setup to use a virtualenv, so you found a link where you found that you need to activate the virtualenv
From there, that's where any PyCharm ran a pip install into, and from which you can run python and try to import any modules.
If you don't activate the environment, you would need to run your scripts by giving the full path to the Python executable - C:\path\to\venv\python script.py
To backup a list of installed modules from a virtualenv, you can do pip freeze > requirements.txt
Then pip install -r requirements.txt will restore those into any freshly created virtualenv
I'm trying to install python and 1 pypi module programmatically via a batch script, to allow me to run a script I wrote that needs that module, on several PCs. A batch script will make it easier to install. I have tried
\\server\share\python-3.6.1.exe /quiet InstallAllUsers=1 Include_pip=1 Include_test=0 PrependPath=1
python -m pip install utm
pause
Without success. When run, this throws the error ‘python’ is not recognized as an internal or external command. I realize this means the system path variables have not been updated correctly, but I thought PrependPath=1 was setting these variables correctly. Any help anyone could provide would be appreciated.
PrependPath will add to the path, you have to re-read the path again in order for the command to work.
You can re-read the path by using RefreshEnv.cmd from chocolatey, or simply customize the path for the installation, and then pass the full path to the Python executable, like this:
\\server\share\python-3.6.1.exe /quiet InstallAllUsers=1 TargetDir=%ProgramFiles%\Python3.6 Include_pip=1 Include_test=0 PrependPath=1
%ProgramFiles%\Python3.6\python.exe -m pip install utm
I have a virtualenv located at /home/user/virtualenvs/Environment. Now I need this environment at another PC. So I installed virtualenv-clone and used it to clone /Environment. Then I copied it to the other PC via USB. I can activate it with source activate, but when I try to start the python interpreter with sudo ./Environment/bin/python I get
./bin/python: 1: ./bin/python: Syntax Error: "(" unexpected
Executing it without sudo gives me an error telling me that there is an error in the binaries format.
But how can this be? I just copied it. Or is there a better way to do this? I can not just use pip freeze because there are some packages in /Environment/lib/python2.7/site-packages/ which I wrote myself and I need to copy them, too. As I understand it pip freeze just creates a list of packages which pip then downloads and installs.
Do the following steps on the source machine:
workon [environment_name]
pip freeze > requirements.txt
copy requirements.txt to other PC
On the other PC:
create a virtual environment using mkvirtualenv [environment_name]
workon [environment_name]
pip install -r requirements.txt
You should be done.
Other Resources:
How to Copy/Clone a Virtual Environment from Server to Local Machine
Pip Freeze Not Applicable For You?
Scenario: you have libraries installed on your current system that are very hard to migrate using pip freeze and am talking really hard, because you have to download and install the wheels manually such as gdal, fiona, rasterio, and then even doing so still causes the project to crash because possibly they were installed in the wrong order or the dependencies were wrong and so on.
This is the experience I had when I was brought on board a project.
For such a case, when you finally get the environment right you basically don't want to go through the same hell again when you move your project to a new machine. Which I did, multiple times. Until finally I found a solution.
Now, disclaimer before I move on:
I don't advocate for this method as the best, but it was the best for my case at the time.
I also cannot guarantee it will work when switching between different OSes as I have only tried it between Windows machine. In fact I don't expect it to work when you move from Windows to other OSs as the structure of the virtualenv folder from Unix-based OS is different from that of Windows.
Finally, the best way to do all of this is to use Docker. My plan is to eventually do so. I have just never used Docker for a non-web-app project before and I needed a quick fix as my computer broke down and the project could not be delayed. I will update this thread when I can once I apply Docker to the project.
THE HACK
So this is what I did:
Install the same base Python on your new machine. If you have 3.9 on the old, install 3.9 on the new one and so on. Keep note of where the executable can be located, usually something like C:\Users\User\Appdata\Local\Programs\Python\PythonXX
Compress your virtual env folder, copy it into the project directory
inside your new machine. Extract all files there
Using text editor of your choice, or preferably IDE, use the 'Search
in all files' feature to look for all occurrences of references to
your old machine paths: C:\Users*your-old-username*
Replace these with your new references. For my case I had to do it in
the following files inside the virtual env folder: pyvenv.cfg, Scripts/activate, Scripts/activate.bat, Scripts/activate.fish and Scripts/activate.nu.
And that's it!
Good luck everyone.
I think what occurs is that you just copy the symbolic links in the source file to the target machine as binary files(no longer links). You should copy it using rsync -l to copy to keep those links.
Usually I use virtualenv to create a new environment, then I go to the environment where I want to copy from, copy all the folders and paste it into the environment folder I just created, but most importantly when asking if you want to replace the Destination files, choose to skip these files. This way you keep your settings.
At least for me, this has worked very well.
I hope it works for you too.
I share my experience.
Suppose another PC does not install Python
Python version: 3.7.3
Platform: Platform: Windows 10, 7 (64bit)
The following is working for me.
Step:
download Windows embeddable zip file
download get-pip.py (because the embeddable zip file does not provide pip)
[Optional] install tkinter, see this article: Python embeddable zip: install Tkinter
Choose a packaging method (I use NSIS: https://nsis.sourceforge.io/Download)
folder artictures:
- main.nsi
- InstallData
- contains: Step1 & Step2
- YourSitePackages # I mean that packages you do not intend to publish to PyPI.
- LICENSE
- MANIFEST.in
- README.rst
- ...
- requirements.txt
- setup.py
The abbreviated content about main.nsi is as follows:
!define InstallDirPath "$PROGRAMFILES\ENV_PYTHON37_X64"
!define EnvScriptsPath "${InstallDirPath}\Scripts"
...
CreateDirectory "${InstallDirPath}" # Make sure the directory exists before the writing of Uninstaller. Otherwise, it may not write correctly!
SetOutPath "${InstallDirPath}"
SetOverwrite on
File /nonfatal /r "InstallData\*.*"
SetOutPath "${InstallDirPath}\temp"
SetOverwrite on
File /nonfatal /r "YourSitePackages\*.*"
nsExec::ExecToStack '"${InstallDirPath}\python.exe" "${InstallDirPath}\get-pip.py"' # install pip
nsExec::ExecToStack '"${InstallDirPath}\Scripts\pip.exe" install "${InstallDirPath}\temp\."' # install you library. same as: `pip install .`
RMDir /r "${InstallDirPath}\temp" # remove source folder.
...
/*
Push ${EnvScriptsPath} # Be Careful about the length of the HKLM.Path. it is recommended to write it to the HKCU.Path, it is difficult for the user path to exceed the length limit
Call AddToPath # https://nsis.sourceforge.io/Path_Manipulation
*/
hope someone will benefit from this.
ERROR: Python 3 is not supported by the Google Cloud SDK. Please use a Python 2.x version that is 2.6 or greater.
If you have a compatible Python interpreter installed, you can use it by setting the CLOUDSDK_PYTHON environment variable to point to it.
I guess the first question we should be asking is "with all the money google makes off of their customers why can't they hire someone to ensure that their cloud sdk works with python 3?"
How exactly to overcome this error on linux? What specific files need to be edited? and where should those files be located?
I searched around, a lot, and found this question about how to fix this on Windows, but the answer is not really that comprehensive.
Thus far I've attempted:
One source of documentationsays to modify a file called app.yaml, but I searched using the command find . -name "app.yaml" and no such file exists.
Specifically I'm using arch linux, I originally tried to use the AUR package but it's disfunctional.
So I installed from the documentation, making sure to edit the ./install.sh file, specifying python2 as per this discussion on the google groups, that doesn't work either. after running the command gcloud auth login I get the same error as posted above.
This is a very easy thing to solve. The native python command on the Arch command line is actually for Python 3. The SDK requires Python2.7 and the
Just go to the google-cloud-sdk folder and open the install.sh file.
Change the CLOUDSDK_PYTHON="python" value to CLOUDSDK_PYTHON="python2.7"
Rerun the install with the command ./install.sh in the same folder and follow the prompts.
That's all.
I had the same issue so I did a little change in the dev_appserver.py. This file is in the following path :
google-cloud-sdk/bin
change the shebang from /usr/bin/env python to /usr/bin/env python2
I see this almost every time I update gcloud SDK, especially when running dev_appserver.py <my app config yaml file>
I found that setting the CLOUDSDK_PYTHON env variable to 'python2' seems to silence the error. E.g on macOS:
export CLOUDSDK_PYTHON=python2
Not sure why they simply cannot make this dev server compatible with Python 3 already