I'm using pandas.read_gbq to pull data from Google Big Query, in an IPython notebook. However, I put my IPython notebooks in a shared Google Drive folder so that other folks can see them as well.
When I've run the read_gbq command, a new browser window opens up asking me to authenticate, which works great. But then my credentials file gets dumped in the same folder where my notebook sits, which is not good!
How can I leave my credentials file somewhere safe (local)?
Related
I have a file on my computer that I want to upload to Google Colab. I know there are numerous ways to do this, including a
from google.colab import files
uploaded = files.upload()
or just uploading manually from the file system. But I want to upload that specific file without needing to choose that file myself.
Something like:
from google.colab import files
file_path = 'path/to/the/file'
files.upload(file_path)
Is there any way to do this?
Providing a file path directly rather than clicking through the GUI for an upload requires access to your local machine's file system. However, when your run cell IPython magic commands such as %pwd in Google collab, you'll notice that the current working directory shown is that of the notebook environment - not that of your machine. The way to eschew the issue are as follows.
1. Local Runtime
Only local runtimes via Jupyter seems to enable such access to the local file system. This necessitates the installation of jupyterlab, a Jupyter server extension for using a WebSocket, and launching a local server. See this tutorial.
2. Google Drive
In case Google Drive is convenient, you can upload files into Google Drive from your local machine without clicking through a GUI.
3. Embracing the GUI
If these options seem overkill, you, unfortunately, have to stick with
from google.colab import files
uploaded = files.upload()
as you alluded to.
I am trying to set up ezsheets for the use with Google Sheets. I followed the instructions from here https://ezsheets.readthedocs.io/en/latest/ and here https://automatetheboringstuff.com/2e/chapter14/
The set up process works quite differently on my computer: Somehow I could download the credentials-sheets.json. I need to download the token-sheets.pickle and token-drive.pickle files. When I run import ezsheets, no browser window is opended as described in the set up instructions. Nothing happens.
Is there another way to download both files?
I followed the steps you referenced and managed to generate the files, but I also encountered the same issue before figuring out the cause. The problem is that there are a few possible causes and the script silently fails without telling you exactly what happened.
Here are a few suggestions:
First off you need to configure your OAuth Consent Screen. You won't be able to create the credentials without it.
Make sure that you have the right credentials file. To generate it you have to go to the Credentials page in the Cloud Console. The docs say that you need an OAuth Client ID. Make sure that you have chosen the correct app at the top.
Then you will be prompted to choose an application type. According to the docs you shared the type should be "Other", but this is no longer available so "Desktop app" is the best equivalent if you're just running a local script.
After that you can just choose a name and create the credentials. You will be prompted to download the file afterwards.
Check that the credentials-sheets.json file has that exact name.
Make sure that the credentials-sheets.json file is located in the same directory where you're running your python script file or console commands.
Check that you've enabled both the Sheets and Drive API in your GCP Project.
Python will try to setup a temporary server on http://localhost:8080/ to retrieve the pickle files. If another application is using port 8080 then it will also fail. In my case a previously failed Python script was hanging on to that port.
To find and close the processes using port 8080 you can refer to this answer for Linux/Mac or this other answer for Windows. Just make sure that the process is not something you're currently using.
I just used the single import ezsheets command to get the files so after getting the token-sheets.pickle I had to run it again to get the token-drive.pickle, but after that the library should detect that you already have the files.
I am new to google colab and i am figuring out if google colab is able to access files on my computer's cdrive directly.
import os
path = 'C:\\Users\\guest\\Desktop\\'
for file in os.listdir(path):
print(file)
The error message that come out is [Errno 2] No such file or directory: 'C:\Users\zhuan.lim\Desktop\script tools\Python Scripts\'
I searched online and some examples said to upload the files first using:
from google.colab import files
uploaded = files.upload()
However, is there another way for google colab to directly read from my drives?
Thanks in advance.
Solution
You can make Google Colab access the files on your computer essentially in three ways:
Upload files to Google Colab.
from google.colab import files
uploaded = files.upload()
Upload your files to your Google Drive account and then mount Google Drive on Colab. In my experience, this has been the most convenient method. Also, note that this allows you to both read and write to Google Drive (as if that is a local drive).
from google.colab import drive
drive.mount('/content/gdrive')
!ls ./content/gdrive
Once loaded, click on Files on the left pane to access the file-structure, as shown in the following screenshot.
Note: Alternatively, click on Files >> Mount Drive and this will insert the code-snippet to mount Google Drive into your Colab Notebook. Once you run that cell, you will see GDrive getting mounted.
Initiate a local runtime and then access it. In this case colab uses your local resources and the local files are accessible to it as well. Please do read the security concerns/warning before initiating this option. I have not personally tried it and your are on your own there.
I will explain option#3 below.
Connecting Colab to Local Runtime
Colab offers you to connect to a local runtime. If you have installed jupyter_http_over_ws as explained here you should be able to just provide the port you used to start the local runtime and connect to it from colab.
Step-1
Click on Reconnect and then select "Connect to local runtime". (Top right corner in colab).
Step-2
Click on hyperlink: these instructions, in the pop-up as shown below (in step-3), to install jupyter_http_over_ws, if not already installed.
Install and enable the jupyter_http_over_ws jupyter extension (one-time).
pip install jupyter_http_over_ws
jupyter serverextension enable --py jupyter_http_over_ws
Start server and authenticate.
New notebook servers are started normally, though you will need to set a flag to explicitly trust WebSocket connections from the Colaboratory frontend.
jupyter notebook \
--NotebookApp.allow_origin='https://colab.research.google.com' \
--port=8888 \
--NotebookApp.port_retries=0
For more details, I encourage you to see these instructions.
Step-3
Provide the correct port number (e.g. 8888) that was used to start the local runtime (jupyter notebook on your local machine).
No, there's no other way then files.upload(), because that is the way. But I think your looking for a more user friendly way of getting your files in. You could drag-on-drop your files into Google Drive, and then mount it in your Google Colab session by inserting following lines in a cell and executing it:
from google.colab import drive
drive.mount('/content/gdrive')
It will prompt you to go to a URL to authenticate yourself. After you've clicked the URL and allowed Google Colab Access to your Google Drive files, you can access your Google Drive files. More elaborate explanation here : Import data into Google Colaboratory
I am working with Azure Cloud Jupyter Notebook but i dont know how to read my data set so i need to know how to upload my csv dataset
Here's what I found in the FAQ online:
How can I upload my data and access it in a notebook?
A file can be added to the project itself from either the web or computer, or uploaded using the File Menu inside a JupyterNotebook if you chose to save under the project/ folder. Files outside the project/ folder will not be persisted. If you have multiple files that add up to over 100mb you'll need to upload them one by one.
You can also download data using the terminal or shell commands inside of a notebook from publicly accessible web sites include GitHub, Azure blob storage, nasa.gov, etc...
I'm looking for a way to automate the authentication process when connecting a colab-session to my google drive.
I'd prefer to use the built-in tools for this one, instead of PyDrive.
In short: have the following cell run without having to manually authenticate by logging in and copying the password from the dialogue
from google.colab import drive
drive.mount('/content/drive/')
Automatically mounting to your Drive files is now supported for Colab notebooks which aren't shared by multiple people.
To enable this for a notebook, create a new Drive notebook, open the file browser, and click the 'Mount Drive' button.
You'll see a permissions dialog like so:
After you complete the permissions once, you'll see your Drive mounted in the file browser.
Better still, if you reload the notebook later and reconnect, your Drive will mount automatically with no more drive.mount copy/paste required. Your Drive files will just be there.