I am looking for a way to access an .csv document that I have registered on drive to perform data analysis. The idea would be to have something similar as pandas' read_csv but to access a remote file, not one registered locally. Note that I don't want to access a Google spreadsheet document : it's a .csv document that I have shared on Google drive. Ideally, I'd like to be able to save it on Drive as well.
Thank you for the help,
Best,
You will want to use Google's File Stream to do this. What it does is basically mount the drive to your computer so that you can access it from anywhere.
So on my windows computer I can open a terminal and then access anything on my drive. (Or if you have a mac you will find it mounted to /Volumes)
>>>ls /mnt/g/
$RECYCLE.BIN My Drive Team Drives
>>>ls /mnt/g/My\ Drive/
test.csv
Related
I am working on a machine learning task and have saved a Keras model and want to deploy it to Github (so that I can host a web demo using Streamlit and/or Flask). However, the model file is so large (> 1 GB), that I cannot upload it to Github for free.
My thought process regarding an alternative is to upload it to a cloud service such as google drive (or dropbox, box etc.) then using some sort of Python module to access it from there.
So my question is, can I upload a pickle file containing a pickled Keras model to Google Drive and then access that object from a Python script? If so, how would I go about doing so?
Thank you!
I believe you can, you'll need to pip oauth2client & gspread. To access the data you would need to enable API manager on your google drive and get credentials in the form of a json file. Then you would need to share the file with the email in the credentials giving it permission. You could then port over the information as you needed to, I'm not sure how Keras works but this would be the first step.
Another important factor is that Google api is very touch when it comes to requests that are coming to fast, to overcome this put in sleep commands between each one, but if you do that this method may become way to slow for your idea.
scope = ["https://spreadsheets.google.com/feeds", 'https://www.googleapis.com/auth/spreadsheets',
"https://www.googleapis.com/auth/drive.file", "https://www.googleapis.com/auth/drive"]
creds = ServiceAccountCredentials.from_json_keyfile_name("Your json file here.json", scope)
client = gspread.authorize(creds)
sheet = client.open("your google sheets name or whatever").sheet1 # Open the spreadhseet
data = sheet.get_all_records() # you can call all the information with this.
I understand that you require a way to upload and download large files* from Drive using Python. If I understood your situation correctly, then you can achieve your goals easily by using the Drive API as #TimothyChen commented. First I highly recommend you to follow the Drive API Python Quickstart tutorial to create a working example. Later, you could modify it to use Files.create() and Files.get() to upload/download files as needed. Don't hesitate to ask me more questions if you have doubts.
*Please, keep in mind that there is a 5 TB size limit in Drive.
I am trying to use tf.keras.utils.get_file("URL from google drive")
When I use URL which has less than 33MB it works well
However, when I try to download file more than 33MB it's not working well.
How can I solve this problem?
_URL = 'URL FROM GOOGLE DRIVE'
path_to_zip = tf.keras.utils.get_file("file_name.zip", origin=_URL, extract=True)
PATH = os.path.join(os.path.dirname(path_to_zip), 'art_filename')
I am following https://www.tensorflow.org/tutorials/images/classification
this for my practice, and I am trying to use my own data for the practice.
In this example, it uses URL as "storage.googleapi.com..." and has large amount of data.
I want to use this code for downloading large data from google drive.
Is there anyway to solve this problem?
I also tried google mounting but since I want to access the folders and files,
I am not used to do with google mounting.
Thanks
Files that are above a certain size pop-up with a notification from Drive letting you know that it cannot be scanned for viruses which needs to be accepted before the file can download. By appending "&confirm=t" to the end of the download URL, you can bypass that message and download your files.
Data is in MS Access and it's in one of the shared drive on the network. I need this data in azure blob storage as CSV files. Can anyone please suggest me how can this be possible?
You can move data to Azure Blob storage in several ways, You could use either Azcopy: located here: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10 , Or Storage Explorer(GUI): https://azure.microsoft.com/en-us/features/storage-explorer/
OR using Python SDK:
block_blob_service.create_blob_from_path(container, file, file)
Python SDK can be found here: https://github.com/Azure/azure-sdk-for-python
When it comes to changing the format from Access to CSV, it's something not related to Azure Storage, you can try existing libraries for that conversion, then upload to blob storage.
How can I download a complete folder hierarchy using python Google drive api. Is it that I have query each files in the folder and then download that. But doing this way, folder hierarchy will be lost. Any way to make it in proper way. Thanks
You can achieve this using Google GAM.
gam all users show filelist >filelist.csv
gam all users show filetree >filetree.csv
I got all the answers from this site. I found it very useful.
https://github.com/jay0lee/GAM/wiki
Default max results of query is 100. Must use pageToken/nextPageToken to repeat it.
see Python Google Drive API - list the entire drive file tree
How do I delete any file from Drive using Python's Google Drive API SDK?
I want to sync my folder with google drive, such that, whenever I delete any file from my local machine, the same file which is uploaded on the drive with same name, should be deleted.
I went through : https://developers.google.com/drive/v2/reference/files/delete
But then, from where do I get fileid?
Any help would be appreciated.
Thanks in advance...
You need to read and understand https://developers.google.com/drive/v2/reference/files#resource and https://developers.google.com/drive/search-parameters and https://developers.google.com/drive/v2/reference/files/list
At the bottom of the last page is a Try It Now feature which you can use to play with the Drive SDK BEFORE you write a single line of code. Do the same with https://developers.google.com/drive/v2/reference/files/delete
Once you understand them, you will know how to trash or delete files from Drive. Personally I prefer trash as it's easier to undo my mistakes during testing. #martineau Don't worry too much about the disk space; Google isn't about to run out of disk :-)
The only catch to using Trash is you need to remember to qualify any queries with 'trashed=false' and users will need to empty Trash if ever they hit quota.