This question already has answers here:
Python: download files from google drive using url
(12 answers)
Closed 2 years ago.
I Have an dataset folder of size 690Mo in my google drive, I would to copy the hole dataset on my google colab notebook to train my model, but the process of copying is very long, so how I can download the folder from Google drive with use of python script ?
Maybe, you have too many files on the root directory of Google Drive or in the dataset folder.
If you have too many files and folders in root directory, then you
should clean up and sort it in less folders.
If you have many files in dataset folder, then try the following
solutions:
Make a compressed file of your dataset folder and save it to drive. Then while running copy that compressed file (will take less than a
minute for 690Mb) and extract it in Colab.
Upload your dataset to any other platform (say OneDrive, Mega, etc.), get the link, and download on Colab using that link.
Related
I have a classifying model, and I have nearly finished turning it into a streamlit app.
I have the embeddings and model on dropbox. I have successfully imported the embeddings as it is one file.
However the call for AutoTokenizer.from_pretrained() takes a folder path for various files, rather than a particular file. Folder contains these files:
config.json
special_tokens_map.json
tokenizer_config.json
tokenizer.json
When using the tool locally, I would direct the function to the folder and it would work.
However I am unable to direct it to the folder on DropBox, and I cannot download a folder from DropBox into Python, only a file (as far as I can see).
Is there a way of creating a temp folder on Python or downloading all the files individually and then running AutoTokenizer.from_pretrained() with all the files?
To get around this, I uploaded the model to HuggingFace so I could use it there.
I.e.
tokenizer = AutoTokenizer.from_pretrained("ScoutEU/MyModel")
I'm trying to read a bunch of Google Docs files into Google collab to work with some text data.
It can't seem to read in the '.gdoc' file format, only the .txt file format.
Do I have to save all of them as .txt files first? Is there an efficient way to do this in python? Or is it possible to work with .gdoc files?
Thanks for any help!
Hi I have been stuck in same problem then the following worked for me.
Go to Drive folder where all gdocs are present.
Now simply right click on it and download whole folder.
Google Drive automatically convert all gdocs to docx during that operation.
Upload it on colab or use them locally.
I am newbie in ML and Deep Learning and am currently working on Jupyter notebook.
I have an image dataset in the form of a zip file containing nearly 28000 images downloaded on my desktop.
However, I am not being able to find any code that will make Jupyter notebook unzip the file and read it, so I'm able to work with it and develop a model.
Any help is appreciated!
Are these threads from the past helpful?
How to unzip files into a directory in Python
How to unzip files into memory in Python (you may encounter issues with this approach if the images are too large to fit in Jupyter's allocated memory)
How to read images from a directory in Python
If you go the directory route, a friendly reminder that you'll need to update the code in each example to match your directory structure. E.g. if you want to save images to and read images from a directory called "image_data", then change the code examples to unzip files into that directory and read images from that directory.
Lastly, for finding out what files are in a directory (so that you can read them into your program), see this thread answering the same question.
i am trying to train YOLOv3 with custom dataset on Google Colab. I uploaded my folders, weights etc. When I run my train.py, I get path error. I run the code like this:
!python3 "drive/TrainYourOwnYolo/2_Training/Train_YOLO.py"
The error says,
content/TrainYourOwnYolo/2_Training/dataset/img20.jpg is not found.
As I understand on Colab, my all folders are under drive folder. I don't understand why yolo is trying to find my dataset under content folder. Do you have any idea?
As it seems, you have uploaded your data to /drive/TrainYourOwnYolo/, and not to /content/TrainYourOwnYolo/, where your script is looking.
The /content folder is normally used by Colab when saving, in case you don't use Google Drive. But you have mounted your Google Drive under /drive, so your notebook unsurprisingly fails to find the files.
You should change the file paths in your Train_YOLO.py" script to replace references to /content with /drive.
If this is not possible, you can find the /content folder on the file catalogue on the left of your Colab notebook:
and by right-clicling on it, you'll see an option for uploading files there.
I have this very weird problem. I have searched across internet, read documentation but am not able to figure out how to do it. So what I want to do is train a classifier using Colab. And for that I have a image dataset of dogs on my local machine.
So what I did was I packed that dataset folder of images into a zip file and uploaded it onto Drive. Then from Colab I mounted the drive and from there I tried to unzip the files. Everything good. But I've realised that after sometime some of the extracted files get deleted. And thing is that those files aren't on Colab storage, but instead on Drive and I dunno why they are getting deleted after sometime. Like about an hour.
So far I've used the following commands to do the extraction -
from google.colab import drive
drive.mount('/content/drive')
from zipfile import ZipFile
filename = 'Stanford Dogs Dataset.zip'
with ZipFile(filename, 'r') as zip:
zip.extractall()
print('Done')
and also tried this -
!unzip filename -d destination
Not sure where I am going wrong. And also, dunno why the extracted files though being extracted to a subfolder within drive, also starts showing up on the main root directory. And no I am not talking about the recent section, because when I want to check their location then they points to the root of the drive. It's all so confusing.
First you mount google drive
from google.colab import drive
drive.mount('/gdrive')
Then you can copy from your drive using !cp
!cp '/gdrive/My Drive/my_file' 'my_file'
then you can work as in your pc, unzip and ...