I am working with Azure Cloud Jupyter Notebook but i dont know how to read my data set so i need to know how to upload my csv dataset
Here's what I found in the FAQ online:
How can I upload my data and access it in a notebook?
A file can be added to the project itself from either the web or computer, or uploaded using the File Menu inside a JupyterNotebook if you chose to save under the project/ folder. Files outside the project/ folder will not be persisted. If you have multiple files that add up to over 100mb you'll need to upload them one by one.
You can also download data using the terminal or shell commands inside of a notebook from publicly accessible web sites include GitHub, Azure blob storage, nasa.gov, etc...
Related
I have a file on my computer that I want to upload to Google Colab. I know there are numerous ways to do this, including a
from google.colab import files
uploaded = files.upload()
or just uploading manually from the file system. But I want to upload that specific file without needing to choose that file myself.
Something like:
from google.colab import files
file_path = 'path/to/the/file'
files.upload(file_path)
Is there any way to do this?
Providing a file path directly rather than clicking through the GUI for an upload requires access to your local machine's file system. However, when your run cell IPython magic commands such as %pwd in Google collab, you'll notice that the current working directory shown is that of the notebook environment - not that of your machine. The way to eschew the issue are as follows.
1. Local Runtime
Only local runtimes via Jupyter seems to enable such access to the local file system. This necessitates the installation of jupyterlab, a Jupyter server extension for using a WebSocket, and launching a local server. See this tutorial.
2. Google Drive
In case Google Drive is convenient, you can upload files into Google Drive from your local machine without clicking through a GUI.
3. Embracing the GUI
If these options seem overkill, you, unfortunately, have to stick with
from google.colab import files
uploaded = files.upload()
as you alluded to.
I am trying to create an automated pipeline that gets files from this api fiftyone and load it to s3. From what I saw the fiftyone package can only download it locally.
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset(
"open-images-v6",
split="validation",
classes=["Cat","Dog"],
max_samples=100,
label_types=["detections"],
seed=51,
dataset_name="open-images-pets"
Thats the code I use to download the files, thing is they download locally. Anyone that has some experience with this and how could this be done?
Thank you!
You're right that the code snippet that you shared will download the files from Open Images to whatever local machine you are working on. From there, you can use something like boto3 to upload the files to s3. Then, you may want to check out the examples for using s3fs-fuse and FiftyOne to see how you can mount those cloud files and use them in FiftyOne.
Directly using FiftyOne inside of a Sagemaker notebook is in development.
Note that FiftyOne Teams has more support for cloud data, with methods to upload/download to the cloud and use cloud objects directly rather than with s3fs-fuse.
I am using an application called Screamingfrog to scan my companies sites for broken images/links. Currently, the scans are ran on a remote PC desktop and are on a scheduler that scans each site daily and stores the scan results as .csv files to a Google Drive. We use a remote desktop because the scans significantly slow down my laptop, so I'm unable to use the scan locally on my laptop. What I want to do is some how change the path to use that specific Google Drive and pull the .csv files from there. Then I run a command in Pycharm that takes those files and puts them in a Google Sheet.
So to summarize I basically want to change the path to use a Google Drive instead of locally from my work laptop.
Thank you in advance to anyone trying to help.
I think you should look at the Google Drive API Documentation, it facilities pushing and pulling files from your storage.
Here is the link: https://developers.google.com/drive/api/v3/quickstart/python
I couldn't comment on your post because of the new user restrictions
I saved file to the same directory using (./folder_name) when I use AzureML jupyter. Now how can I download to my local machine or blob storage?
The folder have a lot of files and sub-directory in it, which I scraped online. So it is not realistic to save one by one.
file_path = "./"
for i in target_key_word:
tem_str = i.replace(' ', '+')
dir_name = file_path + i
if not os.path.exists(dir_name):
os.mkdir(dir_name)
else:
print("Directory " , dir_name , " already exists")
Appreciate this is an old question but for anyone else with this questions this is how I did it.
Open a new terminal window in your Azure ML Notebooks homepage.
Zip up the entire directory with:
zip -r myfiles.zip .
Refresh the folder explorer and download the zip file.
If you are asking how you could download all your files and folders from the Notebooks GUI of Azure ML Studio, I wish there was a straightforward way to do so. As far as I know, you could do the following for now:
You have to download Azure Storage Explorer and navigate to the blob storage linked to your ML workspace in the Azure portal (you can find the precise blob storage account by navigating to the overview section of your ML workspace).
After you open the linked blob storage from the resource group, navigate to the overview section where you should select Open in Explorer.
Once the Storage Explorer is open you can link the Azure blob storage by connecting it to the Explorer (which is pretty straightforward, you just have to fetch the key from the Access Keys section).
After your storage account is linked, navigate to the File Shares folder and find the folder prefixed code-. This is where you will find all your files and folders, which you can download to your local in bulk.
From my understanding you would like to download the project files :
You could do one of the below :
From the project page, click the "Download Project" button to download all the files of a given project.
OR
Go to Azure Notebooks and sign in.
From your public profile page, select My Projects at the top of the
page.
My Projects link on the top of the browser window
Select the project.
Click the download Download button to trigger a zip file download
that contains all of your project files.
Another option for downloading Notebooks is to go through the azure ML studio UI to "Edit in VS Code." Once you have it open in VS Code, right click the folder and there is an option "Download"
You can save these files to local and then re-upload as needed.
After having worked on a Azure Functions application, i have now deployed the app and had it running for a while. Now I want to continue my work on another computer, however I cant seem to identify any way to download the source code in either VS Code nor Azure Portal?
For python function we can not download the content from Azure portal or VS code. It is in read-only mode.
Workaround:
1.Copy your project to another computer.
2.Create a new project on another computer and copy the main files from azure portal.
host.json and requirements.txt files from App files.
init.py and function.json files from Code+Test.