Develop Reference

How to list files from Dataset object in Azure Pipeline

I am trying to list files from Dataset in Azure pipeline (build) but getting error, I tried below two ways
Method #1
from azureml.core import Workspace, Dataset
import os
ws = Workspace.from_config()
dataset = Dataset.get_by_name(ws, "<dataset-name>")
print(os.listdir(str(dataset.as_mount())))
FileNotFoundError: [Errno 2] No such file or directory:
'<azureml.data.dataset_consumption_config.DatasetConsumptionConfig
object at >'
Method #2
from azureml.core import Workspace, Dataset
import os
ws = Workspace.from_config()
dataset = Dataset.get_by_name(ws, "<dataset-name>")
data = dataset.mount()
data.start()
print(os.listdir(data.mount_point))
Error: Mount is only supported on Unix or Unix-like operating systems
with the native package libfuse installed.
Can anyone please help me with this, I am stuck at it from sometime.

No such file or directory: 'final_data_1.npy'

I am trying this code using tensorflow and numpy. However, I am getting an error.
import numpy as np
from tensorflow.python.framework import ops
np.random.seed(1)
ops.reset_default_graph()
ops.reset_default_graph()
#final_data_1 and 2 are the numpy array files for the images in the folder img and annotations.csv file
#total of 5 GB due to conversion of values to int
Z2= np.load('final_data_1.npy')
Z1= np.load('final_data_2.npy')
print(Z2[:,0])
print(Z1.shape)
my error is:
FileNotFoundError: [Errno 2] No such file or directory: 'final_data_1.npy'
Can you suggest a solution?

Like the Error message implies you have to name the right directory where this file "final_data_1.npy" is located at:
Example
import pandas as pd
df = pd.read_csv("./Path/where/you/stored/table/data.csv")
print(df)
Same goes with the function .load()
You have to add the directory of this file
np.load('./User/Desktop/final_data_1.npy')
Without naming the directory where the file is located your computer doesn't know where "final_data_1" is

Why am I getting a file not found error when reading a pdf file on jupyter notebook?

I'm trying to open a file in Jupyter notebook (first time using after only using atom). I'm getting an error that it can't read the file path. I'm sure the file path is correct, though. Your help is appreciated. I tried the regular open method and open with os.
import PyPDF2
import os
path = "M:\2020\BD NY\Month End\01 - Jan 2020\Pershing - LBG\01.31.2020 - LBG - AVERAGE PRICE.pdf"
os.startfile(path)
pdfFileObj = open(path)
And my error:
FileNotFoundError Traceback (most recent call last)
in
2 import os
3 path = "M:\2020\BD NY\Month End\01 - Jan 2020\Pershing - LBG\01.31.2020 - LBG - AVERAGE PRICE.pdf"
----> 4 os.startfile(path)
5 pdfFileObj = open(path)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'M:\x820\BD NY\Month End\x01 - Jan 2020\Pershing - LBG\x01.31.2020 - LBG - AVERAGE PRICE.pdf'

Are you running this on Windows or Linux? If you're on Windows,then you should be use a path like (using \\)
C:\\Users\\apple\\Downloads\train.csv
A simple way is to move your files to be read under the same folder of your python file, then you just need to use the name of the file, without calling another path.

PyDrive Upload and Remove

I am new to Google Drive API and writing a simplest form of a script that automatically upload an image from the local drive on to google drive, then once that image is uploaded, delete the local copy, following is what I have got:
#%%
import os
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from googleapiclient.http import MediaFileUpload
g_login = GoogleAuth()
g_login.LocalWebserverAuth()
drive = GoogleDrive(g_login)
#%%
header = 'images/dice'
path = header + str(i) + '.png'
file = drive.CreateFile()
file.SetContentFile(path)
file.Upload()
if file.uploaded:
print("test")
os.remove(path)
however when attempting in deleting the local copy, following error occurs:
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'images/dice1.png'
I searched it up, thinking it might be the SetContentFile(path) where it did no close the file after Upload(), which according to
https://gsuitedevs.github.io/PyDrive/docs/build/html/pydrive.html
it should close automatically after upload.
What am I overseeing here?
Note: In the end, I want to use a loop that go through all the files within the directory.
This is the output:
1
test
---------------------------------------------------------------------------
PermissionError Traceback (most recent call last)
<ipython-input-21-2aeb578b5851> in <module>
9 if file.uploaded:
10 print("test")
---> 11 os.remove(path)
12
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'images/dice1.png'

Even if PyDrive does not close it for you, from looking into the code, it looks like you can do something like this:
...
try:
file.Upload()
finally:
file.content.close()
if file.uploaded:
...
could you give a try please and see if that helps?

open function returns 404 error on python

I am working on a small python project and found myself having to read a json file. I tried with this little script found on the web, but it gives me a 404 error.
I have a folder containing the json file (datasets.json) and the python file which, for some reason, does not find the json one.
with open('datasets.json', 'r') as file:
dataset = json.loads(file.read())
print(dataset)
Traceback (most recent call last): File "Desktop/proj/ai/index.py", line 4, in with open('datasets.json', 'r') as file: FileNotFoundError: [Errno 2] No such file or directory: 'datasets.json'

The problem is that a relative path depends of the current directory, when you compile a python file the current directory isn't the file's one. Try using an absolute path. You can also transform a relative path to absolute by using os module.
import os
relativePath = './hello/world.py'
absolutePath = os.path.abspath(relativePath)
print(absolutePath)