I'm developing an API using Azure Function Apps. The API works fine locally (using localhost). However, after publishing to Function App, I'm getting this error:
[Errno 30] Read-only file system
This error happens after I made the connection as a function to allow establishing new connection every time the API is requested. The data is taken from Azure Blob Storage container.
The code:
DBConnection.py:
import os, uuid
from azure.storage.blob import BlockBlobService, AppendBlobService
from datetime import datetime
import pandas as pd
import dask.dataframe as dd
import logging
def BlobConnection() :
try:
print("Connecting...")
#Establish connection
container_name = 'somecontainer'
blob_name = 'some_name.csv'
file_path = 'somepath'
account_name = 'XXXXXX'
account_key = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
blobService = BlockBlobService(account_name=account_name, account_key=account_key)
blobService.get_blob_to_path(container_name, blob_name, file_path)
df = dd.read_csv(file_path, dtype={'Bearing': 'int64', 'Speed': 'int64'})
df = df.compute()
return df
except Exception as ex:
print('Unable to connect!')
print('Exception:')
print(ex)
You are probably running in Package or Zip.
If so when you run your code the following line is trying to save the blob and can't. If you update that to use get_blob_to_bytes or get_blob_to_stream you would be fine.
blobService.get_blob_to_path(container_name, blob_name, file_path)
From [https://stackoverflow.com/questions/53630773/how-to-disable-read-only-mode-in-azure-function-app]
Part 1 - Disabling read-only mode
You'll likely find if you're using the latest tools that your function app is in run-from-package mode, which means it's reading the files directly from the uploaded ZIP and so there's no way to edit it. You can turn that off by deleting the WEBSITE_RUN_FROM_ZIP or WEBSITE_RUN_FROM_PACKAGE application setting in the portal. Note this will clear your function app until the next time you publish.
If your tools are a little older, or if you've deployed using the latest tools but with func azure functionapp publish my-app-name --nozip then you can use the App Service Editor in Platform Features in the portal to edit the function.json files and remove the "generatedBy" setting, which will stop them being read-only.
Related
when running the azure CLI command:
az storage account blob-service-properties show --account-name sa36730 --resource-group rg-exercise1
The output json contains the filed isVersioningEnabled.
I am trying to get this field using python sdk.
I wrote this code but the output doesnt contain the version enabled information.
def blob_service_properties():
connection_string = "<connection string>"
# Instantiate a BlobServiceClient using a connection string
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
properties = blob_service_client.get_service_properties()
pprint.pprint(properties)
# [END get_blob_service_properties]
My output looks like:
{'analytics_logging': <azure.storage.blob._models.BlobAnalyticsLogging object at 0x7ff0f8b7c340>,
'cors': [<azure.storage.blob._models.CorsRule object at 0x7ff1088b61c0>],
'delete_retention_policy': <azure.storage.blob._models.RetentionPolicy object at 0x7ff0f8b9b1c0>,
'hour_metrics': <azure.storage.blob._models.Metrics object at 0x7ff0f8b9b700>,
'minute_metrics': <azure.storage.blob._models.Metrics object at 0x7ff0f8b9b3d0>,
'static_website': <azure.storage.blob._models.StaticWebsite object at 0x7ff0f8ba5c10>,
'target_version': None}
Is there a way to get the versioning information using Python SDK for storage blob?
I tried the below steps, it worked for me, in my environment
To get the Blobservice properties you can use azure-mgmt-storage package.
You can use the below to get the blob service properties.
Code:
from azure.mgmt.storage import StorageManagementClient
from azure.identity import DefaultAzureCredential
storage_client=StorageManagementClient(credential=DefaultAzureCredential(),subscription_id="<your subscription Id>")
blob_service_list = storage_client.blob_services.list("<your resourcegrouup name>', '<your account name>')
for items in blob_service_list:
print(items)
Console:
I am using Azure storage account connection string to load a data file into Azure blob storage container, using Python program. Here is the code snippet of my program:
from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
... ...
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
container_name = "test"
# Create the container
container_client = blob_service_client.create_container(container_name)
upload_file_path = "dummy_data.xlsx"
blob_client = blob_service_client.get_blob_client(container=container_name, blob=upload_file_path)
# Upload file
with open(file=upload_file_path, mode="rb") as data:
blob_client.upload_blob(data)
My program successfully created a container in the blog storage, but failed to load data into the container, with error message like this:
ClientAuthenticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:adfasa-asdfa0adfa
Time:2022-10-25T20:32:19.0165690Z
ErrorCode:AuthenticationFailed
authenticationerrordetail:The MAC signature found in the HTTP request 'bacadreRER=' is not the same as any computed signature. Server used following string to sign: 'PUT
I got stuck with the error. I tried to use SAS key and it worked. Why it's not working for a connection string? I am following Microsoft's code example to write my program:
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=managed-identity%2Croles-azure-portal%2Csign-in-azure-cli
Tried to manually upload data file with Azure Portal, and it worked. Using SAS key string in my Python code was also working. But it didn't work with Access Key connection string. It's odd that with the connection string I could create a container successfully.
I tried in my environment and got below results:
I executed the same code and successfully uploaded file in blob storage.
Code:
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
connect_str="DefaultEndpointsProtocol=https;AccountName=storage326123;AccountKey=3Lf7o2+vi3HgGKmUWaIG4xVdyzrzhxW5NxDNaUGVwykBPT5blZNKIyjbQlo0OAfuz0nllLUOGLRs+ASt9gqF+Q==;EndpointSuffix=core.windows.net"
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
container_name = "test"
# Create the container
container_client = blob_service_client.create_container(container_name)
upload_file_path = "C:\\Users\\v-vsettu\\Downloads\\dog.jpg"
blob_client = blob_service_client.get_blob_client(container=container_name, blob=upload_file_path)
# Upload file
with open(file=upload_file_path, mode="rb") as data:
blob_client.upload_blob(data)
Console:
Portal:
ClientAuthenticationError: Server failed to authenticate the request.Make sure the value of Authorization header is formed correctly including the signature. RequestId:adfasa-asdfa0adfa
Time:2022-10-25T20:32:19.0165690ZErrorCode:AuthenticationFailed
authenticationerrordetail:The MAC signature found in the HTTP request'bacadreRER=' is not the same as any computed signature. Server used following string to sign: 'PUT
The above error shows you missing something in connection string also check with signature.
You can get the connection string by
I am trying to upload file to Azure container daily.
I got an Error:"The specified blob already exists" when uploading file with same file( I want to overwrite the file)
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
conn_str = yml['AZURE_BLOB']['CONN_STR']
container_name = yml['AZURE_BLOB']['CONTAINER_NAME']
# Create the BlobServiceClient that is used to call the Blob service for the storage account
blob_service_client = BlobServiceClient.from_connection_string(conn_str=conn_str)
# Create a blob client using the local file name as the name for the blob
blob_client = blob_service_client.get_blob_client(container=container_name, blob=destination_file_name)
# Upload the created file
data = fs.open(source_path,mode='rb').read()
blob_client.upload_blob(data)
print(destination_file_name+'\t......[DONE]')
Error message:
azure.core.exceptions.ResourceExistsError: The specified blob already exists.
RequestId:13d062cd-801e-00a4-77c7-a81c56000000
Time:2019-12-02T04:18:06.0826908Z
ErrorCode:BlobAlreadyExists
Error:None
If you want to overwrite the existing blob using Blob storage client library v12, just add overwrite=True in the upload_blob method.
Here is the sample code:
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
conn_str = "xxx"
container_name = "test6"
blob_service_client = BlobServiceClient.from_connection_string(conn_str=conn_str)
blob_client = blob_service_client.get_blob_client(container=container_name,blob="a1.txt")
with open("F:\\temp\\a1.txt","rb") as data:
blob_client.upload_blob(data,overwrite=True)
print("**completed**")
After executing the code, the new blob is uploaded and the existing blob can be overwritten. Screenshot as below:
Check out this blog post about a known issue.
This is a known issue with development storage. This happens when there are multiple threads launched to upload the blocks (which constitute the blob). Basically what is happening is that development storage makes use of SQL Server as the data store. Now first thing it does is makes an entry into the table which stores blob information. If there are multiple threads working then all of these threads will try to perform the same operation. After the first thread succeeds, the subsequent threads will result in this exception being raised.
I'm currently working on using the Google Cloud Platform Speech-to-Text Api. I've followed the prompts and read some of the possible errors and troubleshooting information on the developer site. But I can't seem to make the env variable needed to access the auth credentials.
The name of the variable is GOOGLE_APPLICATION_CREDENTIALS. I have tried the following:
Setting up the variable using the command prompt: set GOOGLE_APPLICATION_CREDENTIALS=[path]
Setting up the variable using the anaconda power shell: $env:GOOGLE_APPLICATION_CREDENTIALS=[path]
Including the path in PATH variables on the environment variables native option on Windows 10
Including the environment variables on the same native option state above.
Setting up the variable inside Sublime3 using the following:
def explicit():
from google.cloud import storage
# Explicitly use service account credentials by specifying the private key
# file.
storage_client = storage.Client.from_service_account_json(
'path')
# Make an authenticated API request
buckets = list(storage_client.list_buckets())
print(buckets)
The code I am trying to run is the following:
import io
import os
# Imports the Google Cloud client library
from google.cloud import speech
from google.cloud.speech import enums
from google.cloud.speech import types
# Instantiates a client
client = speech.SpeechClient()
# The name of the audio file to transcribe
file_name = os.path.join(
os.path.dirname(__file__),
'resources',
'audio.raw')
# Loads the audio into memory
with io.open(file_name, 'rb') as audio_file:
content = audio_file.read()
audio = types.RecognitionAudio(content=content)
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=16000,
language_code='en-US')
# Detects speech in the audio file
response = client.recognize(config, audio)
for result in response.results:
print('Transcript: {}'.format(result.alternatives[0].transcript))
I have read other answers on setting up and verifying environment variables, but no solution has worked.
I can also see the environment in the env folder in the anaconda shell prompt. Does anyone have any ideas as to why the script cannot find the file?
The error message is:
line 317, in default
raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application.
Hi I was also facing the same issue but resolved it later. So the reason the path is not getting set is probably because the file is blocked by the system. Just go to the location where the file is and right click on the file -> properties. there you will see an option called unblock. click on it and that should resolve your problem.
I'm trying to upload a csv file to a container. It is constantly giving me an error that says - Retry policy did not allow for a retry: , HTTP status code=Unknown, Exception=HTTPSConnectionPool
Here is my code -
from azure.storage.blob import BlockBlobService
block_blob_service = BlockBlobService(account_name='myAccoutName', account_key='myAccountKey')
block_blob_service.get_blob_to_path(container_name='test1', blob_name='pho.csv', file_path = 'C:\\Users\\A9Q5NZZ\\pho.csv')
I am new to Python so if you can answer with a simple language, that would be really helpful.
Forget uploading a CSV file, it doesn't even let me view existing blobs in an existing container! It gives the same 'Retry Policy' error for the below code -
container_name = 'test1'
generator = block_blob_service.list_blobs(container_name)
for blob in generator:
print("\t Blob name: " + blob.name)
I understand I've asked two questions, but I think the error is the same. Any help is appreciated. Again, since I am new to Python, an explanation/code with simpler terms would be great!
The method get_blob_to_path you're using is for downloading blob to local. If you want to upload a local file to azure blob storage, you should use this method block_blob_service.create_blob_from_path(container_name="",blob_name="",file_path="")
The sample code works at my side:
from azure.storage.blob import BlockBlobService
block_blob_service = BlockBlobService(account_name='xxx', account_key='xxxx')
block_blob_service.create_blob_from_path(container_name="mycontainier",blob_name="test2.csv",file_path="D:\\temp\\test2.csv")