I have an Azure blob storage account which is fire walled to selected networks only. I would like to access this storage account from a function app running on a dynamic plan whose outbound IP addresses are known to me. Problem is that I add these outbound ips to the Allowed IP addresses in Firewall and Virtual Network settings of the blob storage but I still continue to get an error which says:
This request is not authorized to perform this operation.
Can someone please point out where I am going wrong?
N.B. I am using PythonSDK for accessing the blob storage with the account name and the account key!
I did some test on my side using consumption function app to access my files in blob storage and it works for me.
There are 2 steps I did :
I enabled storage account firewall and added all function app outbound IPs to it.
Enabling anonymous access on the container that the blob file I would like to access,so that my function app can access the blob file directly (as storage firewall is enabled so that only the specified IPs would be able to access your storage , I think the security here is ok. If you need higher security level in your scenario, as #Marie Hoeger said , you should use private container and SAS token to control blob access).
If you have any further concerns , pls feel free to let me know : )
Related
I work on a solution in order to upload images from a LocalFileSystem to Azure Storage.
For the moment we use a TokenSaS and a BlobClient but we would like to avoid to store locally an expiring SaSToken.
In order to do this, we thought about Azure IoTHub, that allows us to replace this process.
def __upload_file_Azure_IoTHub(self,src_path:str,blob_path:str) ->BlobClient:
if os.path.exists(src_path):
# We start by creating a blobClient from AzureIoTHub
storage_info=self.IoTHub_client.get_storage_info_for_blob(blob_path)
# We create the SAS Url from Client + Token
sas_url="https://{}/{}/{}{}".format(
storage_info["hostName"],
storage_info["containerName"],
storage_info['blobName'],
storage_info["sasToken"])
try:
with BlobClient.from_blob_url(sas_url) as blob_client:
with open(src_path, "rb") as fp:
blob=blob_client.upload_blob(fp,overwrite=True, timeout=self.config.azure_timeout)
self.IoTHub_client.notify_blob_upload_status(storage_info["correlationId"], True, 200, "OK: {}".format(blob_path))
return blob
except Exception as ex:
self.IoTHub_client.notify_blob_upload_status(storage_info["correlationId"], False, 403, "Upload Failed")
raise Exception("AzureUpload_IoTHub")
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/iot-hub/iot-hub-devguide-file-upload.md#device-initialize-a-file-upload
The problem is that when we upload this way, the device name is added as a prefix to the blobName.
It is a problem because it will cause us problems of rectrocompatibility :
We defined a naming convention in our storage and this behavior will break everything.
Let's imagine :
DeviceName = FileWatcherDaemon
BlobPath = YYYY/MM/DD/MyBlob.whatever
# Then :
BlobName = FileWatcherDaemon/YYYY/MM/DD/MyBlob.whatever
# Instead of :
BlobName = YYYY/MM/DD/MyBlob.whatever
I tried replacing the blobName by my blob_path, but it is not working because the generated sasToken is blobLevel and not containerLevel.
Do you have an idea about how to remove this device name ?
For us it is a problem because we will have many different devices uploading in the same storage. We would like the naming convention to fit with our business needs and not to technical information.
Are you using any of the other capabilities of Azure IoT Hub besides the fileUpload? Do you need your device to received messages from the Cloud? For your scenario, it may be overkill using IoT Hub just for uploading files.
For the moment we use a TokenSaS and a BlobClient but we would like to avoid to store locally an expiring SaSToken.
Have a look at Identity and access management Security Recommendations and consider using Azure Key Vault in your scenario.
Microsoft recommends using Azure AD to authorize requests to Azure Storage. However, if you must use Shared Key authorization, then secure your account keys with Azure Key Vault. You can retrieve the keys from the key vault at runtime, instead of saving them with your application. For more information about Azure Key Vault, see Azure Key Vault overview.
For more security "Azure Key Vaults may be either software-protected or, with the Azure Key Vault Premium tier, hardware-protected by hardware security modules (HSMs)."
I intend to use Google Cloud Storage through my own domain name supereye.co.uk .
However,
When I try to associate CNAME on my DNS record for supereye.co.uk with the Google Cloud Bucket production-supereye-co-uk, I get the following message when I try to access to
production-supereye-co-uk.supereye.co.uk/static/default-coverpng :
<Error>
<Code>NoSuchBucket</Code>
<Message>The specified bucket does not exist.</Message>
</Error>
What shall I do ?
Important note : This is not a static site. This is a Django Application that runs on Google Cloud Engine and Django has its own URL routing mechanism. It means Django translates everything after supereye.co.uk/URL , I wonder how this should work.
So you cannot simply just add a CNAME record to redirect some traffic to a give URL.
You are going to have to do one of the following to get your desired result:
Serve traffic to a new sub domain data.supereye.co.uk which will host your content.
Proxy data through your django app, this is not ideal but would allow you to easily protect your data with authentication or authorization.
Proxy content through nginx, using nginx you proxy the request (forward) through to your cloud bucket. Light weight and fairly simple to implement.
Use a gcp loadbalance to split the traffic, you can setup a LB to split requests between the backend group and the bucket using host/ path rules.
I would either got for the LB or the nginx proxy as these will be the easiest to implement (depending on your setup). If you want any form of access control go for proxying the request through your django app.
I need to write a cloud function in GCP, which responds to HTTP requests and has service account access to GCP cloud storage. My function will receive a string and threshold parameters. It will retrieve a csv file from cloud storage, compute similarity between the text string supplied and the entities in the csv file and return the entities that satisfy the threshold requirements.
From Google's cloud function tutorials, I have yet to see anything that gives it cloud storage access, a service account for access therein, etc.
Could anyone link a resource or otherwise explain how to get from A to B?
Let the magic happens!
In fact, nothing is magic. With most of Google Cloud products, you have a service account that you can grant the permission that you want. On Cloud Functions, the default service account is the AppEngine default service account with this pattern <projectID>#appspot.gserviceaccount.com.
When you deploy a Cloud FUnctions you can use a custom service account by using this paramenter --service-account=. It's safer because your Cloud Functions can have his own service account, with limited permissions (App Engine default service account is Project Editor by default, which is too wide!!)
So, this service is loaded automatically with your cloud functions and the Google Cloud auth libraries can access it via the Metadata server. The credentials is taken from the runtime context, it's the default credential of the environment
About your code, keep it as simple as that
from google.cloud import storage
client = storage.Client() # Use default credentials
bucket = client.get_bucket('myBucket')
blobs = bucket.list_blobs()
for blob in blobs:
print(blob.size)
On your workstation, if you want to execute the same code, you can use your own credential by running this command gcloud auth application-default login If you prefer using a service account key file (that I strongly don't recommend, but it's not the topic), you can set the environment variable GOOGLE_APPLICATION_CREDENTIALS with the file path as value
I have created an Azure function which is trigered when a new file is added to my Blob Storage. This part works well !
BUT, now I would like to start the "Speech-To-Text" Azure service using the API. So I try to create my URI leading to my new blob and then add it to the API call. To do so I created an SAS Token (From Azure Portal) and I add it to my new Blob Path .
https://myblobstorage...../my/new/blob.wav?[SAS Token generated]
By doing so I get an error which says :
Authentification failed Invalid URI
What am I missing here ?
N.B : When I generate manually the SAS token from the "Azure Storage Explorer" everything is working well. Plus my token is not expired in my test
Thank you for your help !
You might generate the SAS token with wrong authentication.
Make sure the Object option is checked.
Here is the reason in docs:
Service (s): Access to service-level APIs (e.g., Get/Set Service Properties, Get Service Stats, List Containers/Queues/Tables/Shares)
Container (c): Access to container-level APIs (e.g., Create/Delete Container, Create/Delete Queue, Create/Delete Table, Create/Delete
Share, List Blobs/Files and Directories)
Object (o): Access to object-level APIs for blobs, queue messages, table entities, and files(e.g. Put Blob, Query Entity, Get Messages,
Create File, etc.)
I have multiple micro-services running on our project's GCP (Google Cloud Platform) App Engine. In our case, it would be preferable to minimize permissions on a per-service basis. Currently, we keep all the credential files in Keybase, keeping secrets sectioned off by team membership. So if I am working on one App Engine service, I can't see the secrets for another team's app engine service.
What we have been doing for other secrets, e.g. config files with passwords and secret tokes, we give the app engine service account kms decryption privileges and simply pull an encrypted copy of the config file from firestore and decrypt it. But we don't want to use the default app engine service account everywhere, because different teams using the same service accounts would have access to everyone's secrets. So we want to move to a service account on a per-service basis for each app engine service in development.
From what I can tell in the Google documentation, they want to you to upload up the credential file(s) when we deploy the app, which works, however, from the cloud console, it appears difficult to lock down who can look at the the files deployed to the service, and anyone with access could simply copy/paste all the credentials.
If you have the config in a dictionary, you do something like this:
from google.oauth2 import service_account
from google.cloud import kms_v1
d = {'type': 'service_account',
'project_id': 'my-awesome-project',
'private_key_id': '074139282fe9834ac23401',
'private_key': '-----BEGIN PRIVATE KEY----\n supersecretkeythatnobodyknows==\n-----END PRIVATE KEY-----\n',
'client_email': 'my-cool-address#my-awesome-project.iam.gserviceaccount.com',
'client_id': '1234567890',
'auth_uri': 'https://accounts.google.com/o/oauth2/auth',
'token_uri': 'https://oauth2.googleapis.com/token',
'auth_provider_x509_cert_url':
'https://www.googleapis.com/oauth2/v1/certs',
'client_x509_cert_url': 'https://www.googleapis.com/robot/v1/metadata/x509/my-cool-addres%40my-awesome-project.iam.gserviceaccount.com'}
credentials = service_account.Credentials.from_service_account_info(d)
kms_client = kms_v1.KeyManagementServiceClient(credentials=credentials)
This works, but how do we get the dictionary "d" into the program without it showing up in the code and having accessible by a broad range of people?
If each service in your AppEngine environment have to have their own identity, it's not possible with AppEngine. Newer service like Cloud Function or Cloud run can do this, but the very old service (for cloud era) AppEngine (more than 10 years old) can't.
You have a service account for all AppEngine service. You can use it to decrypt others services account key file, per service and use them in the appropriate service, but the root authorization stay the default App Engine service account, and thereby all services/all teams can have access to it (and to all the encrypted service account key files).
Maybe the solution is to redesign the application and to have a project per team?
What guillaume blaquiere said in his answer is correct and I agree with his idea of having a project per team, also. You cannot make a different service account for each of your App Engine's services.
Although what you want to achieve may be somehow possible in the App Engine flexible. OK, so you cannot find a way to only allow some team for some service and other team for the other. What you can do ( but I do not think that you want that and I would not recommend either ) is blocking certain IPs, by controlling your access with Firewalls. You can create your own firewall rules and allow request from one place and do not from another place.
This is just a trick that may or may not work for you. The real advice is that if you really want to implement the system you have described clean and correctly with more service accounts you should consider migrating to Cloud Run.