Develop Reference

Access Datalake from local python securely

I would like to access the contents in my Azure Datalake gen 2 via my local python editor? What would be the best way to do this?
I googled, but there are multiple ways to do this - SAS, Service principle for instance.
Could someone please provide any pointers in the right direction?
Thank you.

Use this method to link:
https://learn.microsoft.com/en-us/python/api/azure-storage-file-datalake/azure.storage.filedatalake.datalakeserviceclient?view=azure-python
And use this to create credential:
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
And for any one you want it to access, you need to add access role(RBAC, people without corresponding permissions cannot perform specific operations.):
You can try above steps.

How to start with the InstagramAPI in Python?

i want to play with the InstagramAPI and write some code for like getting a list of my follower and something like that. I am really new to that topic.
What is the best way to do this? Is there a Python-Lib for handle those json request or should I send them directly to the (new? graphAPI, displayAPI) InstagramAPI?
Appreciate every advice I can get. Thanks :)

LevPasha's Instagram-API-python, instabot, and many other API's are no longer functional as of Oct 24, 2020 after Facebook deprecated the legacy API and now has a new, authentication-required, API. It now requires registering your app with Facebook to be able to get access to many of the API features (via oembed) that were previously available without any authentication.
See https://developers.facebook.com/docs/instagram/oembed/ for more details on the new implementation and how to migrate.
You should still be able to get a list of your followers, etc. via the new oEmbed API and python--it will require registering the app, making a call to the new GET API with your authentication key via the python requests package, and then processing the result.

There is one library called instabot. This useful library has all the necessary functions/methods to interact with your insta account. Read its documentation here.
The pip installation is: pip install instabot
To get started with, lets say you want to simply login to your account.
from instabot import Bot
bot = Bot()
bot.login(username="YOUR USERNAME", password="YOUR PASSWORD")
To get the list of your followers,
my_followers = bot.followers()
If you want to upload a photo or get your posts,
bot.upload_photo(image, caption="blah blah blah") #the image variable here is a path to that image
all_posts = bot.get_your_medias() #this will return all of your medias of the account
#to get the info of each media, use
for post in all_posts:
print(bot.get_media_info(post))
and there are many other functions/methods available in this library.
It actually very fun to interact with instagram using python. You will have a great time. Enjoy :)

You can use https://github.com/LevPasha/Instagram-API-python to call Instagram APIs
and also if you want to call API directly You can use the requests package.
It also supports graphql APIs.
Here you can see an example:
https://gist.github.com/gbaman/b3137e18c739e0cf98539bf4ec4366ad

It seems like in 2022 this is the only active working and maintained python solution:
https://github.com/adw0rd/instagrapi

How to Create Python Code with Google API Client

I have code below that was given to me to list Google Cloud Service Accounts for a specific Project.
import os
from googleapiclient import discovery
from gcp import get_key
"""gets all Service Accounts from the Service Account page"""
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
service = discovery.build('iam', 'v1')
project_id = 'projects/<google cloud project>'
request = service.projects().serviceAccounts().list(name=project_id)
response = request.execute()
accounts = response['accounts']
for account in accounts:
print(account['email'])
This code works perfectly and prints the accounts as I need them. What I'm trying to figure out is:
Where can I go to see how to construct code like this? I found a site that has references to the Python API Client, but I can't seem to figure out how to make the code above from it. I can see the Method to list the Service Accounts, but it's still not giving me enough information.
Is there somewhere else I should be going to educate myself. Any information you have is appreciated so I don't pull out the rest of my hair.
Thanks, Eric

Let me share with you this documentation page, where there is a detailed explanation on how to build a script such as the one you shared, and what does each line of code mean. It is extracted from the documentation of ML Engine, not IAM, but it is using the same Python Google API Client Libary, so just ignore the references to ML and the rest will be useful for you.
In any case, here it is a commented version of your code, so that you understand it better:
# Imports for the Client API Libraries and the key management
import os
from googleapiclient import discovery
from gcp import get_key
# Look for an environment variable containing the credentials for Google Cloud Platform
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
# Build a Python representation of the REST API
service = discovery.build('iam', 'v1')
# Define the Project ID of your project
project_id = 'projects/<google cloud project>'
"""Until this point, the code is general to any API
From this point on, it is specific to the IAM API"""
# Create the request using the appropriate 'serviceAccounts' API
# You can substitute serviceAccounts by any other available API
request = service.projects().serviceAccounts().list(name=project_id)
# Execute the request that was built in the previous step
response = request.execute()
# Process the data from the response obtained with the request execution
accounts = response['accounts']
for account in accounts:
print(account['email'])
Once you understand the first part of the code, the last lines are specific to the API you are using, which in this case is the Google IAM API. In this link, you can find detailed information on the methods available and what they do.
Then, you can follow the Python API Client Library documentation that you shared in order to see how to call the methods. For instance, in the code you shared, the method used depends on service, which is the Python representation of the API, and then goes down the tree of methods in the last link as in projects(), then serviceAccounts() and finally the specificlist() method, which ends up in request = service.projects().serviceAccounts().list(name=project_id).
Finally, just in case you are interested in the other available APIs, please refer to this page for more information.
I hope the comments I made on your code were of help, and that the documentation shared makes it easier for you to understand how a code like that one could be scripted.

You can use ipython having googleapiclient installed - with something like:
sudo pip install --upgrade google-api-python-client
You can go to interactive python console and do:
from googleapiclient import discovery
dir(discovery)
help(discovery)
dir - gives all entries that object has - so:
a = ''
dir(a)
Will tell what you can do with string object. Doing help(a) will give help for string object. You can do dipper:
dir(discovery)
# and then for instance
help(discovery.re)
You can call your script in steps, and see what is result print it, do some research, having something - do %history to printout your session, and have solution that can be triggered as a script.

Authentication to Google Cloud Python API Library stopped working

I have problems with the authentication in the Python Library of Google Cloud API.
At first it worked for some days without problem, but suddenly the API calls are not showing up in the API Overview of the Google CloudPlatform.
I created a service account and stored the json file locally. Then I set the environment variable GCLOUD_PROJECT to the project ID and GOOGLE_APPLICATION_CREDENTIALS to the path of the json file.
from google.cloud import speech
client = speech.Client()
print(client._credentials.service_account_email)
prints the correct service account email.
The following code transcribes the audio_file successfully, but the Dashboard for my Google Cloud project doesn't show anything for the activated Speech API Graph.
import io
with io.open(audio_file, 'rb') as f:
audio = client.sample(f.read(), source_uri=None, sample_rate=48000, encoding=speech.encoding.Encoding.FLAC)
alternatives = audio.sync_recognize(language_code='de-DE')
At some point the code also ran in some errors, regarding the usage limit. I guess due to the unsuccessful authentication, the free/limited option is used somehow.
I also tried the alternative option for authentication by installing the Google Cloud SDK and gcloud auth application-default login, but without success.
I have no idea where to start troubleshooting the problem.
Any help is appreciated!
(My system is running Windows 7 with Anaconda)
EDIT:
The error count (Fehler) is increasing with calls to the API. How can I get detailed information about the error?!

Make sure you are using an absolute path when setting the GOOGLE_APPLICATION_CREDENTIALS environment variable. Also, you might want to try inspecting the access token using OAuth2 tokeninfo and make sure it has "scope": "https://www.googleapis.com/auth/cloud-platform" in its response.
Sometimes you will get different error information if you initialize the client with GRPC enabled:
0.24.0:
speech_client = speech.Client(_use_grpc=True)
0.23.0:
speech_client = speech.Client(use_gax=True)
Usually it's an encoding issue, can you try with the sample audio or try generating LINEAR16 samples using something like the Unix rec tool:
rec --channels=1 --bits=16 --rate=44100 audio.wav trim 0 5
...
with io.open(speech_file, 'rb') as audio_file:
content = audio_file.read()
audio_sample = speech_client.sample(
content,
source_uri=None,
encoding='LINEAR16',
sample_rate=44100)
Other notes:
Sync Recognize is limited to 60 seconds of audio, you must use async for longer audio
If you haven't already, set up billing for your account

With regards to the usage problem, the issue is in fact that when you use the new google-cloud library to access ML APIs, it seems everyone authenticates to a project shared by everyone (hence it says you've used up your limit even though you've not used anything). To check and confirm this, you can call an ML API that you have not enabled by using the python client library, which will give you a result even though it shouldn't. This problem persists to other language client libraries and OS, so I suspect it's an issue with their grpc.
Because of this, to ensure consistency I always use the older googleapiclient that uses my API key. Here is an example to use the translate API:
from googleapiclient import discovery
service = discovery.build('translate', 'v2', developerKey='')
service_request = service.translations().list(q='hello world', target='zh')
result = service_request.execute()
print(result)
For the speech API, it's something along the lines of:
from googleapiclient import discovery
service = discovery.build('speech', 'v1beta1', developerKey='')
service_request = service.speech().syncrecognize()
result = service_request.execute()
print(result)
You can get the list of the discovery APIs at https://developers.google.com/api-client-library/python/apis/ with the speech one located in https://developers.google.com/resources/api-libraries/documentation/speech/v1beta1/python/latest/.
One of the other benefits of using the discovery library is that you get a lot more options compared to the current library, although often times it's a bit more of a pain to implement.

puppet cert list all using api/python?

I need to get the list of all puppet nodes (basically output of puppet cert list --all). What is the best way to do the same using python (without using exec or similar things on the command itself) in puppet 2.6.18
puppet 2.7.0 onwards has HTTP API to achieve the same.
http://docs.puppetlabs.com/guides/rest_api.html#certificate-request
GET /{environment}/certificate_statuses/no_key
puppetdb also one api but am not sure if the env am working with has puppetdb. (checking on that).
Is there anything like ansible.runner for puppet?
Any other thoughts?

You first need to configure access to the REST API in auth.conf. Then you can use the built-in urllib2 or external requests library to query the API with the appropriate SSL client certificate for authentication.
If you don't want to deal with SSL client certificates, you can use allow_ip in auth.conf. I'd only do that if you're not interested in the more sensitive areas of the API (like requesting a catalog).
I wrote a Python wrapper around the Puppet REST API and posted it on GitHub: pypuppet.
For example,
>>> import puppet
>>> p = puppet.Puppet()
>>> print p.certificates()
See the README and example auth.conf for more info. Let me know how it works out for you.