Develop Reference

Kafka confluent proxy api - send message - Internal server error

I'm trying to wrap the Confluent kafka proxy api in one class that will handle producing and consuming.
Following this link: https://docs.confluent.io/platform/current/kafka-rest/api.html I tried to implement it as follows:
def send(self, topic, data):
try:
r = requests.post(self._url('/topics/' + topic), json=data, headers=headers_v2)
if not r.ok:
raise Exception("Error: ", r.reason)
except Exception as e:
print(" ")
print('Event streams send request failed')
print(Exception, e)
print(" ")
return e
but I ended up working with 2 versions of the api (v2/v3) cause I didn't find some api's in one implementation and vise versa...
For example I didn't find how to create topic in v2, so I implemented it with v3.
My issue now is with the send method, I'm getting Internal server error and I can't find why!
Maybe because the create topic was done with v3 and I'm trying to produce messages with v2.

I changed the data payload for the send to look like:
data = {"records": [{"value": data}]} and send passed,
poll passed when using:
r = requests.get(self._url('/consumers/' + self.consumer_group + '/instances/' + self.consumer + '/records'), headers={'Accept': 'application/vnd.kafka.json.v2+json'})

HttpError 503, while creating subnet using GCP Python API

Hello Everyone,
Need your thoughts on an issue I am getting with a python script to create vpc and subnet.
My script is working fine when creating vpc, but next step of subnet creation is failing with error
googleapiclient.errors.HttpError: <HttpError 503 when requesting https://www.googleapis.com/compute/v1/projects/<projectname>/regions/us-east1/subnetworks?alt=json returned "Internal error. Please try again or contact Google Support.
I am able to create subnet from UI and from rest API page.
Here is the script code I am using for subnet creation-
def create_subnet(compute, project, region, classname):
subnetname = classname
networkpath = 'projects/<projectname>/global/networks/%s' % (classname)
ipCidrRange = "10.0.0.0/16"
config = {
"name": subnetname,
"network": networkpath,
"ipCidrRange": ipCidrRange
}
print('##### Print Config ##### %s' % (config))
return compute.subnetworks().insert(
project=project,
region=region,
body=config).execute()
```
def main(project, classname, zone, region):
compute = googleapiclient.discovery.build('compute', 'v1')
print('Creating vpc')
operation = create_vpc(compute, project, classname)
print('Creating subnet')
operation = create_subnet(compute, project, region, classname)
```
Thanks in advance for comments and suggestions.

I got the root cause of this issue. I was making subnet call without waiting for vpc create operation to complete.
Created new function to wait and call it after vpc creation step resolves the issue.
def wait_for_global_operation(compute, project, operation):
print('Waiting for operation to finish...')
while True:
result = compute.globalOperations().get(
project=project,
operation=operation).execute()
if result['status'] == 'DONE':
print("done.")
if 'error' in result:
raise Exception(result['error'])
return result
time.sleep(1)
Thanks Lozano for your comments and feedback.

This seems to be related to the a wrong label syntax. Try the following syntax for network and region:
"network":
"https://www.googleapis.com/compute/v1/projects/XXXXX/global/networks/XXXXX",
"region":
"https://www.googleapis.com/compute/v1/projects/XXXXX/regions/XXXXX"
The online API Explorer can be pretty helpful1.

Docker Python API - Tagging Containers

I'm using Docker with AWS ECR repo. One of the steps they instruct you to do is to run "docker tag" to tag a built image with a tag that includes a "fully-ish qualified" location of where the image is going to be stored in ECR.
I was working on migrating a script I had to Python API (instead of doing shell calls to the docker client). I'm unable to find the equivalent of "docker tag" in the API docs at https://docker-py.readthedocs.io/en/stable/images.html.
Can somebody point me in the right direction?

For those of you using ECR/ECS in AWS, here is an example of how you go about this.
Amazon provides instructions like this in ECR to push your images:
aws ecr get-login --no-include-email --region us-west-2
docker build -t myproj .
docker tag calclab:latest XXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/myproj:latest
docker push XXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/myproj:latest
Here is the rough equivalent using the Docker Python API and Boto (AWS's Python library). It includes tagging the image twice in ECR, so that I can track each image's version number while tracking what the latest is (so my ECS Task can, by default, always grab the most current image)
import docker
import boto3
def ecrDemo(version_number):
# The ECR Repository URI
repo = XXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/myproj
# The name of the [profile] stored in .aws/credentials
profile = "sandbox"
# The region your ECR repo is in
region = "us-west-2"
# How you want to tag your project locally
local_tag = "myproj"
#Set up a session
session = boto3.Session(profile_name=profile, region_name=region)
ecr = session.client('ecr')
docker_api = docker.APIClient()
print "Building image " + local_tag
for line in docker_api.build(path='.', tag=local_tag, stream=True, \
dockerfile='./Dockerfile.myproj'):
process_docker_api_line(line)
# Make auth call and parse out results
auth = ecr.get_authorization_token()
token = auth["authorizationData"][0]["authorizationToken"]
username, password = b64decode(token).split(':')
endpoint = auth["authorizationData"][0]["proxyEndpoint"]
# print "Make authentication call"
# docker_api.login(username=user, password=password, \
# registry=endpoint, reauth=True)
auth_config_payload = {'username': username, 'password': password}
version_tag = repo + ':latest'
latest_tag = repo + ':' + version_number
print "Tagging version " + version_tag
if docker_api.tag(local_tag, version_tag) is False:
raise RuntimeError("Tag appeared to fail: " + version_tag)
print "Tagging latest " + latest_tag
if docker_api.tag(local_tag, latest_tag) is False:
raise RuntimeError("Tag appeared to fail: " + tag_latest)
print "Pushing to repo " + version_tag
for line in docker_api.push(version_tag, stream=True, auth_config=auth_config_payload):
self.process_docker_api_line(line)
print "Pushing to repo " + latest_tag
for line in docker_api.push(latest_tag, stream=True, auth_config=auth_config_payload):
self.process_docker_api_line(line)
print "Removing taged deployment images"
# You will still have the local_tag image if you need to troubleshoot
docker_api.remove_image(version_tag, force=True)
docker_api.remove_image(latest_tag, force=True)
def process_docker_api_line(payload):
""" Process the output from API stream, throw an Exception if there is an error """
# Sometimes Docker sends to "{}\n" blocks together...
for segment in payload.split('\n'):
line = segment.strip()
if line:
try:
line_payload = json.loads(line)
except ValueError as ex:
print "Could not decipher payload from API: " + ex.message
if line_payload:
if "errorDetail" in line_payload:
error = line_payload["errorDetail"]
sys.stderr.write(error["message"])
raise RuntimeError("Error on build - code " + `error["code"]`)
elif "stream" in line_payload:
sys.stdout.write(line_payload["stream"])

These are the steps you can use to build and tag an image.
import docker
tag = 'latest' # or whatever you want
client = docker.from_env()
# identifier of the image on your system
dockername = "%s:%s" % (<name of the image on your system>, <tag>)
# the target identifier
target = "%s:%d/%s" % (<registry address>, <registry_port>, <id or name of the image>)
# the url is usually unix://var/run/docker.sock' but depends on your environment
cli = docker.APIClient(base_url="<the daemon\'s url or socket>")
# build the image
cli.build(path=..., tag=dockername, pull=..., buildargs=...)
# tag it
image = client.images.get(dockername)
image.tag(target, tag=tag)

This answer helped me a great deal, thank you!
When I was developing my flow, I was confused by the terminology on the docker-py page, which I think would benefit by more examples.
I was not sure during development if I was correctly building, or if I was having issues with authentication or authorization.
I needed to carefully monitor my build results using the Docker CLI, and capture and analyze the output from the different build, tag and push functions.
Another caveat with getting output from these functions is a warning that is clearly stated for the docker-py pull() function but not the others: if you request that the function provide a generator for output from the operation you must consume that generator. I was able to get my flow working with a debug level of verbosity.
Unfortunately, when I toggled off the verbosity in my code and did not consume the generators for build() and push() (tag() only has a boolean result), my flow only appeared to work: it was not throwing errors, but it also was not building or pushing the code! It is better to either not turn on the streaming output if you're not in debug mode, or leave it on and use deque() to consume the output without processing it.
To summarize the differences in how tags are used:
build() takes a 'local tag', which is just the name of the build, e.g. 'myproj'
tag() applies a 'version tag' to a 'local tag' you just built with build(), the version tag will include the registry and a version label, e.g., myregistry.mydomain.com/myname/myproj:latest
push() will push the image in a 'version tag' to the registry designated in the version tag. So in this case, the image you tagged as myregistry.mydomain.com/myname/myproj:latest will be pushed to the registry myregistry.mydomain.com.

Google API batch request returns HttpError 404 for every call through Python client

I have a little application based on google-api-python-client, but the batch request has not been working anymore for a couple of days (error 404).
For example, the following code worked fine until a few days ago.
from apiclient.http import BatchHttpRequest
from apiclient.discovery import build
import json
DEVELOPER_KEY = "foobar"
YOUTUBE_API_SERVICE_NAME = "youtube"
YOUTUBE_API_VERSION = "v3"
channelIds = ['UC2C_jShtL725hvbm1arSV9w','UC2C_jShtL725hvbm1arSV9w']
parts_list = [
"id",
"brandingSettings",
]
fields_list = [
"items/id",
"items/brandingSettings/channel",
]
parts = ",".join(parts_list)
fields = ",".join(fields_list)
request_map = {}
def this_is_the_callback_function(request_id, response, exception):
if exception is not None:
# Do something with the exception
print exception
pass
else:
print request_id
print request_map[int(request_id)]
print json.dumps(response,sort_keys=True, indent=4)
service = build(YOUTUBE_API_SERVICE_NAME, YOUTUBE_API_VERSION, developerKey=DEVELOPER_KEY)
batch = service.new_batch_http_request(callback=this_is_the_callback_function)
channels = service.channels()
i = 0
for c in channelIds:
i += 1
request_map[i] = c
request = channels.list(id=c,part=parts, fields=fields)
batch.add(request)
print request_map
batch.execute()
Now, If i run it, I get:
{1: 'UC2C_jShtL725hvbm1arSV9w', 2: 'UC2C_jShtL725hvbm1arSV9w'}
<HttpError 404 when requesting https://www.googleapis.com/youtube/v3/channels?fields=items%2Fid%2Citems%2FbrandingSettings%2Fchannel&alt=json&part=id%2CbrandingSettings&id=UC2C_jShtL725hvbm1arSV9w&key=foobar returned "Not Found">
<HttpError 404 when requesting https://www.googleapis.com/youtube/v3/channels?fields=items%2Fid%2Citems%2FbrandingSettings%2Fchannel&alt=json&part=id%2CbrandingSettings&id=UC2C_jShtL725hvbm1arSV9w&key=foobar returned "Not Found">
Strange. What it seems odd to me is that If I try to make a simple request to those links (simply by cutting and pasting an URL in the browser), the server returns data as usual.

It seems this problem is fixed with the newest version of the python library.
However, if someone still has to solve it, or is using a different environment (like iOS or android), that's the solution.
it seems google have used before a single batch url, which was:
https://www.googleapis.com/batch
this worked well in my apps, until it suddenly stopped. it seems the solution is to change the url for the batch calls to :
https://www.googleapis.com/batch/youtube/v3
(for youtube apis)
usually the google api objects will expose a property allowing to change this value, but if not it can be changed in source.

How do I return the Instagram Realtime subscription challenge?

I'm trying to subscribe to a tag. It appears that the callback URL is being called correctly with a hub.challenge and hub.mode, and I figured out how to access the challenge using self.request.get('hub.challenge'). I thought I was just supposed to echo the challenge, but that doesn't appear to work since I receive the following errors in the GAE logs:
InstagramAPIError: (400) APISubscriptionError-Challenge verification failed. Sent "647bf6dbed31465093ee970577ce1b72", received "
647bf6dbed31465093ee970577ce1b72
".
Here is the full handler:
class InstagramHandler(BaseHandler):
def get(self):
def process_tag_update(update):
update = update
mode = self.request.get('hub.mode')
challenge = self.request.get('hub.challenge')
verify_token = self.request.get('hub.verify_token')
if challenge:
template_values = {'challenge':challenge}
path = os.path.join(os.path.dirname(__file__), '../templates/instagram.html')
html = template.render(path, template_values)
self.response.out.write(html)
else:
reactor = subscriptions.SubscriptionsReactor()
reactor.register_callback(subscriptions.SubscriptionType.TAG, process_tag_update)
x_hub_signature = self.request.headers.get('X-Hub-Signature')
raw_response = self.request.data
try:
reactor.process('INSTAGRAM_SECRET', raw_response, x_hub_signature)
except subscriptions.SubscriptionVerifyError:
logging.error('Instagram signature mismatch')

So returning it as a string worked. I should have payed closer attention to the error message, but it took a helpful person on the Python IRC to point out the extra line breaks in the message. Once I put the template files on one line, it seemed to work. I can now confirm that my app is authorized via Instagram's list subscription URL.