How to setup Kafka Client within Google App Engine - python

I have followed the google tutorial below and am able to run this on my google app engine no problem.
https://cloud.google.com/appengine/docs/standard/python3/building-app/writing-web-service
I only run into problems when I try to incorporate a kafka client in the main.py script. The main script is below and when all reference to kafka is commented out this script runs fine again in the app engine.
Can Kafka be used in an app engine? I have tried using both the internal and external kafka IP addresses on GCP with no luck... When I run "gcloud app deploy" with requirements.txt app.yaml and main.py in the same directory (as per tutorial), or even "python3 main.py" (for testing) the script hangs, there is no error message it just doesn't start the server. Again if Kafka references removed it runs fine. Kafka is running fine in the GCP project - can pub and sub from ssh terminals fine.
I have been able to make this Kafka setup work when run on a Google Compute Engine with the Flask app run directly from the VM - (running directly from the VM not using gcloud terminal). Doing this seems to allow kafka to use the internal IP address and avoids any problems with external IP addresses and cross network difficulties that Kafka doesn't seem to like.
Please help!! (This is my first ever SOF question - so be kind and be very descriptive:) )
from flask import Flask, render_template, Response
from pykafka import KafkaClient
from pykafka.common import OffsetType
clientKafka = KafkaClient(hosts='10.128.0.3:9092')
app = Flask(__name__)
topick = clientKafka.topics['gps_new']
producer = topick.get_sync_producer()
#app.route('/')
def index():
return(render_template('index.html'))
#app.route('/topic/gps_new/')
def get_messages():\
def events():
for i in
clientKafka.topics["gps_new"].get_simple_consumer(auto_offset_reset=OffsetType.LATEST,reset_offset_on_start=True):
yield 'data:{0}\n\n'.format(i.value.decode())
return Response(events(), mimetype='text/event-stream')
if __name__ == '__main__':
# This is used when running locally only. When deploying to Google App
# Engine, a webserver process such as Gunicorn will serve the app. This
# can be configured by adding an `entrypoint` to app.yaml.
# Flask's development server will automatically serve static files in
# the "static" directory. See:
# http://flask.pocoo.org/docs/1.0/quickstart/#static-files. Once deployed,
# App Engine itself will serve those files as configured in app.yaml.
app.run(host='127.0.0.1', port=8080, debug=True)

Related

How to deploy python application to evennode?

After reading following docs and examples about deploying python app to evennode I've tried to do it with Flask application, but didn't succeed
https://www.evennode.com/docs/git-deployment
https://github.com/evennode/python-getting-started
Here is my main.py module's code:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello, World!"
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8000)
Also I'have created conf.py file with configuration for gunicorn:
workers = 4
bind = "0.0.0.0:8000"
Then I'm running the application with gunicorn --config=conf.py main:app and all works well on my local machine. To run it on evennode I populated requirements.txt and committed above files. Then run following commands:
git remote add evennode git#git.evennode.com:your_app_here
git push evennode master
The output looks next way and I don't know what to do with it:
ssh: connect to host git.evennode.com port 8000: No route to host
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
I have added my public ssh key to evennode app settings as well, so that's can't be an issue
Any help is appreciated

Multiple flask applications served by a single gunicon server

I have a service, which contains 2 Flask-applications - let's name them app and monitoring_app running on different ports. monitoring_app is an utility service which provides metrics collected with prometheus_client. I have faced up with an issue trying run them under the Gunicorn server properly.
I have read through the similar topic:
Multiple Flask Application in single uwsgi
but it seems my problem can't be solved with Dispatcher Middleware.
I can start these applications without Gunicorn like this:
from threading import Thread
import os
from my_project import create_app, create_monitoring_app
def start_app(my_app):
my_app.run(host="localhost",
debug=True,
port=int(os.environ.get("API_PORT", "5000")),
threaded=True,
use_reloader=False)
if __name__ == "__main__":
app = create_app()
app_thread = Thread(target=start_app, daemon=True, args=(app,))
app_thread.start()
monitoring_app = create_monitoring_app()
monitoring_app.run(host="localhost",
port=int(os.environ.get("OPS_PORT", "5001")),
threaded=True,
use_reloader=False,
debug=True)
It works ok for development, but it runs under Flask development server which is not ok for production environment. With Gunicorn I can start them separately:
gunicorn "my_project:create_monitoring_app()" -b "[::]:$OPS_PORT" &
gunicorn "my_project:create_app()" -b "[::]:$API_PORT"
but then they will be in different interpreters and I can't use prometheus_client of app in monitoring_app what is critical for me
What can I do to achieve same behavior as if I run these applications from my development environment? Or may be I am doing something wrong and I should do it in another way?

Python import from a virtual environment when run in apache

I am new in python. I am trying to deploy python code on apache server for i.e i have created flask api. So for apache i have installed XAMPP and changed my httpd.conf to execute python on apache. It works well!! Here is code example which is working
Code working:
#!C:\Users\test.lab\AppData\Local\Continuum\anaconda3\envs\myproject\python.exe
# enable debugging
print("Content-type: text/html\n")
print ("Hello Python Web Browser!! This is cool!!")
But when I tried to import that through 500 Error, Here is the code
#!C:\Users\test.lab\AppData\Local\Continuum\anaconda3\envs\myproject\python.exe
# enable debugging
from flask import Flask
app = Flask(__name__)
#app.route('/')
def index():
return 'testing'
if __name__ == '__main__':
app.run(debug = True)
flask is installed on my environment (myproject). When I run through command like python test.py and it works.
Flask has it's own development web server.
Using python myfile.py which will work properly as a webserver (no need for apache on development).
If you still want do deploy on Apache, Flask have some info on how to do so, docs: http://flask.pocoo.org/docs/1.0/deploying/mod_wsgi/
Special attention to this: http://flask.pocoo.org/docs/1.0/deploying/mod_wsgi/#creating-a-wsgi-file

Predix - Polymer frontend to communicate with Flask Python server?

I'm trying to push a program containing of 2 apps to Cloudfoundry - a Flask server to run a Python API and a Polymer app for frontend. Currently I'm using the following structure:
manifest.yml:
---
applications:
- name: flask_min
path: ./flask_min
buildpack: https://github.com/cloudfoundry/python-buildpack
memory: 512M
- name: pacing_app
memory: 512M
buildpack: nodejs_buildpack
command: node server/app.js
path: ./pacing_app/build/es5-basic
And then in the folder ./flask_min I have a Procfile:
web: python3 app.py
and app.py has a Flask server (plus the decorator for CORS for local testing that I left out from here for brevity):
app = Flask(__name__)
port = int(os.getenv("PORT", 7733))
#app.route('/hello', methods=['GET', 'OPTIONS'])
#crossdomain(origin='*')
def hello():
return "Hello"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=port)
I can cf push it, and the instances show up running. However, when I visit the url of the flask_min app, I don't get the "Hello" it's supposed to print out, I get an HTTP ERROR 503.
As for the pacing_app, I am using the Predix Webapp Starter, except that I removed the elements in seed-app.html and just replaced it with the API call:
<template>
<iron-ajax url="http://localhost:7733/hello" handle-as="text" last-response="{{data}}" auto></iron-ajax>
<p>{{data}}</p>
</template>
Locally this is working, however, on Predix it is not. First question is: How can I link to the Flask API once it's on Predix?
Also, even though the whole thing is working locally, the Polymer frontend also won't load when on Predix. I also get
Failed to load resource: the server responded with a status of 503
(Service Unavailable)
Even though the cf push seems successful, the same IS running locally when I do a gulp in the pacing-app folder and run thy python server "by hand" locally.
Do you know what I'm doing wrong? How can I set up a Polymer frontend with Predix components that uses a Python API also running on Predix?
I'm not set on using two separate apps, I just don't know how to do this with one app. I would prefer to have the Polymer app run on the NodeJS server instead of serving it from Flask because of performance and the Python/Flask server is important because I intend to run some SKLearn code in the background.
I see two options for you.
Use the predix-webapp-starter with NodeJS as your front end server. Use a separate Python microservice as your back end server. This approach might be better, especially if your backend SKLearn processes take a long time.
Run everything in a single Python/Flask server. You could look at the Predix Digital Volcano App as an example. This approach might be easier if you have more experience with Python.
The combination of polymer and flask requires a bit of 'adjusting' for the flask server to serve the correct polymer folder... Starting from a simple flask server as below:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return app.send_static_file('index.html')
#app.route('/<path:the_path>')
def all_other_routes(the_path):
return app.send_static_file(the_path)
if __name__ == '__main__':
app.run(debug=True)
and by using the polymer starter kit in a new folder:
mkdir polymer && cd polymer && polymer init polymer-3-starter-kit && polymer build && cd ..
Then, you either have to change the default flask serving folder (static) or you can simply create a symbolic link to your polymer build folder. I prefer to do the second, by issuing the following command on the root directory (alongside your server file):
ln -s ./polymer/build/es6-bundled ./static

Error connecting to python server running in boot2docker (centos container | Windows 7.1 host)

My use case:
We need to run a watir-webdriver script in headless mode and our dev machines run Windows.
The gist of our problem:
I can stand up the python server inside the container, but I can't connect to it at its expected location from the host: http://192.168.59.103:8084.
The architecture:
A ruby script that "does something useful and prints the output to the console" and a basic python server that imports os and then executes os.system('ruby ourrubyscript.rb') inside our index route. We also brought in flask for api routing and flask.cors to eliminate potential cross origin issues.
Local test (on Host):
running "python server.py" in our host workspace stands up a python server on port 5000. Routing to it from a browser executes our route and prints the expected output to the console.
server.py
from flask import Flask
from flask import request
from flask.ext.cors import CORS
import os
app = Flask(__name__)
CORS(app, resources={r'/': {"origins": "*"}}, headers='Content-Type')
app.config['CORS_HEADERS'] = 'Content-Type'
#app.route('/')
def index():
return "<h1>Hello Stupid</h1>"
#app.route('/ruby/test', methods=['GET'])
def runRubyScript():
return os.system("ruby script.rb")
if __name__ == "__main__":
app.run()
When I nav to localhost:5000, "Hello Stupid" prints to my screen.
When I route to localhost:5000/ruby/test, our script runs and that script prints to my console.
Procedure to move this to docker
We pulled latest centos and installed wget, xvfb, git, ruby, python, pip, flask, flask-cors, and firefox; committed that to a local image named "webdev."
Then gem installed headless and installed phantomjs to $PATH
We then cloned our src repository to "/opt/testapp" in the container. This repository contains our "server.py" file and a "script.rb" file.
docker#boot2docker:~$ docker run -d -p 5000:5000 --name wd webdev python /opt/testapp/server.py
So, I left my boot2docker-vm bound to private ip 192.168.59.103, and I can confirm that this ip is valid by navigating to a different port that is running a different container; however, when I navigate to 5000, I get an ERR_CONNECTION_REFUSED. I had an almost "AHA!!" moment thinking that iptables had the port locked down by default and we needed to go open internal port 5000 in order to map it out to the vm but then I found that the centos image does not have iptables (or firewalld since this is CentOS7) installed by default. When that approach turned out to be incorrect, I decided to post to see if anyone might be able to assist here because I am out of ideas.
So I have figured out why this is not exposed to the host environment and this makes perfect sense.
In NAT mode, the virtual machine gets assigned a private static ip address: 192.168.59.103 (that is...the virtual machine gets assigned that IP)
Each container stands up with 2 adapters: a loopback and an externally facing virtual private bridge accessible to the vm.
When I stand up my server on port 5000, it binds directly to loopback inside the container and is inaccessible to the virtual machine.
Removing the localhost binding from the service and allowing it to
bind to the default gateway 0.0.0.0:5000
causes the service to listen on all adapters and allows communications
between the vm and the container's external adapter through the
virtual private bridge that gets created when the container stands.
For some reason, switching my virtual machine to run in Bridged mode against my NIC is causing the boot2docker-vm to crash on start....so I'm relegated to NAT for the time being until I can figure out how to switch it properly.
[Edit 1 (related to bridging)]
Bridge mode seems to currently be unsupported in boot2docker as of at least October 8, 2014. b2d expects a NAT adapter for ssh bootstrapping and the host adapter for container<->container socket access.
The current, unsupported, undocumented workaround to expose a container to hosts on a network is to add a third network interface to VirtualBox and bridge with it.
[Edit 2 (example server running against default gateway)]
Here is an example of a flask server that attaches to all interfaces:
Note that the only Δ between the server above and the server below is the final line of the configuration.
app.run()
needs to become
app.run(host='0.0.0.0')
from flask import Flask
from flask import request
from flask.ext.cors import CORS
import os
app = Flask(__name__)
CORS(app, resources={r'/': {"origins": "*"}}, headers='Content-Type')
app.config['CORS_HEADERS'] = 'Content-Type'
#app.route('/')
def index():
return "<h1>Hello Stupid</h1>"
#app.route('/ruby/test', methods=['GET'])
def runRubyScript():
return os.system("ruby script.rb")
if __name__ == "__main__":
app.run(host='0.0.0.0')

Categories

Resources