I'm working on building an SQLite3 database to store aperture flux measurements of stars in any given astronomical image. I have one file (star_database.py) containing a Flask app running with two routes that handle selecting from and inserting into that database. There is a separate script (aperture_photometry.py) that will call those routes when incoming images need photometric processing. The crux of my problem is in the interaction between the function to insert data into the SQLite database and the aperture photometry script tasked with passing data to the Flask app. Here are the relevant functions:
# Function in aperture_photometry.py that turns Star object data into a dictionary
# and passes it to the Flask app
from astropy.io import fits
import requests
import json
def measure_photometry(image, bandpass):
df = fits.getdata(image)
hf = fits.getheader(image, ext=1)
date_obs = hf['DATE-OBS']
ra, dec = get_center_coords(hf)
response = requests.get(f'http://127.0.0.1:5000/select/{ra}/{dec}/').content
star_json = json.loads(response)
if star_json is not None:
stars = json_to_stars(star_json)
get_raw_flux(df, df*0.01, hf, stars) # Error array will be changed
star_json = []
# Critical section
for star in stars:
star_json.append({"star_id":star.star_id, "source_id":star.source_id, "flux":star.flux, "flux_err":star.flux_err})
response = requests.put('http://127.0.0.1:5000/insert/', data={'stars':star_json, 'bandpass':bandpass, 'dateobs':date_obs})
print(response.content)
else:
print("ERROR: Could not get star objects from database.")
# Function in star_database.py that handles incoming flux measurements from the
# aperture photometry script, and then inserts data into the SQLite database
from flask import Flask, request
#app.route('/insert/', methods=['PUT'])
def insert_flux_rows():
conn = create_connection(database)
if conn is not None:
c = conn.cursor()
body = request.get_json(force=True)
print(body)
# More comes after this, but it is not relevant to the question
After running the Flask app and calling aperture_photometry.py, the PUT request response.content line prints a 400 Bad Request error with the message, Failed to decode JSON object: Expecting value: line 1 column 1 (char 0). I think that problem is either in the way I have tried to format the star object data as it is being passed into the PUT request in measure_photometry, or if not there is something wrong with doing body = request.get_json(force=True). It is also worth mentioning that the statement print(body) in insert_flux_rows does not print anything to stdout. For all intents and purposes the two scripts must remain separate, i.e. I cannot combine them and remove the requests dependency.
I would really appreciate some help with this, as I have been trying to fix it all day.
Based on the top answer from this question, it seems like your data variable in the measure_photometry function may not be properly convertable to json.
You should try to test it out (maybe run a json.dumps on it) to see if a more detailed error message is provided. There's also the jsonschema package.
Related
I have made a REST API using Python-Flask, now the input for api_call is supposed to be a string and that string needs to have special characters included. The problem occurs here is that when I make the request with http://127.0.0.1:5000/api/<your string>, the browser redirected it as a Google Search Query and sometimes it just throw the Not Found 404 error and this only occurs when I use some special characters in my string like ?. Is there a way to encode the string before calling the api? What I have done is I have made an another program that urlencode the string passed and redirect it to my api call but that doesn't seem like the correct way to call api.
Here's my API code:
# Importing Libraries
from flask import Flask, jsonify, request
# Creating Flask App
app = Flask(__name__)
#app.route('/')
def index():
'''
index function, to inform user of the API route.
params: None
return: None
'''
return f"Visit:<br><strong>{request.url}api/(your string)</strong><br>to get the api response"
#app.route('/api/<string:body>')
def sample(body):
# the code for this function is different and can't be shared but the output works similar
return jsonify({'length': len(body)})
if __name__ == '__main__':
app.run(debug=True)
As I have mentioned the code for the sample() function is different from the original but the output generation is completely similar, the original code is also returning the dictionary which will then be jsonified for response just like the code here. The code for string_reformat is:
# Importing Libraries
from urllib.parse import quote
# URL Encoding String
input_string = input()
print(quote(input_string))
I have a question about python and firestore, I have conected python to fire store and I can add data however when I try to fill in a field in my database using a Varible all I get in the firebase console is a random string.
My code
import urllib
import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore
link = "example.com/test1"
f = urllib.urlopen(link)
uw = f.read()
linky = "Example.com/test.txt"
fa = urllib.urlopen(linky)
uname = fa.read()
print(uname)
unname=uname
# Use a service account
uname="hello"
cred = credentials.Certificate('key.json')
firebase_admin.initialize_app(cred)
#uw="hello"
db = firestore.client()
doc_ref = db.collection(u'treasure').document(uw)
doc_ref.set({
u'found_by': uname
#u'last': u'Lovelace',
#u'born': 1815
})
#print(uuname)
print(uname)
Here is my firebase consoleSorry, I don't have the needed reputation to embed an image but here is the link
noteI am trying to load the data to put into the database from a server however I have verified this is not the issue. the first one of these url ib request is getting a name of the document however this works well but the second one is where I get the field data but the problem is not loading it off a serverThanks!
The data is being base64-encoded.
>>> s = 'aGVsbG8='
>>> base64.b64decode(s)
b'hello'
I don't use Firebase/Firestore but if the data is being automatically base64-encoded on write then most likely it will be automatically decoded when read.
If you need to manually decode it, note that base64.b64decode returns bytes, so you need to call .decode() on the bytes to get a str.
This comment on github suggests that adding the u prefix to string literals will make Firestore encode as UTF-8 instead of base64.
So in your example:
uname = u'Hello'
I'm looking for some advice, or a relevant tutorial regarding the following:
My task is to set up a flask route that POSTs to API endpoint X, receives a new endpoint Y in X's response, then GETs from endpoint Y repeatedly until it receives a certain status message in the body of Y's response, and then returns Y's response.
The code below (irrelevant data redacted) accomplishes that goal in, I think, a very stupid way. It returns the appropriate data occasionally, but not reliably. (It times out 60% of the time.) When I console log very thoroughly, it seems as though I have bogged down my server with multiple while loops running constantly, interfering with each other.
I'll also receive this error occasionally:
SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /book
import sys, requests, time, json
from flask import Flask, request
# create the Flask app
app = Flask(__name__)
# main booking route
#app.route('/book', methods=['POST']) #GET requests will be blocked
def book():
# defining the api-endpoints
PRICING_ENDPOINT = ...
# data to be sent to api
data = {...}
# sending post request and saving response as response object
try:
r_pricing = requests.post(url = PRICING_ENDPOINT, data = data)
except requests.exceptions.RequestException as e:
return e
sys.exit(1)
# extracting response text
POLL_ENDPOINT = r_pricing.headers['location']
# setting data for poll
data_for_poll = {...}
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
# poll loop, looking for 'UpdatesComplete'
j = 1
poll_json = r_poll.json()
update_status = poll_json['Status']
while update_status == 'UpdatesPending':
time.sleep(2)
j = float(j) + float(1)
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
poll_json = r_poll.json()
update_status = poll_json['Status']
return r_poll.text
This is more of an architectural issue more than a Flask issue. Long-running tasks in Flask views are always a poor design choice. In this case, the route's response is dependent on two endpoints of another server. In effect, apart from carrying the responsibility of your app, you are also carrying the responsibility of another server.
Since the application's design seems to be a proxy for another service, I would recommend creating the proxy in the right way. Just like book() offers the proxy for PRICING_ENDPOINT POST request, create another route for POLL_ENDPOINT GET request and move the polling logic to the client code (JS).
Update:
If you cannot for some reason trust the client (browser -> JS) with the POLL_ENDPOINT information in a hidden proxy like situation, then maybe move the polling to a task runner like Celery or Python RQ. Although, it will introduce additional components to your application, it would be the right way to go.
Probably you get that error because of the HTTP connection time out with your API server that is looping. There are some standards for HTTP time connection and loop took more time that is allowed for the connection. The first (straight) solution is to "play" with Apache configs and increase the HTTP connection time for your wsgi. You can also make a socket connection and in it check the update status and close it while the goal was achieved. Or you can move your logic to the client side.
this is a two-part question: I have seen individual pieces discussed, but can't seem to get the recommended suggestions to work together. I want to create a web service to store images and their metadata passed from a caller and run a test call from Postman to make sure it is working. So to pass an image (Drew16.jpg) to the web service via Postman, it appears I need something like this:
For the web service, I have some python/flask code to read the request (one of many variations I have tried):
from flask import Flask, jsonify, request, render_template
from flask_restful import Resource, Api, reqparse
...
def post(self, name):
request_data = request.get_json()
userId = request_data['UserId']
type = request_data['ImageType']
image = request.files['Image']
Had no problem with the data portion and straight JSON but adding the image has been a bugger. Where am I going wrong on my Postman config? What is the actual set of Python commands for reading the metadata and the file from the post? TIA
Pardon the almost blog post. I am posting this because while you can find partial answers in various places, I haven't run across a complete post anywhere, which would have saved me a ton of time. The problem is you need both sides to the story in order to verify either.
So I want to send a request using Postman to a Python/Flask web service. It has to have an image along with some metadata.
Here are the settings for Postman (URL, Headers):
And Body:
Now on to the web service. Here is a bare bones service which will take the request, print the metadata and save the file:
from flask import Flask, request
app = Flask(__name__)
# POST - just get the image and metadata
#app.route('/RequestImageWithMetadata', methods=['POST'])
def post():
request_data = request.form['some_text']
print(request_data)
imagefile = request.files.get('imagefile', '')
imagefile.save('D:/temp/test_image.jpg')
return "OK", 200
app.run(port=5000)
Enjoy!
Make sure `request.files['Image'] contains the image you are sending and follow http://flask.pocoo.org/docs/1.0/patterns/fileuploads/ to save the file to your file system. Something like
file = request.files['Image']
file.save('./test_image.jpg')
might do what you want, while you will have to work out the details of how the file should be named and where it should be placed.
I'm currently playing around with tipfy on Google's Appengine and just recently ran into a problem: I can't for the life of me find any documentation on how to use GET variables in my application, I've tried sifting through both tipfy and Werkzeug's documentations with no success. I know that I can use request.form.get('variable') to get POST variables and **kwargs in my handlers for URL variables, but that's as much as the documentation will tell me. Any ideas?
request.args.get('variable') should work for what I think you mean by "GET data".
Source: http://www.tipfy.org/wiki/guide/request/
The Request object contains all the information transmitted by the client of the application. You will retrieve from it GET and POST values, uploaded files, cookies and header information and more. All these things are so common that you will be very used to it.
To access the Request object, simply import the request variable from tipfy:
from tipfy import request
# GET
request.args.get('foo')
# POST
request.form.get('bar')
# FILES
image = request.files.get('image_upload')
if image:
# User uploaded a file. Process it.
# This is the filename as uploaded by the user.
filename = image.filename
# This is the file data to process and/or save.
filedata = image.read()
else:
# User didn't select any file. Show an error if it is required.
pass
this works for me (tipfy 0.6):
from tipfy import RequestHandler, Response
from tipfy.ext.session import SessionMiddleware, SessionMixin
from tipfy.ext.jinja2 import render_response
from tipfy import Tipfy
class I18nHandler(RequestHandler, SessionMixin):
middleware = [SessionMiddleware]
def get(self):
language = Tipfy.request.args.get('lang')
return render_response('hello_world.html', message=language)