Develop Reference

Pandas not outputting anything after successful deployment

When I run this script, I do not get an output. It appears as if it is successful as I do not get any errors telling me otherwise. When I run the notebook, a cell appears below the 5th cell, indicating that the script ran successfully, but there's nothing populated. All of my auth is correct as when I use the same auth in postman to pull tag data values, it's successful. This script used to run fine and output a table in addition to a graph.
What gives? Any help would be greatly appreciated.
Sample dataset when pulling tag data values from the Azure API
"c": 100,
"s": "opc",
"t": "2021-06-11T16:45:55.04Z",
"v": 80321248.5
#Code
import pandas as pd
from modules.services_factory import ServicesFactory
from modules.data_service import TagDataValue
from modules.model_service import ModelService
from datetime import datetime
import dateutil.parser
pd.options.plotting.backend = "plotly"
#specify tag list, start and end times here
taglist = ['c41f-ews-systemuptime']
starttime = '2021-06-10T14:00:00Z'
endtime = '2021-06-10T16:00:00Z'
# Get data and model services.
services = ServicesFactory('local.settings.production.json')
data_service = services.get_data_service()
tagvalues = []
for tag in taglist:
for tagvalue in data_service.get_tag_data_values(tag, dateutil.parser.parse(starttime), dateutil.parser.parse(endtime)):
tagvaluedict = tagvalue.__dict__
tagvaluedict['tag_id'] = tag
tagvalues.append(tagvaluedict)
df = pd.DataFrame(tagvalues)
df = df.pivot(index='t',columns='tag_id')
fig = df['v'].plot()
fig.update_traces(connectgaps=True)
fig.show()

How to send form data to Flask App API Pandas Dataframe

I am learning API's but I have been using Pandas for data analysis for some time. Can I send data to an API from a Pandas dataframe?
For example, if I make up some time series data in a Pandas df and attempt to use df.to_json(). Ultimate goal is here to make a Flask API that returns the median value of Value in the Pandas df.
import requests
import pandas as pd
import numpy as np
from numpy.random import randint
np.random.seed(11)
rows,cols = 50000,1
data = np.random.rand(rows,cols)
tidx = pd.date_range('2019-01-01', periods=rows, freq='T')
df = pd.DataFrame(data, columns=['Value'], index=tidx)
median_val = df.Value.median()
print('[INFO]')
print(median_val)
print('[INFO]')
print(df.head())
json_data = df.to_json()
print('[Sending to API!]')
url = "http://127.0.0.1:5000/api/v1.0/median_val"
print(requests.post(url, json_data).text)
Is it possible (or bad practice) to send a years worth of time series data to an API to get processed? Or how much data can be sent as FORM on an HTTP POST request?
Here is something simple in Flask on a local route shown below which errors out. This is just something I made up on the fly trying to figure it out.
import numpy as np
import pandas as pd
import time, datetime
from datetime import datetime
import json
from flask import Flask, request, jsonify
#start flask app
app = Flask(__name__)
#Simple flask route to return Value average
#app.route("/api/v1.0/median_val", methods=['POST'])
def med_val():
r = request.form.to_dict()
print(r.keys())
df = pd.json_normalize(r)
print(df)
if r.keys() == {'Date','Value'}:
try:
df = pd.json_normalize(r)
df['Date'] = datetime.fromtimestamp(df['Date'].astype(float))
df = pd.DataFrame(df,index=[0])
df = df.set_index('Date')
df['Value'] = df['Value'].astype(float)
median_val = df.Value.median()
except Exception as error:
print("Internal Sever Error {}".format(error))
error_str = str(error)
return error_str, 500
return json.dumps(median_val)
else:
print("Error on api route, rejected unable to process keys")
print("rejected unable to process keys")
return 'Bad Request', 400
if __name__ == '__main__':
print("Starting main loop")
app.run(debug=True,port=5000,host="127.0.0.1")
I dont get why the print on the flask side the prints are empty. Any tips greatly appreciated there isnt a lot of wisdom here to web server processes/design.
r = request.form.to_dict()
print(r.keys())
df = pd.json_normalize(r)
print(df)
Full trace back on the Flask side.
dict_keys([])
Empty DataFrame
Columns: []
Index: [0]
Error on api route, rejected unable to process keys
rejected unable to process keys
127.0.0.1 - - [10/Feb/2021 07:50:44] "←[31m←[1mPOST /api/v1.0/median_val HTTP/1.1←[0m" 400 -

I got the code to work :) not using df.to_json() but populating an empty Python dictionary baggage_handler = {} with the data to send to the Flask App Api route to process the data.
Also not super sure on best practices for how much data can be sent as an HTTP POST body but this appears to work on local host :)
Flask APP:
import numpy as np
import pandas as pd
import time, datetime
from datetime import datetime
import json
from flask import Flask, request, jsonify
#start flask app
app = Flask(__name__)
#Simple flask route to return Value average
#app.route("/api/v1.0/median_val", methods=['POST'])
def med_val():
r = request.form.to_dict()
df = pd.json_normalize(r)
print('incoming keys')
print(r.keys())
if r.keys() == {'Value'}:
print('keys are good')
try:
df = pd.json_normalize(r)
df['Value'] = df['Value'].astype(float)
median_val = df.Value.median()
print('median value == ',median_val)
except Exception as error:
print("Internal Sever Error {}".format(error))
error_str = str(error)
return error_str, 00
return json.dumps(median_val)
else:
print("Error on api route, rejected unable to process keys")
print("rejected unable to process keys")
return 'Bad Request', 400
if __name__ == '__main__':
print("Starting main loop")
app.run(debug=True,port=5000,host="127.0.0.1")
HTTP Request script:
import requests
import pandas as pd
import numpy as np
from numpy.random import randint
np.random.seed(11)
rows,cols = 50000,1
data = np.random.rand(rows,cols)
tidx = pd.date_range('2019-01-01', periods=rows, freq='T')
df = pd.DataFrame(data, columns=['Value'], index=tidx)
median_val = df.Value.median()
print('[INFO]')
print(median_val)
print('[INFO]')
print(df.head())
#create an empty dictionary
baggage_handler = {}
print('[packaging some data!!]')
values_to_send = df.Value.tolist()
baggage_handler['Value'] = values_to_send
print('[Sending to API!]')
response = requests.post('http://127.0.0.1:5000/api/v1.0/median_val', data=baggage_handler)
print("RESPONCE TXT", response.json())
data = response.json()
print(data)

Sentinel API error on Python: HTTP status 200 OK: API response not valid. JSON decoding failed

I have been trying to follow an easy tutorial on how to get sentinel 2 images for a series of polygons I have. For some reason, no matter what I do I keep running into the same error (detailed above).
from sentinelsat import SentinelAPI, read_geojson, geojson_to_wkt
import geopandas as gpd
import folium
import rasterio as rio
from rasterio.plot import show
from rasterio.mask import mask
import matplotlib.pyplot as plt
from pyproj import Proj, transform
import pandas as pd
import os
from datetime import date
import sentinelhub
user = 'xxxxx'
password = 'xxxxx'
url = 'https://scihub.copernicus.eu/dhus'
api = SentinelAPI(user, password, url)
validation = gpd.read_file('EarthData/tutakoke_permafrost_validation/Tutakoke_permafrost_validation.shp')
plateau_transects = gpd.read_file('EarthData/tutakoke_permafrost_plateau_transects/Tutakoke_Permafrost_Plateau_Transects.shp')
validation = validation.set_crs(epsg=32604, inplace=True, allow_override=True)
validation['imdate']='01-01-2019'
validation['imdate'] = pd.to_datetime(validation2['imdate'])
validation['geometry2'] = validation.geometry.buffer(2, cap_style=3)
footprint=validation['geometry2'][1]
products = api.query(footprint,
date = ('20200109', '20200510'),
platformname = 'Sentinel-2',
processinglevel = 'Level-2A',
cloudcoverpercentage = (0, 20))
The error I keep getting is:
SentinelAPIError: HTTP status 200 OK: API response not valid. JSON decoding failed.

Ah - it was that my footprint was not in the correct lat lon format!

How can I transform geocoding with google in geocoding with Tomtom? (Python?

I have this code that basically give me the lat and long when I give it the direction. It is planted to use the API of google but now I need to use the Tomtom API. I don´t know how to do it.
import openpyxl
from openpyxl import load_workbook
import pandas as pd
ds_address=('C:/Users...
wb = load_workbook(filename = ds_address)
data = wb['Sheet1']
#########################
######### 02 - API GOOGLE
#########################
import googlemaps
YOUR_API_KEY =
gmaps = googlemaps.Client(key=YOUR_API_KEY)
def geo_coding_google(adrss):
geocode_result = gmaps.geocode(adrss)
lat,lng = geocode_result[0]['geometry']['location']['lat'],geocode_result[0]['geometry']['location']['lng']
return(lat,lng)
#########################
######### 03 - Build Data
#########################
out = []
for r in range(2,101):#42207):
try:
id=str(data.cell(row=r, column=1).value).lower().encode('utf-8').strip()
adrss=str(data.cell(row=r, column=2).value).lower().encode('utf-8').strip()
adrss=str(adrss)+",medellin,antioquia,colombia"
print(adrss)
g_c=geo_coding_google(str(adrss))
#### Save Data
out.append({'id':id,'direccion':adrss,'lat':g_c[0],'lon':g_c[1]})
except IndexError:
out.append({'id':"NA",'direccion':"NA",'lat':"NA"[0],'lon':"NA"[1]})
pass
#########################
######### 04 - Export data
#########################
df = pd.DataFrame(out)
file_name="C:/Users
Thanks

You need a library that supports TomTom Geocoding endpoint.
For example: https://geocoder.readthedocs.io/providers/TomTom.html
>>> import geocoder
>>> g = geocoder.tomtom(adrss, key='<API KEY>')
>>> result = g.latlng

How to apply the Python function with predictive model into Flask application

I have a code that consists of two logical parts: receiving POST json request in Flask and function with prediction model. So here how it looks in code representation:
from flask import Flask
from flask import request
import io
import json
import pandas as pd
import numpy as np
from fbprophet import Prophet
app = Flask(__name__)
#app.route('/postjson', methods = ['POST'])
def postJsonHandler():
print (request.is_json)
content = request.get_json()
df = pd.io.json.json_normalize(content, ['Days', 'Orders'])
print (df)
return 'JSON posted'
app.run(host='0.0.0.0', port= 8090)
And here is the part with the function with model:
def TimeSeries ():
df['Days'] = pd.to_datetime(df['Days'])
df = df.rename(columns={'Days': 'ds',
'Orders': 'y'})
my_model = Prophet(interval_width=0.95, yearly_seasonality=False, daily_seasonality=False, weekly_seasonality=True)
df['y'] = np.log(df['y'])
my_model.fit(df)
future_dates = my_model.make_future_dataframe(periods=30)
forecast = my_model.predict(future_dates)
yhat=forecast.yhat
ser=np.exp(yhat)
df_upd=pd.DataFrame(ser[-30:])
df_upd.reset_index(drop=True, inplace=True)
js=df_upd.to_dict(orient='split')
del js['index']
res=json.dumps(js)
return res
My questions will bу following:
How can I transfer the result dataframe df from first part function postJsonHandler () and use it as an input in the second part function TimeSeries()?
How can I integrate my predictive function TimeSeries() in Flask environment to be able to run everything at once- to receive a json request, transform it into pandas dataframe, caluclates the result of prediction in json format and transfer this to server.
Big Thanks!

You combine your functions:
from flask import Flask
from flask import request
import io
import json
import pandas as pd
import numpy as np
from fbprophet import Prophet
app = Flask(__name__)
my_model = Prophet(interval_width=0.95, yearly_seasonality=False, daily_seasonality=False, weekly_seasonality=True)
#app.route('/postjson', methods = ['POST'])
def postJsonHandler():
print (request.is_json)
content = request.get_json()
df = pd.io.json.json_normalize(content, ['Days', 'Orders'])
df['Days'] = pd.to_datetime(df['Days'])
df = df.rename(columns={'Days': 'ds',
'Orders': 'y'})
df['y'] = np.log(df['y'])
my_model.fit(df)
future_dates = my_model.make_future_dataframe(periods=30)
forecast = my_model.predict(future_dates)
yhat=forecast.yhat
ser=np.exp(yhat)
df_upd=pd.DataFrame(ser[-30:])
df_upd.reset_index(drop=True, inplace=True)
js=df_upd.to_dict(orient='split')
del js['index']
res=json.dumps(js)
app.run(host='0.0.0.0', port= 8090