Related
I have a json list in the format below with 70 sources that I have to iterate and load url data in to the backend.
I already have the code and having issues deploying it as an Azure function to run every 10 minutes (timer trigger that has a 10 min timeout). In order to achieve this, I was thinking of iterating and splitting the list into 4 parts of approximately equal length and have azure function execute them within 10 min each. But, I am not able to split the list into equal parts and send subsequent list items to the function to load data. Please assist in achieving the following.
lst=[{
"input_type": "source_1",
"url_load": [
{
"url": "https:..."
"code": "site_1"
},
{
"url": "https:..."
"code": "site_2"
},
{
"url": "https:..."
"code": "site_3"
}]
},
{
"input_type": "source_2",
"url_load": [
{
"url": "https:..."
"code": "site_1"
},
{
"url": "https:..."
"code": "site_2"
},
{
"url": "https:..."
"code": "site_3"
}]
},
{
"input_type": "source_3",
"url_load": [
{
"url": "https:..."
"code": "site_1"
},
{
"url": "https:..."
"code": "site_2"
},
{
"url": "https:..."
"code": "site_3"
}]
},
{
"input_type": "source_4",
"url_load": [
{
"url": "https:..."
"code": "site_1"
},
{
"url": "https:..."
"code": "site_2"
},
{
"url": "https:..."
"code": "site_3"
}]
},.......
{
"input_type": "source_70",
"url_load": [
{
"url": "https:..."
"code": "site_1"
},
{
"url": "https:..."
"code": "site_2"
},
{
"url": "https:..."
"code": "site_3"
}]
}]
for i in range(0,70,5)
if len(lst[:i])>0:
func.load(lst[0:i])
#"schedule": "0 */10 2,3 * * *"
dt=datetime.now()
ten_min_segment = math.floor(dt.minute/10)
hour_seg - dt.hour - 3
ordinal = hour_seg *6 + ten_min_segment
url_list = lst[ordinal]
function.load(url_list)
I am trying to set up some simple billing based on AWS usage so thought that I'd automate it by using AWS's CostExplorer API with the goal of querying the last months AWS bill by client_id and then creating the client invoice automatically based on the total value.
I've set up a tag on a number of AWS products I use for a specific client_id (Lambda, LightSail, SES, API Gateway and SQS).
Then created this simple example:
import os
import json
import boto3
region_name = os.getenv('region_name')
aws_access_key_id = os.getenv('aws_access_key_id')
aws_secret_access_key = os.getenv('aws_secret_access_key') or
client = boto3.client('ce',
region_name=region_name,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key)
response = client.get_cost_and_usage(
TimePeriod={
'Start': '2021-01-01',
'End': '2021-10-22'
},
Filter={
'Tags': {
'Key': 'client_id',
'Values': ['1234567890'],
'MatchOptions': ['EQUALS']
}
},
Granularity='MONTHLY',
Metrics=[
'NormalizedUsageAmount',
]
)
print(json.dumps(response, indent=4, sort_keys=True))
I got a result:
{
"DimensionValueAttributes": [],
"ResponseMetadata": {
"HTTPHeaders": {
"cache-control": "no-cache",
"connection": "keep-alive",
"content-length": "1538",
"content-type": "application/x-amz-json-1.1",
"date": "Thu, 21 Oct 2021 13:25:09 GMT",
"x-amzn-requestid": ""
},
"HTTPStatusCode": 200,
"RequestId": "",
"RetryAttempts": 0
},
"ResultsByTime": [
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-02-01",
"Start": "2021-01-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-03-01",
"Start": "2021-02-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-04-01",
"Start": "2021-03-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-05-01",
"Start": "2021-04-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-06-01",
"Start": "2021-05-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-07-01",
"Start": "2021-06-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-08-01",
"Start": "2021-07-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-09-01",
"Start": "2021-08-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-10-01",
"Start": "2021-09-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": true,
"Groups": [],
"TimePeriod": {
"End": "2021-10-22",
"Start": "2021-10-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
}
]
}
But there are no costs even though there are definitely invoiced bills across this time period. Is what I am trying to achieve not the use case for this or is there something trivial I have missed?
so I am opening a Json file and when I try to load the file to a variable I get an errr because it can't read the file. While I have validated (online) that the Json file is valid. I am using this code:
with open("messagesTest2.json") as json_file:
data = json.load(json_file) <----- ERROR
for p in data['commits']:
print(p['message'])
And I get this error. While I have another json file that is also validated and this code works. But this file doesn't work. My guess is that somewhere in the file there is something that it cannot translate as json? Is the decoder's fault?
in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Do you have any idea how to fix it? Keep in mind that the json file is valid else I'll have to show the file but I'll have to hide some data :D
The Json file (The urls/passwards/logins/etc have been replaced but the format remains the same) :
{
"commits": [{
"sha": "asjdaskldjkalsk",
"node_id": "sakldjaskldjaskldjklas",
"commit": {
"author": {
"name": "korki",
"email": "korki#kth.se",
"date": "2015-09-07T22:06:51Z"
},
"committer": {
"name": "korki",
"email": "korki#kth.se",
"date": "2015-09-07T22:06:51Z"
},
"message": "Added LaTex template and instructions",
"tree": {
"sha": "askdljaskdlajsklda",
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915"
},
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"comment_count": 0,
"verification": {
"verified": "False",
"reason": "unsigned",
"signature": "None",
"payload": "None"
}
},
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"html_url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"comments_url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"author": {
"login": "korki",
"id": 999,
"node_id": "askljdklas==",
"type": "User",
"site_admin": "None"
},
"committer": {
"login": "korki",
"id": 999,
"node_id": "askljdklas==",
"type": "User",
"site_admin": "None"
},
"parents": [{
"sha": "asdaskldjasdklsjl",
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"html_url": "https://gits-15.ds.sd.se/04dd5b226dda1915"
}]
}, {
"sha": "kasdjklasdjklas",
"node_id": "sdklasjdklasjkl",
"commit": {
"author": {
"name": "korki",
"email": "korki#kth.se",
"date": "2015-08-31T10:45:24Z"
},
"committer": {
"name": "korki",
"email": "korki#kth.se",
"date": "2015-08-31T10:45:24Z"
},
"message": "Update README.md",
"tree": {
"sha": "askldjkasldjklas",
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915"
},
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"comment_count": 0,
"verification": {
"verified": "None",
"reason": "unsigned",
"signature": "None",
"payload": "None"
}
},
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"html_url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"comments_url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"author": {
"login": "korki",
"id": 999,
"node_id": "dkasdasdnas==",
"type": "User",
"site_admin": "None"
},
"committer": {
"login": "korki",
"id": 999,
"node_id": "askldaskldja==",
"type": "User",
"site_admin": "None"
},
"parents": [{
"sha": "dlkasdjklas;dlkjas;",
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"html_url": "https://gits-15.ds.sd.se/04dd5b226dda1915"
}]
}, {
"sha": "dsagadsgsgdsa",
"node_id": "sdagfsdgsd",
"commit": {
"author": {
"name": "korki",
"email": "korki#kth.se",
"date": "2015-08-31T10:44:42Z"
},
"committer": {
"name": "korki",
"email": "korki#kth.se",
"date": "2015-08-31T10:44:42Z"
},
"message": "Initial commit",
"tree": {
"sha": "asdasddasdas",
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915"
},
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"comment_count": 0,
"verification": {
"verified": "None",
"reason": "unsigned",
"signature": "None",
"payload": "None"
}
},
"url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"html_url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"comments_url": "https://gits-15.ds.sd.se/04dd5b226dda1915",
"author": {
"login": "korki",
"id": 999,
"node_id": "kjklklj==",
"type": "User",
"site_admin": "None"
},
"committer": {
"login": "korki",
"id": 999,
"node_id": "jhkjkj==",
"gravatar_id": "",
"type": "User",
"site_admin": "None"
},
"parents": []
}]
}
That error means it is reading a blank file. Make sure you are reading the file you think you are reading.
EDIT: Another possibility is that you have already read through all the lines of the file. If you read through all the lines and try to read the file, it will appear as a blank file.
I had the exact same issue. I used a powershell script to create a json file and I tried to read the file from another python script but I kept getting the same error as you, even though the JSON file was properly formatted. The issue was I was using a powershell command, "Out-File" Instead, I used Set-Content and it fixed the issue. I believe it was an encoding difference between the commands. Maybe look at how you created the JSON file and the encoding used. I know this is late but I'll share anyways just in case anyone else is having the same issue.
I am trying to import the Grafana dashboard using HTTP API by following Grafana
Grafana Version: 5.1.3
OS -Windows 10
This is what i tried
curl --user admin:admin "http://localhost:3000/api/dashboards/db" -X POST -H "Content-Type:application/json;charset=UTF-8" --data-binary #c:/Users/Mahadev/Desktop/Dashboard.json
and
Here is my python code
import requests
headers = {
'Content-Type': 'application/json;charset=UTF-8',
}
data = open('C:/Users/Mahadev/Desktop/Dashboard.json', 'rb').read()
response = requests.post('http://admin:admin#localhost:3000/api/dashboards/db', headers=headers, data=data)
print (response.text)
And output of both is:
[{"fieldNames":["Dashboard"],"classification":"RequiredError","message":"Required"}]
It is asking for root property called dashboard in my json payload. Can anybody suggest me how to use that porperty and what data should i provide.
If any one want to dig more here are some links.
https://github.com/grafana/grafana/issues/8193
https://github.com/grafana/grafana/issues/2816
https://github.com/grafana/grafana/issues/8193
https://community.grafana.com/t/how-can-i-import-a-dashboard-from-a-json-file/669
https://github.com/grafana/grafana/issues/273
https://github.com/grafana/grafana/issues/5811
https://stackoverflow.com/questions/39968111/unable-to-post-to-grafana-using-python3-module-requests
https://stackoverflow.com/questions/39954475/post-request-works-in-postman-but-not-in-python/39954514#39954514
https://www.bountysource.com/issues/44431991-use-api-to-import-json-file-error
https://github.com/grafana/grafana/issues/7029
Maybe you should try to download your dashboard from the API so you will a "proper" json model to push after?
You can download it with the following command :
curl -H "Authorization: Bearer $TOKEN" https://grafana.domain.tld/api/dashboards/uid/$DASHBOARD_UID
An other way to do it , you can download a dashboard JSON on grafana website => grafana.com/dashboards and try to upload it with your current code? ;)
The dashboard field contain everything that will be display, alerts, graph etc....
Here is an example of dashboard.json :
{
"meta": {
"type": "db",
"canSave": true,
"canEdit": true,
"canAdmin": false,
"canStar": true,
"slug": "status-app",
"url": "/d/lOy3lIImz/status-app",
"expires": "0001-01-01T00:00:00Z",
"created": "2018-06-04T11:40:20+02:00",
"updated": "2018-06-14T17:51:23+02:00",
"updatedBy": "jean",
"createdBy": "jean",
"version": 89,
"hasAcl": false,
"isFolder": false,
"folderId": 0,
"folderTitle": "General",
"folderUrl": "",
"provisioned": false
},
"dashboard": {
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 182,
"links": [],
"panels": [
{
"alert": {
"conditions": [
{
"evaluator": {
"params": [
1
],
"type": "lt"
},
"operator": {
"type": "and"
},
"query": {
"params": [
"A",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "alerting",
"frequency": "60s",
"handler": 1,
"name": "Status of alert",
"noDataState": "alerting",
"notifications": [
{
"id": 7
}
]
},
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Collectd",
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 0
},
"id": 4,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "Status",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "processes_processes",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT mean(value) FROM \"processes_processes\" WHERE (\"instance\" = '' AND \"host\" = 'Webp01') AND $timeFilter GROUP BY time($interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"value"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "instance",
"operator": "=",
"value": ""
},
{
"condition": "AND",
"key": "host",
"operator": "=",
"value": "Webp01"
}
]
}
],
"thresholds": [
{
"colorMode": "critical",
"fill": true,
"line": true,
"op": "lt",
"value": 1
}
],
"timeFrom": null,
"timeShift": null,
"title": "Status of ",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"refresh": "5m",
"schemaVersion": 16,
"style": "dark",
"tags": [
"web",
"nodejs"
],
"templating": {
"list": []
},
"time": {
"from": "now/d",
"to": "now"
},
"timepicker": {
"hidden": false,
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "Status APP",
"uid": "lOy3lIImz",
"version": 89
},
}
Edit:
Here is a JSON snipper for templating your dashboard :
"templating": {
"list": [
{
"allValue": null,
"current": {
"text": "PRD_Web01",
"value": "PRD_Web01"
},
"datasource": "Collectd",
"hide": 0,
"includeAll": false,
"label": null,
"multi": false,
"name": "host",
"options": [],
"query": "SHOW TAG VALUES WITH KEY=host",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "sda",
"value": "sda"
},
"datasource": "Collectd",
"hide": 0,
"includeAll": false,
"label": null,
"multi": false,
"name": "device",
"options": [],
"query": "SHOW TAG VALUES FROM \"disk_read\" WITH KEY = \"instance\"",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
As I read your answer, I guess you will be OK with this ;). I will try to keep a better eye on this thread
Can you show how your dashboard json looks like ? The json MUST contain a key dashboard in it with all the details inside its value like the following:
{
"dashboard": {
"id": null,
"uid": null,
"title": "Production Overview",
"tags": [ "templated" ],
"timezone": "browser",
"schemaVersion": 16,
"version": 0
},
"folderId": 0,
"overwrite": false
}
I'm wondering if there is a way to display the number of commits/branches/etc made in a repository to a website. I'm using Flask to for the website with jinja2 on an apache2 server and instead of displaying the number through hard coding it I would like it to update the number automatically whenever new commits/branches are made.
You can use the GitHub API to do this.
For example, you can request the amount of commits a repository has by requesting (GET) /repos/:owner/:repo/commits.
An example of a response is as follows (from the API documentation):
[
{
"url": "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e",
"sha": "6dcb09b5b57875f334f61aebed695e2e4193db5e",
"html_url": "https://github.com/octocat/Hello-World/commit/6dcb09b5b57875f334f61aebed695e2e4193db5e",
"comments_url": "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e/comments",
"commit": {
"url": "https://api.github.com/repos/octocat/Hello-World/git/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e",
"author": {
"name": "Monalisa Octocat",
"email": "support#github.com",
"date": "2011-04-14T16:00:49Z"
},
"committer": {
"name": "Monalisa Octocat",
"email": "support#github.com",
"date": "2011-04-14T16:00:49Z"
},
"message": "Fix all the bugs",
"tree": {
"url": "https://api.github.com/repos/octocat/Hello-World/tree/6dcb09b5b57875f334f61aebed695e2e4193db5e",
"sha": "6dcb09b5b57875f334f61aebed695e2e4193db5e"
},
"comment_count": 0,
"verification": {
"verified": true,
"reason": "valid",
"signature": "-----BEGIN PGP MESSAGE-----\n...\n-----END PGP MESSAGE-----",
"payload": "tree 6dcb09b5b57875f334f61aebed695e2e4193db5e\n..."
}
},
"author": {
"login": "octocat",
"id": 1,
"avatar_url": "https://github.com/images/error/octocat_happy.gif",
"gravatar_id": "",
"url": "https://api.github.com/users/octocat",
"html_url": "https://github.com/octocat",
"followers_url": "https://api.github.com/users/octocat/followers",
"following_url": "https://api.github.com/users/octocat/following{/other_user}",
"gists_url": "https://api.github.com/users/octocat/gists{/gist_id}",
"starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/octocat/subscriptions",
"organizations_url": "https://api.github.com/users/octocat/orgs",
"repos_url": "https://api.github.com/users/octocat/repos",
"events_url": "https://api.github.com/users/octocat/events{/privacy}",
"received_events_url": "https://api.github.com/users/octocat/received_events",
"type": "User",
"site_admin": false
},
"committer": {
"login": "octocat",
"id": 1,
"avatar_url": "https://github.com/images/error/octocat_happy.gif",
"gravatar_id": "",
"url": "https://api.github.com/users/octocat",
"html_url": "https://github.com/octocat",
"followers_url": "https://api.github.com/users/octocat/followers",
"following_url": "https://api.github.com/users/octocat/following{/other_user}",
"gists_url": "https://api.github.com/users/octocat/gists{/gist_id}",
"starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/octocat/subscriptions",
"organizations_url": "https://api.github.com/users/octocat/orgs",
"repos_url": "https://api.github.com/users/octocat/repos",
"events_url": "https://api.github.com/users/octocat/events{/privacy}",
"received_events_url": "https://api.github.com/users/octocat/received_events",
"type": "User",
"site_admin": false
},
"parents": [
{
"url": "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e",
"sha": "6dcb09b5b57875f334f61aebed695e2e4193db5e"
}
]
}
]
Since you're using an API, you can request it each time the page is loaded, in a sense, automatically updating it.