Develop Reference

How can I transform or query JSON data in my Github Workflow?

I am using the octokit/request-action action in my Github Workflow to obtain information about a certain Github Release via its tag name:
- name: Get Release Info
uses: octokit/request-action#v2.x
id: release
with:
route: GET /repos/{org_repo}/releases/tags/{tag}
The response JSON structure contains information about all of the assets attached to the release. I need to use this information to build a list of download URLs for all the release artifacts and pass that into a Python script that builds a discord notification.
Using the Github Workflow YAML, how do I "query" the JSON data (similar to JSONata) to obtain a list of all the browser_download_url values inside the assets array? The data returned looks like this (trimmed):
{
"url": "https://api.github.com/repos/octocat/Hello-World/releases/1",
"id": 1,
"assets": [
{
"url": "https://api.github.com/repos/octocat/Hello-World/releases/assets/1",
"browser_download_url": "https://github.com/octocat/Hello-World/releases/download/v1.0.0/example1.zip",
"id": 1
},
{
"url": "https://api.github.com/repos/octocat/Hello-World/releases/assets/2",
"browser_download_url": "https://github.com/octocat/Hello-World/releases/download/v1.0.0/example2.zip",
"id": 2
}
]
}
The end result I want is a way to pass the two download URLs above to my script like so (using a separate step in my workflow):
python discord_notification.py "https://github.com/octocat/Hello-World/releases/download/v1.0.0/example1.zip" "https://github.com/octocat/Hello-World/releases/download/v1.0.0/example2.zip"
(Exact syntax can vary; the above snippet is just an example)
It's possible that what I want to do just can't be achieved in the workflow YAML itself. If that's the case, I'd be OK with a solution that involves passing all or part of the response JSON to the Python script and use Python itself to parse the JSON data. I just don't know if Bash adds a layer of complexity that will make passing a multi-line response string as a parameter difficult.

Apache Spark / PySpark, defining custom JSON Schema for Dynamic Keys

I have a bunch of JSON files, and suppose each have the following structure:
{
"fields": {
"name": "Bob",
"key": "bob"
},
"results": {
"bob": { ... }
}
}
Where by some unfortunate reason, while the structure of the JSON is fairly consistent, there is one dynamic key under "results". Defining the schema for under the fields is fairly straight-forward to me.
So, for several JSON files, the final schema might be:
fieldSchema = StructField(...)
resultSchema = StructField("results", StructType([StructField("bob", ...)]))
finalSchema = StructType([fieldSchema, resultsSchema])
Where the problem is this line: StructField("bob", ...)
Obviously, bob is not the key I'm looking for. This name for the StructField would ideally be some kind of wildcard character, regex pattern, or worst case, some dynamic field based on other fields.
I'm a newbie to Spark and have been scouring the documentation and historical StackOverflow posts, but I've been unable to find anything.
Long story short, I want to be able to pass some kind of wide net for the name parameter in StructField to encompass a variety of different keys, similar to a regex pattern.

Google Docs API programmatically adding a table of content

I have a python script which does some analysis and output the results as text (paragraphs) on a Google Doc. I know how to insert text, update paragraph and text style through batchUpdate.
doc_service.documents().batchUpdate(documentId=<ID>,body={'requests': <my_request>}).execute()
where, for instance, "my_request" takes the form of something like:
request = [
{
"insertText": {
"location": {
"index": <index_position>,
"segmentId": <id>
},
"text": <text>
}
},
{
"updateParagraphStyle": {
"paragraphStyle": {
"namedStyleType": <paragraph_type>
},
"range": {
"segmentId": <id>,
"startIndex": <index_position>,
"endIndex": <index_position>
},
"fields": "namedStyleType"
}
},
]
However, once the script is done updating the table, it would be fantastic if a table of content could be added at the top of the document.
However, I am very new to Google Docs API and I am not entirely sure how to do that. I know I should use "TableOfContents" as a StructuralElement. I also know this option currently does not update automatically after each modification brought to the document (this is why I would like to create it AFTER the document has finished updating and place it at the top of the document).
How to do this with python? I am unclear where to call "TableOfContents" in my request.
Thank you so very much!

After your comment, I was able to understand better what you are desiring to do, but I came across these two Issue Tracker's posts:
Add the ability to generate and update the TOC of a doc.
Geting a link to heading paragraph.
These are well-known feature requests that unfortunately haven't been implemented yet. You can hit the ☆ next to the issue number in the top left on this page as it lets Google know more people are encountering this and so it is more likely to be seen faster.
Therefore, it's not possible to insert/update a table of contents programmatically.

Validate object values against yaml configuration

I have an application where a nested Python dictionary is created based on a JSON document that I get as a response from an API. Example:
colleagues = [
{ "name": "John",
"skills": ["python", "java", "scala"],
"job": "developer"
},
{ "name": "George",
"skills": ["c", "go", "nodejs"],
"job": "developer"
}]
This dictionary can have many more nested levels.
What I want to do is let the user define their own arbitrary conditions (e.g. in order to find colleagues that have "python" among their skills, or whose name is "John") in a YAML configuration file, which I will use to check against the Python dictionary.
I thought about letting them configure that in the following manner in the YAML file, but this would require using exec(), which I want to avoid for security reasons:
constraints:
- "python" in colleagues[x]["skills"]
- colleagues[x]["name"] == "John"
What other options are there for such a problem, so that the user can specify their own constraints for the dictionary values? Again, the dictionary above is just an example. The actual one is much larger in size and nesting levels.

You could use a Lucene query parser to convert queries like "skill:python" and "name:John" to executable predicate functions, and then filter your list of colleagues using those predicates. Googling for "python lucene parser" will turn up several parsing options.

What is the best way to search millions of JSON files?

I've very recently picked up programming in Python and am working on creating a database.
I've already worked out extracting all these files from their source so they are all in a directory on my computer.
All of these files are structured the same way and what I want to do is search these multidimensional dictionaries and locate the value for a specific set of keys.
These json files are all structured similarly,
{
"userid": 34535367,
"result": {
"list": [
{
"name": 264,
"age": 64,
"id": 456345345
},
{
"name": 263,
"age": 42,
"id": 364563463456
}
]
}
}
In my case, I would like to search for the "name" key and return the relevant data(quality, id and the original userid) for the thousands of names just like it from my millions of JSON files.
Basically I'm very new at this and the little programming knowledge I have is in Python. I'm happy to start learning whatever I need to, but I'm not sure which direction to go.

If your goal is to create a database, then you should look on how databases work and solve the same problem you are trying to solve right now :)
NoSQL databases (like mangodb) work also with json documents and implements most likely a whole set of tools to search and filter documents.
Now to answer your question, there is no quick way to do so unless you do some preprocessing, meaning that you store different information about the data (called metadata).
This is a huge subject and I don't have enough expertise to give you all the answers, but I can give you a simple tip: Use indexes.
An index is a sorted key/value map where for every value, we store the documents that contains that value (or the file + position of the Json document) . For example an index for the name property would like this:
{
263: ('jsonfile10.json', '0')
264: ('jsonfile10.json', '30'),
# The json document can be found on the jsonfile10.json file on line 30
}
By keeping an index for the most queried values, you can turn a linear time search into a logarithmic time search not to mention that inserting a new document is much faster. in your case, you seems to only need an index on the name field.
Creating/updating the index is done when you insert, update or remove a document. Using a balanced binary tree can accelerate the updates on the index.

As a suggestion, why don't you just process all the incoming files and insert the data into a database? You will have a toolset to query that database. SQLite for example will do (as well as any other more sophisticated database):
http://www.sqlite.org/
http://docs.python.org/2/library/sqlite3.html
Simple other solution might be to build a file mapping name_id to /file/path. Then you can logarithmically do a binary search by the name id. But I'd still advise using a proper database as maintaining the index will be more cumbersome than doing some inserts/deletes.