Develop Reference

Expanding a Scribunto module that doesn't have a function

I want to get the return value of this Wikimedia Scribunto module in Python. Its source code is roughly like this:
local Languages = {}
Languages = {
["aa"] = {
name = "afarština",
dir = "ltr",
name_attr_gen_pl = "afarských"
},
-- More languages...
["zza"] = {
name = "zazaki",
dir = "ltr"
}
}
return Languages
In the Wiktextract library, there is already Python code to accomplish similar tasks:
def expand_template(sub_domain: str, text: str) -> str:
import requests
# https://www.mediawiki.org/wiki/API:Expandtemplates
params = {
"action": "expandtemplates",
"format": "json",
"text": text,
"prop": "wikitext",
"formatversion": "2",
}
r = requests.get(f"https://{sub_domain}.wiktionary.org/w/api.php",
params=params)
data = r.json()
return data["expandtemplates"]["wikitext"]
This works for languages like French because there the Scribunto module has a well-defined function that returns a value, as an example here:
Scribunto module:
p = {}
function p.affiche_langues_python(frame)
-- returns the needed stuff here
end
The associated Python function:
def get_fr_languages():
# https://fr.wiktionary.org/wiki/Module:langues/analyse
json_text = expand_template(
"fr", "{{#invoke:langues/analyse|affiche_langues_python}}"
)
json_text = json_text[json_text.index("{") : json_text.index("}") + 1]
json_text = json_text.replace(",\r\n}", "}") # remove tailing comma
data = json.loads(json_text)
lang_data = {}
for lang_code, lang_name in data.items():
lang_data[lang_code] = [lang_name[0].upper() + lang_name[1:]]
save_json_file(lang_data, "fr")
But in our case we don't have a function to call.
So if we try:
def get_cs_languages():
# https://cs.wiktionary.org/wiki/Modul:Languages
json_text = expand_template(
"cs", "{{#invoke:Languages}}"
)
print(json_text)
we get <strong class="error"><span class="scribunto-error" id="mw-scribunto-error-0">Chyba skriptu: Musíte uvést funkci, která se má zavolat.</span></strong> usage: get_languages.py [-h] sub_domain lang_code get_languages.py: error: the following arguments are required: sub_domain, lang_code. (Translated as "You have to specify a function you want to call. But when you enter a function name as a parameter like in the French example, it complains that that function does not exist.)
What could be a way to solve this?

The easiest and most general way is to get the return value of the module as JSON and parse it in Python.
Make another module that exports a function dump_as_json that takes the name of the first module as a frame argument and returns the first module as JSON. In Python, expand {{#invoke:json module|dump_as_json|Module:module to dump}} using the expandtemplates API and parse the return value of the module invocation as JSON with json.loads(data["expandtemplates"]["wikitext"]).
Text of Module:json module (call it what you want):
return {
dump_as_json = function(frame)
local module_name = frame.args[0]
local json_encode = mw.text.jsonEncode
-- json_encode = require "Module:JSON".toJSON
return json_encode(require(module_name))
end
}
With pywikibot:
from pywikibot import Site
site = Site(code="cs", fam="wiktionary")
languages = json.loads(site.expand_text("{{#invoke:json module|dump_as_json|Module:module to dump}}")
If you get the error Lua error: Cannot pass circular reference to PHP, this means that at least one of the tables in Module:module to dump is referenced by another table more than once, like if the module was
local t = {}
return { t, t }
To handle these tables, you will have to get a pure-Lua JSON encoder function to replace mw.text.jsonEncode, like the toJSON function from Module:JSON on English Wiktionary.
One warning about this method that is not relevant for the module you are trying to get: string values in the JSON will only be accurate if they were NFC-normalized valid UTF-8 with no special ASCII control codes (U+0000-U+001F excluding tab U+0009 and LF U+000A) when they were returned from Module:module to dump. As on a wiki page, the expandtemplates API will replace ASCII control codes and invalid UTF-8 with the U+FFFD character, and will NFC-normalize everything else. That is, "\1\128e" .. mw.ustring.char(0x0301) would be modified to the equivalent of mw.ustring.char(0xFFFD, 0xFFFD, 0x00E9). This doesn't matter in most cases (like if the table contains readable text), but if it did matter, the JSON-encoding module would have to output JSON escapes for non-NFC character sequences and ASCII control codes and find some way to encode invalid UTF-8.
If, like the module you are dumping, Module:module to dump is a pure table of literal values with no references to other modules or to Scribunto-only global values, you could also get its raw wikitext with the Revisions API and parse it in Lua on your machine and pass it to Python. I think there is a Python extension that allows you to directly use a Lua state in Python.
Running a module with dependencies on the local machine is not possible unless you go to the trouble of setting up the full Scribunto environment on your machine, and figuring out a way to download the module dependencies and make them available to the Lua state. I have sort of done this myself, but it isn't necessary for your use case.

Python to Java script SendMailRequest with SourceArn and FromArn

I have here a part of a code in Python which is for AWS SendEmailRequest(SES)
response = boto3.client('ses').send_raw_email(
FromArn='response = boto3.client('ses').send_raw_email(
FromArn='arn:aws:ses:us-east-1:123456789012:identity/example.com',
SourceArn='arn:aws:ses:us-east-1:123456789012:identity/example.com',
RawMessage={
'Data': msg
},
)
This is working as expected. My problem is that I also need to have this in my Java script but I'm confused how to incorporate it. I've been trying but it seems to be not working. This is the existing Java script part below:
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
message.writeTo(outputStream);
RawMessage rawMessage = new RawMessage(ByteBuffer.wrap(outputStream.toByteArray()));
SendRawEmailRequest rawEmailRequest = new SendRawEmailRequest(rawMessage)
client.sendRawEmail(rawEmailRequest);
I think the FromArn and SourceArn should be incorporated in the rawMessage or rawEmailRequest but I couldn't make it work. On the top of the code, there are values declared like this:
public class SESEMail {
static final String FROM = "example#web.com";
static final String key = Config.key;
static final String privatekey = Config.privateKey;
static Logger logger = Logger.getLogger(SESEMail.class);
public static Variables variables;
I've been reading this one but still confused with how Java language works. http://javadox.com/com.amazonaws/aws-java-sdk-ses/1.10.29/com/amazonaws/services/simpleemail/model/SendRawEmailRequest.html#getSourceArn()

Cannont take in multiple inputs

i have this code in the google apps script.
function createDocument(invoice_id,cust_name)
{
var TEMPLATE_ID = 'practice_link';
var documentId = DriveApp.getFileById(TEMPLATE_ID).makeCopy().getId();
drivedoc = DriveApp.getFileById(documentId);
drivedoc.setName("Invoice " + invoice_id);
doc = DocumentApp.openById(documentId);
var body = doc.getBody();
body.replaceText('{invoice_id}', invoice_id);
body.replaceText('{cust_name}',cust_name);
drivedoc.setSharing(DriveApp.Access.ANYONE_WITH_LINK, DriveApp.Permission.EDIT);
return "https://docs.google.com/document/d/" + documentId + "/export?format=pdf";
}
function doGet(e) {
var invoice_id = e.parameter.invoice_id;
var cust_name = e.parameter.cust_name;
var url = createDocument(invoice_id,cust_name);
return ContentService.createTextOutput(url);
}
when i use the link :
url?invoice_id=1, it runs properly
but when i try :
url?invoice_id=1&cust_name=customer
it runs but does not make changes for cust_name.
and if i try:
url?cust_name=customer
it shows the error
Exception: Invalid argument: replacement (line 12, file "Code")
after many tries i realised that it is showing this error only for when i use, cust_name=str .
please help me out here.i have many more parameters to pass and m stuck here.
i ll be using this app with python to pass on the parameters

Based on the scenarios you've mentioned, error only occurs when the invoice_id parameter is missing and adding cust_name won't affect the code. Those are hints that the URL you are trying to access is outdated or the script changes you've made is not yet deployed.
url?invoice_id=1, it runs properly
but when i try :
url?invoice_id=1&cust_name=customer
it runs but does not make changes for cust_name.
Also, Exception: Invalid argument: replacement (line 12, file "Code") indicates that the 2nd parameter of body.replaceText(pattern, value) in line 12 is undefined. This part should error too for cust_num if the deployed application is updated and the parameter for cust_num is missing.
Follow these steps on how to deploy script as web app:
At the top right of the script project, click Deploy > New deployment.
Next to "Select type," click Enable deployment types settings > Web app.
Enter the information about your web app in the fields under "Deployment configuration."
Click Deploy.
You can also use the test environments to evaluate your application and to make sure that the output of your project is correct before deploying. See Test a deployment.

With Python Kubernetes client, how to replicate `kubectl create -f` generally?

My Bash script using kubectl create/apply -f ... to deploy lots of Kubernetes resources has grown too large for Bash. I'm converting it to Python using the PyPI kubernetes package.
Is there a generic way to create resources given the YAML manifest? Otherwise, the only way I can see to do it would be to create and maintain a mapping from Kind to API method create_namespaced_<kind>. That seems tedious and error prone to me.
Update: I'm deploying many (10-20) resources to many (10+) GKE clusters.

Update in the year 2020, for anyone still interested in this (since the docs for the python library is mostly empty).
At the end of 2018 this pull request has been merged,
so it's now possible to do:
from kubernetes import client, config
from kubernetes import utils
config.load_kube_config()
api = client.ApiClient()
file_path = ... # A path to a deployment file
namespace = 'default'
utils.create_from_yaml(api, file_path, namespace=namespace)
EDIT: from a request in a comment, a snippet for skipping the python error if the deployment already exists
from kubernetes import client, config
from kubernetes import utils
config.load_kube_config()
api = client.ApiClient()
def skip_if_already_exists(e):
import json
# found in https://github.com/kubernetes-client/python/blob/master/kubernetes/utils/create_from_yaml.py#L165
info = json.loads(e.api_exceptions[0].body)
if info.get('reason').lower() == 'alreadyexists':
pass
else
raise e
file_path = ... # A path to a deployment file
namespace = 'default'
try:
utils.create_from_yaml(api, file_path, namespace=namespace)
except utils.FailToCreateError as e:
skip_if_already_exists(e)

I have written a following piece of code to achieve the functionality of creating k8s resources from its json/yaml file:
def create_from_yaml(yaml_file):
"""
:param yaml_file:
:return:
"""
yaml_object = yaml.loads(common.load_file(yaml_file))
group, _, version = yaml_object["apiVersion"].partition("/")
if version == "":
version = group
group = "core"
group = "".join(group.split(".k8s.io,1"))
func_to_call = "{0}{1}Api".format(group.capitalize(), version.capitalize())
k8s_api = getattr(client, func_to_call)()
kind = yaml_object["kind"]
kind = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', kind)
kind = re.sub('([a-z0-9])([A-Z])', r'\1_\2', kind).lower()
if "namespace" in yaml_object["metadata"]:
namespace = yaml_object["metadata"]["namespace"]
else:
namespace = "default"
try:
if hasattr(k8s_api, "create_namespaced_{0}".format(kind)):
resp = getattr(k8s_api, "create_namespaced_{0}".format(kind))(
body=yaml_object, namespace=namespace)
else:
resp = getattr(k8s_api, "create_{0}".format(kind))(
body=yaml_object)
except Exception as e:
raise e
print("{0} created. status='{1}'".format(kind, str(resp.status)))
return k8s_api
In above function, If you provide any object yaml/json file, it will automatically pick up the API type and object type and create the object like statefulset, deployment, service etc.
PS: The above code doesn't handler multiple kubernetes resources in one file, so you should have only one object per yaml file.

I see what you are looking for. This is possible with other k8s clients available in other languages. Here is an example in java. Unfortunately the python client library does not support that functionality yet. I opened a new feature request requesting the same and you can either choose to track it or contribute yourself :). Here is the link for the issue on GitHub.
The other way to still do what you are trying to do is to use java/golang client and put your code in a docker container.

execute_script throws no exception but file not created

Trying to create a function to streamline some automation.
When the javascript is called with the arguments directly it works perfectly and the file is created.
browser.execute_script("HAR.clear({token: \"abcd\"})")
browser.find_element_by_link_text("B").click()
browser.execute_script("HAR.triggerExport({token: \"abcd\", fileName: \"name_of_file\"}).then(result => {})")
When i try to pass it as a variable, there are no errors but the har file is not created.
Call:
simple_find("B",'\\"name_of_file\\"')
Function:
def simple_find (element, filename):
browser.execute_script("HAR.clear({token: \"abcd\"})")
browser.find_element_by_link_text(element).click()
options = '{token: \\"abcd\\", fileName: '+filename+'}'
ret=browser.execute_script("HAR.triggerExport(arguments[0]).then(result => {});return arguments[0]",options)
print ret
I added the return piece to help debug what is being passed and here is the output:
C:>python firefox-Manage.py
{token: \"abcd\", fileName: \"name_of_file\"}
It looks exactly like the call made earlier with the exception of the file not being created. What am I missing?
java version is: 1.8.0_66
selenium version is: 2.48.2
python version is: 2.7.10
thx

Your options object created from Python looks malformed. There's no reason to surround the values with \\":
options = '{token: \\"abcd\\", fileName: '+filename+'}'
My guess would be you want to pass a dictionary directly to selenium instead of a string:
options = {'token': "abcd", 'fileName': filename}