Develop Reference

Expanding a Scribunto module that doesn't have a function

I want to get the return value of this Wikimedia Scribunto module in Python. Its source code is roughly like this:
local Languages = {}
Languages = {
["aa"] = {
name = "afarština",
dir = "ltr",
name_attr_gen_pl = "afarských"
},
-- More languages...
["zza"] = {
name = "zazaki",
dir = "ltr"
}
}
return Languages
In the Wiktextract library, there is already Python code to accomplish similar tasks:
def expand_template(sub_domain: str, text: str) -> str:
import requests
# https://www.mediawiki.org/wiki/API:Expandtemplates
params = {
"action": "expandtemplates",
"format": "json",
"text": text,
"prop": "wikitext",
"formatversion": "2",
}
r = requests.get(f"https://{sub_domain}.wiktionary.org/w/api.php",
params=params)
data = r.json()
return data["expandtemplates"]["wikitext"]
This works for languages like French because there the Scribunto module has a well-defined function that returns a value, as an example here:
Scribunto module:
p = {}
function p.affiche_langues_python(frame)
-- returns the needed stuff here
end
The associated Python function:
def get_fr_languages():
# https://fr.wiktionary.org/wiki/Module:langues/analyse
json_text = expand_template(
"fr", "{{#invoke:langues/analyse|affiche_langues_python}}"
)
json_text = json_text[json_text.index("{") : json_text.index("}") + 1]
json_text = json_text.replace(",\r\n}", "}") # remove tailing comma
data = json.loads(json_text)
lang_data = {}
for lang_code, lang_name in data.items():
lang_data[lang_code] = [lang_name[0].upper() + lang_name[1:]]
save_json_file(lang_data, "fr")
But in our case we don't have a function to call.
So if we try:
def get_cs_languages():
# https://cs.wiktionary.org/wiki/Modul:Languages
json_text = expand_template(
"cs", "{{#invoke:Languages}}"
)
print(json_text)
we get <strong class="error"><span class="scribunto-error" id="mw-scribunto-error-0">Chyba skriptu: Musíte uvést funkci, která se má zavolat.</span></strong> usage: get_languages.py [-h] sub_domain lang_code get_languages.py: error: the following arguments are required: sub_domain, lang_code. (Translated as "You have to specify a function you want to call. But when you enter a function name as a parameter like in the French example, it complains that that function does not exist.)
What could be a way to solve this?

The easiest and most general way is to get the return value of the module as JSON and parse it in Python.
Make another module that exports a function dump_as_json that takes the name of the first module as a frame argument and returns the first module as JSON. In Python, expand {{#invoke:json module|dump_as_json|Module:module to dump}} using the expandtemplates API and parse the return value of the module invocation as JSON with json.loads(data["expandtemplates"]["wikitext"]).
Text of Module:json module (call it what you want):
return {
dump_as_json = function(frame)
local module_name = frame.args[0]
local json_encode = mw.text.jsonEncode
-- json_encode = require "Module:JSON".toJSON
return json_encode(require(module_name))
end
}
With pywikibot:
from pywikibot import Site
site = Site(code="cs", fam="wiktionary")
languages = json.loads(site.expand_text("{{#invoke:json module|dump_as_json|Module:module to dump}}")
If you get the error Lua error: Cannot pass circular reference to PHP, this means that at least one of the tables in Module:module to dump is referenced by another table more than once, like if the module was
local t = {}
return { t, t }
To handle these tables, you will have to get a pure-Lua JSON encoder function to replace mw.text.jsonEncode, like the toJSON function from Module:JSON on English Wiktionary.
One warning about this method that is not relevant for the module you are trying to get: string values in the JSON will only be accurate if they were NFC-normalized valid UTF-8 with no special ASCII control codes (U+0000-U+001F excluding tab U+0009 and LF U+000A) when they were returned from Module:module to dump. As on a wiki page, the expandtemplates API will replace ASCII control codes and invalid UTF-8 with the U+FFFD character, and will NFC-normalize everything else. That is, "\1\128e" .. mw.ustring.char(0x0301) would be modified to the equivalent of mw.ustring.char(0xFFFD, 0xFFFD, 0x00E9). This doesn't matter in most cases (like if the table contains readable text), but if it did matter, the JSON-encoding module would have to output JSON escapes for non-NFC character sequences and ASCII control codes and find some way to encode invalid UTF-8.
If, like the module you are dumping, Module:module to dump is a pure table of literal values with no references to other modules or to Scribunto-only global values, you could also get its raw wikitext with the Revisions API and parse it in Lua on your machine and pass it to Python. I think there is a Python extension that allows you to directly use a Lua state in Python.
Running a module with dependencies on the local machine is not possible unless you go to the trouble of setting up the full Scribunto environment on your machine, and figuring out a way to download the module dependencies and make them available to the Lua state. I have sort of done this myself, but it isn't necessary for your use case.

How to replace email domain misspellings

I manage a booking system where people enter their emails to book appointments. Unfortunately, many of the email addresses have typos, especially in the top-level domain. I'd like to implement a function to identify when the domain should be .com, which should be all cases where it starts with ".c".
I wrote the function and tested the individual lines, which seem to work. But when I run the function on a set of emails, it only returns the email with no ".c", meaning it only returns 'my_name_is#orange.org'. The ideal output would be:
['me.you#perp.com',
'appleorange#msn.edu.com',
'laughable#gmail.com',
'notfunny#pointdexter.com',
'very.latin...word#word.nutherwork.com',
'very.latin..word#word.nutherwork.com',
'my_name_is#orange.org']
Any help would be appreciated!
emails = ['me.you#perp.comp',
'appleorange#msn.edu.cot',
'laughable#gmail.copl',
'notfunny#pointdexter.com',
'very.latin...word#word.nutherwork.com',
'very.latin..word#word.nutherwork.cnm',
'my_name_is#orange.org']
def domain(emails):
for email in emails:
parse_mail = email.split('.')
parse_mail = parse_mail[-1]
if parse_mail[0] == "c":
email = email.replace(parse_mail,"com")
pass
return email
print(domain(emails))

It returns the email without "c" because that's the last one in the array.
Your routine "domain" only returns the last email in the array. If you put the org address higher up, it'll return an email with a "c".

I expect what you're looking for is:
def domain(emails):
for i in range(len(emails)):
parse_mail = emails[i].split('.')
if parse_mail[-1][0] == "c":
parse_mail = emails[i].split('.')[:-1]
parse_mail.append("com")
correct = '.'.join(parse_mail)
emails[i] = correct
pass
return emails
print(domain(emails))
But as Sembei Norimaki, mentioned - this will also auto-correct (erroneously) other extensions beginning with a 'c'.

Updating a custom field using ASANA Python API

I'm trying to update the values of custom fields in my Asana list. I'm using the Official Python client library for the Asana API v1.
My code currently looks like this;
project = "Example Project"
keyword = "Example Task"
print "Logging into ASANA"
api_key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
client = asana.Client.basic_auth(api_key)
me = client.users.me()
all_projects = next(workspace for workspace in me['workspaces'])
projects = client.projects.find_by_workspace(all_projects['id'])
for project in projects:
if 'Example Project' not in project['name']:
continue
print "Project found."
print "\t"+project['name']
print
tasks = client.tasks.find_by_project(project['id'], {"opt_fields":"this.name,custom_fields"}, iterator_type=None)
for task in tasks:
if keyword in task['name']:
print "Task found:"
print "\t"+str(task)
print
for custom_field in task['custom_fields']:
custom_field['text_value'] = "New Data!"
print client.tasks.update(task['id'], {'data':task})
But when I run the code, the task doesn't update. The return of print client.tasks.update returns all the details of the task, but the custom field has not been updated.

I think the problem is that our API is not symmetrical with respect to custom fields... which I kind of find to be a bummer; it can be a real gotcha in cases like this. Rather than being able to set the value of a custom field within the block of values as you're doing above, which is intuitive, you have to set them with a key:value dictionary-like setup of custom_field_id:new_value - not as intuitive, unfortunately. So above, where you have
for custom_field in task['custom_fields']:
custom_field['text_value'] = "New Data!"
I think you'd have to do something like this:
new_custom_fields = {}
for custom_field in task['custom_fields']:
new_custom_fields[custom_field['id']] = "New Data!"
task['custom_fields'] = new_custom_fields
The goal is to generate JSON for the POST request that looks something like
{
"data": {
"custom_fields":{
"12345678":"New Data!"
}
}
}
As a further note, the value should be the new text string if you have a text custom field, a number if it's a number custom field, and the ID of the enum_options choice (take a look at the third example under this header on our documentation site) if it's an enum custom field.

Thanks to Matt, I got to the solution.
new_custom_fields = {}
for custom_field in task['custom_fields']:
new_custom_fields[custom_field['id']] = "New Data!"
print client.tasks.update(task['id'], {'custom_fields':new_custom_fields})
There were two problems in my original code, the first was that I was trying to treat the API symmetrically and this was identified and solved by Matt. The second was that I was trying to update in an incorrect format. Note the difference between client.tasks.update in my original and updated code.

Filtering statistics from facebook ads api using the python sdk

I am trying to use Facebook's ads-api to get data about advertising accounts/campaigns/etc within a specified time range.
Up until now I managed to get overall information (added below) using the official python sdk ,
but I can't figure out how to insert the time filter condition.
The answer is probably here under "Filtering results", but I don't understand how to translate what they are doing there to python...
https://developers.facebook.com/docs/reference/ads-api/adstatistics/v2.2
I would really appreciate any help you can provide,
Thanks!
This is the relevant module (I think) from the official python sdk project:
https://github.com/facebook/facebook-python-ads-sdk/blob/master/facebookads/objects.py
My current code is:
from facebookads.session import FacebookSession
from facebookads.api import FacebookAdsApi
from facebookads import objects
from facebookads.objects import (
AdUser,
AdCampaign,
)
my_app_id = 'APP_ID'
my_app_secret = 'AP_SECRET'
my_access_token = 'ACCESS_TOKEN'
my_session = FacebookSession(my_app_id, my_app_secret, my_access_token)
my_api = FacebookAdsApi(my_session)
FacebookAdsApi.set_default_api(my_api)
me = objects.AdUser(fbid='me')
my_accounts = list(me.get_ad_accounts())
my_account=my_accounts[1]
print(">>> Campaign Stats")
for campaign in my_account.get_ad_campaigns(fields=[AdCampaign.Field.name]):
for stat in campaign.get_stats(fields=[
'impressions',
'clicks',
'spent',
'unique_clicks',
'actions',
]):
print(campaign[campaign.Field.name])
for statfield in stat:
print("\t%s:\t\t%s" % (statfield, stat[statfield]))
and the output I get is (All caps and xxxx are mine):
Campaign Stats
CAMPAIGN_NAME1
impressions: xxxx
unique_clicks: xxxx
clicks: xxxx
actions: {u'mobile_app_install': xxxx, u'app_custom_event': xxxx, u'app_custom_event.fb_mobile_activate_app': xxx}
spent: xxxx
CAMPAIGN_NAME2
impressions: xxxx
unique_clicks: xxxx
clicks: xxxx
actions: {XXXX}
spent: xxxx

The get_stats() method has an additional parameter named params where you can pass in start_time and/or end_time.
params_data = {
'start_time': 1415134405,
}
stats = campaign.get_stats(
params=params_data,
fields=[
'impressions',
'clicks',
...
]
)
for stat in stats:
...
The API accepts a number of different parameters documented here: https://developers.facebook.com/docs/reference/ads-api/adstatistics
More optional reading
The reason for both a params parameter and fields parameter requires a bit of explanation. Feel free to ignore this if you're not interested. :)
The implementation for the params parameter basically just constructs the query string for the API call:
params['start_time'] = 1415134405
creates:
?start_time=1415134405
The Ads API endpoints generally accept a fields parameter to define what data you want to return:
?fields=impressions,clicks&start_time=1415134405
which you have correctly defined, but because it's just fields in the query string, you could also technically do this:
params['fields'] = 'impressions,clicks'
The fields parameter in get_stats() (and other read methods) is simply an easy way to define this fields parameter. The implementation looks something like this:
def remote_read(self, params=[], fields=[]):
...
params['fields'] = fields
...

The "get_stats" method is deprecated in the V2.4 version of the API.
Instead, "get_insights" method should be used. The parameters for that method are liste on the page below:
https://developers.facebook.com/docs/marketing-api/insights/v2.5
From the page above, the replacement for the "start_time" and "end_time" is the "time_ranges" attribute. Example:
"time_ranges": {'since': '2015-01-01', 'until': '2015-01-31'}

Regex to capture the codes between specific function

I have been experimenting with python re, trying to capture specific variables between specific functions. To illustrate let me give an example of file contents of a php file :-
public function executeSomething ()
{
$this->title = 'Edit something';
$this->action = 'Edit';
$this->headerTitle = 'Edit something';
return true;
}
public function executeSomethingEdit ()
{
if (strlen ($this->somethingElse) > 0)
{
$this->titleText = "Update";
$title = 'Edit something';
}
else
{
$this->titleText = "Create";
$title = 'Create something';
}
$this->title = $title;
$this->headerTitle = $title;
$this->formTitle = 'Something details'
return true;
}
What the python script needs to do now is iterate through this file in search for functions that starts with 'public function execute' and get the content within the braces i.e { }. I have already came up with python code to achieve this i.e :-
r = re.compile(r"public function execute(.*?)\(\).*?{(.*?)}", re.DOTALL)
The problem occurs when I have a validation within the function i.e if else statement such as the one in the function executeSomethingEdit. The script doesn't takes into account whatever codes below the if statements closing braces '}'. Therefore I need to further enhance the python code to include the function declaration below i.e something like this :-
r = re.compile(r"public function execute(.*?)\(\).*?{(.*?)}.*?public function", re.DOTALL)
At the moment this code is not working/producing the result that I wanted. I need to used python's re specifically because i need to further analyse the content of {(.*?)}. I'm very new to python so I hope someone could direct me in the right direction or at least tell me what I'm doing wrong. Thanks in advance.

If the input PHP has no bugs and has consistent indentation, you could check for a non-space character before the closing brace.
r = re.compile(r'public function execute(.*?)\(\).*?{(.*?)[^ ]}', re.DOTALL)