Develop Reference

How to create and or edit a page with pyWikiBot

The MediaWiki API has an edit function which is available within pywikibot.
According to https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.site.html
the function is called with a page parameter:
editpage(page, summary=None, minor=True, notminor=False, bot=True, recreate=True, createonly=False, nocreate=False, watch=None, **kwargs)[source]¶
A page needs a source to be constructed. I could not find an example for this.
E.g.: How to add something to edit summary when using Pywikibot?
has just one line as the accepted answer and does not use site or page. I am confused.
What is the proper approach do create and or edit a page with pyWikiBot directly using python code? (not script ...)
https://github.com/wikimedia/pywikibot/blob/master/pywikibot/page/init.py#L2328
has the constructor:
"""Page: A MediaWiki page."""
#deprecated_args(defaultNamespace='ns', insite=None)
def __init__(self, source, title='', ns=0):
"""Instantiate a Page object."""
if isinstance(source, pywikibot.site.BaseSite):
if not title:
raise ValueError('Title must be specified and not empty '
'if source is a Site.')
super(Page, self).__init__(source, title, ns)
Which is unfortunately not properly documented and part of the 6000 line init.py file having all the classes.
When trying
newPage=Page(site,pageTitle)
newPage.text=pageContent
I get
AttributeError: APISite instance has no attribute 'edit'
site.edit(newPage,'summary')

The following code works:
from pywikibot.page import Page
newPage=Page(site,pageTitle)
newPage.text=pageContent
newPage.save("summary")
see also https://www.mediawiki.org/wiki/Manual:Pywikibot/Create_your_own_script
It's just unfortunate that the documentation at https://pypi.org/project/pywikibot/ is not using standard python documentation tools.

PRAW Errors updating to new Python Distro

So I am trying to create a bot that cross posts from a sub (r/pics) to (r/polpics) using a bit of code from u/GoldenSights. I upgraded to a new python distro and I get a ton of errors, I don't even know where to begin. Here is the code (formatting off, error lines bold):
Traceback (most recent call last):
File "C:\Users\tonyc\AppData\Local\Programs\Python\Python36-32\Lib\site-
packages\praw\subdump.py", line 84, in <module>
r = praw.Reddit(USERAGENT)
File "C:\Users\tonyc\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\praw\reddit.py", line 150, in __init__
raise ClientException(required_message.format(attribute))
praw.exceptions.ClientException: Required configuration setting 'client_id'
missing.
This setting can be provided in a praw.ini file, as a keyword argument to the `Reddit` class constructor, or as an environment variable.
This seems to be related to USERAGENT setting. I don't think I have that configured right.
USERAGENT = ""
# This is a short description of what the bot does. For example
"/u/GoldenSights' Newsletter bot"
SUBREDDIT = "pics"
# This is the sub or list of subs to scan for new posts.
# For a single sub, use "sub1".
# For multiple subs, use "sub1+sub2+sub3+...".
# For all use "all"
KEYWORDS = ["It looks like this post is about US Politics."]
# Any comment containing these words will be saved.
KEYDOMAINS = []
# If non-empty, linkposts must have these strings in their URL
This is the error line:
print('Logging in')
r = praw.Reddit(USERAGENT) <--here, this is error line 84
r.set_oauth_app_info(APP_ID, APP_SECRET, APP_URI)
r.refresh_access_information(APP_REFRESH)
Also in Reddit.py :
raise ClientException(required_message.format(attribute)) <--- error
praw.exceptions.ClientException: Required configuration setting 'client_id'
missing.
This setting can be provided in a praw.ini file, as a keyword argument to
the `Reddit` class constructor, or as an environment variable.

Firstly, you're going to want to have your API credentials stored externally in your praw.ini file. This makes things a lot more secure, and looks like it might go some way to fixing your issue. Here's what a completed praw.ini file looks like, including the useragent, so try to replicate this.
[DEFAULT]
# A boolean to indicate whether or not to check for package updates.
check_for_updates=True
# Object to kind mappings
comment_kind=t1
message_kind=t4
redditor_kind=t2
submission_kind=t3
subreddit_kind=t5
# The URL prefix for OAuth-related requests.
oauth_url=https://oauth.reddit.com
# The URL prefix for regular requests.
reddit_url=https://www.reddit.com
# The URL prefix for short URLs.
short_url=https://redd.it
[appname]
client_id=IE*******T14_w
client_secret=SW***********************CLY
password=******************
username=appname
user_agent=web:appname:1.0.0 (by /u/username)
Let me know how things go after you sort this out.

Mock flask.request in python nosetests

I'm writing test cases for code that is called via a route under Flask. I don't want to test the code by setting up a test app and calling a URL that hits the route, I want to call the function directly. To make this work I need to mock flask.request and I can't seem to manage it. Google / stackoverflow searches lead to a lot of answers that show how to set up a test application which again is not what I want to do.
The code would look something like this.
somefile.py
-----------
from flask import request
def method_called_from_route():
data = request.values
# do something with data here
test_somefile.py
----------------
import unittest
import somefile
class SomefileTestCase(unittest.TestCase):
#patch('somefile.request')
def test_method_called_from_route(self, mock_request):
# want to mock the request.values here
I'm having two issues.
(1) Patching the request as I've sketched out above does not work. I get an error similar to "AttributeError: 'Blueprint' object has no attribute 'somefile'"
(2) I don't know how to exactly mock the request object if I could patch it. It doesn't really have a return_value since it isn't a function.
Again I can't find any examples on how to do this so I felt a new question was acceptable.

Try this
test_somefile.py
import unittest
import somefile
import mock
class SomefileTestCase(unittest.TestCase):
def test_method_called_from_route(self):
m = mock.MagicMock()
m.values = "MyData"
with mock.patch("somefile.request", m):
somefile.method_called_from_route()
unittest.main()
somefile.py
from flask import request
def method_called_from_route():
data = request.values
assert(data == "MyData")
This is going to mock the entire request object.
If you want to mock only request.values while keeping all others intact, this would not work.

A few years after the question was asked, but this is how I solved this with python 3.9 (other proposed solutions stopped working with python 3.8 see here). I'm using pytest and pytest-mock, but the idea should be the same across testing frameworks, as long as you are using the native unittest.mock.patch in some capacity (pytest-mock essentially just wraps these methods in an easier to use api). Unfortunately, it does require that you set up a test app, however, you do not need to go through the process of using test_client, and can just invoke the function directly.
This can be easily handled by using the Application Factory Design Pattern, and injecting application config. Then, just use the created app's .test_request_context as a context manager to mock out the request object. using .test_request_context as a context manager, gives everything called within the context access to the request object. Here's an example below.
import pytest
from app import create_app
#pytest.fixture
def request_context():
"""create the app and return the request context as a fixture
so that this process does not need to be repeated in each test
"""
app = create_app('module.with.TestingConfig')
return app.test_request_context
def test_something_that_requires_global_request_object(mocker, request_context):
"""do the test thing"""
with request_context():
# mocker.patch is just pytest-mock's way of using unittest.mock.patch
mock_request = mocker.patch('path.to.where.request.is.used')
# make your mocks and stubs
mock_request.headers = {'content-type': 'application/json'}
mock_request.get_json.return_value = {'some': 'json'}
# now you can do whatever you need, using mock_request, and you do not
# need to remain within the request_context context manager
run_the_function()
mock_request.get_json.assert_called_once()
assert 1 == 1
# etc.
pytest is great because it allows you to easily setup fixtures for your tests as described above, but you could do essentially the same thing with UnitTest's setUp instance methods. Happy to provide an example for the Application Factory design pattern, or more context, if necessary!

with help of Gabrielbertouinataa on this article: https://medium.com/#vladbezden/how-to-mock-flask-request-object-in-python-fdbc249de504:
code:
def print_request_data():
print(flask.request.data)
test:
flask_app = flask.Flask('test_flask_app')
with flask_app.test_request_context() as mock_context:
mock_context.request.data = "request_data"
mock_context.request.path = "request_path"
print_request_data()

Here is an example of how I dealt with it:
test_common.py module
import pytest
import flask
def test_user_name(mocker):
# GIVEN: user is provided in the request.headers
given_user_name = "Some_User"
request_mock = mocker.patch.object(flask, "request")
request_mock.headers.get.return_value = given_user_name
# WHEN: request.header.get method is called
result = common.user_name()
# THEN: user name should be returned
request_mock.headers.get.assert_called_once_with("USERNAME", "Invalid User")
assert result == given_user_name
common.py module
import flask
def user_name():
return flask.request.headers.get("USERNAME", "Invalid User")

What you're trying to do is counterproductive. Following the RFC 2616 a request is:
A request message from a client to a server includes, within the first line of that message, the method to be applied to the resource, the identifier of the resource, and the protocol version in use.
Mocking the Flask request you need to rebuild its structure, what certainly, you will not to want to do!
The best approach should be use something like Flask-Testing or use some recipes like this, and then, test your method.

How do I apply a monkeypatch to GAE?

Can you tell me how I apply this patch to google app engine as in where to put it? Thank you
def _user_init(self, email=None, _auth_domain=None,
_user_id=None, federated_identity=None, federated_provider=None):
if not _auth_domain:
_auth_domain = os.environ.get('AUTH_DOMAIN')
assert _auth_domain
if email is None and federated_identity is None:
email = os.environ.get('USER_EMAIL', email)
_user_id = os.environ.get('USER_ID', _user_id)
federated_identity = os.environ.get('FEDERATED_IDENTITY',
federated_identity)
federated_provider = os.environ.get('FEDERATED_PROVIDER',
federated_provider)
if not email and not federated_identity:
raise UserNotFoundError
self.__email = email
self.__federated_identity = federated_identity
self.__federated_provider = federated_provider
self.__auth_domain = _auth_domain
self.__user_id = _user_id or None
users.User.__init__ = _user_init

Just use it as-is: Put that code in a module that gets imported before you use the relevant User module or datastore functionality. I included the relevant line to patch the code (the last line) with the patch itself.

Overriding the constructor like this is not safe. If the internal implementation of the Users API changes in production, your application could break without warning.
What are you trying to accomplish here? I don't see any custom logic; it looks like you've just copied the constructor from the SDK verbatim. If you need to add custom logic, try subclassing UserProperty and/or wrapping the users API calls instead.

I think, this belongs to some application as a grep within the appengine sdk, for 'federated_identity' does not result any clues. BTW, you should be doing the same. Grep (or WinGrep) for terms like 'federated' to see if this partial patch can be applied to any source.
Thanks for the updated link. The patch can be added to the file google/appengine/api/users.py
You might just need to add the last line: users.User.__init__ = _user_init
I could figure this out after checking the latest code in the svn.

Google Analytics and Python

I'm brand new at Python and I'm trying to write an extension to an app that imports GA information and parses it into MySQL. There is a shamfully sparse amount of infomation on the topic. The Google Docs only seem to have examples in JS and Java...
...I have gotten to the point where my user can authenticate into GA using SubAuth. That code is here:
import gdata.service
import gdata.analytics
from django import http
from django import shortcuts
from django.shortcuts import render_to_response
def authorize(request):
next = 'http://localhost:8000/authconfirm'
scope = 'https://www.google.com/analytics/feeds'
secure = False # set secure=True to request secure AuthSub tokens
session = False
auth_sub_url = gdata.service.GenerateAuthSubRequestUrl(next, scope, secure=secure, session=session)
return http.HttpResponseRedirect(auth_sub_url)
So, step next is getting at the data. I have found this library: (beware, UI is offensive) http://gdata-python-client.googlecode.com/svn/trunk/pydocs/gdata.analytics.html
However, I have found it difficult to navigate. It seems like I should be gdata.analytics.AnalyticsDataEntry.getDataEntry(), but I'm not sure what it is asking me to pass it.
I would love a push in the right direction. I feel I've exhausted google looking for a working example.
Thank you!!
EDIT: I have gotten farther, but my problem still isn't solved. The below method returns data (I believe).... the error I get is: "'str' object has no attribute '_BecomeChildElement'" I believe I am returning a feed? However, I don't know how to drill into it. Is there a way for me to inspect this object?
def auth_confirm(request):
gdata_service = gdata.service.GDataService('iSample_acctSample_v1.0')
feedUri='https://www.google.com/analytics/feeds/accounts/default?max-results=50'
# request feed
feed = gdata.analytics.AnalyticsDataFeed(feedUri)
print str(feed)

Maybe this post can help out. Seems like there are not Analytics specific bindings yet, so you are working with the generic gdata.

I've been using GA for a little over a year now and since about April 2009, i have used python bindings supplied in a package called python-googleanalytics by Clint Ecker et al. So far, it works quite well.
Here's where to get it: http://github.com/clintecker/python-googleanalytics.
Install it the usual way.
To use it: First, so that you don't have to manually pass in your login credentials each time you access the API, put them in a config file like so:
[Credentials]
google_account_email = youraccount#gmail.com
google_account_password = yourpassword
Name this file '.pythongoogleanalytics' and put it in your home directory.
And from an interactive prompt type:
from googleanalytics import Connection
import datetime
connection = Connection() # pass in id & pw as strings **if** not in config file
account = connection.get_account(<*your GA profile ID goes here*>)
start_date = datetime.date(2009, 12, 01)
end_data = datetime.date(2009, 12, 13)
# account object does the work, specify what data you want w/
# 'metrics' & 'dimensions'; see 'USAGE.md' file for examples
account.get_data(start_date=start_date, end_date=end_date, metrics=['visits'])
The 'get_account' method will return a python list (in above instance, bound to the variable 'account'), which contains your data.

You need 3 files within the app. client_secrets.json, analytics.dat and google_auth.py.
Create a module Query.py within the app:
class Query(object):
def __init__(self, startdate, enddate, filter, metrics):
self.startdate = startdate.strftime('%Y-%m-%d')
self.enddate = enddate.strftime('%Y-%m-%d')
self.filter = "ga:medium=" + filter
self.metrics = metrics
Example models.py: #has the following function
import google_auth
service = googleauth.initialize_service()
def total_visit(self):
object = AnalyticsData.objects.get(utm_source=self.utm_source)
trial = Query(object.date.startdate, object.date.enddate, object.utm_source, ga:sessions")
result = service.data().ga().get(ids = 'ga:<your-profile-id>', start_date = trial.startdate, end_date = trial.enddate, filters= trial.filter, metrics = trial.metrics).execute()
total_visit = result.get('rows')
<yr save command, ColumnName.object.create(data=total_visit) goes here>