Develop Reference

Apply function decorator on print function across all files without having to import and/or reapply?

Edit: My first attempt at asking this might be a bit unfocused/poorly worded here's a better explanation of what I'm trying to do:
I'm trying to modify the default behavior of the print function for the entire environment python is running in without having to modify each file that's being run.
I'm attempting to decorate the print function (I know there are many ways to do this such as overriding it but that's not really the question I'm asking) so I can have it print out some debugging information and force it to always flush. I did that like so:
def modify_print(func):
# I made this so that output always gets flushed as it won't by default
# within the environment I'm using, I also wanted it to print out some
# debugging information, doesn't really matter much in the context of this
# question
def modified_print(*args,**kwargs):
return func(f"some debug prefix: ",flush=True,*args,**kwargs)
return modified_print
print = modify_print(print)
print("Hello world") # Prints "some debug prefix Hello World"
However what I'm trying to do is modify this behavior throughout my entire application. I know I can manually decorate/override/import the print function in each file however I'm wondering if there is some way I can globally configure my python environment to decorate this function everywhere. The only way I can think to do this would be to edit the python source code and build the modified version.
EDIT:
Here's the behavior I wanted implemented, thank you Match for your help.
It prints out the line number and filename everywhere you call a print function within your python environment. This means you don't have to import or override anything manually in all of your files.
https://gist.github.com/MichaelScript/444cbe5b74dce2c01a151d60b714ac3a
import site
import os
import pathlib
# Big thanks to Match on StackOverflow for helping me with this
# see https://stackoverflow.com/a/48713998/5614280
# This is some cool hackery to overwrite the default functionality of
# the builtin print function within your entire python environment
# to display the file name and the line number as well as always flush
# the output. It works by creating a custom user script and placing it
# within the user's sitepackages file and then overwriting the builtin.
# You can disable this behavior by running python with the '-s' flag.
# We could probably swap this out by reading the text from a python file
# which would make it easier to maintain larger modifications to builtins
# or a set of files to make this more portable or to modify the behavior
# of more builtins for debugging purposes.
customize_script = """
from inspect import getframeinfo,stack
def debug_printer(func):
# I made this so that output always gets flushed as it won't by default
# within the environment I'm running it in. Also it will print the
# file name and line number of where the print occurs
def debug_print(*args,**kwargs):
frame = getframeinfo(stack()[1][0])
return func(f"{frame.filename} : {frame.lineno} ", flush=True,*args,**kwargs)
return debug_print
__builtins__['print'] = debug_printer(print)
"""
# Creating the user site dir if it doesn't already exist and writing our
# custom behavior modifications
pathlib.Path(site.USER_SITE).mkdir(parents=True, exist_ok=True)
custom_file = os.path.join(site.USER_SITE,"usercustomize.py")
with open(custom_file,'w+') as f:
f.write(customize_script)

You can use usercustomize script from the site module to achieve something like this.
First, find out where your user site-packages directory is:
python3 -c "import site; print(site.USER_SITE)"
/home/foo/.local/lib/python3.6/site-packages
Next, in that directory, create a script called usercustomize.py - this script will now be run first whenever python is run.
One* way to replace print is to override the __builtins__ dict and replace it with a new method - something like:
from functools import partial
old_print = __builtins__['print']
__builtins__['print'] = partial(old_print, "Debug prefix: ", flush=True)
Drop this into the usercustomize.py script and you should see all python scripts from then on being overridden. You can temporarily disable calling this script by calling python with the -s flag.
*(Not sure if this is the correct way of doing this - there may be a better way - but the main point is that you can use usercustomize to deliver whatever method you choose).

There's no real reason to define a decorator here, because you are only intending to apply it to a single, predetermined function. Just define your modified print function directly, wrapping it around __builtins__.print to avoid recursion.
def print(*args, **kwargs):
__builtins.__print(f"some debug prefix: ", flush=True, *args, **kwargs)
print("Hello world") # Prints "some debug prefix Hello World"
You can use functools.partial to simplify this.
import functools
print = functools.partial(__builtins.__print, f"some debug prefix: ", flush=True)

Is there any way to replace the exists default key bindings in IPython Vi mode?

I am using IPython Vi mode, and IPython sets lots of default key mappings here. I did a research here and here, finding that I can create new key bindings by using KeyBindingManager.
However, if the binding I want to create exists, then it cannot works. For example, I write some code in ~/.ipython/profile_default/startup/vikeys.py:
from prompt_toolkit.filters.cli import ViNavigationMode, ViMode
from prompt_toolkit.filters import Always, IsReadOnly
# create `handle`
from IPython import get_ipython
ip = get_ipython()
if getattr(ip, 'pt_cli'):
registry = ip.pt_cli.application.key_bindings_registry
handle = registry.add_binding
navigation_mode = ViNavigationMode() & ViMode() & Always()
#handle('t', filter=navigation_mode)
def _(event):
"""
Go up, but if we enter a new history entry, move to the start of the
line.
"""
event.current_buffer.auto_up(
count=event.arg, go_to_start_of_line_if_history_changes=True)
Since the t is mapped here, my binding doesn't work.
I also tried to replace the handle with the code below, but there was no effects:
# create `handle`
from prompt_toolkit.key_binding.manager import KeyBindingManager
manager = KeyBindingManager.for_prompt()
handle = manager.registry.add_binding
So, is there any way to replace the existing key bindings in IPython?

Answer the question myself, to help those who meet to same problem.
After reading the source code, I find that adding eager=True argument to the handle will override the default keybindings.
Here is an example to replace default navigation keys for Dvorak user.

Using conditional imports as a way to utilize different functions with the same namespace

I am a beginner python programmer, and I am working on a selenium project in python 2.7.
I have a generic scraper script, that basically outlines what I want to do with all the websites that I visit. However, because of the nature of the data that I want to grab, I can't run the same code on each site-- each site needs to run it's own individual code.
I have attempted to solve this by importing inside of an if statement, and this is the solution I came up with:
site = False
if source_website == "Website A":
from website_a import *
site = True
elif source_website == "Website B":
from website_b import *
site = True
else:
print "This is not an acceptable website!"
if site == True:
# main code block
driver = driver_setup(chrome)
driver.get(source_website_URL)
stuff_to_save = do_some_stuff(driver)
xml_file(stuff_to_save)
driver.quit()
where the website_a and website_b modules both have functions named do_some_stuff, and they do stuff specific to the website that they're on. Now, this seems to work, for the most part. I also seem to be able to extend functionality to any number of websites, given that I program a module called website_c with the function do_some_stuff, and add that to the conditional import.
So, my question is, is this a good idea? Is there a better way to do something like this?
I have literally never seen anyone wrap import statements inside of if statements like this-- and generally, if no one seems to do it, there's usually a good reason why.

In general, from somewhere import * is not a good idea due to namespace pollution. If you want the website-specific code in separate modules, why not do something like
import importlib
website_modules = {'Website A': 'website_a', 'Website B': 'website_b'}
# ...
website = importlib.import_module(website_modules[source_website])
# use with website.function_name

Explore the page object model pattern (http://code.google.com/p/selenium/wiki/PageObjects). You should model each page as a unique entity, then have some logic that determines what page type you are displaying (either explicitly by having you specify it or implicitly by inspecting the URL and the contents of the page) and then expose the methods to capture the data you need on those objects rather than working directly with the webdriver instance. You should ultimately aim for something like:
for page_identifier in ['page1', 'page2', 'page3']:
page = navigate_to(page_identifier)
extracted_data = page.get_data()
xml_file.write(extracted_data)

Use entire scope as input in plugin in Sublime Text 2

One of the nice things about Textmate was the ability to pipe the contents of an entire scope into a command, like so:
You could then specify the scope to be used, such as meta.class.python or whatever.
I'm trying to write a small plugin that will pipe the entire current scope in as the input for the plugin (for example (not exactly what I'm trying to do, but close), a function that lets you comment out an entire Python class without selecting it all)
Using the current selection(s) as input is quite easy:
import sublime, sublime_plugin
import re
class DoStuffWithSelection(sublime_plugin.TextCommand):
def run(self, edit):
for region in self.view.sel():
if not region.empty():
changed = region # Do something to the selection
self.view.replace(edit, region, changed) # Replace the selection
I've scoured the Sublime Text plugin API for some way to do something like for region in self.view.scope(), but without success.
Is there a way, then, to use the contents of the current scope under the cursor as input for a plugin function? Or, even better, a way to use the entire scope if there isn't a selection, but use the selection if there is one.
Thanks!

If you want to get text that you select, the following code snippet is an example.
if not region.empty():
selectText = self.view.substr(region)
...
If you want to get text where the cursor is located, the following code snippet is an example.
if region.empty():
lineRegion = self.view.line(region)
lineText = self.view.substr(lineRegion)
...
To get more information, see http://net.tutsplus.com/tutorials/python-tutorials/how-to-create-a-sublime-text-2-plugin/ and http://www.sublimetext.com/docs/api-reference.

What's the official way of storing settings for Python programs?

Django uses real Python files for settings, Trac uses a .ini file, and some other pieces of software uses XML files to hold this information.
Are one of these approaches blessed by Guido and/or the Python community more than another?

Depends on the predominant intended audience.
If it is programmers who change the file anyway, just use python files like settings.py
If it is end users then, think about ini files.

As many have said, there is no "offical" way. There are, however, many choices. There was a talk at PyCon this year about many of the available options.

Don't know if this can be considered "official", but it is in standard library: 14.2. ConfigParser — Configuration file parser.
This is, obviously, not an universal solution, though. Just use whatever feels most appropriate to the task, without any necessary complexity (and — especially — Turing-completeness! Think about automatic or GUI configurators).

I use a shelf ( http://docs.python.org/library/shelve.html ):
shelf = shelve.open(filename)
shelf["users"] = ["David", "Abraham"]
shelf.sync() # Save

Just one more option, PyQt. Qt has a platform independent way of storing settings with the QSettings class. Underneath the hood, on windows it uses the registry and in linux it stores the settings in a hidden conf file. QSettings works very well and is pretty seemless.

There is no blessed solution as far as I know. There is no right or wrong way to storing app settings neither, xml, json or all types of files are fine as long as you are confortable with. For python I personally use pypref it's very easy, cross platform and straightforward.
pypref is very useful as one can store static and dynamic settings and preferences ...
from pypref import Preferences
# create singleton preferences instance
pref = Preferences(filename="preferences_test.py")
# create preferences dict
pdict = {'preference 1': 1, 12345: 'I am a number'}
# set preferences. This would automatically create preferences_test.py
# in your home directory. Go and check it.
pref.set_preferences(pdict)
# lets update the preferences. This would automatically update
# preferences_test.py file, you can verify that.
pref.update_preferences({'preference 1': 2})
# lets get some preferences. This would return the value of the preference if
# it is defined or default value if it is not.
print pref.get('preference 1')
# In some cases we must use raw strings. This is most likely needed when
# working with paths in a windows systems or when a preference includes
# especial characters. That's how to do it ...
pref.update_preferences({'my path': " r'C:\Users\Me\Desktop' "})
# Sometimes preferences to change dynamically or to be evaluated real time.
# This also can be done by using dynamic property. In this example password
# generator preference is set using uuid module. dynamic dictionary
# must include all modules name that must be imported upon evaluating
# a dynamic preference
pre = {'password generator': "str(uuid.uuid1())"}
dyn = {'password generator': ['uuid',]}
pref.update_preferences(preferences=pre, dynamic=dyn)
# lets pull 'password generator' preferences twice and notice how
# passwords are different at every pull
print pref.get('password generator')
print pref.get('password generator')
# those preferences can be accessed later. Let's simulate that by creating
# another preferences instances which will automatically detect the
# existance of a preferences file and connect to it
newPref = Preferences(filename="preferences_test.py")
# let's print 'my path' preference
print newPref.get('my path')

I am not sure that there is an 'official' way (it is not mentioned in the Zen of Python :) )- I tend to use the Config Parser module myself and I think that you will find that pretty common. I prefer that over the python file approach because you can write back to it and dynamically reload if you want.

One of the easiest ways which is use is using the json module.
Save the file in config.json with the details as shown below.
Saving data in the json file:
{
"john" : {
"number" : "948075049" ,
"password":"thisisit"
}
}
Reading from json file:
import json
#open the config.json file
with open('config.json') as f:
mydata = json.load(f) ;
#Now mydata is a python dictionary
print("username is " , mydata.get('john').get('number') , " password is " , mydata.get('john').get('password')) ;

It depends largely on how complicated your configuration is. If you're doing a simple key-value mapping and you want the capability to edit the settings with a text editor, I think ConfigParser is the way to go.
If your settings are complicated and include lists and nested data structures, I'd use XML or JSON and create a configuration editor.
For really complicated things where the end user isn't expected to change the settings much, or is more trusted, just create a set of Python classes and evaluate a Python script to get the configuration.

For web applications I like using OS environment variables: os.environ.get('CONFIG_OPTION')
This works especially well for settings that vary between deploys. You can read more about the rationale behind using env vars here: http://www.12factor.net/config
Of course, this only works for read-only values because changes to the environment are usually not persistent. But if you don't need write access they are a very good solution.

It is more of convenience. There is no official way per say. But using XML files would make sense as they can be manipulated by various other applications/libraries.

Not an official one but this way works well for all my Python projects.
pip install python-settings
Docs here: https://github.com/charlsagente/python-settings
You need a settings.py file with all your defined constants like:
# settings.py
DATABASE_HOST = '10.0.0.1'
Then you need to either set an env variable (export SETTINGS_MODULE=settings) or manually calling the configure method:
# something_else.py
from python_settings import settings
from . import settings as my_local_settings
settings.configure(my_local_settings) # configure() receives a python module
The utility also supports Lazy initialization for heavy to load objects, so when you run your python project it loads faster since it only evaluates the settings variable when its needed
# settings.py
from python_settings import LazySetting
from my_awesome_library import HeavyInitializationClass # Heavy to initialize object
LAZY_INITIALIZATION = LazySetting(HeavyInitializationClass, "127.0.0.1:4222")
# LazySetting(Class, *args, **kwargs)
Just configure once and now call your variables where is needed:
# my_awesome_file.py
from python_settings import settings
print(settings.DATABASE_HOST) # Will print '10.0.0.1'

why would Guido blessed something that is out of his scope? No there is nothing particular blessed.