Develop Reference

Why does my parallel code generate an error?

Issue 1: When sys.stdout.write is not wrapped in a separate function, the code below fails.
Issue 2: When ssys.stdout.write is wrapped in a separate function, the code prints spaces between each letter.
Code (v1):
#!/usr/bin/env python
import pp
import sys
def main():
server = pp.Server()
for c in "Hello World!\n":
server.submit(sys.stdout.write, (c,), (), ("sys",))()
if __name__=="__main__":
main()
Trace:
$ ./parhello.py
Traceback (most recent call last):
File "./parhello.py", line 15, in <module>
main()
File "./parhello.py", line 12, in main
server.submit(write, (c,), (), ("sys",))()
File "/Library/Python/2.7/site-packages/pp.py", line 461, in submit
sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
File "/Library/Python/2.7/site-packages/pp.py", line 639, in __dumpsfunc
sources = [self.__get_source(func) for func in funcs]
File "/Library/Python/2.7/site-packages/pp.py", line 706, in __get_source
sourcelines = inspect.getsourcelines(func)[0]
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 688, in getsourcelines
lines, lnum = findsource(object)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 527, in findsource
file = getsourcefile(object)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 446, in getsourcefile
filename = getfile(object)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 422, in getfile
'function, traceback, frame, or code object'.format(object))
TypeError: <built-in method write of file object at 0x1002811e0> is not a module, class, method, function, traceback, frame, or code object
make: *** [test] Error 1
Code (v2):
#!/usr/bin/env python
import pp
import sys
def hello(c):
sys.stdout.write(c)
def main():
server = pp.Server()
for c in "Hello World!\n":
server.submit(hello, (c,), (), ("sys",))()
if __name__=="__main__":
main()
Trace:
$ ./parhello.py
H e l l o W o r l d !

For the first part, pp wasn't designed to handle built-ins as arguments to submit. The second problem is more complicated. Before pp calls the submitted function, it redirects stdout and stderr to a StringIO object. On completing the task, it prints the value from the StringIO object with
print sout,
This means that it appends a space the contents of sout before printing it. To get around this, don't have your functions use sys.stdout directly, but print either to a file or a queue you manage and handle the printing of in a better way.

Reading from a text file web2py

I'm having an issue with web2py. I have a text file called defVals.txt that's in the modules folder. I try to read from it, using open("defVals.txt") (in a Module in the same director as defVals.txt), but I get the error:
Traceback (most recent call last):
File "/home/jordan/web2py/gluon/restricted.py", line 212, in restricted
exec ccode in environment
File "/home/jordan/web2py/applications/randommotif/controllers/default.py", line 67, in <module>
File "/home/jordan/web2py/gluon/globals.py", line 188, in <lambda>
self._caller = lambda f: f()
File "/home/jordan/web2py/applications/randommotif/controllers/default.py", line 13, in index
defaultData = parse('defVals.txt')
File "applications/randommotif/modules/defaultValParser.py", line 6, in parse
lines = open(fileName)
IOError: [Errno 2] No such file or directory: 'defVals.txt'
What am I doing wrong? Where should I place defVals.txt
I'm using Ubuntu 12.10
Thanks,
Jordan
Update:
This is the source code to defaultValParser.py:
import itertools
import string
import os
from gluon import *
from gluon.custom_import import track_changes; track_changes(True)
#this returns a dictionary with the variables in it.
def parse(fileName):
moduleDir = os.path.dirname(os.path.abspath('defaultValParser.py'))
filePath = os.path.join(moduleDir, fileName)
lines = open(filePath, 'r')
#remove lines that are comments. Be sure to remove whitespace in the beginning and end of line
real = filter(lambda x: (x.strip())[0:2] != '//', lines)
parts = (''.join(list(itertools.chain(*real)))).split("<>")
names = map(lambda x: (x.split('=')[0]).strip(), parts)
values = map(lambda x: eval(x.split('=')[1]), parts)
return dict(zip(names, values))
It works fine if I import it and call it from a terminal (provided I comment out the gluon imports), but if I call it from a web2py controller, it fails completely:
Traceback (most recent call last):
File "/home/jordan/web2py/gluon/restricted.py", line 212, in restricted
exec ccode in environment
File "/home/jordan/web2py/applications/randommotif/controllers/default.py", line 71, in <module>
File "/home/jordan/web2py/gluon/globals.py", line 188, in <lambda>
self._caller = lambda f: f()
File "/home/jordan/web2py/applications/randommotif/controllers/default.py", line 17, in index
defaultData = parse('defVals.txt')
File "applications/randommotif/modules/defaultValParser.py", line 6, in parse
IOError: [Errno 2] No such file or directory: 'defVals.txt'

Use an absolute path based on the __file__ path of the module:
moduledir = os.path.dirname(os.path.abspath('__file__'))
# ..
defaultData = parse(os.path.join(moduledir, 'defVals.txt'))
__file__ is the filename of the current module, using the .dirname() of that gives you the directory the module is in. I used .abspath() to make sure you have an absolute path for your module file at all times, heading off some edgecases you could hit otherwise.
moduledir is a global in your module.

How to open files, web browsers, and URLs in Python, not in IDLE.

I know you can open files, browsers, and URLs in the Python GUI. However, I don't know how to apply this to programs. For example, none of the below work. (The below are snippets from my growing chat bot program):
def browser():
print('OPENING FIREFOX...')
handle = webbroswer.get() # webbrowser is imported at the top of the file
handle.open('http://youtube.com')
handle.open_new_tab('http://google.com')
and
def file():
file = str(input('ENTER THE FILE\'S NAME AND EXTENSION:'))
action = open(file, 'r')
actionTwo = action.read()
print (actionTwo)
These errors occur, in respect to the above order, but in individual runs:
OPENING FIREFOX...
Traceback (most recent call last):
File "C:/Users/RCOMP/Desktop/Programming/Python Files/AI/COMPUTRON_01.py", line 202, in <module>
askForQuestions()
File "C:/Users/RCOMP/Desktop/Programming/Python Files/AI/COMPUTRON_01.py", line 64, in askForQuestions
browser()
File "C:/Users/RCOMP/Desktop/Programming/Python Files/AI/COMPUTRON_01.py", line 38, in browser
handle = webbroswer.get()
NameError: global name 'webbroswer' is not defined
>>>
ENTER THE FILE'S NAME AND EXTENSION:file.txt
Traceback (most recent call last):
File "C:/Users/RCOMP/Desktop/Programming/Python Files/AI/COMPUTRON_01.py", line 202, in <module>
askForQuestions()
File "C:/Users/RCOMP/Desktop/Programming/Python Files/AI/COMPUTRON_01.py", line 66, in askForQuestions
file()
File "C:/Users/RCOMP/Desktop/Programming/Python Files/AI/COMPUTRON_01.py", line 51, in file
action = open(file, 'r')
IOError: [Errno 2] No such file or directory: 'file.txt'
>>>
Am I handling this wrong, or can I just not use open() and webbrowser in a program?

You should read the errors and try to understand them - they are very helpful in this case - as they often are:
The first one says NameError: global name 'webbroswer' is not defined.
You can see here that webbrowser is spelled wrong in the code. It also tells you the line it finds the error (line 38)
The second one IOError: [Errno 2] No such file or directory: 'file.txt' tells you that you're trying to open a file that doesn't exist. This does not work because you specified
action = open(file, 'r')
which means that you're trying to read a file. Python does not allow reading from a file that does not exist.

Why this code doesn't work in parallel python

I tried to use pp(Parallel Python) like this:
import glob
import subprocess
import pp
def run(cmd):
print cmd
subprocess.call(cmd, shell=True)
job_server = pp.Server()
job_server.set_ncpus(8)
jobs = []
for a_file in glob.glob("./*"):
cmd = "ls"
jobs.append(job_server.submit(run, (cmd,)))
for j in jobs:
j()
But encountered such an error that subprocess.call is not a global name.
An error has occured during the function execution
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pp-1.6.1-py2.7.egg/ppworker.py", line 90, in run
__result = __f(*__args)
File "<string>", line 3, in run
NameError: global name 'subprocess' is not defined
I've imported subprocess, why can't it be used here?
According to abarnert's suggestion, I changed my code to this:
import glob
import pp
def run(cmd):
print cmd
subprocess.call(cmd, shell=True)
job_server = pp.Server()
job_server.set_ncpus(8)
jobs = []
for a_file in glob.glob("./*"):
cmd = "ls"
jobs.append(job_server.submit(run, (cmd,),modules=("subprocess",)))
for j in jobs:
j()
But it still doesn't work, it complains like this:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.6/dist-packages/pp-1.6.1-py2.6.egg/pp.py", line 721, in _run_local
job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment

The documentation explains this pretty well, and each example shows you how to deal with it.
Among the params of the submit method is "modules - tuple with module names to import". Any modules you want to be available in the submitted job has to be listed here.
So, you can do this:
jobs.append(job_server.submit(run, (cmd,), (), ('subprocess',)))
Or this:
jobs.append(job_server.submit(run, (cmd,), modules=('subprocess',)))

Sorry, untested, but did you try:
from subprocess import call
Inside the 'run' function?
And then use "call" instead of "subprocess.call" ? That would make 'call' local to the function but accessible.

Setting fabric hosts list from an external hosts file

I need to get fabric to set its hosts list by opening and reading a file to get the hosts.
Setting it this way allows me to have a huge list of hosts without needing to edit my fabfile for this data each time.
I tried this:
def set_hosts():
env.hosts = [line.split(',') for line in open("host_file")]
def uname():
run("uname -a")
and
def set_hosts():
env.hosts = open('hosts_file', 'r').readlines
def uname():
run("uname -a")
I get the following error each time I try to use the function set_hosts:
fab set_hosts uname
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 712, in main
*args, **kwargs
File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 264, in execute
my_env['all_hosts'] = task.get_hosts(hosts, roles, exclude_hosts, state.env)
File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 74, in get_hosts
return merge(*env_vars)
File "/usr/local/lib/python2.7/dist-packages/fabric/task_utils.py", line 57, in merge
cleaned_hosts = [x.strip() for x in list(hosts) + list(role_hosts)]
AttributeError: 'builtin_function_or_method' object has no attribute 'strip'

The problem you're hitting here is that you're setting env.hosts to a function object, not a list or iterable. You need the parens after readlines, to actually call it:
def set_hosts():
env.hosts = open('hosts_file', 'r').readlines()