Develop Reference

Multiprocessing with Django when importing a external module

I built a scraping module "scraper.py" that also has the ability to download file and I imported this module into django views. Issue is that in the scraper.py, this " __name__='__main__" is included where the multiprocessing pool is, so when I import the module and try to run it, it doesn't work because it isn't the main.
This is the script(scraper.py) that uses the pool method.
def download(self, url):
response = self._is_downloadable(url)
if response:
name = response.headers.get('content-disposition')
fname = re.findall('filename=(.+)', name)
if len(fname) != 0:
filename = fname[0]
filename = filename.replace("\"", "")
print(filename)
else :
filename = "Lecture note"
with open(filename, 'wb') as files:
for chunk in response.iter_content(100000):
files.write(chunk)
def download_course_file(self, course):
username = self._login_data["username"]
p = Path(f"{username}-{course}.txt").exists()
if not p:
self.get_download_links(course)
statime = time.time()
if __name__ == "__main__":
with Pool() as p:
with open(f"{username}-{course}.txt", "r") as course_link:
data = course_link.read().splitlines(False)[::2]
p.map(self.download, data)
print(data)
print(f"Process done {time.time()-statime}")
This module is imported in the views and then ran as
import scraper
def download_course(request, id):
course = course = get_object_or_404(Course, id=id)
course_name = (course.course_name)[:6]
person, error = create_session(request)
if "invalid" in error:
data = {"error":error}
return JsonResponse(data)
person.download_course_file(course_name)
data = {"success":"Your notes are being downloaded"}
return JsonResponse(data)
PS: create_session is a function for initialising the scraper object with a username and password.
Is there a workaround for this name statement and even if there isn't, can't I remove it when I am deploying to a server as long as the server don't use windows as its OS.

AssertionError: View function mapping is overwriting an existing endpoint function: EMPTY?

Any suggestion toward a clean solution will be appreciated.
I am trying to add some static files to an existing app, a static folder with other files is already set. I would like to define several routes based on a list, the goal is to be able to easily add files.
The problem is that I receive this instead of a working program, the "endpoint function" isn't properly detected.
Output:
{'/file1.js': <function func at 0x10aedfed8>, '/file2.js': <function func at 0x10aeea050>}
{'/file1.js': <function func at 0x10aedfed8>, '/file2.js': <function func at 0x10aeea050>}
Traceback (most recent call last):
File "flaskweird.py", line 29, in <module>
app.add_url_rule(d, '', app.serv_deps[d])
File "/Library/Python/2.7/site-packages/flask/app.py", line 62, in wrapper_func
return f(self, *args, **kwargs)
File "/Library/Python/2.7/site-packages/flask/app.py", line 984, in add_url_rule
'existing endpoint function: %s' % endpoint)
AssertionError: View function mapping is overwriting an existing endpoint function:
Code :
from flask import Flask
app = Flask(__name__)
def gen_serv_dep(path):
fp = 'ui' + path # path always starts with /
with open(fp) as f:
content = f.read()
if path.endswith('.css'):
mime = "text/css"
elif path.endswith('.js'):
mime = "text/javascript"
else:
mime = "text"
def func():
#this.__name__ = path # doesn't change anything
return Response(response=content, status=200, mimetype=mime)
return func
deps = ['/file1.js', '/file2.js']
app.serv_deps = {}
for d in deps:
app.serv_deps[d] = gen_serv_dep(d)
for d in deps:
print(app.serv_deps)
app.add_url_rule(d, '', app.serv_deps[d])
#app.route('/')
def hello_world():
return 'Hello, World!'
Sample files to serve:
mkdir ui
echo "test1" > ui/file1.js
echo "test2" > ui/file2.js
Flask==0.10.1

"Because you passed an empty string to add_url_rule multiple times in a for loop. Pick a unique name each time." -- davidism
This works fine
for d in deps:
app.add_url_rule(d, d, app.serv_deps[d])
Thanks

Python CherryPy server, what does cherrypy.Application do?

I have here a Python program.
It uses CherryPy to create a server.
# coding:utf-8
import os.path
import cherrypy
from app import application
def main():
try:
currentDir_s = os.path.dirname(os.path.abspath(__file__))
except:
currentDir_s = os.path.dirname(os.path.abspath(sys.executable))
cherrypy.Application.currentDir_s = currentDir_s
configFileName_s = 'server.conf'
if os.path.exists(configFileName_s) == False:
configFileName_s = None
cherrypy.engine.autoreload.unsubscribe()
cherrypy.engine.timeout_monitor.unsubscribe()
cherrypy.quickstart(application.Application_cl(), config=configFileName_s)
if __name__ == '__main__':
main()
And in "server.conf" it configure the server:
[global]
tools.log_headers.on: True
tools.sessions.on: False
tools.encode.on: True
tools.encode.encoding:"utf-8"
server.socket_port: 8080
server.socket_timeout:60
server.thread_pool: 10
server.environment: "production"
log.screen: True
[/]
tools.staticdir.root: cherrypy.Application.currentDir_s
tools.staticdir.on = True
tools.staticdir.dir = '.'
There is one thing, I don't understand, this line (line 13 in the python code):
cherrypy.Application.currentDir_s = currentDir_s
I searched in the internet about this, but I couldn't find anything. What does "cherrypy.Application" do? Why I have to do this assignment (cherrypy.Application.currentDir_s = currentDir_s)?

I searched the cherrypy source code and here is what I found.
In _cptree.py module you will find the Application class. Below that, there is a Tree class which has mount method which we are used to binding applications with (e.g. cherrypy.tree.mount(Root(), "/", config=config))
def mount(self, root, script_name="", config=None):
...
When you look inside this method you will see the code below;
def mount(self, root, script_name="", config=None):
...
if isinstance(root, Application):
app = root
if script_name != "" and script_name != app.script_name:
raise ValueError(
"Cannot specify a different script name and pass an "
"Application instance to cherrypy.mount")
script_name = app.script_name
else:
app = Application(root, script_name)
# If mounted at "", add favicon.ico
if (script_name == "" and root is not None
and not hasattr(root, "favicon_ico")):
favicon = os.path.join(os.getcwd(), os.path.dirname(__file__),
"favicon.ico")
root.favicon_ico = tools.staticfile.handler(favicon)
if config:
app.merge(config)
self.apps[script_name] = app
So, the code says that every object (application) you pass to the mount method is either Application instance or wrapped in an Application instance. So why is that so? When you check Application class above Tree class, you will see a __call__ method like below;
def __call__(self, environ, start_response):
return self.wsgiapp(environ, start_response)
Yes you see it now, it is wsgi interface.
Therefore, Application is a wsgi wrapper for your cherrypy applications.
When you check the source code of cherrypy you may learn lots of things. I hope this answer help you.

unittest - Importerror

I'm following this tutorial on web2py where you get to make a testdriven environment. However when I try to run the test with unittest, selenium I get this error:
$ python functional_tests.py
running tests
Traceback (most recent call last):
File "functional_tests.py", line 56, in <module>
run_functional_tests()
File "functional_tests.py", line 46, in run_functional_tests
tests = unittest.defaultTestLoader.discover('fts')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 202, in discover
raise ImportError('Start directory is not importable: %r' % start_dir)
ImportError: Start directory is not importable: 'fts'
This is how the functional_tests.py looks like:
#!/usr/bin/env python
try: import unittest2 as unittest #for Python <= 2.6
except: import unittest
import sys, urllib2
sys.path.append('./fts/lib')
from selenium import webdriver
import subprocess
import sys
import os.path
ROOT = 'http://localhost:8001'
class FunctionalTest(unittest.TestCase):
#classmethod
def setUpClass(self):
self.web2py = start_web2py_server()
self.browser = webdriver.Firefox()
self.browser.implicitly_wait(1)
#classmethod
def tearDownClass(self):
self.browser.close()
self.web2py.kill()
def get_response_code(self, url):
"""Returns the response code of the given url
url the url to check for
return the response code of the given url
"""
handler = urllib2.urlopen(url)
return handler.getcode()
def start_web2py_server():
#noreload ensures single process
print os.path.curdir
return subprocess.Popen([
'python', '../../web2py.py', 'runserver', '-a "passwd"', '-p 8001'
])
def run_functional_tests(pattern=None):
print 'running tests'
if pattern is None:
tests = unittest.defaultTestLoader.discover('fts')
else:
pattern_with_globs = '*%s*' % (pattern,)
tests = unittest.defaultTestLoader.discover('fts', pattern=pattern_with_globs)
runner = unittest.TextTestRunner()
runner.run(tests)
if __name__ == '__main__':
if len(sys.argv) == 1:
run_functional_tests()
else:
run_functional_tests(pattern=sys.argv[1])

I solved this problem by replacing fts with the full path i.e. /home/simon/web2py/applications/testapp/fts
Hope this helps

I had the same problem and based on an excellent article unit testing with web2py,
I got this to work by doing the following:
Create a tests folder in the tukker directory
Copy/save the amended code(below) into tests folder as alt_functional_tests.py
Alter the web2py path in the start_web2py_server function to your own path
To run, enter the command: python web2py.py -S tukker -M -R applications/tukker/tests/alt_functional_tests.py
I am no expert but hopefully this will work for you also.
import unittest
from selenium import webdriver
import subprocess
import urllib2
execfile("applications/tukker/controllers/default.py", globals())
ROOT = 'http://localhost:8001'
def start_web2py_server():
return subprocess.Popen([
'python', '/home/alan/web2py/web2py/web2py.py', 'runserver',
'-a "passwd"', '-p 8001' ])
class FunctionalTest(unittest.TestCase):
#classmethod
def setUpClass(self):
self.web2py = start_web2py_server()
self.browser = webdriver.Firefox()
self.browser.implicitly_wait(1)
#classmethod
def tearDownClass(self):
self.browser.close()
self.web2py.kill()
def get_response_code(self, url):
"""Returns the response code of the given url
url the url to check for
return the response code of the given url
"""
handler = urllib2.urlopen(url)
return handler.getcode()
def test_can_view_home_page(self):
# John opens his browser and goes to the home-page of the tukker app
self.browser.get(ROOT + '/tukker/')
# He's looking at homepage and sees Heading "Messages With 300 Chars"
body = self.browser.find_element_by_tag_name('body')
self.assertIn('Messages With 300 Chars', body.text)
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(FunctionalTest))
unittest.TextTestRunner(verbosity=2).run(suite)

First you have to do some changes in wrong paths in fts/functional_tests.py
search for
'python', '../../web2py.py', 'runserver', '-a "passwd"', '-p 8001'
and change it to
'python', '../../../web2py.py', 'runserver', '-a "passwd"', '-p 8001'
then
tests = unittest.defaultTestLoader.discover('fts')
to
tests = unittest.defaultTestLoader.discover('.')
then
tests = unittest.defaultTestLoader.discover('fts', pattern=pattern_with_globs)
to
tests = unittest.defaultTestLoader.discover('.', pattern=pattern_with_globs)
and
sys.path.append('fts/lib')
to
sys.path.append('./lib')

CherryPy Custom Tool for user authentication

I'm trying to set up a simple way of decorating methods in my CherryPy controller classes so that a user is redirected to the login page if they haven't authenticated yet. I was going to do a basic Python decorator, but an answer here suggested I use a CherryPy Custom Tool instead. So I'm trying to do that, but I can't get it to work. Here's what I have:
def authenticate():
user = cherrypy.session.get('user', None)
if not user:
raise cherrypy.HTTPRedirect('/?errMsg=Please%20log%20in%20first')
cherrypy.tools.authenticate = cherrypy.Tool('on_start_resource', authenticate)
The /home page is a page that should be restricted to authenticated users, so I have this:
#cherrypy.expose
#cherrypy.tools.authenticate
def home(self, **kwargs):
tmpl = TemplateDir.get_template('home.mako')
return tmpl.render()
However, I get this error when I try to start my web site:
Traceback (most recent call last):
File ".\example.py", line 3, in <module>
from controller.main import Root
File "C:\...\controller\main.py", line 9, in <module>
class Root(BaseModule):
File "C:\...\controller\main.py", line 19, in Root
#cherrypy.tools.authenticate
File "C:\Python26\lib\site-packages\cherrypy\_cptools.py", line 119, in
__call__ % self._name)
TypeError: The 'authenticate' Tool does not accept positional arguments; you must
use keyword arguments.
Edit: okay, if I change my use of the custom tool to have parentheses, I get a different error.
#cherrypy.expose
#cherrypy.tools.authenticate() # Magic parentheses...
def home(self, **kwargs):
...
Now I get:
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\cherrypy\_cprequest.py", line 625, in respond
self.hooks.run('on_start_resource')
File "C:\Python26\lib\site-packages\cherrypy\_cprequest.py", line 97, in run
hook()
File "C:\Python26\lib\site-packages\cherrypy\_cprequest.py", line 57, in __call__
return self.callback(**self.kwargs)
File ".\example.py", line 40, in authenticate
user = cherrypy.session.get('user', None)
AttributeError: 'module' object has no attribute 'session'
Edit: I have sessions turned on:
cherrypy.tools.sessions.storage_type = 'file'
cherrypy.tools.sessions.storage_path = r'%s\sessions' % curDir
cherrypy.tools.sessions.timeout = 60
cherrypy.tree.mount(Root(), "/", config={
'/static': {
'tools.staticdir.on':True,
'tools.staticdir.dir':r'%s\static' % curDir,
},
'/': {
'tools.sessions.on':True,
}
})
When I first load the page with my custom tool decorator on the web method, I get this error:
AttributeError: 'module' object has no attribute 'session'
Then when I reload the page, I get this error:
AttributeError: '_Serving' object has no attribute 'session'
Edit: even trying this much in my controller class, I still get the 'module object has no attribute session' error:
class Root(BaseModule):
_cp_config = {'tools.sessions.on': True}
sess = cherrypy.session # Error here
...

I was using the wrong hook. Changing:
cherrypy.tools.authenticate = cherrypy.Tool('on_start_resource', authenticate)
To:
cherrypy.tools.authenticate = cherrypy.Tool('before_handler', authenticate)
Fixed the problem. Apparently my authenticate method was getting called before sessions had been turned on, so it couldn't access cherrypy.session. I didn't need any session-turn-on stuff in my controllers; all that was necessary was the following in my server-start script:
def authenticate():
...
cherrypy.tools.authenticate = cherrypy.Tool('before_handler', authenticate)
cherrypy.tree.mount(Root(), "/", config={
"/": {
'tools.sessions.on':True,
'tools.sessions.storage_type':'file',
'tools.sessions.storage_path':r'%s\sessions' % curDir,
'tools.sessions.timeout':60
}, ...
})
Then, in my controller on a restricted method:
#cherrypy.expose
#cherrypy.tools.authenticate()
def home(self, **kwargs):
...

Most likely sessions aren't enabled. There's an example config file on the session wiki page, or have a look at tutorial #7.