I'm trying to dynamically load a class from a specific module (called 'commands') and the code runs totally cool on my local setup running from a local Django server. This bombs out though when I deploy to Google App Engine. I've tried adding the commands module's parent module to the import as well with no avail (on either setup in that case). Here's the code:
mod = __import__('commands.%s' % command, globals(), locals(), [command])
return getattr(mod, command)
App Engine just throws an ImportError whenever it hits this.
And the clarify, it doesn't bomb out on the commands module. If I have a command like 'commands.cat' it can't find 'cat'.
I was getting import errors when importing this way when my folder/package was named "commands". I renamed the package to "cmds" and everything worked. I'm guessing there was a conflict with a builtin named "commands". Also, I don't know if it matters, but I only passed a value for the name parameter when calling import:
__import__('cmds.' + command_name)
Nick Johnson from the AppEngine team wrote up a blog post on this topic that may help you:
Webapps on App Engine, part 6: Lazy loading
The whole batch of them are recommended reading.
My AppEngine framework MVCEngine dynamically imports controller classes. The actual code in-context can be browsed on Google Code.
Briefly, here's how I do it:
controller_name = "foo"
controller_path = "app/controllers/%s_controller.py" % controller_name
controller = __import__(controller_path)
controllerClass = classForName(controller_name, namespace=controller.__dict__)
and the classForName function:
def classForName(name, *args, **kw):
ns = kw.get('namespace',globals())
return ns[name](*args)
I haven't read Nick's article on Lazy Loading, referenced above, but he is pretty much the authority on things AppEngine, and he has a better understanding than I do of the (all-important) performance characteristics of different approaches to coding for AppEngine. Definitely read his article.
You may want to have a look on mapreduce.util.for_name which lets you to dynamically import class/function/method. I promise :) I will wrap that in a blogpost.
Related
In the official Quickstart, it's recommended to use __name__ when using a single module:
... If you are using a single module (as in this example), you should use __name__ because depending on if it’s started as
application or imported as module the name will be different
('__main__' versus the actual import name). ...
However, in their API document, hardcoding is recommended when my application is a package:
So it’s important what you provide there. If you are using a single
module, __name__ is always the correct value. If you however are
using a package, it’s usually recommended to hardcode the name of your
package there.
I can understand why it's better to hardcode the name of my package, but why not hardcoding the name of a single module? Or, in other words, what information can Flask get when it receives a __main__ as its first parameter? I can't see how this can make it easier for Flask to find the resources...
__name__ is just a convenient way to get the import name of the place the app is defined. Flask uses the import name to know where to look up resources, templates, static files, instance folder, etc. When using a package, if you define your app in __init__.py then the __name__ will still point at the "correct" place relative to where the resources are. However, if you define it elsewhere, such as mypackage/app.py, then using __name__ would tell Flask to look for resources relative to mypackage.app instead of mypackage.
Using __name__ isn't orthogonal to "hardcoding", it's just a shortcut to using the name of the package. And there's also no reason to say that the name should be the base package, it's entirely up to your project structure.
I started evaluating PyCharm 3 professional edition because I will be working on several Pyramid + SQLAlchemy projects.
One of the things I would really love to have is SQLAlchemy autocomplete.
I created a new starter project with the alchemy scaffold following these instructions. I also installed the SQLAlchemy package for the interpreter and virtual environment I am using for this project.
Also, when I created a new pycharm project for this code, the IDE suggested me to install the pyramid, sqlalchemy and other packages. Of course I accepted the suggestion and let the IDE install all of those packages.
In the models.py file, the DBSession is declared as follows:
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
In the views.py file, the DBSession is used this way:
one = DBSession.query(MyModel).filter(MyModel.name == 'one').first()
So I started playing with the IDE and did something like this: typed DBSession. and the IDE just gave me some few suggestions, within which the 'query' function was not listed. Then I tried typing: DBSession.query(MyModel). and pressed Ctrl+Space to try to get suggestions and a 'No suggestions' message showed up.
I would really like to have the SQLAlchemy suggestions of functions that I could use on my DBSession variable (like filter, filter_by, first, etc). I would say that this is mandatory for me :)
Is there something I am missing? Or, PyCharm doesn't support this?
The solution I've found to this (picked up from somewhere on the web) was to type hint the DBSession instance like this:
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
""":type: sqlalchemy.orm.Session"""
After this the code completion seems to work fine everywhere in the project
Note that the tutorial states:
This guide was written for PyCharm 2.7.3, although many of the topics apply for PyCharm 3.
In PyCharm 3 Professional, it is much easier to install Pyramid and start using a scaffold. See one of my video tutorials Pyramid in PyCharm in 5 minutes at 1:17 specifically.
Also you might want to blow away your project and start fresh if stuff doesn't work as expected.
PyCharm 3 Professional supports SQAlchemy as follows.
Code insight (2.6+)
Possibility to view database structure in a diagram. Refer to the section Working with Diagrams.
Code completion and resolve. (3.0+)
See more information on how to use code completion.
I use a type declaration after a variable assignment:
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
...
engine = create_engine(connect_str, max_overflow=10)
engine: Engine
As a use for variables in for loop, I used:
for table, meta in tables.items():
meta: Table
pass
in which, tables is sqlalchemy.orm.mapper.Mapper, and table is an imported type:
from sqlalchemy import create_engine, Table
If anyone gets here now, the best solution I'v seen for this issue can be found here. To save you the click:
from contextlib import contextmanager
from typing import ContextManager
#contextmanager
def session() -> ContextManager[Session]:
yield Session(...)
I am new to web2py and python both. I am writing a sample blog app in this framework. I want to split the business logic that gets called in each controller method to it's own module, and found this example helpful:
http://www.web2pyslices.com/slice/show/1478/using-modules-in-web2py
Cleaning up web2py my controllers
As you can see, you need to import objects in modules or set them through globals.current. The controller can refer to "db" and "request" instances (for example) without any import. What kind of coding mechanism makes it possible in controller but not elesewhere?
The web2py framework does a lot of behind the scenes work to make all that stuff available.
For example, when you go to a URL like host/app/controller, that controller is called by web2py (starting with something in web2py.py) that handles importing web2py modules, providing request/response objects, etc.
Things placed in modules, however, are intended to be standalone Python code, not necessarily specific to web2py.
Found the answer:
Looks like how web2py works is by compiling the python code for the controllers and models and views on the fly. It runs them in it's special 'environment'
Related snippets of code are:
https://github.com/web2py/web2py/blob/master/gluon/main.py#L205-263
In the file above, look at: build_environment, run_models_in, run_controller_in functions (below):
https://github.com/web2py/web2py/blob/master/gluon/compileapp.py#L385-487
https://github.com/web2py/web2py/blob/master/gluon/compileapp.py#L504-539
https://github.com/web2py/web2py/blob/master/gluon/compileapp.py#L542-607
Which run the python code in a 'restricted' environment:
https://github.com/web2py/web2py/blob/master/gluon/restricted.py#L197-225
I've started looking in to unittest when using google app engine. And it seems to bit tricky from what I've read. Since you can't (and not suppose to) run your test's against the datastore.
I've written an abstract class to emulate a datastore model class. And it quite works pretty nice returning mockup data on get, all, fetch and so on (only tried on a small scale) returning dbModel like results.
The one thing I haven't found a solution I'm satisfied with is how to differentiate which model class to use. I want to use the mock-ups for unit tests and the actual db.Model for when webapp is running.
My current solution looks like this in my .py containing all db.Models:
if 'SERVER_SOFTWARE' in os.environ:
class dbTest(db.Model):
content = db.StringProperty()
comments = db.ListProperty(str)
else:
class dbTest(Abstract):
content = 'Test'
comments = ['test1', 'test2']
And it kinda feels like it could break any minute. Is this the way to go or could one combine these as one class and if the db.Model is invoked properly use that else the mockup?
Check out gaetestbed (docs). It stubs out the datastore (and all the other services like memcache) and makes testing from the command line very easy. It ensures a clean environment before every test runs.
I personally think it is nicer than the other solutions I have seen.
Instead of messing your models.py I would go with gaeunit.
I've used it with success in a couple of projects and the features I like are:
Just one file to add to your project (gaeunit.py) and you are almost done
Gaeunit isolates the test datastore from the development store (i.e. tests don't pollute your development db)
Since you can't (and not suppose to)
run your test's against the datastore.
This is not true. You can and should use the local datastore implementation as a test harness - there's no reason to waste your time creating mocks for every datastore behaviour. You can use a tool such as noseGAE or gaeunit, as suggested by other posters, but if you want to set it up yourself, see this snippet.
There's more than one problem you're trying to address here...
First, for running tests with GAE emulation you can take a look at gaeunit, which I like best. If you don't want to run them from the browser then you can take a look at noseGAE (part of nose). This should give you command-line testing.
Second, regarding your comment about about 'creating an overhead of dependencies' it sounds like you're searching for a good unit testing and mocking framework. These will let you mock out the database for tests which don't need to hit it. Try mox and mockito for python.
I'm new to python, and starting to learn website development with pylons and sqlalchemy.
I've read the document of sqlalchemy and pylons, but still have a lot of problems. I've tried 2 days, but a simple website with basic CRUD operations can't work yet. I met some big problems(for me), that the circular imports problem, and relationship between models. I want to ask them here, but I know little about python, it's a problem for me to ask too.
I'm looking for an example application using pylons and sqlalchemy, I've googled, but not found. Where can I found it? Thanks in advance!
You should read The Pylons Book.
You should probably start looking from here, http://wiki.pylonshq.com/display/pylonscommunity/Sites+Using+Pylons as many of them are open-source.
Another source would be PyPI: http://pypi.python.org/pypi?%3Aaction=search&term=pylons&submit=search
Good (but complex) example on Pylons + SQLA is reddit: http://code.reddit.com/browser/r2/r2/
I met some big problems(for me)
It easier to just ask about these particular problems, though, rather than trying to understand existing code. Sites like reddit use some un-intuitive code.
circular imports problem
Just use single module per class and there will be no problems. When it is absolutely nessesary that class X and class Y able to use each other, use
from .y import Y
Class X(Base):
...
y = relation(Y, backref="x")
Class Y(Base):
...
#classmethod
def get_x(cls):
return cls.x.attr.target_mapper.class_
This is a bit hackish, but lets you create circular reference. Other way would be to add X into module y namespace from module x explicitly.