Develop Reference

How Can we Create a plugin and Play Architecture in Django?

Actually, I want to create a plug and play architecture in Python Django. I have a different kind of scrapers and I am writing more scrapers too. Whenever I build a new scraper I have to again publish my repo in production. What I need I just want to plug that new scraper without actually again deploying my code. I have already used the GitHub versioning system but I am in need of a more clean way. Thanks in advance.

What you should be doing is Generating the Setup.py File which would automatically set it up as a package thus, When the User would run the Setup.py File,
It would deploy it as a package. Second Thing is you can create a Package out of It with the tools already provided by Python to create a Package of an App in Django.
Packaging a Python Script:
https://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/quickstart.html
https://pythonhosted.org/an_example_pypi_project/setuptools.html
For Django, Follow below Steps:
Initilize it with a ReadMe File
Initilize it with a MANIFEST.in to include the text files and static files in our package
run python setup.py build
Reference: https://www.pythoncentral.io/package-python-django-application-reusable-component/
Now Simply Use it anywhere You Wish making it a Reusable App!
Reference for a Package Settled Up:
https://github.com/smfai200/rebound

How do I get xhtml2pdf working on GAE?

I am new to GAE, web dev and python, but am working my way up.
I have been trying to get xhtml2pdf working on GAE for some time now but have had no luck. I have downloaded various packages but keep getting errors of missing modules. These errors vary depending on what versions of these packages and dependencies I use. I have even tried using the xhtml2pdf "required dependency" versions.
I know xhtml2pdf used to be hosted on GAE according to a stackoverflow post from 2010, but I don't know if this is the case anymore. Have they replaced it with something else that the GAE team think is better?
I have also considered that the app.yaml is preventing my app from running. As soon as I try importing the pisca module, my app stops.
Could anyone please give me some direction on how to get this working? In the sense of how to install these packages with dependencies and where they should be placed in my project folder (note that I am using Windows). And what settings I would need to add to my app.yaml file.

I got it now! Don't use XHTML2PDF - use ReportLab on its own instead.

Using Soundcloud Python library in Google App Engine - what files do I need to move?

I want to use the soundcloud python library in a web app I am developing in Google App Engine. However, I can't find any file called "soundcloud.py" in the soundcloud library files I downloaded. When using pip install it works fine on my local computer.
What files exactly do I need to move - or what exact steps do I need to take - in order to be able to "import soundcloud" within Google App Engine.
I already tried moving all the *.py files into my main app directory, but still got this error:
import soundcloud
ImportError: No module named soundcloud

You need to include your soundcloud external library into your App Package. Where exactly to place them within your application is fairly dependant on frameworks your application uses, but the soundcloud must exist on your PYTHONPATH somewhere.
Also, soundcloud isn't a solitary library. You'll also need fudge and simplejson (Included in appengine since 1.4.2). Just grab both package folders from your local pip install, and stick em somewhere in your application that's already loaded in your path.
To see where your local installed pip folders are, just fake that you're uninstalling one of the packages. On my machine, this looks like this:
sparker% sudo pip uninstall fudge
Password:
Uninstalling fudge:
/Library/Python/2.7/site-packages/fudge
/Library/Python/2.7/site-packages/fudge-1.0.3-py2.7.egg-info
Proceed (y/n)? n
You only need to grab the fudge folder to include in your application, but I'd take both. It's always nice later to know which version of applications you're bundling with your app, and that's the easiest way to know.

Ok, I think I figured out. What I needed to do what copy the soundcloud folder, along with the fudge, simplejson and requests folders into the root folder of my webapp. Thank you VooDooNOFX -- although you didn't directly answer the precise question, you sparked the thinking that helped me figure it out.

GAE - Including external python modules without adding them to the repository?

I'm current working on a python based Google App Engine project. Specifically, I'm using Flask for the application. I'm wondering what the accepted method of including external python modules is, specifically when it comes to the repository. From what I can tell, including other people's code in my repository is bad form for several reasons. However, other people will be working on the same repository, so we should be using the same external modules to insure the same results.
Specifically, I need to include Flask (and its dependencies) to my application. The easiest way to do this with Google App Engine is just to throw them into the root level:
MyProject
app.yaml
main.py
MyApp
Flask
...
What is the proper way to bring in these external modules in such a project? Both a generalized answer and one specific to my case would be useful. Also, any other related recommendations would be appreciated. Thank you much.

While it is indeed possible to include third party libraries as submodules or symlinks from external repositories, in practice it's not a good idea. Here are two scenarios on what could go wrong:
If the third party library releases a new version that breaks the functionality, you will have to either make all the necessary changes to meet the new requirements or simply find the previous version to keep working and break the external connection. Usually this happens when you are very close to deadlines.
If the third party library releases a new version and one of your colleagues is upgraded and made all the necessary changes to support the new version, on your side the code will be broken until you will upgrade as well.
The above examples are much more visible in big projects with lots of dependencies and as more people joining the project in the long run it becomes a huge problem! I could come up with more examples, but I think you can see the point.
Your best option is to include the external libraries into your repository, which also has the advantage that you are able to have the whole project up and running on a new machine without many dependencies. There are many ways on how to organize your third party libraries and all of them needs to be included on the same or deeper level with your app.yaml file. Just as #dragonx mentioned include only the core library code.
Also do not afraid putting stuff into your repository cause space is not an issue today and these libraries usually not updating that often so your repository size is not getting too much bigger over time.
Since you mentioned Flask on Google App Engine, you can check out my gae-init project, where you can see in practice how the external libraries are organised.

You're actually asking two questions here.
How do I include the external library in my GAE project?
You've got the right idea. Whatever way you go about it, you must somehow include Flask and its dependencies in the root of your GAE project. One way is to put a copy directly in there.
The second way is to use a symbolic link to the folder that contains the external library. I'm not sure about Flask, but often times external repos contain the actual library code in a subdirectory - so often you don't want the root of the repo in your GAE app, just the root of the actual source. In this case, it's easier to put a symlink that links to the source folder.
How do I manage external libraries in my source repo?
This is a harder question to answer since it depends what source control tool you're using. Yes, you do want to have everyone use the same versions of external libraries, and they should be included in your source control somehow.
If you're using git, git submodule is the way to go. It's a bit confusing to start with but it'll get the job done.
I'd recommend a repo structure that looks something like this
repo/
thirdparty/
flask/
other_dependency/
another_dependency/
README.TXT
setup.py
src/
app/
app.yaml
your_source.py
softlink_to_flask
softlink_to_other_dependency
softlink_to_another_dependency_src
In this example you keep the source to your external libraries in the thirdparty folder. These may be git submodules. In the app folder you have your source, and softlinks to the appropriate files that are actually needed for your app to run. In this case, the actual code for another_dependency may be in the another_dependency/src folder rather than the actual root of another dependency. This way you don't need to include the unnecessary files in your deployment folder, but you can still keep the entire library in your repo.

You can't just create requirements.txt and put it to GAE. Your code must include all pure python libraries that used your project and doesn't supported by GAE (https://developers.google.com/appengine/docs/python/tools/libraries27).
If you look at flask deploy example for GAE (http://flask.pocoo.org/docs/quickstart/#deploying-to-a-web-server and https://github.com/kamalgill/flask-appengine-template) you can find some dependencies like flask, werkzeug and etc. and all this dependencies you must push to GAE server.
So I see three solutions:
Use local requirements for local development and make custom build function that will download all dependencies, put with your application and upload to GAE server.
Add tools for local deployment when you just start project that put required libraries with your application (don't forget about .gitignore).
Use something like git submodules to requirements repositories.

There is two case for using python third party packages in google app engine project:
If your library is one of the supported runtime-provided third-party libraries of GAE section
just add it to your app.yml file under libraries
libraries:
- name: package_name
version: latest
Add your code
import pack_name
Sometimes you need to install the package with
pip install package_name
Make sure you're using the right interpreter, by using
pip freeze
you can make sure the package is installed successfully to the right path.
Otherwise, if GAE does not support you library, you need to download it manually and save it locally under root/Lib directory:
or through GIT
or through pip (pip install package_name -t path/to/your/Lib/dir)
After that, we should declare Lib directory as source dir in pycharm
pycharm->preferences->Project Structure
Choose Lib directory and mark it as source.
Then, import it.
import pack_name
Pay attention that when you're doing the import, you choosing the local path and not your python path.
In general, that's recommended to have requirements.txt file, that includes all the used packages names, and then the pycharm will recognize the uninstalled packages and suggest you to install them.
Good Luck

get the modules locally for django project

Is there a way to import all the modules in the django project itself instead of setting up again and again in all the systems.
I would have used gem freeze or something like that in a rails project.

There's a bit of a terminology confusion here: "modules" refers to individual .py files within a Python package. And importing is what you do within code, to bring modules into the current namespace.
I think what you're asking is how to install Python packages on deployment. The answer to that is, use pip with the freeze command, in conjunction with virtualenv.

First of all you should be using virtualenv. This way the python path of your django app only has stuff relevant to it. This also allows you to run several separate django / python apps on the same server without them bumping heads.
When you have a virtualenv with your django app running in it you need to generate a requirements file.
pip freeze -E virtualenv_path > stable-req.txt
You can then use this file to generate a bundle.
pip bundle mybundle.bundle -r stable-req.txt
This bundle can then be used to deploy with.

Instead of
gem freeze
Try to use
pip bundle
I found this solution here: Django equivalent to "rake rails:freeze:gems" and "rake gems:unpack"

Is there a reason you want to import all modules?
It's a good practice to import only those modules, classes etc. which are needed...