Getting the python path set up properly scrapy/django - python

I have a scrapy project that I put it inside a Django project root folder.
And I have these lines added to the setting :
import sys
import os
sys.path.append('/home/user/Django-projects/first_project')
os.environ['DJANGO_SETTINGS_MODULE'] = 'first_project.settings'
Now when I try to import a model in items.py :
from blog.models import Annonce
I get this message :
ImportError: No module named blog.models
Django project directory:
.
├── assets
│   ├── logo.jpg
│   └── screen.png
├── blog
│   ├── admin.py
│   ├── admin.pyc
│   ├── forms.py
│   ├── forms.pyc
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── models.py
│   ├── models.pyc
│   ├── tests.py
│   ├── urls.py
│   ├── urls.pyc
│   ├── views.py
│   └── views.pyc
├── database.sql
├── first_project
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── settings.py
│   ├── settings.pyc
│   ├── urls.py
│   ├── urls.pyc
│   ├── wsgi.py
│   └── wsgi.pyc
├── manage.py
├── scrapy project
│   ├── marocannonces
│   │   ├── __init__.py
│   │   ├── __init__.pyc
│   │   ├── items.py
│   │   ├── items.pyc
│   │   ├── middlewares.py
│   │   ├── middlewares.pyc
│   │   ├── pipelines.py
│   │   ├── pipelines.pyc
│   │   ├── settings.py
│   │   ├── settings.pyc
│   │   └── spiders
│   │      └── some spiders here...
│   ├── scrapy.cfg
└── templates
└── blog
├── contact.html
├── home.html
└── read.html
How can I fix this please ?

Related

How to run django in a subpackage

My project results are as follows, django in automl/service, in automl/service/worker/suggester.py file has an import from automl.core import util and this file imported by views.py, when I run python service/manage.py runserver will raise the exception
from automl.core import util
ModuleNotFoundError: No module named 'automl'
how to solve this problem, does django can not run in a sub package?
├── automl
│   ├── __init__.py
│   ├── core
│   │   ├── __init__.py
│   │   └── base.py
│   ├── example
│   │   ├── __init__.py
│   │   └── demo.py
│   └── service
│   ├── __init__.py
│   ├── manage.py
│   ├── master
│   │   ├── __init__.py
│   │   └── urls.py
│   ├── settings.py
│   ├── worker
│   │   ├── __init__.py
│   │   ├── admin.py
│   │   ├── apps.py
│   │   ├── exceptions.py
│   │   ├── models.py
│   │   ├── suggester.py
│   │   ├── tests.py
│   │   ├── urls.py
│   │   └── views.py
│   ├── urls.py
│   └── wsgi.py
├── bin
├── build.sh
├── ci.yml
├── conf
│   └── log_config.json
└── docs

How to import python function from another file into django views

I am trying to call a function from a python file that is located outside the django project directory from views.py. I have tried importing the python file but running the django server says "no module named xxxx".
Tree structure is given below.
├── auto_flash
│   ├── __init__.py
│   └── test.py
└── fireflash
├── detection_reports
│   ├── admin.py
│   ├── admin.pyc
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── migrations
│   │   ├── __init__.py
│   │   └── __init__.pyc
│   ├── models.py
│   ├── models.pyc
│   ├── __pycache__
│   │   └── __init__.cpython-35.pyc
│   ├── templates
│   │   └── detection_reports
│   │   └── home.html
│   ├── tests.py
│   ├── views.py
│   └── views.pyc
├── fireflash
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── __pycache__
│   │   ├── __init__.cpython-35.pyc
│   │   └── settings.cpython-35.pyc
│   ├── settings.py
│   ├── settings.pyc
│   ├── ui_cgi.py
│   ├── urls.py
│   ├── urls.pyc
│   ├── wsgi.py
│   └── wsgi.pyc
├── __init__.py
└── manage.py
Project name here is "fireflash" and there is an app in that project named "detection_reports". There is another directory named "auto_flash" that has same location as "fireflash". What I want to do is to import test.py file from "auto_flash" in detection_reports.views and call a function from views. Importing like this "from auto_flash import test" throws error "no module named auto_flash".
I have tried all of the above mentioned solutions, but the cleaner solution was to append the path for auto_flash to syspath and the issue was resolved. Thanks to all of you for the efforts.
Move auto_flash to fireflash
In detection_reports add from auto_flash import test.py

Gunicorn giving me Module Not Found Error

I'm attempting to build a Django blog.
I have a couple of apps in there, but for some reason Gunicorn is giving me trouble with the error - ModuleNotFoundError: No module named 'blog.wsgi'
├── Procfile
├── blog
│   ├── blog
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-36.pyc
│   │   │   ├── settings.cpython-36.pyc
│   │   │   ├── urls.cpython-36.pyc
│   │   │   └── wsgi.cpython-36.pyc
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   ├── db.sqlite3
│   ├── manage.py
│   ├── media
│   │   ├── 1820-3-large.jpg
│   │   ├── 1820-3-large_XGHcfcZ.jpg
│   │   ├── 1820-3-large_ZTmLkYt.jpg
│   │   ├── 1820-3-large_dPbPsPW.jpg
│   │   └── paul-morris-144777.jpg
│   ├── posts
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-36.pyc
│   │   │   ├── admin.cpython-36.pyc
│   │   │   ├── models.cpython-36.pyc
│   │   │   └── views.cpython-36.pyc
│   │   ├── admin.py
│   │   ├── apps.py
│   │   ├── migrations
│   │   │   ├── 0001_initial.py
│   │   │   ├── 0002_auto_20171217_0000.py
│   │   │   ├── __init__.py
│   │   │   └── __pycache__
│   │   │   ├── 0001_initial.cpython-36.pyc
│   │   │   ├── 0002_auto_20171217_0000.cpython-36.pyc
│   │   │   └── __init__.cpython-36.pyc
│   │   ├── models.py
│   │   ├── static
│   │   │   └── posts
│   │   │   ├── css
│   │   │   ├── img
│   │   │   │   └── home.jpg
│   │   │   └── js
│   │   ├── templates
│   │   │   └── posts
│   │   │   ├── home.html
│   │   │   └── post_details.html
│   │   ├── tests.py
│   │   └── views.py
│   └── sitepages
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   ├── admin.cpython-36.pyc
│   │   ├── models.cpython-36.pyc
│   │   └── views.cpython-36.pyc
│   ├── admin.py
│   ├── apps.py
│   ├── migrations
│   │   ├── __init__.py
│   │   └── __pycache__
│   │   └── __init__.cpython-36.pyc
│   ├── models.py
│   ├── static
│   │   └── sitepages
│   │   ├── css
│   │   │   └── about.css
│   │   ├── img
│   │   └── js
│   ├── templates
│   │   └── sitepages
│   │   └── about.html
│   ├── tests.py
│   └── views.py
├── requirements.txt
└── venv
`
This is the bulk of my file tree as I see it. I don't see any issues, but I'm definitely overlooking something. Let me know if you'd like to see any other pieces of code.
My Procfile says -
web: gunicorn blog.wsgi --log-file -
This is my wsgi.py file -
This is wsgi.py -
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "blog.settings")
application = get_wsgi_application()

Django ModuleNotFound For scripts

This has been driving me up the wall. I have a Django project with the following tree structure, and am trying to run python helper_scripts/load_professors_into_db.py from the root directory
load_professors_into_db.py has the following code:
## TODO: FIX THIS DAMN IMPORT PATH. THE SCRIPT DOESNT RUN CAUSE OF IT
from ocubulum_dashboard.models import Researcher
import pandas as pd
df = pd.read_csv("helper_scripts/soc_myaces_list.csv")
df = df.dropna()
df = df[~pd.isnull(df["scopus_id"])]
df = df[df["scopus_id"] != 'None']
However, it keeps trying ModuleNotFound errors. I've tried adding __init__.py files everywhere, but that doesn't work either.
Traceback (most recent call last):
File "helper_scripts/load_professors_into_db.py", line 10, in <module>
from ocubulum_dashboard.models import Researcher
ModuleNotFoundError: No module named 'ocubulum_dashboard'
The problem doesn't only occur for this. For other scripts that I want to run such as scopus_scraper.py, I face this ridiculous import issue as well.
Traceback (most recent call last):
File "data_collectors/scopus/scopus_scraper.py", line 1, in <module>
from ocubulum_dashboard.models import Researcher
ModuleNotFoundError: No module named 'ocubulum_dashboard'
Can someone point me as to how to solve this problem? I'm on python 3.6.
Entire Folder Structure:
├── data_aggregators
│   ├── myaces_aggregator.py
│   └── scopus_aggregator.py
├── data_collectors
│   ├── execute_all.py
│   ├── __init__.py
│   ├── journals
│   │   ├── __init__.py
│   │   ├── journal_scraper.py
│   │   ├── master.py
│   │   ├── __pycache__
│   │   │   └── __init__.cpython-36.pyc
│   │   └── test.json
│   ├── nus_myaces
│   │   ├── __init__.py
│   │   ├── master.py
│   │   └── __pycache__
│   │   └── __init__.cpython-36.pyc
│   ├── __pycache__
│   │   └── __init__.cpython-36.pyc
│   └── scopus
│   ├── __init__.py
│   ├── master.py
│   ├── __pycache__
│   │   └── __init__.cpython-36.pyc
│   ├── scopus_scraper.py
│   └── scopus_wrapper
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   └── scopus_wrapper.cpython-36.pyc
│   └── scopus_wrapper.py
├── environment.yml
├── helper_scripts
│   ├── __init__.py
│   ├── load_professors_into_db.py
│   ├── __pycache__
│   │   └── __init__.cpython-36.pyc
│   └── soc_myaces_list.csv
├── __init__.py
├── manage.py
├── ocubulum
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   ├── settings.cpython-36.pyc
│   │   ├── settings_development.cpython-36.pyc
│   │   ├── urls.cpython-36.pyc
│   │   ├── views.cpython-36.pyc
│   │   └── wsgi.cpython-36.pyc
│   ├── settings_development.py
│   ├── settings.py
│   ├── static
│   ├── urls.py
│   ├── views.py
│   └── wsgi.py
├── ocubulum_dashboard
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   ├── 0001_initial.py
│   │   ├── __init__.py
│   │   └── __pycache__
│   │   ├── 0001_initial.cpython-36.pyc
│   │   └── __init__.cpython-36.pyc
│   ├── models.py
│   ├── __pycache__
│   │   ├── admin.cpython-36.pyc
│   │   ├── apps.cpython-36.pyc
│   │   ├── __init__.cpython-36.pyc
│   │   ├── models.cpython-36.pyc
│   │   ├── tests.cpython-36.pyc
│   │   └── views.cpython-36.pyc
│   ├── static
│   │   ├── css
│   │   │   ├── custom.css
│   │   │   └── side-menu.css
│   │   ├── img
│   │   │   └── logo.png
│   │   └── js
│   │   └── ui.js
│   ├── templates
│   │   └── ocubulum
│   │   ├── dashboard.html
│   │   └── layout.html
│   ├── tests.py
│   └── views.py
├── Procfile
├── __pycache__
│   └── __init__.cpython-36.pyc
├── README.md
├── requirements.txt
└── runtime.txt
Try from ..ocubulum_dashboard.models import Researcher.
Or add the folder containing ocubulum_dashboard to your PYTHONPATH.

How set setting environment with DJANGO_SETTINGS_MODULE

I am new in Python \ Django and I have some question about settings environment. I use git for edit my project and on server I have production project. But when I edit project on local machine I have my own settings(DATABASE) so if I pull edited project on server I need to change my settings, so I want to set DJANGO_SETTINGS_MODULE, but I don't know what path I must set and where I must allocate my default setting.
This is my hierarchy:
wow
├── core
│   ├── admin.py
│   ├── api_views.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   ├── 0001_initial.py
│   │   ├── __init__.py
│   │   └── __pycache__
│   ├── models.py
│   ├── __pycache__
│   │   ├── admin.cpython-35.pyc
│   │   ├── api_views.cpython-35.pyc
│   │   ├── __init__.cpython-35.pyc
│   │   ├── models.cpython-35.pyc
│   │   ├── sitemap.cpython-35.pyc
│   │   ├── urls.cpython-35.pyc
│   │   └── views.cpython-35.pyc
│   ├── sitemap.py
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── manage.py
├── media
│   └── core
│   ├── iUcZCekdW68_FrqsEHv.jpg
│   ├── iUcZCekdW68.jpg
│   ├── qWI5I5NuIeg.jpg
│   ├── server1.png
│   └── server2.png
├── not_found
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   ├── __init__.py
│   │   └── __pycache__
│   ├── models.py
│   ├── __pycache__
│   │   ├── admin.cpython-35.pyc
│   │   ├── __init__.cpython-35.pyc
│   │   ├── models.cpython-35.pyc
│   │   ├── urls.cpython-35.pyc
│   │   └── views.cpython-35.pyc
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── requirements.txt
├── settings.py
├── static
│   ├── admin
│   │   ├── css
│   │   ├── fonts
│   │   ├── img
│   │   └── js
│   └── core
│   ├── css
│   ├── img
│   └── js
├── staticfiles
│   ├── admin
│   │   ├── css
│   │   ├── fonts
│   │   ├── img
│   │   └── js
│   └── core
│   ├── css
│   ├── img
│   └── js
├── templates
│   ├── 404.html
│   ├── admin
│   │   ├── base.html
│   │   ├── base_site.html
│   │   └── index.html
│   ├── character.html
│   ├── characters.html
│   ├── checkout.html
│   ├── index.html
│   └── server.html
├── url.xml
└── wow
├── gunicorn.conf.py
├── __init__.py
├── __pycache__
│   ├── __init__.cpython-35.pyc
│   ├── settings.cpython-35.pyc
│   ├── urls.cpython-35.pyc
│   └── wsgi.cpython-35.pyc
├── settings.py
├── urls.py
├── views.py
└── wsgi.py
What i must do, end where i must write os.environ["DJANGO_SETTINGS_MODULE"] = wow.settings, or something like this?

Categories

Resources