I'm trying to build a very simple Web Scraper that scrapes from popular Job Boards and posts the jobs in a simple Django website.
In my Scrapy spider, I am trying to create a Django model object for every job scraped. But I'm having trouble importing the model in my Scrapy spider file.
This is my folder structure:
F:.
│ db.sqlite3
│ manage.py
│
├───jobs_board
│ asgi.py
│ settings.py
│ urls.py
│ wsgi.py
│ __init__.py
│
├───main
│ │ admin.py
│ │ apps.py
│ │ models.py
│ │ tests.py
│ │ urls.py
│ │ views.py
│ │ __init__.py
│ │
│ ├───migrations
│ │ 0001_initial.py
│ │ __init__.py
│ │
│ └───templates
│ └───main
│ home.html
│
└───scraper
│ scrapy.cfg
│ t.py
│ testing.md
│
└───scraper
│ items.py
│ middlewares.py
│ pipelines.py
│ settings.py
│ __init__.py
│
└───spiders
indeed.py
__init__.py
What I want to do is, if I run the spider "indeed.py", I want it to be able to import the main/models.py file. How would I achieve this?
Thank you for your help! :)
Related
I am struggling with the implementation of unittest for subdirectories. I have the following project
project
│ README.md
│ __init__.py
│
└───common
│ __init__.py
│ └───my_func
│ │ __init__.py
│ │ func1.py
│ │ func2.py
│
└───scripts
│ __init__.py
│ └───folder_1
│ │ __init__.py
│ │ code1.py
│ │ code2.py
│
│ └───folder_2
│ │ __init__.py
│ │ code3.py
│ │ code4.py
│ │
│ └───tests
│ │ └───test1
│ │ │ __init__.py
│ │ │ test_code3.py
│ │ │ test_code4.py
│ │ │
│ │ └───test2
│ │ │ __init__.py
│ │ │ test_code3.py
│ │ │ test_code4.py
I set the working directory to be ./project. In my code1.py file I import common.my_func.func1 and it runs normally.
Now, I am trying to implement some unittest functions in ./project/scripts/tests. To do so, I am using the coverage package and running the command: coverage run --source=scripts -m unittest discover scripts/tests. However, when doing so, I get the following error:
ModeluNotFoundError: No module named common.my_func
Weirdly speaking, the scripts works perfectly when I try to run it for one test folder only, and not the whole folder coverage run --source=scripts -m unittest discover scripts/tests/test1.
I tried multiple combinations of removing the source, getting more specific with the folder and on. Have any of you faced similar problems with python 3.8?
Thank you very much in advance,
I recently upgraded streamlit to version 1.10.0. After upgradation I ran streamlit only to find that there is a side navigation bar displaying python file names present in my directory. I have attached images for the same.
import streamlit as st
import pandas as pd
from services.multiapp import MultiApp
from functools import partial
from pages import show, analysis, help, custom, download, three_d
def refresh():
pass
file = st.file_uploader('Upload CSV file',
type='csv', help="Format")
if file is not None:
# verification
# Prediction
st.button("Refresh", on_click=refresh)
data = pd.read_csv(file)
app = MultiApp()
app.add_app('Show data', partial(show.app, data))
app.add_app('Analysis', analysis.app)
app.add_app('Custom Plots', custom.app)
app.add_app('3-D Plots', partial(three_d.app, data))
app.add_app('Download', partial(download.app, data, file.name))
app.add_app('Help', help.app)
app.run()
Please help me to remove the side navigation bar.
My current working directory has the following structure
├── app.py
├── pages
│ ├── analysis.py
│ ├── custom.py
│ ├── download.py
│ ├── help.py
│ ├── __pycache__
│ │ ├── analysis.cpython-38.pyc
│ │ ├── analysis.cpython-39.pyc
│ │ ├── custom.cpython-38.pyc
│ │ ├── custom.cpython-39.pyc
│ │ ├── download.cpython-38.pyc
│ │ ├── download.cpython-39.pyc
│ │ ├── get_df.cpython-38.pyc
│ │ ├── help.cpython-38.pyc
│ │ ├── help.cpython-39.pyc
│ │ ├── iplots.cpython-38.pyc
│ │ ├── plot.cpython-38.pyc
│ │ ├── show.cpython-38.pyc
│ │ ├── show.cpython-39.pyc
│ │ ├── show_d.cpython-38.pyc
│ │ ├── three_d.cpython-38.pyc
│ │ └── three_d.cpython-39.pyc
│ ├── show.py
│ └── three_d.py
├── __pycache__
│ ├── multiapp.cpython-38.pyc
│ └── plots.cpython-38.pyc
├── README.md
├── services
│ ├── multiapp.py
│ ├── __pycache__
│ │ ├── multiapp.cpython-38.pyc
│ │ └── multiapp.cpython-39.pyc
│ └── services.py
├── Templates
│ ├── links.txt
│ ├── multi-page-app-main
│ │ ├── app.py
│ │ ├── apps
│ │ │ ├── data.py
│ │ │ ├── home.py
│ │ │ ├── model.py
│ │ │ └── __pycache__
│ │ │ ├── data.cpython-38.pyc
│ │ │ ├── home.cpython-38.pyc
│ │ │ └── model.cpython-38.pyc
│ │ ├── multiapp.py
│ │ ├── __pycache__
│ │ │ └── multiapp.cpython-38.pyc
│ │ ├── README.md
│ │ └── requirements.txt
│ ├── streamlit-dashboard-template-main
│ │ ├── app.py
│ │ ├── dashboard.py
│ │ ├── README.md
│ │ ├── stdashdark.png
│ │ └── stdashlight.png
│ ├── streamlit-geospatial-master
│ │ ├── app.py
│ │ ├── apps
│ │ │ ├── basemaps.py
│ │ │ ├── census.py
│ │ │ ├── cesium.py
│ │ │ ├── deck.py
│ │ │ ├── device_loc.py
│ │ │ ├── gee_datasets.py
│ │ │ ├── gee.py
│ │ │ ├── heatmap.py
│ │ │ ├── home.py
│ │ │ ├── housing.py
│ │ │ ├── plotly_maps.py
│ │ │ ├── raster.py
│ │ │ ├── rois.py
│ │ │ ├── timelapse.py
│ │ │ ├── vector.py
│ │ │ ├── wms.py
│ │ │ └── xy.py
│ │ ├── data
│ │ │ ├── cog_files.txt
│ │ │ ├── html
│ │ │ │ └── sfo_buildings.html
│ │ │ ├── realtor_data_dict.csv
│ │ │ ├── us_counties.geojson
│ │ │ ├── us_metro_areas.geojson
│ │ │ ├── us_nation.geojson
│ │ │ └── us_states.geojson
│ │ ├── environment-bk.yml
│ │ ├── index.html
│ │ ├── LICENSE
│ │ ├── multiapp.py
│ │ ├── packages.txt
│ │ ├── Procfile
│ │ ├── README.md
│ │ ├── requirements.txt
│ │ └── setup.sh
│ ├── Streamlit-master
│ │ ├── app.py
│ │ ├── data
│ │ │ ├── demo.wav
│ │ │ ├── Salary_Data.csv
│ │ │ ├── sal.jpg
│ │ │ ├── snippets
│ │ │ └── virtual.mp4
│ │ ├── data.py
│ │ ├── demo.py
│ │ ├── layout.py
│ │ ├── plots.py
│ │ ├── Procfile
│ │ ├── README.md
│ │ ├── requirements.txt
│ │ ├── sidebar.py
│ │ └── widget.py
│ └── streamlit template
│ ├── JC-202103-citibike-tripdata.csv
│ ├── logo.png
│ ├── requirements.txt
│ └── streamlit_template.py
├── test.csv
└── test.py
Streamlit sidebar picks everything up that is located in the "pages" folder (see here: Blog Streamlit)
You could either rename your pages folder to something else or if you like the side bar, you can just collapse it initially:
import streamlit as st
st.set_page_config(initial_sidebar_state="collapsed")
That way the sidebar will be closed on start but can still be opened if needed.
A little late to this question, but I added the following snippet to the pages that I wanted to hide the lists on.
no_sidebar_style = """
<style>
div[data-testid="stSidebarNav"] {display: none;}
</style>
"""
st.markdown(no_sidebar_style, unsafe_allow_html=True)
This got rid of the Streamlit generated page components on the sidebar and let me keep all my other components.
Add this to all the pages you want to hide the page list from, but there should not be any ‘ul’ in your page except for the page list.
st.markdown("<style> ul {display: none;} </style>", unsafe_allow_html=True)
tree
├───pzApi
│ │ asgi.py
│ │ db.sqlite3
│ │ manage.py
│ │ settings.py
│ │ urls.py
│ │ wsgi.py
│ │ __init__.py
│ │
│ └───api
│ │ apps.py
│ │ funcs.py
│ │ urls.py
│ │ views.py
│ │ __init__.py
Procfile
release: python pzApi/manage.py migrate
web: gunicorn --bind 127.0.0.1:8000 pzApi.wsgi:application
I'm still trying to deploy this web app and after fixing the wsgi.py not found error by putting it in the same dir as manage.py, now I have another problem where the 'api' folder is not being found, any help is appreciated!
in Django, my model.py is out of my App modules I want to import all the classes from that model.py to all modules. I searched a lot I didn't get anything can anyone help me
here is my directory
C:
│ manage.py
│ **models.py**
│ mysqlclient-1.4.6-cp39-cp39-win_amd64.whl
│
│
├───myshop
│ │ asgi.py
│ │ settings.py
│ │ urls.py
│ │ wsgi.py
│ │ __init__.py
│ │
├───products
│ │ admin.py
│ │ apps.py
│ │ tests.py
│ │ views.py
│ │ __init__.py
│ │
│ ├───migrations
│ │ │ __init__.py
1Trying to use Django 1.10 to create a file upload system (similar to this example here. My main problem is, no matter how hard I try, Django is unable to show my webpage (404 error). I have no idea why. I'm following the 1.9 example and it should be working, as far as I can tell. I've attached the error and my data tree
[D:.
│ db.sqlite3
│ manage.py
│
├───.idea
│ courseworkupload.iml
│ misc.xml
│ modules.xml
│ workspace.xml
│
├───courseworkupload
│ │ settings.py
│ │ urls.py
│ │ wsgi.py
│ │ __init__.py
│ │
│ └───__pycache__
│ settings.cpython-35.pyc
│ urls.cpython-35.pyc
│ wsgi.cpython-35.pyc
│ __init__.cpython-35.pyc
│
├───upload
│ │ admin.py
│ │ apps.py
│ │ forms.py
│ │ models.py
│ │ tests.py
│ │ urls.py
│ │ views.py
│ │ __init__.py
│ │
│ ├───migrations
│ │ │ 0001_initial.py
│ │ │ __init__.py
│ │ │
│ │ └───__pycache__
│ │ 0001_initial.cpython-35.pyc
│ │ __init__.cpython-35.pyc
│ │
│ ├───templates
│ │ Final.html
│ │ upload.html
│ │
│ ├───uploadedfiles
│ └───__pycache__
│ admin.cpython-35.pyc
│ forms.cpython-35.pyc
│ models.cpython-35.pyc
│ urls.cpython-35.pyc
│ views.cpython-35.pyc
│ __init__.cpython-35.pyc
│
├───Uploadedfiles
└───__pycache__
manage.cpython-35.pyc][2]
Views.py below
from django.http import HttpResponseRedirect
from django.http import HttpResponse
from django.shortcuts import render
from django.core.urlresolvers import reverse
from .forms import docfieldform
from .models import Document
def upload(request):
if request.method == 'POST':
form = docfieldForm(request.POST, request.FILES)
if form.is_valid():
newdoc = Document (docfile=request.FILES['newfile'])
newdoc.save()
# Redirect to the document list after POST
return HttpResponseRedirect(reverse('upload'))
else:
form = docfieldform()
return render( request,'Final.html',)
You need to remove the .html in your URL conf so that it becomes url(r'^upload/$', upload, name='upload'). So if your browser url is http://127.0.0.1:8000/upload/upload/ it should go to the upload view.
To display content in upload.html replace render(request, 'Final.html') with render(request, 'upload.html') in your upload view