Develop Reference

Python class with a decorator is being flagged by pylint as unused-import

For example, let's say I have three Python files; one file is the init.py which has routines for creating the namespaces of my Flask API and initializing the Flask app:
from flask import Flask
app = Flask(...)
... initialize the namespace etc. ...
... and another Python file with the definitions of the API Resource subclasses that exist in the enpoint.py file. It contains a few Python classes that make use of decorators from Flask-RESTX to wire up the endpoints:
#namespace.route("/path")
class EndpointApi:
def get():
...
The third file is main.py which simply starts the Flask server running. Unfortunately for me though, it contains an import which is flagged by pylint.
My app is working fine, but when I run pylint, it tells me there are unused imports. If I remove the imports, then the logic in the decorator that adds the route to the Flask API does not execute, and the result is that the endpoint is no longer added to the API.
Is there some way to add a class file (like endpoints.py) without importing it? I want pylint to stop warning me about unused imports, when clearly I'm using the decorator to call some global function that adds the API Resource handlers to Flask.
Sure, I could ignore the pylint error with a comment, but is there a better way? I am truly disgusted with placing a comment on every line of an import statement which I'm sure is not an "unused-import" (I have about 30).
Obviously, I could just refactor the decorator pattern into its constituent parts, and extract the relevant code to be included inside the main.py file. The equivalent code would look like this in main.py:
from endpoint import EndpointApi
EndpointApi = namespace.route("/path")(EndpointApi)
This is exactly the same code that's run in the decorator, so pylint considers my EndpointApi to be unused even though the decorator is using it to append a "/path" route to the namespace. Removing the decorator and adding the equivalent code to main.py decreases maintainability because now the relevant parts of EndpointApi are in two different files instead of all being defined one.
Edit
No, from endpoint import * makes it worse:
main.py:3:0: W0614: Unused import(s) EndpointApi, Resource and ns from wildcard import of endpoint (unused-wildcard-import)
Minimal example
flask-restx-hello $ pylint *py; for x in *py; do echo $x; cat $x; done
************* Module main
main.py:3:0: W0611: Unused EndpointApi imported from endpoint (unused-import)
------------------------------------------------------------------
Your code has been rated at 9.29/10 (previous run: 7.86/10, +1.43)
endpoint.py
"""docstring"""
from flask_restx import Resource
from init import ns
#ns.route('/hello')
class EndpointApi(Resource):
"""docstring"""
def get(self):
"""docstring"""
return {'hello': 'world'}
init.py
"""docstring"""
from flask import Flask
from flask_restx import Api
app = Flask(__name__)
api = Api(app)
ns = api.namespace('sick', description='crazy', path='/root/haha')
main.py
"""docstring"""
from init import app
from endpoint import EndpointApi
if __name__ == '__main__':
app.run(debug=True)
$ cat requirements.txt
aniso8601==9.0.1
attrs==22.1.0
click==8.0.4
dataclasses==0.8
Flask==2.0.3
flask-restx==0.5.1
importlib-metadata==4.8.3
itsdangerous==2.0.1
Jinja2==3.0.3
jsonschema==4.0.0
MarkupSafe==2.0.1
pkg-resources==0.0.0
pyrsistent==0.18.0
pytz==2022.2.1
six==1.16.0
typing-extensions==4.1.1
Werkzeug==2.0.3
zipp==3.6.0

Pylint isn't wrong. You aren't using endpoint anywhere in main.py. The only reason you're importing endpoint is to execute the decorator. Which is fine, but there is no way for pylint to know that.
In this case, it's ok to ignore the warning.

If you don't want to disable each of the import errors on every single link like this:
# pylint: disable=no-name-in-module
you could use:
# pylint: disable=import-error
Commenting this once in the file will disable pylint import errors for the entire file.
Additionally, this post on stack overflow may help:
Is it possible to ignore one single specific line with Pylint?
For more info on W0611 pylint unused imports:
https://pylint.pycqa.org/en/latest/user_guide/messages/warning/unused-import.html

Why can't I reference this class from another app in my Django project?

I'm using Django and Python 3.7. I have two applications in my project -- common and mainsite . In common, I have a file "model_utils.py" at the root of my application (right next to models.py). It contains this code
class RawCol(Expression):
def __init__(self, model, field_name):
field = model._meta.get_field(field_name)
self.table = model._meta.db_table
self.column = field.column
super().__init__(output_field=CharField())
def as_sql(self, compiler, connection):
sql = f'"{self.table}"."{self.column}"'
return sql, []
How do I reference this class from my other application, "mainsite"? I put this at the top of one of my files in mainsite ..
from common import RawCol
but when I run some tests, I get this error ...
ImportError: cannot import name 'RawCol' from 'common' (/Users/davea/Documents/workspace/mainsite_project/common/__init__.py)
Edit: Project structure at the top level directories looks like ...
+ common
+ mainsite
+ mainsite_project
+ manage.py
+ templates
+ venv

Try from common.model_utils import RawCol
instead of from common import RawCol
You always need to specify the exact .py file (without the .py ending) to import from.
If it still doesn't work, it can be a circular import problem.
If you try to import something from mainsite.model into common.model_utils and the other way around, you are creating an impossible import loop.
You can fix this by creating a seperate file like common/dependent_model.py and putting only the RawCol() class in there without any import from mainsite. Like this, the both files are not importing from each other (which doesn't work).

Locating Cookiecutter Extensions

I'm working on creating my first cookiecutter. By and large, this has gone well, but I now want to add a jinja2 filter of my own.
In line with the comments in this issue, I've created a new Jinja2 extension much like the one here. Full code for this extension is here:
https://github.com/seclinch/sigchiproceedings-cookiecutter/commit/5a314fa7207fa8ab7b4024564cec8bb1e1629cad#diff-f4acf470acf9ef37395ef389c12f8613
However, the following simple example demonstrates the same error:
# -*- coding: utf-8 -*-
from jinja2.ext import Extension
def slug(value):
return value
class PaperTitleExtension(Extension):
def __init__(self, environment):
super(PaperTitleExtension, self).__init__(environment)
environment.filters['slug'] = slug
I've dropped this code into a new jinja2_extensions directory and added a simple __init__.py as follows:
# -*- coding: utf-8 -*-
from paper_title import PaperTitleExtension
__all__ = ['PaperTitleExtension']
Based on this piece of documentation I've also added the following to my `cookiecutter.json' file:
"_extensions": ["jinja2_extensions.PaperTitleExtension"]
However, running this generates the following error:
$ cookiecutter sigchiproceedings-cookiecutter
Unable to load extension: No module named 'jinja2_extensions'
I'm guessing that I'm missing some step here, can anyone help?

Had the same issue, try executing with python3 -m option
My extension in extensions/json_loads.py
import json
from jinja2.ext import Extension
def json_loads(value):
return json.loads(value)
class JsonLoadsExtension(Extension):
def __init__(self, environment):
super(JsonLoadsExtension, self).__init__(environment)
environment.filters['json_loads'] = json_loads
cookiecutter.json
{
"service_name": "test",
"service_slug": "{{ cookiecutter.service_name.replace('-', '_') }}",
...
"_extensions": ["extensions.json_loads.JsonLoadsExtension"]
}
Then I executed with python3 -m cookiecutter . no_input=True timestamp="123" extra_dict="{\"features\": [\"redis\", \"grpc_client\"]}" -f and it works fine.

I ran into a similar error earlier.
Unable to load extension: No module named 'cookiecutter_repo_extensions'
The problem was that in my case there was a dependency to the 'cookiecutter-repo-extension' which I had not installed in my Virtual Environment.

The directory containing your extension needs to be on your PYTHONPATH.
https://github.com/cookiecutter/cookiecutter/issues/1211#issuecomment-522226155
A PR to improve the docs would be appreciated 📖 ✍️ 🙏

Getting scrapy project settings when script is outside of root directory

I have made a Scrapy spider that can be successfully run from a script located in the root directory of the project. As I need to run multiple spiders from different projects from the same script (this will be a django app calling the script upon the user's request), I moved the script from the root of one of the projects to the parent directory. For some reason, the script is no longer able to get the project's custom settings in order to pipeline the scraped results into the database tables. Here is the code from the scrapy docs I'm using to run the spider from a script:
def spiderCrawl():
settings = get_project_settings()
settings.set('USER_AGENT','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)')
process = CrawlerProcess(settings)
process.crawl(MySpider3)
process.start()
Is there some extra module that needs to be imported in order to get the project settings from outside of the project? Or does there need to be some additions made to this code? Below I also have the code for the script running the spiders, thanks.
from ticket_city_scraper.ticket_city_scraper import *
from ticket_city_scraper.ticket_city_scraper.spiders import tc_spider
from vividseats_scraper.vividseats_scraper import *
from vividseats_scraper.vividseats_scraper.spiders import vs_spider
tc_spider.spiderCrawl()
vs_spider.spiderCrawl()

Thanks to some of the answers already provided here, I realised scrapy wasn't actually importing the settings.py file. This is how I fixed it.
TLDR: Make sure you set the 'SCRAPY_SETTINGS_MODULE' variable to your actual settings.py file. I'm doing this in the __init__() func of Scraper.
Consider a project with the following structure.
my_project/
main.py # Where we are running scrapy from
scraper/
run_scraper.py #Call from main goes here
scrapy.cfg # deploy configuration file
scraper/ # project's Python module, you'll import your code from here
__init__.py
items.py # project items definition file
pipelines.py # project pipelines file
settings.py # project settings file
spiders/ # a directory where you'll later put your spiders
__init__.py
quotes_spider.py # Contains the QuotesSpider class
Basically, the command
scrapy startproject scraper was executed in the my_project folder, I've added a run_scraper.py file to the outer scraper folder, a main.py file to my root folder, and quotes_spider.py to the spiders folder.
My main file:
from scraper.run_scraper import Scraper
scraper = Scraper()
scraper.run_spiders()
My run_scraper.py file:
from scraper.scraper.spiders.quotes_spider import QuotesSpider
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
import os
class Scraper:
def __init__(self):
settings_file_path = 'scraper.scraper.settings' # The path seen from root, ie. from main.py
os.environ.setdefault('SCRAPY_SETTINGS_MODULE', settings_file_path)
self.process = CrawlerProcess(get_project_settings())
self.spider = QuotesSpider # The spider you want to crawl
def run_spiders(self):
self.process.crawl(self.spider)
self.process.start() # the script will block here until the crawling is finished
Also, note that the settings might require a look-over, since the path needs to be according to the root folder (my_project, not scraper).
So in my case:
SPIDER_MODULES = ['scraper.scraper.spiders']
NEWSPIDER_MODULE = 'scraper.scraper.spiders'
And repeat for all the settings variables you have!

It should work , can you share your scrapy log file
Edit:
your approach will not work
because ...when you execute the script..it will look for your default settings in
if you have set the environment variable ENVVAR
if you have scrapy.cfg file in you present directory from where you are executing your script and if that file points to valid settings.py directory ,it will load those settings...
else it will run with vanilla settings provided by scrapy ( your case)
Solution 1
create a cfg file inside the directory (outside folder) and give it a path to the valid settings.py file
Solution 2
make your parent directory package , so that absolute path will not be required and you can use relative path
i.e python -m cron.project1
Solution 3
Also you can try something like
Let it be where it is , inside the project directory..where it is working...
Create a sh file...
Line 1: Cd to first projects location ( root directory)
Line 2 : Python script1.py
Line 3. Cd to second projects location
Line 4: python script2.py
Now you can execute spiders via this sh file when requested by django

I have used this code to solve the problem:
from scrapy.settings import Settings
settings = Settings()
settings_module_path = os.environ.get('SCRAPY_ENV', 'project.settings.dev')
settings.setmodule(settings_module_path, priority='project')
print(settings.get('BASE_URL'))

this could happen because you are no longer "inside" a scrapy project, so it doesn't know how to get the settings with get_project_settings().
You can also specify the settings as a dictionary as the example here:
http://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

I used the OS module for this problem.
The python file you are running is in one directory and your scrapy project is in a different directory. You can not simply just import the python spider and run on this python script because the current directory you are working in does not have the settings.py file or the scrapy.cfg.
import os
To show the current directory you are working in use the following code:
print(os.getcwd())
From here you are going to want to change the current directory:
os.chdir(\path\to\spider\folder)
Lastly, tell os which command to execute.
os.system('scrape_file.py')

This is an addition to the answer of malla.
You can configure the settings, pipelines, spiders, etc modules variable. You dont need to pass them as strings. The big advantage is that you can run that spider from different places and you dont need to adjust the strings in the settings. You can do both: run from script (from anywhere, even from multiple different roots) and run with scrapy crawl without adjusting:
from ticket_city_scraper.ticket_city_scraper import settings # your setting module
def run_spider():
os.environ.setdefault('SCRAPY_SETTINGS_MODULE', settings.__name__)
process = CrawlerProcess(get_project_settings())
process.crawl(MySpider3)
process.start()
You can make the setting itself variable:
from . import spiders # your spiders module
from . import pipelines # your pipelines module
def get_full_package_name_for_class(clazz) -> str:
return ".".join([clazz.__module__, clazz.__name__])
SPIDER_MODULES = [spiders.__name__]
NEWSPIDER_MODULE = spiders.__name__
ITEM_PIPELINES = {
get_full_package_name_for_class(pipelines.YourScrapyPipeline): 300
}

importing error in python while importing two classes

I am working with a python script, and i face importing problem when i try to import a class from another python script. Here is how my python project folder looks:
Mysql_Main/
checks.py
Analyzer/
config.py
ip.py
op.py
__init__.py
Now i want to import two classes named: Config() and Sqlite() from config.py into the checks.py script.How do i do it?
This is what i tried, but its resulting in an error!
inside checks.py:
from Analyzer import config
config = config.Config()
sqlite = config.Sqlite()
The problem is that Config class is imported properly, but Sqlite class is not getting imported.It is showing error - Config instance has no attribute 'Sqlite'

When you do:
config = config.Config()
You write over the variable config and it no longer points to the module config. It stores the new Config instance.
Try:
from Analyzer import config
config_instance = config.Config()
sqlite_instance = config.Sqlite()