Develop Reference

How can I create a dynamic logging structure in scrapy?

I am building a scraper using scrapy framework. I want to log every run in a structured manner. I already created dynamic logging using the timestamp in the setting.py file
LOG_DIR = os.path.join(BASE_DIR, 'logs')
if not os.path.exists(LOG_DIR):
try:
os.mkdir(LOG_DIR)
except OSError as e:
pass
LOG_FILE = os.path.join(LOG_DIR, f'{datetime.now().timestamp()}.log')
but I further want to store logs in a nested directory structure. that can help me access the accurate logs more easily
i.e.
- ROOT
|- logs
| |- 21-02-2022
| |- 22-02-2022
| | |- us
| | |- UK
| | | |- t-shirt
| | | |- hoodie
| | | | |- 1653029099.520938.log
Can someone please direct me on how can I achieve this?

Python Flask session variables aren't recognized by imported functions

My Flask application.py was getting a bit large and I wanted to spread the content (classes/functions) across a couple of additional py files. I placed class definitions in an appClasses.py and that is imported fine by application.py using
from appClasses import *
And, I placed some function definitions into appFunction.py and import those into application.py using
from appFunction import *
The functions in appFunction.py make use of flask session variables. When the same functions are in application.py there is no issue with session variable references (and app behaves as expected) but as soon as I cut/paste into appFunction.py and import functions I get a name error exception when the first function using session vars is called:
NameError: name 'session' is not defined
Here's a sample function:
def load_hgw_dict():
print("LOAD HGW DICT: called")
if 'hgw_dict_loaded' in session:
print("LOAD HGW DICT: hgw dict is available")
hgw_dict=session['hgw_dict']
else:
print("LOAD HGW DICT: hgw dict not available...adding")
session['hgw_dict_loaded']=True
list_of_hgws="/home/stbweb/LIST_OF_HENBGW"
hgw_file = open(list_of_hgws, "r")
hgw_dict=dict()
for line in hgw_file:
hgw_common, hgw_ip, hgw_hostname = line.split()
hgw_dict[hgw_common]= { 'ip':hgw_ip, 'hostname':hgw_hostname }
hgw_file.close()
print("LOAD HGW DICT: hgw dict =", hgw_dict)
return hgw_dict
Session is created by flask itself when flask application.py is run.
Here's the project dir structure...some files removed for compactness
.
|-- appClasses.py
|-- appHeNBGW.py
|-- application.ini
|-- application.py
|-- application.pyc
|-- cert.pem
|-- flask_session
| -- 2029240f6d1128be89ddc32729463129
|-- __init.py__
|-- key.pem
|-- __pycache__
|<snip>
|-- README.md
|-- set_dev_env
|-- static |<snip>
|-- templates
| <snip>
-- wsgi.py

Function not defined when running tests in Python package

I am creating a Python package/library. My directory structure looks like this:
my_package/
|-- my_package/
| |-- tests/
| | |-- __init__.py
| | |-- my_tests.py
| |
| |-- __init__.py
| |-- main.py
|
|-- setup.py
I have all my functions in the main.py file:
def sum_nums(a,b):
res = a + b
return(res)
def mult_nums(a,b):
res = a * b
return(res)
def sub_nums(a,b):
res = a - b
return(res)
my_tests.py looks like this:
from unittest import TestCase
import my_package
def test_sum():
assert sum_nums(3,4) == 7
def test_mult():
assert mult_nums(3,4) == 12
def test_sub():
assert sub_nums(3,4) == -1
When I run my tests from the package root directory as follows:
python setup.py test
... I get the following error:
NameError: name 'sum_nums' is not defined
Is my package directory structure correct?
Am I missing an _ init _.py file?
Does every directory require an _ init _.py file?
Is it okay to place all my functions inside a single main.py file
without using if name == "main"?

You need to indicate that the functions under test came for the my_package package like:
from unittest import TestCase
import my_package
def test_sum():
assert my_package.sum_nums(3,4) == 7
def test_mult():
assert my_package.mult_nums(3,4) == 12
def test_sub():
assert my_package.sub_nums(3,4) == -1

tests should not be in the package so move it up one dir. See sanic or any other module in github for example. Your functions need to be available in init.py. You can import them like is done in sanic.
https://github.com/huge-success/sanic
You also need
from my_package import sum_nums, mult_nums, sub_nums
Or prefix with my_package.

Get absolute path in Python

I'm having a problem with assigning static files containing resources.
My working directory structure is:
|- README.md
|- nlp
| |-- morpheme
| |-- |-- morpheme_builder.py
| |-- fsa_setup.py
| - tests
| |-- test_fsa.py
| - res
| |-- suffixes.xml
The code for fsa_setup.py is:
class FSASetup():
fsa = None
def get_suffixes():
list_suffix = list()
file = os.path.realpath("../res/suffixes.xml")
.....
if __name__ == "__main__":
FSASetup.get_suffixes()
The code for morpheme_builder.py is:
class MorphemeBuilder:
def get_all_words_from_fsa(self):
......
if __name__ == "__main__":
FSASetup.get_suffixes()
When it is called in fsa_setup.py, the file path's value is '\res\suffixes.xml' and that is correct, but when the other case realized, the file path value is '\nlp\res\suffixes.xml'.
I know how it works like this. So how can I give the path of the resource to the file.

The problem is that morpheme_builder.py is in the directory morphem. So when you say ../res/suffixes.xml it will go on directory back ... so it will go to nlp/res/suffixes.xml. What about if you use os.path.abspath("../res/suffixes.xml")?

How to structure celery tasks

I have 2 types of task: async tasks and schedule tasks. So, here is my dir structure:
proj
|
-- tasks
|
-- __init__.py
|
-- celeryapp.py => celery instance defined in this file.
|
-- celeryconfig.py
|
-- async
| |
| -- __init__.py
| |
| -- task1.py => from proj.tasks.celeryapp import celery
| |
| -- task2.py => from proj.tasks.celeryapp import celery
|
-- schedule
|
-- __init__.py
|
-- task1.py => from proj.tasks.celeryapp import celery
|
-- task2.py => from proj.tasks.celeryapp import celery
But when I run celery worker like below, it does not work. It can not accept the task from celery beat scheduler.
$ celery worker --app=tasks -Q my_queue,default_queue
So, is there any best practice on multiple task files organization?

Based on celery documentation you can import a structure of celery tasks like this:
For example if you have an (imagined) directory tree like this:
|
|-- foo
| |-- __init__.py
| |-- tasks.py
|
|-- bar
|-- __init__.py
|-- tasks.py
Then calling app.autodiscover_tasks(['foo', bar']) will result in the modules foo.tasks and bar.tasks being imported.

Celery tasks can be async, sync or scheduled depends on its invocation
task.delay(arg1,arg2) #will be async
task.delay(arg1,arg2).get() #will be sync
task.delay(arg1,arg2).get() #will be sync
task.apply_async(args = [arg1,arg2], {'countdown' : some_seconds}) #async with delay
There's a lot of invocations depending on your needs
However, you must start celery with -B flag to enable celery scheduler
$ celery worker --app=tasks -B -Q my_queue,default_queue
So the way you take to organize your tasks is something personal and it deppends on your project complexity, but I think that organize them by its type of synchronism wouldn't be the best option.
I've googled this topic and I haven't found any guide or advise, but I've read some cases that organize their task by their functionality.
I've followed this advise, because this isn't a pattern, in my projects. Here one example of how I did
your_app
|
-- reports
|
-- __init__.py
-- foo_report.py
-- bar_report.py
-- tasks
|
-- __init__.py
-- report_task.py
-- maintenance
|
-- __init__.py
-- tasks
|
-- __init__.py
-- delete_old_stuff_task.py
-- twitter
|
-- __init__.py
-- tasks
|
-- __init__.py
-- batch_timeline.py