Develop Reference

Duplicate Logs are generated while using Pytest Parametrize fixture in Python 3.7 [duplicate]

This question already has answers here:
Duplicate log output when using Python logging module
(17 answers)
Closed 2 years ago.
I am getting duplicate logs for each iteration in Pytest Parametrize fixture.
Sample File -
input1 = 1
input2 = 2
input3 = 3
#pytest.mark.parametrize("input",[(input1),(input2),(input3)])
def test_numbers(input):
file_name = inspect.stack()[0][3]
log = customLogger(file_name=file_name)
log.info(input)
Logger Code -
def customLogger(file_name,log_level=logging.INFO):
'''
Important Logging Information
CRITICAL = 50
FATAL = CRITICAL
ERROR = 40
WARNING = 30
WARN = WARNING
INFO = 20
DEBUG = 10
NOTSET = 0
'''
# Gets the name of the class / method from where this method is called
logger_name = inspect.stack()[1][3]
logger = logging.getLogger(logger_name)
# By default, log all messages
logger.setLevel(logging.DEBUG) # dont change this setting
file_handler = logging.FileHandler("../logs/"+file_name +".log", mode='a')
file_handler.setLevel(log_level)
formatter = logging.Formatter('%(asctime)s - %(funcName)s - %(levelname)s: %(message)s',
datefmt='%m/%d/%Y %I:%M:%S %p')
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
return logger
I get a response in my log file like this -
07/07/2020 02:01:52 AM - test_number - INFO: 1
07/07/2020 02:01:52 AM - test_number - INFO: 2
07/07/2020 02:01:52 AM - test_number - INFO: 2
07/07/2020 02:01:52 AM - test_number - INFO: 3
07/07/2020 02:01:52 AM - test_number - INFO: 3
07/07/2020 02:01:52 AM - test_number - INFO: 3
I am out of my mind in resolving this, I am thinking to write a Log cleaner method to remove duplicates from the logs. Please let me know if I am missing something here ?

Each time you call customLogger, you create and add a new logging handler, that will also log the line, thus the multiplication of logs.
There are several possibilities to ensure that, the easiest is probably just to clear the handlers before adding a new one:
...
file_handler.setFormatter(formatter)
logger.handlers = []
logger.addHandler(file_handler)

Airflow BashOperator OSError: [Errno 2] No such file or directory

I keep getting the same error from a scheduled BashOperator that is currently back-filling (it's over a month "behind").
[2018-06-10 22:06:33,558] {base_task_runner.py:115} INFO - Running: ['bash', '-c', u'airflow run dag_name task_name 2018-03-14T00:00:00 --job_id 50 --raw -sd DAGS_FOLDER/dag_file.py']
Traceback (most recent call last):
File "/anaconda/bin//airflow", line 27, in <module>
args.func(args)
File "/anaconda/lib/python2.7/site-packages/airflow/bin/cli.py", line 387, in run
run_job.run()
File "/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line 198, in run
self._execute()
File "/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line 2512, in _execute
self.task_runner.start()
File "/anaconda/lib/python2.7/site-packages/airflow/task_runner/bash_task_runner.py", line 29, in start
self.process = self.run_command(['bash', '-c'], join_args=True)
File "/anaconda/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py", line 120, in run_command
universal_newlines=True
File "/anaconda/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/anaconda/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
[2018-06-10 22:06:33,633] {sequential_executor.py:47} ERROR - Failed to execute task Command 'airflow run dag_name task_name 2018-03-14T00:00:00 --local -sd /var/lib/airflow/dags/dag_file.py' returned non-zero exit status 1.
I remember seeing something that suggested this might be a permissions issue, but I can't figure out which permissions might be involved.
I'm using a systemd configuration--and at my wit's end--I've taken to running the airflow webserver and scheduler as root.
I can take the list in the first line and enter it verbatim in an ipython shell as args to a subprocess.Popen instance (as it is in airflow/task_runner/base_task_runner.py; save no envs) and not only does it run but it correctly informs the airflow db that the task is complete. I can do this as user Airflow, root, or ubuntu.
I've added /anaconda/bin to the PATH in .bashrc for Airflow, root, ubuntu, and /etc/bash.bashrc in addition to the value for AIRFLOW_HOME which is also in my env file /etc/airflow.
This is what my systemd entry looks like:
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/airflow
User=root
Group=root
Type=simple
ExecStart=/anaconda/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
My env file:
PATH=$PATH:/anaconda/bin/
AIRFLOW_HOME=/var/lib/airflow
AIRFLOW_CONFIG=$AIRFLOW_HOME/airflow.cfg
Using apache-airflow==1.9.0 and desperate for a solution. Thanks in advance.
Airflow.cfg:
[core]
airflow_home = /var/lib/airflow
dags_folder = /var/lib/airflow/dags
base_log_folder = /var/lib/airflow/logs
remote_log_conn_id =
encrypt_s3_logs = False
logging_level = INFO
logging_config_class =
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
executor = SequentialExecutor
sql_alchemy_conn = {actual value hidden}
sql_alchemy_pool_size = 5
sql_alchemy_pool_recycle = 3600
parallelism = 4
dag_concurrency = 2
dags_are_paused_at_creation = True
non_pooled_task_slot_count = 16
max_active_runs_per_dag = 1
load_examples = False
plugins_folder = /var/lib/airflow/plugins
fernet_key = {actual value hidden}
donot_pickle = False
dagbag_import_timeout = 30
task_runner = BashTaskRunner
default_impersonation =
security =
unit_test_mode = False
task_log_reader = file.task
enable_xcom_pickling = True
killed_task_cleanup_time = 60
[cli]
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8080
[api]
auth_backend = airflow.api.auth.backend.default
[operators]
default_owner = root
default_cpus = 1
default_ram = 512
default_disk = 512
default_gpus = 0
[webserver]
base_url = http://localhost:8080
web_server_host = 0.0.0.0
web_server_port = 8080
web_server_ssl_cert =
web_server_ssl_key =
web_server_worker_timeout = 120
worker_refresh_batch_size = 1
worker_refresh_interval = 60
secret_key = temporary_key
workers = 1
worker_class = sync
access_logfile = -
error_logfile = -
expose_config = False
authenticate = False
filter_by_owner = False
owner_mode = user
dag_default_view = tree
dag_orientation = LR
demo_mode = False
log_fetch_timeout_sec = 5
hide_paused_dags_by_default = False
page_size = 100
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
smtp_port = 25
smtp_mail_from = airflow#example.com
[celery]
...
[dask]
cluster_address = 127.0.0.1:8786
[scheduler]
job_heartbeat_sec = 120
scheduler_heartbeat_sec = 120
run_duration = -1
min_file_process_interval = 0
dag_dir_list_interval = 300
print_stats_interval = 300
child_process_log_directory = /var/lib/airflow/logs/scheduler
scheduler_zombie_task_threshold = 900
catchup_by_default = True
max_tis_per_query = 0
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
max_threads = 1
authenticate = False
[ldap]
...
[mesos]
...
[kerberos]
...
[github_enterprise]
...
[admin]
hide_sensitive_variable_fields = True
Adding ls -hal
root#ubuntu:/var/lib/airflow# ls -hal /var
total 52K
drwxr-xr-x 13 root root 4.0K Jun 3 11:58 .
root#ubuntu:/var/lib/airflow# ls -hal /var/lib
total 164K
drwxr-xr-x 42 root root 4.0K Jun 10 19:00 .
root#ubuntu:/var/lib/airflow# ls -hal
total 40K
drwxr-xr-x 4 airflow airflow 4.0K Jun 11 06:41 .
drwxr-xr-x 42 root root 4.0K Jun 10 19:00 ..
-rw-r--r-- 1 airflow airflow 13K Jun 11 06:41 airflow.cfg
-rw-r--r-- 1 airflow airflow 579 Jun 10 19:00 airflow.conf
drwxr-xr-x 2 airflow airflow 4.0K Jun 10 21:27 dags
drwxr-xr-x 4 airflow airflow 4.0K Jun 10 20:31 logs
-rw-r--r-- 1 airflow airflow 1.7K Jun 10 19:00 unittests.cfg
root#ubuntu:/var/lib/airflow# ls -hal dags/
total 16K
drwxr-xr-x 2 airflow airflow 4.0K Jun 10 21:27 .
drwxr-xr-x 4 airflow airflow 4.0K Jun 11 06:41 ..
-rw-r--r-- 1 airflow airflow 3.4K Jun 10 21:26 dag_file.py
-rw-r--r-- 1 airflow airflow 1.7K Jun 10 21:27 dag_file.pyc
and contents of dag_file.py:
import airflow
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'root',
'run_as': 'root',
'depends_on_past': True,
'start_date': datetime(2018, 2, 20),
'email': ['myemail#gmail.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'end_date': datetime(2018, 11, 15),
}
env = {
'PSQL': '{obscured}',
'PATH': '/anaconda/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin',
'PWD': '/home/ubuntu/{obs1}/',
'HOME': '/home/ubuntu',
'PYTHONPATH': '/home/ubuntu/{obs1}',
}
dag = DAG(
'dag_name',
default_args=default_args,
description='',
schedule_interval=timedelta(days=1))
t1 = BashOperator(
env=env,
task_id='dag_file',
bash_command='export PYTHONPATH=/home/ubuntu/{obs1} && /anaconda/bin/ipython $PYTHONPATH/{obs2}/{obs3}.py {{ ds }}',
dag=dag)
And I remind you that this runs correctly as airflow, root, and ubuntu: airflow run dag_name dag_file 2018-03-17T00:00:00 --job_id 55 --raw -sd DAGS_FOLDER/dag_file.py

It looks like python version mismatch, edit you .bashrc with proper python version and run:
source .bashrc
This will resolve you issue.
For my case we are usingexport PATH="/opt/miniconda3/bin":$PATH
Also to check how I can to this:
/opt/miniconda3/bin/python /opt/miniconda3/bin/airflow
This is how I used to run the airflow.

On Airflow v1.10.0 you just specify the filepath, without the space at the end anymore.
Example:
compact_output_task = BashOperator(**{
'task_id': 'compact_output',
'bash_command': './compact_output.sh',
'xcom_push': True,
})

Systemd EnvironmentFile won't expand the variable inside it so your PATH will only look at /anaconda/bin if you just want to extend your PATH it's better to use
ExecStart=/bin/bash -c 'PATH=/path/to/venv/bin/:$PATH exec /path/to/airflow scheduler
this solved my issue with No such file or directory, because airflow couldn't find the binary that I was calling inside my bash operator.

using flask_apscheduler with gunicorn

I am trying to schedule tasks within my flask app to fire now and then... this works great with flask development server, but once I fire the same app with gunicorn it is not working as intended, no errors, but no tasks are being fired..
from flask import Flask
from flask_apscheduler import APScheduler
class Config(object):
JOBS = [
{
'id': 'job1',
'func': 'myapp:job1',
'args': (1, 2),
'trigger': 'interval',
'seconds': 5
}
]
SCHEDULER_API_ENABLED = True
def job1(a, b):
print(str(a) + ' ' + str(b))
app = Flask(__name__)
if __name__ == '__main__':
app.config.from_object(Config())
scheduler = APScheduler()
scheduler.init_app(app)
scheduler.start()
app.run()
For the purpose I include this sample, python myapp.py works fine and prints 1 2 every fifth second, however, running gunicorn -w 1 -b 0.0.0.0:5000 myapp:app it is not printing 1 2 every fifth second, why?

so, 2 things
1) use logging, you get a LOT more info:
2)
you need to declare outside your if __name__=='__main__': block:
1 from flask import Flask
2 from flask_apscheduler import APScheduler
3 import logging
4
5 logging.basicConfig(level=logging.DEBUG,
6 format='%(asctime)s %(levelname)s %(message)s')
7
8 logger = logging.getLogger(__name__)
9
10 class Config(object):
11 JOBS = [
12 {
13 'id': 'job1',
14 'func': 'scratch:job1',
15 'args': (1, 2),
16 'trigger': 'interval',
17 'seconds': 5
18 }
19 ]
20
21 SCHEDULER_API_ENABLED = True
22
23
24 def job1(a, b):
25 logger.info(str(a) + ' ' + str(b))
26
27 app = Flask(__name__)
28 app.config.from_object(Config())
29
30 scheduler = APScheduler()
31 scheduler.init_app(app)
32 scheduler.start()
33
34
35 if __name__ == '__main__':
36 app.run()
~
~
output:
[2017-02-15 14:29:39 +0000] [25122] [INFO] Booting worker with pid: 25122
2017-02-15 14:29:39,817 INFO Adding job tentatively -- it will be properly scheduled when the scheduler starts
2017-02-15 14:29:39,819 INFO Added job "job1" to job store "default"
2017-02-15 14:29:39,819 INFO Scheduler started
2017-02-15 14:29:39,820 DEBUG Looking for jobs to run
2017-02-15 14:29:39,820 DEBUG Next wakeup is due at 2017-02-15 14:29:44.816072-05:00 (in 4.995362 seconds)
2017-02-15 14:29:44,816 DEBUG Looking for jobs to run
2017-02-15 14:29:44,817 INFO Running job "job1 (trigger: interval[0:00:05], next run at: 2017-02-15 14:29:44 EST)" (scheduled at 2017-02-15 14:29:44.816072-05:00)
2017-02-15 14:29:44,817 INFO 1 2
The issue:
The job didn't fire originally. The issue was that your scheduler definition (was inside of your block if __name__ == '__main__' This code only gets hit when you call that python file directly (python myapp.py)
When you launched it with Gunicorn, the __name__ was no longer main, it was "myapp.py" so it never got to the part of the code that created and added the scheduler :)

Python Logging: Not showing on terminal

logging3.py
import sys
import logging
import first
import Logger
root = logging.getLogger()
root.addHandler(Logger.get_streamhandler())
root.warning('we did something in info')
hi = first.Foo()
hi.baz()
first.py
import logging
import Logger
first_logger = logging.getLogger(__name__)
first_logger.parent = False
first_logger.addHandler(Logger.get_streamhandler())
first_logger.info('in first')
class Foo:
def __init__(self):
print 'making sure we are in first.Foo'
self.logger = logging.getLogger('foo')
self.logger.addHandler(Logger.get_streamhandler())
self.logger.info('created Foo obj')
def baz(self):
self.logger.info('i dont know what baz is')
Logger.py
1 import logging
2 import sys
3
4
5 '''
6 NOTSET means inherit the log level from the parent logger
7 '''
8
9 LEVELS = { 'debug' : logging.DEBUG,
10 'info' : logging.INFO,
11 'warning' : logging.WARNING,
12 'error' : logging.ERROR,
13 'critical': logging.CRITICAL,
14 }
15
16 def getLevel(lvl):
17 return LEVELS.get(lvl) or logging.DEBUG
18
19 def get_streamhandler(lvl=None):
20 sh = logging.StreamHandler()
21 fmt = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
22 sh.setFormatter(logging.Formatter(fmt))
23 sh.setLevel(getLevel(lvl))
24 return sh
~
OUTPUT:
python logging3.py
2013-10-21 14:18:09,687 - first - INFO - in first
2013-10-21 14:18:09,687 - root - WARNING - we did something in info
making sure we are in first.Foo
Where is logging info for Foo object? <---------------
Also, can someone confirm that logging tree for the above is
root
----first
---------foo
or is it
root
----root.first
--------------root.first.foo

that's intentional. The logger has a loglevel (separate for console and file); you can set them with
foo.setConsoleLevel(logging.ERROR)
foo.setFileLevel(logging.INFO)
etc. If you change your loglevel to logging.INFO (which is 0x14 on my python 2.6 session) or below, then you will see the log messages.
Logging messages that are below the current loglevel are suppressed; only messages at or above the current level are passed through. This means that info messages can go to the file, but not to the screen, or that you can change the level to debug to get additional output while debugging, etc.

stdout and stderror from fabric task wrapped in celery

im running a simple task, triggered from a django view:
task = mock_deploy.delay()
mock_deploy is defined as:
from celery.decorators import task as ctask
from project.fabscripts.task.mock import *
#ctask(name="mock_deploy")
def mock_deploy():
print "hi form celery task b4 mockdeploy 1234"
output = execute(mock_deploy2)
return "out: %s" % (output)
And the fabric task itself is defined as:
#task
def mock_deploy2():
lrun("ls -l /")
lrun("ifconfig eth0")
# i need to get the full output from those commands and save them to db
And now... I was trying to substitute stdout, overwriting fabric execute function:
def execute(task):
output = StringIO()
error = StringIO()
sys.stdout = output
sys.stderr = error
task()
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__
return (output.getvalue(), error.getvalue())
And I was trying to substitute stdout within fabric task. No matter what i did, the only output i was getting was a first line of "what fabric wants to do"
out: [localhost] local: ls -l /
Then, the whole output of the ls command was printed perfectly fine in celery log. Except for the missing one line of out: [localhost] local: ls -l / `9the one i managed to get as output)
[2012-06-14 21:33:56,587: DEBUG/MainProcess] TaskPool: Apply <function execute_and_trace at 0x36710c8> (args:('mock_deploy', '2a90d920-130a-4942-829b-87f4d5ebe80f', [], {}) kwargs:{'hostname': 's16079364', 'request': {'retries': 0, 'task': 'mock_deploy', 'utc': False, 'loglevel': 10, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'args': [], 'expires': None, 'is_eager': False, 'eta': None, 'hostname': 's16079364', 'kwargs': {}, 'logfile': None, 'id': '2a90d920-130a-4942-829b-87f4d5ebe80f'}})
[2012-06-14 21:33:56,591: DEBUG/MainProcess] Task accepted: mock_deploy[2a90d920-130a-4942-829b-87f4d5ebe80f] pid:22214
hi form celery task b4 mockdeploy 1234
total 3231728
-rw-r--r-- 1 root root 3305551148 2012-06-13 14:43 dumpling.sql
drwxr-xr-x 2 root root 4096 2012-05-09 17:42 bin
drwxr-xr-x 4 root root 4096 2012-02-14 15:21 boot
drwxr-xr-x 2 root root 4096 2012-03-09 14:10 build
drwxr-xr-x 2 root root 4096 2010-05-11 19:58 cdrom
-rw------- 1 root root 2174976 2012-05-23 11:23 core
drwxr-xr-x 15 root root 4080 2012-06-11 12:55 dev
drwxr-xr-x 135 root root 12288 2012-06-14 21:15 etc
drwxr-xr-x 6 root root 77 2012-05-21 14:41 home
...
A horrible horrible workaround is wrapping up a fabric run command to add a "> /tmp/logfile.log" on each command, then when the task is finished ill retrieve the file with scp...
My question in short is how do i get the full output of a fabric task when its triggered with celery?
The following did the trick:
#ctask(name="mock_deploy")
def mock_deploy():
env.roledefs.update({'remote': ['root#1.1.1.1',]})
output = StringIO()
sys.stdout = output
execute(mock_deploy2)
sys.stdout = sys.__stdout__
return output.getvalue()