Develop Reference

Logger that logs to console and file at different levels

I have a bootstrap script for a Raspberry Pi that runs in python. I am looking to create a logger that logs to a file as well as to the console.
I was going to do something like this:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s] %(message)s",
handlers=[
logging.FileHandler("{0}/{1}.log".format(logPath, fileName)),
logging.StreamHandler()
])
But what I would really like is to log INFO to the StreamHandler and DEBUG to the FileHandler... I cannot seem to figure that out.
Can anyone help me out?
Using Python 3.7.5

You could build the logger yourself (either through a config file or in pure python)
The tricky thing that I have wasted several hours on is forgetting to set the log level on the logger as well as on each of the handlers. Ensure that the logger is as permissive as the most permissive handler.
example script
# emits the info line to the console and
# both the info & debug lines to the log file
# test_pylog.py
import logging
log_format = logging.Formatter(
'%(asctime)s %(threadName)s %(levelname)s %(message)s'
)
logger = logging.getLogger(__name__)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(log_format)
logger.addHandler(console_handler)
file_handler = logging.FileHandler('logfile.txt')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(log_format)
logger.addHandler(file_handler)
logger.setLevel(logging.DEBUG)
if __name__ == '__main__':
logger.debug('Panic! at the disco')
logger.info('Weezer')

Logging with info level doesn't produce any output

I ran the following using both the Python shell, and run it as a Python file from the command line. I don's see my log output at all.
import logging
formatter = logging.Formatter('%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s')
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
stream_handler.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(stream_handler)
logger.info(("info logging"))

Your logging output was almost correct with the exception of setLevel. The logging level needs to be defined on the logger instance instead of the handler instance. Your code therefore only needs a very small tweak to make it work:
import logging
formatter = logging.Formatter(
'%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s'
)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(stream_handler)
logger.setLevel(logging.INFO)
logger.info("info logging")
This piece of code produces the following output:
2019-08-21 15:04:55,118,118 INFO [testHandler.py:11] info logging
Note, I also removed the double brackets on the logger.info call as these are not necessary.

Use logging.basicConfig() to initialize the logging system.

You need to set the level of your logger,
logger.setLevel(logging.INFO)
Below statement can be removed from your code,
stream_handler.setLevel(logging.INFO)

A logger and a handler can have different levels. You have only set the level for the handler, not the logger itself.
import logging
formatter = logging.Formatter('%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s')
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
stream_handler.setFormatter(formatter)
logger = logging.getLogger()
logger.setLevel(logging.INFO) # Required
logger.addHandler(stream_handler)
logger.info(("info logging"))

python log pops up on console

I am using below configuration for logging in python,
logger = logging.getLogger(__name__)
hdlr = logging.FileHandler(r'/var/log/resource_checker.log')
formatter = logging.Formatter('%(asctime)s : %(name)s : %(levelname)s : %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.INFO)
This works well for me. However if I add logging.basicConfig(level=logging.DEBUG) with existing logging configuration, all my logs pops up on screen screen (with additional DEBUG logs, which is good to have for me) and logfile does not populate at all.
how can i push those logs back to log file as my older config.?

This will log everything to a file, all from the logging.basicConfig:
import logging
logging.basicConfig(
level=logging.DEBUG,
filename='/var/log/resource_checker.log',
format='%(asctime)s : %(name)s : %(levelname)s : %(message)s'
)
logger = logging.getLogger(__name__)

Logging Python/NumPy console messages

I'm trying to figure out how to capture messages generated by Python/NumPy intractive shell when running my script. I would like to log all generated by console messages (errors, warnings) to same file as defined in my code log messages with time stamps:
def LogToFile():
global logger
logger = logging.getLogger('MyApp')
logger.setLevel(logging.DEBUG)
file_log_handler = RotatingFileHandler('logfile.log', maxBytes=1024, backupCount=5)
logger.addHandler(file_log_handler)
stderr_log_handler = logging.StreamHandler()
logger.addHandler(stderr_log_handler)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_log_handler.setFormatter(formatter)
stderr_log_handler.setFormatter(formatter)
return logger

Afaik, you'd have to specify this in basicConfig, not in your logger:
logging.basicConfig(filename=LOG_FILE,
level=logging.DEBUG)
before you do
logger = logging.getLogger('MyApp')
logger.setLevel(logging.DEBUG)

Python logging messages appears in stdout when script is launched

I'm setting a logging feature in my test script, but when I run it all messages are appearing on the console(stdout), the log file is written ok, how can I avoid this behaviour? I only want the messages are in the log file. Thanks.
import logging
import logging.handlers
LOG_FILE = "/var/log/mylog.log"
logger = logging.getLogger('mylog')
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes=10*1024*1024, backupCount=5)
file_handler.setFormatter(formatter)
file_handler.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
console_handler.setLevel(logging.DEBUG)
logger.addHandler(console_handler)
....

As #alecxe said, you should change your console_handler code to:
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
console_handler.setLevel(logging.ERROR)
logger.addHandler(console_handler)

You're setting it yourself here:
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
console_handler.setLevel(logging.DEBUG)
logger.addHandler(console_handler)
Check out the docs: https://docs.python.org/2.7/library/logging.handlers.html