when i receive input like url
Then want to logging in cloud run
I want to see my input with python
You can log the input URL in Python using the built-in logging module. You can use this example
import logging
def log_url(url):
logging.basicConfig(filename='url.log', level=logging.INFO)
logging.info('Received URL: %s', url)
if __name__ == '__main__':
url = 'https://www.example.com'
log_url(url)
This code will log the input URL in a file named url.log in the same directory as the script. The logging level is set to logging.INFO, so only informational messages will be logged. You can adjust the logging level as needed to control the verbosity of the log output.
To log the input URL in the cloud, you will need to configure the logging backend to write to a cloud-based log service. The specific service you use will depend on the cloud provider you are using. For example, if you are using Google Cloud, you could use Stackdriver Logging to store and manage your logs.
Related
I am trying to send logs (with jsonPayload) to GCP using StructuredLogHandler with below python code.
rootlogger = logging.getLogger()
client = google.cloud.logging.Client(credentials=xxx, project=xxx)
h = StructuredLogHandler()
rootlogger.addHandler(h)
logger = logging.getLogger('test')
logger.warning('warning')
I see that logs are being printed on console (in json format) but the logs are not sent to GCP Log Explorer. Can someone help?
Since v3 of Python Cloud Logging Library it's now easier than ever as it integrates with the Python standard logging library with client.setup_logging:
import logging
import google.cloud.logging
# Instantiate a client
client = google.cloud.logging.Client()
# Retrieves a Cloud Logging handler based on the environment
# you're running in and integrates the handler with the
# Python logging module. By default this captures all logs
# at INFO level and higher
client.setup_logging()
Name your logger as usual. E.g.
logger = logging.getLogger('test')
Then if you want to send structured log messages to Cloud Logging then you can use one of two methods:
Use the json_fields extra argument:
data_dict = {"hello": "world"}
logging.info("message field", extra={"json_fields": data_dict})
Use a JSON-parseable string (requires importing the json module):
import json
data_dict = {"hello": "world"}
logging.info(json.dumps(data_dict))
This will see your log messages sent to Google Cloud and the JSON payload available under the jsonPayload field of the expanded log entry:
I was wondering whether cloud functions has restrictions to export logging from python out of Google Cloud Platform for long messages (over 1500 characters).
The below are multiple examples from Papertrail on how to send log messages from Python, but the one I tried is below which worked sending messages to Papertrail on Google Compute Engine, but did not work on Google Cloud Functions. Google Cloud Function instead logs them in Google Cloud Operation Logging Service.
import logging
import socket
from logging.handlers import SysLogHandler
class ContextFilter(logging.Filter):
hostname = socket.gethostname()
def filter(self, record):
record.hostname = ContextFilter.hostname
return True
syslog = SysLogHandler(address=('logsN.papertrailapp.com', XXXXX))
syslog.addFilter(ContextFilter())
format = '%(asctime)s %(hostname)s YOUR_APP: %(message)s'
formatter = logging.Formatter(format, datefmt='%b %d %H:%M:%S')
syslog.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(syslog)
logger.setLevel(logging.INFO)
logger.info("This is a message")
In particular, it seems that for the above code, the address I placed on the handler does not work in Google Cloud Function if the message I pass is too long.
syslog = SysLogHandler(address=('logsN.papertrailapp.com', XXXXX))
Is this normal or my cloud function has some issues?
Is there a way I can pass an address in my python logging within Google Cloud Functions that can be compatible for long messages?
I am using Python and I was wondering if there is any package/simple way for logging directly to Azure?
I found a package (azure-storage-logging) that would be really nice, however it is not being maintained and not compatible with the new Azure API.
Any help is welcome.
You should use Application Insights which will send the logs to Azure Monitor (previously Log Analytics).
https://learn.microsoft.com/en-us/azure/azure-monitor/app/opencensus-python
I had the same requirement to log error and debug messages for small application and store the logs to Azure data lake. We did not want to use Azure Insight as our was not a web application and we just needed logs to debug the code.
To solve this I created temp.log file.
logging.basicConfig(filename='temp.log', format='%(asctime)s %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
datefmt='%Y-%m-%d:%H:%M:%S')
At the end of program I uploaded the temp.log to azure using,
DataLakeFileClient.append_data
local_file = open("temp.log",'r')
file_contents = local_file.read()
file_client.append_data(data=file_contents, offset=0, length=len(file_contents))
file_client.flush_data(len(file_contents))
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-python
You can just create your own handler. I show you how to log it to an azure table. Storing in blob can be similar. The biggest benefit is that you emit the log as you log it instead of sending logs at the end of a process.
Create a table in azure table storage and then run the following code.
from logging import Logger, getLogger
from azure.core.credentials import AzureSasCredential
from azure.data.tables import TableClient, TableEntity
class _AzureTableHandler(logging.Handler):
def __init__(self, *args, **kwargs):
super(_AzureTableHandler, self).__init__(*args, **kwargs)
credentials: AzureSasCredential = AzureSasCredential(signature=<sas-token>)
self._table_client: TableClient = TableClient(endpoint=<storage-account-url>, table_name=<table-name>, credential=credentials)
def emit(self, record):
level = record.levelname
message = record.getMessage()
self._table_client.create_entity(TableEntity({'Severity': level,
'Message': message,
'PartitionKey': f'{datetime.now().date()}',
'RowKey': f'{datetime.now().microsecond}'}))
if __name__ == "__main__":
logger: Logger = getLogger(__name__)
logger.addHandler(_AzureTableHandler())
logger.warning('testing azure logging')
In this approach, you also have the benefit of creating custom columns for your table. For example, you can have separate columns for the project name which is logging, or the username of the dev who is running the script.
logger.addHandler(_AzureTableHandler(Username="Hesam", Project="Deployment-support-tools-client-portal"))
Make sure to add your custom column names to the table_entity dictionary. Or you can put the project name as partition key.
In the google example, it gives the following:
logger.log_struct({
'message': 'My second entry',
'weather': 'partly cloudy',
})
How would I do the equivalent in python's logger. For example:
import logging
log.info(
msg='My second entry',
extra = {'weather': "partly cloudy"}
)
When I view this in stackdriver, the extra fields aren't getting parsed properly:
2018-11-12 15:41:12.366 PST
My second entry
Expand all | Collapse all
{
insertId: "1de1tqqft3x3ri"
jsonPayload: {
message: "My second entry"
python_logger: "Xhdoo8x"
}
logName: "projects/Xhdoo8x/logs/python"
receiveTimestamp: "2018-11-12T23:41:12.366883466Z"
resource: {…}
severity: "INFO"
timestamp: "2018-11-12T23:41:12.366883466Z"
}
How would I do that?
The closest I'm able to do now is:
log.handlers[-1].client.logger('').log_struct("...")
But this still requires a second call...
Current solution:
Update 1 - user Seth Nickell improved my proposed solution, so I update this answer as his method is superior. The following is based on his response on GitHub:
https://github.com/snickell/google_structlog
pip install google-structlog
Used like so:
import google_structlog
google_structlog.setup(log_name="here-is-mylilapp")
# Now you can use structlog to get searchable json details in stackdriver...
import structlog
logger = structlog.get_logger()
logger.error("Uhoh, something bad did", moreinfo="it was bad", years_back_luck=5)
# Of course, you can still use plain ol' logging stdlib to get { "message": ... } objects
import logging
logger = logging.getLogger("yoyo")
logger.error("Regular logging calls will work happily too")
# Now you can search stackdriver with the query:
# logName: 'here-is-mylilapp'
Original answer:
Based on an answer from this GitHub thread, I use the following bodge to log custom objects as info payload. It derives more from the original _Worker.enqueue and supports passing custom fields.
from google.cloud.logging import _helpers
from google.cloud.logging.handlers.transports.background_thread import _Worker
def my_enqueue(self, record, message, resource=None, labels=None, trace=None, span_id=None):
queue_entry = {
"info": {"message": message, "python_logger": record.name},
"severity": _helpers._normalize_severity(record.levelno),
"resource": resource,
"labels": labels,
"trace": trace,
"span_id": span_id,
"timestamp": datetime.datetime.utcfromtimestamp(record.created),
}
if 'custom_fields' in record:
entry['info']['custom_fields'] = record.custom_fields
self._queue.put_nowait(queue_entry)
_Worker.enqueue = my_enqueue
Then
import logging
from google.cloud import logging as google_logging
logger = logging.getLogger('my_log_client')
logger.addHandler(CloudLoggingHandler(google_logging.Client(), 'my_log_client'))
logger.info('hello', extra={'custom_fields':{'foo': 1, 'bar':{'tzar':3}}})
Resulting in:
Which then makes it much easier to filter according to these custom_fields.
Let's admit this is not good programming, though until this functionality is officially supported there doesn't seem to be much else that can be done.
This is not currently possible, see this open feature request on google-cloud-python for more details.
Official docs: Setting Up Cloud Logging for Python
You can write logs to Logging from Python applications
by using the Python logging handler included with the Logging client library, or
by using Cloud Logging API Cloud client library for Python directly.
I did not get the Python logging module to export jsonPayload, but the cloud logging library for Python works:
google-cloud-logging >= v.3.0.0 can do it
No need for Python logging workaround anymore, the only thing you need is to install Python >= 3.6 and
pip install google-cloud-logging
Then, you can use Python logging with
import os
# have the environment variable ready:
# GOOGLE_APPLICATION_CREDENTIALS
# Then:
GOOGLE_APPLICATION_CREDENTIALS = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
client = google.cloud.logging.Client.from_service_account_json(
GOOGLE_APPLICATION_CREDENTIALS
)
log_name = "test"
gcloud_logger = client.logger(log_name)
# jsonPayloads
gcloud_logger.log_struct(entry)
# messages
gcloud_logger.log_text('hello')
# generic (can find out whether it is jsonPayload or message)!
gcloud_logger.log(entry or 'hello')
You run these commands in a Python file outside of GCP and reach GCP with more or less a one-liner, you only need the credentials.
You can use the gcloud logger even for printing and logging in one go, taken from Writing structured logs, untested.
Python logging to log jsonPayload into GCP logs (TL/DR)
I could not get this to run!
You can also use the built-in logging module of Python with the workaround mentioned in the other answer, but I did not get it to run.
It will not work if you pass a dictionary or its json.dumps() directly as a parameter, since then, you get a string output of the whole dictionary which you cannot read as a json tree.
But it also did not work for me when I used the logger.info() to log the jsonPayload / json.dumps in an example parameter called extras.
import json
import os
#...
# https://googleapis.dev/python/logging/latest/stdlib-usage.html
GOOGLE_APPLICATION_CREDENTIALS = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
client = google.cloud.logging.Client.from_service_account_json(
GOOGLE_APPLICATION_CREDENTIALS
)
log_name = "test"
handler = CloudLoggingHandler(client, name=log_name)
setup_logging(handler)
logger = logging.getLogger()
logger.setLevel(logging.INFO) # Set default level.
#...
# Complete a structured log entry.
entry = dict(
severity="NOTICE",
message="This is the default display field.",
# Log viewer accesses 'component' as jsonPayload.component'.
component="arbitrary-property"
)
# Python logging to log jsonPayload into GCP logs
logger.info('hello', extras=json.dumps(entry))
I also tried the google-structlog solution of the other answer, that only threw the error:
google_structlog.setup(log_name=log_name) TypeError: setup() got an unexpected keyword argument 'log_name'
I used Python v3.10.2 and
google-auth==1.35.0
google-cloud-core==1.7.2
google-cloud-logging==1.15.0
googleapis-common-protos==1.54.0
Research steps gcloud logging (TL/DR)
Following the fixed issue (see the merge and close in the end) on github at googleapis / python-logging: Logging: support sending structured logs to stackdriver via stdlib 'logging'. #13
you find feat!: support json logs #316 :
This PR adds full support for JSON logs, along with standard text
logs. Now, users can call logging.error({'a':'b'}), and they will get
a JsonPayload in Cloud Logging, Or call logging.error('test') to
receive a TextPayload
As part of this change, I added a generic logger.log() function, which
serves as a generic entry-point instead of logger.log_text or
logger.log_struct. It will infer which log function is meant based on
the input type
Previously, the library would attach the python logger name as part of
the JSON payload for each log. Now, that information will be attached
as a label instead, giving users full control of the log payload
fields
Fixes #186, #263, #13
With the main new version listing the new feature:
chore(main): release 3.0.0 #473
I am using Stackdriver Logging for Python and a Python logger at the same time. I am using the function google.cloud.logging.logger.log_struct (https://gcloud-python.readthedocs.io/en/latest/logging/logger.html) to log a JSON to the Stackdriver.
I am able to view the logs as expected in the log viewer with the selected resource Global when I am running my script using a Google Compute Engine VM instance. The struct I am logging is recorded properly in jsonPayload.
However, when the logging is coming from a Google Kubernetes Engine, the logger view does not show the structs I passed, but rather what is printed on stdout. How do I make sure I observe the same behaviour from the Google Compute Engine VM instance and a Google Kubernetes Engine?
This is a snippet showing how I am doing the logging:
import google.cloud.logging
import logging
logging_client = google.cloud.logging.Client()
# connects the logger to the root logging handler
cloud_logger = logging_client.logger('email_adaptor')
formatter = logging.Formatter(
'%(asctime)s - %(levelname)s - %(message)s - %(lineno)d - %(filename)s')
# get the logger a name
logger = logging.getLogger('email_adaptor')
# set a level for the logger
logger.setLevel(logging.DEBUG)
# make the logger write on stdout
stdout_alarm_log = logging.StreamHandler(sys.stdout)
stdout_alarm_log.setFormatter(formatter)
logger.addHandler(stdout_alarm_log)
struct = {'message':'Processed Alarm', 'valid': True}
cloud_logger.log_struct(struct, severity='INFO')
logger.info(str(struct))
This is an example of what I get on the STDOUT on both the VM instance and the Kubernetes Engine:
2018-10-26 12:30:20,008 - INFO - Processed Alarm {'valid': True} - 79 - log.py
INFO:email_adaptor:Processed Alarm {'valid': True}
This is what I see under the resource Global in the Google Log Viewer (the logs are ones from my VM instance, and do not correspond to the example I gave in the snippet code):
This is what I see under the resource Google Kubernetes Engine: The struct do not show, instead I see what is printed on STDOUT:
The Stackdriver library calls in the code write against “global” and the structured log entries from your GKE containers will show under “Global” resource.
The “GKE Container” resource will show logging written to stdout and stderr, which are ingested by default.
To write structured logs to stdout/stderr and access them in Stackdriver, the only structured format that the logging agent will accept is JSON. You must configure your logger to output JSON for it to be picked up as a structured log entry. More information at https://cloud.google.com/logging/docs/agent/configuration#process-payload .