Home:ALL Converter>Make fields of JSON logs from Docker container available in GCP Logs via google-fluentd

Make fields of JSON logs from Docker container available in GCP Logs via google-fluentd

Ask Time:2020-09-10T03:29:40         Author:datosh

Json Formatter

First let me explain the setup/system:

I have a virtual Ubuntu 18.04 machine running via GCP Compute Engine. I have followed the GCP documentation and have installed the structured logging agent via apt-get.

On this machine I have a golang webservice, running inside a Docker container, which uses Logrus to put out logs that look like this:

{"message":"some interesting thing occured","severity":"INFO","timestamp":{"seconds":1599668988,"nanos":567943787}}

In order to get these logs from the machine / container to GCP Logs I have created a new fluentd configuration file /etc/google-fluentd/config.d/docker.conf, as described in fluentd documentation:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter docker.**>
  @type parser
  format json
  key_name log
  reserve_data true
</filter>

As well as the Docker daemon to send logs for this container to fluentd, as described in Docker documentation: docker run --log-driver=fluentd ...

The limitation I am facing is that the logs do arrive in GCP Logs Viewer, but the fields of my service are not correctly parsed, and therefore I am not able to sort by severity and so on. The logs that arrive in GCP look like this:

{
  "insertId": "xxxxxxxxxxxxxxxx",
  "jsonPayload": {
    "stream": "stderr",
    "log": "{\"message\":\"some interesting thing occured\",\"severity\":\"INFO\",\"timestamp\":{\"seconds\":1599668988,\"nanos\":567943787}}\n"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "instance_id": "22222222222222222",
      "project_id": "xxxxxxx-333333",
      "zone": "us-east1-b"
    }
  },
  "timestamp": "2020-09-09T16:29:48.568042072Z",
  "labels": {
    "compute.googleapis.com/resource_name": "xxxxxxxx"
  },
  "logName": "some/folder/logs/docker.var.lib.docker.containers.7ec67d861ea6d2dbf20db456dad4e7efc43b91a498580dda325f76170b502509.7ec67d861ea6d2dbf20db456dad4e7efc43b91a498580dda325f76170b502509-json.log",
  "receiveTimestamp": "2020-09-09T16:33:29.159377657Z"
}

What additional steps do I need to take so that the logs are understood by GCP Logs Viewer? I though that the filter/parser configuration for fluentd is enough to achieve this, in particular the section about.

  format json
  key_name log

Author:datosh,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/63818358/make-fields-of-json-logs-from-docker-container-available-in-gcp-logs-via-google
yy