First let me explain the setup/system:
I have a virtual Ubuntu 18.04 machine running via GCP Compute Engine. I have followed the GCP documentation and have installed the structured logging agent via apt-get
.
On this machine I have a golang webservice, running inside a Docker container, which uses Logrus to put out logs that look like this:
{"message":"some interesting thing occured","severity":"INFO","timestamp":{"seconds":1599668988,"nanos":567943787}}
In order to get these logs from the machine / container to GCP Logs I have created a new fluentd configuration file /etc/google-fluentd/config.d/docker.conf
, as described in fluentd documentation:
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<filter docker.**>
@type parser
format json
key_name log
reserve_data true
</filter>
As well as the Docker daemon to send logs for this container to fluentd, as described in Docker documentation: docker run --log-driver=fluentd ...
The limitation I am facing is that the logs do arrive in GCP Logs Viewer, but the fields of my service are not correctly parsed, and therefore I am not able to sort by severity and so on. The logs that arrive in GCP look like this:
{
"insertId": "xxxxxxxxxxxxxxxx",
"jsonPayload": {
"stream": "stderr",
"log": "{\"message\":\"some interesting thing occured\",\"severity\":\"INFO\",\"timestamp\":{\"seconds\":1599668988,\"nanos\":567943787}}\n"
},
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "22222222222222222",
"project_id": "xxxxxxx-333333",
"zone": "us-east1-b"
}
},
"timestamp": "2020-09-09T16:29:48.568042072Z",
"labels": {
"compute.googleapis.com/resource_name": "xxxxxxxx"
},
"logName": "some/folder/logs/docker.var.lib.docker.containers.7ec67d861ea6d2dbf20db456dad4e7efc43b91a498580dda325f76170b502509.7ec67d861ea6d2dbf20db456dad4e7efc43b91a498580dda325f76170b502509-json.log",
"receiveTimestamp": "2020-09-09T16:33:29.159377657Z"
}
What additional steps do I need to take so that the logs are understood by GCP Logs Viewer? I though that the filter/parser configuration for fluentd is enough to achieve this, in particular the section about.
format json
key_name log