Photo by Barrett Ward on Unsplash

Containers Logging Guide

Get Started with Containers Logging with Journald, Loki and Grafana

From a DevOps standpoint, logging is one of the most important things to get right in applications. Logging helps DevOps teams troubleshoot issues faster through the logging tools like ELK, Grafana, and Loki. this guide will explain how we send container logs to Journald and then push it to Loki with promtail and create a simple dashboard with Grafana.

I will use podman in this guide, the same commands and the same options are available on docker.

Podman Logging Driver with journald

There are several different log drivers you can use with the containers like json-file and journald. to get logging and info from the running containers.

let’s take Nginx container with access logs as example, on the podman run command i will add options --log-driver and --log-opt to specify the driver to be journald and the log file path to be access log.

$ podman run -d --name nginx -p 8080:80 --log-driver journald --log-opt path=/var/log/nginx/access.log nginx

you can pass the options to podman-compose like the below code

logging:
   driver: journald
   options:
     LOG_PATH: /var/log/nginx/access.log

reading container logs

now you can read the logs using the podman logs command or using journalctl command

$ podman logs --tail 10 -f nginx
$ journalctl CONTAINER_NAME=nginx

Collect logs with Promtail

edit your promtail config and add this lines under scrape_configs:to collect containers logs

scrape_configs:
- job_name: journal
  journal:
    max_age: 12h
    json: false
    labels:
      job: journal
  relabel_configs:
    - source_labels: [__journal_container_name]
      target_label: container

note: this config will collect logs from all containers that use journald log driver

Grafana and Loki

after the last step, you can explore the logs on Grafana and Loki