Analyze Pacemaker Events In Cloud Logging

Customer’s deploying SAP on Google Cloud often leverage Pacemaker for high availability to support their most critical systems. Let’s take a look at how you can use Cloud Logging to easily conduct root cause analysis of Pacemaker clusters.

When there are multiple Pacemaker clusters running in Google Cloud platform, a central logging place can help to store the Pacemaker logs and offer an easy way to analyze Pacemaker events such as fencing or resource failover.

Partner with aster.cloud
for your next big idea.
Let us know here.

From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.

CYBERPOGO.COM :: For the Arts, Sciences, and Technology.

DADAHACKS.COM :: Parenting For The Rest Of Us.

ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.

TAKUMAKU.COM :: For The Hearth And Home.

ASTER.CLOUD :: From The Cloud And Beyond.

LIWAIWAI.COM :: Intelligence, Inside and Outside.

GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.

FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.

ASTERCASTER.COM :: Supra Astra. Beyond The Stars.

BARTDAY.COM :: Prosperity For Everyone.

The Ops Agent is the primary agent for collecting telemetry from your Compute Engine instances. Combining logging and metrics into a single agent, the Ops Agent uses Fluent Bit for logs, which supports high-throughput logging, and the OpenTelemetry Collector for metrics.

Install the Agent

Follow this guide to install the Ops Agent on a single VM via command line or using the Google Cloud Console. To install the agent to multiple VMs, use gcloud or automation tools. Ensure your VM doesn’t have the legacy Cloud Logging Agent or Cloud Monitoring Agent installed on it.

Configure the Agent

By default, the Ops Agent’s build-in configuration collects file-based syslog log. Pacemaker resource agents such as SAPHana write logs to system log /var/log/messages in SAP certified OS SUSE and RedHat.

Add below configuration elements to the user configuration file /etc/google-cloud-ops-agent/config.yaml to stream Pacemaker logs to Cloud Logging. The path defined below covers all default log files that Pacemaker writes logs to in SUSE and RedHat.

Pacemaker-log is the receiver ID which defines the logName “projects/[PROJECT_ID]/logs/pacemaker-log” of the log entries streamed to Cloud Logging.

Note: If there are existing configurations defined in the logging section, then only add the bold parts.

logging:
  receivers:
    pacemaker-log:
      type: files
      include_paths: 
      - /var/log/pacemaker.log
      - /var/log/cluster/corosync.log
      - /var/log/pacemaker/pacemaker.log
  service:
    pipelines:
      pacemaker-pipeline:
        receivers: [pacemaker-log]

Restart the agent

Restart the agent to apply the user-specified configuration

sudo service google-cloud-ops-agent restart

Validate the agent

Validate in logging module log

/var/log/google-cloud-ops-agent/subagents/logging-module.log to ensure the Pacemaker logs are activated, you should see similar entries as below listing Pacemaker logs. Follow the troubleshooting guide for any issues.

[2021/08/19 04:34:26] [ info] [sp] stream processor started
[2021/08/19 04:34:26] [ info] [input:tail:tail.3] inotify_fs_add(): inode=51595418 watch_fd=1 name=/var/log/cluster/corosync.log
[2021/08/19 04:34:26] [ info] [input:tail:tail.0] inotify_fs_add(): inode=16814304 watch_fd=1 name=/var/log/messages
[2021/08/19 04:34:26] [ info] [input:tail:tail.2] inotify_fs_add(): inode=51506 watch_fd=1 name=/var/log/google-cloud-ops-agent/subagents/logging-module.log
[2021/08/19 04:34:26] [ info] [input:tail:tail.3] inotify_fs_add(): inode=51595416 watch_fd=2 name=/var/log/pacemaker/pacemaker.log

Validate cloud logging

Use below log filter (replace PROJECT_ID) in Cloud Logging Logs Explorer to validate the Pacemaker logs are being streamed there.

logName="projects/[PROJECT_ID]/logs/pacemaker-log"

Now you can use Cloud Logging Logs Explorer to analyze Pacemaker events. Below sample log filter can help to filter the critical Pacemaker actions and events. Replace the INSTANCE_ID_NODE1/2 with the actual instance IDs of the two cluster nodes. The filter captures

Actions of the cluster nodes, cluster resources such as start, stop or promote
Failed resource operations, such as start, stop or promote
Fencing actions, reasons (loss of cluster nodes, resource failure etc.) and results
Corosync communication errors
Cluster membership changes, member joins or leaves

resource.type="gce_instance"
(resource.labels.instance_id="[INSTANCE_ID_NODE1]" OR resource.labels.instance_id="[INSTANCE_ID_NODE2]")
(protoPayload.methodName="v1.compute.instances.reset"
OR jsonPayload.message: ("LogNodeActions"
OR ("remote_op_done" "Operation")
OR ("notice" "LogAction")
OR ("TOTEM" "failed" OR "membership" OR "Restransmit")
OR ("Result of" "operation" NOT "ok" NOT "Cancelled" NOT "probe")
OR ("SAPHana(" "WARNING" OR "ERROR")
OR ("SAPInstance(" OR "stonith-ng:" OR "gcp:stonith:" OR "gcp:alias:" OR "gcp-vpc-move-vip:" OR "fence_gce:" "ERROR" OR "Failed")
)
)

Now Pacemaker logs from all your clusters are stored in Cloud Logging, you can analyze Pacemaker events happening to any of your clusters in one central place. If further support is needed from Google Cloud Customer Care Team, efforts and time are saved to collect and transfer logs to the support agent.

To monitor Pacemaker clusters and receive alerts, read this blog What’s happening in your SAP systems? Find out with Pacemaker Alerts.

By Cherry Legler, Senior Technical Solution Engineer
Source Google Cloud