Monitor Only

How to deploy pigsty monitoring system without provisioning solution?

剧本概览

部署监控系统:可以在现有集群中创建新的用户或修改现有用户pgsql-monitor.yml

日常管理

# 在 pg-test 集群中部署监控
./pgsql-monitor.yml -l pg-test

剧本说明

#!/usr/bin/env ansible-playbook
---
#==============================================================#
# File      :   pgsql-monitor.yml
# Ctime     :   2021-02-23
# Mtime     :   2021-02-27
# Desc      :   deploy monitor components only
# Path      :   pgsql-monitor.yml
# Copyright (C) 2018-2021 Ruohang Feng
#==============================================================#

# this is pgsql monitor setup playbook for MONITOR ONLY mode

# MONITOR-ONLY (monly) mode is a special deployment mode for
# integration with exterior provisioning solution or existing
# postgres clusters.
# with limited functionalities

# For monly deployment, The infra part is still the same.
# You MUST use static services discovery for prometheus
# You CAN NOT use services_registry


#------------------------------------------------------------------------------
# Deploy monitor on selected targets
#------------------------------------------------------------------------------
- name: Monitor Only Deployment
  become: yes
  hosts: all
  gather_facts: no
  tags: monitor
  roles:
    - role: monitor                         # init monitor system
  vars:
    #------------------------------------------------------------------------------
    # RECOMMEND CHANGES
    #------------------------------------------------------------------------------
    # You'd better change those options in your main config file
    # prometheus_sd_method: static          # MUST use static sd for monitor only mode
    service_registry: none                  # MUST NOT register services
    exporter_install: binary                # none|yum|binary, none by default

    # exporter_install controls how node_exporter & pg_exporter are installed
    #    none   : I've already installed manually
    #    yum    : Use yum install, `exporter_repo_url` will be added if specified
    #    binary : Copy binary to /usr/bin. You must have binary in your `files` dir

    #------------------------------------------------------------------------------
    # MONITOR PROVISION
    #------------------------------------------------------------------------------
    # - install - #
    # exporter_install: none                        # none|yum|binary, none by default
    # exporter_repo_url: ''                         # if set, repo will be added to /etc/yum.repos.d/ before yum installation

    # - collect - #
    # exporter_metrics_path: /metrics               # default metric path for pg related exporter

    # - node exporter - #
    # node_exporter_enabled: true                   # setup node_exporter on instance
    # node_exporter_port: 9100                      # default port for node exporter
    # node_exporter_options: '--no-collector.softnet --collector.systemd --collector.ntp --collector.tcpstat --collector.processes'

    # - pg exporter - #
    # pg_exporter_config: pg_exporter-demo.yaml     # default config files for pg_exporter
    # pg_exporter_enabled: true                     # setup pg_exporter on instance
    # pg_exporter_port: 9630                        # default port for pg exporter
    # pg_exporter_url: ''                           # optional, if not set, generate from reference parameters

    # - pgbouncer exporter - #
    # pgbouncer exporter require pgbouncer to work, so it is disabled by default in monitor-only mode
    # pgbouncer_exporter_enabled: false             # setup pgbouncer_exporter on instance (if you don't have pgbouncer, disable it)
    # pgbouncer_exporter_port: 9631                 # default port for pgbouncer exporter
    # pgbouncer_exporter_url: ''                    # optional, if not set, generate from reference parameters

    # - postgres variables reference - #
    # pg_dbsu: postgres
    # pg_port: 5432                                 # postgres port (5432 by default)
    # pgbouncer_port: 6432                          # pgbouncer port (6432 by default)
    # pg_localhost: /var/run/postgresql             # localhost unix socket dir for connection
    # pg_default_database: postgres                 # default database will be used as primary monitor target
    # pg_monitor_username: dbuser_monitor           # system monitor username, for postgres and pgbouncer
    # pg_monitor_password: DBUser.Monitor           # system monitor user's password
    # service_registry: consul                      # none | consul | etcd | both



#------------------------------------------------------------------------------
# update static inventory in meta node and reload
#------------------------------------------------------------------------------
- name: Update prometheus static sd files
  become: yes
  hosts: meta
  tags: prometheus
  gather_facts: no
  vars:
    #------------------------------------------------------------------------------
    # RECOMMEND CHANGES
    #------------------------------------------------------------------------------
    prometheus_sd_method: static                  # service discovery method: static|consul|etcd

  tasks:
    - include_tasks: roles/prometheus/tasks/targets.yml
    - include_tasks: roles/prometheus/tasks/reload.yml

...

使用样例

./pgsql-monitor.yml -l pg-test

执行结果

$ ./pgsql-monitor.yml -l pg-test -e pg_user=test
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

PLAY [Create user in cluster] *****************************************************************************************************************************************************

TASK [Check parameter pg_user] ****************************************************************************************************************************************************
ok: [10.10.10.11] => {
    "changed": false,
    "msg": "All assertions passed"
}
ok: [10.10.10.12] => {
    "changed": false,
    "msg": "All assertions passed"
}
ok: [10.10.10.13] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Fetch user definition] ******************************************************************************************************************************************************
ok: [10.10.10.11]
ok: [10.10.10.12]
ok: [10.10.10.13]

TASK [debug] **********************************************************************************************************************************************************************
ok: [10.10.10.11] => {
    "msg": {
        "comment": "default test user for production usage",
        "name": "test",
        "password": "test",
        "pgbouncer": true,
        "roles": [
            "dbrole_readwrite"
        ]
    }
}
ok: [10.10.10.12] => {
    "msg": {
        "comment": "default test user for production usage",
        "name": "test",
        "password": "test",
        "pgbouncer": true,
        "roles": [
            "dbrole_readwrite"
        ]
    }
}
ok: [10.10.10.13] => {
    "msg": {
        "comment": "default test user for production usage",
        "name": "test",
        "password": "test",
        "pgbouncer": true,
        "roles": [
            "dbrole_readwrite"
        ]
    }
}

TASK [Check user definition] ******************************************************************************************************************************************************
ok: [10.10.10.11] => {
    "changed": false,
    "msg": "All assertions passed"
}
ok: [10.10.10.12] => {
    "changed": false,
    "msg": "All assertions passed"
}
ok: [10.10.10.13] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [include_tasks] **************************************************************************************************************************************************************
included: /Volumes/Data/pigsty/roles/postgres/tasks/monitor.yml for 10.10.10.11, 10.10.10.12, 10.10.10.13

TASK [Render user test creation sql] **********************************************************************************************************************************************
skipping: [10.10.10.12]
skipping: [10.10.10.13]
changed: [10.10.10.11]

TASK [Execute user test creation sql on primary] **********************************************************************************************************************************
skipping: [10.10.10.12]
skipping: [10.10.10.13]
changed: [10.10.10.11]

TASK [Add user to pgbouncer] ******************************************************************************************************************************************************
changed: [10.10.10.11]
changed: [10.10.10.13]
changed: [10.10.10.12]

TASK [Reload pgbouncer to add user] ***********************************************************************************************************************************************
changed: [10.10.10.11]
changed: [10.10.10.12]
changed: [10.10.10.13]

PLAY RECAP ************************************************************************************************************************************************************************
10.10.10.11                : ok=9    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
10.10.10.12                : ok=7    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
10.10.10.13                : ok=7    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

任务详情

默认任务如下:

playbook: ./pgsql-monitor.yml

  play #1 (all): Monitor Only Deployment	TAGS: [monitor]
    tasks:
      monitor : Install exporter yum repo	TAGS: [exporter_install, exporter_yum_install, monitor]
      monitor : Install node_exporter and pg_exporter	TAGS: [exporter_install, exporter_yum_install, monitor]
      monitor : Copy node_exporter binary	TAGS: [exporter_binary_install, exporter_install, monitor]
      monitor : Copy pg_exporter binary	TAGS: [exporter_binary_install, exporter_install, monitor]
      monitor : Create /etc/pg_exporter conf dir	TAGS: [monitor, pg_exporter]
      monitor : Copy default pg_exporter.yaml	TAGS: [monitor, pg_exporter]
      monitor : Config /etc/default/pg_exporter	TAGS: [monitor, pg_exporter]
      monitor : Config pg_exporter service unit	TAGS: [monitor, pg_exporter]
      monitor : Launch pg_exporter systemd service	TAGS: [monitor, pg_exporter]
      monitor : Wait for pg_exporter service online	TAGS: [monitor, pg_exporter]
      monitor : Register pg-exporter consul service	TAGS: [monitor, pg_exporter_register]
      monitor : Reload pg-exporter consul service	TAGS: [monitor, pg_exporter_register]
      monitor : Config pgbouncer_exporter opts	TAGS: [monitor, pgbouncer_exporter]
      monitor : Config pgbouncer_exporter service	TAGS: [monitor, pgbouncer_exporter]
      monitor : Launch pgbouncer_exporter service	TAGS: [monitor, pgbouncer_exporter]
      monitor : Wait for pgbouncer_exporter online	TAGS: [monitor, pgbouncer_exporter]
      monitor : Register pgb-exporter consul service	TAGS: [monitor, node_exporter_register]
      monitor : Reload pgb-exporter consul service	TAGS: [monitor, node_exporter_register]
      monitor : Copy node_exporter systemd service	TAGS: [monitor, node_exporter]
      monitor : Config default node_exporter options	TAGS: [monitor, node_exporter]
      monitor : Launch node_exporter service unit	TAGS: [monitor, node_exporter]
      monitor : Wait for node_exporter online	TAGS: [monitor, node_exporter]
      monitor : Register node-exporter service to consul	TAGS: [monitor, node_exporter_register]
      monitor : Reload node-exporter consul service	TAGS: [monitor, node_exporter_register]

  play #2 (meta): Update prometheus static sd files	TAGS: [prometheus]
    tasks:
      include_tasks	TAGS: [prometheus]
      include_tasks	TAGS: [prometheus]

Last modified 2021-03-28: update en docs (f994b54)