基础设施初始化
如何使用剧本初始化基础设施
剧本概览
Pigsty需要首先完成元节点的基础设施部署，基础设施的部署通过infra.yml完成。
infra.yml 将元节点（默认分组名为meta）作为部署目标，用户需要先行在配置文件的meta分组中指定元节点。
./infra.yml
注意事项
您必须完成元节点的初始化后，才能正常执行普通节点的初始化工作。
infra.yml 固定会作用于配置文件中 名为 meta 的分组
您可以将元节点当成普通节点复用，即在元节点上定义并创建PostgreSQL数据库。
我们建议按照默认配置在元节点上创建一个pg-meta元数据库，但不要施加过多负载。
完整执行一遍初始化流程可能花费2～8分钟，视您的机器配置而定。
选择性执行
您可以通过ansible的标签机制，可以选择执行剧本的一个子集。
例如，如果您只想执行本地源初始化的部分，则可以通过以下命令进行
./infra.yml --tags=repo
具体的标签请参考任务详情
一些常用的任务子集包括：
./infra.yml --tags=repo -e repo_rebuild=true # 强制重新创建本地源
./infra.yml --tags=prometheus_reload # 重新加载Prometheus配置
./infra.yml --tags=nginx_haproxy # 重新生成Nginx Haproxy索引页
./infra.yml --tags=prometheus_targets,prometheus_reload # 重新生成Prometheus静态监控对象文件并应用
剧本说明
infra.yml 主要完成以下工作
- 部署并启用本地源
- 完成元节点的初始化
- 完成元节点基础设施初始化
- CA基础设施
- DNS Nameserver
- Nginx
- Prometheus & Alertmanger
- Grafana
- 将Pigsty本体拷贝至元节点
- 在元节点上完成数据库初始化（可选，用户可以通过标准的数据库集群初始化流程复用元节点）
#!/usr/bin/env ansible-playbook
---
#==============================================================#
# File : infra.yml
# Ctime : 2020-04-13
# Mtime : 2020-07-23
# Desc : init infrastructure on meta nodes
# Path : infra.yml
# Copyright (C) 2018-2021 Ruohang Feng
#==============================================================#
#------------------------------------------------------------------------------
# init local yum repo (only run on meta nodes)
#------------------------------------------------------------------------------
- name: Init local repo
become: yes
hosts: meta
gather_facts: no
tags: repo
roles:
- repo
#------------------------------------------------------------------------------
# provision nodes
#------------------------------------------------------------------------------
- name: Provision Node
become: yes
hosts: meta
gather_facts: no
tags: node
roles:
- node
#------------------------------------------------------------------------------
# init meta service (only run on meta nodes)
#------------------------------------------------------------------------------
- name: Init meta service
become: yes
hosts: meta
gather_facts: no
tags: meta
roles:
- role: ca
tags: ca
- role: nameserver
tags: nameserver
- role: nginx
tags: nginx
- role: prometheus
tags: prometheus
- role: grafana
tags: grafana
#------------------------------------------------------------------------------
# init dcs on nodes
#------------------------------------------------------------------------------
- name: Init dcs
become: yes
hosts: meta
gather_facts: no
roles:
- role: consul
tags: dcs
#------------------------------------------------------------------------------
# copy scripts to meta node
#------------------------------------------------------------------------------
- name: Copy ansible scripts
become: yes
hosts: meta
gather_facts: no
ignore_errors: yes
tags: ansible
tasks:
- name: Copy ansible scritps
when: node_admin_setup is defined and node_admin_setup|bool and node_admin_username != ''
block:
# create copy of this repo
- name: Create ansible tarball
become: no
connection: local
run_once: true
command:
cmd: tar -cf files/meta.tgz roles templates ansible.cfg infra.yml pgsql.yml pgsql-rm.yml pigsty.yml Makefile
chdir: "{{ playbook_dir }}"
- name: Create ansible directory
file: path="/home/{{ node_admin_username }}/meta" state=directory owner={{ node_admin_username }}
- name: Copy ansible tarball
copy: src="meta.tgz" dest="/home/{{ node_admin_username }}/meta/meta.tgz" owner={{ node_admin_username }}
- name: Extract tarball
shell: |
cd /home/{{ node_admin_username }}/meta/
tar -xf meta.tgz
chown -R {{ node_admin_username }} /home/{{ node_admin_username }}
rm -rf meta.tgz
chmod a+x *.yml
#------------------------------------------------------------------------------
# meta node database (optional)
#------------------------------------------------------------------------------
# this play will create database clusters on meta nodes.
# it's good to reuse meta node as normal database nodes too
# but it's always better to leave it be.
#------------------------------------------------------------------------------
#- name: Pgsql Initialization
# become: yes
# hosts: meta
# gather_facts: no
# roles:
# - role: postgres # init postgres
# tags: [pgsql, postgres]
#
# - role: monitor # init monitor system
# tags: [pgsql, monitor]
#
# - role: haproxy # init haproxy
# tags: [proxy, haproxy]
#
# - role: vip # init haproxy
# tags: [proxy, vip]
#
...
任务详情
使用以下命令可以列出所有基础设施初始化会执行的任务，以及可以使用的标签：
./infra.yml --list-tasks
默认任务如下：
playbook: ./infra.yml
play #1 (meta): Init local repo TAGS: [repo]
tasks:
repo : Create local repo directory TAGS: [repo, repo_dir]
repo : Backup & remove existing repos TAGS: [repo, repo_upstream]
repo : Add required upstream repos TAGS: [repo, repo_upstream]
repo : Check repo pkgs cache exists TAGS: [repo, repo_prepare]
repo : Set fact whether repo_exists TAGS: [repo, repo_prepare]
repo : Move upstream repo to backup TAGS: [repo, repo_prepare]
repo : Add local file system repos TAGS: [repo, repo_prepare]
repo : Remake yum cache if not exists TAGS: [repo, repo_prepare]
repo : Install repo bootstrap packages TAGS: [repo, repo_boot]
repo : Render repo nginx server files TAGS: [repo, repo_nginx]
repo : Disable selinux for repo server TAGS: [repo, repo_nginx]
repo : Launch repo nginx server TAGS: [repo, repo_nginx]
repo : Waits repo server online TAGS: [repo, repo_nginx]
repo : Download web url packages TAGS: [repo, repo_download]
repo : Download repo packages TAGS: [repo, repo_download]
repo : Download repo pkg deps TAGS: [repo, repo_download]
repo : Create local repo index TAGS: [repo, repo_download]
repo : Copy bootstrap scripts TAGS: [repo, repo_download, repo_script]
repo : Mark repo cache as valid TAGS: [repo, repo_download]
play #2 (meta): Provision Node TAGS: [node]
tasks:
node : Update node hostname TAGS: [node, node_name]
node : Add new hostname to /etc/hosts TAGS: [node, node_name]
node : Write static dns records TAGS: [node, node_dns]
node : Get old nameservers TAGS: [node, node_resolv]
node : Truncate resolv file TAGS: [node, node_resolv]
node : Write resolv options TAGS: [node, node_resolv]
node : Add new nameservers TAGS: [node, node_resolv]
node : Append old nameservers TAGS: [node, node_resolv]
node : Node configure disable firewall TAGS: [node, node_firewall]
node : Node disable selinux by default TAGS: [node, node_firewall]
node : Backup existing repos TAGS: [node, node_repo]
node : Install upstream repo TAGS: [node, node_repo]
node : Install local repo TAGS: [node, node_repo]
node : Install node basic packages TAGS: [node, node_pkgs]
node : Install node extra packages TAGS: [node, node_pkgs]
node : Install meta specific packages TAGS: [node, node_pkgs]
node : Install node basic packages TAGS: [node, node_pkgs]
node : Install node extra packages TAGS: [node, node_pkgs]
node : Install meta specific packages TAGS: [node, node_pkgs]
node : Node configure disable numa TAGS: [node, node_feature]
node : Node configure disable swap TAGS: [node, node_feature]
node : Node configure unmount swap TAGS: [node, node_feature]
node : Node setup static network TAGS: [node, node_feature]
node : Node configure disable firewall TAGS: [node, node_feature]
node : Node configure disk prefetch TAGS: [node, node_feature]
node : Enable linux kernel modules TAGS: [node, node_kernel]
node : Enable kernel module on reboot TAGS: [node, node_kernel]
node : Get config parameter page count TAGS: [node, node_tuned]
node : Get config parameter page size TAGS: [node, node_tuned]
node : Tune shmmax and shmall via mem TAGS: [node, node_tuned]
node : Create tuned profiles TAGS: [node, node_tuned]
node : Render tuned profiles TAGS: [node, node_tuned]
node : Active tuned profile TAGS: [node, node_tuned]
node : Change additional sysctl params TAGS: [node, node_tuned]
node : Copy default user bash profile TAGS: [node, node_profile]
node : Setup node default pam ulimits TAGS: [node, node_ulimit]
node : Create os user group admin TAGS: [node, node_admin]
node : Create os user admin TAGS: [node, node_admin]
node : Grant admin group nopass sudo TAGS: [node, node_admin]
node : Add no host checking to ssh config TAGS: [node, node_admin]
node : Add admin ssh no host checking TAGS: [node, node_admin]
node : Fetch all admin public keys TAGS: [node, node_admin]
node : Exchange all admin ssh keys TAGS: [node, node_admin]
node : Install public keys TAGS: [node, node_admin]
node : Install ntp package TAGS: [node, ntp_install]
node : Install chrony package TAGS: [node, ntp_install]
node : Setup default node timezone TAGS: [node, ntp_config]
node : Copy the ntp.conf file TAGS: [node, ntp_config]
node : Copy the chrony.conf template TAGS: [node, ntp_config]
node : Launch ntpd service TAGS: [node, ntp_launch]
node : Launch chronyd service TAGS: [node, ntp_launch]
play #3 (meta): Init meta service TAGS: [meta]
tasks:
ca : Create local ca directory TAGS: [ca, ca_dir, meta]
ca : Copy ca cert from local files TAGS: [ca, ca_copy, meta]
ca : Check ca key cert exists TAGS: [ca, ca_create, meta]
ca : Create self-signed CA key-cert TAGS: [ca, ca_create, meta]
nameserver : Make sure dnsmasq package installed TAGS: [meta, nameserver]
nameserver : Copy dnsmasq /etc/dnsmasq.d/config TAGS: [meta, nameserver]
nameserver : Add dynamic dns records to meta TAGS: [meta, nameserver]
nameserver : Launch meta dnsmasq service TAGS: [meta, nameserver]
nameserver : Wait for meta dnsmasq online TAGS: [meta, nameserver]
nameserver : Register consul dnsmasq service TAGS: [meta, nameserver]
nameserver : Reload consul TAGS: [meta, nameserver]
nginx : Make sure nginx package installed TAGS: [meta, nginx, nginx_install]
nginx : Create local html directory TAGS: [meta, nginx, nginx_dir]
nginx : Update default nginx index page TAGS: [meta, nginx, nginx_dir]
nginx : Copy nginx default config TAGS: [meta, nginx, nginx_config]
nginx : Copy nginx upstream conf TAGS: [meta, nginx, nginx_config]
nginx : Fetch haproxy facts TAGS: [meta, nginx, nginx_config, nginx_haproxy]
nginx : Templating /etc/nginx/haproxy.conf TAGS: [meta, nginx, nginx_config, nginx_haproxy]
nginx : Templating haproxy.html TAGS: [meta, nginx, nginx_config, nginx_haproxy]
nginx : Launch nginx server TAGS: [meta, nginx, nginx_reload]
nginx : Restart meta nginx service TAGS: [meta, nginx, nginx_launch]
nginx : Wait for nginx service online TAGS: [meta, nginx, nginx_launch]
nginx : Make sure nginx exporter installed TAGS: [meta, nginx, nginx_exporter]
nginx : Config nginx_exporter options TAGS: [meta, nginx, nginx_exporter]
nginx : Restart nginx_exporter service TAGS: [meta, nginx, nginx_exporter]
nginx : Wait for nginx exporter online TAGS: [meta, nginx, nginx_exporter]
nginx : Register cosnul nginx service TAGS: [meta, nginx, nginx_register]
nginx : Register consul nginx-exporter service TAGS: [meta, nginx, nginx_register]
nginx : Reload consul TAGS: [meta, nginx, nginx_register]
prometheus : Install prometheus and alertmanager TAGS: [meta, prometheus, prometheus_install]
prometheus : Wipe out prometheus config dir TAGS: [meta, prometheus, prometheus_clean]
prometheus : Wipe out existing prometheus data TAGS: [meta, prometheus, prometheus_clean]
prometheus : Create postgres directory structure TAGS: [meta, prometheus, prometheus_config]
prometheus : Copy prometheus bin scripts TAGS: [meta, prometheus, prometheus_config]
prometheus : Copy prometheus rules scripts TAGS: [meta, prometheus, prometheus_config]
prometheus : Copy altermanager config TAGS: [meta, prometheus, prometheus_config]
prometheus : Render prometheus config TAGS: [meta, prometheus, prometheus_config]
prometheus : Config /etc/prometheus opts TAGS: [meta, prometheus, prometheus_config]
prometheus : Fetch prometheus static monitoring targets TAGS: [meta, prometheus, prometheus_config, prometheus_targets]
prometheus : Render prometheus static targets TAGS: [meta, prometheus, prometheus_config, prometheus_targets]
prometheus : Launch prometheus service TAGS: [meta, prometheus, prometheus_launch]
prometheus : Launch alertmanager service TAGS: [meta, prometheus, prometheus_launch]
prometheus : Wait for prometheus online TAGS: [meta, prometheus, prometheus_launch]
prometheus : Wait for alertmanager online TAGS: [meta, prometheus, prometheus_launch]
prometheus : Reload prometheus service TAGS: [meta, prometheus, prometheus_reload]
prometheus : Copy prometheus service definition TAGS: [meta, prometheus, prometheus_register]
prometheus : Copy alertmanager service definition TAGS: [meta, prometheus, prometheus_register]
prometheus : Reload consul to register prometheus TAGS: [meta, prometheus, prometheus_register]
grafana : Make sure grafana is installed TAGS: [grafana, grafana_install, meta]
grafana : Check grafana plugin cache exists TAGS: [grafana, grafana_plugin, meta]
grafana : Provision grafana plugins via cache TAGS: [grafana, grafana_plugin, meta]
grafana : Download grafana plugins from web TAGS: [grafana, grafana_plugin, meta]
grafana : Download grafana plugins from web TAGS: [grafana, grafana_plugin, meta]
grafana : Create grafana plugins cache TAGS: [grafana, grafana_plugin, meta]
grafana : Copy /etc/grafana/grafana.ini TAGS: [grafana, grafana_config, meta]
grafana : Remove grafana provision dir TAGS: [grafana, grafana_config, meta]
grafana : Copy provisioning content TAGS: [grafana, grafana_config, meta]
grafana : Copy pigsty dashboards TAGS: [grafana, grafana_config, meta]
grafana : Copy pigsty icon image TAGS: [grafana, grafana_config, meta]
grafana : Replace grafana icon with pigsty TAGS: [grafana, grafana_config, grafana_customize, meta]
grafana : Launch grafana service TAGS: [grafana, grafana_launch, meta]
grafana : Wait for grafana online TAGS: [grafana, grafana_launch, meta]
grafana : Update grafana default preferences TAGS: [grafana, grafana_provision, meta]
grafana : Register consul grafana service TAGS: [grafana, grafana_register, meta]
grafana : Reload consul TAGS: [grafana, grafana_register, meta]
play #4 (meta): Init dcs TAGS: []
tasks:
consul : Check for existing consul TAGS: [consul_check, dcs]
consul : Consul exists flag fact set TAGS: [consul_check, dcs]
consul : Abort due to consul exists TAGS: [consul_check, dcs]
consul : Clean existing consul instance TAGS: [consul_clean, dcs]
consul : Stop any running consul instance TAGS: [consul_clean, dcs]
consul : Remove existing consul dir TAGS: [consul_clean, dcs]
consul : Recreate consul dir TAGS: [consul_clean, dcs]
consul : Make sure consul is installed TAGS: [consul_install, dcs]
consul : Make sure consul dir exists TAGS: [consul_config, dcs]
consul : Get dcs server node names TAGS: [consul_config, dcs]
consul : Get dcs node name from var TAGS: [consul_config, dcs]
consul : Get dcs node name from var TAGS: [consul_config, dcs]
consul : Fetch hostname as dcs node name TAGS: [consul_config, dcs]
consul : Get dcs name from hostname TAGS: [consul_config, dcs]
consul : Copy /etc/consul.d/consul.json TAGS: [consul_config, dcs]
consul : Copy consul agent service TAGS: [consul_config, dcs]
consul : Get dcs bootstrap expect quroum TAGS: [consul_server, dcs]
consul : Copy consul server service unit TAGS: [consul_server, dcs]
consul : Launch consul server service TAGS: [consul_server, dcs]
consul : Wait for consul server online TAGS: [consul_server, dcs]
consul : Launch consul agent service TAGS: [consul_agent, dcs]
consul : Wait for consul agent online TAGS: [consul_agent, dcs]
play #5 (meta): Copy ansible scripts TAGS: [ansible]
tasks:
Create ansible tarball TAGS: [ansible]
Create ansible directory TAGS: [ansible]
Copy ansible tarball TAGS: [ansible]
Extract tarball TAGS: [ansible]
最后修改 2021-02-23: update deploy doc (ed50323)