Infra Provision
How to provision infrastructure with playbooks
剧本概览
Pigsty需要首先完成元节点的基础设施部署，基础设施的部署通过infra.yml完成。
./infra.yml
注意事项
您必须完成元节点的初始化后，才能正常执行普通节点的初始化工作。
infra.yml 固定会作用于配置文件中 名为 meta 的分组
您可以将元节点当成普通节点复用，即在元节点上定义并创建PostgreSQL数据库。
我们建议按照默认配置在元节点上创建一个pg-meta元数据库，但不要施加过多负载。
完整执行一遍初始化流程可能花费2～8分钟，视您的机器配置而定。
选择性执行
您可以通过ansible的标签机制，可以选择执行剧本的一个子集。
例如，如果您只想执行本地源初始化的部分，则可以通过以下命令进行
./infra.yml --tags=repo
具体的标签请参考任务详情
剧本说明
infra.yml 主要完成以下工作
- 部署本地源
- 完成元节点的初始化
- 完成Meta基础设施初始化
- CA基础设施
- DNS Nameserver
- Nginx
- Prometheus & Alertmanger
- Grafana
- 将Pigsty本体拷贝至元节点
- 在元节点上完成数据库初始化（可选）
#------------------------------------------------------------------------------
# init local yum repo (only run on meta nodes)
#------------------------------------------------------------------------------
- name: Init local repo
become: yes
hosts: meta
gather_facts: no
tags: repo
roles:
- repo
#------------------------------------------------------------------------------
# provision nodes
#------------------------------------------------------------------------------
- name: Provision Node
become: yes
hosts: meta
gather_facts: no
tags: node
roles:
- node
#------------------------------------------------------------------------------
# init meta service (only run on meta nodes)
#------------------------------------------------------------------------------
- name: Init meta service
become: yes
hosts: meta
gather_facts: no
tags: meta
roles:
- role: ca
tags: ca
- role: nameserver
tags: nameserver
- role: nginx
tags: nginx
- role: prometheus
tags: prometheus
- role: grafana
tags: grafana
#------------------------------------------------------------------------------
# init dcs on nodes
#------------------------------------------------------------------------------
- name: Init dcs
become: yes
hosts: meta
gather_facts: no
roles:
- role: consul
tags: dcs
#------------------------------------------------------------------------------
# copy scripts to meta node
#------------------------------------------------------------------------------
- name: Copy ansible scripts
become: yes
hosts: meta
gather_facts: no
ignore_errors: yes
tags: ansible
tasks:
- name: Copy ansible scritps
when: node_admin_setup is defined and node_admin_setup|bool and node_admin_username != ''
block:
# create copy of this repo
- name: Create ansible tarball
become: no
connection: local
run_once: true
command:
cmd: tar -cf files/meta.tgz roles templates ansible.cfg infra.yml pgsql.yml pgsql-rm.yml pigsty.yml Makefile
chdir: "{{ playbook_dir }}"
- name: Create ansible directory
file: path="/home/{{ node_admin_username }}/meta" state=directory owner={{ node_admin_username }}
- name: Copy ansible tarball
copy: src="meta.tgz" dest="/home/{{ node_admin_username }}/meta/meta.tgz" owner={{ node_admin_username }}
- name: Extract tarball
shell: |
cd /home/{{ node_admin_username }}/meta/
tar -xf meta.tgz
chown -R {{ node_admin_username }} /home/{{ node_admin_username }}
rm -rf meta.tgz
chmod a+x *.yml
任务详情
使用以下命令可以列出所有基础设施初始化会执行的任务，以及可以使用的标签：
./infra.yml --list-tasks
默认任务如下：
playbook: ./infra.yml
play #1 (meta): Init local repo TAGS: [repo]
tasks:
repo : Create local repo directory TAGS: [repo, repo_dir]
repo : Backup & remove existing repos TAGS: [repo, repo_upstream]
repo : Add required upstream repos TAGS: [repo, repo_upstream]
repo : Check repo pkgs cache exists TAGS: [repo, repo_prepare]
repo : Set fact whether repo_exists TAGS: [repo, repo_prepare]
repo : Move upstream repo to backup TAGS: [repo, repo_prepare]
repo : Add local file system repos TAGS: [repo, repo_prepare]
repo : Remake yum cache if not exists TAGS: [repo, repo_prepare]
repo : Install repo bootstrap packages TAGS: [repo, repo_boot]
repo : Render repo nginx server files TAGS: [repo, repo_nginx]
repo : Disable selinux for repo server TAGS: [repo, repo_nginx]
repo : Launch repo nginx server TAGS: [repo, repo_nginx]
repo : Waits repo server online TAGS: [repo, repo_nginx]
repo : Download web url packages TAGS: [repo, repo_download]
repo : Download repo packages TAGS: [repo, repo_download]
repo : Download repo pkg deps TAGS: [repo, repo_download]
repo : Create local repo index TAGS: [repo, repo_download]
repo : Copy bootstrap scripts TAGS: [repo, repo_download, repo_script]
repo : Mark repo cache as valid TAGS: [repo, repo_download]
play #2 (meta): Provision Node TAGS: [node]
tasks:
node : Update node hostname TAGS: [node, node_name]
node : Add new hostname to /etc/hosts TAGS: [node, node_name]
node : Write static dns records TAGS: [node, node_dns]
node : Get old nameservers TAGS: [node, node_resolv]
node : Truncate resolv file TAGS: [node, node_resolv]
node : Write resolv options TAGS: [node, node_resolv]
node : Add new nameservers TAGS: [node, node_resolv]
node : Append old nameservers TAGS: [node, node_resolv]
node : Node configure disable firewall TAGS: [node, node_firewall]
node : Node disable selinux by default TAGS: [node, node_firewall]
node : Backup existing repos TAGS: [node, node_repo]
node : Install upstream repo TAGS: [node, node_repo]
node : Install local repo TAGS: [node, node_repo]
node : Install node basic packages TAGS: [node, node_pkgs]
node : Install node extra packages TAGS: [node, node_pkgs]
node : Install meta specific packages TAGS: [node, node_pkgs]
node : Install node basic packages TAGS: [node, node_pkgs]
node : Install node extra packages TAGS: [node, node_pkgs]
node : Install meta specific packages TAGS: [node, node_pkgs]
node : Node configure disable numa TAGS: [node, node_feature]
node : Node configure disable swap TAGS: [node, node_feature]
node : Node configure unmount swap TAGS: [node, node_feature]
node : Node setup static network TAGS: [node, node_feature]
node : Node configure disable firewall TAGS: [node, node_feature]
node : Node configure disk prefetch TAGS: [node, node_feature]
node : Enable linux kernel modules TAGS: [node, node_kernel]
node : Enable kernel module on reboot TAGS: [node, node_kernel]
node : Get config parameter page count TAGS: [node, node_tuned]
node : Get config parameter page size TAGS: [node, node_tuned]
node : Tune shmmax and shmall via mem TAGS: [node, node_tuned]
node : Create tuned profiles TAGS: [node, node_tuned]
node : Render tuned profiles TAGS: [node, node_tuned]
node : Active tuned profile TAGS: [node, node_tuned]
node : Change additional sysctl params TAGS: [node, node_tuned]
node : Copy default user bash profile TAGS: [node, node_profile]
node : Setup node default pam ulimits TAGS: [node, node_ulimit]
node : Create os user group admin TAGS: [node, node_admin]
node : Create os user admin TAGS: [node, node_admin]
node : Grant admin group nopass sudo TAGS: [node, node_admin]
node : Add no host checking to ssh config TAGS: [node, node_admin]
node : Add admin ssh no host checking TAGS: [node, node_admin]
node : Fetch all admin public keys TAGS: [node, node_admin]
node : Exchange all admin ssh keys TAGS: [node, node_admin]
node : Install public keys TAGS: [node, node_admin]
node : Install ntp package TAGS: [node, ntp_install]
node : Install chrony package TAGS: [node, ntp_install]
node : Setup default node timezone TAGS: [node, ntp_config]
node : Copy the ntp.conf file TAGS: [node, ntp_config]
node : Copy the chrony.conf template TAGS: [node, ntp_config]
node : Launch ntpd service TAGS: [node, ntp_launch]
node : Launch chronyd service TAGS: [node, ntp_launch]
play #3 (meta): Init meta service TAGS: [meta]
tasks:
ca : Create local ca directory TAGS: [ca, ca_dir, meta]
ca : Copy ca cert from local files TAGS: [ca, ca_copy, meta]
ca : Check ca key cert exists TAGS: [ca, ca_create, meta]
ca : Create self-signed CA key-cert TAGS: [ca, ca_create, meta]
nameserver : Make sure dnsmasq package installed TAGS: [meta, nameserver]
nameserver : Copy dnsmasq /etc/dnsmasq.d/config TAGS: [meta, nameserver]
nameserver : Add dynamic dns records to meta TAGS: [meta, nameserver]
nameserver : Launch meta dnsmasq service TAGS: [meta, nameserver]
nameserver : Wait for meta dnsmasq online TAGS: [meta, nameserver]
nginx : Make sure nginx package installed TAGS: [meta, nginx, nginx_install]
nginx : Create local html directory TAGS: [meta, nginx, nginx_dir]
nginx : Update default nginx index page TAGS: [meta, nginx, nginx_dir]
nginx : Copy nginx default config TAGS: [meta, nginx, nginx_config]
nginx : Copy nginx upstream conf TAGS: [meta, nginx, nginx_config]
nginx : Fetch haproxy facts TAGS: [meta, nginx, nginx_config, nginx_haproxy]
debug TAGS: [meta, nginx, nginx_config, nginx_haproxy]
nginx : Templating /etc/nginx/haproxy.conf TAGS: [meta, nginx, nginx_config, nginx_haproxy]
nginx : Templating haproxy.html TAGS: [meta, nginx, nginx_config, nginx_haproxy]
nginx : Launch nginx server TAGS: [meta, nginx, nginx_reload]
nginx : Restart meta nginx service TAGS: [meta, nginx, nginx_launch]
nginx : Wait for nginx service online TAGS: [meta, nginx, nginx_launch]
nginx : Make sure nginx exporter installed TAGS: [meta, nginx, nginx_exporter]
nginx : Config nginx_exporter options TAGS: [meta, nginx, nginx_exporter]
nginx : Restart nginx_exporter service TAGS: [meta, nginx, nginx_exporter]
nginx : Wait for nginx exporter online TAGS: [meta, nginx, nginx_exporter]
nginx : Register cosnul nginx service TAGS: [meta, nginx, nginx_register]
nginx : Register consul nginx-exporter service TAGS: [meta, nginx, nginx_register]
nginx : Reload consul TAGS: [meta, nginx, nginx_register]
prometheus : Install prometheus and alertmanager TAGS: [meta, prometheus, prometheus_install]
prometheus : Wipe out prometheus config dir TAGS: [meta, prometheus, prometheus_clean]
prometheus : Wipe out existing prometheus data TAGS: [meta, prometheus, prometheus_clean]
prometheus : Recreate prometheus data dir TAGS: [meta, prometheus, prometheus_config]
prometheus : Copy /etc/prometheus configs TAGS: [meta, prometheus, prometheus_config]
prometheus : Copy /etc/prometheus opts TAGS: [meta, prometheus, prometheus_config]
prometheus : Overwrite prometheus scrape_interval TAGS: [meta, prometheus, prometheus_config]
prometheus : Overwrite prometheus evaluation_interval TAGS: [meta, prometheus, prometheus_config]
prometheus : Overwrite prometheus scrape_timeout TAGS: [meta, prometheus, prometheus_config]
prometheus : Overwrite prometheus pg metrics path TAGS: [meta, prometheus, prometheus_config]
prometheus : Launch prometheus service TAGS: [meta, prometheus, prometheus_launch]
prometheus : Launch alertmanager service TAGS: [meta, prometheus, prometheus_launch]
prometheus : Wait for prometheus online TAGS: [meta, prometheus, prometheus_launch]
prometheus : Wait for alertmanager online TAGS: [meta, prometheus, prometheus_launch]
grafana : Make sure grafana is installed TAGS: [grafana, grafana_install, meta]
grafana : Check grafana plugin cache exists TAGS: [grafana, grafana_plugin, meta]
grafana : Provision grafana plugins via cache TAGS: [grafana, grafana_plugin, meta]
grafana : Download grafana plugins from web TAGS: [grafana, grafana_plugin, meta]
grafana : Download grafana plugins from web TAGS: [grafana, grafana_plugin, meta]
grafana : Create grafana plugins cache TAGS: [grafana, grafana_plugin, meta]
grafana : Copy /etc/grafana/grafana.ini TAGS: [grafana, grafana_config, meta]
grafana : Remove grafana provision dir TAGS: [grafana, grafana_config, meta]
grafana : Copy provisioning content TAGS: [grafana, grafana_config, meta]
grafana : Copy pigsty dashboards TAGS: [grafana, grafana_config, meta]
grafana : Copy pigsty icon image TAGS: [grafana, grafana_config, meta]
grafana : Replace grafana icon with pigsty TAGS: [grafana, grafana_config, grafana_customize, meta]
grafana : Launch grafana service TAGS: [grafana, grafana_launch, meta]
grafana : Wait for grafana online TAGS: [grafana, grafana_launch, meta]
grafana : Update grafana default preferences TAGS: [grafana, grafana_provision, meta]
grafana : Register consul grafana service TAGS: [grafana, grafana_register, meta]
grafana : Reload consul TAGS: [grafana, grafana_register, meta]
play #4 (meta): Init dcs TAGS: []
tasks:
consul : Check for existing consul TAGS: [consul_check, dcs]
consul : Consul exists flag fact set TAGS: [consul_check, dcs]
consul : Abort due to consul exists TAGS: [consul_check, dcs]
consul : Clean existing consul instance TAGS: [consul_check, dcs]
consul : Stop any running consul instance TAGS: [consul_check, dcs]
consul : Remove existing consul dir TAGS: [consul_check, dcs]
consul : Recreate consul dir TAGS: [consul_check, dcs]
consul : Make sure consul is installed TAGS: [consul_install, dcs]
consul : Make sure consul dir exists TAGS: [consul_config, dcs]
consul : Get dcs server node names TAGS: [consul_config, dcs]
consul : Get dcs node name from var TAGS: [consul_config, dcs]
consul : Get dcs node name from var TAGS: [consul_config, dcs]
consul : Fetch hostname as dcs node name TAGS: [consul_config, dcs]
consul : Get dcs name from hostname TAGS: [consul_config, dcs]
consul : Copy /etc/consul.d/consul.json TAGS: [consul_config, dcs]
consul : Copy consul agent service TAGS: [consul_config, dcs]
consul : Get dcs bootstrap expect quroum TAGS: [consul_server, dcs]
consul : Copy consul server service unit TAGS: [consul_server, dcs]
consul : Launch consul server service TAGS: [consul_server, dcs]
consul : Wait for consul server online TAGS: [consul_server, dcs]
consul : Launch consul agent service TAGS: [consul_agent, dcs]
consul : Wait for consul agent online TAGS: [consul_agent, dcs]
play #5 (meta): Copy ansible scripts TAGS: [ansible]
tasks:
Create ansible tarball TAGS: [ansible]
Create ansible directory TAGS: [ansible]
Copy ansible tarball TAGS: [ansible]
Extract tarball TAGS: [ansible]
Last modified 2021-02-08: update en deploy doc (c50b13b)