基础设施初始化

如何使用剧本初始化基础设施

剧本概览

Pigsty需要首先完成元节点的基础设施部署,基础设施的部署通过infra.yml完成。

./infra.yml

注意事项

您必须完成元节点的初始化后,才能正常执行普通节点的初始化工作。

infra.yml 固定会作用于配置文件中 名为 meta 的分组

您可以将元节点当成普通节点复用,即在元节点上定义并创建PostgreSQL数据库。

我们建议按照默认配置在元节点上创建一个pg-meta元数据库,但不要施加过多负载。

完整执行一遍初始化流程可能花费2~8分钟,视您的机器配置而定。

选择性执行

您可以通过ansible的标签机制,可以选择执行剧本的一个子集。

例如,如果您只想执行本地源初始化的部分,则可以通过以下命令进行

./infra.yml --tags=repo

具体的标签请参考任务详情

剧本说明

infra.yml 主要完成以下工作

  • 部署本地源
  • 完成元节点的初始化
  • 完成Meta基础设施初始化
    • CA基础设施
    • DNS Nameserver
    • Nginx
    • Prometheus & Alertmanger
    • Grafana
  • 将Pigsty本体拷贝至元节点
  • 在元节点上完成数据库初始化(可选)
#------------------------------------------------------------------------------
# init local yum repo (only run on meta nodes)
#------------------------------------------------------------------------------
- name: Init local repo
  become: yes
  hosts: meta
  gather_facts: no
  tags: repo
  roles:
    - repo
#------------------------------------------------------------------------------
# provision nodes
#------------------------------------------------------------------------------
- name: Provision Node
  become: yes
  hosts: meta
  gather_facts: no
  tags: node
  roles:
    - node
#------------------------------------------------------------------------------
# init meta service (only run on meta nodes)
#------------------------------------------------------------------------------
- name: Init meta service
  become: yes
  hosts: meta
  gather_facts: no
  tags: meta
  roles:
    - role: ca
      tags: ca

    - role: nameserver
      tags: nameserver

    - role: nginx
      tags: nginx

    - role: prometheus
      tags: prometheus

    - role: grafana
      tags: grafana
#------------------------------------------------------------------------------
# init dcs on nodes
#------------------------------------------------------------------------------
- name: Init dcs
  become: yes
  hosts: meta
  gather_facts: no
  roles:
    - role: consul
      tags: dcs
#------------------------------------------------------------------------------
# copy scripts to meta node
#------------------------------------------------------------------------------
- name: Copy ansible scripts
  become: yes
  hosts: meta
  gather_facts: no
  ignore_errors: yes
  tags: ansible
  tasks:
    - name: Copy ansible scritps
      when: node_admin_setup is defined and node_admin_setup|bool and node_admin_username != ''
      block:
        # create copy of this repo
        - name: Create ansible tarball
          become: no
          connection: local
          run_once: true
          command:
            cmd: tar -cf files/meta.tgz roles templates ansible.cfg infra.yml pgsql.yml pgsql-rm.yml pigsty.yml Makefile
            chdir: "{{ playbook_dir }}"

        - name: Create ansible directory
          file: path="/home/{{ node_admin_username }}/meta" state=directory owner={{ node_admin_username }}

        - name: Copy ansible tarball
          copy: src="meta.tgz" dest="/home/{{ node_admin_username }}/meta/meta.tgz" owner={{ node_admin_username }}

        - name: Extract tarball
          shell: |
            cd /home/{{ node_admin_username }}/meta/
            tar -xf meta.tgz
            chown -R {{ node_admin_username }} /home/{{ node_admin_username }}
            rm -rf meta.tgz
            chmod a+x *.yml            

任务详情

使用以下命令可以列出所有基础设施初始化会执行的任务,以及可以使用的标签:

./infra.yml --list-tasks

默认任务如下:

playbook: ./infra.yml

  play #1 (meta): Init local repo	TAGS: [repo]
    tasks:
      repo : Create local repo directory	TAGS: [repo, repo_dir]
      repo : Backup & remove existing repos	TAGS: [repo, repo_upstream]
      repo : Add required upstream repos	TAGS: [repo, repo_upstream]
      repo : Check repo pkgs cache exists	TAGS: [repo, repo_prepare]
      repo : Set fact whether repo_exists	TAGS: [repo, repo_prepare]
      repo : Move upstream repo to backup	TAGS: [repo, repo_prepare]
      repo : Add local file system repos	TAGS: [repo, repo_prepare]
      repo : Remake yum cache if not exists	TAGS: [repo, repo_prepare]
      repo : Install repo bootstrap packages	TAGS: [repo, repo_boot]
      repo : Render repo nginx server files	TAGS: [repo, repo_nginx]
      repo : Disable selinux for repo server	TAGS: [repo, repo_nginx]
      repo : Launch repo nginx server	TAGS: [repo, repo_nginx]
      repo : Waits repo server online	TAGS: [repo, repo_nginx]
      repo : Download web url packages	TAGS: [repo, repo_download]
      repo : Download repo packages	TAGS: [repo, repo_download]
      repo : Download repo pkg deps	TAGS: [repo, repo_download]
      repo : Create local repo index	TAGS: [repo, repo_download]
      repo : Copy bootstrap scripts	TAGS: [repo, repo_download, repo_script]
      repo : Mark repo cache as valid	TAGS: [repo, repo_download]

  play #2 (meta): Provision Node	TAGS: [node]
    tasks:
      node : Update node hostname	TAGS: [node, node_name]
      node : Add new hostname to /etc/hosts	TAGS: [node, node_name]
      node : Write static dns records	TAGS: [node, node_dns]
      node : Get old nameservers	TAGS: [node, node_resolv]
      node : Truncate resolv file	TAGS: [node, node_resolv]
      node : Write resolv options	TAGS: [node, node_resolv]
      node : Add new nameservers	TAGS: [node, node_resolv]
      node : Append old nameservers	TAGS: [node, node_resolv]
      node : Node configure disable firewall	TAGS: [node, node_firewall]
      node : Node disable selinux by default	TAGS: [node, node_firewall]
      node : Backup existing repos	TAGS: [node, node_repo]
      node : Install upstream repo	TAGS: [node, node_repo]
      node : Install local repo	TAGS: [node, node_repo]
      node : Install node basic packages	TAGS: [node, node_pkgs]
      node : Install node extra packages	TAGS: [node, node_pkgs]
      node : Install meta specific packages	TAGS: [node, node_pkgs]
      node : Install node basic packages	TAGS: [node, node_pkgs]
      node : Install node extra packages	TAGS: [node, node_pkgs]
      node : Install meta specific packages	TAGS: [node, node_pkgs]
      node : Node configure disable numa	TAGS: [node, node_feature]
      node : Node configure disable swap	TAGS: [node, node_feature]
      node : Node configure unmount swap	TAGS: [node, node_feature]
      node : Node setup static network	TAGS: [node, node_feature]
      node : Node configure disable firewall	TAGS: [node, node_feature]
      node : Node configure disk prefetch	TAGS: [node, node_feature]
      node : Enable linux kernel modules	TAGS: [node, node_kernel]
      node : Enable kernel module on reboot	TAGS: [node, node_kernel]
      node : Get config parameter page count	TAGS: [node, node_tuned]
      node : Get config parameter page size	TAGS: [node, node_tuned]
      node : Tune shmmax and shmall via mem	TAGS: [node, node_tuned]
      node : Create tuned profiles	TAGS: [node, node_tuned]
      node : Render tuned profiles	TAGS: [node, node_tuned]
      node : Active tuned profile	TAGS: [node, node_tuned]
      node : Change additional sysctl params	TAGS: [node, node_tuned]
      node : Copy default user bash profile	TAGS: [node, node_profile]
      node : Setup node default pam ulimits	TAGS: [node, node_ulimit]
      node : Create os user group admin	TAGS: [node, node_admin]
      node : Create os user admin	TAGS: [node, node_admin]
      node : Grant admin group nopass sudo	TAGS: [node, node_admin]
      node : Add no host checking to ssh config	TAGS: [node, node_admin]
      node : Add admin ssh no host checking	TAGS: [node, node_admin]
      node : Fetch all admin public keys	TAGS: [node, node_admin]
      node : Exchange all admin ssh keys	TAGS: [node, node_admin]
      node : Install public keys	TAGS: [node, node_admin]
      node : Install ntp package	TAGS: [node, ntp_install]
      node : Install chrony package	TAGS: [node, ntp_install]
      node : Setup default node timezone	TAGS: [node, ntp_config]
      node : Copy the ntp.conf file	TAGS: [node, ntp_config]
      node : Copy the chrony.conf template	TAGS: [node, ntp_config]
      node : Launch ntpd service	TAGS: [node, ntp_launch]
      node : Launch chronyd service	TAGS: [node, ntp_launch]

  play #3 (meta): Init meta service	TAGS: [meta]
    tasks:
      ca : Create local ca directory	TAGS: [ca, ca_dir, meta]
      ca : Copy ca cert from local files	TAGS: [ca, ca_copy, meta]
      ca : Check ca key cert exists	TAGS: [ca, ca_create, meta]
      ca : Create self-signed CA key-cert	TAGS: [ca, ca_create, meta]
      nameserver : Make sure dnsmasq package installed	TAGS: [meta, nameserver]
      nameserver : Copy dnsmasq /etc/dnsmasq.d/config	TAGS: [meta, nameserver]
      nameserver : Add dynamic dns records to meta	TAGS: [meta, nameserver]
      nameserver : Launch meta dnsmasq service	TAGS: [meta, nameserver]
      nameserver : Wait for meta dnsmasq online	TAGS: [meta, nameserver]
      nginx : Make sure nginx package installed	TAGS: [meta, nginx, nginx_install]
      nginx : Create local html directory	TAGS: [meta, nginx, nginx_dir]
      nginx : Update default nginx index page	TAGS: [meta, nginx, nginx_dir]
      nginx : Copy nginx default config	TAGS: [meta, nginx, nginx_config]
      nginx : Copy nginx upstream conf	TAGS: [meta, nginx, nginx_config]
      nginx : Fetch haproxy facts	TAGS: [meta, nginx, nginx_config, nginx_haproxy]
      debug	TAGS: [meta, nginx, nginx_config, nginx_haproxy]
      nginx : Templating /etc/nginx/haproxy.conf	TAGS: [meta, nginx, nginx_config, nginx_haproxy]
      nginx : Templating haproxy.html	TAGS: [meta, nginx, nginx_config, nginx_haproxy]
      nginx : Launch nginx server	TAGS: [meta, nginx, nginx_reload]
      nginx : Restart meta nginx service	TAGS: [meta, nginx, nginx_launch]
      nginx : Wait for nginx service online	TAGS: [meta, nginx, nginx_launch]
      nginx : Make sure nginx exporter installed	TAGS: [meta, nginx, nginx_exporter]
      nginx : Config nginx_exporter options	TAGS: [meta, nginx, nginx_exporter]
      nginx : Restart nginx_exporter service	TAGS: [meta, nginx, nginx_exporter]
      nginx : Wait for nginx exporter online	TAGS: [meta, nginx, nginx_exporter]
      nginx : Register cosnul nginx service	TAGS: [meta, nginx, nginx_register]
      nginx : Register consul nginx-exporter service	TAGS: [meta, nginx, nginx_register]
      nginx : Reload consul	TAGS: [meta, nginx, nginx_register]
      prometheus : Install prometheus and alertmanager	TAGS: [meta, prometheus, prometheus_install]
      prometheus : Wipe out prometheus config dir	TAGS: [meta, prometheus, prometheus_clean]
      prometheus : Wipe out existing prometheus data	TAGS: [meta, prometheus, prometheus_clean]
      prometheus : Recreate prometheus data dir	TAGS: [meta, prometheus, prometheus_config]
      prometheus : Copy /etc/prometheus configs	TAGS: [meta, prometheus, prometheus_config]
      prometheus : Copy /etc/prometheus opts	TAGS: [meta, prometheus, prometheus_config]
      prometheus : Overwrite prometheus scrape_interval	TAGS: [meta, prometheus, prometheus_config]
      prometheus : Overwrite prometheus evaluation_interval	TAGS: [meta, prometheus, prometheus_config]
      prometheus : Overwrite prometheus scrape_timeout	TAGS: [meta, prometheus, prometheus_config]
      prometheus : Overwrite prometheus pg metrics path	TAGS: [meta, prometheus, prometheus_config]
      prometheus : Launch prometheus service	TAGS: [meta, prometheus, prometheus_launch]
      prometheus : Launch alertmanager service	TAGS: [meta, prometheus, prometheus_launch]
      prometheus : Wait for prometheus online	TAGS: [meta, prometheus, prometheus_launch]
      prometheus : Wait for alertmanager online	TAGS: [meta, prometheus, prometheus_launch]
      grafana : Make sure grafana is installed	TAGS: [grafana, grafana_install, meta]
      grafana : Check grafana plugin cache exists	TAGS: [grafana, grafana_plugin, meta]
      grafana : Provision grafana plugins via cache	TAGS: [grafana, grafana_plugin, meta]
      grafana : Download grafana plugins from web	TAGS: [grafana, grafana_plugin, meta]
      grafana : Download grafana plugins from web	TAGS: [grafana, grafana_plugin, meta]
      grafana : Create grafana plugins cache	TAGS: [grafana, grafana_plugin, meta]
      grafana : Copy /etc/grafana/grafana.ini	TAGS: [grafana, grafana_config, meta]
      grafana : Remove grafana provision dir	TAGS: [grafana, grafana_config, meta]
      grafana : Copy provisioning content	TAGS: [grafana, grafana_config, meta]
      grafana : Copy pigsty dashboards	TAGS: [grafana, grafana_config, meta]
      grafana : Copy pigsty icon image	TAGS: [grafana, grafana_config, meta]
      grafana : Replace grafana icon with pigsty	TAGS: [grafana, grafana_config, grafana_customize, meta]
      grafana : Launch grafana service	TAGS: [grafana, grafana_launch, meta]
      grafana : Wait for grafana online	TAGS: [grafana, grafana_launch, meta]
      grafana : Update grafana default preferences	TAGS: [grafana, grafana_provision, meta]
      grafana : Register consul grafana service	TAGS: [grafana, grafana_register, meta]
      grafana : Reload consul	TAGS: [grafana, grafana_register, meta]

  play #4 (meta): Init dcs	TAGS: []
    tasks:
      consul : Check for existing consul	TAGS: [consul_check, dcs]
      consul : Consul exists flag fact set	TAGS: [consul_check, dcs]
      consul : Abort due to consul exists	TAGS: [consul_check, dcs]
      consul : Clean existing consul instance	TAGS: [consul_check, dcs]
      consul : Stop any running consul instance	TAGS: [consul_check, dcs]
      consul : Remove existing consul dir	TAGS: [consul_check, dcs]
      consul : Recreate consul dir	TAGS: [consul_check, dcs]
      consul : Make sure consul is installed	TAGS: [consul_install, dcs]
      consul : Make sure consul dir exists	TAGS: [consul_config, dcs]
      consul : Get dcs server node names	TAGS: [consul_config, dcs]
      consul : Get dcs node name from var	TAGS: [consul_config, dcs]
      consul : Get dcs node name from var	TAGS: [consul_config, dcs]
      consul : Fetch hostname as dcs node name	TAGS: [consul_config, dcs]
      consul : Get dcs name from hostname	TAGS: [consul_config, dcs]
      consul : Copy /etc/consul.d/consul.json	TAGS: [consul_config, dcs]
      consul : Copy consul agent service	TAGS: [consul_config, dcs]
      consul : Get dcs bootstrap expect quroum	TAGS: [consul_server, dcs]
      consul : Copy consul server service unit	TAGS: [consul_server, dcs]
      consul : Launch consul server service	TAGS: [consul_server, dcs]
      consul : Wait for consul server online	TAGS: [consul_server, dcs]
      consul : Launch consul agent service	TAGS: [consul_agent, dcs]
      consul : Wait for consul agent online	TAGS: [consul_agent, dcs]

  play #5 (meta): Copy ansible scripts	TAGS: [ansible]
    tasks:
      Create ansible tarball	TAGS: [ansible]
      Create ansible directory	TAGS: [ansible]
      Copy ansible tarball	TAGS: [ansible]
      Extract tarball	TAGS: [ansible]

最后修改 2021-01-09: add tutorial (c0b4a87)