Fork me on GitHub

基于Consul+upsync构建动态Nginx配置系统

发表于 | 分类于 | | 阅读次数   |   字数统计: 1,627(字)   |   阅读时长: 9(分)
基于consul+upsync构建动态Nginx配置系统

基于Consul+upsync构建动态Nginx配置系统

前言

本文介绍如何利用Consul+upsync来实现动态的管理Nginx后端服务器，实现动态增删后端服务器而无需reload Nginx服务。

环境说明

IPosroleremark
192.168.200.101CentOS 7.4nginx-lb and consul
192.168.200.102CentOS 7.4nginx-server
192.168.200.103CentOS 7.4nginx-server
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
# uname -r
3.10.0-327.el7.x86_64
# uname -m
x86_64
# nginx -V
nginx version: nginx/1.14.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' --add-module=/data0/src/nginx_upstream_check_module --add-module=/data0/src/nginx-upsync-module
# consul version
Consul v1.2.2
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Nginx-Server安装

1
2
3
4
5
6
7
8
9
10
11
12
# yum install -y pcre-devel openssl-devel
# cat /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
# yum install -y nginx
# systemctl start nginx
# systemctl enable nginx
# echo "linux-node03.example.com:192.168.47.102" >/usr/share/nginx/html/index.html
# echo "linux-node04.example.com:192.168.47.103" >/usr/share/nginx/html/index.html

Nginx-LB配置

Nginx部署

1
2
3
4
5
6
7
8
9
10
# yum install -y pcre-devel openssl-devel
# cat /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
# yum install -y nginx
# systemctl start nginx
# systemctl enable nginx

nginx-upsync-module及nginx_upstream_check_module模块配置

upsync是微博开源的基于nginx的动态流量管理方案. github地址: https://github.com/weibocom/nginx-upsync-module.

nginx-upsync-module，它的功能是拉取consul的后端server的列表，并更新Nginx的路由信息。此模块不依赖于任何第三方模块。 consul作为Nginx的db，利用consul的KV服务，每个Nginx work进程独立的去拉取各个upstream 的配置，并更新各自的路由。

模块相关信息可以参考GitHub文档信息：https://github.com/weibocom/nginx-upsync-module/tree/master/doc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cd /data0/src
git clone https://github.com/weibocom/nginx-upsync-module.git
git clone https://github.com/xiaokai-wang/nginx_upstream_check_module.git
wget http://nginx.org/download/nginx-1.14.0.tar.gz
tar -zxf nginx-1.14.0.tar.gz
cd nginx-1.14.0
patch -p1 < /data0/src/nginx_upstream_check_module/check_1.12.1+.patch
./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' --add-module=/data0/src/nginx_upstream_check_module --add-module=/data0/src/nginx-upsync-module
make
which nginx
cp /usr/sbin/nginx{,.20180804}
systemctl stop nginx
cp /data0/src/nginx-1.14.0/objs/nginx /usr/sbin/
systemctl start nginx
nginx -V

Consul配置

1
2
3
4
5
6
cd /data0/src/
wget https://releases.hashicorp.com/consul/1.2.2/consul_1.2.2_linux_amd64.zip
unzip /data0/src/consul_1.2.2_linux_amd64.zip
install /data0/src/consul /usr/local/bin/
mkdir -p /data0/consul/{data,logs}
consul agent --server --bind=192.168.200.101 --data-dir=/data0/consule/data -bootstrap-expect=1 -node=nginx-lb -client 0.0.0.0 -ui

调整Nginx配置并测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
# cat /etc/nginx/nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    upstream test {
        server 127.0.0.1:11111;
        upsync 192.168.200.101:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
        upsync_dump_path /etc/nginx/server/server.conf;
    }

    upstream bar {
        server 192.168.200.102:80 weight=1 fail_timeout=10 max_fails=3;
    }


    server {
        listen 80;

        location = / {
        proxy_pass http://test;
          proxy_set_header  Host  $host;
          proxy_set_header  X-Real-IP  $remote_addr;
          proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
          add_header    real $upstream_addr;
        }

        location ~ /Content|Scripts/ {
          proxy_pass http://test;
          proxy_set_header  Host  $host;
          proxy_set_header  X-Real-IP  $remote_addr;
          proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
        }

        location = /bar {
            proxy_pass http://bar;
        }

        location = /upstream_show {
            upstream_show;
        }

        location = /upstream_status {
            stub_status on;
            access_log off;
        }
    }
}

创建upsync_dump_path

1
mkdir -p /etc/nginx/server/

添加后端

1
2
curl -X PUT http://192.168.200.101:8500/v1/kv/upstreams/test/192.168.200.102:80
curl -X PUT http://192.168.200.101:8500/v1/kv/upstreams/test/192.168.200.103:80

查看通过consul添加的后端服务器信息

1
2
3
# cat /etc/nginx/server/server.conf
server 192.168.200.103:80 weight=1 max_fails=2 fail_timeout=10s;
server 192.168.200.102:80 weight=1 max_fails=2 fail_timeout=10s;

验证nginx代理信息

1
2
3
4
5
6
7
8
9
10
11
12
13
# for i in `seq 100` ;do curl -s -I http://192.168.200.101|tail -2|head -1;done
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80

real地址就是后端服务器的地址

删除后端

1
2
curl -X DELETE http://192.168.200.101:8500/v1/kv/upstreams/test/192.168.200.102:80
curl -X DELETE http://192.168.200.101:8500/v1/kv/upstreams/test/192.168.200.103:80

测试并验证


======================================================
希望各位朋友支持一下

本文作者：dongsheng
本文地址： https://mds1455975151.github.io/archives/c92739f9.html
版权声明：转载请注明出处！

坚持技术分享，您的支持将鼓励我继续创作！