Fork me on GitHub

基于Consul+upsync构建动态Nginx配置系统

基于Consul+upsync构建动态Nginx配置系统

前言

本文介绍如何利用Consul+upsync来实现动态的管理Nginx后端服务器,实现动态增删后端服务器而无需reload Nginx服务。

环境说明

IPosroleremark
192.168.200.101CentOS 7.4nginx-lb and consul
192.168.200.102CentOS 7.4nginx-server
192.168.200.103CentOS 7.4nginx-server
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
# uname -r
3.10.0-327.el7.x86_64
# uname -m
x86_64
# nginx -V
nginx version: nginx/1.14.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' --add-module=/data0/src/nginx_upstream_check_module --add-module=/data0/src/nginx-upsync-module
# consul version
Consul v1.2.2
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Nginx-Server安装

1
2
3
4
5
6
7
8
9
10
11
12
# yum install -y pcre-devel openssl-devel
# cat /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
# yum install -y nginx
# systemctl start nginx
# systemctl enable nginx
# echo "linux-node03.example.com:192.168.47.102" >/usr/share/nginx/html/index.html
# echo "linux-node04.example.com:192.168.47.103" >/usr/share/nginx/html/index.html

Nginx-LB配置

Nginx部署

1
2
3
4
5
6
7
8
9
10
# yum install -y pcre-devel openssl-devel
# cat /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
# yum install -y nginx
# systemctl start nginx
# systemctl enable nginx

nginx-upsync-module及nginx_upstream_check_module模块配置

upsync是微博开源的基于nginx的动态流量管理方案. github地址: https://github.com/weibocom/nginx-upsync-module.

nginx-upsync-module,它的功能是拉取consul的后端server的列表,并更新Nginx的路由信息。此模块不依赖于任何第三方模块。 consul作为Nginx的db,利用consul的KV服务,每个Nginx work进程独立的去拉取各个upstream 的配置,并更新各自的路由。

模块相关信息可以参考GitHub文档信息:https://github.com/weibocom/nginx-upsync-module/tree/master/doc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cd /data0/src
git clone https://github.com/weibocom/nginx-upsync-module.git
git clone https://github.com/xiaokai-wang/nginx_upstream_check_module.git
wget http://nginx.org/download/nginx-1.14.0.tar.gz
tar -zxf nginx-1.14.0.tar.gz
cd nginx-1.14.0
patch -p1 < /data0/src/nginx_upstream_check_module/check_1.12.1+.patch
./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' --add-module=/data0/src/nginx_upstream_check_module --add-module=/data0/src/nginx-upsync-module
make
which nginx
cp /usr/sbin/nginx{,.20180804}
systemctl stop nginx
cp /data0/src/nginx-1.14.0/objs/nginx /usr/sbin/
systemctl start nginx
nginx -V

Consul配置

1
2
3
4
5
6
cd /data0/src/
wget https://releases.hashicorp.com/consul/1.2.2/consul_1.2.2_linux_amd64.zip
unzip /data0/src/consul_1.2.2_linux_amd64.zip
install /data0/src/consul /usr/local/bin/
mkdir -p /data0/consul/{data,logs}
consul agent --server --bind=192.168.200.101 --data-dir=/data0/consule/data -bootstrap-expect=1 -node=nginx-lb -client 0.0.0.0 -ui

调整Nginx配置并测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
# cat /etc/nginx/nginx.conf

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;


events {
worker_connections 1024;
}


http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

sendfile on;
#tcp_nopush on;

keepalive_timeout 65;

#gzip on;

include /etc/nginx/conf.d/*.conf;

upstream test {
server 127.0.0.1:11111;
upsync 192.168.200.101:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
upsync_dump_path /etc/nginx/server/server.conf;
}

upstream bar {
server 192.168.200.102:80 weight=1 fail_timeout=10 max_fails=3;
}


server {
listen 80;

location = / {
proxy_pass http://test;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
add_header real $upstream_addr;
}

location ~ /Content|Scripts/ {
proxy_pass http://test;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

location = /bar {
proxy_pass http://bar;
}

location = /upstream_show {
upstream_show;
}

location = /upstream_status {
stub_status on;
access_log off;
}
}
}

创建upsync_dump_path

1
mkdir -p /etc/nginx/server/

添加后端

1
2
curl -X PUT http://192.168.200.101:8500/v1/kv/upstreams/test/192.168.200.102:80
curl -X PUT http://192.168.200.101:8500/v1/kv/upstreams/test/192.168.200.103:80

查看通过consul添加的后端服务器信息

1
2
3
# cat /etc/nginx/server/server.conf
server 192.168.200.103:80 weight=1 max_fails=2 fail_timeout=10s;
server 192.168.200.102:80 weight=1 max_fails=2 fail_timeout=10s;

验证nginx代理信息

1
2
3
4
5
6
7
8
9
10
11
12
13
# for i in `seq 100` ;do curl -s -I http://192.168.200.101|tail -2|head -1;done
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80
real: 192.168.200.102:80
real: 192.168.200.103:80

real地址就是后端服务器的地址

删除后端

1
2
curl -X DELETE http://192.168.200.101:8500/v1/kv/upstreams/test/192.168.200.102:80
curl -X DELETE http://192.168.200.101:8500/v1/kv/upstreams/test/192.168.200.103:80

测试并验证


======================================================
希望各位朋友支持一下

本文作者:dongsheng
本文地址https://mds1455975151.github.io/archives/c92739f9.html
版权声明:转载请注明出处!

坚持技术分享,您的支持将鼓励我继续创作!