Nginx Kubernetes 源码阅读 Lua Ingress OpenResty

接上篇博客介绍的Ingress的使用, 本文将会就Nginx公司与Kubernetes社区的两种Ingress实现进行说明, 而后会将焦点放在Kubernetes社区的Ingress实现, 给出相关的代码逻辑, 而后介绍如何对其进行调试.

Ingress基本实现原理

在上篇博客的开头就介绍到了这个问题, 基于Nginx或是OpenResty的Ingress实现使用 了类似下方的多域名配置, 通过不同的server_name选择合适的后端服务.

1
2
3
4
5
6
7
8
9
10
server {
    listen       80;
    server_name  app1.example.org;
    ...
}
server {
    listen       80;
    server_name  app2.example.org;
    ...
}

目前我用过的有Nginx公司的实现:

https://github.com/nginxinc/kubernetes-ingress

以及Kubernetes社区的实现

https://github.com/kubernetes/ingress-nginx

查看Ingress容器中的配置

即使不会自己手动去改容器中的Nginx配置文件, 至少也该了解下配置文件的结构, 以便未来的调试, 本部分将会介绍两种Ingress实现的Nginx配置.

当我们启动Ingress容器后, 请自己确认一下自己的Ingress容器所在的机器.

1
2
3
4
5
6
7
8
# nginx公司的Ingress容器
07b350a2a1d5        nginx/nginx-ingress                                         "/nginx-ingress -ngi…"   16 seconds ago      Up 16 seconds                                 k8s_nginx-ingress_nginx-ingress-9l5sj_nginx-ingress_26f8244c-84eb-4ff9-8040-11b788439f8c_0

# kubernetes社区的Ingress容器
58af9c7aafd1        29024c9c6e70           "/usr/bin/dumb-init …"   17 hours ago        Up 17 hours                             k8s_nginx-ingress-controller_nginx-ingress-controller-5f96cfcb6-4fcxj_ingress-nginx_6498182d-755a-4ce6-b0d1-a3693b59bc11_0

# 进入容器内部
docker exec -ti 07b350a2a1d5 /bin/bash

Nginx公司的Nginx配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 配置文件入口:
#  /etc/nginx/nginx.conf
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    
    access_log  /var/log/nginx/access.log  main;

    # ...

    include /etc/nginx/config-version.conf;
    include /etc/nginx/conf.d/*.conf;

    # ...
}

从上方的配置文件来看是比较简单的, 是通过include /etc/nginx/conf.d/*.conf; 来增加对于不同的域名的支持.

我们随机找一个配置文件来看:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# configuration for default/arya-prod-canary

upstream default-arya-prod-canary-arya.example.com-arya-develop-canary-80 {
	random two least_conn;
	server 172.32.3.87:6000 max_fails=1 fail_timeout=10s;
	server 172.32.3.88:6000 max_fails=1 fail_timeout=10s;
}

server {
	listen 80;
	server_tokens on;

	server_name arya.example.com;

	location / {
		proxy_http_version 1.1;
		proxy_connect_timeout 60s;
		proxy_read_timeout 60s;
		client_max_body_size 1m;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Port $server_port;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_buffering on;
		
		proxy_pass http://default-arya-prod-canary-arya.example.com-arya-develop-canary-80;
	}
}

该配置文件比较中规中矩, 就是普通的Nginx反向代理配置, 通过不同的server_name匹配域名, 并使用proxy_pass转发给后端pod对应的IP.

Kubernetes社区的Nginx配置

我们再来看看Kubernetes社区的Nginx配置, 或者说OpenResty配置, 它的/etc/nginx/nginx.conf文件相当的长, 截取其中的一段说明一下:

下面是arya.example.com的配置, 可以说你在upstream文件中完全看不到pod信息, 所有的请求都由lua来处理了, 不过不用担心, 后面的几个部分将会介绍Lua脚本如何 获得pod以及如何处理请求.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
upstream upstream_balancer {
	### Attention!!!
	#
	# We no longer create "upstream" section for every backend.
	# Backends are handled dynamically using Lua. If you would like to debug
	# and see what backends ingress-nginx has in its memory you can
	# install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin.
	# Once you have the plugin you can use "kubectl ingress-nginx backends" command to
	# inspect current backends.
	#
	###
	
	server 0.0.0.1; # placeholder
	
	balancer_by_lua_block {
		balancer.balance()
	}
	
	keepalive 32;
	
	keepalive_timeout  60s;
	keepalive_requests 100;
	
}

## start server arya.example.com
server {
	server_name arya.example.com ;
	
	listen 80  ;
	listen 443  ssl http2 ;
	
	set $proxy_upstream_name "-";
	
	ssl_certificate_by_lua_block {
		certificate.call()
	}
	
	location / {
		
		set $namespace      "default";
		set $ingress_name   "arya-prod";
		set $service_name   "arya-prod";
		set $service_port   "80";
		set $location_path  "/";
		
		rewrite_by_lua_block {
			lua_ingress.rewrite({
				force_ssl_redirect = false,
				ssl_redirect = true,
				force_no_ssl_redirect = false,
				use_port_in_redirects = false,
			})
			balancer.rewrite()
			plugins.run()
		}
		
		header_filter_by_lua_block {
			plugins.run()
		}
		body_filter_by_lua_block {
			
		}
		
		log_by_lua_block {
			balancer.log()
			monitor.call()
			plugins.run()
		}
		
		port_in_redirect off;
		
		set $balancer_ewma_score -1;
		set $proxy_upstream_name "default-arya-prod-80";
		set $proxy_host          $proxy_upstream_name;
		set $pass_access_scheme  $scheme;
		set $pass_server_port    $server_port;
		set $best_http_host      $http_host;
		set $pass_port           $pass_server_port;
		
		set $proxy_alternative_upstream_name "";
		
		client_max_body_size                    1m;
		proxy_set_header Host                   $best_http_host;
		...
		# mitigate HTTPoxy Vulnerability
		...
		
		proxy_pass http://upstream_balancer;
		
		proxy_redirect                          off;
		
	}
	
}
## end server arya.example.com

K8s社区的Ingress实现是如何读写配置文件的

官方文档How it works中介绍简要介绍了它的实现: Ingress控制器的主要目标是 生成配置文件(Nginx.conf), 这一行为就要求在配置文件改变之后就重载Nginx. 但是 控制器能够做到在只改变upstream时不会重载Nginx配置, 这项功能主要使用了 lua-nginx-module实现.

接下来我会就Ingress的代码分析一下, 是如何做到不重载Nginx而刷新upstream的.

参考代码https://github.com/kubernetes/ingress-nginx/tree/nginx-0.26.1: 版本 0.26.1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// internal/ingress/controller/controller.go
// syncIngress collects all the pieces required to assemble the NGINX
// configuration file and passes the resulting data structures to the backend
// (OnUpdate) when a reload is deemed necessary.
func (n *NGINXController) syncIngress(interface{}) error {
	// ...
	// 获取每个应用的upstream信息
	hosts, servers, pcfg := n.getConfiguration(ings)

	if !n.IsDynamicConfigurationEnough(pcfg) {
		//...
		// 如果不能直接通过lua修改upstream, 需要重新写入Nginx.conf
		err := n.OnUpdate(*pcfg)
		// ...
	}

	err := wait.ExponentialBackoff(retry, func() (bool, error) {
		// 动态刷新upstream
		err := n.configureDynamically(pcfg)
		if err == nil {
			klog.V(2).Infof("Dynamic reconfiguration succeeded.")
			return true, nil
		}
		// ...
	})
	}
	// ...

	n.runningConfig = pcfg

	return nil
}

我们再看下configureDynamically函数做了哪些操作.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// internal/ingress/controller/nginx.go

// configureDynamically encodes new Backends in JSON format and POSTs the
// payload to an internal HTTP endpoint handled by Lua.
func (n *NGINXController) configureDynamically(pcfg *ingress.Configuration) error {
	backendsChanged := !reflect.DeepEqual(n.runningConfig.Backends, pcfg.Backends)
	if backendsChanged {
		// backend发生了改变, 修改backend
		// 此处的backend就是指http中upstream对应的后端的ip与端口
		err := configureBackends(pcfg.Backends)
		if err != nil {
			return err
		}
	}

	streamConfigurationChanged := !reflect.DeepEqual(n.runningConfig.TCPEndpoints, pcfg.TCPEndpoints) || !reflect.DeepEqual(n.runningConfig.UDPEndpoints, pcfg.UDPEndpoints)
	if streamConfigurationChanged {
		// stream发生了改变
		// 此处的stream指的是Nginx中直接反向代理的TCP与UDP服务
		err := updateStreamConfiguration(pcfg.TCPEndpoints, pcfg.UDPEndpoints)
		if err != nil {
			return err
		}
	}
	// ...

	return nil
}

既然后看到这里了, 我们再以backend的更新为例, 检查下它是如何与Nginx交互的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
// internal/ingress/controller/nginx.go
func configureBackends(rawBackends []*ingress.Backend) error {
	backends := make([]*ingress.Backend, len(rawBackends))

	// 初始化backend信息
	for i, backend := range rawBackends {
		var service *apiv1.Service
		if backend.Service != nil {
			service = &apiv1.Service{Spec: backend.Service.Spec}
		}
		luaBackend := &ingress.Backend{
			Name:                 backend.Name,
			Port:                 backend.Port,
			SSLPassthrough:       backend.SSLPassthrough,
			SessionAffinity:      backend.SessionAffinity,
			UpstreamHashBy:       backend.UpstreamHashBy,
			LoadBalancing:        backend.LoadBalancing,
			Service:              service,
			NoServer:             backend.NoServer,
			TrafficShapingPolicy: backend.TrafficShapingPolicy,
			AlternativeBackends:  backend.AlternativeBackends,
		}

		var endpoints []ingress.Endpoint
		for _, endpoint := range backend.Endpoints {
			endpoints = append(endpoints, ingress.Endpoint{
				Address: endpoint.Address,
				Port:    endpoint.Port,
			})
		}

		luaBackend.Endpoints = endpoints
		backends[i] = luaBackend
	}

	// 通过发送post请求给Nginx服务器, 请求Nginx更新backend
	statusCode, _, err := nginx.NewPostStatusRequest("/configuration/backends", "application/json", backends)
	if err != nil {
		return err
	}

	if statusCode != http.StatusCreated {
		return fmt.Errorf("unexpected error code: %d", statusCode)
	}

	return nil
}

// 发送请求的代码比较简单, 就直接贴到下面了
// internal/nginx/main.go
// NewPostStatusRequest creates a new POST request to the internal NGINX status server
func NewPostStatusRequest(path, contentType string, data interface{}) (int, []byte, error) {
	url := fmt.Sprintf("http://127.0.0.1:%v%v", StatusPort, path)

	buf, err := json.Marshal(data)
	if err != nil {
		return 0, nil, err
	}

	client := http.Client{}
	res, err := client.Post(url, contentType, bytes.NewReader(buf))
	if err != nil {
		return 0, nil, err
	}
	defer res.Body.Close()

	body, err := ioutil.ReadAll(res.Body)
	if err != nil {
		return 0, nil, err
	}

	return res.StatusCode, body, nil
}

而OpenResty又是如何处理这个信息呢?

读者可以去OpenResty容器中检查一下Nginx.conf配置文件, 里面会有几个用于Ingress 操作的接口, 我这里就不列Nginx配置了, 直接贴一下相关的Lua文件.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
-- rootfs/etc/nginx/lua/configuration.lua
function _M.call()
  if ngx.var.request_method ~= "POST" and ngx.var.request_method ~= "GET" then
    ngx.status = ngx.HTTP_BAD_REQUEST
    ngx.print("Only POST and GET requests are allowed!")
    return
  end

  if ngx.var.request_uri == "/configuration/servers" then
    handle_servers()
    return
  end

  if ngx.var.request_uri == "/configuration/general" then
    handle_general()
    return
  end

  if ngx.var.uri == "/configuration/certs" then
    handle_certs()
    return
  end

  if ngx.var.request_uri ~= "/configuration/backends" then
    ngx.status = ngx.HTTP_NOT_FOUND
    ngx.print("Not found!")
    return
  end

  -- 这里, 如果是以GET请求发送, 你可以获得当前的backend数据
  if ngx.var.request_method == "GET" then
    ngx.status = ngx.HTTP_OK
    ngx.print(_M.get_backends_data())
    return
  end

  -- 非GET请求会执行下面的语句
  local backends = fetch_request_body()
  if not backends then
    ngx.log(ngx.ERR, "dynamic-configuration: unable to read valid request body")
    ngx.status = ngx.HTTP_BAD_REQUEST
    return
  end

  -- 这里对backend进行修改.
  local success, err = configuration_data:set("backends", backends)
  if not success then
    ngx.log(ngx.ERR, "dynamic-configuration: error updating configuration: " .. tostring(err))
    ngx.status = ngx.HTTP_BAD_REQUEST
    return
  end

  ngx.status = ngx.HTTP_CREATED
end

通过这几段代码的观察, 可以看到Ingress是如何将配置文件刷入到Lua的变量中的, 如果你看的更加仔细, 那么也能了解到何时会更新nginx.conf. 在使用Ingress时, 需要尽量够避免那些操作, 使得Nginx的更新尽可能快.

另外有一点, 当前的部分只讲了Ingress如何刷入upstream配置, 但是没有描述当我们的 请求到来时, 如何读取相关的配置, 并发给后端. 这部分有兴趣的读者自行查看. 你可以 通过这部分代码了解到上一篇所讲的灰度发布(canary)的原理, 以及一些负载均衡策略.

如何调试K8s社区的Ingress实现

上一节我们介绍了K8s社区的Ingress的部分逻辑, 由于这个Ingress的实现并不是直接在配置 文件中写入upstream, 所以我们在调试时, 没法直接cat出文件, 这里我提供两个思路.

进入OpenResty容器查看配置

不知道读者是否忘记, 在上一部分中, 以GET方式请求backend, lua脚本会返回当前的配置信息, 这个配置信息里面肯定是包括后端的ip以及端口的.

1
curl 127.0.0.1:10246/configuration/backends

kubectl插件使用

这个插件我最近搜索找到的, 你可以通过它来读取Ingress配置, 算是比较通用的一种方法.

https://kubernetes.github.io/ingress-nginx/kubectl-plugin/

1
kubectl ingress-nginx backends -n ingress-nginx

总结

这篇博客中, 主要介绍了两种Ingress的实现, 并且就K8s社区的Ingress实现做了详细的说明, 基本将更新backend的代码呈现给了大家. 当然谁也无法保证未来的Ingress会不会有大的变化, 就目前来看, lua脚本实现的功能已经相当多了, 我的使用方向应该是在此基础上增加一些 访问策略的限制, 例如频率限制或是ip封禁.