Consulを再度学んだ

memo

consulを少し触ったままだったので、ちゃんと学んでみた。
と言っても一通り触ったくらいであるが。

あらためて試すにあたり、Official Consul Docker Imageを使用してクラスタを構成した。

counsulのおさらい

consulをざっくりとおさらい。
複数のサーバーをクラスタ化して、オーケストレーションするツール。
クラスタ内のサーバー同士をDNSインターフェイスを通してIPベース相互検出し(Discovery)、健全性を確認し(Health Check)、特定のサーバーへのトラフィックを回避する(Load Balancing)を行う。

RESTベースの外部API(HTTP API)を提供しており、サーバー情報はキーバリュースストアで保持されている。

dockerでクラスタ構成を作る

consulはサーバとクライアントという2つのコンポーネントで構成され、いずれかの状態で起動する必要がある。
クライアントモードでは各サーバーと通信をする。
サーバーモードでは奇数台で構成されることを期待されていて、それぞれがGossip Protocolで通信し、Raftアルゴリズムに基づいてリーダーを選出する。

今回はdockerを使って以下のような構成を使っておさらいする

1
2
3
4
5
6
7
8
9
10
# server
consul --
         `- consul1 (leader) 172.16.238.10
         `- consul2          172.16.238.11
         `- consul3          172.16.238.12
# agent
nginx  --
         `- nginx1           172.16.238.13
         `- nginx2           172.16.238.14
         `- nginx3           172.16.238.15

ということでserver 3コンテナを起動してみる。

ちなみにconsulでは次のポートをデフォルトで使用する。

機能 protocol port 内容
Server RPC TCP 8300 他のAgentからのリクエストを受付る
Serf LAN TCP and UDP 8301 LAN上のゴシッププロトコル。すべてのAgentが使用する。
Serf WAN TCP and UDP 8302 WAN上のゴシッププロトコル。他のサーバーとの通信に使用。
CLI RPC TCP 8400 CLIからの入力を処理する。すべてのAgentが使用する。
HTTP API TCP 8500 クライアントがHTTP API を受付る
DNS Interface TCP and UDP 8600 DNSクエリを解決する

今回は外からも利用するので一つだけ書くポートをフォワードしておく。

リーダー選出

指定数以上のサーバーが起動したことで、リーダーが選出されていることを確認する。

1
2
3
4
5
6
7
8
9
10
11
# リーダーのnodeの情報取得
curl -s http://localhost:8500/v1/status/leader | jq .
"172.16.238.10:8300"

# agentのnode情報取得
curl -s http://localhost:8500/v1/status/peers | jq .
[
  "172.16.238.10:8300",
  "172.16.238.12:8300",
  "172.16.238.11:8300"
]

agentの追加

3台起動してるnginxコンテナをagentモードでクラスタに追加する。
(追加時に合わせて幾つか死活監視を試せるようにしている)

起動しているコンテナは以下の状態

1
2
3
4
5
6
7
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                                                                                                                                       NAMES
c48eb5e1c99a        learnconsul_nginx2   "nginx -g 'daemon ..."   24 hours ago        Up 24 hours         443/tcp, 0.0.0.0:8000->80/tcp                                                                                                               nginx2
1bd432bd0b98        consul:latest        "docker-entrypoint..."   24 hours ago        Up 24 hours         8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp                                                                        consul2
7917ff526a10        learnconsul_nginx1   "nginx -g 'daemon ..."   24 hours ago        Up 24 hours         443/tcp, 0.0.0.0:8080->80/tcp                                                                                                               nginx1
f72330ed0182        consul:latest        "docker-entrypoint..."   24 hours ago        Up 24 hours         8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp                                                                        consul3
5c40e078481e        learnconsul_nginx3   "nginx -g 'daemon ..."   24 hours ago        Up 24 hours         443/tcp, 0.0.0.0:8888->80/tcp                                                                                                               nginx3
f7270595eda4        consul:latest        "docker-entrypoint..."   24 hours ago        Up 24 hours         0.0.0.0:8300-8302->8300-8302/tcp, 0.0.0.0:8400->8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 8600/udp, 8600/tcp, 0.0.0.0:8600->53/udp   consul1

nodeを追加していく

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# configファイルをコンテナにコピー
docker cp config/nginx1 nginx1:/etc/consul.d
# clusterにjoinする
docker exec -d -t nginx1 consul agent -config-file=/etc/consul.d/nginx1
# ttl health check用のスクリプトを起動
docker exec -d -t nginx1 bash -c './http_request nginx1'
 
consul1    |     2017/01/25 07:50:51 [INFO] serf: EventMemberJoin: nginx1 172.16.238.13
consul1    |     2017/01/25 07:50:51 [INFO] consul: member 'nginx1' joined, marking health alive
consul2    |     2017/01/25 07:50:51 [INFO] serf: EventMemberJoin: nginx1 172.16.238.13
consul3    |     2017/01/25 07:50:51 [INFO] serf: EventMemberJoin: nginx1 172.16.238.13
 
# 以下nginx1と同じ処理を行う
 
# nginx2
docker cp config/nginx2 nginx2:/etc/consul.d
docker exec -d -t nginx2 consul agent -config-file=/etc/consul.d/nginx2
docker exec -d -t nginx2 bash -c './http_request nginx2'
 
consul1    |     2017/01/25 07:51:31 [INFO] serf: EventMemberJoin: nginx2 172.16.238.14
consul1    |     2017/01/25 07:51:31 [INFO] consul: member 'nginx2' joined, marking health alive
consul2    |     2017/01/25 07:51:31 [INFO] serf: EventMemberJoin: nginx2 172.16.238.14
consul3    |     2017/01/25 07:51:31 [INFO] serf: EventMemberJoin: nginx2 172.16.238.14
 
# nginx3
docker cp config/nginx3 nginx3:/etc/consul.d
docker exec -d -t nginx3 consul agent -config-file=/etc/consul.d/nginx3
docker exec -d -t nginx3 bash -c './http_request nginx3'
 
consul1    |     2017/01/25 07:52:06 [INFO] serf: EventMemberJoin: nginx3 172.16.238.15
consul1    |     2017/01/25 07:52:06 [INFO] consul: member 'nginx3' joined, marking health alive
consul2    |     2017/01/25 07:52:06 [INFO] serf: EventMemberJoin: nginx3 172.16.238.15
consul3    |     2017/01/25 07:52:06 [INFO] serf: EventMemberJoin: nginx3 172.16.238.15

追加されていることをapiを通じて確認する。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
# ローカルエージェント(consul1)で既知のノードを返します
curl -s http://localhost:8500/v1/catalog/nodes | jq .
[
  {
    "ID": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
    "Node": "consul1",
    "Address": "172.16.238.10",
    "TaggedAddresses": {
      "lan": "172.16.238.10",
      "wan": "172.16.238.10"
    },
    "Meta": {},
    "CreateIndex": 5,
    "ModifyIndex": 9
  },
  {
    "ID": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
    "Node": "consul2",
    "Address": "172.16.238.11",
    "TaggedAddresses": {
      "lan": "172.16.238.11",
      "wan": "172.16.238.11"
    },
    "Meta": {},
    "CreateIndex": 4,
    "ModifyIndex": 20
  },
  {
    "ID": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
    "Node": "consul3",
    "Address": "172.16.238.12",
    "TaggedAddresses": {
      "lan": "172.16.238.12",
      "wan": "172.16.238.12"
    },
    "Meta": {},
    "CreateIndex": 6,
    "ModifyIndex": 8
  },
  {
    "ID": "",
    "Node": "nginx1",
    "Address": "172.16.238.13",
    "TaggedAddresses": {
      "lan": "172.16.238.13",
      "wan": "172.16.238.13"
    },
    "Meta": null,
    "CreateIndex": 28,
    "ModifyIndex": 29
  },
  {
    "ID": "",
    "Node": "nginx2",
    "Address": "172.16.238.14",
    "TaggedAddresses": {
      "lan": "172.16.238.14",
      "wan": "172.16.238.14"
    },
    "Meta": null,
    "CreateIndex": 119,
    "ModifyIndex": 120
  },
  {
    "ID": "",
    "Node": "nginx3",
    "Address": "172.16.238.15",
    "TaggedAddresses": {
      "lan": "172.16.238.15",
      "wan": "172.16.238.15"
    },
    "Meta": null,
    "CreateIndex": 132,
    "ModifyIndex": 134
  }
]
 
# ローカルエージェント(consul1)で登録された全ての service を返す
curl -s http://localhost:8500/v1/agent/services | jq .

{
  "consul": {
    "ID": "consul",
    "Service": "consul",
    "Tags": [],
    "Address": "",
    "Port": 8300,
    "EnableTagOverride": false,
    "CreateIndex": 0,
    "ModifyIndex": 0
  }
}
 
# ローカルエージェント(consul1) のserf エージェントが見えているメンバを返す
curl -s http://localhost:8500/v1/agent/members | jq .
[
  {
    "Name": "nginx1",
    "Addr": "172.16.238.13",
    "Port": 8301,
    "Tags": {
      "build": "0.7.2:'a9afa0c",
      "dc": "dc1",
      "role": "node",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 4,
    "DelegateCur": 4
  },
  {
    "Name": "nginx2",
    "Addr": "172.16.238.14",
    "Port": 8301,
    "Tags": {
      "build": "0.7.2:'a9afa0c",
      "dc": "dc1",
      "role": "node",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 4,
    "DelegateCur": 4
  },
  {
    "Name": "nginx3",
    "Addr": "172.16.238.15",
    "Port": 8301,
    "Tags": {
      "build": "0.7.2:'a9afa0c",
      "dc": "dc1",
      "role": "node",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 4,
    "DelegateCur": 4
  },
  {
    "Name": "consul1",
    "Addr": "172.16.238.10",
    "Port": 8301,
    "Tags": {
      "build": "0.7.5:'21f2d5a",
      "dc": "dc1",
      "expect": "3",
      "id": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
      "port": "8300",
      "role": "consul",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 5,
    "DelegateCur": 4
  },
  {
    "Name": "consul3",
    "Addr": "172.16.238.12",
    "Port": 8301,
    "Tags": {
      "build": "0.7.5:'21f2d5a",
      "dc": "dc1",
      "id": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
      "port": "8300",
      "role": "consul",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 5,
    "DelegateCur": 4
  },
  {
    "Name": "consul2",
    "Addr": "172.16.238.11",
    "Port": 8301,
    "Tags": {
      "build": "0.7.5:'21f2d5a",
      "dc": "dc1",
      "id": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
      "port": "8300",
      "role": "consul",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 5,
    "DelegateCur": 4
  }
]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
# service consulのhealth check
curl -s http://localhost:8500/v1/health/service/consul | jq .
[
  {
    "Node": {
      "ID": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
      "Node": "consul1",
      "Address": "172.16.238.10",
      "TaggedAddresses": {
        "lan": "172.16.238.10",
        "wan": "172.16.238.10"
      },
      "Meta": {},
      "CreateIndex": 5,
      "ModifyIndex": 9
    },
    "Service": {
      "ID": "consul",
      "Service": "consul",
      "Tags": [],
      "Address": "",
      "Port": 8300,
      "EnableTagOverride": false,
      "CreateIndex": 5,
      "ModifyIndex": 9
    },
    "Checks": [
      {
        "Node": "consul1",
        "CheckID": "serfHealth",
        "Name": "Serf Health Status",
        "Status": "passing",
        "Notes": "",
        "Output": "Agent alive and reachable",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 5,
        "ModifyIndex": 5
      }
    ]
  },
  {
    "Node": {
      "ID": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
      "Node": "consul2",
      "Address": "172.16.238.11",
      "TaggedAddresses": {
        "lan": "172.16.238.11",
        "wan": "172.16.238.11"
      },
      "Meta": {},
      "CreateIndex": 4,
      "ModifyIndex": 35
    },
    "Service": {
      "ID": "consul",
      "Service": "consul",
      "Tags": [],
      "Address": "",
      "Port": 8300,
      "EnableTagOverride": false,
      "CreateIndex": 4,
      "ModifyIndex": 35
    },
    "Checks": [
      {
        "Node": "consul2",
        "CheckID": "serfHealth",
        "Name": "Serf Health Status",
        "Status": "passing",
        "Notes": "",
        "Output": "Agent alive and reachable",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 8,
        "ModifyIndex": 8
      }
    ]
  },
  {
    "Node": {
      "ID": "e8ded508-98bb-4f9b-b0a2-ea872a0361d6",
      "Node": "consul3",
      "Address": "172.16.238.12",
      "TaggedAddresses": {
        "lan": "172.16.238.12",
        "wan": "172.16.238.12"
      },
      "Meta": {},
      "CreateIndex": 6,
      "ModifyIndex": 32
    },
    "Service": {
      "ID": "consul",
      "Service": "consul",
      "Tags": [],
      "Address": "",
      "Port": 8300,
      "EnableTagOverride": false,
      "CreateIndex": 6,
      "ModifyIndex": 32
    },
    "Checks": [
      {
        "Node": "consul3",
        "CheckID": "serfHealth",
        "Name": "Serf Health Status",
        "Status": "passing",
        "Notes": "",
        "Output": "Agent alive and reachable",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 7,
        "ModifyIndex": 7
      }
    ]
  }
]

health check(死活監視)について

consulのhealth checkには次の種類のものがある。

check 内容
Script + Interval Exit code 0でpassing、Exit code 1でWarning、それ以外でcriticalとなるスクリプトを実行する
HTTP + Interval curlなどでのHTTPリクエスト。2xxステータスでpassing。429 Too Many Requestsでwarning。それ以外はcritical。
TCP + Interval 指定ホストにTCPで接続を試みる。接続成功でpassing、それ以外はciritcalとなる。
Time to Live (TTL) アプリケーションが自身の状態をチェックして通知する。通知状態はTTLの間保持される。
Docker + Interval DockerコンテナにHTTPまたはSocket通信し、適切な終了コードかをチェックする

今回は以下の機能を試す。

  • HTTP + Interval
  • TCP + Interval
  • Time to Live (TTL)

ちなみにTime to Liveはちょっとわかりにくいが、自身のconsul http apiを実行する。
例えばpass状態を送る場合は http://localhost:8500/v1/agent/check/pass/{CHECK_ID} にリクエストする。

今回のhealth checkの設定は以下のようにしている。
(すでにclusterにjoinするときに設定は行っている)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// nginx1の場合

{
"checks": [
    {
      "name": "HTTP Nginx on port 80",
      "http": "http://localhost:80",
      "interval": "10s",
      "timeout": "1s"
    },
    {
      "id": "nginx1",
      "name": "Nginx Status",
      "ttl": "30s"
    },
    {
      "name": "SSH TCP on port 80",
      "tcp": "172.16.238.13:80",
      "interval": "10s",
      "timeout": "1s"
    }
  ]
}

service(nginx1、nginx2、nginx3)ごとのstatusを取得してみる。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
# service nginxのhealth check
curl -s http://localhost:8500/v1/health/service/proxy | jq .
[
  {
    "Node": {
      "ID": "",
      "Node": "nginx1",
      "Address": "172.16.238.13",
      "TaggedAddresses": {
        "lan": "172.16.238.13",
        "wan": "172.16.238.13"
      },
      "Meta": null,
      "CreateIndex": 12,
      "ModifyIndex": 13
    },
    "Service": {
      "ID": "proxy",
      "Service": "proxy",
      "Tags": [
        "nginx"
      ],
      "Address": "172.16.238.13",
      "Port": 80,
      "EnableTagOverride": false,
      "CreateIndex": 13,
      "ModifyIndex": 13
    },
    "Checks": [
      {
        "Node": "nginx1",
        "CheckID": "HTTP Nginx on port 80",
        "Name": "HTTP Nginx on port 80",
        "Status": "passing",
        "Notes": "",
        "Output": "HTTP GET http://172.16.238.13: 200 OK Output: <!DOCTYPE html>\n<html>\n<head>\n<title>Welcome to nginx!</title>\n<style>\n    body {\n        width: 35em;\n        margin: 0 auto;\n        font-family: Tahoma, Verdana, Arial, sans-serif;\n    }\n</style>\n</head>\n<body>\n<h1>Welcome to nginx!</h1>\n<p>If you see this page, the nginx web server is successfully installed and\nworking. Further configuration is required.</p>\n\n<p>For online documentation and support please refer to\n<a href=\"http://nginx.org/\">nginx.org</a>.<br/>\nCommercial support is available at\n<a href=\"http://nginx.com/\">nginx.com</a>.</p>\n\n<p><em>Thank you for using nginx.</em></p>\n</body>\n</html>\n",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 14,
        "ModifyIndex": 17
      },
      {
        "Node": "nginx1",
        "CheckID": "nginx1",
        "Name": "Nginx Status",
        "Status": "passing",
        "Notes": "",
        "Output": "",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 15,
        "ModifyIndex": 227
      },
      {
        "Node": "nginx1",
        "CheckID": "serfHealth",
        "Name": "Serf Health Status",
        "Status": "passing",
        "Notes": "",
        "Output": "Agent alive and reachable",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 12,
        "ModifyIndex": 12
      },
      {
        "Node": "nginx1",
        "CheckID": "SSH TCP on port 80",
        "Name": "SSH TCP on port 80",
        "Status": "passing",
        "Notes": "",
        "Output": "TCP connect 172.16.238.13:80: Success",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 16,
        "ModifyIndex": 18
      }
    ]
  },
  {
    "Node": {
      "ID": "",
      "Node": "nginx2",
      "Address": "172.16.238.14",
      "TaggedAddresses": {
        "lan": "172.16.238.14",
        "wan": "172.16.238.14"
      },
      "Meta": null,
      "CreateIndex": 22,
      "ModifyIndex": 25
    },
    "Service": {
      "ID": "proxy",
      "Service": "proxy",
      "Tags": [
        "nginx"
      ],
      "Address": "172.16.238.14",
      "Port": 80,
      "EnableTagOverride": false,
      "CreateIndex": 25,
      "ModifyIndex": 25
    },
    "Checks": [
      {
        "Node": "nginx2",
        "CheckID": "HTTP Nginx on port 80",
        "Name": "HTTP Nginx on port 80",
        "Status": "passing",
        "Notes": "",
        "Output": "HTTP GET http://172.16.238.14: 200 OK Output: <!DOCTYPE html>\n<html>\n<head>\n<title>Welcome to nginx!</title>\n<style>\n    body {\n        width: 35em;\n        margin: 0 auto;\n        font-family: Tahoma, Verdana, Arial, sans-serif;\n    }\n</style>\n</head>\n<body>\n<h1>Welcome to nginx!</h1>\n<p>If you see this page, the nginx web server is successfully installed and\nworking. Further configuration is required.</p>\n\n<p>For online documentation and support please refer to\n<a href=\"http://nginx.org/\">nginx.org</a>.<br/>\nCommercial support is available at\n<a href=\"http://nginx.com/\">nginx.com</a>.</p>\n\n<p><em>Thank you for using nginx.</em></p>\n</body>\n</html>\n",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 26,
        "ModifyIndex": 30
      },
      {
        "Node": "nginx2",
        "CheckID": "nginx2",
        "Name": "Nginx Status",
        "Status": "passing",
        "Notes": "",
        "Output": "",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 27,
        "ModifyIndex": 223
      },
      {
        "Node": "nginx2",
        "CheckID": "serfHealth",
        "Name": "Serf Health Status",
        "Status": "passing",
        "Notes": "",
        "Output": "Agent alive and reachable",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 22,
        "ModifyIndex": 22
      },
      {
        "Node": "nginx2",
        "CheckID": "SSH TCP on port 80",
        "Name": "SSH TCP on port 80",
        "Status": "passing",
        "Notes": "",
        "Output": "TCP connect 172.16.238.14:80: Success",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 28,
        "ModifyIndex": 29
      }
    ]
  },
  {
    "Node": {
      "ID": "",
      "Node": "nginx3",
      "Address": "172.16.238.15",
      "TaggedAddresses": {
        "lan": "172.16.238.15",
        "wan": "172.16.238.15"
      },
      "Meta": null,
      "CreateIndex": 37,
      "ModifyIndex": 38
    },
    "Service": {
      "ID": "proxy",
      "Service": "proxy",
      "Tags": [
        "nginx"
      ],
      "Address": "172.16.238.15",
      "Port": 80,
      "EnableTagOverride": false,
      "CreateIndex": 38,
      "ModifyIndex": 38
    },
    "Checks": [
      {
        "Node": "nginx3",
        "CheckID": "HTTP Nginx on port 80",
        "Name": "HTTP Nginx on port 80",
        "Status": "passing",
        "Notes": "",
        "Output": "HTTP GET http://172.16.238.15: 200 OK Output: <!DOCTYPE html>\n<html>\n<head>\n<title>Welcome to nginx!</title>\n<style>\n    body {\n        width: 35em;\n        margin: 0 auto;\n        font-family: Tahoma, Verdana, Arial, sans-serif;\n    }\n</style>\n</head>\n<body>\n<h1>Welcome to nginx!</h1>\n<p>If you see this page, the nginx web server is successfully installed and\nworking. Further configuration is required.</p>\n\n<p>For online documentation and support please refer to\n<a href=\"http://nginx.org/\">nginx.org</a>.<br/>\nCommercial support is available at\n<a href=\"http://nginx.com/\">nginx.com</a>.</p>\n\n<p><em>Thank you for using nginx.</em></p>\n</body>\n</html>\n",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 41,
        "ModifyIndex": 42
      },
      {
        "Node": "nginx3",
        "CheckID": "nginx3",
        "Name": "Nginx Status",
        "Status": "passing",
        "Notes": "",
        "Output": "",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 39,
        "ModifyIndex": 228
      },
      {
        "Node": "nginx3",
        "CheckID": "serfHealth",
        "Name": "Serf Health Status",
        "Status": "passing",
        "Notes": "",
        "Output": "Agent alive and reachable",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 37,
        "ModifyIndex": 37
      },
      {
        "Node": "nginx3",
        "CheckID": "SSH TCP on port 80",
        "Name": "SSH TCP on port 80",
        "Status": "passing",
        "Notes": "",
        "Output": "TCP connect 172.16.238.14:80: Success",
        "ServiceID": "",
        "ServiceName": "",
        "CreateIndex": 40,
        "ModifyIndex": 44
      }
    ]
  }
]

consul serverのnodeを一つダウンしてみる。

1
2
docker stop consul3
consul3

consul3の状態を確認する

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
curl -s http://localhost:8500/v1/health/node/consul3 | jq .
[
  {
    "Node": "consul3",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "critical",
    "Notes": "",
    "Output": "Agent not live or unreachable",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 7,
    "ModifyIndex": 748
  }
]

consul3 を復帰させる

1
2
docker start consul3
consul3

再び状態を確認してみる。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
curl -s http://localhost:8500/v1/health/node/consul3 | jq .
[
  {
    "Node": "consul3",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "passing",
    "Notes": "",
    "Output": "Agent alive and reachable",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 7,
    "ModifyIndex": 800
  }
]

nginx1のnodeのhttpチェックをcriticalな状態にしてみる。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
docker exec nginx1 mv /usr/share/nginx/html/index.html /usr/share/nginx/html/index.txt
curl -s http://localhost:8500/v1/health/node/nginx1 | jq .
[
  {
    "Node": "nginx1",
    "CheckID": "HTTP Nginx on port 80",
    "Name": "HTTP Nginx on port 80",
    "Status": "critical",
    "Notes": "",
    "Output": "HTTP GET http://localhost:80: 403 Forbidden Output: <html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.11.10</center>\r\n</body>\r\n</html>\r\n",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 614,
    "ModifyIndex": 803
  },
  {
    "Node": "nginx1",
    "CheckID": "nginx1",
    "Name": "Nginx Status",
    "Status": "passing",
    "Notes": "",
    "Output": "",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 612,
    "ModifyIndex": 806
  },
  {
    "Node": "nginx1",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "passing",
    "Notes": "",
    "Output": "Agent alive and reachable",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 610,
    "ModifyIndex": 610
  },
  {
    "Node": "nginx1",
    "CheckID": "SSH TCP on port 80",
    "Name": "SSH TCP on port 80",
    "Status": "passing",
    "Notes": "",
    "Output": "TCP connect 172.16.238.13:80: Success",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 613,
    "ModifyIndex": 613
  }
]

apiからciriticalなものだけ取得してみる。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
curl -s http://localhost:8500/v1/health/state/critical | jq .
[
  {
    "Node": "nginx1",
    "CheckID": "HTTP Nginx on port 80",
    "Name": "HTTP Nginx on port 80",
    "Status": "critical",
    "Notes": "",
    "Output": "HTTP GET http://localhost:80: 403 Forbidden Output: <html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.11.10</center>\r\n</body>\r\n</html>\r\n",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 614,
    "ModifyIndex": 803
  }
]

メンテナンスモード

当たり前ではなるが、nodeをメンテナス状態にできる。
nginx1をメンテナンスモードにする。

1
2
3
4
5
6
7
8
9
10
11
docker exec -it nginx1 /bin/bash

# maintenance modeをenableにして、理由にtestとする
consul maint -reason test -enable
Node maintenance is now enabled

# maintenance の状態を確認
consul maint
Node:
  Name:   nginx1
  Reason: test

nginx1の状態を確認する。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
curl -s http://localhost:8500/v1/health/node/nginx1 | jq .
[
  {
    "Node": "nginx1",
    "CheckID": "_node_maintenance",
    "Name": "Node Maintenance Mode",
    "Status": "critical",
    "Notes": "test",
    "Output": "",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 894,
    "ModifyIndex": 894
  },
  {
    "Node": "nginx1",
    "CheckID": "HTTP Nginx on port 80",
    "Name": "HTTP Nginx on port 80",
    "Status": "passing",
    "Notes": "",
    "Output": "HTTP GET http://localhost:80: 200 OK Output: ",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 614,
    "ModifyIndex": 869
  },
  {
    "Node": "nginx1",
    "CheckID": "nginx1",
    "Name": "Nginx Status",
    "Status": "passing",
    "Notes": "",
    "Output": "",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 612,
    "ModifyIndex": 928
  },
  {
    "Node": "nginx1",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "passing",
    "Notes": "",
    "Output": "Agent alive and reachable",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 610,
    "ModifyIndex": 610
  },
  {
    "Node": "nginx1",
    "CheckID": "SSH TCP on port 80",
    "Name": "SSH TCP on port 80",
    "Status": "passing",
    "Notes": "",
    "Output": "TCP connect 172.16.238.13:80: Success",
    "ServiceID": "",
    "ServiceName": "",
    "CreateIndex": 613,
    "ModifyIndex": 613
  }
]

メンテナンスモードを解除する。

1
2
3
4
5
6
# maintenance modeをdisableにする
consul maint -disable
Node maintenance is now disabled

consul maint
# なし

event処理

イベントを監視して、発火時に任意の処理を実行することができる。

1
2
3
4
5
6
7
8
9
# testイベントをwatchする
docker exec -it consule1 /bin/sh
consul watch -type=event -name=test echo "test"

# testイベントを発行
docker exec -it nginx1 /bin/bash

# consul1
test

watchコマンドで状態が変化した際に特定の処理をしてみる。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
docker exec -it consule1 /bin/sh
consul watch -type=checks -state=critical "jq ."
[]
 
# nginx1を落とす
docker stop nginx1
 
# criticalが表示される
[
  {
    "Node": "nginx1",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "critical",
    "Notes": "",
    "Output": "Agent not live or unreachable",
    "ServiceID": "",
    "ServiceName": ""
  }
]

consul-templateを試してみる

consul-templateも試しておく。
consul-templateは特定のイベントに応じて、登録しているtemplateからファイルを生成することができる。
イベントを監視して、特定のイベントが発火すると登録しているtemplateからファイルを再生成する感じである。

consul-templateで監視できるイベントは

  • datacenter(detacenterの)
  • file(fileの変更)
  • key(ストアのkey変更・存在)
  • ls(kvストアの最上位のkey)
  • node
  • secret
  • service
  • tree(kvストアのkvペア)

例えば、次のようなnodeが追加されるたびにhost名とipを記載したファイルが出力されるもので試してみる。

1
2
3
4
{{ range nodes }}
# {{.Node}}
{{ .Address }} {{.Node}}
{{ end }}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
docker exec -it nginx1 /bin/bash
 
# -dry でdry run
consul-template -consul-addr 127.0.0.1:8500 -template "/etc/consul-template/config.d/templates/hosts.ctmpl:/hosts" -dry
 
> /hosts
 
# consul1
172.16.238.10 consul1
 
# consul2
172.16.238.11 consul2
 
# consul3
172.16.238.12 consul3
 
# nginx1
172.16.238.13 nginx1

node nginx2を追加する。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
> /hosts
 
# consul1
172.16.238.10 consul1
 
# consul2
172.16.238.11 consul2
 
# consul3
172.16.238.12 consul3
 
# nginx1
172.16.238.13 nginx1
 
# nginx2
172.16.238.14 nginx2

テンプレートが更新されている。

まとめ

おさらいとして、今回はdockerの公式イメージでクラスタを構成してみた。
consulによるクラスタ構成を以前より意識できた感がある。

repository

docker-consul-example

Comments