Translated from 日本語 • View original Hello. The title is long, but I don't know if there is demand because it has been successful for a long time, but I will record it.
Please do it when you have time.
The purpose is to external ELB→nginx (SSL termination and client authentication) → internal ELB→app
In other words, I want to authenticate the client, but ELB does not have that function, so I forward tcp443 and do it with nginx, but since the source IP that can be taken by the higher server will be the IP of ELB, I want to enable ProxyProtocol in ELB and listen to ProxyProtocol and log the source IP while ssl termination with nginx.
Omit parts such as securityGroup and creating an instance.
1.Creating an elb in awscli, configuring proxy-protocol
Reference:
Enable or Disable Proxy Protocol Support - Elastic Load Balancing Now Try ELB Across Multiple AZs in a VPC (2) - Using awscli - Yohei no Days Devotion XP
- ELB creation
profile=xxxxx
elbname-ext=xxxx-elb
securitygrops="sg-xxxxxxxx sg-yyyyyyyy"
subnets="subnet-xxxxxxxx subnet-yyyyyyyy"
sudo bash -c "aws elb create-load-balancer --load-balancer-name ${elbname-ext} --listeners Protocol=TCP,LoadBalancerPort=443,InstanceProtocol=TCP,InstancePort=443 --subnets ${ subnets} --security-groups ${securitygrops} --profile ${profile}"
*The external ELB is set to only transfer from TCP443 to TCP443 and do not include a certificate.
elbname-int=yyyy-elb
securitygrops="sg-xxxxxxxx sg-zzzzzzzz"
sudo bash -c "aws elb create-load-balancer --load-balancer-name ${elbname-int} --listeners Protocol=TCP,LoadBalancerPort=80,InstanceProtocol=TCP,InstancePort=8xxx --subnets ${ subnets} --security-groups ${securitygrops} --scheme internal --profile ${profile}"
・helthcheck settings
sudo bash -c "aws elb configure-health-check --load-balancer-name ${elbname-ext} --health-check Target="TCP:443",Interval=30,Timeout=5,UnhealthyThreshold=2,HealthyThreshold=10 -- profile ${profile}"
sudo bash -c "aws elb configure-health-check --load-balancer-name ${elbname-int} --health-check Target="TCP:80",Interval=30,Timeout=5,UnhealthyThreshold=2,HealthyThreshold=10 -- profile ${profile}"
Instance Registration
instances-ext="i-xxxxxxxx i-yyyyyyyyy"
instances-int="i-zzzzzzzz i-aaaaaaaaa"
sudo bash -c "aws elb register-instances-with-load-balancer --load-balancer-name ${elbname-ext} --instances ${instances-ext} --profile ${profile}"
sudo bash -c "aws elb register-instances-with-load-balancer --load-balancer-name ${elbname-int} --instances ${instances-int} --profile ${profile}"
Proxy protocol activation
Create a policy
sudo bash -c "aws elb create-load-balancer-policy --load-balancer-name ${elbname-ext} --policy-name EnableProxyProtocol --policy-type-name ProxyProtocolPolicyType -- policy-attributes AttributeName=ProxyProtocol,AttributeValue=true --profile ${profile}"
Review your policy
sudo bash -c "aws elb set-load-balancer-policies-for-backend-server --load-balancer-name ${elbname-ext} --instance-port 443 --policy-names EnableProxyProtocol --profile ${profile} "
If you want to disable it, you can specify an empty specification of '--policy-names []' to disable it
Ensure Proxy Protocols Are Enabled
sudo bash -c "aws elb describe-load-balancers --load-balancer-name ${elbname-ext} --profile ${profile}"
"BackendServerDescriptions": [
{
"InstancePort": 443,
"PolicyNames": [
"EnableProxyProtocol"
]
}
2. About Nginx Configuration
I will omit the details and only add the relevant settings. The 'proxy-protocol' setting needs to be 'restarted' instead of 'reload', so you need to perform a 'restart' of nginx when confirming your login.
Reference:
Using Proxy Protocol With Nginx | chris lea Module ngx_http_realip_module #355 (Proxy Protocol Support) - nginx
Actual nginx configuration file
(The version at the time of success is 1.7.4-1)
/etc/nginx/nginx.conf
-----------------------------------------------
log_format main '$proxy_protocol_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"'
'$http_x_userid - $http_x_signature - $http_x_sessionkey "$request_body"';
-----------------------------------------------
*Changed to '$proxy_protocol_addr' instead of '$remote_addr' *Excerpts only where relevant are (copy and paste from 'mastering Nginx' such as http_x_sessionkey, etc., but please remove it if it is unnecessary)
/etc/nginx/conf.d/ssl.conf
-----------------------------------------------
server {
# listen 443 default ssl;
listen 443 default ssl proxy_protocol;
server_name <%= node['nginx']['servername1'] %>;
set_real_ip_from <%= node['nginx']['set_real_ip_from'] %>;
real_ip_header proxy_protocol;
ssl_certificate /etc/nginx/<%= node['nginx']['sslcrt1'] %>;
ssl_certificate_key /etc/nginx/<%= node['nginx']['sslkey1'] %>;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_session_cache shared:WEB:10m;
ssl_session_timeout 10m;
ssl_ciphers RC4:HIGH:!aNULL:! MD5:@STRENGTH;
## client-auth configuration
ssl_verify_client on;
ssl_verify_depth 3;
ssl_client_certificate /etc/nginx/<%= node['nginx']['clientcrt'] %>;
# ssl_crl /etc/nginx/<%= node['nginx']['clientcrl'] %>;
resolver <%= node['nginx']['resolver'] %> valid=5s;
resolver_timeout 3s;
location / {
proxy_set_header X-FORWARDED-PROTO https;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_connect_timeout 10s;
proxy_read_timeout 10s;
proxy_pass http://<%= @upstream1 %>:<%= @upstreamport %>$request_uri;
## Pass client-cert to backend-server.
# proxy_set_header ssl_client_cert $ssl_client_cert;
# root /usr/share/nginx/html;
# index index.html index.htm;
}
}
-----------------------------------------------
*Among the above, the directives related to 'ProxyProtocol' are as follows: 'listen', 'set_real_ip_from', 'real_ip_header'.
※<%= .. Please note that the part enclosed in %> is an attribute (variable) and is replaced with the value set when the chef recipe was executed, and is different from the actual value. @の変数はroleとenvironmentで吸収しきれずdata_bags pulling from the table. *'set_real_ip_from' specifies the segment address of the VPC (e.g., 10.0.0.0/17). I need to write down the actual address of the ELB, but I can't identify it and it will be automatically shaken. *'ssl_crl' (list of proof of destruction) comments are subject to indefinite circumstances.
**The version that can interpret 'ProxyProtocol' is 1.5.12 or higher. Below that is an error with 'configtest'. **There is no need to build from the source with '--with-proxy-protocol' on the 1.4 system, and although it is 1.7, I was able to realize it using a bare package without building it. *I stopped upstream and added resolver. The name solution of nginx is solved every time you put it in a variable in the mastering Nginx P94 set set, and it seems that you will go to listen to it at the timing of the ttl specified in the varid of the resolver. It seems that the resolver is only needed to resolve the names specified in upstream and *_pass the first time.
yum repository for the 1.7 series of nginx (mainline one)
The contents are as follows. From the chef point of view, you need to write 'options' "--disablerepo=epel"' to prevent it from entering from the epel in the package resource, etc.
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/mainline/centos/6/$basearch/
gpgcheck=0
enable=1
3. Check the Source IP of Access Logs
Login
ssh nginx-srv1-dev
ssh nginx-srv2-dev
Confirmation
rpm -qa|grep nginx
sudo service nginx configtest
sudo service nginx restart
sudo service nginx status
sudo netstat -lnptu
chkconfig --list nginx
If it becomes InService after adding it to the elb, access it from the browser and check the logs
$ sudo tail -f /var/log/nginx/access.log
119.xxx.xxx.xxx - - [12/Aug/2014:04:39:28 +0000] "GET / HTTP/1.1" 400 648 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" "-"- - - - - "-"
119.xxx.xxx.xxx - - [12/Aug/2014:04:39:28 +0000] "GET /favicon.ico HTTP/1.1" 400 648 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" "-"- - - - - "-"
Since the global IP of the connection source came out like this, it seems to be a success in terms of 'ProxyProtocol'. I won't go into any more about client authentication here, but please understand.
Personally, I had a hard time with the twists and turns until I realized that I just had to do what is written on Chris Lea's page linked in the article, and the wrong designation of the ELB protocol. I couldn't find an article that explained it together, so I wrote it. At an event at the nginx user meeting, I learned firsthand what it means for developer Igor to recommend a new version if you want to use SSL. It may be useful when you want to get rid of a specific device.
Thank you for reading. I would be happy if it could be of help to someone.