Unable to load balance using Docker, Consul and nginx

Question

What I want to achive is load balancing using this stack: Docker, Docker Compose, Registrator, Consul, Consul Template, NGINX and, finally, a tiny service that prints out "Hello world" in browser. So, at this moment I have a docker-compose.yml file. It looks like so:

version: '2'
services:
accent:
    build:
        context: ./accent
    image: accent
    container_name: accent
    restart: always
    ports:
        - 80
consul:
    image: gliderlabs/consul-server:latest
    container_name: consul
    hostname: ${MYHOST}
    restart: always
    ports:
        - 8300:8300
        - 8400:8400
        - 8500:8500
        - 8600:53/udp
    command: -advertise ${MYHOST} -data-dir /tmp/consul -bootstrap -client 0.0.0.0
registrator:
    image: gliderlabs/registrator:latest
    container_name: registrator
    hostname: ${MYHOST}
    network_mode: host
    restart: always
    volumes:
        - /var/run/docker.sock:/tmp/docker.sock
    command: -ip ${MYHOST} consul://${MYHOST}:8500
nginx:
    container_name: nginx
    image: nginx:latest
    restart: always
    volumes:
        - /etc/nginx
    ports:
        - 8181:80
consul-template:
    container_name: consul-template
    build:
        context: ./consul-template
    network_mode: host
    restart: always
    volumes_from:
        - nginx
    volumes:
        - /var/run/docker.sock:/tmp/docker.sock
    command: -consul=${MYHOST}:8500 -wait=5s -template="/etc/ctmpl/nginx.ctmpl:/etc/nginx/nginx.conf:docker kill -s HUP nginx"

The first service - accent - is that my web service that I need to load balance. When I run this command:

$ docker-compose up

I see that all services start to run and I see no error messages. It looks as if everything is just perfect. When I run

$ docker ps

I see this in the console:

... NAMES              STATUS            PORTS
consul-template    Up 45 seconds     
consul             Up 56 seconds     0.0.0.0:8300->8300/tcp, 0.0.0.0:8400->8400/tcp, 8301-8302/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 8600/tcp, 8600/udp, 0.0.0.0:8600->53/udp    
nginx              Up 41 seconds     0.0.0.0:8181->80/tcp   
registrator        Up 56 seconds
accent             Up 56 seconds     0.0.0.0:32792->80/tcp

Please, pay attention to the last row and especially to PORTS column. As you can see, this service publishes 32792 port. To check that my web service is achievable I go to 127.0.0.1:32972 on my host machine (the machine where I run docker compose up) and see this in browser:

Hello World

This is exactly what I wanted to see. However, it is not what I finally want. Please, have a look at the output of docker ps command and you will see, that my nginx service published 8181 port. So, my expectation is that when I go to this address - 127.0.0.1:8181 - I will see exactly the same "Hello world" page. However, it is not. In browser I see Bad Gateway error message and in nginx logs I see this error message

nginx | 2017/01/18 06:16:45 [error] 5#5: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:32792/index.php", host: "127.0.0.1:8181"

It is really interesting, because nginx does what I expect it to do - upstreams to "http://127.0.0.1:32792/index.php". But I'm not sure why does it fail. By the way, this is how nginx.conf (created automatically with Consul Template) looks like:

worker_processes 1;
events {
    worker_connections 1024;
}
http {
    sendfile on;

    upstream app_servers {
        server 127.0.0.1:32792;
    }

    server {
        listen 80;
        root /code;
        index index.php index.html;

        location / {
            try_files $uri/ $uri/ /index.php;
        }

        location ~ .php$ {
         proxy_pass http://app_servers;
         proxy_redirect off;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Host $server_name;
        }

        location ~ /.ht {
        deny all;
        }

    }
}

I wouldn't change anything, since this nginx.conf looks good to me. Trying to understand why it does not work, I shelled to nginx container and made a couple of commands:

$ curl accent
Hello World

$ curl 127.0.0.1:32972
curl: (7) Failed to connect to 127.0.0.1 port 32972: Connection refused

$ curl accent:32972
curl: (7) Failed to connect to accent port 32972: Connection refused

Again, it is interesting, because nginx container sees my web service under port 80 and not under its published 32972 port. Anyway, at this stage I do not know why it does not work and how to fix it. I just have a guess, that it is somehow connected to the way, how network is configured in docker-compose.yml. I tried various combinations of network_mode: host on accent and nginx service, but to no avail - either accent stops working or nginx or both. So, I need some help.

Answer 1

When you do port binding it publish some port from container (80 in accent e.g.) and some port on your host (random 32792 on host e.g.).Containers in same network as your accent container can access your container port 80 by accent (same as accent:80) due to docker-compose services name resolving. You can access accent:80 from your host with accent:32792. When you are requesting 127.0.0.1:32792 from your nginx container you can access only nginx container 32792 port, not accent. accent:32792 is not correct url from anyway (80 port open on accent, 32792 on host). But 127.0.0.1:32792 should work when you add nginx container to host network. But I noticed that you use incorrect port in curl call. Your accent:80 published to host 32792 but you request 32972.