Postgres Docker - unable to connect from remote server

I am using postgres:9.5.3 docker image. I am starting the containers and then trying to connect to the psql database from a remote host but each time it fails with the error:

psql: could not connect to server: Connection refused
Is the server running on host "172.18.0.2" and accepting
TCP/IP connections on port 5432?

In my docker-compose file, I am mounting the pg_hba.conf. This is my docker-compose file:

services:
    db:
      networks:
        - test
      image: postgres:9.5.3
      expose:
        - 5432
      volumes:
        - ./pgdata/:/var/lib/postgresql/data
        - ./pg_hba.conf/:/var/lib/postgresql/data/pg_hba.conf

I have modified my pg_hba.conf file to accept remote connections from all hosts based on the instructions here. My pg_hba.conf is as follows:

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             0.0.0.0/0               trust
# IPv6 local connections:
host    all             all             ::0/0                   trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local   replication     postgres                                trust
#host    replication     postgres        127.0.0.1/32            trust
#host    replication     postgres        ::1/128                 trust

host all all 0.0.0.0/0 md5

And my postgresql.conf has the following line too: listen_addresses = '*'

When I try to connect to the database from the host I am running the container on, it connects successfully. But when I try to connect from any remote machine using the command psql -h 172.18.0.2 -U postgres -d postgres -p 5432, it gives me the connection error that means remote connections are not working. With all these settings, I would expect it to connect. What am I missing here?