Golang TCP connection close doesn't give an EOF

This is actually a follow up on this question: How to know TCP connection is closed in Golang net package? which doesn't work in my implementation, and I'm trying to figure out why.

The idea is, I have a server that accepts TCP connections and waits for the client to close them. It waits for 5 seconds, and every second it tries to read from the TCP connection, if this gives an EOF error, that would indicate that the client has closed the connection.

This is the servers code:

func server(done chan bool) {
    l, _ := net.Listen("tcp", serverAddress)

    // Listen for incomming TCP connections
    conn, err := l.Accept()
    if err != nil {
        log.Fatalf("l.Accept(): got error on accept: %v", err)
    }
    defer conn.Close()

    // Read periodically from the TCP connection to see if we can find
    // out when the client has closed his end.
    go func() {
        for {
            conn.SetReadDeadline(time.Now().Add(10 * time.Millisecond))
            b, err := conn.Read([]byte{})
            fmt.Printf("Read %v bytes with err %v \n", b, err)
            if err == io.EOF {
                conn.Close()
                conn = nil
                break
            } else {
                var zero time.Time
                conn.SetReadDeadline(zero)
            }
            time.Sleep(1 * time.Second)
        }
        done <- true
    }()

    // Do this for 5 seconds, than quit
    time.Sleep(5 * time.Second)
    done <- true
}

So the client sets up a connection, waits for two seconds and then closes it. The clients code is as follows:

func main() {
    done := make(chan bool, 1)
    go server(done)

    // Give the listener some time to set up
    time.Sleep(200 * time.Millisecond)

    // Dial the server
    conn, err := net.Dial("tcp", serverAddress)
    if err != nil {
        log.Fatalf("net.Dial(): got %v, want no error", err)
    }
    defer conn.Close()

    go func() {
        // After two seconds, close the TCP connection to the server
        time.Sleep(2 * time.Second)
        fmt.Printf("Client closes conn now with err %v\n", conn.Close())
    }()

    // Wait untill server is done
    <-done
}

I would expect that the server would get an EOF error when reading, after the client has closed the connection, however the output I get is:

Read 0 bytes with err <nil> 
Read 0 bytes with err <nil> 
Client closes conn now with err <nil>
Read 0 bytes with err <nil> 
Read 0 bytes with err <nil> 
Read 0 bytes with err <nil> 

I have the wireshark output here. It actually shows the FIN was send by the client (package 11), but only 3 seconds later the server sends it's own FIN: Wireshark output

So basically I can't figure out why I'm not getting any error on the read on the servers side.