Parallax mapping - only works in one direction

I'm working on parallax mapping (from this tutorial: http://sunandblackcat.com/tipFullView.php?topicid=28) and I seem to only get good results when I move along one axis (e.g. left-to-right) while looking at a parallaxed quad. The image below illustrates this:

You can see it clearly at the left and right steep edges. If I'm moving to the right the right steep edge should have less width than the left one (which looks correct on the left image) [Camera is at right side of cube]. However, if I move along a different axis (instead of west to east I now move top to bottom) you can see that this time the steep edges are incorrect [Camera is again on right side of cube].

I'm using the most simple form of parallax mapping and even that has the same problems. The fragment shader looks like this:

void main()
{
    vec2 texCoords = fs_in.TexCoords;   
    vec3 viewDir = normalize(viewPos - fs_in.FragPos);
    vec3 V = normalize(fs_in.TBN * viewDir);
    vec3 L = normalize(fs_in.TBN * lightDir);


    float height = texture(texture_height, texCoords).r;
    float scale = 0.2;
    vec2 texCoordsOffset = scale * V.xy * height;

    texCoords += texCoordsOffset;

    // calculate diffuse lighting
    vec3 N = texture(texture_normal, texCoords).rgb * 2.0 - 1.0;
    N = normalize(N); // normal already in tangent-space
    vec3 ambient = vec3(0.2f);
    float diff = clamp(dot(N, L), 0, 1);
    vec3 diffuse = texture(texture_diffuse, texCoords).rgb * diff;
    vec3 R = reflect(L, N);
    float spec = pow(max(dot(R, V), 0.0), 32);
    vec3 specular = vec3(spec);
    fragColor = vec4(ambient + diffuse + specular, 1.0);
}

TBN matrix is created as follows in the vertex shader:

vs_out.TBN = transpose(mat3(normalize(tangent), normalize(bitangent), normalize(vs_out.Normal)));

I use the transpose of the TBN to transform all relevant vectors to tangent space. Without offsetting the TexCoords, the lighting looks solid with normal mapped texture so my guess is that it's not the TBN matrix that's causing the issues. What could be causing this that it only works in one direction?

edit

Interestingly, If I invert the y coordinate of the TexCoords input variable parallax mapping seems to work. I have no idea why this works though and I need it to work without the inversion.

vec2 texCoords = vec2(fs_in.TexCoords.x, 1.0 - fs_in.TexCoords.y);