Vulkan normal mapping direction problems

I am currently trying to implement normal mapping into my Vulkan renderer. However, for some reason my TBN matrix is transforming the normal map incorrectly. I am following the learnopengl[dot]com tutorial for implementation of normal maps. In OpenGL (which I used before) I had no problems with normal mapping. However, here you can see the problem in two videos:

1: https://www.youtube.com/watch?v=RWLkka939w0&feature=youtu.be

2: https://www.youtube.com/watch?v=fkPp7wYwNGI&feature=youtu.be

The first one shows the effect when rendering my diffuse colour (at the moment its a forward renderer with one directional light) the second one shows the normal output of the renderer. The white sphere and torus are using the vertex normal. As you can see with the vertex normal used, the lighting is correct (I also tested it with the textured sphere and block). When applying the TBN matrix to the normal map the light goes off and seams to change direction when rotating the cube. Here is the glsl code to calculate the TBN matrix:

Vertex shader:

void main() {

  vec4 pos = u_main.model * vec4(position, 1.0);

  mat3 normal_matrix = transpose(inverse(mat3(u_main.model)));

  vec3 T = normalize(normal_matrix * tangent.xyz);
  vec3 N = normalize(normal_matrix * normal);

  vec3 B = normalize(cross(N, T) * tangent.w);

  v_TBN = mat3(T, B, N);

  FragmentPosition = vec3(pos);
  v_position = pos.xyz / pos.w;
  v_TexCoord = tex_coord;
  v_normal = normalize(normal_matrix * normal);

  //The proj has been manipulated like here: https://matthewwellings.com/blog/the-new-vulkan-coordinate-system/
  gl_Position = u_main.proj * u_main.view * u_main.model * vec4(position, 1.0);
}

Fragment shader:

vec3 N = texture(t_Normal, v_TexCoord).rgb;

N = v_TBN * (N * 2 - 1);

Things I already tried:

• I use Gltf to load my models which uses +y as up, so I multiplied all y-coordinates with -1.
• I tried some formula to calculate the TBN matrix in the shader. But I didn't understand it correctly and the normal wasn't working correctly either so I discarded it.
• converting my normal map from srgb to linear colours via a glsl function:

vec3 srgb_to_linear(vec3 c) {
    return mix(c / 12.92, pow((c + 0.055) / 1.055, vec3(2.4)), step(0.04045, c));
}

I hope someone has an idea what this could be. I am struggling for around 2 weeks now ^^. If you need more information let me know. Thanks in advance!

-siebencorgie