Making tangent-space normal maps agree with the renderer

Diagnose a coordinate agreement problem

A tangent-space normal map stores directions relative to a surface frame. It is not a standalone description of world-space bumps. A clean bake can therefore show seams or inverted detail when the imported mesh or shader interprets that frame differently. Begin diagnosis with the mesh, bake, texture, and shader together; repainting a seam can conceal the actual mismatch.

Trace the basis and encoding

The frame contains tangent T, bitangent B, and normal N. A common convention reconstructs B = sign * cross(N, T) and transforms the sampled direction using the resulting TBN basis. MikkTSpace documents the handedness sign and the need for matching baking and rendering transforms. [1]

Unsigned normal textures commonly encode signed direction components into the 0–1 range; Microsoft documents this representation and a two-channel reconstruction example. [3] The conventions called OpenGL and DirectX normal formats differ in the direction represented by the green channel. [2]

Test a mirrored panel

Imagine a panel containing one ordinary UV island and one mirrored island. Apply a synthetic normal pointing toward positive tangent Y. With correct handedness, both islands respond according to their own surface frame. If the mirrored island alone reacts oppositely, inspect the stored sign and vertex splits before globally flipping green.

Now substitute a neutral encoded normal (0.5, 0.5, 1.0) for an unsigned RGB map. If a seam remains, investigate mesh normals, transformations, or interpolation rather than the detailed bake. This is a diagnostic construction, not a universal texture format prescription.

Make the asset pipeline reproducible

Freeze the low-poly mesh before baking. Record triangulation, UV set, vertex normals, tangent generator, and normal convention. Preserve required discontinuities during export instead of merging vertices solely because their positions match.

Import the texture as directional data and verify the decoder used by the shader. Rotate a single light across a curved surface, hard edge, and mirrored seam. Test an uncompressed texture before judging compression artifacts. Compare the runtime result with the baker using the same mesh, then retain this asset as a regression fixture.

Respect the method's limits

Matching tangent generators alone is insufficient if triangulation or the interpolation convention changes; MikkTSpace specifically warns about differing quad diagonals. [1] Green-channel inversion fixes an axis convention mismatch, not arbitrary tangent errors.

Normal mapping changes shading direction, so it cannot supply a missing geometric silhouette. When a close-up requires a projecting rim, decide whether geometry or another surface representation is needed. Keep that decision separate from repairing a broken tangent contract.

Sources

• [1] MikkTSpace interface and integration notes — Morten S. Mikkelsen.
• [2] OpenGL and DirectX normal formats — Adobe Substance 3D.
• [3] NormalMapEffect normal-map encoding — Microsoft DirectX Tool Kit.