Lines and errors after normal leveling
Thanks for the reply mate,
The normal was created from a High Poly mesh and the result doesn't change if i re-normalise the map.
I usually add a levels to the normal map to get the full range out of the texture and normally set it up like this
Not the values specifically, just where the arrows are in relation too the information in the channel (apart from the blue which is clamped 127-255)
Thanks for the help 
I usually add a levels to the normal map to get the full range out of the texture and normally set it up like this
Not the values specifically, just where the arrows are in relation too the information in the channel
Yep, that technique can produce a more steep normal map, but it's mathematically incorrect. 
Perhaps working in 32bits ( floating point ) mode or saving the normal map using OpenEXR/HDR/TIFF Float point could mitigate the effect... but I think some Photoshop plugins can work only in 8/16 bits mode.
Another thing you can try is to adjust the contrast/bright of the normal map's channel ( usually the blue one.... make it darker ) instead of editing the output levels.... Or try with the curves instead.
Other idea ... but I 've not tested and I have serious doubts about it... Perhaps you could multiply the normal map by using a convexity map. That's equivalent to smooth and self-occlude the normal map using the cosine of the lowpoly normal and the highpoly normal. Perhaps could work.
BUT ... hmmm... I think I could add an option in xNormal or some kind of "special normal map tone mapper" in a way the color range precision could be better used... I could investigate a bit this.
Great tips jogshy. I didn't know adjusting the RGB levels could render a mathematically incorrect result. I will try using the brightness/contrast trick and see what kind of results I can get. Thanks for all the support!
-Jeremy-
Thanks for the reply Jogshy 
I rendered the normal out as an .exr and this fixed the problem.
I was able to do the levels adjustment and then merge the 32bit image down to 8bits without any artifacting but after merging the 127,127,255 becomes 188,188,255.
I would love to try out the contrast method more but because i cant see the channel information its mainly guesswork when im editing.
Would the Levels method still be ok if the map was normalized after the editing was done as ive seen great results with this method and didn't realize that it was mathematically incorrect...what makes it incorrect btw?
It would be amazing if you could figure out a way to maximise the channels but still provide a correct result mathematically.
This is a very interesting topic and im eager to hear more about your thoughts 
Would the Levels method still be ok if the map was normalized after the editing was done as ive seen great results with this method and didn't realize that it was mathematically incorrect...what makes it incorrect btw?
Two things make it mathematically incorrect:
1. De-normalization. If you touch manually a pixel, the normals will be denormalized ( R^2+G^2+B^2 != 1 ).
you can fix this easily just applying a xNormal->Normalize filter in Photoshop... not a big problem. Also, if the pixel shader normalizes the normal ( which is discouraged to save some speed in the HLSL/GLSL shader code ) it will fine too.
2. Alteration of the tangent basis. The TS-normal maps are computed in a way that
normalize ( mul ( normalTS, vertex.invTangentBasisMatix ) ) = outOSNormal
If you alter the normalTS pixel then the resulting object-space wil be altered indirectly... Not a big problem in theory, but in practice it can be.
An example: altering the normal will cause an incorrect vector displacement mapping... so the projectedPos=lowpoly.pos + norm*dist WON'T match the original highpoly.position. Even if you aren't using displacement mapping the outOSNormal will differ from the original highpoly's normal.. Remind the point of retopo/projective normal mapping is to capture the highpoly's normals as accurately as possible ( if not, just use the height-map to normal-map 2D method! ).
Other effect is the introduction of inexaplicable seams due to a pixel shader mismatch. The normal map can look ok in Photoshop but you can break the light continuum ( specially around seams, hard edges or around mirrored/tiled parts ). This is specially noticeable for objects with a very shiny specular factor ( because the specular is usually computed with a pow() function... so the error will be exponentially increased ).
For example, if you blur some pixels near a triangle with a hard edge(imagine a box's edge), a lighting artifact will apear for sure.... because the blur gonna average two neighbor pixels which should have a complete different normal !
Other example: if you have a value near 126, the Photoshop output levels could convert it to 129. That's a critical error, because when you compute the diffuse lighting using a dot(normal,lightDir) gonna pass from a negative value(backfaced,unlit) to a positive value(frontfaced, lit)... so you can introduce a black line or halo artifact there. Making this will flip the normals!
Other nasty collateral effect: the curves can modify the background color ( that should be 127,127,255 or 128,128,255 ). That can make artifacts when the 3D engine paints the objects with mipmapping ( due to, again, flipped normals, >0 to <0 dots, etc... ) or strangely skewed lighting ( if the light is on the front of the objects... why I see that gradient? )
However, a mathematically-incorrect method can look good... in fact, a lot of films use "biased" renderers which aren't physically correct but they look good-enough so....
Wow, what a reply!
Do you think its possible to add something to xNormal so it would render a normal map, but automatically level the image while still being correct mathematically?
Do you think its possible to add something to xNormal so it would render a normal map, but automatically level the image while still being correct mathematically?
Well, it's a hard problem to solve... but perhaps if I perform a special way of importance sampling around the highpoly's normal it could be possible to capture better the detail, which should lead to a normal map with more "contrast" or "cavities" around. It's something like a cavity map but applied to the normal instead to the AO.
Other thing that can help is the antialiasing. Currently, in xn3 the antialiasing is very...hmmm... simple. For xn4.0 I'm playing with much more advanced sampling methods. I definitely gonna investigate this for xn4. For xn3 it will be too complicated to implement currently.
Btw... have you tried to multiply the TS-normal map by a convexity map? That should weight the normal map with the cosine of the lowpoly/highpoly normals enhacing the result.... or perhaps not, who know!










Have you generated that normal using a height map?
Caution about adjusting the levels(curves) or a normal map. At some point it will be denormalized andmathematically incorrect... so it can show some kind of weird things.
Ideally, the R^2+G^2+B^2 should be equal to 1 or it will be denormalized.
What happens if you pass a xNormal->Normalize filter after editing it?
My advice is to touch the normal map as few as you can. If you need more steep/marked normals play a bit with the "Smooth" and "Method" params of the HM2NM tool. If this does not give you enough detail, try to alter the output levels in Photoshop... but remember to pass a xNormal->Normalize filter at the end.
------------------------------------------------------
http://www.xnormal.net
http://santyhammer.blogspot.com
http://www.ratgpu.com