To be honest, this is exactly what I'd expect this shader to do. It's kinda hard to explain, but this is totally what you're calculating right now. The calculation part is actually calculating the rate of change of the normals' direction, but that's exactly where you hit a major problem.
You are outputting the IN.Normal from the vertex shader, which is (as the name implies) per vertex. For each face, there'll be 3 vertices, so 3 normals that are part of this equation. After the vertex shader, the normals will be applied to all of the polygon in a linear interpolation process, which means the rate of change between all 3 normals actually stays the same all over the polygon.
Imagine the simplest function of all: f(x) = x. If you'd be taking the derative of that (like you're doing), it'd mean that f'(x) = 1, meaning that there's a constant change of 1 every 'step'.
In this case, since the normals are linear interpolated across the triangle, the rate of change won't actually change at all, since that's linear across the triangle.
I don't know exactly what you're going for, but I think you'll be able to do it, the thinking part is just hard

Please share it on the forums, so we can help you think

Greets,
ZaPP