Culling, normals and face winding

The internet is an amazing resource for programmers, especially when using a language or platform that has a massive community behind it. It’s almost guaranteed that someone has experienced your current problem and found a solution to it, a solution that’s shared on a forum post or mailing list somewhere. Unity3D is one such platform.

Unfortunately though, if you don’t use the correct search terms you can end up going in circles for a while and such a thing happened to me last week when trying to extrude my OSM buildings into 3D. I kept telling myself that it was not important for gameplay (and it isn’t) but I was so curious to see what it looked like that I’ve spent just over a weeks worth of free time trying to implement it.

Inevitably trying something new (or even doing something that was done ages ago but is now being reimplemented in some new manifestation) results in all sorts of problems, puzzles, frustrations, mindless hacking and misinterpretations.

Mesh extrusion should be easy, right? Just duplicate the initial mesh, offset it then create new triangles between the two layers to create the walls?


Fun with normals

It turns out that doing a ‘sandwich’ style join between two layers of the 2D mesh (one at a y offset to the other) was a bad idea because of the way shading (from lighting) is done in the grahics pipeline. If there is only one normal per vertex, the buildings would look like they have rounded edges instead of flat shaded (hard) edges. What’s required is to have three copies of a vertex in order to hold the three normals that represent each face. Consider a cube. There aren’t 8 vertices, there are really 24 vertices with 24 corresponding normals (3 per corner).

I had to break the code down into its most basic form and try an extrude just a box into a cube so that I could figure out where my code was breaking.

Fun with normals part 2

Oh, so Unity uses a left-hand coordinate system? That right hand rule for cross product I learnt at Uni and is all over the internet doesn’t actually apply here? Grrr. If I’d been more diligent I’d have noticed that the Unity3D docs clearly state the use of left-hand rule when using Vector3.Cross but it took a while to ‘get it’. My Vector3 normal = Vector3.Cross(b-a, d-a).normalized; should have been Vector3 normal = Vector3.Cross(d-a, b-a).normalized; (I didn’t want to multiply by -1).

With the normals facing the wrong way, the walls would look dark despite the intensity of the lighting.

Fun with normals and culling

Upon investigating the ‘why are the walls dark’ issue, I experimented with some Unity3D provided shaders, but oh no!

Why oh why are some of the walls of my buildings being culled out?! The normals are definitely outward facing!

I stumbled across this forum post (see, someone had a similar issue). Normals are only used for shading and colour, the triangle winding order determines what triangles are back facing.

*sigh* I felt like I’d had that very issue before when my collision meshes didn’t work; the triangles had to be drawn with CCW winding instead of CW (or was it vice versa) for the raycasts to ‘hit’. Anyway, duh! I felt stupid for wasting an evening.

Fun with normals and culling part 2

But holy crap! It still didn’t work! Some of the buildings had walls disappearing but others were fine.

Think, think…

Oh, the OSM data has some polygons defined in CCW order and others in CW order. I had to make them use the same ordering. More searching and I found this. Yay for pseudo code. It works. Ship it.

And now for a screenshot.

Oh look, I’ve got placeholder art from Shenandoah Studio’s Battle of the Bulge; I hope they don’t mind. Their games looks amazing.