gluUnProject for iPhone / OpenGL ES

I wasn't surprised that gluUnProject did not exist in the iPhone SDK since they did not provide a glu library implementation. However as I have migrated the gluLookAt, it was time to migrate the gluUnProject. It has been ages since I had used this method. The only thing I remebered was that it was used to know where you have clicked in your 3d world, by using a 2d device, i.e. a mouse. In our case since we're using iPhone, it's the capacitive touch screen which gives us back 2d coordinates of where we touched on the screen.

Since I'm using a perspective view, I needed to translate those coordinates into world coordinates. In my case, on the current project I'm working on, I would finally want to know which button the user click from a grid.

I got the gluUnProject code from MESA. However the code needed some minor adjustments, namely converting everything from double to float:

• any GLdouble had to be replaced with GLfloat
• any double numbers, e.g. 0.0 or 1.0 I converted them to their respective float counterpart, e.g. 0.0f or 1.0f
• and math functions which accepted/returned double, I replaced them with their float versions, e.g. fabs -> fabsf

The next problem was that gluUnProject takes the 3 coordinates, 2 of them are retrieved from the touch event, but the Z coordinate we don't know it. On the desktop openGL usually the depth buffer is queried to retrieve the depth value (between 0 and 1) at the x-y coordinate. But we cannot do this on the iPhone as it uses tile rendering.

So I thought to use the value of zero, but in effect it was always giving the coordinates of the center of the screen. In reality it was giving me the coordinates of the camera position. When I tried 1 instead of 0, it was giving me coordinates that made more sense but still not 100% precise.

The solution was to unproject twice, one time at the near plane (z = 0) and one time at the far plane (z = 1) as discussed in a forum. That gives you a ray which can then be used to make an intersection with a plane and get the exact coordinates. Thus converting from 2d to 3d.

Here's the method I have used which uses the migrated gluUnProject using just floats:

-(CGPoint) getOGLPos:(CGPoint)winPos
{
// I am doing this once at the beginning when I set the perspective view
// glGetFloatv( GL_MODELVIEW_MATRIX, __modelview );
// glGetFloatv( GL_PROJECTION_MATRIX, __projection );
// glGetIntegerv( GL_VIEWPORT, __viewport );

//opengl 0,0 is at the bottom not at the top
winPos.y = (float)__viewport[3] - winPos.y;
// float winZ;
//we cannot do the following in openGL ES due to tile rendering
// glReadPixels( (int)winPos.x, (int)winPos.y, 1, 1, GL_DEPTH_COMPONENT24_OES, GL_FLOAT, &winZ );

float cX, cY, cZ, fX, fY, fZ;
//gives us camera position (near plan)
gluUnProject( winPos.x, winPos.y, 0, __modelview, __projection, __viewport, &cX, &cY, &cZ);
//far plane
gluUnProject( winPos.x, winPos.y, 1, __modelview, __projection, __viewport, &fX, &fY, &fZ);

//We could use some vector3d class, but this will do fine for now
//ray
fX -= cX;
fY -= cY;
fZ -= cZ;
float rayLength = sqrtf(cX*cX + cY*cY + cZ*cZ);
//normalize
fX /= rayLength;
fY /= rayLength;
fZ /= rayLength;

//T = [planeNormal.(pointOnPlane - rayOrigin)]/planeNormal.rayDirection;
//pointInPlane = rayOrigin + (rayDirection * T);

float dot1, dot2;

float pointInPlaneX = 0;
float pointInPlaneY = 0;
float pointInPlaneZ = 0;
float planeNormalX = 0;
float planeNormalY = 0;
float planeNormalZ = -1;

pointInPlaneX -= cX;
pointInPlaneY -= cY;
pointInPlaneZ -= cZ;

dot1 = (planeNormalX * pointInPlaneX) + (planeNormalY * pointInPlaneY) + (planeNormalZ * pointInPlaneZ);
dot2 = (planeNormalX * fX) + (planeNormalY * fY) + (planeNormalZ * fZ);

float t = dot1/dot2;

fX *= t;
fY *= t;
//we don't need the z coordinate in my case

return CGPointMake(fX + cX, fY + cY);
}