## Saturday, July 2, 2011

### Fix for Android 2.3

A new version of Photos Around is available in Android Market, with a fix for a startup crash on Android 2.3. Actually, I think it is a fix, because I have not been able to test it on a real device...

## Sunday, June 13, 2010

### Thanks for feedback!

Thanks a lot to all of you who give feedback on Photos Around, either as comments in Android Market or by other means. It's a great help to improve the app further.

### Version 1.3

A new version of Photos Around has been released. This version fixes loading problems and detects if networking or location detection is unavailable. Have fun!

## Tuesday, June 8, 2010

### Small screens supported

A new verison of Photos Around is now available on the Android Market. Support for small screens has been added, which means that you can now download the app if you have Sony Ericsson X10 mini or another small screen Android phone.

## Monday, May 31, 2010

### Technical: Orientation sensing in Photos Around

This post is purely technical and probably only interesting to those who are creating their own augmented reality apps.

Location based augmented reality apps, like Photos Around, need to find out where the mobile device is and how it is oriented in the world. Finding out where the device is, is quite straight forward using the LocationManager in Android, so I will not address this further here. How the device is oriented, on the other hand, is a harder calculation to make.

When I first started to work with orientation sensing in my app, I googled to see if someone else had solved the problem. There were some helpful blog posts, especially from Rud Merriam, which got me started, but I soon discovered that the provided information wasn't enough to give full orientation sensing for all orientation situations. Typically, the calculated angles are unstable when you aim the camera in some directions. Since then I have solved the problem with a slightly different approach, which I would like to share back to those interested.

In Photos Around, I use the orientation of the device for three distinct tasks:

The sensor data is then fed back to my implementation of the method onSensorChanged(), in which most of the calculations take place:

Now let's see what's going on here. First the type of the incoming sensor data is determined. Then it is filtered and copied to the corresponding work array for either magnetic field data or accelerometer data. The filtering is done using a simple mean filter:

I have experimented with more elaborate filters, but I like the natural feeling in the mean filter. It's a bit noisy and slow though, so it can probably be improved more.

Next step is to calculate the rotation matrix using the method SensorManager.getRotationMatrix() provided by the platform. Then I get the orientation of the device in the world, assuming that the screen orientation of the app is in portrait mode.

However, when using the camera preview it seems to be a very good idea to use landscape orientation for your app rather than portrait. I first tried with portrait orientation, and it worked on some phone models but not all (see older blog posts), so I decided to switch to landscape orientation. Since I use landscape orientation I need to remap the coordinate system of the rotation matrix, and thankfully, there is a platform method for that as well: SensorManager.remapCoordinateSystem().

If you compare my code with the documentation of the method, you can see that I change the coordinate system to one which is rotated 90 degrees around the z axis of the device, which is exactly what happens when you change from portrait mode to landscape mode. However, the documentation provides a different setting for augmented reality apps, but I recommend not using it. Changing the camera's axis to y is not a good idea in my experience. I only got full orientation sensing to work when the camera is kept in the negative z direction.

After remapping, the resulting rotation matrix can be used as it is for rotating the OpenGL world and I have solved task 1 from above. Here is a snippet from the code running in the OpenGL thread:

In order to use the rotation matrix directly like this you need to transfer the matrix from the main thread, in which the onSensorChanged() method is called, to the OpenGL thread. This can be done either by synchronizing the rotation matrix object itself or copy it in a method synchronized with the rendering. I have implemented the latter way.

There is one more thing you need to think of before using the rotation matrix for rotating the OpenGL world. The OpenGL world needs to be oriented in the way described by the documentation for SensorManager.getRotationMatrix(), which states that the x component is positive in east direction, the y component is positive in north direction and the z component is positive in upwards direction. In other words, geographic longitude should be mapped to x, latitude should be mapped to y and altitude should be mapped to z.

I found it helpful to really think through the two coordinate systems involved. First the world's coordinate system as described above and then the device's coordinate system where x is going from left to right on the screen, y is going from bottom to top of the screen and z is going straight out of the screen.

The remapped rotation matrix is connecting these two coordinate systems. It is used to transform features in the world to the device's coordinate system, which is what is done in the OpenGL thread. The 3D features there are primarily textured quads representing photos at a certain distance away from the device. By multiplying their coordinates with the rotation matrix, you get their coordinates in the device's system. Of course, this is all handled by the OpenGL implementation. But I will soon do the same thing myself as well.

Another interesting matrix is the inverse of the rotation matrix, which can be used to transform features of the device to the world's coordinate system. More on that later when I calculate the azimuth angle.

Now, place your device on a table in front of you with the screen facing up, and the short edges of the screen facing west and east respectively. In this case, the world's coordinate system is equal to the device's coordinate system. They share directions for x, y and z. If you would be running my code now, and recording the remapped rotation matrix, you would notice that it is very close to the identity matrix.

OK, so now I have the rotation matrix which allows OpenGL to rotate the world correctly while rendering. I will also use it to solve the remaining two tasks from above. As you can see in the onSensorChanged() method above, the first thing I calculate using the matrix is the orientation of the screen.

In order to understand how the screen orientation is calculated, imagine an up vector in the world coordinate system, going from the ground up and through the center point of the screen of your device. Depending on how you hold the device, it would come out of the screen (or go into the screen) at a certain angle. If your device is still lying on the table, the vector would come straight out of the screen at no angle at all. If you are holding the device in front of you, the screen facing horizontally, the up vector would be parallell with the screen.

In any case, now think about the projection of this up vector onto the plane of the screen. This projection vector is always pointing towards the upmost part of the screen, so it's ideal for calculating where up is in screen coordinates. Also, the projection vector is equal to the x and y components (with z = 0) of the up vector based in the device's coordinate system. If I can get hold of those x and y components, I can calculate the orientation angle of the screen.

Since the up vector is known in world coordinates as the positive unit z vector, and I have the rotation matrix of the device, the first thing I do is transforming the up vector to device coordinates by multiplying the rotation matrix with the unit z vector. After that we can simply pick the x and y components.

I have the orientation of the screen described as an x,y vector in device coordinates, which is good, but I want to know the corresponding orientation angle. In other words, I want to change the orientation vector from rectangular coordinates to polar coordinates and use the angular coordinate. As you can see in the code, this is easily done in Java using the Math.atan2() function. Task 2 solved.

So, how is the orientation angle used then? As you might have noticed in Photos Around, all user interface components (except the initial splash screen) orients automatically so they are in level. Up is up. I use the orientation angle to accomplish this by rotating the model view before rendering the component in orthographic projection (screen projection):

All components are defined with their centers in origo and are then translated to their locations on the screen, and rotated according to the orientation angle, when rendered.

All right, last task then: calculating the azimuth angle, i.e. the camera direction's deviation from looking straight north in the plane of the earth (it is flat, right?). This is useful when rendering the radar (or a compass), since the targets should be rotated with the azimuth angle.

Targets in the radar are rendered at their relative location to the device, which is located at the center of the radar. So a target directly to the north of the device should be rendered directly above the center when the camera is looking north. If the camera direction deviates from north at a certain angle, i.e. the azimuth, the target should rotate the same angle in the reverse direction. This means that I can use the calculated azimuth angle in the same way as the orientation angle to rotate the radar targets when rendering them. Actually, when I render the radar, it is rotated with the sum of the orientation and azimuth angles (except for the field of view part, which is only rotated with the orientation angle).

So, how do I calculate the azimuth angle? It is done in a similar way as the orientation angle. For screen orientation I projected physical up on the screen of the device and then calculated the resulting orientation angle using Math.atan2(). For the azimuth I need to project the camera direction on the face of the earth and then calculate the resulting azimuth angle. A major difference is that I need to transform from device coordinates to world coordinates, so the inverse rotation matrix is needed.

As you can see in the code, I transform the same positive unit z vector I used for the orientation calculation. In fact the camera direction is along the negative z axis, but in order to save some variable loading I reuse the positive z vector and then compensate with 180 degrees in the resulting azimuth angle. In other words, I rather calculate the direction of the screen, than the direction of the camera. That's fine as long as the result is compensated.

And that solves task 3 as well!

In the beginning of this post I mentioned that my solution is different from others'. Most solutions I have found remap the rotation matrix so the camera direction is along the y axis and then uses SensorManager.getOrientation() to get the orientation angles of the device. I have experimented a lot with this approach, but never got it to work for every situation. Typically the orientation angles are unstable when you aim straight up or down. Also, I found it very difficult to use the rotation matrix with OpenGL. Remapping like I do above solves this part, but then the angles provided by getOrientation() are unstable when you aim the camera horizontally, which is the main use case...

I hope someone will find this very long post both readable and useful. If you have questions, please let me know.

Location based augmented reality apps, like Photos Around, need to find out where the mobile device is and how it is oriented in the world. Finding out where the device is, is quite straight forward using the LocationManager in Android, so I will not address this further here. How the device is oriented, on the other hand, is a harder calculation to make.

When I first started to work with orientation sensing in my app, I googled to see if someone else had solved the problem. There were some helpful blog posts, especially from Rud Merriam, which got me started, but I soon discovered that the provided information wasn't enough to give full orientation sensing for all orientation situations. Typically, the calculated angles are unstable when you aim the camera in some directions. Since then I have solved the problem with a slightly different approach, which I would like to share back to those interested.

In Photos Around, I use the orientation of the device for three distinct tasks:

- Rotating the OpenGL 3D representation of the world.
- Rotating the user interface so it is always in level (up is up, so to speak).
- Find out where north is in relation to the device, so the radar screen can be rotated correctly.

mSensorManager.registerListener(this, mSensorManager

.getDefaultSensor(Sensor.TYPE_ACCELEROMETER),

SensorManager.SENSOR_DELAY_GAME);

mSensorManager.registerListener(this, mSensorManager

.getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD),

SensorManager.SENSOR_DELAY_GAME);

The sensor data is then fed back to my implementation of the method onSensorChanged(), in which most of the calculations take place:

public void onSensorChanged(SensorEvent event) {

Sensor sensor = event.sensor;

int type = sensor.getType();

float[] mags = mMags;

float[] accels = mAccels;

/*

* Determine the type of sensor data and filter and copy it.

*/

if (type == Sensor.TYPE_MAGNETIC_FIELD) {

mMagFilter.filter(event.values, mags);

} else {

mAccelFilter.filter(event.values, accels);

}

/*

* Calculate the rotation matrix.

*/

float[] rotationMatrixA = mRotationMatrixA;

if (SensorManager.getRotationMatrix(rotationMatrixA, null, accels, mags)) {

float[] rotationMatrixB = mRotationMatrixB;

/*

* Remap the rotation matrix, since I use landscape orientation

* of the screen

*/

SensorManager.remapCoordinateSystem(rotationMatrixA,

SensorManager.AXIS_Y, SensorManager.AXIS_MINUS_X,

rotationMatrixB);

/*

* Calculate the current orientation of the screen

*/

float[] orientationVector = mOrientationVector;

float[] zVec = sZVector;

Matrix.multiplyMV(orientationVector, 0, rotationMatrixB, 0, zVec, 0);

float orientation = (float) (-Math.atan2(orientationVector[0],

orientationVector[1]) * RADIANS_TO_DEGREES);

/*

* Calculate the azimuth angle (deviation from north)

*/

float[] azimuthVector = mAzimuthVector;

Matrix.invertM(rotationMatrixA, 0, rotationMatrixB, 0);

Matrix.multiplyMV(azimuthVector, 0, rotationMatrixA, 0, zVec, 0);

float azimuth = (float) (180 + Math.atan2(azimuthVector[0],

azimuthVector[1])

* RADIANS_TO_DEGREES);

/*

* Use rotationMatrixB, orientation and azimuth

*/

}

}

Now let's see what's going on here. First the type of the incoming sensor data is determined. Then it is filtered and copied to the corresponding work array for either magnetic field data or accelerometer data. The filtering is done using a simple mean filter:

public void filter(float[] inVector, float[] outVector) {

float[] lastVector = mLastVector;

for (int i = 0; i < 3; i++) {

float value = 0.075f * inVector[i] + (1 - 0.075f)

* lastVector[i];

outVector[i] = value;

lastVector[i] = value;

}

}

I have experimented with more elaborate filters, but I like the natural feeling in the mean filter. It's a bit noisy and slow though, so it can probably be improved more.

Next step is to calculate the rotation matrix using the method SensorManager.getRotationMatrix() provided by the platform. Then I get the orientation of the device in the world, assuming that the screen orientation of the app is in portrait mode.

However, when using the camera preview it seems to be a very good idea to use landscape orientation for your app rather than portrait. I first tried with portrait orientation, and it worked on some phone models but not all (see older blog posts), so I decided to switch to landscape orientation. Since I use landscape orientation I need to remap the coordinate system of the rotation matrix, and thankfully, there is a platform method for that as well: SensorManager.remapCoordinateSystem().

If you compare my code with the documentation of the method, you can see that I change the coordinate system to one which is rotated 90 degrees around the z axis of the device, which is exactly what happens when you change from portrait mode to landscape mode. However, the documentation provides a different setting for augmented reality apps, but I recommend not using it. Changing the camera's axis to y is not a good idea in my experience. I only got full orientation sensing to work when the camera is kept in the negative z direction.

After remapping, the resulting rotation matrix can be used as it is for rotating the OpenGL world and I have solved task 1 from above. Here is a snippet from the code running in the OpenGL thread:

gl.glMatrixMode(GL10.GL_MODELVIEW);

gl.glLoadMatrixf(mRotationMatrix, 0);

gl.glTranslatef(0, 0, -mAltitude);

In order to use the rotation matrix directly like this you need to transfer the matrix from the main thread, in which the onSensorChanged() method is called, to the OpenGL thread. This can be done either by synchronizing the rotation matrix object itself or copy it in a method synchronized with the rendering. I have implemented the latter way.

There is one more thing you need to think of before using the rotation matrix for rotating the OpenGL world. The OpenGL world needs to be oriented in the way described by the documentation for SensorManager.getRotationMatrix(), which states that the x component is positive in east direction, the y component is positive in north direction and the z component is positive in upwards direction. In other words, geographic longitude should be mapped to x, latitude should be mapped to y and altitude should be mapped to z.

I found it helpful to really think through the two coordinate systems involved. First the world's coordinate system as described above and then the device's coordinate system where x is going from left to right on the screen, y is going from bottom to top of the screen and z is going straight out of the screen.

The remapped rotation matrix is connecting these two coordinate systems. It is used to transform features in the world to the device's coordinate system, which is what is done in the OpenGL thread. The 3D features there are primarily textured quads representing photos at a certain distance away from the device. By multiplying their coordinates with the rotation matrix, you get their coordinates in the device's system. Of course, this is all handled by the OpenGL implementation. But I will soon do the same thing myself as well.

Another interesting matrix is the inverse of the rotation matrix, which can be used to transform features of the device to the world's coordinate system. More on that later when I calculate the azimuth angle.

Now, place your device on a table in front of you with the screen facing up, and the short edges of the screen facing west and east respectively. In this case, the world's coordinate system is equal to the device's coordinate system. They share directions for x, y and z. If you would be running my code now, and recording the remapped rotation matrix, you would notice that it is very close to the identity matrix.

OK, so now I have the rotation matrix which allows OpenGL to rotate the world correctly while rendering. I will also use it to solve the remaining two tasks from above. As you can see in the onSensorChanged() method above, the first thing I calculate using the matrix is the orientation of the screen.

In order to understand how the screen orientation is calculated, imagine an up vector in the world coordinate system, going from the ground up and through the center point of the screen of your device. Depending on how you hold the device, it would come out of the screen (or go into the screen) at a certain angle. If your device is still lying on the table, the vector would come straight out of the screen at no angle at all. If you are holding the device in front of you, the screen facing horizontally, the up vector would be parallell with the screen.

In any case, now think about the projection of this up vector onto the plane of the screen. This projection vector is always pointing towards the upmost part of the screen, so it's ideal for calculating where up is in screen coordinates. Also, the projection vector is equal to the x and y components (with z = 0) of the up vector based in the device's coordinate system. If I can get hold of those x and y components, I can calculate the orientation angle of the screen.

Since the up vector is known in world coordinates as the positive unit z vector, and I have the rotation matrix of the device, the first thing I do is transforming the up vector to device coordinates by multiplying the rotation matrix with the unit z vector. After that we can simply pick the x and y components.

private static final float[] sZVector = { 0, 0, 1, 1 };

...

float[] orientationVector = mOrientationVector;

float[] zVec = sZVector;

Matrix.multiplyMV(orientationVector, 0, rotationMatrixB, 0, zVec, 0);

float orientation = (float) (-Math.atan2(orientationVector[0],

orientationVector[1]) * RADIANS_TO_DEGREES);

I have the orientation of the screen described as an x,y vector in device coordinates, which is good, but I want to know the corresponding orientation angle. In other words, I want to change the orientation vector from rectangular coordinates to polar coordinates and use the angular coordinate. As you can see in the code, this is easily done in Java using the Math.atan2() function. Task 2 solved.

So, how is the orientation angle used then? As you might have noticed in Photos Around, all user interface components (except the initial splash screen) orients automatically so they are in level. Up is up. I use the orientation angle to accomplish this by rotating the model view before rendering the component in orthographic projection (screen projection):

gl.glPushMatrix();

gl.glTranslatef(dx, dy, 0);

gl.glRotatef(orientation, 0, 0, 1f);

// Render here...

gl.glPopMatrix();

All components are defined with their centers in origo and are then translated to their locations on the screen, and rotated according to the orientation angle, when rendered.

All right, last task then: calculating the azimuth angle, i.e. the camera direction's deviation from looking straight north in the plane of the earth (it is flat, right?). This is useful when rendering the radar (or a compass), since the targets should be rotated with the azimuth angle.

Targets in the radar are rendered at their relative location to the device, which is located at the center of the radar. So a target directly to the north of the device should be rendered directly above the center when the camera is looking north. If the camera direction deviates from north at a certain angle, i.e. the azimuth, the target should rotate the same angle in the reverse direction. This means that I can use the calculated azimuth angle in the same way as the orientation angle to rotate the radar targets when rendering them. Actually, when I render the radar, it is rotated with the sum of the orientation and azimuth angles (except for the field of view part, which is only rotated with the orientation angle).

So, how do I calculate the azimuth angle? It is done in a similar way as the orientation angle. For screen orientation I projected physical up on the screen of the device and then calculated the resulting orientation angle using Math.atan2(). For the azimuth I need to project the camera direction on the face of the earth and then calculate the resulting azimuth angle. A major difference is that I need to transform from device coordinates to world coordinates, so the inverse rotation matrix is needed.

float[] azimuthVector = mAzimuthVector;

Matrix.invertM(rotationMatrixA, 0, rotationMatrixB, 0);

Matrix.multiplyMV(azimuthVector, 0, rotationMatrixA, 0, zVec, 0);

float azimuth = (float) (180 + Math.atan2(azimuthVector[0],

azimuthVector[1])

* RADIANS_TO_DEGREES);

As you can see in the code, I transform the same positive unit z vector I used for the orientation calculation. In fact the camera direction is along the negative z axis, but in order to save some variable loading I reuse the positive z vector and then compensate with 180 degrees in the resulting azimuth angle. In other words, I rather calculate the direction of the screen, than the direction of the camera. That's fine as long as the result is compensated.

And that solves task 3 as well!

In the beginning of this post I mentioned that my solution is different from others'. Most solutions I have found remap the rotation matrix so the camera direction is along the y axis and then uses SensorManager.getOrientation() to get the orientation angles of the device. I have experimented a lot with this approach, but never got it to work for every situation. Typically the orientation angles are unstable when you aim straight up or down. Also, I found it very difficult to use the rotation matrix with OpenGL. Remapping like I do above solves this part, but then the angles provided by getOrientation() are unstable when you aim the camera horizontally, which is the main use case...

I hope someone will find this very long post both readable and useful. If you have questions, please let me know.

## Friday, May 28, 2010

### New version

A new version of Photos Around is available in the Android Market. The issue with rotated camera view on some models has been addressed, and is hopefully resolved (fingers crossed...).

## Wednesday, May 26, 2010

### Known issues

There seems to be a problem running the app on Nexus One with Froyo (Android 2.2). It's not yet clear if it's a general problem with Nexus One or if it's related to just Froyo.

I have tested extensively on HTC Hero with Android 1.5 and HTC Desire with Android 2.1, and it works fine with these. However, there are some small issues with these phones, that should be known:

- If you don't have GPS manually enabled (or you don't have GPS coverage) the app will likely remain at the splash screen.

- Sometimes the overlayed graphics just disappear on HTC Desire. You should press HOME and restart the app to recover the graphics.

I have tested extensively on HTC Hero with Android 1.5 and HTC Desire with Android 2.1, and it works fine with these. However, there are some small issues with these phones, that should be known:

- If you don't have GPS manually enabled (or you don't have GPS coverage) the app will likely remain at the splash screen.

- Sometimes the overlayed graphics just disappear on HTC Desire. You should press HOME and restart the app to recover the graphics.

Subscribe to:
Posts (Atom)