Augmented Reality Control of the Telerobot 2003

6.2 Design and Implementation

To be able to model a 3D environment it is desirable to have some internal model of what a 3D object is, we are modelling rectangular wire frames, so some 3D representation of a rectangular prism was needed.

There were a number of choices for the internal representation of the prism wire frame some of the most promising were:

1. 
   A 3D rectangle, i.e. 4 3D points, with a distance value representing the depth that the prism extends.
   
2. 
   2 3D rectangles representing parallel faces of the prism, from this it is possible to calculate the depth that the 3D rectangles are apart and subsequently draw them, along with adjoining lines.
   
3. 
   A single 3D point and dimensions for length height and width.
   
4. 
   8 3D points which specify the vertices of the prism
   

The labView control is shown below, it consists of a cluster of 8 3D points, where each 3D point consists of a cluster of 3 doubles, x, y and z.

Computer graphics, however, operate in a 2 Dimensional environment, i.e. the computer monitor. So it is necessary to have some way of representing the above 3D rectangular prism (RP) in 2D as well for the purpose of display and interactions with the mouse, also a 2D device. Since we had chosen to represent the 3D RP as 8 points it seems a logical step to also represent the 2D RP as 8 2D points, where each point in 3D is mapped to the corresponding point in 2D.

There are three operations that we wish to be able to perform in order to manipulate the prism to accurately model the working environment. These are:

1. 
   Scaling, in x y and z directions but z scaling is limited to moving the top of the prism, we wish to always limit the bottom of the prism to the top surface of the workspace.
   
2. 
   Translation, in x and y directions, again there is no z translation as we wish to limit the wire frame so that it remains on the top surface of the workspace.
   
3. 
   Rotating, about the z axis. Rotations in other direction are not considered, as again we wish to keep the wire frame on the top surface of the workspace. This also simplifies the process of a user modelling a real object with the wire frame model, as they can simply place the wire frame over the extremities of the object.
   

All three translations, x y and z, can be done in a single stage by inputting a translation vector (x,y,z) and adding the components of this vector to each of the 8 points of the 3D RP. That is

Where F is the final position, P is the initial point of the 3D prism and T is the translation vector.

Scaling needs to be done about a fixed point, and in order to satisfy that the bottom face of the RD doesn’t ever leave the top face of the workspace that point should be the geometric centre of the bottom face of the RP. So a function is needed to find that point based on the 3D prism. Once this point is known then it can be used as an input, a scaling vector is also needed. We can then apply some simple scaling equations.

Where (x’, y’, z’) is the position of the scaled point

F is the fixed point at the bottom centre of the RP

S is the scaling vector and

(x ,y ,z) is the original point

6.2.3.3 Rotation

Rotation, as already discussed, is performed about the z axis. A matrix that rotates a given point about the z axis about the origin is given by:

So if we first translate to the origin, rotate the point, and then translate back the same amount, then we can effectively rotate a point about the z axis about some given fixed point F.

Where (x’, y’, z’) is the new rotated point

(x, y, z) is the point o rotate and

F is the fixed point to rotate about.

6.2.4 3D to 2D mapping equations

A 3D prism can now be manipulated using scaling, translation and rotation, but at present there is no way of representing this 3D RP on the screen. Some form of conversion between 3D and 2D is needed. We tried several different equations before settling on one that didn’t visibly distort the 3D RP and gave us the ability to manipulate where the vanishing point of the 2D representation of the 3D RP was to occur. Our first equations had the vanishing point at the origin (0, 0), these we quickly discarded in favour of equations that allowed us to specify a variable vanishing point.

Where (Px, Py, Pz) is the 3D point to be mapped

(Ex, Ey, Ez) is the 3D eye position, this is where your eye appears to be viewing the 3D point from. Ey is a dummy depth variable, and represents how far the 3D point is away from the Eye position, we have used a default 1000. If this value is too big then foreshortening is negligible and the prism appears as if all parallel lines are actually parallel. If this value is too small then the 3D RP becomes distorted, and looks as if we are viewing it through a fish bowl. The constant 1000 was chosen after experimentation with many values and viewing the best results, more on this in the testing section of this module, 6.3. The 2D point (Ex, Ez) is the 2D vanishing point of the picture as determined by camera calibration, see the relevant section.

S_x,2D and S_y,2D are the x and y points respectively that the mapped 3D point should appear at in order to produce an accurate image.

6.2.5 Drawing the Prism

Now that we have a 2D representation of a 3D RP we are able to draw the RP in labView. In order to achieve this we have had to assign he various 8 points to somewhere in the RP and build up some form of relationship between them. The prism that we have defined is as follows:

Once we know the relationship between the points it is possible to explicitly code the order in which to draw the points. The way we have chosen to achieve this is just an arbitrary assignment and matters little but is mentioned here as a matter of completeness.

The drawing of the RP can be broken up into two parts. Those that can be drawn without lifting the pen, the so called continuous drawing parts, and those that are left over which involve drawing discrete lines and lifting the pen between each line.

The continuous drawing is done in this order:

Draw point 1 to 2

Draw point 2 to 3

Draw point 3 to 4

Draw point 4 to 1

Draw point 1 to 5

Draw point 5 to 6

Draw point 6 to 7

Draw point 7 to 8

Draw point 8 to 2

There are then 3 discrete line segments left over after the continuous drawing in order to complete the RP. These are to

Draw point 5 to 8

Draw point 3 to 7

Draw point 4 to 6

Three colours are also needed to be inputted for the various modes of viewing the prism, which are regular, highlighted and selected. Regular is when the prism is sitting idle so to speak and the mouse is not over any control point and the mouse has not been clicked on a control point. Highlighted is when the mouse is over a control point but the button is not pressed. Selected is when the mouse has been clicked when over a control point and the prism is being manipulated.

This method of drawing the prism all in one colour, albeit a changing colour depending on the state of the prism is fine, however, when the drawing of the RP is viewed it is occasionally possible, if you look lines in the wrong way, to get what is known as geometric inversion. This is when the prism is viewed as if it was in a different perspective to that which it is intended to be viewed from. To assist in the understanding of this matter and why it is a problem take a look at the following figures

This prism has had its visible edges emphasised with a heavy line and the hidden edges are left as thin lines. This is the same prism to that above in 6.2.5 except that it has had the geometric inversion emphasised. Compare this to the same prism below with the intended viewing mode emphasised.

Although it is still possible to view the geometric inversion it is clearer what is intended to be viewed if these emphasises are shown on the screen as part of the drawing of the RP.

The way that this is overcome is by identifying what we have called the “view mode” of the RP, shown as “left”, “right” and “centre” above. If the view mode is known then the different cases for drawing are also known, and so, in particular, the number of lines that need greying out is known.

If we look at the bottom face of the prism and consider the usual 2D Cartesian co-ordinate system of

y



x

then we notice that each of the 3 view modes have unique configuration of their points in terms of the relations between maximum and minimum x values. By using this, and having knowledge of the structure of the RP and the way that it is drawn, i.e. point 1 cannot be adjacent to point 8 and Point 3 cannot be on the bottom of the RP etc, it becomes possible to identify the view mode of the prism, and also to identify which point of the prism is closest to the viewer. From this information it is possible to identify the bottom face of the prism. This information is used later on to identify control points. More on that in the relevant section under control points.

7. 
   Control Points
   

As mentioned previously control points are points that allow the RP to be manipulated. There are 6 control points in total but only ever 5 active at any one time. When the RP is sitting idle and has not been selected then it is known that the user does not wish to manipulate this RP, and to reduce clutter on the image many of the control points are hidden and inactive. Only one remains a control point at the geometric centre of the RP, this is the control point (CP) that is used to select the RP. Once the RP has been selected the remaining 5 CPs become visible and active. These 5 CPs are responsible for:

1. 
   Rotation
   
2. 
   Length stretching
   
3. 
   Width stretching
   
4. 
   Height stretching and
   
5. 
   Translation (x&y)
   

All control points have a constant radius, but this radius is easily changeable on start-up. The purpose of the control points is to manipulate the prism and the names chosen are indicative of that purpose. So rotation rotates the prism, height stretches the prism taller or shorter, and length/width stretches the prism in the direction that the control point is on. See below for a clearer idea of the intended design, this is where a Length/Width control point is stretching the RP.

But in order for the control points to remain in an easy to reach place, i.e. not unnecessarily on any points on the back of the prism, obviously one point needs to be otherwise we could only ever manipulate one face of the prism, we need to be able to swap the control points over as the prism rotates and otherwise changes its appearance. Again due to the different viewing modes of the RP this is made more difficult than it sounds, fortunately we have already almost completely solved this problem earlier, in the Greying back of Prism 6.2.6 section above. If we look at the view modes again but with control points added as well then this will hopefully become clearer.

As is evident form this diagram the control points, for the left and right view modes is simple defined as the opposite corner to the greyed out lines, this if we can find one then we can find the other. The centre view mode is however different, and deserves special mention. We notice that the configuration for control points in this mode could occur in one of two ways for this orientation of RP.

But however if the prism is rotated slightly then one of the points will have a lower x coordinate, based on the coordinate system in 6.2.6 Greying Back of Prism, this is then identified as the rotate point and the configuration for the other CPs is based on that rotate point. We note that all CPs are adjacent to the rotate point, except for translation, which is covered later, so if the rotate point is known then all other CPs can be found. But the case above still requires attention.

The choice is ultimately arbitrary as to which configuration we use, centre 1 or centre 2. This is because both configurations convey the same meaning and neither is a clearly better representation, also neither is easier or harder to understand. As such we have chosen to, when this case arrives, assign the CPs the configuration shown in centre 1.

2. 
   Drawing control points
   

There are a few choices to make regarding the appearance of the control points, there are several options to choose from:

1. 
   Squares whose geometric centres are at the CP.
   
2. 
   Circles whose centre is at the CP
   
3. 
   Ovals whose centre is at the CP, and are elongated in such a direction as to try to indicate some sense of perspective
   

The ring around the CP was added after experimentation and testing showed that the edges of the CP were not clearly defined, and also to highlight the presence of the CP and make it stand out. The chosen CP the looks something like this:

Size

Translation of the prism is also possible, apart from the translation point on the bottom face of the prism, via translation lines. These are lines that pass through the bottom centre control point of the prism, and are drawn in the same realistic looking perspective that the prism is drawn in. They only appear if the prism is active, that is the control points are visible and engaged. The user may click on the line and the prism will translate in a direction perpendicular to the line, following the mouse movement.

The RP we have been discussing in the preceding sections, in both 3D and 2D, is merely a construct created for our convenience, it allows convenient representation of 3D models on a 2D screen, however, it so far has no relevance or meaning when dealing with the real world. Meaning that a 3D RP of length 100 units is not a 3D real world RP of length 100 mm, or cm or whatever. Also, when dealing with the translation vector T as discussed in 6.2.3.1, a translation vector of (100, 100, 0) does not translate the prism in the positive x direction 100 mm and the positive y direction 100 mm. These issues need to be addressed in order to continue, and they are covered further in . This section covers solving the relative translation discrepancy but assumes that the origin is at the bottom left of the workspace. Where the workspace is given by ABCD below.

In other words it assumes that the point B shown above is (0,0,0) with the coordinate system as defined above. So we require an algorithm that, for any given workspace, and knowledge of the 3D and 2D prism, and manipulations, and any additional information that is required to map given 3D RP to a 2D RP, translates the origin to point B. We also include in this algorithm a way of specifying the desired dimensions of the RP.

We feel that this is a suitable starting size for the RP; this is because it is roughly of equivalent size to the long rectangular prism blocks that exist in the tele-Robot enclosure thus less manipulation is required to model a real block with the wire frame.

• 
  The arbitrary 3D RP we are given is of a cube of side 100 with point 1 at the origin.
  
• 
  The grid we are calibrating the RP to, in the real world is a square, that is
  

1. 
   Since we are given the 2D coordinates of points A, B, C and D we know the magnitude of BC, we can therefore calculate the desired magnitude of the RP.
   
2. 
   As we are given some initial arbitrary 3D RP and parameters necessary to calculate the equivalent 2D RP we can calculate the magnitude of B’C’, and the 2D RP
   
3. 
   From 1. and 2. we are able to calculate an x scaling factor, S_x, where
   
4. 
   As we started with a cubic prism and we know that the workspace ABCD is square we can say that . As dictated by our desired prism dimensions.
   

1. 
   As the prism is now of desired dimensions we can concentrate only on one point in the prism, and that is B’. If we translate this point so that is coincides with B then the prism appears in the correct position.
   
2. 
   First a few observations regarding the relationship between the 3D RP translation vector and the corresponding 2D RP.
   
   1. 
      If we translate a given 3D RP by applying the translation vector (0, y, 0) and look at one point in the 2D RP, we note that this point has moved in both its x and y coordinates.
      
   2. 
      If we translate a given 3D RP by applying the translation vector (x, 0, 0) and look at one point in the 2D RP, we note that this point has moved only in its x coordinate
      
3. 
   Form this we realise the possibility of applying a numerical iterative technique where we translate first with the translation vector (0, y, 0) until the y coordinate of B’ corresponds to the y coordinate of B. Incrementing or decrementing y in the translation vector based on need, until the two y coordinates of B’ and B are equal, or within some allowed error range.
   
4. 
   From observation 2.2 we can know that we can translate the 3D RP in the x direction with the translation vector (x, 0, 0) without effecting the now correct y position of the RP. We do this iteratively until the x coordinate of B’ is within some allowable error range of the x coordinate of B.