JME 3 Tutorial (8) - Hello Picking

Previous: Hello Animation, Next: Hello Collision


Typical interactions in games include shooting, picking up objects, and opening doors. From an implementation point of view, these apparently different interactions are very similar: The user first aims and selects a target in the 3D scene, and then triggers an action on it. We call this process picking. You can pick something by either pressing a key on the keyboard, or by clicking with the mouse. In either case, you identify the target by aiming a ray –a straight line– into the scene. This method to implement picking is called ray casting (which is not the same as ray tracing). This tutorial relies on what you have learned in the Hello Input tutorial.

Sample Code

package jme3test.helloworld;
import com.jme3.app.SimpleApplication;
import com.jme3.collision.CollisionResult;
import com.jme3.collision.CollisionResults;
import com.jme3.font.BitmapText;
import com.jme3.input.KeyInput;
import com.jme3.input.controls.ActionListener;
import com.jme3.input.controls.KeyTrigger;
import com.jme3.input.controls.MouseButtonTrigger;
import com.jme3.material.Material;
import com.jme3.math.ColorRGBA;
import com.jme3.math.Ray;
import com.jme3.math.Vector3f;
import com.jme3.scene.Geometry;
import com.jme3.scene.Node;
import com.jme3.scene.shape.Box;
import com.jme3.scene.shape.Sphere;
/** Sample 8 - how to let the user pick (select) objects in the scene
 * using the mouse or key presses. Can be used for shooting, opening doors, etc. */
public class HelloPicking extends SimpleApplication {
  public static void main(String[] args) {
    HelloPicking app = new HelloPicking();
    app.start();
  }
  Node shootables;
  Geometry mark;
  @Override
  public void simpleInitApp() {
    initCrossHairs(); // a "+" in the middle of the screen to help aiming
    initKeys();       // load custom key mappings
    initMark();       // a red sphere to mark the hit
    /** create four colored boxes and a floor to shoot at: */
    shootables = new Node("Shootables");
    rootNode.attachChild(shootables);
    shootables.attachChild(makeCube("a Dragon",    -2f, 0f, 1f));
    shootables.attachChild(makeCube("a tin can",    1f,-2f, 0f));
    shootables.attachChild(makeCube("the Sheriff",  0f, 1f,-2f));
    shootables.attachChild(makeCube("the Deputy",   1f, 0f,-4f));
    shootables.attachChild(makeFloor());
  }
  /** Declaring the "Shoot" action and mapping to its triggers. */
  private void initKeys() {
    inputManager.addMapping("Shoot",
      new KeyTrigger(KeyInput.KEY_SPACE), // trigger 1: spacebar
      new MouseButtonTrigger(0));         // trigger 2: left-button click
    inputManager.addListener(actionListener, "Shoot");
  }
  /** Defining the "Shoot" action: Determine what was hit and how to respond. */
  private ActionListener() {
    @Override
    public void onAction(String name, boolean keyPressed, float tpf) {
      if (name.equals("Shoot") && !keyPressed) {
        // 1. Reset results list.
        CollisionResults results = new CollisionResults();
        // 2. Aim the ray from cam loc to cam direction.
        Ray ray = new Ray(cam.getLocation(), cam.getDirection());
        // 3. Collect intersections between Ray and Shootables in results list.
        shootables.collideWith(ray, results);
        // 4. Print the results.
        System.out.println("----- Collisions? " + results.size() + "-----");
        for (int i = 0; i < results.size(); i++) {
          // For each hit, we know distance, impact point, name of geometry.
          float dist = results.getCollision(i).getDistance();
          Vector3f pt = results.getCollision(i).getContactPoint();
          String hit = results.getCollision(i).getGeometry().getName();
          System.out.println("* Collision #" + i);
          System.out.println("  You shot " + hit + " at " + pt + ", " + dist + " wu away.");
        }
        // 5. Use the results (we mark the hit object)
        if (results.size() > 0){
          // The closest collision point is what was truly hit:
          CollisionResult closest = results.getClosestCollision();
          // Let's interact - we mark the hit with a red dot.
          mark.setLocalTranslation(closest.getContactPoint());
          rootNode.attachChild(mark);
        } else {
        // No hits? Then remove the red mark.
          rootNode.detachChild(mark);
        }
      }
    }
  };
  /** A cube object for target practice */
  protected Geometry makeCube(String name, float x, float y, float z) {
    Box(new Vector3f(x, y, z), 1, 1, 1);
    Geometry cube = new Geometry(name, box);
    Material mat1 = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
    mat1.setColor("Color", ColorRGBA.randomColor());
    cube.setMaterial(mat1);
    return cube;
  }
  /** A floor to show that the "shot" can go through several objects. */
  protected Geometry makeFloor() {
    Box(new Vector3f(0,-4,-5), 15,.2f,15);
    Geometry floor = new Geometry("the Floor", box);
    Material mat1 = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
    mat1.setColor("Color", ColorRGBA.Gray);
    floor.setMaterial(mat1);
    return floor;
  }
  /** A red ball that marks the last spot that was "hit" by the "shot". */
  protected void initMark() {
    Sphere sphere = new Sphere(30, 30, 0.2f);
    mark = new Geometry("BOOM!", sphere);
    Material mark_mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
    mark_mat.setColor("Color", ColorRGBA.Red);
    mark.setMaterial(mark_mat);
  }
  /** A centred plus sign to help the player aim. */
  protected void initCrossHairs() {
    guiNode.detachAllChildren();
    guiFont = assetManager.loadFont("Interface/Fonts/Default.fnt");
    BitmapText ch = new BitmapText(guiFont, false);
    ch.setSize(guiFont.getCharSet().getRenderedSize() * 2);
    ch.setText("+"); // crosshairs
    ch.setLocalTranslation( // center
      settings.getWidth()/2 - guiFont.getCharSet().getRenderedSize()/3*2,
      settings.getHeight()/2 + ch.getLineHeight()/2, 0);
    guiNode.attachChild(ch);
  }
}

You should see four colored cubes floating over a gray floor, and cross-hairs. Aim the cross-hairs and click, or press the spacebar to shoot. The last hit will be marked with a red dot. Keep an eye on the application's output stream, it will give you more details: The name of the mesh that was hit, the coordinates of the hit, and the distance.

Understanding the Helper Methods

The methods makeCube(), makeFloor(), initMark(), and initCrossHairs, are custom helper methods. We call them from simpleInitApp() to initialize the scenegraph with sample content.

  1. makeCube() creates simple colored boxes for "target practice".
  2. makeFloor() creates a gray floor node for "target practice".
  3. initMark() creates a red sphere ("mark"). We will use it later to mark the spot that was hit.
    • Note that the mark is not attached and therefor not visible at the start.
  4. initCrossHairs() creates simple cross-hairs by printing a "+" sign in the middle of the screen.
    • Note that the cross-hairs are attached to the guiNode, not to the rootNode.

In this example, we attached all "shootable" objects to one custom node, Shootables. This is an optimization so the engine only has to calculate intersections with objects we are actually interested in. The Shootables node is attached to the rootNode as usual.

Understanding Ray Casting for Hit Testing

Our goal is to determine which box the user "shot" (picked). In general, we want to determine which mesh the user has selected by aiming the cross-hairs at it. Mathematically, we draw a line from the camera and see whether it intersects with objects in the 3D scene. This line is called a ray. Here is our simple ray casting algorithm for picking objects:

  1. Reset the results list.
  2. Cast a ray from cam location into the cam direction.
  3. Collect all intersections between the ray and Shootable nodes in the results list.
  4. Use the results list to determine what was hit:
    1. For each hit, JME reports its distance from the camera, impact point, and the name of the mesh.
    2. Sort the results by distance.
    3. Take the closest result, it is the mesh that was hit.

Implementing Hit Testing

Loading the scene

First initialize some shootable nodes and attach them to the scene. You will use the mark object later.

  Node shootables;
  Geometry mark;
  @Override
  public void simpleInitApp() {
    initCrossHairs();
    initKeys();
    initMark();
    shootables = new Node("Shootables");
    rootNode.attachChild(shootables);
    shootables.attachChild(makeCube("a Dragon",    -2f, 0f, 1f));
    shootables.attachChild(makeCube("a tin can",    1f,-2f, 0f));
    shootables.attachChild(makeCube("the Sheriff",  0f, 1f,-2f));
    shootables.attachChild(makeCube("the Deputy",   1f, 0f, -4));
    shootables.attachChild(makeFloor());
  }

Setting Up the Input Listener

Next you declare the shooting action. It can be triggered either by clicking, or by pressing the space bar. The initKeys() method is called from simpleInitApp() to set up these input mappings.

  /** Declaring the "Shoot" action and its triggers. */
  private void initKeys() {
    inputManager.addMapping("Shoot",      // Declare...
      new KeyTrigger(KeyInput.KEY_SPACE), // trigger 1: spacebar, or
      new MouseButtonTrigger(0));         // trigger 2: left-button click
    inputManager.addListener(actionListener, "Shoot"); // ... and add.
  }

Defining the Picking Action

Next we implement the ActionListener that responds to the Shoot trigger with an action. The action follows the ray casting algorithm described above:

  1. For every click or press of the spacebar, the Shoot action is triggered.
  2. The action casts a ray forward and determines intersections with shootable objects (= ray casting).
  3. For any target that has been hit, it prints name, distance, and coordinates of the hit.
  4. Finally it attaches a red mark to the closest result, to highlight the spot that was actually hit.
  5. When nothing was hit, the results list is empty, and the red mark is removed.

Note how it prints a lot of output to show you which hits were registered.

  /** Defining the "Shoot" action: Determine what was hit and how to respond. */
  private ActionListener() {
    @Override
    public void onAction(String name, boolean keyPressed, float tpf) {
      if (name.equals("Shoot") && !keyPressed) {
        // 1. Reset results list.
        CollisionResults results = new CollisionResults();
        // 2. Aim the ray from cam loc to cam direction.
        Ray ray = new Ray(cam.getLocation(), cam.getDirection());
        // 3. Collect intersections between Ray and Shootables in results list.
        shootables.collideWith(ray, results);
        // 4. Print results.
        System.out.println("----- Collisions? " + results.size() + "-----");
        for (int i = 0; i < results.size(); i++) {
          // For each hit, we know distance, impact point, name of geometry.
          float dist = results.getCollision(i).getDistance();
          Vector3f pt = results.getCollision(i).getContactPoint();
          String hit = results.getCollision(i).getGeometry().getName();
          System.out.println("* Collision #" + i);
          System.out.println("  You shot " + hit + " at " + pt + ", " + dist + " wu away.");
        }
        // 5. Use the results (we mark the hit object)
        if (results.size() > 0){
          // The closest collision point is what was truly hit:
          CollisionResult closest = results.getClosestCollision();
          mark.setLocalTranslation(closest.getContactPoint());
          // Let's interact - we mark the hit with a red dot.
          rootNode.attachChild(mark);
        } else {
        // No hits? Then remove the red mark.
          rootNode.detachChild(mark);
        }
      }
    }
  };

Tip: Notice how you use the provided method results.getClosestCollision().getContactPoint() to determine the closest hit's location. If your game includes a "weapon" or "spell" that can hit multiple targets, you would instead loop over the list of results, and interact with each of them.

Exercises

After a hit was registered, the closest object is identified as target, and marked with a red dot. Modify the code sample to solve these exercises:

Magic Spell

Change the color of the closest clicked target!
Here are some tips:

  1. Go to the line where the closest target is indentified, and add you changes after that.
  2. To change an object's color, you must first know its node. Identify the node by identifying the target's name.
    • Use rootNode.getChild(closest.getGeometry().getName())
  3. Create a new color material and set the node's Material to this color.
    • Look inside the makeCube() method for an example of how to set random colors.

Shoot at a Character

Shooting boxes isn't very exciting – can you add code that loads and positions a model in the scene, and shoot at it?

Pick up into Inventory

Change the code as follows to simulate the player picking up objects into the inventory: When you click once, the closest target is identified and detached from the scene. When you click a second time, the target is reattached at the location that you have clicked. Here are some tips:

  1. Create an inventory node to store the detached nodes.
  2. The inventory node is not attached to the rootNode.
  3. You can make the inventory visible by attaching the inventory node to the guiNode (which attaches it to the HUD). Note the following caveats:
    • If your nodes use a lit Material (not "Unshaded.j3md"), also add a light to the guiNode.
    • Size units are pixels in the HUD, therefor a 2-wu cube is displayed only 2 pixels wide. – Scale it.
    • Position the nodes: The bottom left corner of the HUD is (0f,0f), and the top right corner is at (settings.getWidth(),settings.getHeight()).

Conclusion

You have learned how to use ray casting to solve the task of determining what object a user selected on the screen. You learned that this can be used for a variety of interactions, such as shooting, opening, picking up and dropping items, pressing a button or lever, etc.
Use your imagination from here:

Now, wouldn't it be nice if those targets and the floor were solid objects and you could walk among them? Let's continue with Collision Detection.

beginner, beginner,, documentation, intro, node, ray, click, collision, keyinput, input

view online version