General information



Download 57.62 Kb.
Date25.06.2017
Size57.62 Kb.
#21835
Collaborative Mobile Image Processing developer documentation

Molnár Zoltán

zmolnar@inf.u-szeged.hu

General information:


Eclipse configuration

  1. Download ADT Bundle for Windows

http://developer.android.com/sdk/index.html

  1. Download Android NDK Library

http://developer.android.com/tools/sdk/ndk/index.html#download

  1. Install Android NDK Plugin & Android Development Tools

http://developer.android.com/sdk/installing/installing-adt.html

  1. Download OpenCV Android SDK 2.4.8

http://sourceforge.net/projects/opencvlibrary/files/opencv-android/2.4.8/OpenCV-2.4.8-android-sdk.zip/download

  1. Install Android 4.0 (API 14) SDK Platfrom using Android SDK Manager

For using devices via Eclipse usually USB driver is required.

For example: If you want to compile to a Samsung Galaxy S3, you have to download and install the Samsung USB driver for mobile phones tool from this website:



http://developer.samsung.com/android/tools-sdks/Samsung-Andorid-USB-Driver-for-Windows

Android project configuration

Download the sources zip file, unzip it, and import to eclipse.

System environment variables



  1. NDKROOT: \android-ndk-r9c

  2. OPENCV_SDK: \OpenCV-2.4.8-android-sdk\sdk

These paths should not contain any spaces.

Device environment



  1. Minimum API level: 14 (Android 4.0)

  2. Target API level: 1a9 (Android 4.4)

  3. Download & install OpenCV Manager from Google Play

https://play.google.com/store/apps/details?id=org.opencv.engine&hl=hu

Demo modes:


This defines how the application will work, what interfaces will appear or stay hidden. This parameter can only be changed in the source code. If you want to run (compile) a different version you need to change "demo_mode" parameter in hu.inf.u_szeged.inf.library.SettingsReader class in the SZTE-RGVC Library project.

none: normal, full application.

functions:



  • taking picture

  • 3 camera reconstruction from gallery images (fig. C.)

  • 2 camera reconstruction from gallery images. local or network based on settings preference (fig. B.)

  • 2 camera test from test cases. See creating test folders for adding test cases. local (fig. A.) or network (fig. B.) based on settings preference

  • 2 camera reconstruction with image taken from camera in place. local (fig. A.) or network (fig. B.).

online demo: 2 camera network reconstruction with images taken from camera in place. Application starts with camera preview. (fig. B.)

offline demo: 2 camera local reconstruction, images taken from a list of image pairs (see creating data for demos) (fig. B.)

obj demo: select image pair from a list, and show the result previously made. See creating data for demos

3cameras demo: 3 camera network reconstruction with images taken from camera in place. Application starts with camera preview. (fig. C.)

full demo: multi camera (2-3-4+) network recontruction with images taken from cameras in place. Application starts with camera preview, uses vision graph to filter devices. Not finished. (fig. C.)

online demo no camera: 2 camera network reconstruction with images taken from a list of image pairs (see creating data for demos). (fig. B.)

3cameras demo no camera: 3 camera network reconstruction with images taken from gallery or list of images. Not finished. (fig. C.)

full demo no camera: multi camera (2-3-4+) network recontruction with images taken from gallery or list of images. application starts with camera preview, uses vision graph to filter devices. Not finished. (fig. C.)

bundle adjustement test: network, 3 camera reconstruction with fixed input for testing bundle adjustement and merging multiple reconstruction results

Settings


Ip of other device: This option shows a list of devices with ip addresses. The selected item will be used in online demo mode ONLY IF demo_ip.txt (see needed files) is missing or empty. This list can be changed in NotSoDummy.java

Automatic sensor service: Toggles if phone sensors should be turned on automatically when starting the app. Useful when you are planning on taking photos.

Run on local pipeline: Toggles local or network mode for 2 camera test and reconstruction

Protocol mode: This option changes communication behavior for 2 camera network test and reconstruction. All other network modes (like 3 camera reconstruction, etc.) use unchecked version.

Calculate matchings: You may uncheck this when reconstructing with predefined tests to make it faster. If not calculated matchings will be read from _matchings.txt (see creating test cases).

Calculate keypoints: You may uncheck this when reconstructing with predefined tests to make it faster. If not calculated keypoints will be read from _kp_desc.dat (see creating test cases). Keypoint detector is much faster now, so just leave it checked.

Run patchfinders parallel: In 2 camera test or reconstruction mode you can run patchfinder sequentally for less memory usage or parallel for faster result. Parallel mode handles out of memory cases. Parallel mode is much faster on multi-core devices.

Save as obj: Save reconstruction result to a Wavefront OBJ format file or not.

Show viewer: Show recontruction result on screen or not. Useful to turn off when running batch tests.

Needed files and folders


Collaborative folder: This is the main folder for the application. It should be located in the root folder of storage (SD card or built in memory).

Files folder: This folder contains device specific files. Should be located in Collaborative.

  • demo_ip.txt: Put ip addresses here of other devices to use in demo modes. In 2 camera modes only the first address will be used. One address per line.

  • camera_matrix.txt: The device`s camera calibration matrix. 9 values, one per line.

  • pose.txt, file.txt: used in BA test mode only.

Images- folders: Pictures taken with the app will be placed in these folder with the sensor data and the calibration matrix of the camera. Should be located in Collaborative.

Images folder: This folder contains the folders for test cases and the last taken image. Should be located in Collaborative.

  • last.jpg: This is the last image taken with the app on this device. This image`s keypoints are also saved here in a file named: last_kp_desc.dat

  • test folders: These folders will appear when u select test in the main screen. See creating test cases.

Results: This folder only contains one subfolder called reconstruction. Should be located in Collaborative.

Reconstuction: This folder contains all the result of reconstructions.

  • result folders: These folders are named after the image the reconstruction used. These folders contains all the results and debug data generated by the application.

    • test_.txt: This file cotains the log of the reconstruction.

    • results.txt: This file cotains the final result of the reconstruction. one plane per line. one normal and a distance per plane.

    • roi_.txt: contains the x and y coordinates of the corner points of the selected plane (x)

    • _patch_.png: segmented grayscale patch

    • patch_rect_uncalib_: rectified segmented patch

    • matchings.txt

    • rectified_b.png

    • .obj: 3d model of the result

    • _filtered.obj: 3d result of the model, planes are filtered with uncertainty module



    • resultPattchFinderImage:

  • Temporary directories for pose and scale estimation modules. These folders will contain the input and output of these modules.

off_demo folder: This folder contains data for offline demo modes, where you can select input from a list of image pairs. Should be located in Collaborative. See creating data for demos.

Creating data for demos


The off_demo folder should contain subfolders. These folders will contain all the data used for offline demo modes. The name of these folders should be numbers only. The images will appear in ascending order of these numbers (See user interfaces: DemoMainActivity).

X is the name of the folder.

X folder should contain the following files:


  • X_a_rgb.jpg, X_b_rgb.jpg: the images of the scene. These images will appear in the main screen. (See user interfaces: DemoMainActivity)

  • camamtx.txt, cambmtx.txt: camera calibration matrices for a and b image.

  • X.obj: previously saved 3d model of the scene (+matX.mtl). Used for OBJ demo.

  • matchings.txt: previously saved keypoint matchings of the images.

Creating test cases/folders


The Images folder should contain subfolders. These folders will contain all the data used for tests. The name of these folders should be numbers only. The tests will appear when you select test in the main screen. (See user interfaces: MainActivity).

X is the name of the folder.

X folder should contain the following files:


  • X_a_rgb.jpg, X_b_rgb.jpg: the images of the scene

  • camamtx.txt, cambmtx.txt: camera calibration matrices for a and b image.

  • roi_.txt: contains the x and y coordinates of the corner points of plane i

  • X_a_patch_.png: segmented grayscale image of plane i

If the above 2 files are missing you will have to select all planes manually every time. (See user interfaces: SegmentationActivity)

  • X_matchings.txt: Needed if you deselect Calculate matchings in Settings.

  • X_a_kp_desc.dat, X_b_kp_desc.dat: Needed if you deselect Calculate keypoints in Settings.




User interfaces


Main Activity:

  • 3 camera reconstruction: Always select secondary devices first. Uses images from gallery.

  • Connect: Press this to participate in 2 camera network reconstruction or test as secondary device.

  • Reconstruction: 2 camera reconstruction (local or network) If local mode is selected in Settings, you may have to select only one image, as the application automatically tries to find its pair based on its name. (123_a_rgb.jpg -> 123_b_rgb.jpg)

  • Test: 2 camera test (local or network) In test mode you dont have to select cornerpoints for planes (if needed files are present, See creating test cases).


Settings:

See Settings.

Segmentation Activity:

There are two states of this activity. The one shown above is the default. It is used to select cornerpoints of the planes to reconstruct by tapping the screen.


  • Delete last point button deletes the last added point of a not closed plane.

  • Restart deletes all points of a not closed plane.

  • Next closes the current plane. It only appears when at least 3 point is added. In network and demo modes it automatically send the closed plane to the reconstruction pipeline.

  • Reconstruct button includes next button`s functionality if possible (at least 3 points and not closed plane), and also finishes this activity.

  • Zoom locked: this button toggles the states of this activity.

The secondary state of this activity is zoom mode. In this mode Zoom locked button changes to Zoom unlocked button (shown below). All other buttons are removed. In this mode you can zoom and scroll the image.



Camera activity:
You can take picture with the big camera icon on the middle right.

After taking an image apreview screen appears with two buttons. The drop button will discard the current image and take you back to the camera. The 3d reconstruction button will save the image and continue the application.

You can access the camera settings with the 2nd icon from the top right (9 white dots).


Demo Main Activity:


This activity uses the data from the offl_demo folder. Image pairs are shown. You can select an image by tapping on it. The behavior depends on the demo mode. (See demo modes)
Reconstruction activity:
The appearance of this activity may vary based on application mode (demo mode), but is will always tell you the current state of the process.


Result activity:
This screen shows the 3d model created from the selected planes. you can rotate the model with touch controls and adjust the zoom with the buttons.
There may be planes not shown on the result. These may be hidden because of the uncertainty module. Planes and the camera objects can be toggled in the menu.

Connecting to other devices


There are two ways of telling the application where to connect in network modes.

The following modes are using the demo_ip.txt (see Needed files) for ip addresses to connect to:



  • online demo

  • 3 camera demo

  • full demo

  • online demo no camera

  • 3 camera demo no camera

  • full demo no camera

  • BA test

  • 3 camera reconstruction in normal mode (See user interfaces: Main activity)

The following modes are using the list found in NotSoDummy.java:

  • 2 camera reconstruction in normal mode (See user interfaces: Main activity)

  • 2 camera test in normal mode (See user interfaces: Main activity)

Important classes


  • hu.u_szeged.inf.camera.main.CameraActivity: Camera activity. It is used for taking images. It has some functions for demo modes.

  • hu.u_szeged.inf.communication.Conn_new: Network communication functions

  • hu.u_szeged.inf.main.MainActivity: The main activity. It has functions to initialize reconstructions and tests too.

  • hu.u_szeged.inf.main.ReconstructionPipeline: The actual pipeline for 2 camera reconstruction.

  • hu.u_szeged.inf.main.demo: This package contains the different demo versions of ReconstructionPipeline and MainActivity.

  • hu.inf.u_szeged.inf.modules: This package contains the actual modules like Asift detection or rectification...

figure A.: Model, local pipeline




ASIFT keypoint detection

Keypoint detection

Keypoint matching

Pose Estimation

Rectification

Patch rectification

Patch rectification

...

PatchFinder

PatchFinder

Reconstruction

Reconstruction

View and save

Reconstruction pipeline



Segmentation

f
Primary device



Secondary device

Keypoint detection

Keypoint matching

Pose Estimation

Rectification

Patch rectification

Patch rectification

...

PatchFinder

PatchFinder

Reconstruction

Reconstruction

View and save

Keypoint detection

View and save

Segmentation
igure B.: 2 camera network reconstruction pipeline

f
Secondary devices



Secondary devices

Keypoint detection

Keypoint matching

Pose Estimation

Rectification

Patch rectification

Patch rectification

...

PatchFinder

PatchFinder

Reconstruction

Reconstruction

View and save

Primary device

Keypoint detection

View and save

Segmentation

Scale Estimation

Merging reconstruction results
igure C.: 3 (or more) camera reconstruction pipeline
Download 57.62 Kb.

Share with your friends:



The database is protected by copyright ©ininet.org 2024
send message
    Main page
current state
final result