Tutorial 5: GUI interactions (masks, lines and drawing tools)

Phenopype uses OpenCV’s High GUI module to display images and to allow users to interact with images. High GUI has a few pros and cons:

+ native OpenCV GUI (no extra GUI libraries required)
+ the module is extremely fast in displaying images
+ it can display very large image-arrays (> 10000x10000 pixels)
+ it can display multiple images side by side
+ interactions (drawing and measuring) are possible

- sometimes unstable (e.g. windows are not closed but freeze)
- issues with cross plattform stability (e.g. on macOS)
- displaying instructions is hacky (text is "painted" onto a displayed image)
- user input (key strokes and mouse clicks) sometimes isn't captured properly

Currently Phenopype uses the standard GUI libraries that ship with the most recent precompiled opencv-contrib-python that are listed on the Python package index, which is Qt for Linux and macOS, and Win32 UI on Windows. The Qt GUI is a bit more userfriendly with builtin buttons, scrollbars, RGB info and zoom (see the OpenCV docs), but you don’t actually need those things for Phenopype GUI interactions.

Notes regarding High GUI windows in Phenopype

• If you want to close a High GUI window, use Enter

• If you want to close a High GUI window and quit the Phenoype process that invoked it, use Esc

• If your High GUI window is frozen, use Esc. If the window still does’t close, type cv2.destroyAllWindows() into the console

• If the last step doesn’t work, restart the kernel

Show images

To show an image, it first has to be loaded as an array using load_image. The array can then be passed on to show_image, which simply displays an image (no interactions, except zooming in using the mousewheel). The window is closed by keystroke (Enter, or Esc which closes the window and ends ongoing processes).

import phenopype as pp
import os

img = pp.load_image("images/stickle1.jpg")
pp.show_image(img,
              window_aspect="free" # resize the window
             )

However, show_image can also handle multiple arrays. Here we loop through the images folder, attach all images to a list, and pass that list of arrays to the functions - it will give a warning if more than 10 images are being opened at the same time.

img_list = []
for i in os.listdir("images"):
    img = pp.load_image(os.path.join("images",i))
    img_list.append(img)
pp.show_image(img_list)

The function has a few more options:

pp.show_image(img_list,
              max_dim=250,           # maximu dimension (in either direction) for the windows
              check=False,           # don't issue warning if more than 10 images are opened
              position_offset=50,    # window offset if multiple windows are displayed
              position_reset=False)  # don't reset position of windows (i.e. window position will be remembered)

Create masks

Masking, i.e. removing unwanted parts of an image that contain noise by including or excluding certain parts of the image, is an important preprocessing step in any computer vision workflow. Phenopype’s create_mask tool provides flexibility when drawing masks.

Fig. 1: Phenopype’s mask tool in action. You can include or exclude certain parts of the image; the resulting coordinates are recognized in subsequent computer vision steps (e.g. thresholding)

Using create_mask results in a DataFrame object that contains coordinates of the created mask. You can add multiple “submasks” that belong to the same mask layer that don’t have to be connected. Finish with Enter.

img = pp.load_image(os.path.join("images",'isopods.jpg'))
masks = pp.preprocessing.create_mask(img, label="tray")
masks

The masks DataFrame can also be attached to an existing DataFrame, when provided via df_image_data.

img,df = pp.load_image(os.path.join("images",'isopods.jpg'), df=True)
masks = pp.preprocessing.create_mask(img, label="tray",df_image_data=df)
masks

Creating masks like this corresponds to the prototyping workflow, which explicitly requires a subsequent call of visualization functions to show the mask that was just created.

canvas = pp.visualization.draw_masks(img, df_masks=masks)
pp.show_image(canvas)

Existing mask DataFrames can be updated if they are provided via df_mask.

masks = pp.preprocessing.create_mask(img,
                                     df_masks=masks,
                                     label="scale",
                                     include=False # will be automatically drawn in red
                                    )
canvas = pp.visualization.draw_masks(img,
                                     df_masks=masks,
                                     label=True,
                                     label_colour="black",
                                     label_size=3,
                                     line_width=2,
                                     label_width=4)
pp.show_image(canvas)

Try the polygon tool - single polygons are finished with Ctrl, finish with Enter - the last open polyon will automatically be completed.

masks = pp.preprocessing.create_mask(img, df_masks=masks, tool="polygon")
canvas = pp.visualization.draw_masks(img, df_masks=masks)
pp.show_image(canvas)

Landmarks, lines, and drawings

Functional morphology of organisms is often measured by placing landmarks at specific points that show structural, functional or developmental significance. In Phenopype, this is done using the landmark tool.

Fig. 2: Phenopype’s landmark tool (here called with the high throughput method).

img = pp.load_image(os.path.join("images",'stickle1.jpg'))
df_lm = pp.measurement.landmarks(img)

There are a few more options …

df_lm = pp.measurement.landmarks(img,
                                 point_size=15,
                                 point_colour="green",
                                 label_size=2,
                                 label_width=3,
                                 label_colour="blue")

… and like for the masks, we need to explicitly call a drawing function.

canvas = pp.visualization.draw_landmarks(img, df_landmarks=df_lm,
                                         point_size=15,
                                         point_colour="green",
                                         label_size=2,
                                         label_width=3,
                                         label_colour="blue")
pp.show_image(canvas)

Similiar to the polygon tool in the create_mask function, one can use the draw_polyline tool to measure an object:

df_lines = pp.measurement.polylines(img,
                                 line_width=2,
                                 line_colour="green")
canvas = pp.visualization.draw_polylines(canvas, df_polylines=df_lines,
                                         line_width=2,
                                         line_colour="green")
pp.show_image(canvas)

Similar to this function is the draw tool, which is useful to separate objects after binarization. Here, for example, use a combination of create_mask and draw to detect and separate connected stickleback plates: first draw a mask, then do the segmentation. The draw tool is used directly on the binary image, and always creates lines with two points (or rectangles when tool="rectangle" is selected. Then find the contours, and show the result.

image = pp.load_image(os.path.join("images",'stickle1.jpg'))
mask = pp.preprocessing.create_mask(image, tool="polygon")

- create mask mask1

image_bin = pp.segmentation.threshold(image,
                                      method="adaptive",
                                      channel="red",
                                      blocksize=199,
                                      constant=5,
                                      df_masks=mask)
image_morph = pp.segmentation.morphology(image_bin,
                                         operation="open",
                                         shape="ellipse",
                                         kernel_size=3,
                                         iterations=1)

- include mask "mask1" pixels

image_bin_draw = pp.segmentation.draw(image_morph)

- draw polylines

contours = pp.segmentation.find_contours(image_bin_draw,  retrieval="ext", min_area=150)
image_drawn = pp.visualization.draw_contours(image, df_contours=contours)
pp.show_image(image_drawn)

Data entry

For high data quality it is important to verify the ID of the specimen in the current picture. Often, the picture name contains the ID, but typically an label is placed inside the image. Using the enter_data tool we open the image and an entry prompt that will create a column with a name of our chosing inside all exported results.

image, df_img_data = pp.load_image(os.path.join("images",'stickle1.jpg'), df=True)

df_img_data = pp.preprocessing.enter_data(image, df_img_data, columns="ID")

- add column ID

	filename	width	height	size_ratio_original	ID
0	stickle1.jpg	2400	1600	1	142501