Tutorial 5: GUI interactions (masks, lines and drawing tools)¶
Phenopype uses OpenCV’s High GUI module to display images and to allow users to interact with images. High GUI has a few pros and cons:
+ native OpenCV GUI (no extra GUI libraries required) + the module is extremely fast in displaying images + it can display very large image-arrays (> 10000x10000 pixels) + it can display multiple images side by side + interactions (drawing and measuring) are possible - sometimes unstable (e.g. windows are not closed but freeze) - issues with cross plattform stability (e.g. on macOS) - displaying instructions is hacky (text is "painted" onto a displayed image) - user input (key strokes and mouse clicks) sometimes isn't captured properly
Currently Phenopype uses the standard GUI libraries that ship with the most recent precompiled
opencv-contrib-python that are listed on the Python package index, which is
Qt for Linux and macOS, and
Win32 UI on Windows. The
Qt GUI is a bit more userfriendly with builtin buttons, scrollbars, RGB info and zoom (see the OpenCV docs), but you don’t actually need those
things for Phenopype GUI interactions.
Notes regarding High GUI windows in Phenopype
If you want to close a High GUI window, use
If you want to close a High GUI window and quit the Phenoype process that invoked it, use
If your High GUI window is frozen, use
Esc. If the window still does’t close, type
cv2.destroyAllWindows()into the console
If the last step doesn’t work, restart the kernel
To show an image, it first has to be loaded as an array using
load_image. The array can then be passed on to
show_image, which simply displays an image (no interactions, except zooming in using the mousewheel). The window is closed by keystroke (
Esc which closes the window and ends ongoing processes).
import phenopype as pp import os
img = pp.load_image("images/stickle1.jpg") pp.show_image(img, window_aspect="free" # resize the window )
show_image can also handle multiple arrays. Here we loop through the images folder, attach all images to a list, and pass that list of arrays to the functions - it will give a warning if more than 10 images are being opened at the same time.
img_list =  for i in os.listdir("images"): img = pp.load_image(os.path.join("images",i)) img_list.append(img) pp.show_image(img_list)
The function has a few more options:
pp.show_image(img_list, max_dim=250, # maximu dimension (in either direction) for the windows check=False, # don't issue warning if more than 10 images are opened position_offset=50, # window offset if multiple windows are displayed position_reset=False) # don't reset position of windows (i.e. window position will be remembered)
Masking, i.e. removing unwanted parts of an image that contain noise by including or excluding certain parts of the image, is an important preprocessing step in any computer vision workflow. Phenopype’s
create_mask tool provides flexibility when drawing masks.
Fig. 1: Phenopype’s mask tool in action. You can include or exclude certain parts of the image; the resulting coordinates are recognized in subsequent computer vision steps (e.g. thresholding)
create_mask results in a DataFrame object that contains coordinates of the created mask. You can add multiple “submasks” that belong to the same mask layer that don’t have to be connected. Finish with
img = pp.load_image(os.path.join("images",'isopods.jpg')) masks = pp.preprocessing.create_mask(img, label="tray") masks
The masks DataFrame can also be attached to an existing DataFrame, when provided via
img,df = pp.load_image(os.path.join("images",'isopods.jpg'), df=True) masks = pp.preprocessing.create_mask(img, label="tray",df_image_data=df) masks
Creating masks like this corresponds to the prototyping workflow, which explicitly requires a subsequent call of visualization functions to show the mask that was just created.
canvas = pp.visualization.draw_masks(img, df_masks=masks) pp.show_image(canvas)
Existing mask DataFrames can be updated if they are provided via
masks = pp.preprocessing.create_mask(img, df_masks=masks, label="scale", include=False # will be automatically drawn in red ) canvas = pp.visualization.draw_masks(img, df_masks=masks, label=True, label_colour="black", label_size=3, line_width=2, label_width=4) pp.show_image(canvas)
polygon tool - single polygons are finished with
Ctrl, finish with
Enter - the last open polyon will automatically be completed.
masks = pp.preprocessing.create_mask(img, df_masks=masks, tool="polygon") canvas = pp.visualization.draw_masks(img, df_masks=masks) pp.show_image(canvas)
Landmarks, lines, and drawings¶
Functional morphology of organisms is often measured by placing landmarks at specific points that show structural, functional or developmental significance. In Phenopype, this is done using the
Fig. 2: Phenopype’s landmark tool (here called with the high throughput method).
img = pp.load_image(os.path.join("images",'stickle1.jpg')) df_lm = pp.measurement.landmarks(img)
dirpath defaulted to file directory - E:\git_repos\phenopype\tutorials\images Directory to save files set at - E:\git_repos\phenopype\tutorials\images - setting landmarks
There are a few more options …
df_lm = pp.measurement.landmarks(img, point_size=15, point_colour="green", label_size=2, label_width=3, label_colour="blue")
- setting landmarks
… and like for the masks, we need to explicitly call a drawing function.
canvas = pp.visualization.draw_landmarks(img, df_landmarks=df_lm, point_size=15, point_colour="green", label_size=2, label_width=3, label_colour="blue") pp.show_image(canvas)
Similiar to the
polygon tool in the
create_mask function, one can use the
draw_polyline tool to measure an object:
df_lines = pp.measurement.polylines(img, line_width=2, line_colour="green") canvas = pp.visualization.draw_polylines(canvas, df_polylines=df_lines, line_width=2, line_colour="green") pp.show_image(canvas)
- draw polylines
Similar to this function is the
draw tool, which is useful to separate objects after binarization. Here, for example, use a combination of
draw to detect and separate connected stickleback plates: first draw a mask, then do the segmentation. The
draw tool is used directly on the binary image, and always creates lines with two points (or rectangles when
tool="rectangle" is selected. Then find the contours, and show the result.
image = pp.load_image(os.path.join("images",'stickle1.jpg')) mask = pp.preprocessing.create_mask(image, tool="polygon")
dirpath defaulted to file directory - E:\git_repos\phenopype\tutorials\images Directory to save files set at - E:\git_repos\phenopype\tutorials\images - create mask
image_bin = pp.segmentation.threshold(image, method="adaptive", channel="red", blocksize=199, constant=5, df_masks=mask) image_morph = pp.segmentation.morphology(image_bin, operation="open", shape="ellipse", kernel_size=3, iterations=1)
- include mask "mask1" pixels
image_bin_draw = pp.segmentation.draw(image_morph)
- draw polylines
contours = pp.segmentation.find_contours(image_bin_draw, retrieval="ext", min_area=150) image_drawn = pp.visualization.draw_contours(image, df_contours=contours) pp.show_image(image_drawn)
Found 20 contours that match criteria.
For high data quality it is important to verify the ID of the specimen in the current picture. Often, the picture name contains the ID, but typically an label is placed inside the image. Using the
enter_data tool we open the image and an entry prompt that will create a column with a name of our chosing inside all exported results.
image, df_img_data = pp.load_image(os.path.join("images",'stickle1.jpg'), df=True)
df_img_data = pp.preprocessing.enter_data(image, df_img_data, columns="ID")
- add column ID