Tutorial 5: GUI interactions (masks, lines and drawing tools)¶
Phenopype uses OpenCV’s High GUI module to display images and to allow users to interact with images. High GUI has a few pros and cons:
+ native OpenCV GUI (no extra GUI libraries required)
+ the module is extremely fast in displaying images
+ it can display very large image-arrays (> 10000x10000 pixels)
+ it can display multiple images side by side
+ interactions (drawing and measuring) are possible
- sometimes unstable (e.g. windows are not closed but freeze)
- issues with cross plattform stability (e.g. on macOS)
- displaying instructions is hacky (text is "painted" onto a displayed image)
- user input (key strokes and mouse clicks) sometimes isn't captured properly
Currently Phenopype uses the standard GUI libraries that ship with the most recent precompiled opencv-contrib-python
that are listed on the Python package index, which is Qt
for Linux and macOS, and Win32 UI
on Windows. The Qt
GUI is a bit more userfriendly with builtin buttons, scrollbars, RGB info and zoom (see the OpenCV docs), but you don’t actually need those
things for Phenopype GUI interactions.
Notes regarding High GUI windows in Phenopype
If you want to close a High GUI window, use
Enter
If you want to close a High GUI window and quit the Phenoype process that invoked it, use
Esc
If your High GUI window is frozen, use
Esc
. If the window still does’t close, typecv2.destroyAllWindows()
into the consoleIf the last step doesn’t work, restart the kernel
Show images¶
To show an image, it first has to be loaded as an array using load_image
. The array can then be passed on to show_image
, which simply displays an image (no interactions, except zooming in using the mousewheel). The window is closed by keystroke (Enter
, or Esc
which closes the window and ends ongoing processes).
[1]:
import phenopype as pp
import os
[ ]:
img = pp.load_image("images/stickle1.jpg")
pp.show_image(img,
window_aspect="free" # resize the window
)
However, show_image
can also handle multiple arrays. Here we loop through the images folder, attach all images to a list, and pass that list of arrays to the functions - it will give a warning if more than 10 images are being opened at the same time.
[ ]:
img_list = []
for i in os.listdir("images"):
img = pp.load_image(os.path.join("images",i))
img_list.append(img)
pp.show_image(img_list)
The function has a few more options:
[ ]:
pp.show_image(img_list,
max_dim=250, # maximu dimension (in either direction) for the windows
check=False, # don't issue warning if more than 10 images are opened
position_offset=50, # window offset if multiple windows are displayed
position_reset=False) # don't reset position of windows (i.e. window position will be remembered)
Create masks¶
Masking, i.e. removing unwanted parts of an image that contain noise by including or excluding certain parts of the image, is an important preprocessing step in any computer vision workflow. Phenopype’s create_mask
tool provides flexibility when drawing masks.
Fig. 1: Phenopype’s mask tool in action. You can include or exclude certain parts of the image; the resulting coordinates are recognized in subsequent computer vision steps (e.g. thresholding)
Using create_mask
results in a DataFrame object that contains coordinates of the created mask. You can add multiple “submasks” that belong to the same mask layer that don’t have to be connected. Finish with Enter
.
[ ]:
img = pp.load_image(os.path.join("images",'isopods.jpg'))
masks = pp.preprocessing.create_mask(img, label="tray")
masks
The masks DataFrame can also be attached to an existing DataFrame, when provided via df_image_data
.
[ ]:
img,df = pp.load_image(os.path.join("images",'isopods.jpg'), df=True)
masks = pp.preprocessing.create_mask(img, label="tray",df_image_data=df)
masks
Creating masks like this corresponds to the prototyping workflow, which explicitly requires a subsequent call of visualization functions to show the mask that was just created.
[ ]:
canvas = pp.visualization.draw_masks(img, df_masks=masks)
pp.show_image(canvas)
Existing mask DataFrames can be updated if they are provided via df_mask
.
[ ]:
masks = pp.preprocessing.create_mask(img,
df_masks=masks,
label="scale",
include=False # will be automatically drawn in red
)
canvas = pp.visualization.draw_masks(img,
df_masks=masks,
label=True,
label_colour="black",
label_size=3,
line_width=2,
label_width=4)
pp.show_image(canvas)
Try the polygon
tool - single polygons are finished with Ctrl
, finish with Enter
- the last open polyon will automatically be completed.
[ ]:
masks = pp.preprocessing.create_mask(img, df_masks=masks, tool="polygon")
canvas = pp.visualization.draw_masks(img, df_masks=masks)
pp.show_image(canvas)
Landmarks, lines, and drawings¶
Functional morphology of organisms is often measured by placing landmarks at specific points that show structural, functional or developmental significance. In Phenopype, this is done using the landmark
tool.
Fig. 2: Phenopype’s landmark tool (here called with the high throughput method).
[2]:
img = pp.load_image(os.path.join("images",'stickle1.jpg'))
df_lm = pp.measurement.landmarks(img)
dirpath defaulted to file directory - E:\git_repos\phenopype\tutorials\images
Directory to save files set at - E:\git_repos\phenopype\tutorials\images
- setting landmarks
There are a few more options …
[3]:
df_lm = pp.measurement.landmarks(img,
point_size=15,
point_colour="green",
label_size=2,
label_width=3,
label_colour="blue")
- setting landmarks
… and like for the masks, we need to explicitly call a drawing function.
[4]:
canvas = pp.visualization.draw_landmarks(img, df_landmarks=df_lm,
point_size=15,
point_colour="green",
label_size=2,
label_width=3,
label_colour="blue")
pp.show_image(canvas)
Similiar to the polygon
tool in the create_mask
function, one can use the draw_polyline
tool to measure an object:
[5]:
df_lines = pp.measurement.polylines(img,
line_width=2,
line_colour="green")
canvas = pp.visualization.draw_polylines(canvas, df_polylines=df_lines,
line_width=2,
line_colour="green")
pp.show_image(canvas)
- draw polylines
Similar to this function is the draw
tool, which is useful to separate objects after binarization. Here, for example, use a combination of create_mask
and draw
to detect and separate connected stickleback plates: first draw a mask, then do the segmentation. The draw
tool is used directly on the binary image, and always creates lines with two points (or rectangles when tool="rectangle"
is selected. Then find the contours, and show the result.
[6]:
image = pp.load_image(os.path.join("images",'stickle1.jpg'))
mask = pp.preprocessing.create_mask(image, tool="polygon")
dirpath defaulted to file directory - E:\git_repos\phenopype\tutorials\images
Directory to save files set at - E:\git_repos\phenopype\tutorials\images
- create mask
[7]:
image_bin = pp.segmentation.threshold(image,
method="adaptive",
channel="red",
blocksize=199,
constant=5,
df_masks=mask)
image_morph = pp.segmentation.morphology(image_bin,
operation="open",
shape="ellipse",
kernel_size=3,
iterations=1)
- include mask "mask1" pixels
[8]:
image_bin_draw = pp.segmentation.draw(image_morph)
- draw polylines
[9]:
contours = pp.segmentation.find_contours(image_bin_draw, retrieval="ext", min_area=150)
image_drawn = pp.visualization.draw_contours(image, df_contours=contours)
pp.show_image(image_drawn)
Found 20 contours that match criteria.
Data entry¶
For high data quality it is important to verify the ID of the specimen in the current picture. Often, the picture name contains the ID, but typically an label is placed inside the image. Using the enter_data
tool we open the image and an entry prompt that will create a column with a name of our chosing inside all exported results.
[9]:
image, df_img_data = pp.load_image(os.path.join("images",'stickle1.jpg'), df=True)
[11]:
df_img_data = pp.preprocessing.enter_data(image, df_img_data, columns="ID")
- add column ID
[11]:
filename | width | height | size_ratio_original | ID | |
---|---|---|---|---|---|
0 | stickle1.jpg | 2400 | 1600 | 1 | 142501 |
[ ]: