[Python] top lip to nose ratio aspect to detect chewing activity not working as expected

I want to know whether a person is chewing in front of a camera or not using dlib's facemarks. For this end I compute the distance from nose to top lip and derive the mean, then if the that value is bigger than a threshold then we could say that the person is chewing.

To compute the nose to top lip distance I do the following:

def lip_nose_distance(shape):
top_lip = shape[50:54]


nose = shape[32:36]
nose = np.mean(nose, axis=0)
top_lip = np.mean(top_lip, axis=0)

distance = abs(nose[0] - top_lip[0])
return distance


Then iterating of the buffered frame from the video like so:

old_lip_mar1 = 0.0
for (i, landmark) in enumerate(landmark_buffer):

lip_dist = lip_nose_distance(shape)
if old_lip_mar1>0.0:
print("old_lip_mar1 ", old_lip_mar1)
print("lip_dist ", lip_dist)
if abs(lip_dist-old_lip_mar1) > 0.015:
print("lip is moving around too much! so the user is chewing")
else:
print("lip is not moving around too much!")


old_lip_mar1 = lip_dist


The buffers are obtained using:

video_list = os.listdir(VIDEO_PATH) # Read video list

# for vid_name in video_list: # Iterate on video files
vid_path = VIDEO_PATH + "chewing_2.avi"
vid = cv2.VideoCapture(vid_path) # Read video

# Parse into frames
frame_buffer = [] # A list to hold frame images
frame_buffer_color = [] # A list to hold original frame images
while (True):
success, frame = vid.read() # Read frame
if not success:
break # Break if no frame to read left
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) # Convert image into grayscale
frame_buffer.append(gray) # Add image to the frame buffer

vid.release()


landmark_buffer = [] # A list to hold face landmark information
for (i, image) in enumerate(frame_buffer): # Iterate on frame buffer


result = get_rotated_mouth_loc_with_height(image)
landmark = predictor(image, result['rect']) # Detect face landmarks
landmark = shape_to_list(landmark)
landmark_buffer.append(landmark)
shape = result['shape']


The code to work required the following constants and imports:

import dlib
import cv2
import os
from collections import OrderedDict
import numpy as np
from scipy.spatial import distance as dist
FACIAL_LANDMARKS_IDXS = OrderedDict([
("mouth", (48, 68)),
("inner_mouth", (60, 68)),
("right_eyebrow", (17, 22)),
("left_eyebrow", (22, 27)),
("right_eye", (36, 42)),
("left_eye", (42, 48)),
("nose", (27, 36)),
("jaw", (0, 17))
])


Finally an example on which the code could run is found in:

https://streamable.com/lrlchs


For now the error I am getting is that the results are not the ones that are expected (e.g., I get lip is not moving around too much! while they are!) plus the values of the distances don't change accross frames (e.g., ('old_lip_mar1 ', 0.75),('lip_dist ', 0.75) )

Continue reading...