OpenCV基于OpenPose的手部關(guān)鍵點(diǎn)檢測

概述

?? 手部關(guān)鍵點(diǎn)檢測，旨在找出給定圖片中手指上的關(guān)節(jié)點(diǎn)及指尖關(guān)節(jié)點(diǎn), 其中手部關(guān)鍵點(diǎn)檢測的應(yīng)用場景主要包括：

• 手勢識別
• 手語識別與理解
• 手部的行為識別

?? Opencv的DNN手部識別主要基于 CMU Perceptual Computing Lab 開源的手部關(guān)鍵點(diǎn)檢測模型OpenPose。

手部關(guān)鍵點(diǎn)檢測器的實(shí)現(xiàn)主要是基于論文：Hand Keypoint Detection in Single Images using Multiview Bootstrapping - CVPR2017

其中，如下：

?? 論文中，首先采用少量標(biāo)注的人手部關(guān)鍵點(diǎn)圖像數(shù)據(jù)集，訓(xùn)練類似于人體姿態(tài)關(guān)鍵點(diǎn)所使用的CPM - Convolutional Pose Machines 網(wǎng)絡(luò)，以得到手部關(guān)鍵點(diǎn)的粗略估計(jì). 采用了 31個(gè) HD 高清攝像頭從不同的視角對人手部進(jìn)行拍攝。然后，將拍攝圖像送入手部關(guān)鍵點(diǎn)檢測器，以初步得到許多粗略的關(guān)鍵點(diǎn)檢測結(jié)果。一旦有了同一手部的不同視角的關(guān)鍵點(diǎn)，則構(gòu)建關(guān)鍵點(diǎn)測量(Keypoint triangulation)，以得到關(guān)鍵點(diǎn)的3D位置。關(guān)鍵點(diǎn)的3D位置被從3D重新投影到每一幅不同視角的 2D 圖片，并采用2D圖像和關(guān)鍵點(diǎn)，進(jìn)一步訓(xùn)練網(wǎng)絡(luò)，以魯棒的預(yù)測手部關(guān)鍵點(diǎn)位置，這對于關(guān)鍵點(diǎn)難以預(yù)測的圖片而言是尤其重要的。采用這種方式，通過少量幾次迭代，即可得到較為準(zhǔn)確的手部關(guān)鍵點(diǎn)檢測器.

? 總之，關(guān)鍵點(diǎn)檢測器和多視角圖像(multi-view images) 一起構(gòu)建了較為準(zhǔn)確的手部關(guān)鍵點(diǎn)檢測模型. 采用的檢測網(wǎng)絡(luò)類似于人體關(guān)鍵點(diǎn)中所用的網(wǎng)絡(luò)結(jié)構(gòu). 進(jìn)度提升的主要因素是采用了多視角圖片標(biāo)注圖片數(shù)據(jù)集.

?? 手部關(guān)鍵點(diǎn)檢測模型共輸出 22 個(gè)關(guān)鍵點(diǎn)，其中包括手部的 21 個(gè)點(diǎn)，第 22 個(gè)點(diǎn)表示背景. 如圖：

手部關(guān)鍵點(diǎn)

代碼示例

??? ?模型文件準(zhǔn)備：

[1] - hand/pose_deploy.prototxt

[2] - hand/pose_iter_102000.caffemodel

??? python代碼：

import os import cv2 import time import numpy as np import matplotlib.pyplot as pltclass general_pose_model(object):def __init__(self, modelpath):self.num_points = 22self.point_pairs = [[0,1],[1,2],[2,3],[3,4],[0,5],[5,6],[6,7],[7,8],[0,9],[9,10],[10,11],[11,12],[0,13],[13,14],[14,15],[15,16],[0,17],[17,18],[18,19],[19,20]]self.inHeight = 368self.threshold = 0.1self.hand_net = self.get_hand_model(modelpath)# 模型加載def get_hand_model(self, modelpath):prototxt = os.path.join(modelpath, "pose_deploy.prototxt")caffemodel = os.path.join(modelpath, "../pose_iter_102000.caffemodel")hand_model = cv2.dnn.readNetFromCaffe(prototxt, caffemodel)return hand_model# 預(yù)測def predict(self, imgfile):img_cv2 = cv2.imread(imgfile)img_height, img_width, _ = img_cv2.shapeaspect_ratio = img_width / img_heightinWidth = int(((aspect_ratio * self.inHeight) * 8) // 8)inpBlob = cv2.dnn.blobFromImage(img_cv2, 1.0 / 255, (inWidth, self.inHeight), (0, 0, 0), swapRB=False, crop=False)self.hand_net.setInput(inpBlob)output = self.hand_net.forward()# vis heatmapsself.vis_heatmaps(imgfile, output)points = []for idx in range(self.num_points):probMap = output[0, idx, :, :] # confidence map.probMap = cv2.resize(probMap, (img_width, img_height))# Find global maxima of the probMap.minVal, prob, minLoc, point = cv2.minMaxLoc(probMap)if prob > self.threshold:points.append((int(point[0]), int(point[1])))else:points.append(None)return points# heatmap可視化def vis_heatmaps(self, imgfile, net_outputs):img_cv2 = cv2.imread(imgfile)plt.figure(figsize=[10, 10])for pdx in range(self.num_points):probMap = net_outputs[0, pdx, :, :]probMap = cv2.resize(probMap, (img_cv2.shape[1], img_cv2.shape[0]))plt.subplot(5, 5, pdx+1)plt.imshow(cv2.cvtColor(img_cv2, cv2.COLOR_BGR2RGB))plt.imshow(probMap, alpha=0.6)plt.colorbar()plt.axis("off")plt.show()# 手部關(guān)鍵點(diǎn)可視化def vis_pose(self, imgfile, points):img_cv2 = cv2.imread(imgfile)img_cv2_copy = np.copy(img_cv2)for idx in range(len(points)):if points[idx]:cv2.circle(img_cv2_copy, points[idx], 8, (0, 255, 255), thickness=-1,lineType=cv2.FILLED)cv2.putText(img_cv2_copy, "{}".format(idx), points[idx], cv2.FONT_HERSHEY_SIMPLEX,1, (0, 0, 0), 2, lineType=cv2.LINE_AA)# 繪制連接點(diǎn)for pair in self.point_pairs:partA = pair[0]partB = pair[1]if points[partA] and points[partB]:cv2.line(img_cv2, points[partA], points[partB], (0, 255, 255), 3)cv2.circle(img_cv2, points[partA], 8, (0, 0, 255), thickness=-1, lineType=cv2.FILLED)plt.figure(figsize=[10, 10])plt.subplot(1, 2, 1)plt.imshow(cv2.cvtColor(img_cv2, cv2.COLOR_BGR2RGB))plt.axis("off")plt.subplot(1, 2, 2)plt.imshow(cv2.cvtColor(img_cv2_copy, cv2.COLOR_BGR2RGB))plt.axis("off")plt.show()if __name__ == '__main__':print("[INFO]Pose estimation.")os.listdir(imgs_path)]img_files = ['hand.jpg']start = time.time()modelpath = ""pose_model = general_pose_model(modelpath)print("[INFO]Model loads time: ", time.time() - start)for img_file in img_files:start = time.time()res_points = pose_model.predict(img_file)print("[INFO]Model predicts time: ", time.time() - start)pose_model.vis_pose(img_file, res_points)print("[INFO]Done.")

[1] - 輸出heatmap：

[2] - 輸出關(guān)鍵點(diǎn)：

OpenCV于OpenPose的人體姿態(tài)估計(jì)

?? 同樣，Opencv也可以結(jié)合OpenPose進(jìn)行人體姿態(tài)估計(jì)，具體實(shí)現(xiàn)和手部關(guān)鍵點(diǎn)檢測類似，只是調(diào)用的模型函數(shù)有所區(qū)別，具體代碼實(shí)現(xiàn)可以參考下文。

?? 模型文件下載：

OpenPose 人體姿態(tài)模型下載路徑：

BODY25: http://posefs1.perception.cs.cmu.edu/OpenPose/models/pose/body_25/pose_iter_584000.caffemodel COCO: http://posefs1.perception.cs.cmu.edu/OpenPose/models/pose/coco/pose_iter_440000.caffemodel MPI: http://posefs1.perception.cs.cmu.edu/OpenPose/models/pose/mpi/pose_iter_160000.caffemodelCOCO prototxt：https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/models/pose/coco/pose_deploy_linevec.prototxt

?? 代碼實(shí)現(xiàn)：

import cv2 import time import numpy as np import matplotlib.pyplot as plt import osclass general_pose_model(object):def __init__(self, modelpath, mode="BODY25"):# 指定采用的模型# Body25: 25 points# COCO: 18 points# MPI: 15 pointsself.inWidth = 368self.inHeight = 368self.threshold = 0.1if mode == "BODY25":self.pose_net = self.general_body25_model(modelpath)elif mode == "COCO":self.pose_net = self.general_coco_model(modelpath)elif mode == "MPI":self.pose_net = self.get_mpi_model(modelpath)def get_mpi_model(self, modelpath):self.points_name = { "Head": 0, "Neck": 1, "RShoulder": 2, "RElbow": 3, "RWrist": 4,"LShoulder": 5, "LElbow": 6, "LWrist": 7, "RHip": 8, "RKnee": 9, "RAnkle": 10, "LHip": 11, "LKnee": 12, "LAnkle": 13, "Chest": 14, "Background": 15 }self.num_points = 15self.point_pairs = [[0, 1], [1, 2], [2, 3], [3, 4], [1, 5], [5, 6], [6, 7], [1, 14],[14, 8], [8, 9], [9, 10], [14, 11], [11, 12], [12, 13]]prototxt = os.path.join(modelpath,"pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt")caffemodel = os.path.join(modelpath, "pose/mpi/pose_iter_160000.caffemodel")mpi_model = cv2.dnn.readNetFromCaffe(prototxt, caffemodel)return mpi_modeldef general_coco_model(self, modelpath):self.points_name = {"Nose": 0, "Neck": 1, "RShoulder": 2, "RElbow": 3, "RWrist": 4,"LShoulder": 5, "LElbow": 6, "LWrist": 7, "RHip": 8, "RKnee": 9, "RAnkle": 10, "LHip": 11, "LKnee": 12, "LAnkle": 13, "REye": 14, "LEye": 15, "REar": 16, "LEar": 17, "Background": 18}self.num_points = 18self.point_pairs = [[1, 0], [1, 2], [1, 5], [2, 3], [3, 4], [5, 6], [6, 7], [1, 8], [8, 9],[9, 10], [1, 11], [11, 12], [12, 13], [0, 14], [0, 15], [14, 16], [15, 17]]prototxt = os.path.join(modelpath, "openpose_pose_coco.prototxt")caffemodel = os.path.join(modelpath, "../pose_iter_440000.caffemodel")print(prototxt, caffemodel)coco_model = cv2.dnn.readNetFromCaffe(prototxt, caffemodel)return coco_modeldef general_body25_model(self, modelpath):self.num_points = 25self.point_pairs = [[1, 0], [1, 2], [1, 5], [2, 3], [3, 4], [5, 6], [6, 7], [0, 15], [15, 17], [0, 16], [16, 18], [1, 8],[8, 9], [9, 10], [10, 11], [11, 22], [22, 23], [11, 24],[8, 12], [12, 13], [13, 14], [14, 19], [19, 20], [14, 21]]prototxt = os.path.join(modelpath, "pose/body_25/pose_deploy.prototxt")caffemodel = os.path.join(modelpath, "pose/body_25/pose_iter_584000.caffemodel")coco_model = cv2.dnn.readNetFromCaffe(prototxt, caffemodel)return coco_modeldef predict(self, imgfile):img_cv2 = cv2.imread(imgfile)img_height, img_width, _ = img_cv2.shapeinpBlob = cv2.dnn.blobFromImage(img_cv2, 1.0 / 255, (self.inWidth, self.inHeight),(0, 0, 0), swapRB=False, crop=False)self.pose_net.setInput(inpBlob)self.pose_net.setPreferableBackend(cv2.dnn.DNN_BACKEND_OPENCV)self.pose_net.setPreferableTarget(cv2.dnn.DNN_TARGET_OPENCL)output = self.pose_net.forward()H = output.shape[2]W = output.shape[3]print(output.shape)# vis heatmapsself.vis_heatmaps(img_file, output)#points = []for idx in range(self.num_points):probMap = output[0, idx, :, :] # confidence map.# Find global maxima of the probMap.minVal, prob, minLoc, point = cv2.minMaxLoc(probMap)# Scale the point to fit on the original imagex = (img_width * point[0]) / Wy = (img_height * point[1]) / Hif prob > self.threshold:points.append((int(x), int(y)))else:points.append(None)return pointsdef vis_heatmaps(self, imgfile, net_outputs):img_cv2 = cv2.imread(imgfile)plt.figure(figsize=[10, 10])for pdx in range(self.num_points):probMap = net_outputs[0, pdx, :, :]probMap = cv2.resize(probMap, (img_cv2.shape[1], img_cv2.shape[0]))plt.subplot(5, 5, pdx+1)plt.imshow(cv2.cvtColor(img_cv2, cv2.COLOR_BGR2RGB))plt.imshow(probMap, alpha=0.6)plt.colorbar()plt.axis("off")plt.show()def vis_pose(self, imgfile, points):img_cv2 = cv2.imread(imgfile)img_cv2_copy = np.copy(img_cv2)for idx in range(len(points)):if points[idx]:cv2.circle(img_cv2_copy, points[idx], 3, (0, 0, 255), thickness=-1,lineType=cv2.FILLED)cv2.putText(img_cv2_copy, "{}".format(idx), points[idx], cv2.FONT_HERSHEY_SIMPLEX,.6, (0, 255, 255), 1, lineType=cv2.LINE_AA)# Draw Skeletonfor pair in self.point_pairs:partA = pair[0]partB = pair[1]if points[partA] and points[partB]:cv2.line(img_cv2, points[partA], points[partB], (0, 255, 0), 3)cv2.circle(img_cv2, points[partA], 3, (0, 0, 255), thickness=-1, lineType=cv2.FILLED)

[1] - 輸出heatmap：

[2] - 輸出姿態(tài)：

------------------------------------------可愛の分割線------------------------------------------

更多Opencv教程可以 Follow github的opencv教程，中文&English 歡迎Star??????

JimmyHHua/opencv_tutorials?github.com

總結(jié)

以上是生活随笔為你收集整理的opencv 图像雾检测_OpenCV图像处理-基于OpenPose的关键点检测的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。