树莓派+OpenCV追踪小球

,

OpenCV是一个基于BSD许可发现的跨平台计算机视觉库，可以运行在Linux、Windows、Android和Mac OS等操作系统上，它轻量级而且高效，由一系列C函数和少量C++类构成，同时提供了Python、Java、Matlab等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法。OpenCV在以下领域有着广泛的应用：

  • 物体识别
  • 人机互动
  • 图像分割
  • 人脸识别
  • 动作识别
  • 运动跟踪
  • 机器人
  • 运动分析
  • 机器视觉
  • 结构分析
  • 汽车安全驾驶

本文主要介绍使用树莓派如何通过摄像头识别颜色，以及实现通过二自由度云台来识别跟踪圆球。主要涉及到的硬件有：

  • 树莓派3B+
  • 500万像素CSI接口摄像头
  • 二自由度舵机云台
  • PAC9685舵机驱动模块
  • 杜邦线若干

树莓派安装OpenCV库参考: Install OpenCV 4 on your Raspberry Pi

安装玩OpenCV之后，测试一下是否正常：

1
2
3
4
workon cv
python
import cv2
cv2.__version__

正常情况下输出如下：

1
'4.4.0'

安装好摄像头之后，需要启用摄像头，通过执行rasp-config选择camera启动，启动之后重启树莓派。可以通过raspivid和raspistill操作摄像头，捕捉视频片段或图像，捕捉到的视频需要通过mplayer播放。也可以使用motion实时查看摄像头内容，如何配置motion本文略。

接下来我们使用OpenCV来测试摄像头，如下python代码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import numpy as np
import cv2

cap = cv2.VideoCapture(0)

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()

    # Flip camera vertically
    frame = cv2.flip(frame, -1)

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    cv2.imshow('frame', frame)
    cv2.imshow('gray', gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

颜色空间

在介绍如何通过OpenCV识别颜色之前，先介绍一下色彩空间(color space)的概念，色彩空间又称为”色域”，每种色彩空间都有自己的色彩模型，OpenCV主要涉及到两种颜色空间：RGB和HSV。

RGB颜色空间以R(red)、G(green)、B(blue)三种颜色为基础，计算机中，每个像素由三个值表示：red、green、blue，每个值范围0-255。例如计算机显示的纯蓝色，RGB表示为(0,0,255)。OpenCV中默认的色彩空间是RGB颜色空间，但是顺序是BGR。

HSV(Hue, Saturation, Value)是根据颜色的直观特性，由A. R. Smith在1978年创建的一种颜色空间，也称六角锥体模型(Hexcone Model)。色调H用角度度量，取值范围为0°～360°,S表示饱和度，也就是色彩的深浅度(0-100%) ,V表示色彩的亮度(0-100%) 。

从人眼分辨的角度来看，HSV颜色空间更容易区分不同颜色，opencv也是使用HSV来进行物体识别。

BGR转换为HSV

以下是来自Color Detection in Python with OpenCV这篇文章的一段转换代码，保存文件为bgr_hsv_converter.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import sys
import numpy as np
import cv2
 
blue = sys.argv[1]
green = sys.argv[2]
red = sys.argv[3]  
 
color = np.uint8([[[blue, green, red]]])
hsv_color = cv2.cvtColor(color, cv2.COLOR_BGR2HSV)
 
hue = hsv_color[0][0][0]
 
print("Lower bound is :"),
print("[" + str(hue-10) + ", 100, 100]\n")
 
print("Upper bound is :"),
print("[" + str(hue + 10) + ", 255, 255]")

这个脚本以BRG值作为参数，输出为一个HSV颜色范围，用于物体识别。有如下颜色的一个球，BGR为(23 136 109)：

执行转换:

1
python bgr_hsv_converter.py 23 136 109

输出一个HSV颜色范围，这个值将在下文用于识别上图中的球体:

1
2
3
4
5
Lower bound is :
[27, 100, 100]

Upper bound is :
[47, 255, 255]

颜色识别

上文已经获取到了小球的颜色区域，下面这段代码将识别到小球并对小球进行颜色遮罩(mask)，代码来自Color Detection in Python with OpenCV:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import cv2
import numpy as np

# Read the picure - The 1 means we want the image in BGR
img = cv2.imread('ball.png', 1) 

# resize imag to 20% in each axis
#img = cv2.resize(img, (0,0), fx=0.2, fy=0.2)
# convert BGR image to a HSV image
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV) 

# NumPy to create arrays to hold lower and upper range 
# The “dtype = np.uint8” means that data type is an 8 bit integer

lower_range = np.array([27, 100, 100], dtype=np.uint8) 
upper_range = np.array([47, 255, 255], dtype=np.uint8)

# create a mask for image
mask = cv2.inRange(hsv, lower_range, upper_range)

# display both the mask and the image side-by-side
cv2.imshow('mask',mask)
cv2.imshow('image', img)

# wait to user to press q
while(1):
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
 
cv2.destroyAllWindows()

遮罩(mask)后的效果:

追踪小球

接下来这段代码是通过摄像头来追踪小球，并且会画出一条小球运动的轨迹，了解详情可阅读 Ball Tracking with OpenCV 这篇文章:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
from collections import deque
import numpy as np
import argparse
from imutils.video.pivideostream import PiVideoStream
import imutils
import cv2
import time

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-b", "--buffer", type=int, default=64,
    help="max buffer size")
args = vars(ap.parse_args())

# define the lower and upper boundaries of the "yellow object"
# (or "ball") in the HSV color space, then initialize the
# list of tracked points
colorLower = (27, 100, 100)
colorUpper = (47, 255, 255)
pts = deque(maxlen=args["buffer"])
 
camera = PiVideoStream(resolution=(320, 240), framerate=32).start()
time.sleep(2.0)

# keep looping
while True:
    # grab the current frame
    frame = camera.read()
 
    # resize the frame, inverted ("vertical flip" w/ 180degrees),
    # blur it, and convert it to the HSV color space
    frame = imutils.resize(frame, width=600)
    frame = imutils.rotate(frame, angle=180)
    blurred = cv2.GaussianBlur(frame, (11, 11), 0)
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    
    # construct a mask for the color "green", then perform
    # a series of dilations and erosions to remove any small
    # blobs left in the mask
    mask = cv2.inRange(hsv, colorLower, colorUpper)
    mask = cv2.erode(mask, None, iterations=2)
    mask = cv2.dilate(mask, None, iterations=2)
    
    # find contours in the mask and initialize the current
    # (x, y) center of the ball
    cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL,
        cv2.CHAIN_APPROX_SIMPLE)[-2]
    center = None
    
    # only proceed if at least one contour was found
    if len(cnts) > 0:
        # find the largest contour in the mask, then use
        # it to compute the minimum enclosing circle and
        # centroid
        c = max(cnts, key=cv2.contourArea)
        ((x, y), radius) = cv2.minEnclosingCircle(c)
        M = cv2.moments(c)
        center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))
        
        # only proceed if the radius meets a minimum size
        if radius > 10:
            # draw the circle and centroid on the frame,
            # then update the list of tracked points
            cv2.circle(frame, (int(x), int(y)), int(radius),
                (0, 255, 255), 2)
            cv2.circle(frame, center, 5, (0, 0, 255), -1)
    
    # update the points queue
    pts.appendleft(center)
    
    # loop over the set of tracked points
    for i in range(1, len(pts)):
        # if either of the tracked points are None, ignore
        # them
        if pts[i - 1] is None or pts[i] is None:
        	continue
        
        # otherwise, compute the thickness of the line and
        # draw the connecting lines
        thickness = int(np.sqrt(args["buffer"] / float(i + 1)) * 2.5)
        cv2.line(frame, pts[i - 1], pts[i], (0, 0, 255), thickness)
    
    # show the frame to our screen
    cv2.imshow("Frame", frame)
    key = cv2.waitKey(1) & 0xFF
    
    # if the 'q' key is pressed, stop the loop
    if key == ord("q"):
        break
 
# cleanup the camera and close any open windows
camera.stop()
cv2.destroyAllWindows()

确定小球坐标

以下代码用来确定小球在窗口的坐标，用于下文追踪小球确定是否偏离中心点，使用600 * 450分辨率：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
# import the necessary packages
from __future__ import print_function
from imutils.video.pivideostream import PiVideoStream
import imutils
import time
import cv2
import os
import RPi.GPIO as GPIO

# print object coordinates
def mapObjectPosition(x, y):
    print ("[INFO] Object Center coordenates at X0 = {0} and Y0 =  {1}".format(x, y))

# initialize the video stream and allow the camera sensor to warmup
print("[INFO] waiting for camera to warmup...")
#vs = VideoStream(0).start()
vs = PiVideoStream(resolution=(320, 240), framerate=32).start()
time.sleep(2.0)

# define the lower and upper boundaries of the object
# to be tracked in the HSV color space
colorLower = (27, 100, 100)
colorUpper = (47, 255, 255)

# loop over the frames from the video stream
while True:
    # grab the next frame from the video stream, Invert 180o, resize the
    # frame, and convert it to the HSV color space
    frame = vs.read()
    frame = imutils.resize(frame, width=600)
    frame = imutils.rotate(frame, angle=180)
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    
    # construct a mask for the object color, then perform
    # a series of dilations and erosions to remove any small
    # blobs left in the mask
    mask = cv2.inRange(hsv, colorLower, colorUpper)
    mask = cv2.erode(mask, None, iterations=2)
    mask = cv2.dilate(mask, None, iterations=2)
    
    # find contours in the mask and initialize the current
    # (x, y) center of the object
    cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL,
        cv2.CHAIN_APPROX_SIMPLE)[-2]
    #cnts = cnts[0] if imutils.is_cv2() else cnts[1]
    center = None
    
    # only proceed if at least one contour was found
    if len(cnts) > 0:
        # find the largest contour in the mask, then use
        # it to compute the minimum enclosing circle and
        # centroid
        c = max(cnts, key=cv2.contourArea)
        ((x, y), radius) = cv2.minEnclosingCircle(c)
        M = cv2.moments(c)
        center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))
        
        # only proceed if the radius meets a minimum size
        if radius > 10:
            # draw the circle and centroid on the frame,
            # then update the list of tracked points
            cv2.circle(frame, (int(x), int(y)), int(radius),
              (0, 255, 255), 2)
            cv2.circle(frame, center, 5, (0, 0, 255), -1)
            
            # position Servo at center of circle
            mapObjectPosition(int(x), int(y))
            
    # if the ball is not detected
    else:
        print("ball not detected")
    
    # show the frame to our screen
    cv2.imshow("Frame", frame)
    
    # if [ESC] key is pressed, stop the loop
    key = cv2.waitKey(1) & 0xFF
    if key == 27:
        break

# do a bit of cleanup
print("\n[INFO] Exiting Program and cleanup stuff \n")
cv2.destroyAllWindows()
vs.stop()

效果如下:

追踪小球

通过上节获取到小球的坐标，如果小球坐标便宜窗口中心超出一定范围，就驱动舵机转动摄像头，代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
# import the necessary packages
from __future__ import print_function
from imutils.video.pivideostream import PiVideoStream
import Adafruit_PCA9685
import argparse
import imutils
import time
import cv2
import os
import RPi.GPIO as GPIO

#define Servos
panServo = 4
tiltServo = 5

pwm = Adafruit_PCA9685.PCA9685()
pwm.set_pwm_freq(50)

#position servos 
def positionServo(channel, angle):
    pulse = int(4096*((angle*11)+500)/20000)
    pwm.set_pwm(channel, 0, pulse)
    print("[INFO] Positioning servo at channel {0} to {1} degrees\n".format(channel, angle))

# position servos to present object at center of the frame
def mapServoPosition(x, y):
    global panAngle
    global tiltAngle
    if (x < 270):
        panAngle += 5
        print(panAngle)
        if panAngle > 160:
            panAngle = 160
        positionServo(panServo, panAngle)
 
    if (x > 310):
        panAngle -= 5
        print(panAngle)
        if panAngle < 20:
            panAngle = 20
        positionServo(panServo, panAngle)

    if (y < 190):
        tiltAngle -= 5
        print(tiltAngle)
        if tiltAngle > 160:
            tiltAngle = 160
        positionServo(tiltServo, tiltAngle)
 
    if (y > 240):
        tiltAngle += 5
        print(tiltAngle)
        if tiltAngle < 20:
            tiltAngle = 20
        positionServo(tiltServo, tiltAngle)

# initialize the video stream and allow the camera sensor to warmup
print("[INFO] waiting for camera to warmup...")
vs = PiVideoStream(resolution=(320, 240), framerate=32).start()
time.sleep(2.0)

# define the lower and upper boundaries of the object
# to be tracked in the HSV color space
colorLower = (27, 100, 100)
colorUpper = (47, 255, 255)

# Initialize angle servos at 90-90 position
global panAngle
panAngle = 90
global tiltAngle
tiltAngle =90

# positioning Pan/Tilt servos at initial position
positionServo(panServo, panAngle)
positionServo(tiltServo, tiltAngle)

# loop over the frames from the video stream
while True:
    # grab the next frame from the video stream, Invert 180o, resize the
    # frame, and convert it to the HSV color space
    frame = vs.read()
    frame = imutils.resize(frame, width=600)
    frame = cv2.flip(frame, -1)
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

    # construct a mask for the object color, then perform
    # a series of dilations and erosions to remove any small
    # blobs left in the mask
    mask = cv2.inRange(hsv, colorLower, colorUpper)
    mask = cv2.erode(mask, None, iterations=2)
    mask = cv2.dilate(mask, None, iterations=2)
    
    # find contours in the mask and initialize the current
    # (x, y) center of the object
    cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL,
        cv2.CHAIN_APPROX_SIMPLE)[-2]
    #cnts = cnts[0] if imutils.is_cv2() else cnts[1]
    center = None
    
    # only proceed if at least one contour was found
    if len(cnts) > 0:
        # find the largest contour in the mask, then use
        # it to compute the minimum enclosing circle and
        # centroid
        c = max(cnts, key=cv2.contourArea)
        ((x, y), radius) = cv2.minEnclosingCircle(c)
        M = cv2.moments(c)
        center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))
    
        # only proceed if the radius meets a minimum size
        if radius > 10:
            # draw the circle and centroid on the frame,
            # then update the list of tracked points
            cv2.circle(frame, (int(x), int(y)), int(radius),
                (0, 255, 255), 2)
            cv2.circle(frame, center, 5, (0, 0, 255), -1)
    
            # position Servo at center of circle
            mapServoPosition(int(x), int(y))
            
    # if the ball is not detected
    else:
        print("ball not detected")
        
    # show the frame to our screen
    cv2.imshow("Frame", frame)
    
    # if [ESC] key is pressed, stop the loop
    key = cv2.waitKey(1) & 0xFF
    if key == 27:
        break

# do a bit of cleanup
print("\n[INFO] Exiting Program and cleanup stuff \n")
positionServo(panServo, 90)
positionServo(tiltServo, 90)
cv2.destroyAllWindows()
vs.stop()

效果如下：

本文主要参考Automatic Vision Object Tracking，代码有做些改进调整。

参考