Real-Time Object Detection using TensorFlow API
Computer vision is one of the important and demanding fields in Artificial intelligence. Computer vision is an interdisciplinary scientific field that deals with how computers can gain a high-level understanding of digital images or videos. Computer Vision is also composed of various aspects such as image recognition, object detection, image generation, image super-resolution, and more. Object detection is probably the most profound aspect of computer vision due to the number of practical use cases.
Object detection refers to the capability of computer and software systems to locate objects in an image/scene and identify each object. Object detection has been widely used for face detection, vehicle detection, pedestrian counting, web images, security systems, and driverless cars. There are many ways object detection can be used as well in many fields of practice. Like every other computer technology, a wide range of creative and amazing uses of object detection will definitely come from the efforts of computer programmers and software developers.
To perform object detection using TensorFlow API, all you need to do is
Install Python and other required packages on your computer system
Install TensorFlow object detection API
import all the libraries in the program
load detection model
test model with an image
Experience real-time object detection
for complete source code click here
Step 1: Install Required Libraries
1. Install the following dependencies via pip:
TensorFlow
pip install tensorflow
b. OpenCV
pip install opencv-pythonc. pycocotools
pip install pycocotools
Step 2: Install TensorFlow Object Detection API
Clone the TensorFlow object detection repository on the github.com/tensorflow/models page.
Download the zip and Extract it in the project directory.
For setting up the environment for object detection we have to do some stuff to install TensorFlow object detection API in the environment
from IPython.conftest import get_ipython get_ipython().run_cell_magic('bash', '', 'cd /models/research/\nprotoc object_detection/protos/*.proto --python_out=.') get_ipython().run_cell_magic('bash', '', 'cd /models/research\npip install .')
Step 3: Import required python libraries
import installed required python libraries and packages
import numpy as np import os import six.moves.urllib as urllib import sys import tarfile import tensorflow as tf import zipfile import pathlib from collections import defaultdict from io import StringIO from matplotlib import pyplot as plt from PIL import Image from IPython.display import display get_ipython().system('pip install --user tf_slim')
Step 4: Import object detection libraries
import all required packages from object_detection library
from object_detection.utils import ops as utils_ops from object_detection.utils import label_map_util from object_detection.utils import visualization_utils as vis_util
Patches:
# patch tf1 into `utils.ops` utils_ops.tf = tf.compat.v1 # Patch the location of gfile tf.gfile = tf.io.gfile
Step 5: Download Pre-Trained Model
To begin with, we need to download the latest pre-trained network for the model we wish to use. This can be done by simply clicking on the name of the desired model in the table found in the TensorFlow 2 Detection Model Zoo. Clicking on the name of your model should initiate a download for a *.tar.gz file.
Once the *.tar.gz file has been downloaded, open it using a decompression program of your choice (e.g. 7zip, WinZIP, etc.). Next, open the *.tar folder that you see when the compressed folder is opened, and extract its contents inside the folder training_demo/pre-trained-models. Since we downloaded the SSD ResNet50 V1 FPN 640x640 model, our training_demo directory should now look as follows:
training_demo/ ├─ ... ├─ pre-trained-models/ │ └─ ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/ │ ├─ checkpoint/ │ ├─ saved_model/ │ └─ pipeline.config └─ ...
The function to load the model in the environment by passing the name of the saved model:
def load_model(model_name): path = './pre-trained-models/' model_dir = path + model_name model_dir = pathlib.Path(model_dir) / "saved_model" model = tf.saved_model.load(str(model_dir)) model = model.signatures['serving_default'] return model
Step 6: Loading Label Map
Label maps map indices to category names so that when our convolution network predicts 5, we know that this corresponds to the airplane. Here we use internal utility functions, but anything that returns a dictionary mapping integers to appropriate string labels would be fine
# List of the strings that is used to add correct label for each box. PATH_TO_LABELS = './models/research/object_detection/data/mscoco_label_map.pbtxt' category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)
Step 7: Generate Inference Function
Add a wrapper function to call the model, and cleanup the outputs:
def run_inference_for_single_image(model, image): image = np.asarray(image) # The input needs to be a tensor, convert it using `tf.convert_to_tensor`. input_tensor = tf.convert_to_tensor(image) # The model expects a batch of images, so add an axis with `tf.newaxis`. input_tensor = input_tensor[tf.newaxis, ...] # Run inference output_dict = model(input_tensor) # All outputs are batches tensors. # Convert to numpy arrays, and take index [0] to remove the batch dimension. # We're only interested in the first num_detections. num_detections = int(output_dict.pop('num_detections')) output_dict = {key: value[0, :num_detections].numpy() for key, value in output_dict.items()} output_dict['num_detections'] = num_detections # detection_classes should be ints. output_dict['detection_classes'] = output_dict['detection_classes'].astype(np.int64) # Handle models with masks: if 'detection_masks' in output_dict: # Reframe the the bbox mask to the image size. detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks( output_dict['detection_masks'], output_dict['detection_boxes'], image.shape[0], image.shape[1]) detection_masks_reframed = tf.cast(detection_masks_reframed > 0.5, tf.uint8) output_dict['detection_masks_reframed'] = detection_masks_reframed.numpy() return output_dict
Step 8: Test model with sample images
For the sake of simplicity we will test on 2 images:
# If you want to test the code with your images, just add path to the images to the TEST_IMAGE_PATHS. PATH_TO_TEST_IMAGES_DIR = pathlib.Path('./models/research/object_detection/test_images') TEST_IMAGE_PATHS = sorted(list(PATH_TO_TEST_IMAGES_DIR.glob("*.jpg"))) TEST_IMAGE_PATHS
Load the model using load_model function
# load model in variable. model = load_model('ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8')
Run it on each test image and show the results:
def show_inference(model, image_path): # the array based representation of the image will be used later in order to prepare the # result image with boxes and labels on it. image_np = np.array(Image.open(image_path)) # Actual detection. output_dict = run_inference_for_single_image(model, image_np) # Visualization of the results of a detection. vis_util.visualize_boxes_and_labels_on_image_array( image_np, output_dict['detection_boxes'], output_dict['detection_classes'], output_dict['detection_scores'], category_index, instance_masks=output_dict.get('detection_masks_reframed', None), use_normalized_coordinates=True, line_thickness=8) display(Image.fromarray(image_np))
Test the program
for image_path in TEST_IMAGE_PATHS: show_inference(detection_model, image_path)
Result:
Step 9: Implement Real-Time object detection using OpenCV
import cv2 import numpy as np cap = cv2.VideoCapture(0) while True: ret, image_np = cap.read() image_np = show_inference(detection_model, image_np) cv2.imshow('Object Detection', image_np) if cv2.waitKey(25) & 0xFF == ord('q'): cv2.destroyAllWindows() break
For complete source code click here.
