Measuring the size of objects in an image with OpenCV Python to assess the damage

Introduction

Today I would like to tell you how, using the OpenCV — Python library, I dealt with a task that consisted of measuring objects visible in photos, while taking into account the perspective of the photo. It was supposed to determine the size of damages such as scratches or flooding.

In short, using OpenCV, we can measure objects contained in photos. OpenCV, a comprehensive Python library for computer vision, has provided us with a rich set of image processing and analysis functions, including perspective correction. In the example shown below, our algorithm, in addition to measuring objects visible in photos, also corrects perspective.

These capabilities of the library can be used for various purposes, e.g. in the insurance industry to assess damage when claiming compensation. In this article, we focus on assessing things like scratches and other damage (caused by, for example, flooding) because OpenCV allows us to precisely determine the extent of the damage visible in the photo. These functions even work on photos taken from difficult angles. Reliable calculation of actual dimensions was possible by mapping pixels to millimeters. In the case shown below it worked on a test number of 40 different photos of various elements, and these photos were taken from different perspectives.

What is OpenCV Python?

OpenCV is a Python library for computer vision that allows you to perform image processing and computer vision tasks. It has an API in many different languages and it makes various functions available, functions used for working with images, such as object detection, face recognition, and tracking.

To show you exactly how it works, I created a simple scheme.

How to read an image in Python OpenCV

As you can see in the picture, the assumption was that the insurance agent would mark the application. Mark 4 points of the reference object (it could be a piece of paper or a credit card). Then they’d mark the damage (the scratch) with two points. Based on this information, the system will be able to show that the scratch is up to 130 mm. We created an algorithm that manages these operations, and added OpenCV to it in Python.

The algorithm first takes the points of the reference objects and organizes them in a specified order, because many algorithms for images require the points to be in order.

Then, the main thing the algorithm does is correct the perspective.

Image perspective correction — OpenCV

Let’s say the floor is flooded, it’s very extensive, and we can’t take a photo parallel to this damage. The insurance agent would take a photo at an angle. This angle could harm the real results of the algorithm. That’s why we conducted a correction of perspective, which is also available in the OpenCV library.

Then, when we had a corrected image, we defined the number of millilitres for each pixel of the image. And knowing how it looks, knowing what the length of the damage is in pixels, we were able to state the actual size of the damage. We prepared a simple demo for this presentation, to test it easily.

And it looks like this:

We mark the corners of the reference object and the size of the damage too. And here I wanted to show you how the perspective correction algorithm works, what it does in this case. Knowing that the object is a rectangle, we make OpenCV cram it into a flat rectangle.

Based on this, OpenCV generates an image transformation matrix which we apply to the whole image. Then we get something like this:

So the whole image is flattened out in relation to a piece of paper or an ATM card, and we get the actual value as a result of this algorithm’s work. The client measured the actual damage to be 47 millimeters. The algorithm said 48. The requirement was for the calculation to be no less than 85 percent of the actual size for it to be within the margin of error. We tested it on 40 images, and it worked.

Conclusion

Regarding the potential possibilities of this project, we wanted to add the ability to measure the surface visible in the photo. It’s supposed to work in such a way that the person adding the photo (in our case the person reporting the damage) places a point cloud limiting a certain area of the image, and we then calculate the actual area (an operation which is also possible in OpenCV — we have already investigated it). We also planned to add automatic detection of the location of an object or the type of damage using AI algorithms.