Which Type of Data Annotation Is Right For You? How to Get it done?

Teams must decide what type of data annotation is right for their application. This is an important question because data labeling can be expensive and time-consuming, but it is critical to the model’s success. So, teams are stuck with an often complicated cost-benefit analysis when it comes time to annotate their data. While it might be tempting to settle for image classification – it’s probably the cheapest and easiest to achieve – its applications are very limited. If we think about an autonomous vehicle computer vision model looking out into a complex urban environment, we begin to see that just recognizing whether there is a human in its sight or not will not be enough. To avoid running the person over, the car also needs to know where the human is. If we take a medical computer vision application – identifying the shape of cancerous cells, we need instance segmentation, to differentiate between different instances of cells. Defining the whole image as “cells” won’t help us localize the problematic cells or to understand the extent of any problems.

But there are many cases where it’s not obvious what type of data annotation you need. This is a high-risk decision for teams. If they use the wrong annotation method or add the wrong information to their images, their model may not work and they’ll need to start the data labeling process anew. Simulated Data can relieve a lot of the stress associated with this type of decision by automatically and flexibly adding a wider range of annotations with perfect ground truth, but more on this later.

Different Annotation Techniques

Once you’ve chosen your annotation method, there are even more choices to make; now you have to select an annotation technique. This is the actual method that annotators will use to attach annotations to your data. For instance, they may draw squares around objects, do multi-sided polygons, or attach landmarks. It is important to understand these techniques because, again, there is often a tradeoff between cost, time, and effectiveness.

Bounding Boxes – The most basic type of data annotation. It consists of drawing a rectangle or square around the target object and is very commonly used, due to its simplicity and versatility. This is useful when objects are relatively symmetrical – such as boxes of foods or road signs – or when the exact shape of the object is of less interest. On the other hand, complex objects don’t have right angles, and achieving ground truth annotation using bounding boxes is impossible. Additionally, we have no annotations for what’s happening “inside” the box. For instance, if we care about a person’s movement, posture, gait, or other dynamic indicators, bounding boxes are unlikely to be helpful.

Polygon Annotation – a variation of the bounding box technique. By using complex shapes (polygons) and not only the right angles of bounding boxes, the target object’s location, and boundaries are defined more accurately. Increased accuracy cuts out irrelevant pixels that can confuse the classifier. This is good for more irregularly shaped objects – cars, people, logos, and animals. While polygons are more accurate than bounding boxes, overlapping objects may be captured within a single polygon and therefore not distinguishable from each other.

Polylines – this plots continuous lines made of one or more segments. It is best used when important features have a linear appearance. This is common in an autonomous vehicle context, as it is easily applied to define lanes and sidewalks. But, for most use cases this simply isn’t relevant because the object is not linear and more than a single pixel wide.

Landmarking – This is also known as dot annotation. It involves creating dots across the image. These small dots help detect and quantify characteristics in the data. It is used often in facial recognition to detect facial features, emotions, and expressions. It can be used to help annotate human bodies, align posture, and explore the relationship between different body parts. Another interesting use case is to find objects of interest in aerial footage such as cars, buildings, and more. Clearly, this approach is both time-consuming and prone to inaccuracy. For instance, manually landmarking facial features like irises across thousands of images is very difficult to do consistently and accurately.

Tracking – This is a data labeling technique used to plot an object’s movement across multiple frames. Some tools include interpolation, which enables the annotator to label one frame, skip frames, and then annotate the new position. The annotating tools automatically fill in the movement and track the objects through the frames. While this is great theoretically, it takes a lot of work and high levels of accuracy to successfully annotate. In general, the cost of annotating video data quickly becomes cost prohibitive because of the need to annotate frame-by-frame.

Often, your use case will dictate the technique that’s right for you. But, even if you have little choice in which technique to adopt, it is critical to be aware of the constraints of each one. Expensive techniques may limit the amount of data you want to collect. While techniques with inherent variation may force you to pay extra attention to the effects of minor inconsistencies on your model’s performance.

Getting the Annotation Done

Now, you’ve gathered your data and decided on the method and techniques of data annotation that work best for your model. It’s time to get the annotations added to your images. This annotating process involves people sitting, and manually marking image after image. Ideally, you might be assisted by some automation tools, but in general, it is a manual and labor-intensive process. In today’s annotation landscape, there are a couple of different solutions available to you:

Crowdsourcing – Crowdsourcing involves paying workers often distributed globally and working as freelancers to perform a micro-task or assignment. They are generally paid a small sum based on the volume of work they complete. Crowdsourced labor tends to be of low quality and consistency for obvious reasons. The workers are lightly vetted or may have little idea of what they are doing or common pitfalls. The burden of managing them falls on you. There are also platforms that crowdsource work but manage the workflow and sourcing of workers. Additionally, quality assurance post-labeling requires resources and validation, and without it is impossible to guarantee high-quality results. Because these tend to be one-off relationships, there is no feedback loop with the people working on your project and there is no way to train them over time. Data security is also a challenge as these people are often working independently on unsecured computers.

In-House Solutions – Some companies choose to try and solve data annotation needs in-house. For small, easy-to-annotate datasets, this may be a great option. But, many companies often assign this low-level work to their data scientists and engineers, which is not a good use of their time. The alternative of hiring annotators in-house – which brings benefits of process control and QA carries significant overhead costs. Generally, this method is not scalable, as you invest in hiring, managing, and training employees while your data needs may fluctuate wildly over time. Teams that try to automate these processes or build in-house tech solutions often find themselves distracting valuable development teams with projects that would be more efficient to outsource.

Outsourcing – There are many data labeling companies often based in low-cost markets like India – that employee teams focused on data annotation. Some suppliers leverage certain ML models to accelerate the process and do QA. By virtue of employing the annotators, these companies are better able to control quality, can improve quality over time as they learn about your specific needs, and can provide better time estimates than the other options. But, ultimately this is still a manual process, and any cost-savings come from the cheap cost of labor. You still have to devote operational resources to managing this relationship and, at the end of the day, you are still dependent on a third-party vendor that is subject to all kinds of delays, inconsistencies, and challenges.

As you can see, all of these options have significant operational, quality control, and process challenges. They generally force you to devote time and energy to things outside your core technological mandate.

TagX Data Annotation Services

Since data annotation is very important for the overall success of your AI projects, you should carefully choose your service provider. TagX offers data annotation services for machine learning. Having a diverse pool of accredited professionals, access to the most advanced tools, cutting-edge technologies, and proven operational techniques, we constantly strive to improve the quality of our client’s AI algorithm predictions.

We have experts in the field who understand data and its allied concerns like no other. We could be your ideal partners as we bring to the table competencies like commitment, confidentiality, flexibility, and ownership to each project or collaboration. So, regardless of the type of data you intend to get annotations for, you could find that veteran team in us to meet your demands and goals. Get your AI models optimized for learning with us.