AI Image Annotation Solution for Medical Assessment
AI Development •Data Annotation •Data Science •Healthcare •Machine Learning

AI Image Annotation Solution for Medical Assessment

We worked with a US healthcare organization on decision support for skin conditions. Annotators labeled the images, certified dermatologists added the diagnoses, and the trained models suggest a ranked list of possible conditions that a clinician reviews.

Facing a similar challenge? Tell us about the task, and we will show how we would build it for your business.

Build Something Like This for Me

Free 30-min call · One-page solution outline within three business days · No commitment

Service
Machine Learning Consulting Services / Data Science Consulting Services / Agentic AI Development Services / Machine Learning Development Services / Computer Vision Development Services & Solutions / Data Annotation Services
Location
United States
Timelines
9 months
Team
1 PM, 2 ML engineers, 2 Data Scientists, 2 Data Annotators, 2 Full-stack developers, 1 QA engineer

Overview

Our client is a US healthcare organization. It wanted a machine learning tool that helps clinicians assess skin conditions from images: the tool suggests a list of possible conditions, and the clinician reviews it together with the patient's history. Our part was the annotated dataset, the models and the platform that annotators and clinicians work in.

Solution

We divided the project into two stages:

  • Image labeling and filtering: our data annotation team labeled images with a focus on regions of interest (ROI): the body part, the skin type on the Fitzpatrick scale and the visible lesions. Images that failed the quality check were left out of the training set.
  • ROI evaluation and diagnostic input: certified dermatologists reviewed the annotated images together with the patient's demographic data and case history. For each image they gave a differential diagnosis, with a confidence for each possible condition, and could add details by hand when needed.

On this dataset our machine learning engineers trained convolutional neural networks (CNNs) that rank the likely conditions for a new image. The ranking is shown to a clinician as decision support. Patient data was protected with encryption and role-based access throughout the project.

Features

Detailed image annotation:

  • Fine-grained labeling of ROIs in medical images
  • Skin type classification on the Fitzpatrick scale
  • Lesion marking for focused analysis

Clinical review:

  • Diagnostic labels from board-certified dermatologists
  • Patient demographics and medical history available to reviewers
  • Differential diagnosis with a confidence per condition and manual override

Machine learning models:

  • CNN models that rank likely conditions from an image
  • Patient data used alongside the image
  • Retrained on reviewed data in versioned releases

Data protection:

  • Patient data stored and processed with encryption and role-based access
  • Secure authentication for every user

Interface:

  • One platform for annotators and medical professionals
  • Dashboards for review progress and model performance

Infrastructure:

  • Cloud deployment on AWS
  • Modular design, so more conditions can be added later
fw-image

Outcome

  • Clinicians get a ranked list of possible conditions to review, built from images labeled by dermatologists
  • A labeling scheme that records all six Fitzpatrick skin types, so model results can be checked for each one
  • A design that can be extended to more medical conditions
  • An annotation and review pipeline that supplies reviewed data for each new model release
9 months

project timeline

2

annotation stages: region labelling, then review by certified dermatologists

6

Fitzpatrick skin types covered by the labelling scheme

How We Build Medical Image Annotation Pipelines

Annotation for a diagnostic model is a clinical task. Our pipelines separate what trained annotators can decide from what only a physician can:

  • Image quality gate. Every image is checked for focus, lighting, framing and duplicates before anyone labels it. Rejected images are logged with a reason, so the filter itself can be audited.
  • Layered labels. Annotators mark regions of interest, body site and skin type. Diagnostic labels come only from licensed clinicians, with a confidence for each candidate condition.
  • A written labelling protocol. Each label has a definition, borderline examples and a rule for when to escalate. Clinicians approve the protocol before the first batch is labelled.
  • Measured agreement. A share of cases is labelled by two clinicians independently. Disagreements go to adjudication and feed back into the protocol.
  • Stratified evaluation. Test sets are split by Fitzpatrick type, body site and condition, and model metrics are reported for each group, not only as one average.
  • Privacy from the start. Identifiers are removed before images reach annotators, and every access is logged per user and per image.

We train vision models in PyTorch and run annotation in CVAT, or in a purpose-built tool when the clinical workflow needs one. The clinician-facing review screens are built on a TypeScript stack with Next.js and Node.js.

Where Diagnostic Image Models Fall Short

Accuracy on a held-out test set says little about how a model behaves on a skin type, camera or condition it rarely saw. Dermatology makes this visible. In a Nature Medicine study of physicians using AI decision support, board-certified dermatologists reached 38% diagnostic accuracy and primary-care physicians 19%, and both groups were about four percentage points less accurate on images of dark skin. Decision support from a deep learning system improved accuracy for both groups, yet widened the skin-tone gap for the generalists.

Two lessons follow for anyone building such a system. Balanced coverage of skin types has to be planned at data collection, because it cannot be added at the end. And the way a model's output is shown to a clinician changes outcomes as much as the model's own accuracy, so the interface and its confidence display need testing with the people who will use them.

Some decisions stay manual: the final diagnosis, rare conditions with few training examples, and any case where the image and the patient history point in different directions. The model narrows the list. The clinician decides. See our computer vision development services and healthcare pages for how we approach medical imaging.

Other cases

AI Agent Platform With Governed Model Access: Many Agents, One Runtime

AI Agent Platform With Governed Model Access: Many Agents, One Runtime

Read more ›
AI Candidate Screening That Checks CV Claims Against Public Evidence

AI Candidate Screening That Checks CV Claims Against Public Evidence

Read more ›
ChatGPT-Powered Content Generation Platform

ChatGPT-Powered Content Generation Platform

Read more ›
AI Data Analyst in Microsoft Teams: Answers From Live Operational Data

AI Data Analyst in Microsoft Teams: Answers From Live Operational Data

Read more ›
AI Data Verification Layer for Property Data: Rules First, a Person on Borderline Cases

AI Data Verification Layer for Property Data: Rules First, a Person on Borderline Cases

Read more ›
AI Delegation Agent for Executive Teams: Assignments, Statuses and Scores in One Loop

AI Delegation Agent for Executive Teams: Assignments, Statuses and Scores in One Loop

Read more ›
ML/Computer Vision + Data Annotation for ADAS

ML/Computer Vision + Data Annotation for ADAS

Read more ›
AI/ML-Driven Telecom Network Optimization

AI/ML-Driven Telecom Network Optimization

Read more ›
Custom GenAI Medical Drug Search Accelerator

Custom GenAI Medical Drug Search Accelerator

Read more ›
ML/Computer Vision-Powered Data Annotation for Sports

ML/Computer Vision-Powered Data Annotation for Sports

Read more ›
OCR Retail Receipt Data Extraction Tool

OCR Retail Receipt Data Extraction Tool

Read more ›
Computer Vision + Image Annotation for Insurance

Computer Vision + Image Annotation for Insurance

Read more ›
AI Presale Assistant for a Marketing Agency: From Meeting Transcript to Sourced Draft

AI Presale Assistant for a Marketing Agency: From Meeting Transcript to Sourced Draft

Read more ›
AI Pricing Engine for an Equipment Distributor: The Model Proposes, a Person Approves

AI Pricing Engine for an Equipment Distributor: The Model Proposes, a Person Approves

Read more ›
AI Trade Contractor Sourcing: Verified Profiles and Pre-Screening for a Home Builder

AI Trade Contractor Sourcing: Verified Profiles and Pre-Screening for a Home Builder

Read more ›
Cinema Scheduling and Pricing Optimization: A Week of Showtimes, Computed

Cinema Scheduling and Pricing Optimization: A Week of Showtimes, Computed

Read more ›
Computer Vision + ChatGPT-Powered Recommendations

Computer Vision + ChatGPT-Powered Recommendations

Read more ›
Data Annotation for AI-Driven Sports Analytics

Data Annotation for AI-Driven Sports Analytics

Read more ›
Generative AI Bot for Customer Service

Generative AI Bot for Customer Service

Read more ›
Generative AI Solution for Code Compliance

Generative AI Solution for Code Compliance

Read more ›
SaaS Marketing Content Generation Platform for Healthcare

SaaS Marketing Content Generation Platform for Healthcare

Read more ›
Interactive Chatbot with an Animated Avatar

Interactive Chatbot with an Animated Avatar

Read more ›
Managed Video Annotation Pipeline for a Client's Own Human-Motion Footage

Managed Video Annotation Pipeline for a Client's Own Human-Motion Footage

Read more ›
ML/AI Technical Due Diligence for Startup Acquisition

ML/AI Technical Due Diligence for Startup Acquisition

Read more ›
OCR Solution for Invoice Processing

OCR Solution for Invoice Processing

Read more ›
Property Management Platform With an AI Voice Assistant for Multiple HOAs

Property Management Platform With an AI Voice Assistant for Multiple HOAs

Read more ›
AI-Powered Platform for Real Estate

AI-Powered Platform for Real Estate

Read more ›
AI Assistant for Report Generation & Visualization

AI Assistant for Report Generation & Visualization

Read more ›
Sitecore Migration

Sitecore Migration

Read more ›
ML/AI Technical Due Diligence for SporTech

ML/AI Technical Due Diligence for SporTech

Read more ›
Voice Interview to Social Posts: AI Content in the Professional's Own Voice

Voice Interview to Social Posts: AI Content in the Professional's Own Voice

Read more ›

Let's talk about your project

We'll help you harness AI and ML technologies to create innovative software solutions

File requirements: pdf, doc, docx, rtf, ppt, pptx — max 15 MB

No file selected