Overview
Our client is a US healthcare organization. It wanted a machine learning tool that helps clinicians assess skin conditions from images: the tool suggests a list of possible conditions, and the clinician reviews it together with the patient's history. Our part was the annotated dataset, the models and the platform that annotators and clinicians work in.
Solution
We divided the project into two stages:
- Image labeling and filtering: our data annotation team labeled images with a focus on regions of interest (ROI): the body part, the skin type on the Fitzpatrick scale and the visible lesions. Images that failed the quality check were left out of the training set.
- ROI evaluation and diagnostic input: certified dermatologists reviewed the annotated images together with the patient's demographic data and case history. For each image they gave a differential diagnosis, with a confidence for each possible condition, and could add details by hand when needed.
On this dataset our machine learning engineers trained convolutional neural networks (CNNs) that rank the likely conditions for a new image. The ranking is shown to a clinician as decision support. Patient data was protected with encryption and role-based access throughout the project.
Technology Stack
Features
Detailed image annotation:
- Fine-grained labeling of ROIs in medical images
- Skin type classification on the Fitzpatrick scale
- Lesion marking for focused analysis
Clinical review:
- Diagnostic labels from board-certified dermatologists
- Patient demographics and medical history available to reviewers
- Differential diagnosis with a confidence per condition and manual override
Machine learning models:
- CNN models that rank likely conditions from an image
- Patient data used alongside the image
- Retrained on reviewed data in versioned releases
Data protection:
- Patient data stored and processed with encryption and role-based access
- Secure authentication for every user
Interface:
- One platform for annotators and medical professionals
- Dashboards for review progress and model performance
Infrastructure:
- Cloud deployment on AWS
- Modular design, so more conditions can be added later

Outcome
- Clinicians get a ranked list of possible conditions to review, built from images labeled by dermatologists
- A labeling scheme that records all six Fitzpatrick skin types, so model results can be checked for each one
- A design that can be extended to more medical conditions
- An annotation and review pipeline that supplies reviewed data for each new model release
project timeline
annotation stages: region labelling, then review by certified dermatologists
Fitzpatrick skin types covered by the labelling scheme
How We Build Medical Image Annotation Pipelines
Annotation for a diagnostic model is a clinical task. Our pipelines separate what trained annotators can decide from what only a physician can:
- Image quality gate. Every image is checked for focus, lighting, framing and duplicates before anyone labels it. Rejected images are logged with a reason, so the filter itself can be audited.
- Layered labels. Annotators mark regions of interest, body site and skin type. Diagnostic labels come only from licensed clinicians, with a confidence for each candidate condition.
- A written labelling protocol. Each label has a definition, borderline examples and a rule for when to escalate. Clinicians approve the protocol before the first batch is labelled.
- Measured agreement. A share of cases is labelled by two clinicians independently. Disagreements go to adjudication and feed back into the protocol.
- Stratified evaluation. Test sets are split by Fitzpatrick type, body site and condition, and model metrics are reported for each group, not only as one average.
- Privacy from the start. Identifiers are removed before images reach annotators, and every access is logged per user and per image.
We train vision models in PyTorch and run annotation in CVAT, or in a purpose-built tool when the clinical workflow needs one. The clinician-facing review screens are built on a TypeScript stack with Next.js and Node.js.
Where Diagnostic Image Models Fall Short
Accuracy on a held-out test set says little about how a model behaves on a skin type, camera or condition it rarely saw. Dermatology makes this visible. In a Nature Medicine study of physicians using AI decision support, board-certified dermatologists reached 38% diagnostic accuracy and primary-care physicians 19%, and both groups were about four percentage points less accurate on images of dark skin. Decision support from a deep learning system improved accuracy for both groups, yet widened the skin-tone gap for the generalists.
Two lessons follow for anyone building such a system. Balanced coverage of skin types has to be planned at data collection, because it cannot be added at the end. And the way a model's output is shown to a clinician changes outcomes as much as the model's own accuracy, so the interface and its confidence display need testing with the people who will use them.
Some decisions stay manual: the final diagnosis, rare conditions with few training examples, and any case where the image and the patient history point in different directions. The model narrows the list. The clinician decides. See our computer vision development services and healthcare pages for how we approach medical imaging.