New Project-level Dashboards for Tracking Annotation Quality, Throughput, and Team Performance
Sheree Zhang
Sr. Product Manager
Sr. Product Manager
AI data teams need to know whether their annotation projects are on track long before final delivery. Will the project finish on time? How reliable are my annotators, and are they in agreement? Does the data distribution look the way I expect? The answers to these questions matter most when you're delivering data to researchers or clients on a schedule, or racing a product release deadline.
Label Studio Enterprise has always given you visibility into your projects, but getting deeper answers often meant exporting data and building your own reporting, which is time intensive and hard to scale as annotation teams, projects, and stakeholders multiply.
This release brings new and updated dashboards for project velocity, data quality, and member performance, with live metrics so you can spot issues earlier and iterate faster. Let's look at what's new.
The Throughput dashboard answers the questions that land in your inbox every Monday: Are we on track? Where are tasks backing up? How long is this actually taking?
You can see task completion pace over time, how work is distributed across states in your queue, and where the pipeline is starting to bottleneck before it becomes a real problem.
Throughput is the first thing you see when you open a project's Dashboard, because it answers the first question anyone asks.
The worst time to learn your annotators disagree on a label is after 50,000 tasks are done. The Data Quality dashboard surfaces agreement problems while they're still cheap to fix.
The Data Quality dashboard provides real-time insight into the contents and quality of your labeled data so you don’t receive any surprises at the end of your project. Task-level and dimension-level agreement scores help pinpoint edge scenarios or areas that need additional guidance.
Confusion pairs surface the most commonly misclassified labels. For example, if annotators keep confusing 'pedestrian' and 'cyclist,' you'll see that pair ranked at the top, and you know exactly what to clarify in your guidelines.
Label distributions are also included, showing annotation and prediction counts per dimension so you can catch data imbalances early and take action.
The Members dashboard tells you who is labeling, how fast they're working, and how their annotations compare to the rest of the team.
Evaluate and sort members by key metrics like agreement, acceptance score, performance score, ground truth agreement, and time spent to identify who is aligned or needs support.
You can also track a member’s annotation and review progress to see how many annotations have been submitted, accepted, rejected, or are still pending review for both annotators and reviewers.
The Agreement Matrix gives you inter-rater reliability between annotators and models in one view. You’ll know if anyone is consistently out of step with everyone else, which model predicts most similarly with annotators, and whether reliability is lower for certain dimensions.
One thing worth noting: if you’ve already built automated workflows on top of Label Studio’s legacy dashboard metrics, those are all still available through the API. Some metrics have been updated as part of this release. Reference the docs to learn more about names and endpoints.The new Dashboards in Label Studio Enterprise don’t replace existing Analytics in the app. Analytics continues to give a cross-project summary. Dashboards give you the deep dive inside a single project. Visit the Analytics docs to learn more.
New Dashboards are live for Label Studio Enterprise cloud customers today with on-prem support coming soon.
Try new Dashboards today in Label Studio Enterprise or contact us today.
Now you can have the speed and flexibility of coding agents with the powerful backend of Label Studio Enterprise.
Turn agent traces into structured human evaluations to uncover quality issues and drive improvements.
Now Label Studio Enterprise has the most flexible and granular metrics to measure agreement and compare agreement across sannotators, models and ground truth.
