Seriously, we tested the performance of BasicAI Cloud* in handling image frame series annotation

You are the manager of an annotation team, tasked with overseeing a complex, collaborative project. The challenge: accurately annotate a sequence of 1,000 consecutive images, extracted from a video, with up to 50 objects in each frame requiring precise annotation. This project demands seamless coordination between various worker roles, including annotators, reviewers, inspectors, and admins. As a team leader, your primary concerns are efficiency and quality, so you need a platform that not only handles the intricacy and volume of data without missing a beat but also fosters smooth collaboration. Finally, you make the choice: BasicAI Cloud*.

At the BasicAI marketing team, we've always been confident in our platform's ability to support large-scale annotation projects and facilitate seamless teamwork. However, we realized that it was not enough to just say it – we needed to prove it with concrete data. That's why we decided to put our platform to the test, pushing it to the limit to see just how many consecutive images it could handle without breaking a sweat.

To ensure impartiality, we reached out to our production department to conduct an unbiased performance test. We were eager to discover the true capabilities of BasicAI Cloud* when it comes to handling frame series annotation and supporting smooth task flow among various worker roles.

And the result? We were blown away. Our platform didn't merely meet our expectations – it surpassed them. BasicAI Cloud* demonstrated its prowess by supporting the annotation of a staggering 1,000 consecutive images, each containing up to 50 objects, all while facilitating seamless collaboration between multiple roles, without a hint of lag.

Curious about the details of this performance test? Fasten your seatbelts and join us as we dive into the comprehensive testing report that showcases BasicAI Cloud*'s unrivaled performance in consecutive image annotation and team collaboration.

Report date: May 15th, 2023

Testing Environment

Four test machines were used in this testing, all running Chrome browsers with version numbers 113.0.5672.93 (official version) (64-bit), 113.0.5672.92 (official version) (64-bit), 112.0.5615.140 (official version) (64-bit), and 112.0.5615.140 (official version) (64-bit).

*Detailed information on the test machine configurations is not provided here.

Testing Strategy

This round of testing was conducted simultaneously in the same environment. According to the product requirements, the maximum number of consecutive frames should support up to 1,000, with each frame being 1MB in size and containing an average of 100 results. The following testing plan was determined:

Testing Scenarios

Four testing scenarios were conducted in total, with the following content:

Scenario 1

We utilized a machine with 16GB of memory and an Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz 2.11 GHz processor. There were four annotation scenarios. The first scenario involved 500-frame data with 5MB per frame, an average of 100 results per frame, no label restrictions, and auto-loading enabled for smooth annotation, review, and acceptance workflows. Scenarios two and three featured additional task flows and involved 500-frame data with 10MB per frame and 50 results per frame, as well as 500-frame data with 8MB per frame and 50 results per frame (Skeleton), both without label restrictions and with auto-loading enabled. Both scenarios followed a workflow of annotation, multiple review stages, and acceptance, and included tasks for publishing continuous frame reflection tasks.

Scenario 2

We used a machine with 16GB of memory and an 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz 2.80 GHz processor. There were three annotation scenarios. The second scenario involved 1,000-frame data with 3MB per frame and an average of 50 results per frame, no label restrictions, and auto-loading enabled. Scenarios one and three included additional task flows, both involving 1,000-frame data with 1MB per frame and either 100 results per frame or 50 results per frame (Skeleton), no label restrictions, and auto-loading enabled. These scenarios followed a workflow of annotation, multiple review stages, and acceptance.

Scenario 3

We utilized a machine with 16GB of memory and an AMD Ryzen 7 5800U with a Radeon Graphics 1.90 GHz processor. The annotation scenario involved publishing continuous frame tasks with 1,500-frame data, 1MB per frame, and an average of 70 results per frame. The workflow included annotation, multiple review stages, and acceptance.

Scenario 4

We used a machine with 8GB of memory and an Apple M1 processor. The annotation scenario involved 1,500-frame data with 1MB per frame and an average of 70 results per frame, no label restrictions, and auto-loading enabled. The workflow included annotation, multiple review stages, and acceptance, with an additional task flow for publishing continuous frame reflection tasks with 1,500 frames, 1MB per frame, and 70 results per frame, without label restrictions and auto-loading disabled.

Testing Conclusion

Based on the results of this testing, the consecutive frame annotation tool page can support the number of frames and data size required for smooth annotation. The platform can fluently handle 500 frames with 5MB images, up to 100 results per frame, and 1,000 frames with 3MB images, up to 50 results per frame. The platform supports a smooth task flow among various worker roles throughout every stage of the project. The stable operation of the image annotation tool is supported on computers with an Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz 2.11 GHz and 16GB RAM, and Chrome browser 64-bit, version 112 or higher.

Our performance testing of BasicAI Cloud* has showcased its ability to handle large-scale, complex annotation projects with ease. As a team leader, imagine the possibilities offered by such a platform that doesn't compromise quality or efficiency.

BasicAI Cloud* not only supports image annotation but also 3D LiDAR point cloud and 2D & 3D sensor fusion data, further expanding its capabilities. We're excited to announce that BasicAI Cloud* has just gone free! Each new user will now enjoy 50 seats, 100GB storage, and 1,000 model calls (for auto-annotation, auto-segmentation, and object tracking) at no cost.

Experience the power of image annotation on BasicAI Cloud* for yourself today by clicking the button:

Discover firsthand how our platform can revolutionize your annotation projects. Transform your team's workflow and elevate your annotation projects to new heights with BasicAI Cloud*. Together, let's embark on a path toward extraordinary accomplishments in the AI world.

* To further enhance data security, we discontinue the Cloud version of our data annotation platform since 31st October 2024. Please contact us for a customized private deployment plan that meets your data annotation goals while prioritizing data security.