Our human-in-the-loop verification system has been stable for about 6 hours, so we are resolving this incident. Our verifiers will continue working on the backlog until it's cleared.
Posted Jan 22, 2024 - 22:57 EST
Update
We have made significant progress on the backlog, and virtually all documents uploaded before 5pm ET have been verified.
Posted Jan 22, 2024 - 20:56 EST
Monitoring
With the increased volume, the size of the backlog has remained relatively constant throughout the day. Consequently, the average turnaround time for documents that require human verification has risen to above one hour from a norm of under 10 minutes.
Posted Jan 22, 2024 - 18:23 EST
Identified
The performance issues were caused by a spike in human verification tasks for documents that had been uploaded over the weekend as part of a bulk data import job, which we have suspended. We also received an exceptionally high volume of uploads today between 10am and 12pm ET. Both events have contributed to a large backlog of documents that will take some time to work through. Our current estimate is 4 to 5 hours.
Posted Jan 22, 2024 - 14:30 EST
Investigating
We are experiencing performance issues with our human-in-the-loop verification system, which will lead to increased turnaround times and overdue documents for documents that require human verification. The Ocrolus API and Dashboard are not affected by this incident. We are still investigating the root cause.
Posted Jan 22, 2024 - 13:40 EST
This incident affected: Human-in-the-loop Verification System.