Evaluating the Quality of Smart Contract Auditing

May 23, 2022 - We added an experimental indicator to the Web3 Hack and Scam database to evaluate the flag security incident as they related to projects that have been audited by specific auditors.

While the total records as the result of the query remains the same along with the aggregated losses, by specifying the auditor in the dropdown as part of the query, the Hack and Scam DB will now return the number of incidents where the selected auditor participated. In addition, we also compute the average loss from those incidents.

As shown in the example above, we queried from incidents in 2022 where Certik was involved. The results indicated that Certik was labeled as auditor for 13 incidents with an average loss of $5.2M. Keep in mind that this is out of 211 records with an aggregated loss of $2B for the same period. In other words, incidents with Certik as auditor account for roughly 3.4% (or 13 x $5.2M / $2B) of the aggregated loss.

By contrasting the above with the example for Consensys Diligence shown below,

one may infer that the quality of the audits conducted Consensys Diligence, perhaps may be more complete and likely to result in less losses. Keep in mind that this is simply an indicator and may not be a consistent predictor of future audits. Ideally we want to compare on equal counts or as closely as possible.

By considering the combination of the auditor-related count and the resulting average loss within the aggregated loss of the query, the metric provides insight to the quality of the audit conducted by specific auditors within certain periods or conditions. Keep in mind that our data on this specific metric are still being updated and will be completed over the next two weeks.