Slashing Log Query I/O: How SigNoz Cut ClickHouse Scanning to 0.85% with Resource Fingerprinting
SigNoz engineers tackled a pervasive log query bottleneck by rethinking ClickHouse data organization, introducing a deterministic 'resource fingerprint' to group related logs. This schema change leveraged ClickHouse's sparse primary index to reduce block scanning from 99.5% to under 1%, dramatically lowering I/O and latency while remaining extensible across Kubernetes, AWS, and Docker environments. The optimization underscores how aligning storage with access patterns can yield transformative performance gains without complex secondary indexes.