Kian Kyars

Yes, it's done for each transformer block in an LM because each transformer block has different attention heads. If you do it for only one transformer block across all blocks, then you don't get the same representation.

I think I got lost around the standard inference versus Kv caching section because I couldn't understand the matmuls happening based on each flashing repetition of those yellow blocks. But perhaps I just need to go through the blog post once again to try to better understand it.

I didn't understand the explanation

This technique is not a better lesson pilled at all. Waste of time when the model will just learn to do this anyways.