Hi! Founder of Pangram here. Thank you for putting out this piece, I think it treats Pangram fairly and correctly calls out ways in which it fails to be as informative as it could be.

Today, Pangram 3.3 operates on segments of roughly 150-350 words. As you pointed out, this causes confusion when somebody puts in a 200 word segment with 50 words written by AI, and Pangram sees "this segment has major signs of AI writing, so we're going to call this segment AI." I know this is not ideal, and in the next version of Pangram we are working on some architectural changes that we expect will result in major improvements on this front.

I also believe we can do better at communicating our results on short text. On 50-word segments of text, Pangram simply has a lot less information to work with than 5000-word documents. How do we make Pangram useful to those who care about short text while communicating that a 100% AI result from Pangram on a 5,000 word essay carries a lot more weight than a 100% AI result on a single paragraph? Still thinking through this one but if you have any thoughts I'd love to hear them.

On the topic of induced false positives, I think that if you look into any industry and any classifier, it is simply a fact of life that these exist. If I want to induce a false positive in TSA's airport scanner, I can put a gun-shaped object in my bag. If I want to induce a false positive in Waymo's stop sign detection system, I can paint my own sign and put it up on a pole. I'm not sure this can ever be solved in a way that satisfies everybody, but I also think it's important that adversarial cases like this are addressed separately from population-level false positive rates.

I appreciate the nuanced discussion on the topic, and I hope we can impress you soon with some of our upcoming releases!