Beware The Cuckoo

One of these is not like the others, but good enough to blend in

Security professionals are inherently aware that every backdoor in software intended for "the good guys" becomes a backdoor for "the bad guys". This is part of why Hypocrisy is the Worst Crime.

Why Cuckoos Can Exist

For most interactions we need to apply heuristics to determine what is okay and what is not okay. And depending on the cost of false positives or false negatives, these heuristics are tight or loose. If we were to exhaustively verify everything, then we would get very little done. Or more commonly, we'd get replaced by someone who gets a lot done by verifying little.

This opening between surplus and cost + cost to mimic creates a margin that can be exploited by the cuckoo. In the limit, the surplus is eroded by verification.

Who Forgets The Cuckoo?

But people not in these fields and not familiar with these ideas frequently propose backdoors in process. Inevitably, exploiters arise in these backdoors. A famous example in the US was the Believe All Women slogan, which was followed pretty quickly by someone accusing President Joe Biden of sexual assault. That person then moved to Russia alongside someone convicted of spying on the US for Russia, which does make it less likely that they were not a cuckoo.

Defence

Cuckoos work where verification is costly but mimicking verification is cheap. In a positive-sum interaction, the participants are incentivized to reduce the cost of closing the deal. Costly verification reduces the available surplus. Cheap verification reduces the risk of being exploited.

This is just a security problem at its heart, and over time participants in some market evolve mechanisms to handle cuckoos at some acceptable rate. The slow way is to allow for exploitation that doesn't kill you and participate in the arms race.

The quick way is to see how other participants in other arms races behaved and jump to the end result. For that, a few things have to be realized:

1. Total cost is hyperbolic as error rate decreases from 1 to 0
2. It takes time for the cuckoo to evolve, so short-term interactions can be verification free
3. If you're counting on the steady state, you should anticipate the cost of the cuckoo in the steady state

Robotbeat @Robotbeat

Folks don’t know that when airliner manufacturers make composite airframes: they have to use rivets as a secondary bonding method as an adhesive bond isn’t typically inspectable and “certifiable,” and the rivets introduce a hole in the composite which ~halves the usable strength.

May 19, 2024

An example of this is how inspections are in the US: for homes and buildings, errors sufficiently internal to the structure go undetected since it would be prohibitively costly to build otherwise; but in aerospace composites, inspectability is so valuable that we're willing to accept diminished utility to get it.

Another example is the yard sale: because of its short duration, and the generally low value of the items being sold, yard sales are poorly-policed with theft being quite easy. Someone who runs a thrift store, on the other hand, usually runs a tighter shop. Anyone can steal from a thrift store any time, but you have to detect a yard sale and then arrive there during the time it's running.

A third example is the difference between a mutual insurance company and a startup: the former is aiming at the steady state (a certain loss ratio and a certain amount of policy premiums) while the latter will accept massive fraud in order to get growth. If you're not aiming for the steady state from the start, the arms race is something you want to participate in, and you want to adapt, since high costs will kill growth.

When faced with $10 million in monthly fraud losses, PayPal’s team rallied to create groundbreaking solutions. This “all hands on deck” approach led to innovations like CAPTCHA and micro-deposit verification—technologies still widely used today. Embrace crises as opportunities for creative problem-solving and rapid innovation.

— Sequoia Capital, Crucible Moments: PayPal, Podcast on Sequoia Capital

Conclusion

The ultimate thing to remember, though, is that social and governmental policy is also just like code and it also has security vulnerabilities that can be exploited. The cuckoo is just fake credentials. And long-term things eventually reach the state where cuckoos will exist. Beware The Cuckoo.