Editing Blog/2025-01-16/Superheap

[[File:{{#setmainimage:Superheap Bloomberg RSS Feed Example.png}}|thumb|Overall, not too bad]]When friends got on Substack, one of the things that annoyed me is that it doesn't have an RSS feed. This is completely untrue<ref name=hn/>. It does have an RSS feed, it's just that my friend [https://porterwestling.substack.com/ Porter] had chosen the Twitter-like version of Substack instead<ref name=openrss/>. But I didn't realize this and decided that I'd prefer if I received this stuff as an RSS feed anyway. Since Substack refuses to accept [https://kill-the-newsletter.com/ Kill The Newsletter] emails, I wondered to myself how easy it would be to write my own, even though [https://github.com/leafac/kill-the-newsletter KTN is open source]

[https://github.com/roshan/superheap This ended up being really easy to do because of LLMs today]. I ended up using a mix of Cursor, RustRover, and Claude 3.5 Sonnet. Cursor is pretty amazing as a tool to get going, though it seems to be more of an AI text-editor than an IDE for Rustlang: it has pretty slow comprehension of code and jumping to definition and so on don't work for me out of the box.

In the end, I built the initial scaffolding with Cursor and then used RustRover to write some of the code and used a copy-paste workflow vs. Claude.

The whole thing seems fairly simple to me:<ref name=o1tasks/>

# An application mail server that writes to SQLite
# An application that generates feeds from the emails collected
# <code>stunnel</code> to do TLS in front of the app mail server
# <code>nginx</code> to serve the resulting feeds with TLS

Overall, it didn't take very much time to get the whole thing working - and honestly that's simply because LLMs have accelerated development so much! In fact, the email protocol is mostly text-based and so a plain-text version of this could be 200 lines of code in Rust without invoking any libraries. To start with, Cursor just wrote up the protocol and mail parsing as we were going and I got to 50% quite fast. At this point I was getting the impression that some class of software libraries are not that useful: with known protocols, instant re-implementation is not that costly.

But TLS is a big exception. I'm sure it's possible to easily do it at the application server, but the experience of using <code>stunnel</code> to handle TLS for the mail server and <code>nginx</code> to do so likewise for web-serving was actually quite smooth. Claude and ChatGPT made a few missteps here recommending that I use <code>nginx</code> with <code>mail_proxy</code> enabled, but ultimately this wasn't so bad. I anticipate that if I tried to support TLS direct with <code>rusttls</code> it would not be as simple as a 20 line config. And I definitely used <code>certbot</code> to get some Let's Encrypt certificates rather than having the LLM write to the protocol.

When I first went to test this it worked fine since I sent plaintext emails from Gmail. But when I tried Protonmail, it sends email base64-encoded with a <code>Content-Type-Encoding</code> header that contains that specification! And of course most senders use <code>text/html</code> rather than <code>text/plain</code>

I'm confident that either Claude directly or Cursor w/ Claude could have written me a minimal parser for all of this in half an hour, but it seemed to be less bothersome for me to just use the <code>mail-parser</code> crate.

Overall, this would have taken me a much longer time to discover and handle without an LLM to assist me. They're really phenomenal tools. They reduce the cost to write software so much that it is well possible that we might arrive at the future most software engineers I know have always wanted: with individuals running their own software. The one difference being, of course, that the software is mostly machine written. One can imagine this opens up all sorts of creative pursuits now that skill and creativity are decoupled.

In the end, I used this tool for Matt Levine's Money Stuff feed since that one actually ships out a minimal RSS feed with content on the website but a full-featured email with content in the email. So I use this to get the full content in the RSS feed instead.

== Notes ==

<references>
<ref name=hn>I didn't actually know this till I'd finished writing the software, though. Haha! [https://news.ycombinator.com/item?id=42727814 It was a Hacker News comment that informed me]. I was far too eager to try these things out and didn't even check.</ref>
<ref name=openrss>OpenRSS actually does have a feed for Substack notes. For example, [https://openrss.org/substack.com/@porterwestling here's Porter's feed].</ref>
<ref name=o1tasks>Another funny thing I tried for the first time here is that I asked ChatGPT o1 to keep a list of tasks for me on the subject and I'd tell it what I'd done and what needed to be done and it did a fairly good job of keeping the to-do items correct and what needed to be done next and so on.</ref>
</references>

[[Category:Blog]]