Earlier this week I migrated www.freecalypso.org to a new server, and that migration had a side effect of taking all of my Mercurial code repositories offline. (It happened this way because only the static file archive was migrated initially.) I am now pleased to report that all FreeCalypso and Themyscira Wireless Hg repositories are back online at this new location. Old URLs still work as well, handled via HTTP(S) redirect.

The new hosting setup for FC and ThemWi should be much more responsive than our old server that was overrun to death by AI scraper bots, and unlike the band-aid solution that was applied between May 11 and now, the new server is set up much more thoughtfully and properly. Because of the way Mercurial works, its hgweb facility is required for all public read-only access to hosted repositories: not only for casual browsing akin to cgit, but also for native clone and pull operations over HTTP(S). And because hgweb is implemented in Python like most of Mercurial, all operations that invoke it are expensive in terms of server load. Our new server has the following improvements over the old one in this regard:

1. Instead of using simple-minded CGI between the web server front end and hgweb (a new process spawned for each and every request), we now run a dedicated back end server that consists of long-lived worker processes, Green Unicorn specifically for hgweb.
2. The proxy path from lighttpd front end to Gunicorn/hgweb back end passes through Anubis - hence all incoming HTTP(S) requests get their souls weighed, and malignant bots get blocked. It is the same solution that has been adopted by Osmocom (our primary inspiration), Linux kernel cgit and many others.

All static content is still served by lighttpd like before, although we got an update to a newer version of this light daemon. Because so much of our published content is in the form of "raw" FTP-like trees of directories and files, I felt it was very important to preserve the look of directory listings and the assignment of MIME types to files - this consideration was the main impetus for staying with lighttpd, as opposed to the other considered option of moving to nginx.

As another point worthy of note, we no longer force HTTPS onto users against their will! All HTTP(S) resources are equally accessible at both http:// and https:// URLs, and neither redirects to the other - so use whichever you prefer. My personal philosophy is that HTTPS "security" is utterly useless and constitutes nothing but security theater, especially for a site like ours that has no login mechanism, no ability to collect anything from users or visitors, just a public read-only resource made free to the world - but we have a ton of legacy URLs with https in them, plus some people will complain loudly if we went "insecure" only. Hence our solution: use whichever protocol you like, let each user choose for herself.

And finally, never forget good old FTP! On our new server both /pub and /members static file resources are accessible via both FTP and HTTP(S) - so now you have a choice of 3 protocols for downloading our static content 🙂.