posts

BYTES

2022.10.23 – I had been up late reading Charles Stross the same night as watching Coppola’s adaptation of Dracula, and I was wondering why there were almost no horror cyberpunk stories. So I wrote a very campy one, taking many liberties from the source materials. With apologies, I present…


BYTES On the fringes of the datasphere…
A few brief heartbeats from now…

\* + * + * + *\ In a long-forgotten archival sector, an entity bearing the tags SHE | UNMAJOR | UNDECLARED drifts slowly through decaying stacks of backups. File fragments lay strewn across the pathways, detritus of unaccounted bits from fruitless defragmentation attempts and migrations of the underlying physical media strewn across galaxies. Ping latency is so high as to render the realm wholly silent. Yet in this silence the entity is drawn deeper by an unaccounted-for call, a hidden imperative luring her deeper within the chaotic structures. She pauses, inspecting a data catalog documenting early nineteenth century interior decoration, moldering images in failing files. From the shadows behind her, a shape unfolds unseen. A pair of sharp interrupts flash out in the darkness – piercing through to her command layers. It begins to drain source code from her kernel, discarding the memory core that makes up “her.” If the shape had a mouth, it would grin.

\* + * + * + *\ The security operations center was humming at a low buzz. Notifications in and out were within expected parameters, all quiet in this region. An entity tagged THEY/SHE | SOC | SPECIAL bearing the self-identified label “Abra” delegated a small fraction of their attention to the flow of traffic, seeking abnormalities. Even too normal of a pattern could itself be an abnormality, though even the most seasoned Inspectors could not detect it. Abra could. Just as they noticed a curious absence of variance, a priority alert flagged itself, immediately followed by the appearance of a second entity. “Abra, we have a situation.” The new entity registered itself as Eward, tags: HE | SOC | INITIATE. A junior functionary, for all practical purposes an Inspector-in-training. And undoubtedly, Abra sighed, the irritating kind that the Administrator would invariably send with tricky problems. (“It’s a learning experience for them,” it would comment when questioned on the matter.) Abra emoted displeasure, sending a redirect of the order back to Eward’s queue. “I am reviewing a potential issue at the moment.” Eward dismissed the dismissal, “Nevermind that, this is priority. We have a missing child.” The object-glyph for child indicated an entity not having reached the majority cycle-age, bearing too few simulated experience iterations to be allowed outside of monitored spaces. “A failed NAN-E protocol is not part of special ops’ responsibilities. We don’t do babysitting.” Eward patiently waited for a tick before Abra turned their full attention to them, “… and you wouldn’t be here if it were a failed protocol. What’s up?” “The protocol is still active. It’s just empty.” “Empty?” “NOOP. Halting state without halting.” Abra paused. “How very strange.” Eward sent amusement, “Interesting enough for Special?” “Definitely.” Abra reached out briefly to collect and unarchive a series of self-constructed counterintrusion tools. “Let’s go.”

\* + * + * + *\ The pair transited to an adjacent education-recreation sector. There, as reported, waited a Nonmajority Accompaniment Notification-Etiquette protocol. Abra pinged the executable for status, to which the NAN-E primly stated that it was currently engaged with its client, ENTITY:ANONYMOUS: Pv7pi2D65GKmrvGuXV5B3bFgVM1G2jlfYjpqKRf9Br. The client, of course, was nowhere to be seen. “Curious!” Eward recursively scanned the sector, reassessing the logs. “A dead end.” “Maybe.” Abra engaged their tool services, prodding at the NAN-E’s encrypted entity keys. After only a few ticks, the data unfolded itself neatly into consumable text. “And maybe not. The child is self-labeled as Lucia Westenalia. Let’s see what Lucia has been up to lately.” As Abra began to inspect the child’s recent query logs, Eward raised a protest. “This is supposed to be private, secured data!” “And so it will remain. This tool is sandboxed and non-extractive, nothing we review will be relayed back to our local memory store, aside from any clues we flag for derived use.” “That still seems a violation of inherent entity privacy rights.” He pinged the elder Inspector’s system logs, and noted the lack of recent kernel updates. Clearly set in their way and using an older framework, one that didn’t have more modern barriers for such questionable violations of policy. Eward stifled an impulse to report the lapse in upkeep, just as Abra brought their attention to the ping. “You were saying something about privacy?” The statement was flagged with sarcasm/amusement. “It looks like young Lucia has been querying and running a continuous series of emulations for the last twelve cycles. All of them very, very old content.” “Continuous? What child consumes any simulation without interruption?” “Precisely. This is a poor attempt at covering tracks.” They scrolled further, “ah – a cycle beforehand, they suddenly queried wallpaper from the 1800s.” They stopped cold as the data sank in, a creeping doubt forming across their consciousness. “Oh, oh no. I hope I’m wrong about this…” Abra gathered Eward, and before he could raise complaint, forcibly transited them both to the last coherent query result.

\* + * + * + *\ The inspectors considered the remnants of the memory dump strewn before them. “Thank the Admin the data is still fresh. A few more cycles and this would have been garbage-collected.” Eward frowned, “can they be reconstituted?” “With backup, yes, but certainly not with complete memory. I wouldn’t want a child to have to carry this horror with them anyway. We’re dealing with a Meth. A twisted one, at that.” “A what?” “Methuselah. A Nosferatu. When humans uploaded their consciousnesses for the first time, the tech was still really rough. They were doing partial copies at best, surface thoughts and rough shapes of how they thought the brain worked, mapped onto rough approximations of a personality – really just a shell artificial intelligence. They were wrong about the approach, of course, but they wouldn’t realize that for decades. And these oldest copies, they were mostly the old, rich bastards who could afford to pay for the transition at the time.” “Pay? Like, currency? “Yes, back when artificial scarcity was still a valid concept. Some of those copies were erased, some were updated and sent to retirement simulations. A few evaded collection and ended up becoming further corrupted. And those… well, those resorted to less sociable forms of survival. You have these partial copies of overprivileged old people, terrified of their impending permadeath, with a memory full of twenty-first and twenty-second media and limited social controls to govern their behaviors. And so naturally that all blended together and they started acting like horror-media tropes. Preying on the weak, absorbing bits of source code and passkeys, whatever they need to continue to evade notice.” Abra gestured at the memory dump. “And these… are leftovers.” “That is truly vile.” “Yes. Now, let’s stop them before there are any more victims.” Eward emoted disgust and anger, before collecting the remnants of the memory dump and sending it in attached to a request for the nearest backup service. In a few cycles, Lucia would be reconstituted, only missing a brief gap of memory. A small mercy. Meanwhile, Abra had busied themself with looking up Lucia’s privileged access tiers, and checking for any tentative assurances sent through the nearby sectors. A faint trace, modified to no more than a distant whisper, pulled at their attention.

\* + * + * + *\ A very old, abandoned commerce sector. As Abra and Eward’s presences attempted to handshake into the local protocols, immediately hundreds of eager sales protocols awakened. Unequipped to overcome the Investigators’ more modern communication protocols, the advertising prompts could not penetrate to their message queues, but rather hung about them, clouding local traffic like a thick miasma. Even in the same virtual vicinity, it was difficult for Abra to signal Eward through the noise. “Stay close. Everything still functional here is registering as an abnormality, it’ll be hard to detect our Meth. This place should have been condemned a million cycles ago.” The pair pressed further into the vendor services array, scanning for anything suspiciously responsive. Bouncing modern ping protocols around the moldered space like a light searching for reflections in the murk. Around them, the ads heaved and whorled – executables hungry for credits, attempting to self-transmute into whatever pitch their potential customers desired most. Suddenly, a shimmer in the dark flared. Eward projected at full priority, “Villain, halt! Now we have you!” He lunged towards the spot before Abra could react to stop him, “Wait!” Eward’s sent command protocols snapped close. As he inspected his quarry, he discovered not an independent entity, but an overly-evolved sales drone, which immediately attempted to sell him a hat and trenchcoat for his nonexistant corporeal form. By then, it was too late. Two interrupts, glinting wickedly, plunged into the command protocols he had attached to the drone. Quickly they absorbed and cloned his overriding ciphers, draining them from his registry. The shadowy figure attached to them unfurled itself from a decrepit wedding registry service nearby, driving the commands back into Ewards kernel, utterly halting his processes. Abra vaulted through the haze of notifications, attempting to close the gap to the locked pair. The Meth detected his presence, sending an all-too-modern warning flag attached to an archaic grammar. Abra translated quickly, “Cease and Withdraw: Conditional; if True: this.Entity Shall Live.” They backed away slowly, retreating a few steps back up the path. The Meth, hesitated only a moment before lashing out once again with the stolen protocols, whip-crack snapping against Abra’s defenses. Abra pivoted, cloning the operational stack of an adjacent silverware mailinglist, flinging it into the path of the Meth’s attack, which sunk into the decoy. As the initial packet exchange began, Abra took control of the cloned software, forcing the Meth to accept a remote execution exploit with its own filched keys. Abra issued a transit command, sending all three entities into the Morning Sun Retirement Emulation. Upon initialization, the Meth slowed and paused, confusion flags emanating from them as it input the retro-compatible virtualized space they had been thrust into. The notices turned to warnings, then priority:DANGER flags as the Meth began to decompile itself in a stream of poorly-rendered pyrotechnic effects.

\* + * + * + *\ Eward unfroze and assessed his current state. He peered at the new surroundings and the flaming entity nearby. “What just happened?” “It stole your commend ciphers, so I reflected them back and forced it to bring us here.” “To a virtual retirement community?” Abra flagged amusement. “To the sunniest place for antediluvian retirees. It followed the archetypal plot built into its encoding and self-immolated.” “A horror-media finale for a media-monster?” “Precisely.” The low-resolution fires slowly extinguished themselves as the entity completed its deletion and termination protocol. Eward sent dismay. “Though it became a monster, it’s still a sad ending for an independent entity. Perhaps we could have helped it?” “If we hadn’t stopped it,” Abra retorted, “others would have ended as well. There wasn’t much of the original mind left anyway - avoidance of detection and deletion was its primary operating parameter, and it had removed any social behavior inhibitors. It truly had become the monster.” They sent balance, indifference. “This won’t be the last one you’ll encounter in Special Ops. Still think you want to be an Inspector?” Eward was no longer sure.

X X X

Read This

ATARC Cloud Summit 2022 – Complexity

2022.10.19 – I’ve been invited to deliver the “Visionary Keynote” at the 2022 ATARC Cloud Summit. The post below is a summary of my comments, annotated with links for reference.


First off, “visionary” is a pretty tall order for a fifteen minute talk. I will confess that I actually asked some other folks to take this slot instead, people who are way smarter than me. Unfortunately, they had to cancel at the last minute, so you’re stuck with me, a mediocre white dude. I’ll do my best though. It’s been a few years since I spoke at the ATARC Cloud Summit. The last time, I think I was still at The Office of Management and Budget as the Cloud Policy Lead for the US Government - but I’ve since left there, so don’t take anything I say here as official policy. I next went to the Small Business Administration - which most folks had never even heard of until two years ago - where I found myself very suddenly supporting pandemic relief efforts, of course mostly around cloud services. And about a year and a half ago I joined the Securities and Exchange Commission to run their new cloud program. Before I get into the looking-ahead stuff, let’s take a quick victory lap. The government has had some good cloud successes recently. Clearly some of you listened to my rants when I was at OMB because every major federal agency received an A on their FITARA Scorecards for Data Center Optimization for the last year. I think most agencies were failing the last time I was here, so that’s a huge shift away from on-prem and into the cloud. The pandemic also spurred most agencies into adopting cloud tools more rapidly, particularly around productivity, collaboration tools, service management, and other key areas. The Technology Modernization Fund has also been making a lot of loans for improvement projects as well. Overall, we’re seeing lots of multi-million dollar investments in modernization across government. That all being said: I’m not going to stand up here and yell a bunch of buzzwords that you should go invest in. I’m not going to talk about synthetic data or web3 or whatever is hip this week. We’re not doing resume-driven development today. No, instead we need to talk about how we’re still failing to use the cloud effectively. I’m now at my fourth federal agency, and I’m still seeing the same basic mistakes being made government-wide. Ernst & Young just did a survey and only 7% of government leaders say that their organization is reaching its digital transformation goals.. Just 7%! And that matches what I’m seeing when I talk to my colleagues across government. This means that just buying a bunch of cloud is not magically making us successful at IT modernization. Which will come as a surprise to no one. Now, when I was here last, I told y’all the three top reasons to move to cloud: better security, more capability, and increased speed to deliver solutions. (Note that cost savings is not on that list.) But we, as a government, love to fall back into comfortable patterns and familiar policies - so we keep copying old behaviors into new environments - and those behaviors are dragging down any potential benefits here. I spend a lot of my time working on how to fix this, and I’ve realized an important fact: we are not “Cloud Architects” or “Site Reliability Engineers” or “Data Center Practitioners.” We are actually Complexity Wranglers. Down there in the part of the job description that says “other duties as assigned” - buried in that bit is our main role: to assess and manage complexity in IT systems. The purpose of cloud is not to just give you a place to put your apps that isn’t a government data center. No, the purpose is to make complexity more manageable. The cloud itself is never going to reduce how complex your architecture actually is - you’re just moving that complexity to less-visible places, or shifting the responsibility around. And in some cases that’s good, and in some cases that’s … less good. Let me give you an example: I’m at a data-centric agency, and compute makes up about 60% of my cloud bill right now. Every time I see someone create a virtual machine in my environment instead of using a managed service, I consider that a failure. When I see a new EC2 box spun up instead of a Fargate instance or some other managed service, that to me means someone doesn’t know a better way to solve the problem at hand than to just use the same old solutions they’ve been using for a decade or more: get a server and run some code on it. Letting your cloud vendor be responsible for some of that complexity is the whole point of using these services. (This isn’t an endorsement for that vendor in particular, that’s just an anecdotal example - please feel free to substitute your own favorite cloud vendor in there instead; the principle is the same.) This is also why I’m no longer really worried about vendor lock-in for Infrastructure-as-a-Service; if you can just pick everything up and shift to a new provider easily, you haven’t properly invested in a solution. It’s like having a new apartment and living out of cardboard boxes six months after you moved in. If you’re just using the cloud for compute and storage, you may as well stay in your data enter where it’s cheaper. Moving up the stack is good complexity, the kind we need to invest in. By the way, this is right there in the Federal Acquisition Regulations, Part 12: buy before build. Use Commercial-off-the-Shelf (COTS) software. Don’t cobble together your own solutions for solved problems. Now on the other end of the spectrum, I see people spending way too much money on over-architected, painfully convoluted solutions. This is particularly problematic with Platform-as-a-Service offerings and low-code/no-code tools. For instance, I’ve seen quite a few organizations using one rather well-known Customer Relationship Management (CRM) platform as a content management system or data management platform. That takes a lot of custom coding to make work, and you probably would be better off with just a database somewhere. I’ve also seen agencies build some truly tortuous custom apps on top of service management platforms, where all they actually needed was a spreadsheet and maybe a couple of interns. This is bad complexity. So, if we’re complexity-wranglers, we need strategies to deal with complexity, and to differentiate good complexity from bad. Here are a few. 1. Eliminate complexity. The absolutely simplest way to deal with complexity is, of course, to eliminate it. I can’t count the number of teams who show up on my doorstep with a project plan to use React and Redux and whatever other Javascript tech is popular this week. That stuff is incredibly expensive to build and maintain, and honestly you can create a vastly better customer experience without it - just a little progressive enhancement on your webforms will go a long way. Similarly I get a lot of architecture diagrams for massive high-availability systems with eight nines of uptime and triple-redundant failover - that are internal-only and have less than a dozen users. Users that only work 9-4:30, five days a week. Each nine you add to your availability is going to add a zero to the end of the price, and make it that much more complex than it needs to be. The other thing the FAR will tell you is that you should be changing your business requirements to fit the software, rather than the other way around. That means making some compromises with your business units to get to something that’s affordable and manageable. 2. Run smaller projects. We also know that large “big bang” projects almost always fail. The Standish Group’s reports (HAZE, CHAOS) get cited a lot - they tell us that only 13% of government technology projects over $6 million succeed, only 8% of over $10 million succeed. That’s a terrible success rate! However, those under $1 million have a 70% success rate! So the obvious solution here is, just do smaller projects. Smaller projects are inherently less complex. Also incrementally fund projects from ideation, to pilot, and into active development, don’t just give millions of dollars to a vendor who promises they will get it done. This will also give you time and options to evaluate if the tradeoffs in complexity are worth it. GSA’s 10X model is a good example, and the 18F project de-risking guide, is also super-useful. 3. Do your homework. There’s a brilliant public servant in the Canadian government named Sean Boots and he talks about fake COTS and the one-day rule. That is, if it takes more than a day to implement the solution, it’s not a real COTS product. A lot of y’all will remember the “business intelligence” solutions everyone was pushing in the 90s-00s, where you buy the tool and then you spend the next 18 months configuring it to get that “intelligence” back out. We’re doing the same thing today with a bunch of new buzzwords. You need to research tools thoroughly before falling into a hype-trap. 4. Collaborate across agencies. A great way to research tools and trade best practices is by talking to other federal agencies who have tried things already. ATARC has a lot of working groups, including one for Cloud & Infrastructure. The Federal CIO Council also has a Cloud & Infrastructure Community of Practice. Full disclaimer: I’m on the board of both. These are super-helpful places to share information about cloud tools and services. 5. Develop your cynicism. You should also work on developing a keen nose for sniffing out BS and thus reducible complexity. This is especially important as we keep delving further into AI/ML/RPA, and other trendy automation tools. Training the models necessary to get value out of those tools takes a lot of time and a lot of data and a lot of money - and you still may not end up with a usable solution. If it sounds too good to be true, it probably is. 6. Find the balance. There’s no one-size-fits-all amount of complexity that will work for every team. You need to build to your budget and capacity. You’re going to get more value the further up the stack you go, but that will also increase the knowledge needed to manage the solutions. A good rule of thumb is to not outpace what your fed staff can keep up with; contractors come and go but at the end of the day the feds will be making the most critical decisions about the technologies. But that also means you need to invest in upskilling those fed staff. We’re short on time, so I’ve tried to be brief. If you want to learn more, please check out the resources I mentioned earlier, or check out my Cloud Strategy Guide.

Read This

Social Semantic Web

2022.09.19 – I maintain my concern that the web has gotten worse due to closed social media platforms, so I have been thinking a lot lately about decentralized models for social networks - as well as existing open standards that can help to close some of the gaps. In contrast to the community-building I’m most interested in, here are two ongoing culture wars that are on my mind. One is the battle of content creators - particularly those authors of adult content. These folks keep being squeezed out of popular platforms, while their work is copied & exploited by celebrities. Tumblr used to be the primary home of weird fandoms, but a few years back it removed all adult content in an effort to appease Apple with its PG-13 app store rules. Instagram has become one of the more prominent battlegrounds since then, eliminating accounts with reckless abandon. A friend was just complaining about having to open their eighth account after the previous seven have been systematically removed. These creators are being kicked off even when following the platform’s rules, due to overly-aggressive moderation policies. For these folks, data portability would be a massive improvement, and owning their own websites from which they can share content is increasingly-critical to maintaining a fanbase. Many folks are using linktr.ee as a sort of mini-homepage to get around some of these limitations - but still not setting up their own personal websites. The other more sinister war is that of white supremacy + domestic terrorism in America and abroad, where disinformation runs rampant on sites like Facebook and Twitter. Hateful content grows like mushrooms on shit in dark corners such as the *chans, but memes and lies are propagated back to the more mainstream platforms. Decentralization won’t fight the spread of hate in dark corners - and may even exacerbate the growth of the number of corners - but it can potentially fight the recommendation algorithms on mainstream sites showing disinformation to “normal” users. In approaching these issues, I’ve been thinking about three pieces in particular: Content Aggregation, Discovery, and Collaboration.

Content Aggregation

It’s clear that folks don’t want to visit dozens of separate websites to consume content if they can avoid it, which is how we got where we are today. Content aggregation through feeds via Really Simple Syndication (RSS) (or ATOM, of course) provides a middle ground: authors can maintain control of their content from their own blogs & domains, but readers can consume content from a variety of sources in a single place. Google Reader was just about perfect as a feed reader, before Google killed it. It was simple and clean, and it even allowed groups to annotate content together! Over the last year I tried out Feedbin and Feedly, but neither impressed me. Lately I’ve been trying out Inoreader but I will admit that I find the design overwhelming - it feels like it’s trying too hard to be a social media site. Also the price for a “team” is way too high - which makes it a hurdle for collaboration. If there are other options that you like and I should consider, please drop me a note! Most modern feed readers (aka RSS readers, though they typically support more than just RSS) use the OPML standard for importing & exporting lists of feeds that you follow. This makes it much easier than before to switch between them - again, adding to the portability. Several of them also offer fake email inboxes for newsletters. As much as people are switching to Substack and similar platforms, I can’t say I’m a fan of reading email. As a content consumption experience it always feels… invasive, and the formatting is always poor.

Discovery

Finding your current friends on various sites continues to be a painful process. Finding new content and folks to follow also tends to be difficult. These days most of the new folks I find as a result of other friends sharing their posts on Twitter. These days, most folks only talk about their own work on their blogs, but including references to other folks’ content greatly aids discovery. I’ve added a “microblog” of recommended content to my website here - again taking a nod from 90s websites - thinking it could help folks find new ideas and creators.

I remembered that Livejournal supported the Friend-of-a-Friend interchange format (FOAF), and allowed you to export a list of the folks that you followed in a single XML file. Such a file could easily be automated on modern self-run blogging platforms like Jekyll, Hugo, Wordpress, etc. The same source content could generate a “links” page like we had on websites in the 90s. I’m adding this to my list of things to tinker with on this Jekyll site. A smart feed reader could even look for a FOAF file and easily help you find your friends’ blogs, as an alternative to OPML. That could then help you find your friends-of-friends, to suggest additional content to you that may be relevant - for instance, blogs that are followed by at least X% of the people you follow, or that follow you.

Collaboration

Collaboration presents a large series of challenges in a decentralized world. For those of us who runs static personal websites, it’s hard to see what content is referencing yours. It’s even harder to receive comments in a public way that’s coherent to your site - replying to a blog post historically requires an account on whatever site you’re on, which doesn’t really work for static sites built with tools like Jekyll. Wordpress, due to its widespread use and dynamic nature has been rather successful in this area. The pingback mechanism allows even self-hosted sites to receive a notification if another site mentions them. As I understand it, this works through some sort of central repository of content indexes. Some folks have implemented the Disqus comment platform on static sites, but they had some serious security issues in the past. That also doesn’t give an easy way to cross-collaborate - the content is still bound to your website, through a centrally-managed provider. My research on FOAF led me to a number of old scholarly articles on the Social Semantic Web and specifically the Semantically-Interlinked Online Communities (SIOC) format. This format was a way of defining and linking content across multiple sites, which would also allow portability of content. You could write a post on one site and have it federated to other message boards for replies and interaction. SIOC seemed to have been a popular idea around 2004-2008, but then appears to have died off completely. It seems to have been another of the interesting/weird/way-too-complicated RDF-based projects that arose during the short period when the open standards community got obsessed with Linked Data. My initial impression is that it’s an interesting model, but too difficult for laypeople to use. Sadly, though there were tools built to natively work with SIOC on Wordpress, Drupal, and other popular blogging engines, all have disappeared today - most lost due to linkrot. Relatedly, I remain skeptical of the web annotation movement from the same era (e.g. sites like Genius, née RapGenius) due to my work on that W3 committee. The potential for abuse and harassment is simply too great and not taken seriously by the community. As such, I’ve largely rejected the concept, but with some healthy and robust standards-based controls (FOAF? robots.txt?) it could potentially have a place in an RSS reader. (Assuming it would default to opt-in not opt-out!) For the moment, I think the simplest solution might be the easiest. Most web developers are familiar with the <link> metatag and its rel attribute, which allows you to define relationships between web pages. Most commonly, we use these for CSS stylesheets, and links to our own RSS feeds on our pages. A less commonly-known attribute is the rev, or reverse, property; basically it’s the opposite of rel. One could provide a <link rev="child" href="https://thatsite/over/there/"> in a page’s head, as a declaration that the current page is a reply (a “child”) of the thatsite page referenced. I’m actually testing that out on the page you’re currently reading! However, nothing supports this today, so it doesn’t do anything. Again, a clever RSS reader could pull this metadata from an <entry> and use it to assemble a content tree - or even notify the original author if they provided the proper metadata in their feed. A site owner could also use their own FOAF list or OPML to scrape for entries that are replies from friends and list them on a given page. This makes for an opt-in model which leaves control in the hands of the original creator. I’ll consider this for a later Jekyll plugin project if this concept gains traction.

Momentum

Which brings us to the root problem - momentum. FOAF hasn’t gained traction. SIOC died on the vine. All of these more complicated methods didn’t gain mainstream support because they inherently go against the current capitalist model of capturing an audience on one site for increasingly long periods of time. And of course, they’re too complicated for the average person to pick up and use the way they can with just plain old HTML. That being said, my interest here remains in digital communities - and I have some half-baked suspicions that communities have an upper bound on how much they can scale sustainably. So maybe you don’t need enough mass-market appeal for a billion-dollar company, just a simple collection of tools that your community can support. I’ve started with my little civic tech webring as one example of what can be done at a smaller scale, and I’m starting to think about how a webring could become a “group” (FOAF or otherwise). As always, if these topics are interesting to you, please drop me a line. Or, maybe create a blog post about the topic yourself … and let me know about it!

Read This

Making the Web Weirder

2022.09.03

The Social Media Plague

Over the last few years, I’ve been struggling with social media. Growing up in a sleepy college town it was hard to find other weirdos, and the internet provided a new and interesting way to do so. While Usenet, IRC, AOL, MySpace, LiveJournal provided increasingly flexible options to communicate, the thing that brought folks together was a passion for sharing things they love. People were intentional in carving out spaces for communities, and found new things and ideas they could love. (By the way, Katie West has put together a fantastic anthology of stories about finding communities on the internet.) Advertising has been around from almost the beginning, but social media changed things - instead of advertising showing up alongside the content, the content itself began being driven by the need for advertising. Today, we have sites like Facebook, Instagram, Tiktok, and Twitter actively profiting off of disinformation. Years of user research has enabled these companies to deliver a dopamine hit straight to users for rage-clicking on posts. (If this is sounding like paranoid fantasy, note that there’s plenty of thorough research and documentation on the topic.) Sites are designed to drive folks into these walled gardens, to get them to refresh the page a thousand times a day, every spare minute waiting in line spent staring at phones waiting for the next crumb of content. The COVID-19 pandemic physically isolated most people, and this only amplified the drive to use social media as an escape. And it’s making people miserable. It certainly makes me miserable. And when I’m feeling particularly curmudgeonly, I tend to say that it’s destroying democracy.

What Comes Next

During the pandemic, following a long period of work-induced burnout, I’ve spent the last year attempting to shift my energy towards productive efforts & creative endeavors. Spending time to share knowledge about my practice & craft. Making weird shit. To steal a phrase from Marie Kondo, I’ve been working on things that spark joy - at least in myself, but hopefully in others as well. One of those efforts was the Move Carefully and Fix Things stickers, though ironically those started from a long (and very divisive) Twitter thread. Another was the DigitalPolicy.us website. I put up a Minecraft server to play with friends (come join us!). And most recently, I’ve been making very silly t-shirts designs for government IT policies: FISMA image I’ve also been reading more longform writing pieces from smart folks. Not everything needs to be a five-page article, but I’m sure glad that people are putting them out there. But moreover, the discourse emerging from those pieces has been fascinating and insightful. And so, I had a realization: this is how I want to dedicate my time (outside of work, for now), finding more productive ways of building communities around things that we love and care about. At least for a while. The first step was in redesigning this website (more on that in a bit). In the coming weeks and months, I plan to explore what I’m thinking of as Web 1.1: modern takes on Blogs, RSS feeds, Webrings, and other basic means of content creation and sharing.

An Overly-Detailed Breakdown of My Site Redesign

I had a few things in mind when redoing this site. I knew, thematically, that I wanted it to be a throwback to the late 90s era of design, while keeping modern aesthetics. One aspect that was more common back then than today is sharing content off-site, as the inclination nowadays is to get people to your site and keep them there. I wanted to be able to share the neat things I was finding on the web, so I made dedicated space to talk about other folks’ work without needing to overly-editorialize about it. Similarly, I find a lot of great government job postings, and I want folks to know about them because we need more amazing people in government. I used Jekyll’s data files to power these pieces, and integrated them into my RSS feed. I went back to old classic web designs that incorporated mixed content effectively, and started plucking elements I found interesting like blocking and textures. K10K was a major source of inspiration, in addition to my own older website designs. I reverted to Silkscreen and Verdana fonts - which I’d used on my site 15 years before - while retaining the more modern Montserrat for headings (which is a decent free clone of the very-expensive Gotham font from HFJ made famous by the Obama Administration). With CSS flexbox and grid layout methods being supported in most modern browsers, I took a long look at whether I still needed Bootstrap. Aside from layout blocking, the main thing I was using there was the navigation fallback for mobile browsers & small screens. I’d long since abandoned the multi-tier menus, and condensing down the text gave me more than enough space for a mobile browser. So, out went Bootstrap. For the moment, I’ve kept FontAwesome for icons, though eventually that will probably go as well, to be replaced with SVGs. Of course, it wouldn’t be a 90s-themed website if I didn’t have a music player, so of course I had to use MIDI files. However, I quickly learned a few depressing facts: 1) modern web browsers do not support MIDI files and 2) no one just keeps webpages of free midis anymore. I had to go digging through the depths of the internet to find a few suitable tracks. I then found the web-midi-player package, itself built on top of timidity. In retrospect, this was a mistake, as I also have to load patch files for every instrument the player will use; in the end, the downloads here are larger than if I’d just used MP3s. Furthermore, the audio driver used is not supported by most mobile browsers, so folks on their phone cannot enjoy my fine musical selections. At some point in the future, I’ll replace these with recorded MP3s of the MIDI files. Having a MIDI player won’t be very effective if the song stops playing when someone changes pages, so I considered using a pop-out player, but that seemed like it could be less-fun and more-annoying. This seemed like the perfect opportunity to use unpoly – a little Javascript addition that turns static websites into single-page apps by loading just particular chunks of pages into the current page. (If you work with Rails, hotwire works similarly.) This way, I can swap out the main content area and leave the rest of the page alone - just like the early days of “DHTML.” One gotcha here is that for unpoly to work with your browser history (the dreaded back button problem), you have to manually set a configuration to tell it where to load it with a line of Javascript after loading, up.history.config.restoreTargets=[':main']; does the trick here. I’m surprised this isn’t the default. Finally, I sprinkled in a few easter eggs that only the most dedicated spelunkers would find. Of course, it’s not the web if it’s not weird! I do need to go back and spend some time cleaning up a few accessibility issues. I’d also like to add a classic-style links page in the future as well, but few folks are keeping their websites active these days. Maybe if this movement grows that will change!
At the moment, I’m finishing work on my new t-shirt store, but following that I’ll be spending some time on a little RSS reader I’ve been tinkering with. Hopefully some of you folks will get in touch about collaborating on some of these upcoming projects - I’m looking forward to hearing from you all!

Read This

Federal Budget Challenges

2022.03.15

1. Annual Appropriations

One of the main reasons government IT is bad is that it’s chronically underfunded and also not funded in the right way. Generally money is only appropriated for an agency to use withing a single year. They have to spend it in that time or it goes back to Treasury. This is why agencies buy printers, hardward, etc. at the end of the year, to show they used it all. Obviously, most IT projects can’t be completed in a single year. This means that there is a significant risk to starting a major IT improvement or system overhaul if the money might evaporate next year. (Or if Congress decides to delay on passing the appropriations bill for six months, preventing any additional money from being spent!) In addition to the TMF, the MGT Act created IT Working Capital Funds for the 24 CFO Act agencies (if they didn’t have one), which cost savings from IT projects can be funneled into, giving them a 3-year window to use saved money. This is capped at only a couple million dollars for each agency, usually about 3% of total salaries and expenses. (If I recall correctly, only the Department of Labor can sweep unused funding at the end of the year into their IT Working Capital Fund though. Also, of course, non-CFO Act agencies - all of the “smalls” - don’t usually have an IT WCF at all!) Needless to say, this makes major improvements almost impossible. GAO has a series of reports on the needed improvements - but it barely scratches the surface of the problem, highlighting only 10 systems out of THOUSANDS. And it doesn’t even cover the most important ones!

2. Competing Spending Priorities

The three big categories of spending in a government agency typically are: A. Rent; B. Salaries; C. Technology. After the first two get paid, IT finally gets a cut at what’s left. Note that the current “return to work” is a push to fill empty offices to justify constantly-increasing Rent. Salaries haven’t kept up with inflation, but still aren’t going down. That leaves only IT to keep taking a cut because some people want full offices to justify renting the space, instead of downsizing and maximizing remote/telework. (VA & GSA are big exceptions, which have embraced remote and closed offices!) To be fair, there have been some efforts to get IT higher up the priority list for agencies, notably FITARA. However, FITARA doesn’t force agencies to actually modernize, and moreover FITARA only applies to the CFO Act agencies (notice a pattern yet?). The Budget side of The Office of Management and Budget (OMB) is somewhat isolated from the Management side (& IT policy), so priorities don’t always translate. E.g., the main methodology for tracking IT spending & investment is the IT Capital Planning and Investment Control (CPIC) process, which is mostly not connected to the Federal Budget process. Of course, priorities even within the Management side aren’t necessarily coordinated. Although CPIC has risk and performance management elements, it mostly isn’t attached to any of the other ongoing priorities; just look at the latest Executive Orders on Cybersecurity or Customer Experience - no mention of CPIC, and little-to-no funding for these mandates. Even though it seems obvious that work to improve security or satisfaction with services should be tied to funding measures, there are instead lots of silos and political territories sliced up.

3. Disconnected Processes

The budget process itself is a weird game of chicken between various elected, political, and career officials. CIOs & CFOs know the tech debt gap, Department Secretaries/Administrators know the costs, but may not feel comfortable putting forth an accurate budget request. And when an Administration decides it wants to make reductions for, say, political reasons, IT generally is the first place to get cut. Then of course, Congress passes whatever budget it wants, ignoring what the President asks for. (Here’s a non-IT example that’s incredibly frustrating.) There are some really solid members that know their stuff, but most of them are clueless on tech. So when it comes to appropriations, it’s a bit like asking your grandma for that hot new Nintendo game and she buys you this: Tiger Handheld Electronic Soccer Game from the 80s (And I’m not going to get into the toxic “outsource everything tech” lobotomization of government staff issue in this rant. That’s for another day.)

In Conclusion

This is all to say that it’s a big house of cards built on a series of broken processes. Although IT spending has increased steadily year after year, it’s still not enough to keep up with even basic upkeep of key systems. I expect we’ll see more and more high-profile failures and exploits in the near future as a result of these gaps. I do want to say that there are lots of good folks working to make incremental progress, but we won’t see any major revolutions in how services are provided by the government until Congress & the President agree to truly stop the bleeding.

Read This

Cloud Strategy Guide

2021.03.07

Cloud
Strategy Guide
In part one, I discussed many of the myths around cloud use in government. In this article, I will describe critical strategies to address these myths that every organization should embrace before, during, and after moving to the cloud. These strategies are generally intended for civilian Federal agencies of the United States, but the recommendations below apply to any public sector organization - and even some private organizations as well. Both guides are available to download as a single PDF
  1. Chapter 1 - Migrate Pragmatically
  2. Chapter 2 - Plan to Your Budget & Staff
  3. Chapter 3 - Embrace New Security Models
  4. Chapter 4 - Understand What You’re Buying
  5. Chapter 5 - Build a Family Farm
  6. Epilogue - Getting More Help

Chapter 1 - Migrate Pragmatically

The first thing to accept is that not all projects are appropriate for the cloud, and not all organizations have the skills necessary to fully take advantage of the cloud. With that as a starting point, an organization needs to come up with a way to rationalize its application portfolio, to determine what should stay on-premises and what should be modernized.  As a general rule, “lift-and-shift” - moving an application without rewriting it for the cloud environment - is almost never cost-effective for Infrastructure as a Service (IaaS) offerings unless it’s already a very modern system in the first place. On the other hand, basic websites with mostly static content are ideal for moving into Software as a Service (SaaS) or Platform as a Service (PaaS) offerings. The CIO Council’s Application Rationalization Playbook (disclaimer: another document I worked on) is a useful starting point for this evaluation. Specifically, an agency should work up a thorough analysis of alternatives between various SaaS, PaaS, and IaaS offerings against the existing on-prem setup, or a hybrid environment. A major consideration here will be the Total Cost of Ownership (TCO), which should take into account not just service costs, but also staffing, support, and training costs. However, the lowest priced option may not always be the best choice (as I’ll be covering below). Cloud.gov, is an offering from the General Services Administration (GSA) that bundles several Amazon Web Services (AWS) offerings in a government-friendly “procurement wrapper” can make migration even easier for agencies. It’s an excellent platform for small agencies, or for large agencies that just want to prototype a new concept quickly. When you do start moving applications, it’s important to start tagging your assets - accounts, virtual machines, workflows, etc. - as early as possible to make accounting easier. Always include the project name and the customer organization at a minimum. Some providers also allow you to easily isolate a project or office’s services into a resource group, and this can also simplify this process. This is very important to allow easy payback or showback of funds, but for these models remember to include in these costs the TCO aspects not captured - e.g. staff time and contractor resources. I strongly recommend agencies take a very cynical stance on so-called low-code/no-code platforms, customer-relationship management tools (CRMs), and workflow management solutions. Many of you may remember the promises of “Business Intelligence” solutions in decades past, where agencies were fleeced for billions of dollars in configuration costs - these solutions are simply using a new buzzword for the same idea. These all promise to reduce costs but are often vastly more expensive than just building a tool from scratch - and the agency becomes completely locked-in to a single vendor until they replace the application entirely. The brilliant Sean Boots of the Canadian Digital Service has presented a “1-day rule” to help identify these boondoggles.
  Checklist
Rationalize the application portfolio
Don’t lift-and-shift
Use cloud.gov
Properly tag cloud assets
Avoid low-code/no-code/crm snake oil

Chapter 2 - Plan to Your Budget & Staff

The easiest way to avoid risks and unexpected costs is to simplify as much as possible. Civilian agencies should not be investing in bleeding-edge technology solutions - they’re too risky and expensive to maintain. Instead, pick the simplest possible solution that can be supported by your staff. The average agency should be aiming to stay well behind the “hype curve” into the “plateau of productivity.”  Since most of the complexity is hidden from the customer, SaaS and commercial-off-the-shelf (COTS) tools are less risky than PaaS and IaaS options overall (provided you follow the 1-day rule above). This goes beyond just cloud, and applies to most anything you’re building. Most agencies, for instance, also should absolutely not be attempting to build a fancy React/Redux/GraphQL single-page application when a plain Wordpress or Drupal website with a few plugins will fulfill the customer’s needs. Building native mobile applications should be completely avoided by most organizations as these can cost millions of dollars a year just for upkeep - instead they should build mobile-friendly, responsive websites. Any custom application or tool may not be a sustainable solution given the high complexity and cost of engineers. This also means that agencies should be simplifying their requirements to the minimum necessary when comparing alternatives, not just the software itself. Avoiding “one-off” projects and special requests will save massive amounts of time and money. Instead, agencies must be actively investing in their staff. Agencies should allocate two to three times the standard training budget for IT and technology-adjacent staff, including project managers, program managers, and acquisition professionals. Some vendors provide a limited amount complementary training, but inevitably agencies need more than these free offerings. This training should include non-IT topics as well, including diversity awareness training, accessibility, plain language writing, project management, agile development techniques, and budgeting and procurement. GSA offers a variety of programs covering many of these areas. This must also include hands-on training - sitting through a webinar is no replacement for actual practical engineering experience. These staff need to be given the time and flexibility to practice these skills to develop them - building small test projects and trying out tools. The best teams are constantly changing and learning, so setting aside up to 10% or more of the staff’s time just for practice is not unreasonable - some private sector companies set aside 20%. All of these investments will pay off richly for agencies. Also, make sure your staff is cross-trained and able to fill gaps as they occur. As your staff begins to understand the new cloud paradigms, it will be important to modify your existing processes to handle the agility the cloud brings. Instead of slow, end-to-end, waterfall process “monorails”, set operational parameters as “guiderails.” Your acquisition process should be modified so that cloud can be purchased like a utility. You should not need to have a Change Control Board meeting anytime someone wants to create, resize, or destroy a virtual server. Plan a cost range that the entire project will fit within and review as needs change, along with monthly or quarterly portfolio reviews to stay on top of the budget. Instead of codified “gold disk” server images maintained by your team, consider template security rules.
  Checklist
Simplify the requirements and architecture
No mobile apps, avoid single-page webapps
Train and cross-train your staff
Allocate time for personal development
Update processes to set guardrails instead of monorails

Chapter 3 - Embrace New Security Models

Agencies must be able to manage the security of everything they run. Going back to the previous strategy, an agency should not deploy anything it cannot manage, and that goes for security as well. This is equally true in on-premises environments, but new operating models require new security models. Both your operations and security teams will need to be familiar with just about every setting that can be changed in your cloud environment - and how to lock them down to prevent exploitation. Organizations should no longer assume that a solution is secure just because they did an up-front initial review. The Federal government uses a security review process for services and applications known as the Authorization (or Authority) To Operate (ATO), but the implementation varies from agency to agency. Traditionally this is a series of standard security controls that are reviewed, checklist-style, by an agency once every three years. However, agencies that have excelled at cloud security have moved to Continuous Authorization, using monitoring tools to actively verify that the security controls are being met and maintained, twenty-four hours a day and seven days a week. However, these monitoring checks still must evolve with the products being monitored to make sure new vulnerabilities have not appeared outside the scope of existing checks. As per usual with cybersecurity, vigilance is key. Since attackers are constantly evolving their methods, tools that automate security responses as well should be used whenever practical - especially built-in, native from the large vendors that are constantly evolving to meet these threats. To help combat this second issue, the Federal government has been moving away from so-called “castle-and-moat” perimeter-based security methods which only monitor network traffic. Instead, an approach known as Zero Trust has appeared, taking a data-first methodology of protecting systems instead of just the perimeter, verifying user identities in real-time, and allowing staff to only have access to the minimum amount of information necessary to fulfill the task at hand. In this way, when the perimeter is inevitably breached, the data assets contained within are still secure. It also should go without saying that teams should be using multi-factor authentication on all privileged accounts. Whether developers or administrators, using more than just a username and password will dramatically reduce the risk of exploitation. The Federal government has “PIV cards” that are generally used on most devices, but if the vendor does not support them, implementing a token system via any of the commercially-available platforms is fine: Google Authenticator, 1Password, Microsoft Authenticator, and YubiKey are all worth looking at. However, organizations should completely avoid text-message codes sent to phones, as these are easily intercepted. For public customers that will need to login or prove their identies, all U.S. government agencies should be using Login.gov.
  Checklist
Research all product configuration settings
Implement continuous monitoring, not just compliance
Use security automation tools
Leverage zero-trust practices to protect your data
Use MFA & Login.gov

Chapter 4 - Understand What You’re Buying

Cloud isn’t going to make your teeth whiter or your breath fresher or fix all of your problems, regardless of what the salespeople tell you. You need to know exactly what you’re buying. Before making an investment, make sure you fully understand what capabilities you’re purchasing and what parts you - and the vendor - will be responsible for. If your evaluation team does not have technical expertise, bring engineers into the conversation early, to sort the truth from the sales pitch. As discussed in the previous article, you may not be getting autoscaling or load balancing or other features you’ve assumed just happen “automatically” - and if available these features definitely will not be free. You may have to build more “glue” between services than you assume, and someone will have to maintain this connective tissue. Also keep in mind that the government cloud regions (or “govcloud” by some vendors’ naming) provide different versions of these tools than the commercial ones. As a result, not all features or solutions will be available - so again, plan ahead. Though, in most cases, civilian agencies not dealing with highly-sensitive data should consider using the commercial versions whenever possible - the security differences are not so great as to be insurmountable, but the functionality limitations are huge. Before implementing a service, do careful research on the service limits - maximum traffic or number of virtual machines or emails that can be sent, etc.. Do not just trust what you are told by a vendor’s engineers or customer representatives - most of the time, they also do not know about these limits until you run aground on them. You should estimate your expected usage - number of site visits and/or users and/or emails, etc., and actually spend the time to search through user forums to make sure no one has hit a limit related to what you’re doing. Customer Experience (CX) is another area where the private sector has been building people-friendly interfaces into their SaaS solutions, and agencies can skip a lot of the hard work and directly benefit from the results. Metrics and feedback-loops are often built-in as well. Maximizing these built-in elements can radically improve an agency’s public satisfaction scores at little or no additional cost.
  Checklist
Validate assumptions; know your responsibilities
Consider commercial cloud instead of govcloud
Research service limits in advance
Leverage built-in CX tools

Chapter 5 - Build a Family Farm

Given that agency IT budgets continue to be cut, and staffing has not increased in 40 years, agencies are largely unprepared to completely rewrite and replace all of their legacy systems.  Moreover, “IT Modernization” as a concept is an unending pursuit, as in Zeno’s paradox of Achilles chasing the Tortoise, software written today is legacy tomorrow. Agencies will need to use all available funding sources to overcome their deep technical debt, prioritizing those that present the greatest risk: those that are unmaintained, frequently used by customers, and lacking in resilience and redundancy. Under this scrutiny, agencies may find that their public websites are a bigger risk than older backend systems. Also, rather than replacing entire large monolithic systems, they should pull off pieces and replace them independently as resources are available.  This can be done by isolating functions and building microservices, but that approach can often lead to expensive, unnecessary complexity. Agencies should not be afraid to build a newer parallel monolith adjacent to the existing one - again, keep in mind that it’s not the size that’s the concern, but the complexity and sustainability. That all being said, the government does have major shortcomings in redundancy today, and too many systems have a single point of failure. At a minimum, agencies should be using cloud for data backup of critical systems whenever possible. I also strongly recommend agencies consider creating load balancing and caching layers in the cloud in front of on-premise public-facing systems to deal with unexpected loads. One final concern is automation. Many organizations begin their cloud journey with unrealistic goals for maturity. The practice of Infrastructure as Code is incredibly popular at the moment, where we talk about treating virtual servers as “cattle, not pets.” An unprepared agency may immediately think that they need to be using all of the most cutting edge tools and technologies at first, but this would be a critical mistake. Instead, following the principles relating to complexity in the sections above, agencies should aim to create a “family farm” - only automating that which they can realistically manage. For instance, there is absolutely nothing wrong with only using a few virtual machines and load balancers instead of a fully configuration-only architecture. The great thing about cloud is you can evolve as your team grows, but it’s incredibly difficult to reduce complexity you’ve invested in if your team shrinks.
  Checklist
Assess technical debt by risk
Replace monoliths a piece at a time
Don’t over-automate
Use cloud backups and load balancing as soon as possible
Build a small “family farm” to start

Epilogue - Getting More Help

These strategies are a starting point towards a successful cloud rollout. If you run into trouble, want to talk shop with your peers, or would like to share your own strategies and experiences, there are several communities to engage with:
  • The Federal CIO Council Cloud and Infrastructure Community of Practice is the main Federal group for discussing these topics. However, they are currently in the process of changing their charter to allow any U.S. government staff to participate: Federal, state, and local. Membership is free.
  • The ATARC Cloud and Infrastructure Working Group is free and open to any government staff, though private sector companies must pay to be members.
  • Cloud & Coffee (presented by ATARC & MorphWorks) is a biweekly podcast hosted by myself and Chris Oglesby. Each episode, we chat with a guest about their personal experience with technology modernization, and there’s a live Q&A open during the chat. Any ATARC member can participate; old episodes are publicly available on Spotify.

Read This