Hiring and Programming Tests

2020.12.11 – I don’t mean to call anyone out, but I want to talk about programming tests in hiring and contracting. Stop doing them. This post originally appeared as a thread on my Twitter account. I’ve reposted it here for posterity with additional context. I understand that people want to assess whether or not someone is capable of doing the job at hand. The problem is that every engineer, no matter how senior, is learning on the job, EVERY DAY. All of us lookup solutions on the internet, all the time. Your assumption of what should be “baseline knowledge” for one person is based on your own experience. If you learned programming in college, a bubble sort algorithm is probably just muscle-memory by now; for the rest of us, it’s completely irrelevant knowledge. It also has nothing to do with 99.9999% of jobs. Unless you’re working on some bleeding-edge optimization for a major, that sort of fiddly info is way less relevant that principles and practices. It’s just a great way of showing your biases towards “traditional” candidates. The worst interview a company ever gave me was to implement a hashing algorithm. I’ve been writing code for 35 years, I’ve never had to do this manually. I failed. It also told me the company didn’t know what to ask me, I immediately withdrew my application. Tech isn’t about algo knowledge. Bad news to my younger-self: it’s also not about passion. It’s about knowing how to solve problems. Instead of coding exercises here are a few ways to figure out if someone will be a match for the job you’ve got. In short: don’t ask questions that have a single, expected answer. If you know what the answer is, you’ve created a knowledge-based test - and just like with authentication, you’ve created a bad one.

  • Ask them about a problem they’ve solved recently, have them outline the steps they went through.
  • Ask them about a piece of technology that has made their life easier in some way. Whether it’s a build tool or kitchen gadget, how does this relate to what we do?
  • Ask them what new tools they are excited about, and what makes their life as a developer better? (I specifically avoid saying “fun” here, because working for a paycheck not passion is a perfectly valid reason to be a technologist.)
If you want to hire senior developers:
  • Ask them about building for audiences that are not like them? How do we create solutions for inclusion?
  • Ask them about why they chose to not use a piece of technology. How did they evaluate it? Cost, maturity, stability, community?
  • Ask them what makes a good culture for a team. What makes a team productive and empowered?
  • h/t @justgrimes: Ask them to explain how the internet works. Details don’t matter, you want to see if they can read a room and communicate to the level of their audience.
  • My favorite EQ question is “what’s the last good thing you’ve read.” Book/blog/T&Cs/etc. I always add the answers to my reading list - but also high EQ people will usually reflect the question and ask you back. (Unless they’re nervous, so DON’T go on this alone!)
Here are a few other ideas I received on Twitter:
  • via @abbeysuekos: “I’m also starting to get into questions about how people work with those above them (managers/leaders/executives) and those below them (ICs). There are never enough ?s about how you treat people with less power than you… I love questions about projects that didn’t go well. Plenty of good candidates will struggle or even get defensive, and that’s a flag for me. Important to handle the framing carefully and create an explicit safe space, but when done well it’s golden.”
  • via @HimalMandalia: “Had an interesting experience with a tech test years ago. Turned out it was one I’d done already, so told them and said “how about instead I show you and talk through what I’m working on at my current place?” They agreed. So did just that. Got the job.”
Do you have other questions you use to help identify talented candidates, without jumping through programming test hoops? Tell me about it here!

Read This

Welcome Home

2020.11.09 – I see many of my former technology-colleagues now suddenly eager to return to government- or join for the first time- and I’m very excited to work with you all again! That being said, here are a few thoughts from someone who stuck around for the hard parts over the last 4 years.

  1. “Move fast and break things” failed. As a result, we inherited a lot of fast-moving broken things. Sustainability is the most important principle in government tech today. “Move carefully and fix things.”
  2. “Build with, not for” - Cuán McCann (that one is still the most important and gets its own tweet) Note: Cuán’s talk at CFA Summit in 2014 begins with “At the risk of creating a massive existential crisis…” and the following five minutes created one for me. It completely changed how I look at the world and approach The Work.
  3. Technology is almost never the solution to the problem. You need a deep understanding of culture, policy, budget, acquisitions, etc. to be successful. We don’t need ANY more shiny new websites or hackathons. Your first year should be spent understanding the systems.
  4. Fam, choose boring tech over shiny. Those mainframes and COBOL still work just fine after 50 years of service. Those React apps you’re writing are legacy before they launch, at a hundred times the cost, and no one can run them when you leave - making them abandonware.
  5. Government doesn’t need disruption, or even innovation. Many of us who came in as “innovators” are now the career bureaucrats just keeping the place from burning down. Listen to our expertise and work with us.
  6. People don’t want to hear this, but… this isn’t a job for tourists. Building relationships to cause change takes time. If people know you have one foot out the door, they’re not going to trust you. Think about what you’re willing to sacrifice before signing up.
That all being said… Welcome Home. I’m looking forward to collaborating with you all soon.

Instructions for the Zine

Read This

The Secret Equation of Job Satisfaction

2018.12.16 – Many lengthy books and articles have been written on how to be fulfilled at work, good management, and keeping your staff engaged and excited. However, I’ve found one simple equation that is the secret to all of these. Satisfaction = Volition / Friction If you’re not satisfied with your job, it’s probably because you don’t feel effective or that your work has the desired outcome – your volition, or because there’s too much resistance, bureaucracy, barriers, and day-to-day minutiae for you to enjoy the work – the friction.  Satisfaction is a measure of your volition over the job’s friction. More volition increases satisfaction, as does less friction. Let’s say that your control over your job is around 5 on some magical undefined scale – you can do the things you need to some of the time. However, the amount of paperwork you have to fill out to do the job is around a 10.  5/10 = ½, so you’re probably not going to be very satisfied. On the other hand, if you have lots of control and can make all the decisions you need – let’s call this 10 – and you have no daily design-by-committee meetings to wrangle – let’s give this one a 2, you’re at 10/2 = 5, which is looking pretty good. You don’t really need the scale or the points here, it’s just to show how the two relate to each other. The ratio of volition to friction will determine your level of job satisfaction. Research suggests that people feel most fulfilled when they are being challenged just beyond their current capacity. Too little challenge or too much challenge become either boring or overwhelming. Although being able to direct your work is the most obvious component of volition, it can include many supplementary factors. The alignment of your skills and background to your work can impact your volition. Feeling that your work has an impact on the world can also be a major element. Friction is comprised of several components as well. If your organization has unnecessary processes and procedures, those barriers to productivity will cause friction. If you work more than 40 hours per week or your commute is long that will likely cause friction by cutting into personal time. If your organization requires extensive reviews and buy-in from multiple stakeholders to accomplish tasks that may cause friction. Anything that causes you to become quickly exhausted at your job is likely adding to the friction. To increase your job satisfaction, you’ll need to increase your volition in the job or decrease your friction. To increase your volition you can take many steps in your current job.  This includes outsourcing or delegating unwanted tasks, taking on different projects that you enjoy more than your current projects, learn key information or skills, increase the level of challenge by taking on harder projects, decrease the level of challenge by collaborating with experts.  To dramatically increase your volition you may need to find a new job. To decrease the level of friction there are many things to do as well.  You can outsource and delegate tasks to decrease your weekly hours. You can work from home several days to decrease your weekly commute time.  You can decrease the number of weekly meetings with your teams. Any of these can make the job feel easier even if the workload hasn’t decreased. This isn’t to say that there aren’t terrible jobs in the world that no one would ever enjoy. You may be in a job where neither of these variables can be changed enough to make a major impact on your happiness. It’s a good idea to regularly assess where you’re at on this scale just to make sure your satisfaction isn’t slipping away. If you’re familiar with Agile methodologies, this should sound somewhat familiar. “Individuals and interactions over processes and tools” is a core tenet of the Agile approach. Putting the individual in control and reducing the cruft of processes is just good practice. As a manager, I often use this equation to help increase my staff’s satisfaction as well. I may not be able to always give a staff member more control over their work, but I can drop half of our check-in meetings to free up their time, or find ways to reduce the amount of paperwork they have to file to do their job. However, in many cases I can do both at once – by allowing staff to act independently in their projects with less oversight. This brings up another equation that I’m fond of: Trust + Autonomy = Delivery In general, if you hire good, talented people, all you need to do is trust _them to do their job and give them the _autonomy they need to do it, and they’ll deliver good work. (Buckminster Fuller talks about this in Operating Manual for Spaceship Earth, in describing the “Great Pirates.”) Giving your people the space they need to execute and support they need to see their work through will build trust and autonomy. Best of all, trust can increase volition, and autonomy can decrease friction, resulting in delivery and satisfaction. Each one of these components can be broken down into various topics for in depth study.  The high level equation helps identify what the missing component is to getting the desired results.  I hope you’ll keep this equation in mind when you’re approaching your own work, and see if you can find ways of creating a better balance.

Read This

The Hype Market

2018.10.24 It takes time, money, and other resources to execute on any new information technology initiative. In theory, there should be a return on investment for any new IT development. However, in the current market landscape, a lot of effort is being wasted on reinventing the wheel or misapplying solutions, rather than driving towards proving greater value to citizens and customers. The relatively low cost of experimentation and implementation of processes result in a fetishization of abstraction, leading to further and further complexity even if a proportional benefit is not achieved.

Technology is a Bridge

Most businesses and government services aim to connect a person’s intention to an outcome. Whether we’re helping someone call a restaurant to make a reservation, or helping a Veteran to understand their benefits, we’re aiding someone to solve a problem. In short, we’re bridging the gap between a customer and a service. Technology is introduced into the process of delivering a service to a customer to automate one or more steps, it effectively shortens the gap. In general, technology is not seamless in this process – that is, there are almost always steps between the customer’s intent and the resulting action that have to be filled through human intervention or additional services. You can’t think “pizza” and have a pizza instantly appear  – but an app to order pizza removes some of the burden of calling the pizza place and having them take your order, prepare it, and deliver it to your doorstep. Today there are new automation tools available to practitioners who want to create access to services through technology. If you want to create a service so that someone is able to order a pizza, you can set up a website or a mobile app to do so, because there’s already computers, and the Internet, and web browsers on the computers, and protocols for everything to talk to each other. So for anyone trying to solve a problem, the real value is in making the process simpler or easier through further automation – say, having your website be able to actually show the order automatically at the pizza place on a screen, or automatically send you a text when the pizza is out for delivery with an ETA. Based on the previous diagram, the goal is to expand the boundaries of technology to further fill the gaps between the customer and the service. However, that gap is not where the technology industry focus most of its time and money. Software engineers are spending more and more of our resources mucking about with tools that are simply thinner and thinner slices of micromanagement around already-solved problems. In practice, the industry is just creating additional layers that are largely unnecessary for the end customer. Here’s a real world example, again using a website.  In 1998, it took a few hours to set up a website. Most of our time was spent on designing it, then hand-coding HTML to post on a webserver.  As our tools evolved, the time to perform these steps decreased, but the complexity of the tools themselves has increased dramatically. Today, it can take ten times as long to create a basic website as it did decades ago. After twenty years, technology has moved a little closer to the person requesting the information due to improved User Experience, and the end user can get to the information a little faster and easier, but mostly there’s lots of additional complexity in the middle layers. These middle layers have created a very rich market of expensive services and skills based on these bleeding-edge technologies.

Blockchain: A Parable

As a result, the world has seen the rise of numerous solutions with overblown promises of impact, fueled by speculation of venture capital. Blockchain (or distributed ledgers) is a popular example, which has elevated itself with an innate brand of “value” due to being associated with Bitcoin, a popular cryptocurrency using the technology. Most computer systems take data from one person, put it into a storage location (usually a database), and then output it for someone else’s review. One problem that people have with these systems is that there’s no way to validate that the data submitted by the first person is the same as what the person on the other end receives. There are numerous points along that path where a malicious actor – or broken process – can modify or corrupt the data. That can be a hacker exploiting a vulnerable system, data corruption due to failed hardware, or any number of other failures. The submitting person could also just be lying about the data, or the person receiving it could lie about the results. Companies selling blockchain as a solution make the promise that these points of failure can be eliminated by having an authoritative record – or ledger – of all changes to the data. However, in reality, blockchain fulfills only a small piece of this process. In most cases, it simply replaces the database with another type of database, which provides additional checks. This still doesn’t solve the original problem of ensuring that the data is valid on both ends. To visualize this, let’s say I have four apples.  I write down on paper that I have six apples. You read the paper and double check that I do in fact say that I have six apples. You have no proof I ever had six apples, only the promise that I did.  You could then go report that I have eight apples and that you double-checked that fact and you promise that it’s true. This is how blockchain gives a false sense of security.

Addressing the Market Forces

And still the sales pitch still works!  Companies are spending millions to adopt “blockchain” technologies – many of which are so stripped of features that they can’t even be called a distributed ledger anymore. But even if purchasers don’t know the difference, technologists must know that much of this additional complexity is merely hype. Why would software engineers want to use it in the first place? One way to consider this is that engineers obsess on generating higher and higher orders of abstraction. Abstraction, however, generally increases complexity by moving concerns to a higher order function.  Think of going from a wheelbarrow to a bicycle. Bicycles can reduce the amount of energy needed to move a load around due to gears – but those gears are also more complex than just an axle and wheels. Suddenly you have extra parts to keep up and grease, and to build them from scratch you’d need to understand the math behind gearing ratios. With physical processes, resource limitations and cost-effectiveness become natural restrictions on automation. But in software, there are rarely such restrictions; rather, software developers are actively encouraged to create more complex tools, often as “side projects” outside of their main working hours – though frequently eschewing any hope of a healthy work-life balance. These developers and tools are given prominence on stage at most technology conferences, becoming superstars of the technology world and further adding to the hype. Very few people garner attention for solving common problems with common tools. (E.g., there are four major competing Javascript bundlers today, while make, written in 1976, can do just about everything.) To look at this cynically, extra complexity means higher cost.  Technology as a business is largely about hype, the newest and coolest tools are a selling point for most tech-reliant industries (which is most industries). The rise of outsourcing overseas, to areas where costs are much lower, continues to drive down the cost of technology development in the west. But using all of these rapidly-evolving technologies increases the cost to develop, so engineers can demand higher salaries.  It’s only by continuing to evolve the market of solutions to newer and trendier alternatives that the system can maintain its value, otherwise everyone would all settle on a low-bar minimum baseline and scaffold from there – like using dumb “feature” phones with mail and chat features, instead of smart phones. But that wouldn’t allow anyone to maintain the higher prices. These solutions are, as a result, extremely short-lived. Newer model iPhones come out annually.  And a new major Javascript framework establishes market dominance on the web every 18 to 36 months or so. Although at the time of writing we are in the height of React’s reign, we’ve already seen that market splinter and fragment, adding new layers within React, including Redux, Sagas, and others. Every piece of software released today is the legacy technology of tomorrow. This market churn makes it difficult for any new engineers to enter the workforce, as any skills learned in a developed curriculum will largely be out of date by the time the student graduates. Only full-time software engineers are able to compete, and they must dedicate a considerable part of their time to staying current, lest they fall behind and become unemployable.  Though, this will eventually give way to a secondary market of expensive engineers specializing in legacy systems, as seen during the Y2K panic. The Federal Government also continues to provide a welcome market for outdated technology skills.

Moving Past Hype Through Accountability & Simplicity

The hype cycle and automation cult isn’t going away anytime soon.  But we all – government and private sector alike – can work to be more cynical about the technologies presented to us, to dive deep on the topics, and look to experts to inform our opinions.  We can demand proven, reliable solutions with longevity instead of just the newest buzzwords. And we must plan for the inevitable economic downturns in the future that will impact the IT market, like we saw in the “dot com” crash of the late 1990s.  There will always be money to be had in IT, but the bubble must burst (again) eventually. Another thing we can do to combat this trend is to stand up for simplification; using the easiest – and most well known – solution will often be the most cost-effective.  This will inevitably erode the market, driving down costs and associated salaries, and reduce technology practitioners to skilled tradespeople like plumbers or electricians.  In this way, we can plan for a more stable future technology market, with far greater longevity. Through this dedicated effort and planning, we can focus our efforts on those areas of greater value instead, creating a more seamless Customer Experience which actually solves problems for people more efficiently and cheaply, instead of just making increasingly impressive profit margins for private companies.

Read This

The Myth of the Flat Organization


“Floor 1997–2000” by artist Do-Ho Suh

Over the last few years, an increasing number of technology companies and agencies have adopted a flat hierarchical structure. This organizational system promises to improve the efficiency of organizations by removing management structures, leaving as few management layers as possible. I believe that it is also almost entirely imaginary.

Since the popular game company Valve published its employee handbook online in 2012, companies have looked to the text for ways to improve their own processes. Growing companies stopped hiring middle level managers, project managers, and other critical roles in favor of letting engineers manage themselves. This was a very attractive move for smaller organizations, since it reduced the number of positions they needed to hire for, reducing their operating costs dramatically. For engineers it can be initially compelling to be able to set their own priorities amongst themselves — and potentially get an inflated title such as “Lead” or “VP” (but often with no additional compensation).

Management is best done by managers

However, in practice, it’s rarely so simple. By serving dual roles on a team, performance often suffers as staff try to context shift. Good engineers do not automatically make good managers — working with computers and working with people require very different skillsets. A company wouldn’t expect a business manager to write code, so why should the reverse be encouraged?

“Flattening the org chart just means creating a hierarchy of emotional labor”
Steven Reilly

In a flat organization, it is no one’s dedicated job to handle many of the complex human interactions that a business must handle. At one organization, I found myself staying up late nights, writing human resource policies, vacation plans, and codes of conduct. From project management to conflict resolution, functions that are filled by dedicated staff in other businesses frequently become after-hours extra labor by engineering staff. So-called soft skills and human-oriented problems are treated as secondary to achieving product goals.

It may be necessary for staff to serve multiple roles initially for very small organizations to be able to function, but beyond the first dozen staff this is not a practical or ethical way to accomplish tasks. If you’re large enough to be thinking about insurance plans and snack delivery, you’re large enough to hire management. A good project manager or human resources officer is a much better investment than that a lavish office space will be.

Even in large companies, it’s very common for women to be expected to perform administrative duties outside of their roles, such as taking notes or scheduling meetings. In flat organizations, it’s very common for many of the “extra” tasks to be assigned to women and minorities first. The inherent cultural biases in technology only increases the odds that these groups will be expected to do more than their share. It’s also common for more junior staff to be assigned the less exciting work, for instance being assigned to fix bugs instead of writing new code.

Hierarchies form anyway, and unfairly

“A flat org just replaces vertical hierarchy with concentric levels of inner circleness… If you remove formality you can avoid accountability and responsibility with policy of openness that is a convention of silence in practice.”
Ozzy Johnson

In many organizations that aim to be flat, a hierarchy emerges based on social cliques and personal relationships, instead of an officially established order. The ideas of staff who spend time socially with owners and executives tend to be adopted more readily. In many organizations I’ve worked for, I’ve frequently seen talented staff passed over for promotion in favor of friends of company owners and managers.

People, by nature, surround themselves with like-minded — and like-cultured — individuals, creating echo chambers and consolidating power in in-groups. This almost always puts women and minorities at a disadvantage. In a structured organization formal policies on hiring and process can help to prevent the biases and inequalities that come from such in-groups, but a flat organization has no such defense from becoming a good ol’ boys club.

It has been reported that this was the situation at Valve as well. Since there is no official hierarchy, there is often no way to call out the favoritism that comes with these factions, and rarely any formal process for resolution. Even when group-based decision making is a part of the process, individuals outside of the power centers tend to speak up less, adding to the asymmetry.

Although I was initially very excited about the prospect of flat organizations, I have yet to work in one that was effective. Modern business practices and the laws that govern them have mostly evolved to help create a more level playing field for employees. Although tech culture fetishizes rule breaking, disdain for authority, and meritocracy, these only contribute to the very toxic culture that has evolved. For the good of our industry, we could all do with a few more rules — and a few more managers.

Read This

Lessons from the Sunset of Sunlight Labs


A few months ago, I joined Sunlight Foundation as their Senior Technology Adviser. Kat Duffy, who was then the Sunlight Labs director, brought me on to help bolster their programming efforts and lend support across the organization. However, only a couple of weeks later we learned that Sunlight Labs was being shut down entirely. With the time we had until the official closure, we quickly shifted our focus to preserving the legacy of Labs’ work over the last decade.

A quick inventory of our technology assets revealed hundreds of repositories and over fifty servers in various states of decay. A lot of effort would be required to preserve the data and tools for future use. What follows are a few of the lessons from this work.

It’s important to note that Sunlight Labs was never a traditional software development shop – it was always an incubator in which interesting ideas were cultivated. Resources were frequently limited, and delivering new tools and concepts was the top priority. Many of the tools created were never designed to be released. As anyone who’s worked under these constraints can attest, best methods and rigorous practice are often put aside to focus on delivery.

Begin with an end

The most important thing I’ve learned in the last few years is to start all projects with an end in mind. I’ve written previously about the uncertain future of civic technology and nonprofit funding.  We can no longer take for granted that our projects will continue forever. Far too many civic technology projects have effectively become abandonware as a result of lack of foresight.

One effective way of preventing this is to begin your project by thinking about your end goals. Do you want to create a tool for other people to take and use? Is it good enough to simply release some data once and then move on? Is your project one that you’re willing to independently run indefinitely? How long can you expect to be able to work on it? What happens to the project if something happens to you or your organization?

Writing down all of your contingency plans in a public way can help to mitigate these problems when they inevitably come up later. Make sure that you would feel comfortable if you had to walk away from your work tomorrow. Setting a deadline for the end of your work, and communicating this information with stakeholders, can be useful as well.


We all are aware of the critical importance of documenting our projects, yet few projects have good documentation. It can be very time-consuming to write down everything that a new person needs to know about your project, but this is absolutely critical work. I recommend doing as much of this work as possible up-front, and making it an integral part of your process moving forward. This is still useful even if your work isn’t writing code for computers – always take the time to write for humans.

In the same way that you have your peers review your work or that you write your tests, make sure to include writing documentation as a step in the process. Make a new rule for your team: if a feature doesn’t have documentation, it doesn’t get merged and it doesn’t ship.

Assume nothing. Although today you may have the luxury of showing every new team member how every system works as part of their onboarding process, that may change suddenly and without warning. Write as if your end user probably knows nothing about your internal process. Make sure you explain every step of using your tools – from getting the data to running and deploying the software. Give examples as frequently as possible, with screenshots and exact commands wherever it makes sense.


As a general rule, if your code and/or data doesn’t have a license, no one else can use it. There are huge exceptions to this, but it is almost always easier to be explicit than to hire a legal team.

Since a large number of Sunlight’s projects did not have licenses, I created a tool to add licenses in bulk to all projects that were missing them.  We used the GPLv3 license for most of these, as it is a strong license that requires sharing of modifications to code, helping to preserve these tools for future use.  There are lots of other good licenses to consider, however.

Keep a clean house

One of the more laborious tasks was preparing the many projects to be shared publicly on GitHub. Since a large number of these project had only been stored on an internal GitLab server with no public access, many of them had secure secrets – passwords and API credentials – hardcoded into the source. Frequently, these credentials were stored outside of the usual settings files, and since the credentials were different on every project, tools like BFG were not enough to find all of them. Again, I ended up having to write a collection of tools to migrate repositories from GitLab to GitHub, to check for potentially problematic files, and to automatically remove credentials from those files. Even after that, it still required many hours of manually reviewing code to check for stray credentials.

For most projects, it’s a better idea to never check in any secure credentials to your repositories in the first place. Storing these in the environment instead is a common way of avoiding this situation. If that’s too complicated for any reason, you can always simply use standard configuration files and have Git (or whatever version control tool you prefer) ignore them.

Do remember to keep an example of your configuration file in the repository and up-to-date, though. In one case we were unable to deploy updated code to our server due to an incomplete Fabric configuration, and had to update the code manually on the live server as a result.

Pick the simplest solution

One of the biggest challenges was the large number of simple “brochure” sites that were custom applications written from scratch, instead of using a common off-the-shelf solution. These were difficult for staff to maintain and keep running – often taking three different people to update a single page. In one case, we lost all of the content of a site due to it being built on Heroku using a database service that ceased to exist – we had to scrape the Internet Archive to get the content back. The main Sunlight Foundation website itself was one of the most complicated applications that Labs had created, pulling data from seven different databases – each one a completely different type (including MySQL, PostgreSQL, Mongo, Redis, Memcache, and others) – to deliver the content.

Since Sunlight would no longer have any dedicated technical team, keeping these running wouldn’t be viable. After discussing with the rest of the staff, I picked WordPress as a simple platform for people to be able to manage the content moving forward. We picked WPEngine for managed hosting, so staff could rely on them to handle any technical issues that might arise.

For future projects, I would recommend that developers avoid reinventing the wheel – especially if all that is required is a simple content management system. Working with your staff to identify the needs of a project and finding the simplest solution helps to ensure that things will work long after the developer of that project is gone or is too busy to help. And in most cases, one database is plenty.

… But plan for the worst

A constant pain point was that the architecture of Labs’ software used individual, huge servers that hosted critical pieces of infrastructure shared by all applications, with almost no redundancy. This meant that when the only DNS server or database server went down, most of our applications all failed immediately. This happened several times over the few months while we were shutting down Labs.  This also meant that servers were consistently under-utilized while being incredibly expensive – the monthly AWS bill was over $6,000 per month!

A little bit of extra redundancy can go a long way. Moreover, designing your architecture with small “atomic” pieces can reduce shared dependencies and provide greater stability. This is usually cheaper than running huge servers as well. Running one application per server is a good rule of thumb for smaller tools, and you can always split out services (databases, task runners, etc) as your needs grow.

As a quick aside – over the years, the number of servers I’ve seen fail due to logs filling a hard drive is astonishing. Spend the time to learn how to use tools like logrotate and set up your servers so that something simple doesn’t take down your application.

The final steps

When the time came to shut things down and transfer content, we had the help of many partners. Most notably, the Internet Archive was incredibly helpful in scraping all of our content from a list of sites we provided. Their fantastic 301Works project took our our in-house url shortener domain name and a spreadsheet of our links, to preserve those forever so we were able to shut it down entirely (see above about not reinventing the wheel).

One thing that made the transfer process much, much easier was services such as Heroku and Namecheap which allow you to easily transfer a property to a new owner. For both of these, you simply need to provide a username and the transferee needs only to accept – it’s a very straightforward process. GitHub proved to be more complicated, as we first had to add the new owner as an admin of the repositories and then have them manually move the repositories to the new Organization owner – this took a while with so many repositories moving over.

Amazon was the most difficult piece, as we had to create AMIs of our EC2 instances (a very slow process with the aforementioned huge servers) and then grant access (by account number) to the new owners. I was never able to grant access to an S3 bucket to a separate account – whether the documentation is out of date or there was a system bug, I’ll never know –   for most of these I just made the content public wherever possible and sent out links.

Wrapping up

We should always hope that our projects don’t suddenly come to an end without warning.  But we also must plan for the future as it our work may stop tomorrow. Taking steps up early in a project can help to make sure your work lasts beyond the length of your involvement, and that it can be used by as many people as possible. Make sure you invest the time now, so that you won’t need to later.

Read This