#talks

Almost everyone thinks they’re acting rationally. No matter how illogical (or uneven unhinged) an action may appear to outsiders, there’s almost always an internal logic that is at least understandable to the person making that decision, whether it’s an individual or an organization.

And it’s especially apparent in organizations. How many times has a company you liked or respected at one time made a blunder so mystifying that even you, as a fan, have no idea what could possibly have caused the chain of events that led to it? Yet if you were to ask the decision-makers, the reasoning is so clear they’re baffled as to why everyone is not in total lockstep with them.

There are any number of reasons why something that’s apparent to an outsider might be opaque to an insider, and I won’t even try to go over all of them. Instead, I want to focus on a specific categorical error: the misuse of data to drive decisions and outcomes.

A lot of companies say they are data-driven. Who wouldn’t want to be? The implication is that the careful, judicious analysis of data will yield only perfectly logical outcomes as to a company’s next steps or long-term plan. And it’s true that the use of data to inform your judgment can lead to better outcomes. But it can also lead to bad outcomes, for any number of reasons that we’ll discuss below.

But first, definitions.

Data: Individual, separate facts. These tend to be qualitative – if quantitative, they tend to be reduced to qualitative data for analysis.

Story: Connective framework for linking and explaining data.

Narrative: A well-reasoned story that tries to account for as much of the data and context as possible. It is entirely possible (and, in most cases, probable) that multiple narratives can be drawn from the same set of data. Narratives should have a minimum of assumptions, and all assumptions and caveats should be explicitly stated.

Fairytale: A story that is unsupported by the data, connecting data that does not relate to one another or using false data.

I have worked in a number of different industries, all of which pull different kinds of data and analytics to inform different aspects of their business. I cannot thing of a single one that avoided writing fairytales, though some were better systemically than others. What I’m going to do in this blog is go over a number of the different pitfalls you can fall into when writing stories that lead you astray from narrative to fairytale, and how you can overcome them.

I’ll try to use at least one real-world example for each so you can hopefully see how these same types of errors might crop up in your own owrk.

Why fairytales get written

1. Inventing or inferring explanations for specific data

I used to work in daily newspapers back when that was still thought to be a viable enterprise on the internet. The No. 1 problem (as I’m sure you’ve seen looking at any news site) is the chasing of a trend. A story would come across our analytics dashboard that appeared to be “doing numbers,” so immediately the original writer (and, often, a cabal of editors) would convene to try to figure out why that particular story had gone viral.

Oftentimes the real reason was something as ultimately uncontrollable as “we happened to get in the Google News carousel for that story” or “we got linked from Reddit” – phenomena that were not under our control. But because our mandate was to get big numbers, we would try to tease out the smallest things. More stories on the same topic, maybe ape the style (single-sentence paragraphs), try to time stories to go out at the same time every day …

It’s very similar to a cargo cult – remote villages who received supply drops during WWII came to believe that such goods were from a cargo “god,” and by following the teachings of a cargo “leader” (which typically involved re-enacting the steps that led up to the first drops, or mimicking European styles and activities) the cargo would return in abundance. When, in reality, the actions of the native peoples had little to no effect on whether more cargo would come.

This commonly happens when you’re asked to explain the reason for a trend or an outcome, a “why” about user behavior. It is nearly impossible to know why a user does something absent them explicitly telling you either through asynchronous feedback or user interviews. Everything else is conjecture.

But we’re often called upon (as noted above) to make decisions based on these unknowable reasons. What to do?

The correct way to handle these types of questions is:

• Be clear that an explanation is a guess.

• Treat that guess as a hypothesis.

• Test that hypothesis.

• Allow for the possibility that it’s wrong or that there is no right answer, period.

2. Load-bearing single data point

I see this all the time in engineering, especially around productivity metrics. There is an eternal debate as to whether you can accurately measure the productivity of a development team; my response to this is, “kinda.” You can measure any number of metrics that you want in order, but those metrics only measure what they measure. Most development teams use story points in order to gauge roughly how long a given chunk of development will take. Companies like to measure expected vs. actual story points, and then make actions based on those numbers.

Except that the spectrum of actions one can take based on those numbers is unknowably vast, and those numbers in and of themselves don’t mean anything. I worked on a development team where the CTO was reporting velocity up the chain to his superiors as a measure of customer value that was being provided. That CTO also refused to give story point assignments to bug tickets, since that wasn’t “delivering customer value.” I don’t know what definition of customer value you use in your personal life, but to me “having software that works properly” is delivering value.

But because bugs weren’t pointed, they were given lower priority (because we had to meet our velocity numbers). This increased focus on velocity numbers meant that tickets were getting pushed through to production without having gone through thorough testing, because the important thing was to deliver “customer value.” This, as you can imagine, led to more bug tickets that weren’t prioritized, rinse and repeat, until the CTO was let go and the whole initiative was dramatically restructured because our customers, shockingly enough, didn’t feel they were getting enough value in a broken product.

I want to introduce you to two of my favorite “laws” that I use frequently. The first, from psychology, is called Campbell’s Law, after the man who coined it, Donald Campbell. It states:

• The more emphasis you put on a data point for decision-making, the more likely it will wind up being gamed.

We saw this happen in a number of different ways. When story points got so important, suddenly story point estimates started going way up. Though we had a definition of done that including things like code review and QA testing, those things weren’t tracked or considered analytically, so they were de-emphasized when it was perceived that including them would hurt the number. Originally, the velocity stood for “number of story points in stories that were fully coded, tested and QA’ed.” By the end, they stood for “the maximum number of points we could reasonably assign to the stories that we rushed through at the end of the week to make velocity go up.”

The logical conclusion of Campbell’s Law is Goodhart’s Law, named after economist Charles Goodhart:

• When a measure becomes a target, it ceases to be a good measure.

Now, I am not saying you should ignore SPACE or DORA metrics. They can provide some insight into how your development / devops team is functioning. But you should use any of them, collectively or individually, as targets that you need or should meet. They are quantitative data that should be used in conjunction with other, qualitative, data garnered from talking and listening to your team. If someone’s velocity is down over a number of weeks, don’t go to them demanding it come up. Instead, talk to them and find out what’s going on. Have they noticed? Are they doing something differently?

My personal story point numbers tend to be all over the place, because some weeks my IC time is spent powering through my own stories, but then for months at a time I will devote the majority of my time to unblocking others or serving as the coordinator / point person for my team so they can spend their time head-down in the code. If you measured me solely by story points, I would undoubtedly be lacking. But the story points don’t capture all the value I bring to a team.

3. Using data because it’s available

This is probably the number one problem I see in corporate environments. We want to know the answer to x question, we have y data, so we’re going to use y to answer x even if the two are only tangentially (or, sometimes not even that closely) related.

I co-managed the web presence for a large research institution’s college of medicine. On the education side, our number one goal was to increase the quality and number of qualified applicants for our various programs. Except, on the web, it’s kind of hard to draw a direct line between “quality of website” and “quality of applicants.” Sure, if we got lucky someone would actually go through our website to the student application form, and we could see that in the analytics. But much like any major life decision, people made the decision to apply or not after weeks or months of deliberation, visiting the site sporadically. This, in addition to any number of other factors in their life that might affect their choice.

But you have to have KPIs, else how would you know that your workers aren’t slacking? So the powers that be decided the most salient data point was “number of visitors from the surrounding geographic area,” as measured by the geographic identification in Google Analytics (back when GA was at least pretending to provide useful data).

Now, some useful demographic information for you, the listener, to know is that in the year that mandate started being enforced, 53% of the incoming MD class was in-state. So, at best, our primary metric affected very slightly over half of our applicants to our flagship program. That’s to say nothing of the fact that people looking on the website might also just be members of the general public (since the college of medicine was colocated with a major hospital). It’s also not even true, if we were somehow able to discern who of the visitors were high-value applicants, that the website had anything to do with them applying or not to the program! That’s just not something you accurately track through analytics.

This is not an uncommon phenomenon. Because they had a given set of quantitative data to work with, that was the data they used to answer all the questions that were vital to the business.

I get it! It’s hard to say “no” or “you can’t” or “that’s impossible” to your boss when you’re asked to give information or justification. But that is the answer sometimes. The way to get around it is to 1) identify the data you’d actually need to answer the question, and 2) devise a method for capturing that data.

I also want to point out that it is vital to collect data with intent. Not intent as in “bias your data to the outcome you want,” but in the sense that you need to know what questions you’re going to ask of the data in order to be assured you’re collecting the right data. Going back after the fact to interrogate the data with different answers veers dangerously close to p-hacking, where you keep twisting and filtering data until you get some answer to some question, even if it’s not even close to the question you started with.

4. Discounting other possible explanations

I once sat in on a meeting where they were trying to impart to us the importance of caution. They told us about the story of Icarus; in Ancient Greece, the great inventor Daedalus was imprisoned in the Labyrinth he had built for the minotaur. Desperate to escape, he fashioned a set of wings from candle wax and feathers for him and his son, Icarus. Before leaving, he warned Icarus not to fly too close to the sea (for fear the spray would weigh down the wings and cause them to crash) nor too close to the sun, for the heat would melt the wax and cause them to crash. The pair successfully escaped the Labyrinth and the island, but Icarus, caught up in the exhilaration of flight, soared ever higher … until his wings melted and he came crashing down to the sea and drowned.

We were asked to reflect on the moral of the story. “The importance of swimming lessons!” I cracked, “Or, more generally, the importance of always having a backup plan.” Because, of course, Daedalus was worried that his son would fly too high or too low; rather than prepare for that possibility by teaching him how to swim (or fashioning a boat), Daedaelus did the bare minimum and caught the consequences.

Both my explanation and the traditional, “don’t fly too close to the sun” are valid takeaways; this is what I mean when I say that multiple valid narratives can arise from the same set of facts. Were we presenting a report to Daedalus, Inc., on the viability of his new AirWings, I would argue the most useful thing to do would be to present both. Both provide plausible outcomes and actionable information that can be taken away to inform the next stages of the product.

On a more realistic note, I was once asked to do an after-action analysis of a network incursion. In my analysis, I pointed out which IP ranges were generally agreed to be from the same South American country (where there was no legitimate business activity for the targeted company); those access logs seemed to match up with suspicious activity in Florida as well as another South Asian country.

I did not tie those things together. I did not state that they were definitively working together, or even knew of one another. I laid out possibilities including a coordinated attack by the Florida and South American entities (based on timestamps and accounts used); I also posited it was possible the attack originated in South Asia and they passed the compromised credentials to their counterparts (or even sold them to another group) in South America/Florida. It’s also possible that they were all independent actors either getting lucky or acting on the same tip.

The important thing was to not assume facts I did not (and could not) know, and make it very clear when I was extrapolating or assuming facts I did not have. One crucial difference between fairytale and narrative is the acknowledgment of doubt. Do not assert things you cannot know, and point out any caveats or assumptions you made in the formulation of your story. This will not only protect your reputation should any of those facts be wrong, but it makes it easier for others to both conceive of other, additional narratives you might not have, and leaves room / signposts as to what data might be collected in order to verify underlying assumptions.

Summary

It can be easy to get sucked into writing a fairytale when you started out writing a narrative. Data can be hard, deadlines can be short and pressure can be immense. Do you what you can to make sure you’re collecting good data with intent, asking and answering questions that are actually relevant to that data, and not discounting other explanations just because you finished yours. Through the application of proper data analysis, we can get better at providing good products to our customers and treating employees with respect and compassion while still maintaining productivity. It just requires diligence and a willingness to explore beyond superficial numbers to ensure the data you’re analyzing is accurately reflecting reality.

Software requirements are rather straightforward - if we look at the requirements document, we see simple, declarative statements like "Users can log out," or "Users can browse and create topics." And that's when we're lucky enough to get an actual requirements document.

This is not legal advice

None of the following is intended to be legal advice. I am not a lawyer, have not even read all that many John Grisham novels, and am providing this as background for you to use. If you have actual questions, please take them to an actual lawyer. (Or you can try calling John Grisham, but I doubt he'd pick up.)

But there are other requirements in software engineering that aren't as cut-and-dried. Non-functional requirements related to things like maintainability, security, scalability and, most importantly for our purposes, legality.

For the sake of convenience, we're going to use "regulations" and other derivations of the word to mean "all those things that carry the weight of law," be they laws, rules, directives, court orders or what have you.

Hey, why should I care? Isn't this why we have lawyers?

Hopefully your organization has excellent legal representation. Also hopefully, those lawyers are not spending their days watching you code. That's not going to be fun for them or you. You should absolutely use lawyers as a resource when you have questions or aren't sure if something would be covered under a specific law. But you have to know when to ask those questions, and possess enough knowledge when your application could be running afoul of some rule or another.

It's also worthwhile to your career to know these things! Lots of developers don't, and your ability to point them out and know about them will make you seem more knowledgeable (because you are!). It will also make you seem more competent and capable than another developer who does not – again, because you are! This stuff is a skillset just like knowing Django.

While lawyers may be domain experts, they aren't always (especially at smaller organizations) and there are lots of regulations that specifically cover technology/internet-capable software that domain experts likely would not (and should not) be expected to be on top of. Further, if you are armed with foreknowledge, you don't have to wait for for legal review after the work has been completed.

Also, you know, users are people, too. Most regulations wind up being bottom-of-the-barrel expectations that user data is safeguarded and restricting organizations from tricking users into doing things they wouldn't have otherwise. In the same way I would hope my data and self-determination are respected, I also want to do the same for my users.

Regulatory environments

The difference in the regulatory culture between the US and the European Union is vast. I truly cannot stress how different they are, and that's an important thing to know about because it can be easy to become fluent in one and assume the other is largely the same. It's not. Trust me.

United States

The US tends, for the most part, to be a reactionary regulator. Something bad happens, laws or rules (eventually) get written to stop that thing from happening again.

Also, the interpretations of those rules tend to fluctuate more than in the EU, depending on things seemingly as random as which political party is in power (and controlling the executive branch, specifically) or what jurisdiction a lawsuit is filed in. We will not go in-depth into those topics, for they are thorny and leave scars, but it's important to note. The US also tends to give wide latitude to the defense of, "but it's our business model!" The government will not give a full pass on everything, but they tend to phrase things in terms of "making fixes" rather than "don't do that."

Because US regulations tend to be written in response to a specific incident or set of incidents, they tend for the most part to be very narrowly tailored or very broad ("e.g., TikTok is bad, let's give the government the ability to jail you for 20 years for using a VPN!"), leaving little guidance to those of us in the middle. This leaves lots of room for unintended consequences or simply failing to achieve the stated goals. In 2003, Congress passed the CAN-SPAM Act to "protect consumers and businesses from unwanted email." As anyone who ever looks at their spam box can attest, CAN-SPAM's acronym unfortunately seems to have meant "can" as in "grant permission," not "can" as in "get rid of."

European Union

In contrast, the EU tends to issue legislation prescriptively; that is, they identify a general area of concern, and then issue rules about both what you can and cannot do, typically founded in some fundamental right.

This technically is what the US does on a more circumspect level, but the difference is the right is the foundational aspect in the EU, meaning it's much more difficult to slip through a loophole.

From a very general perspective, this leads to EU regulations being more restrictive in what you can and can't do, and the EU is far more willing to punish punitively those companies who run afoul of the law.

Global regulations

There are few regulations that apply globally, and usually they come about backwards - in that a standard is created, and then adopted throughout the world.

Accessibility

In both the US and the EU, the general standard for digital accessibility is WCAG 2.1, level AA. If your website or app does not meet (most) of that standard, and you are sued, you will be found to be out of compliance.

In the US, the reason you need to be compliant comes from a variety of places. The federal government (and state governments) need to be compliant because of the Rehabilitation Act of 1974, section 508. Entities that receive federal money (including SNAP and NSF grants) need to be compliant because of the RA of 1974, section 504. All other publicly accessible organizations (companies, etc.) need to have their websites compliant because of the Americans with Disabilities Act and various updates. And all of the above has only arisen through dozens of court cases as they wound their way through the system, often reversing each other or finding different outcomes with essentially the same facts. And even then, penalties for violating the act are quite rare, with the typical cost being a) the cost of litigation, and b) the cost of remediation and compliance (neither of which are small, but they're also not punitive, either).

In the EU, they issued the Web Accessibility Directive that said access to digital information is a right that all persons, including those with disabilities, should have, so everything has to be accessible.

See the difference?

WCAG provides that content should be

• Perceivable - Your content should be able to be consumed in more than one of the senses. The most common example of this is audio descriptions on videos (because those who can't see the video still should be able to glean the relevant information from it).

• Operable - Your content should usable in more than one modality. This most often takes the form of keyboard navigability, as those with issues of fine motor control cannot always handle a mouse dextrously.

• Understandable - Your content should be comprehensible and predictable. I usually give a design example here, which is that the accessibility standard actually states that your links need to be perceivable, visually, as links. Also, the "visited" state is not just a relic of CSS, it's actually an accessibility issue for people with neurological processing differences who want to be able to tell at a glance what links they've already been to.

Robust - Very broadly, this tenet states you should maximize your compliance with accessibility and other web standards, so that current and future technologies can take full advantage of them without requiring modification to existing content.

Anyway, for accessibility, there's a long list of standards you should be meeting. The (subjectively) more important ones most frequently not followed are:

1. Provide text alternatives for all non-text content: This means alt text for images, audio descriptions for video and explainer text for data/tables/etc. Please also pay attention to the quality – the purpose of the text is to provide a replacement for when the non-text content can't be viewed, so "picture of a hat" is probably not an actual alternative.

2. Keyboard control/navigation: Your site should be navigable with a keyboard, and all interactions (think slideshows, videos) should be controllable by a keyboard.

3. Color contrast: Header text should have a contrast ratio of 3:1 between the foreground and background; smaller text should have a ratio of 4.5:1.

4. Don't rely on color for differentiation: You cannot rely solely on color to differentiate between objects or types of objects. (Think section colors for a newspaper website: You can't just have all your sports links be red, it has to be indicated some other way.)

5. Resizability: Text should be able to be resized up to 200% larger without loss of content or functionality

6. Images of text: Don't use 'em.

7. Give the user control: You can autoplay videos or audio if you must, but you also have to give the user the ability to stop or pause it.

There are many more, but these are the low-hanging fruit that lots of applications still can't manage to pick off

PCI DSS

The Payment Card Industry Data Security Standard is a set of standards that govern how you should store credit card data, regulated by credit card companies themselves. Though some individual US states require adherence to the standards (and fine violators appropriately), federal and EU law does not require you to follow these standards (at least, not specifically these standards). However, the credit card companies themselves can step in and issue fines or, more critically, cut off access to their payment networks if they find the breaches egregious enough.

In most cases, organizations offload their payment processing to a third party (e.g., Stripe, Paypal), who is responsible for maintaining compliance with the specification. However, you as the merchant or vendor need to make sure you’re storing the data from those transactions in the manner provided by the payment processor; it’s not uncommon to find places that are storing too much data on their own infrastructure that technically falls under the scope of PCI DSS.

Some of the standards are pretty basic - don’t use default vendor passwords on hardware and software, encrypt your data transmissions. Some are more involved, like restricting physical access to cardholder data, or monitoring and logging access to network resources and data.

EU regulations

GDPR

The EU's General Data Privacy Regulation caused a big stir when it was first released, and for good reason. It completely changed the way that companies could process and store user data, and severely restricted what sort of shenanigans companies can get up to.

The GDPR states that individuals have the right to not have their information shared; that individuals should not have to hand over their information in order to access goods or services; and that individuals have further rights to their information even once it's been handed over to another organization.

For those of us on the side of building things, it means a few things are now requirements that used to be more "nice-to-haves."

• You must get explicit consent to collect data If you're collecting data on people, you have to explicitly ask for it. You have to specify exactly what information you're collecting, the reason you're collecting it, how long you plan on storing it and what you plan to do with it (this is the reason for the proliferation of all those cookie banners a few years ago). Furthermore, you must give your users the right to say no. You can't just pop up a full-screen non-dismissable modal that doesn't allow them to continue without accepting it.

• You can only collect data for legitimate purposes Just because someone's willing to give you data doesn't mean you're allowed to take it. One of my biggest headaches I got around GDPR was when a client wanted to gate some white papers behind an email signup. I patiently explained multiple times that you can't require an email address for a good or service unless the email address was required to provide said good or service. No matter how many times the client insisted that he had seen someone else doing the same thing, I stood firm and refused to build the illegal interaction.

• Users have the right to ask for the data you have stored, and to have it deleted Users can ask to see what data you have stored on them, and you're required to provide it (including, again, why you have that data stored). And, unless it's being used for legitimate processing purposes, you have to delete that data if the user requests it (the "right to be forgotten").

And all of this applies to any organization or company that provides a good or service to any person in the EU. Not just paid, either – it explicitly says that you do not have to charge money to be covered under the GDPR. So if your org has an app in the App Store that can be downloaded in Ireland, Italy, France or any other EU country, it and likely a lot more of your company's services will fall under GDPR.

As for enforcement, organizations can be fined up to €20 million, or up to 4% of the annual worldwide turnover of the preceding financial year, whichever is greater. Amazon Europe got docked €746 million for what was alleged "[manipulation of] customers for commercial means by choosing what advertising and information they receive[d]" based on the processing of personal data. Meta was fined a quarter of a billion dollars a few different times.

But it's not just the big companies. A translation firm got hit with fines of €20K for "excessive video surveillance of employees" (a fine that's practically unthinkable in the US absent cameras in a private area such as the bathroom), and a retailer in Belgium had to pay €10K for forcing users to submit an ID card to create a loyalty account (since that information was not necessary to creating a loyalty account).

Digital Markets Act

The next wave of regulation to hit the tech world was the Digital Markets Act. which is aimed specifically at large corporations that serve a “gatekeeping functionality” in digital markets in at least three EU countries. Although it is not broadly applicable, it will change the way that several major platforms will work with their data.

The directive’s goal is to break up the oversized share that some platforms have in digital sectors like search, e-commerce, travel, media streaming, and more. When a platform controls sufficient traffic in a sector, and facilitates sales between businesses and users, it must comply with new regulations about how data is provisioned and protected.

Specifically, those companies must:

• Allow third parties to interoperate with their services

• Allow businesses to access the data generated on the platform

• Provide advertising partners with the tools and data necessary to independently verify claims

• Allow business users to promote and conduct business outside of the platform

Additionally, the gatekeepers cannot:

• Promote internal services and products over third parties

• Prevent consumers from linking up with businesses off their platforms

• Prevent users from uninstalling preinstalled software

• Track end users for the purpose of targeted advertising without users’ consent

If it seems like these are aimed at the Apple App Store and Google Play Store, well, congrats, you cracked the code. The DMA aims to help businesses have a fairer environment in which to operate (and not be completely beholden to the gatekeepers), and allow for smaller companies to innovate without being hampered or outright squashed by established interests.

US regulations

The US regulatory environment is a patchwork of laws and regulations written in response to various incidents, and with little forethought for the regulatory environment as a whole. It’s what allows you as a developer to say, “Well, that depends …” in response to almost any question, to buy yourself time to research the details.

HIPAA

Likely the most well-known US privacy regulation, HIPAA covers almost none of the things that most people commonly think it does. We'll start with the name: Most think it's HIPPA, for Health Information Privacy Protection Act. It actually stands for Healthcare Insurance Portability and Accountability Act, because most of the law has nothing to do with privacy.

It is very much worth noting that HIPAA only applies to health plans, health care clearinghouses, and those health care providers that transmit health information electronically in connection with certain administrative or financial transactions where health plan claims are submitted electronically. It also applies to contractors and subcontractors of the above.

That means most of the time when people publicly refuse to comment on someone's health status because of HIPAA (like, in a sports context or something), it's nonsense. They're not required to disclose it, but it's almost certainly not HIPAA that's preventing them from doing so.

What is relevant to us as developers is the HIPAA Privacy Rule. The HIPAA privacy rule claims to "give patients more control over their health information, set boundaries on the use of their health records, establish appropriate safeguards for the privacy of their information."

What it does in practice is require that you have to sign a HIPAA disclosure form for absolutely every medical interaction you have (and note, unlike GDPR, that they do not have to let you say "no"). Organizations are required to keep detailed compliance policies around how your information is stored and accessed. While the latter is undoubtedly a good thing, it does not rise to the level of reverence indicated by its stated goals.

What you as a developer need to know about HIPAA is you need to have very specific policies (think SOC II [official link] [more useful link]) around data access, operate using the principle of least privileged access (only allow those who need to see PHI to be able to access it), and specific security policies related to the physical facility where the data is stored.

HIPAA’s bottom line is that you must keep safe Protected Health Information (PHI), which covers both basic forms of personally identifiable information (PII) such as name, email, address, etc., as well as any health conditions those people might have. This seems like a no-brainer, but it can get tricky when you get to things like disease- or medicine-specific marketing (if you’re sending an email to someone’s personal email address on a non-HIPAA-compliant server about a prostate cancer drug, are you disclosing their illness? Ask your lawyer!).

There are also pretty stringent requirements related to breach notifications (largely true of a lot of the compliance audits as well). These are not things you want to sweep under the rug. It’s true that HIPAA does not see many enforcement acts around the privacy aspects as some of the other, jazzier regulations. But health organizations also tend to err on the side of caution and use HIPAA-certified hosting and tech stacks, as any medical provider will be sure to complain about to you if you ask them how they enjoy their Electronic Medical Records system.

Section 230 of the Communications Decency Act

Also known as the legal underpinnings of the modern internet, Section 230 provides that "No provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider."

In practice, this means that platforms that publish user-generated content (UGC) will not be treated as the "publisher," in the legal sense, of that content for the purposes of liability for libel, etc. This does not mean they are immune from copyright or other criminal liabilities but does provide a large measure of leeway in offering UGC to the masses.

It's also important to note the title of the section, "Protection for private blocking and screening of offensive material." That's because Section 230 explicitly allows for moderation of private services without exposing the provider to any liability for failing to do so in some instances. Consider a social media site that bans Nazi content; if that site lets a few bad posts go through, it does not mean they are on the hook for those posts, at least legally speaking. Probably a good idea to fix the errors lest they be found guilty in the court of public opinion, though.

GLBA

The Graham-Leach-Biley Act is a sort of privacy protection policy for financial institutions. It doesn’t lay out anything particular novel or onerous - financial institutions need to provide a written privacy policy (what data is collected, how it’s used, how to opt-out), and provides some guidelines companies need to meet about safeguarding sensitive customer information. The most interesting, to me, requirement is Pretext Protection, which actually enshrines in law that companies need to have policies in place for how to prevent and mitigate social engineering attacks, both of the phishing variety as well as good old-fashioned impersonation.

COPPA

The Children's Online Privacy Protection Rule (COPPA, and yes, it’s infuriating that the acronym doesn’t match the name) is one of the few regulations with teeth, largely because it is hyperfocused on children, an area of lawmaking where overreaction is somewhat common.

COPPA provides for a number of (now) common-sense rules governing digital interactions that companies can have with children under 13 years old. Information can only be collected with:

• Explicit parental consent.

• Separate privacy policies must be drafted and posted for data about those under 13.

• A reasonable means for parents to review their children's data.

• Establish and maintain procedures for protecting that data, including around sharing that data.

• Limits on retention of that data.

• Prohibiting companies from asking for more data than is necessary to provide the service in question.

Sound weirdly familiar, like GDPR? Sure does. Wondering why only children in the US are afforded such protections? Us too!

FERPA

The Family Educational Rights Protection Act is sort of like HIPAA, but for education. Basically, it states that the parents of a child have a right to the information collected about their child by the school, and to have a say in the release of said information (within reason; they can't squash a subpoena or anything). When the child reaches 18, those rights transfer to the student. Most of FERPA comes down to the same policy generation around retention and access discussed in the section on HIPAA, though the disclosure bit is far more protective (again, because it's dealing with children).

FTC Act

The Federal Trade Commission Act of 1914 is actually the law that created the Federal Trade Commission, and the source of its power. You can think of the FTC as a quasi-consumer protection agency, because it can (and, depending on the political party in the presidency, will) go after companies for what aren't even really violations of law so much as they are deemed "unfair." The FTC Act empowers the commission to prevent unfair competition, as well as protect consumers from unfair/deceptive ads (though in practice, this has been watered down considerably by the courts).

Nevertheless, of late the FTC has been on a roll, specifically targeting digital practices. An excellent recent example was the settlement by Epic Games, makers of Fortnite. The FTC sued over a number of allegations, including violations of COPPA, but it also explicitly called out the company for using dark patterns to trick players into making purchases. The company’s practice of saving any credit cards used (and then making that card available to the kids playing), confusing purchasing prompts and misleading offers were specifically mentioned in the complaint.

CAN-SPAM

Quite possibly the most useless technology law on the books, CAN-SPAM (Controlling the Assault of Non-Solicited Pornography And Marketing Act) clearly put more time into the acronym than the legislation. The important takeaways are that emails need:

• Accurate subjects

• To disclose themselves as an ad

• Unsubscribe links

• A physical address for the company

And as your spam box will tell you, it solved the problem forever. This does not, however, mean you can ignore its strictures! As a consultant at a company that presumably wishes to stay on the right side of the law, you should still follow its instructions.

CCPA and Its Ilk

The California Consumer Privacy Act covers, as its name suggests, California residents in their dealings with technology companies. Loosely based on the GDPR, CCPA requires that businesses disclose what information they have about you and what they do with it. It covers items such as name, social security number, email address, records of products purchased, internet browsing history, geolocation data, fingerprints, and inferences from other personal information that could create a profile about your preferences and characteristics.

It is not as wide-reaching or thorough as GDPR, but it’s better than the (nonexistent) national privacy law.

The CCPA applies to companies with gross revenues totaling more than $25 million, businesses with information about more than 50K California residents, or businesses who derive at least 50% of their annual revenue from selling California residents’ data. There are similar measures that have already been made law in Connecticut, Virginia, Colorado, and Utah, as well as other states also considering relevant bills.

Other state regulations

The joy of the United States’ federalist system is that state laws can be different (and sometimes more stringent!) than federal law, as we see with CCPA. It would behoove you to do a little digging into the state regulations when you’re working with specific areas — e.g., background checks, where the laws differ from state to state, as even though you’re not based there, you may be subject to its jurisdiction.

There are two different approaches companies can take to dealing with state regulations: Either treat everyone under the strictest regulatory approach (e.g., treat every user like they’re from California) or make specific carve-outs based on the state of residence claimed by the user.

It is not uncommon, for example, to have three or four different disclosures or agreements for background checks ready to show a user based on what state they reside in. The specific approach you choose will vary greatly depending on the type of business, the information being collected, and the relevant state laws.

How to implement

Data compliance is critical, and the punitive aspects of GDPR’s enforcement means your team must have a solid strategy for compliance.

The most important aspect of dealing with any regulatory issue is first knowing what’s required for your business. Yes, you’re collecting emails, but to what end? If that data is necessary for your business to function, then you have your base-level requirements.

Matching those up against the relevant regulations will provide you with a starting point from which you can begin to develop the processes, procedures and applications that will allow your business to thrive. Don’t rely on “that’s how we’ve always done it” or “we’ve seen other people do x” as a business strategy.

The regulatory environment is constantly shifting, and it’s important to both keep abreast of changes as well as always knowing what data and services are integral to your business’s success. Keeping up with the prevalent standards will aid you not only in not getting sued, but also ensuring your companies that you’re a trustworthy and reliable partner.

How to keep up

It all seems a little daunting, no?

But you eat the proverbial regulatory elephant the same way you do any other large food item: one bite at a time. In the same way you didn’t become an overnight expert in securing your web applications against cross-site scripting attacks or properly manage your memory overhead, becoming a developer who’s well-versed in regulatory environments is a gradual process.

Now that you know about some of the rules that may apply to you, you know what to keep an eye out for. You know potential areas to research when new projects are pitched or started, and you know where to ask questions. You know to both talk to and listen to your company’s legal team when they start droning on about legalistic terms

People always seem confused by the title of this: "What does scrum have to do with measuring productivity?" they ask. And I smile contentedly, because that's the whole point.

Scrum is supposed to be a system for managing product work, iterating and delivering value to the customer. What usually winds up happening is scrum gets used for the management of software development work as a whole, from decisions about promotion to hiring and firing to everything else. That's not what scrum is designed to do, and it shows.

Now, I love to talk about process improvement, completely agnostic of whatever process framework you're using. I would much rather have a discussion about the work you're doing and what blockers you're hitting rather than discussing abstract concepts.

However, if you keep running into the same issues and blockers over and over again, it's usually worth examining your workflows to find out if you're actually applying the theory behind your framework to the actual work you're doing. The concept of Agile specifically is not about the processes involved, but you need to know and understand the rules before you should feel comfortable breaking them.

Processes

I want to start with a quick overview of a few key terms to make sure everyone's on the same page.

• **Waterfall
**

In waterfall development, every item of work is scheduled out in advance. This is fantastic for management, because they can look at the schedule to see exactly what should be worked on, and have a concrete date by which everything will done.

This is horrible for everyone, including management, because the schedule is predicated upon developers being unerring prophets who are able to forecast not only the exact work that needs to be done to develop a release, but also the exact amount of time said work will take.

The ultimate delimiter of the work to be done is the schedule - usually there’s a specific release date (hopefully but not always far out enough to even theoretically get all the work done); whatever can get done by that date tends to be what’s released.

Waterfall also suffers greatly because it’s completely inflexible. Requirements are gathered months ahead of time; any changes require completely reworking the schedule, so changes are frowned upon. Thus, when the product is released, it’s usually missing features that would have been extremely beneficial to have.

• Agile

Agile can be viewed as a direct response to waterfall-style development; rather than a rigid schedule, the agile approach embraces iteration and quick releases. The three primary “laws” of agile are:

• **Law of the customer** - The customer is the number one priority. Rather than focusing on hitting arbitrary milestones or internal benchmarks, agile teams should be focused on delivering products to customers that bring them additional value. A single line of code changed can be more worthwhile than an entirely new product if that line brings extra value to the customer.

  • Law of small teams - Developers are grouped into small teams that are given autonomy in how they implement the features they’re working on. When work is assigned to a team, it’s not done so prescriptively. In the best agile teams, the assignment is, “Here’s the problem we have, go solve it.”

  • Law of the network - There are differing interpretations on how to implement this, but essentially I view of the network as “the whole organization has to buy in to what the agile teams are doing.” The entire organization doesn’t need to have the same structure as the agile teams, but neither can it be structured in a manner antithetical to the processes or outcomes. The easiest counterexample is the entire dev department is using scrum, but the CTO still feels (by virtue of their title) the ability to step in and make changes or contribute code or modify stories on a whim. Just because the CTO is the manager doesn’t mean they have full control over every decision. Basically, law of the network means “respecting the agile method, even if you’re not directly involved.”

  It’s worth noting that agile is a philosophy, not a framework in and of itself. Both kanban and scrum are implementations of the agile philosophy.

• **Kanban**

  This is usually the most confusing, because both scrum and kanban can use kanban boards (the table of stories, usually denoted by physical or virtual “post-its” that represent the team’s work). Kanban board splits up work into different “stages” (e.g., to-do, doing, done), and provides a visual way to track progression of stories.

  The primary difference between scrum and kanban as a product development methodology is that kanban does not have specific “sprints” of work - the delimiter of work is how many items are in a given status at a given time. For example, if team limits “doing” to four cards and there are already four cards in there, no more can be added until one is moved along to the next stage (usually this means developers will pair or mob on a story to get it through).

• **Scrum**

  Scrum, by contrast, delimits its work by sprints. Sprints are the collection of work the team feels is necessary to complete to deliver value. They can be variable in their length (though in practice, they tend to be a specified time length, which causes its own issues).

  Scrum requires each team to have at least two people - a product owner and a scrum master. Usually there are also developers, QA and devops people on the team as well, but at a minimum you need the PO and SM.

  The product owner has the vision for what the product should be - they should be in constant contact with customers, potential customers and former customers to figure out how value can be added. The scrum master’s job is to be the sandpaper for the developers - not (as the name implies) their manager or boss, but the facilitator for ceremonies and provide coaching/guidance on stories and blockers.

## Other reasons processes fail

I will note that a lot of the reasons I will list below may also apply to other product management methodologies; however, I’m specifically limiting the scope to how they impact scrum teams.

Lack of product vision

I don’t want to lay the blame entirely on product owners for this issue - very often the problem is with how the role is designed and hired for. Product owners should be the final arbiters for product decisions. They should absolutely consult design, UX and customer service experts for their opinions, but the decisions ultimately lies with them.

Unfortunately, the breadth of skills required to be a good product owner are not in abundant supply, and product owners are, bafflingly, often considered afterthoughts at many organizations.

More than specific skills, though, product owners need to have a vision for what the product could be, as well as the flexibility to adapt that vision when new information comes in. Usually, this requires domain knowledge (that can be acquired, but needs to be done so systematically and quickly upon hiring), steadfastness of conviction and the ability to analyze data properly to understand what customers want.

Far too often product owners essentially turn into feature prioitizers, regurgitating nearly everything customers say they want and assigning a ranking to it. This often comes at the expense of both the product’s conceptual integrity as well as relationships with developers, who are supposed to be given problems to solve, not features to develop. This is the classic feature factory trap.

Mistake the rules for the reason

Far too often, people will adopt the ceremonies or trappings of scrum without actually accepting an agile mindset. This is where my favorite tagline, “that’s just waterfall with sprints” comes from.

If you’ve ever started a project by first projecting and planning how long it’s going to take you to deliver a given set of features, congratulations, you’re using waterfall.

To use scrum, you need to adopt the iterative mindset to how you view your product. If you’re developing Facebook, you don’t say, “we’re going to build a system that allows you to have an activity feed that shows posts from your friends, groups and advertisters, and have an instant messaging product, and ..”

Instead, you’d say, “we’re going to develop a platform that helps people connect to one another.” Then you’d figure out the greatest value you can add in one sprint (e.g., users can create profiles and upload their picture.). You know once you have profiles you’ll probably want the ability to post on others’ profiles, so that’s in the backlog.

That’s it. That’s the planning you do. Because once those releases get into customers’ hands, you’ll then have better ideas for how to deliver the next increment of value.

Simply because an organization has “sprints” and a “backlog” and do “retros” doesn’t mean its’ using scrum, it means it’s using the language of scrum.

Lack of discipline/iteration

Tacking on to the last point, not setting up your team for success in an agile environment can doom the product overall. Companies tend to like hiring more junior developers, because they’re cheaper, but not realizing that a junior developer is not just a senior developer working at 80% speed. Junior developers need to have mentoring and code reviews, and those things take time. If the schedule is not set up to allow for that necessary training and code quality checks to happen, the product will suffer overall.

Similarly, development teams are often kept at a starting remove from everyday users and their opinions/feedback. While I by no means advocate a direct open firehose of feedback, some organizations don’t ever let their devs see actual users using the product, which creates a horrible lack of feedback loop from a UX and product design perspective.

Properly investing in the team and the processes is essential to any organization, but especially one that uses scrum.

Lack of organizational shift

The last ancillary reason I want to talk about in terms of scrum failure is aligning the organization with the teams that are using scrum (we’re back to the law of network, here). Scrum does not just rely on the dev team buying in, it also requires the larger organization to at least respect the principles of scrum for the team.

The most common example of this I see is when the entire dev department is using scrum, but the CTO still feels (by virtue of their title) the ability to step in and make changes or contribute code or modify stories on a whim. Just because the CTO is the manager doesn’t mean they have full control over every decision. Removing the autonomy for the team messes with the fundamental principles of scrum, and usually indicates there will be other issues as well (and I guarantee that CTO will also be mad when the scrum is now unbalanced or work doesn’t get done, even though they’re the direct cause).

No. 1 reason scrum fails: It’s used for other purposes

By far, the biggest reason I see scrum failing to deliver is when the ceremonies or ideas or data generated by scrum gets used for something other than delivering value to the end users.

It’s completely understandable! Management broadly wants predictability, the ability to schedule a release months out so that marketing and sales can create content and be ready to go.

But that’s not how scrum works. Organizations are used to being able to dictate schedules for large releases of software all at once (via waterfall), and making dev deliver on those schedules. If you’re scheduling a featureset six months out, it’s almost guaranteed you’re not delivering in an agile manner.

Instead of marketing-driven development, why not flip the script and have development-driven marketing? There is absolutely no law of marketing that says you have to push a new feature the second it’s generally available. If the marketing team keeps up with what’s being planned a sprint in an advance, that means they’d typically have at least a full month of leadtime to prepare materials for release.

Rather than being schedulable, what dev teams should shoot for is reliability and dependability. If the dev team commits to solving an issue in a given sprint, it’d better be done within that sprint (within reason). If it’s not, it’s on the dev team to improve its process so the situation doesn’t happen again.

But why does scrum get pulled off track? Most often, it’s because data points in scrum get used to mean something else.

Estimates

The two hardest problems in computer science are estimates, naming things, and zero-based indexes. Estimates are notoriously difficult to get right, especially when developing new features. Estimates get inordinately more complex when we talk about story pointing.

Story points are a value assigned to a given story. They are supposed to be relative to other stories in the sprint - e.g., a 2 is bigger than a 1, or a medium is bigger than a small, whatever. Regardless of the scale you’re using, it is supposed to be a measure of complexity for the story for prioritization purposes only.

Unfortunately, what usually winds up happening is teams adopt some sort of translation scale (either direct or indirect), something like 1 = finish in an afternoon, 2 = finish in a day, 3 = multiple days, 5 = a week, etc. But then management wants to make sure everyone is pulling their fair share, so people are gently told that 10 is the expectation for the number of points they should complete in a two-week sprint, and now we are completely off the rails.

Story points are not time estimates. Full stop.

It’s not a contract, you’re not a traffic cop trying to make your quota. Story points are estimates of the complexity of a story for you to use in prioritization. That’s it.

I actually dislike measuring sprint velocity sprint-to-sprint, because I don’t think it’s helpful in most cases. It actually distorts the meaning of a sprint. Remember, sprints are supposed to be variable in length; if your increment of value is small, have a small sprint. But because sprint review and retro have to happen every second Friday, sprints have to be two weeks. Because the sprint is two weeks, now we have two separate focii, and the scrum methodology drifts further and further away.

Campbell’s law is one of my favorite axioms. Paraphrased, it states:

The more emphasis placed on a metric, the more those being measured will be incentivized to game it.

In the case above, if developers are told they should be getting 10 points per sprint, suddenly their focus is no longer on the customer. It’s now on the number of story points they have completed. They may be disincentivized to pick up larger stories, fearing they might get bogged down. They’re almost certainly going to overestimate the complexity of stories, because now underestimates mean they’re going to be penalized in terms of hitting their targets.

This is where what I call the Concilio Corollary (itself a play on the uncertainty principle) comes into play:

You change the outcome of development by measuring it.

It’s ultimately a question of incentives and focus. If you start needing to worry about metrics other than “delivering value to the user,” then your focus drifts from same. This especially comes into play when organizations worry about individual velocity.

I don’t believe in the practice of “putting stories down” or “pick up another story when slightly blocked.” If a developer is blocked, it’s on the scrum master and the rest of the team to help them get unblocked. But I absolutely understand the desire to do so if everybody’s expected to maintain a certain momentum, and other people letting their tasks lie to help you is detrimental to their productivity stats. How could we expect teamwork to flourish in such an environment?

So how do we measure productivity?

Short answer: don’t.

Long answer: Don’t measure “productivity” as if it’s a value that can be computed from a single number. Productivity on its own is useless.

I used to work at a college of medicine, and after a big website refresh they were all excited reporting how many pageviews the new site was getting. And it makes sense, because when we think of web analytics, we think page views and monthly visitors and time on site, all that good stuff.

Except … what’s the value of pageviews to a college? They’re not selling ads, where more views works out to more money. In fact, the entire point of the website was to get prospective students to apply. So rather than track “how many people looked at this site,” what they should have been doing was looking at “how many come to this site and then hit the ‘apply now’ button,” and comparing that to the previous incarnation.

First, you need to figure out what the metrics are being used for. There are any number of different reasons you might want to measure “productivity” on a development team. Some potential reasons include performance reviews, deciding who to lay off, justifying costs, figuring out where/whether to invest more, or fixing issues on the development team.

But each of those reasons has a completely different dataset you should be using to make that decision. If you’re talking about performance reviews, knowing the individual velocity of a developer is useless. If it’s a junior, taking on a 5-point story might be a huge accomplishment. If you’re looking at a principal or a senior, you might actually expected a lower velocity, because they’re spending more time pairing with other developers to mentor them or help them get unblocked.

Second, find the data that answers the question. When I worked at a newspaper, we used to have screens all over the place that showed how many pageviews specific articles were getting. Except, we didn’t sell ads based on total pageviews. We got paid a LOT of money to sell ads to people in our geographical area, and a pittance for everything else. A million pageviews usually meant we had gone viral, but most of those hits were essentially worthless to us. To properly track and incentivize for best return, we should have been tracking local pageviews as our primary metric.

Similarly, if you’re trying to justify costs for your development team, just throwing the sprint velocity out there as the number to look at might work at the beginning, but that now becomes the standard you’re measured against. And once you start having to maintain features or fix bugs, those numbers are going to go down (it’s almost always easier to complete a high-point new-feature story than a high-point maintenance story, simply because you don’t have to understand or worry about as much context).

There are a number of newer metrics that have been proposed as standards that dev teams should be using. I don’t have an inherent problem with most of these metrics, but I do want to caution not to just adopt them wholesale as a replacement for sprint velocity. Instead, carefully consider what you’re trying to use the data for, then select those metrics that provide that data. Those metrics are SPACE and DORA. Please note that these are not all individual metrics; some of them (such as “number of handoffs”) are team-based.

SPACE

• Satisfaction and well-being

◦ This involves developer satisfaction surveys, analyzing retention numbers, things of that nature. Try to quantify how your developers feel about their processes.

• Performance

◦ This might include some form of story points shipped, but would also include things like number and quality of code reviews.

• Activity

◦ Story points completed, frequency of deployments, code reviews completed, or amount of time spent coding vs. architecting, etc.

• Communication/collaboration

◦ Time spent pairing, writing documentation, slack responses, on-call/office hours

• Efficiency/flow

◦ Time to get code reviewed, number of handoffs, time between acceptance and deployment

DORA

DORA, or DevOps Research and Assessment, are mostly team-based metrics. They include:

• Frequency of deployments

• Time between acceptance and deployment

• How frequently deployments fail

• How long it takes to recover/restore from failed

Focus on impact

But all of these metrics should be secondary, as the primary purpose of a scrum team is to deliver value. Thus, the primary metrics should measure direct impact of work: How much value did we deliver to customers?

This can be difficult to ascertain! It requires a lot of setup and analysis around observability, but these are things that a properly focused scrum team should already be doing. When the dev team is handed a story for a new feature, one factor of that story should be success criterion: e.g., at least 10% of active users use this feature in the first 10 days. That measurement should be what matters most. And failing to meet that mark doesn’t mean the individual developer failed, it means some underlying assumption (whether it’s discoverability or user need) is flawed, and should be corrected for the next set of iterations.

It comes down to outcome-driven-development vs. feature-driven-development. In scrum, you should have autonomous teams working to build solutions that provide value to the customer. That also includes accountability for the decisions that were made, and a quick feedback loop coupled with iteration to ensure that quality is being delivered continuously.

TL;DR

In summation, these are the important bits:

• Buy in up and down the corporate stack - structure needs to at least enable the scrum team, not work against it

• Don’t estimate more than you need to, and relatively at that

• Know what you’re measuring and why

Now, I know individual developers are probably not in a position to take action at the level “stop using metrics for the wrong reasons.” That’s why I have a set of individual takeaways you can use.

• Great mindset for performance review

◦ I am a terrible self-promoter, but keeping in mind the value I was creating made it easy for me come promotion time to say, “this is definitively what I did and how I added value to the team.” It made it much easier for me than trying to remember what specific stories I had worked on or which specific ideas were mine.

• Push toward alignment

◦ Try to push your leaders into finding metrics that answer the questions they’re actually asking. You may not be able to get them to abandon sprint velocity right off the bat, but the more people see useful, actionable metrics the less they focus on useless ones.

• Try to champion customer value

◦ It’s what scrum is for, so using customer value as your North Star usually helps cut through confusion and disagreement.

• Get better at knowing what you know / don't know

◦ This is literally the point of sprint retros, but sharing understanding of how the system works will help your whole team to improve the process and produce better software.

Since this post is long enough on its own, I also have a separate post from when I gave this talk in Michigan of questions people asked and my answers.

I grew up on Clean Code, both the book and the concept. I strove for my code to be “clean,” and it was the standard against which I measured myself.

And I don’t think I was alone! Many of the programmers I’ve gotten to know over the years took a similar trajectory, venerating CC along with Code Complete and Pragmatic Programmer as the books everyone should read.

But along the way, “clean” started to take on a new meaning. It’s not just from the context of code, either; whether in interior design or architecture or print design, “clean” started to arise as a synonym for “minimalism.”

This was brought home to me when I was working with a junior developer a couple years ago. I refactored a component related to one we working on together to enable necessary functionality, and I was showing him the changes. This was a 200-line component, and he skimmed it about 45 seconds before saying “Nice, much cleaner.”

And it bugged me, but I wasn’t sure why. He was correct - it was cleaner, but it felt like that shouldn’t have been something he was accurately able to identify simply by glancing at it. Or at least, if that was the metric he was using, “clean” wasn’t cutting it.

Because the fact of the matter is you can’t judge the quality of code without reading it and understanding what it’s trying to do, especially without considering it in the context of its larger codebase. You can find signifiers (e.g., fewer lines of code, fewer methods in a class), but “terse” is not a direct synonym of “clean.” Sometimes less code is harder to understand or maintain than more code.

I wanted to find an approach, a rubric, that allowed for more specificity. When I get feedback, I much prefer hearing the specific aspects that are being praised or need work on - someone telling me “that code’s clean” or not isn’t particularly actionable.

So now I say code should be Comprehensible, Predictable and Maintainable. I liked those three elements because they’re important on their own, but also each builds on the others. You cannot have predictable and maintainable code unless it’s also comprehensible, for example.

• Comprehensible - People other than the author, at the time the code is written, can understand both what the code is doing and why.

• Predictable - If we look at one part of the code (a method, a class, a module), we should be able to infer a number of properties about the rest.

• Maintainable - Easy to modify and keep up, as code runs forever

Comprehensibility is important because we don’t all share the context - even if you’re the only person who’s ever going to read the code, the you of three weeks from now will have an entirely different set of issues you’re focusing on, and will not bring the same thoughts to bear when reasoning about the code. And, especially in a professional context, rare is the code that’s only ever read by one other person.

Predictability speaks to cohesion and replicability across your codebase. If I have a method load on a model responsible for pulling that object’s information from the database, all the other models should use load when pulling object info from the DB. Even though you could use get or loadFromDb or any number of terms that are still technically comprehensible, the predictability of using the same word to mean the same thing reduces overall cognitive load when reasoning about the application. If I have to keep track of which word means the action I’m trying to take based on which specific model I’m using, that’s a layer of mental overhead that’s doing nothing toward actually increasing the value or functionality of the software.

Maintainability is the sort of an extension of comprehensibility - how easy is it the code to change or fix down the road? Maintainability includes things like the “open to extension, closed to modification” principle from SOLID, but also things like comments (which we’ll get to, specifically, later on). Comprehensibility is focused on the “what” the code is doing, which often requires in-code context and clear naming. Maintainability on the other hand, focuses on the “why” - so that, if I need to modify it later on, I know what the intent of the method/class/variable was, and can adjust accordingly.

The single most important aspect of CPM code is naming things. Naming stuff right is hard. How we name things influences how we reason about them, how we classify them, and how others will perceive them. Because those names eventually evolve to carry meaning on their own, which can be influenced by outside contexts, and that whole messy ball of definition is what the next person is going to be using when they think about the thing.

I do believe most programmers intellectually know the importance of naming things, but it’s never given the proper level of respect and care its importance would suggest. Very rarely do I see code reviews that suggest renaming variables or methods to enhance clarity - basically, the rule is if it’s good enough that the reviewer understands it at that moment, that’s fine. I don’t think it is.

A class called User should contain all the methods related to the User model. This seems like an uncontroversial stance. But you have to consider that model in the context of its overall codebase. If there is (and there should be) also a class called Authorization in that codebase,  there are already inferences we should be able to draw simply from the names of those two things.

We should assume User and Authorization are closely related; I would assume that some method in Authorization is going to be responsible for verifying that the user of the application is a User allowed to access parts of the application. I would assume these classes are fairly tightly coupled in some respects, and it would be difficult to use one without the other, in some respect.

Names provide signposts and architecture hints of the broader application, and the more attuned to them you are (as both a writer and reader of code), the more information will be able to be conveyed simply by paying attention to them.

If naming is the single most important aspect of CPM, the single most important aspect of naming things is consistency. I personally don’t care about most styling arguments (camelCase vs. snake_case, tabs vs. spaces, whatever). If there’s a style guide for your language or framework, my opinion is you should follow it as closely as possible, deviating only if there’s an actual significant benefit to doing so.

Following style conventions has the two advantages: allowing for easier interoperability of code from different sources, and enabling the use of linters and formatters.

Code is easier to share (both out to others and in from others) if they use same naming conventions and styles, because you’re not adding an extra layer of reasoning atop the code. If you have to remember that Library A uses camelCase for methods but Framework B uses snake_case, that’s however large a section of your brain that is focusing on something other than the logic of what the code is doing.

And enabling linters and formatters means there’s a whole section of code maintenance you no longer have to worry about - you can offload that work to the machine. Remember, computers exist to help us solve problems and offload processing. A deterministic set of rules that can be applied consistently is literally the class of problems computers are designed to handle.

Very broadly, my approach to subjective questions is: Be consistent. Anything that doesn’t directly impact comprehensibility is a subjective decision. Make a decision, set your linter or formatter, and never give another thought to it. Again, consistency is the most important aspect of naming.

But a critically under-appreciated aspect of naming is the context of the author. Everyone sort of assumes we all share the same context, in lots of ways. “Because we work on the same team/at the same company, the next developer will know the meaning of the class PayGrimples.” That may be very broadly true, in that they’ve probably heard of PayGrimples, but it doesn’t mean they share the same context.

A pop-culture example of this is pretty easy - think of the greatest spaceship pilot in the universe, one James Tiberius Kirk. Think about all his exploits, all the strange new worlds he’s discovered. Get a good picture of him in your head.

Which one did you pick? Was it The Original Series’ William Shatner? The new movies’ Chris Pine? Or was it Strange New Worlds’ Paul Wesley?

You weren’t wrong in whatever you picked. Any of those is a valid and correct answer. But if we were talking about Kirk in conversation, you likely would have asked to clarify which one I meant. If we hadn’t, we could talk about two entirely different versions of the same concept indefinitely until we hit upon a divergence point when one of us realized.

Code has that same issue, except whoever’s reading it can’t ask for that clarification. And they can only find out they’re thinking about a different version of the concept if they a) read and digest the code in its entirety before working on it, or b) introduce or uncover a bug in the course of changing it. So when we name things, we should strive for the utmost clarity.

⛔ Unclear without context

type User = { id: number; username: string; firstName: string; lastName: string; isActive: boolean; }

The above is a very basic user model, most of whose properties are clear enough. Id, username, firstName and lastName are all pretty self-explanatory. But then we get to the boolean isActive.

This could mean any number of things in context. They include, but are not limited to:

• The user is moving their mouse on the screen right now

• The user has a logged-in session

• The user has an active subscription

• The user has logged in within the last 24 hours

• The user has performed an authenticated activity in the last 24 hours

• The user has logged in within the last 60 days

All of those are things we may want to know about the user of any application, depending on what we’re trying to do. Even similar-sounding events with the same time horizon (logged in within the last 24 hours vs. authenticated activity in the last 24 hours) give us different information - I can infer the maximum age of the authentication token in the logged-in case, but without knowing the token exchange process, I cannot make the same inference for authenticated activity.

So why not just provide the meaning with the name?

✅ Clarity without context

type User = { id: number; username: string; firstName: string; lastName: string; loggedInPrevious24Hours: boolean; }

Clarity comes through naming things explicitly. Ambiguity is the enemy of clarity, even when you assume the person reading the code should know something.

It’s reasonable to assume that the people reading your code are developers - that is, people familiar with coding concepts. Every other context (industry/domain, organization) is not a safe assumption. Therefore, if you have names or terms that are also used in coding, you should clarify the other meaning. (You should do this generally, as well, but specifically with programming-related terms.)

⛔ Ambiguity kills comprehension

class Class {} class Post {}

The word “class” is generally understanding in programming as an object-oriented prototype. Outside of programming, it could refer to a classroom of children; a classification system; a group of children in the same grade (e.g., junior class); or a social hierarchy (e.g., upper-class, lower-class).

Post is even worse, because it can be a verb or a noun even in a programming context. Blogs usually have posts, but you can also post content (or the HTTP verb, POST). Non-tech-wise, we have places to which you can be sent (“I’m being sent to our post in London”), referring to the mail system, or even structural support for fences.

✅ Specificity aids everyone

class Classroom {} class BlogPost {}

All of this is important because being clear matters more than being concise or clever. After consistency, the most important aspect of naming is being descriptive. The name should describe what the code is doing (vs. why or how) - what a method is doing, or what purpose a variable serves.

For the most part, Classes should be nouns, because they’re describing their domain of influence. Methods should include verbs, because they’re performing actions. Variables should be nouns, reflective of whatever purpose they’re serving.

If you find yourself struggling with the length of your names of methods, variables or classes, that’s not a bad thing. It’s usually a sign you need to consider refactoring (more on this a bit later).

To the point of clarity, be sure to use properly spelled real words and names.

⛔ Abbreviations and shortcuts

class DateUtil { static function dateStrFrmo(date: Date): string { ... } }

Humans have surprisingly good short- and medium-term recall around words and names. Using real words and names makes the concept easier for us to reason about, and easier to keep track of in our heads.

I took the example above from a GitHub code search. I think the original example may have been written by a native German speaker, because if we assume “Frmo” is supposed to be “From,” it’s using the German sentence structure that puts the verb at the end of the sentence. That makes sense! But if someone isn’t familiar with that sentence construction, the name of the method becomes functionally useless.

The misspelling part is important in two respects: one, it can introduce confusion (is it supposed to be “from” or ”form”?). The other is relying on the computer - searches, within the IDE or if you’re GREPing, are looking for specific terms. If it’s spelled wrong, it’s not going to get caught in the search.

✅ Use properly spelled real words and names

class DateUtil { static function getStringFromDate(date: Date): string { ... } }

Here we’ve modified it so we essentially have an English sentence - get the string from the date. I know what’s being passed in (the date), and I know what’s coming out (the string), and I know overall what’s happening (I’m getting the string from the date).

Beyond naming, there is one other “big” rule that gets us to comprehensible, predictable and maintainable code, an old adage: “Keep it simple, sweetheart.” I’m not speaking to system complexity here - your overall architecture should be as complex as needed to do the job. It’s closer to SOLID’s single-responsibility principle, writ large: Every module, every class, every method, every variable should have one job.

To our earlier example of Users and Authorization, users will take care of the users while authorization handles auth. Neither of them should care about the internal workings of the other; Authorization just needs to know it can call User::load to return the user object.

At the method level, this is how we keep our names to a manageable length. You should be able to describe what the method in a very short sentence. If you need more length (or you leave out things it’s doing), it’s probably a sign that the method is trying to do too much.

Smaller methods enable reusability - if the method is only doing a specific thing, we are more likely to be able to use it somewhere else. If the method is doing multiple things, we’d likely need to add a parameter in the other cases where we want to use it, because we don’t want all of those things to happen all the time.

Keeping each method to a single task means we can decompose complex methods into multiple individual methods. This also makes it easier to read the code.

Literally just reading the names of methods allows us to infer what’s going on, divorced of context. For the example below, we would know from the file name this is Typescript, and I’ll give one hint that it’s frontend.

✅ Keep it simple, even for complex actions

function constructor() { this.assignElements(); this.setInterval(); this.getNewArt(); this.listenForInstructions(); }

Initializing this class assigns elements and sets an interval (meaning there are actions that happen on a set schedule); then we get new art, and listen for instructions. Without even knowing the name of the class, we can pretty confidently assume this has to do with art, and that art gets changed frequently (hence the interval). But there also appears to be a manual interruption possible, with listen for instructions.

If we were debugging an issue related to keyboard commands, I would first look to listenForInstructions. If there’s an issue with art not showing up, I would check getNewArt.

Each method is narrowly scoped, even if a lot happens. Keeping things simple aids comprehension and predictability, but it’s also vital for maintainability. It makes it much easier to write tests.

We cannot confidently change code without tests. If we’re making modifications to code without tests, we can read, guess and hope that it won’t create any downstream issues, but unless we know exactly what should happen with a given method in all the ways it can be used, we cannot be certain of the impact of any change. A downstream issue is the definition of a regression; determining the output of a method in changing circumstances is the definition of a test. Thus why we test to avoid regressions.

A good unit test is like a science experiment - a hypothesis is proffered, and borne out or disproven through data, accounting for variables. In programming, variables are literally variables.

If we know exactly what our code will do, we have the flexibility to use it in different circumstances. That may sound tautological, but the confidence we know “exactly” what it will do comes through tests, not an internal sense of “I know what that function does." I would argue most bugs arise through either typos or incorrect assumptions. Most of the typo bugs are caught before release. Most of the insidious bugs that take forever to debug are because someone made an assumption along the way that you have to find and correct.

If all functions perform as we expect, integration issues are drastically reduced. Good unit testing reduces the amount of integration and regression testing you have to do. Maintenance overall becomes easier because we have solid bulwarks of functionality, rather than needing to reason through all possible eventualities fresh every time (or worse, assuming).

I’m not a huge believer in code coverage as a benchmark for quality. I think it can be helpful to have a minimal coverage requirement as you’re starting to remind yourself to write tests, but 100% coverage means absolutely nothing on its own. Quality is much more important than quantity when it comes to testing, especially that you’re testing the right things.

Keeping it simple also relates to abstractions. Code is a series of abstractions (unless you’re writing assembly, in which case, vaya con Dios), but I’m referring specifically to abstractions in the codebase that you write. The cardinal sin of object-oriented programming is a simple rule: “Don’t Repeat Yourself.” It’s not … bad advice, but neither is it a simple panacea we could automate away with, say, a linter or a formatter (or, god forbid, AI).

DRY is overemphasized, possibly because it’s such an easy heuristic to apply. “Hey this looks like other code” is easy to see at a glance, and if you just have an automatic reaction of “I shouldn’t repeat myself ever,” you’ll automatically push that logic up to single method that can be used in multiple places.

But deduplication requires an abstraction. In most cases, you’re not performing exactly the same logic in two places, but two minor variations (or the same logic on two different types of objects). Those variations then require you to include a parameter, to account for a slight branch.

Having that abstracted method hinders comprehensibility. Even if it’s easier/faster to read a one-line reference to the abstracted method, the actual logic being performed now happens out-of-sight.

I am much less concerned with duplication of code than I am making sure we find the right abstraction. Thus, I want to propose a different model for thinking about repetition, two rules  (because again, simpler != terse) to replace the one DRY rule: we’ll call it the Concilio Corollary to the DRY rule, or the damp dyad.

1. Don’t repeat yourself repeating yourself

2. The wrong abstraction will cost you more than repetition

DRYRY is a little tongue-in-cheek, but essentially don’t worry about trying to find an abstraction until you’ve implemented similar logic at least three times. Twice is a coincidence, three times is a pattern. Once you’ve seen the code being used in three different places, you now have the context to know whether it’s a) actually doing the same work, and b) how to abstract it to work in different scenarios.

If you find yourself adding a parameter that changes the flow of logic in each scenario, it’s probably more correct to abstract only those small parts that are the same, and implement the particular logic in the context it’s being used. That’s how we find the right abstraction.

All of this is important because existing code has inertia. This is relevant whether you’re a more senior developer or just starting out in your career.

Those with less experience tend to be terrified to change existing code, and understandably so. That code already exists, it’s doing a job. Even if you’re going in to fix a bug, presumably that code was working well enough when it was written that no one noticed it. And no one wants to be the one to create an error, so the existing code is treated as something close to sacrosanct.

For more experienced developers, know that when you’re writing code you’re creating the building blocks of the application. You’re setting the patterns that other developers later will try to implement, because it’s “correct” and becomes that codebase’s style. Heck, that’s literally the predictability maxim - we want to it look similar when it does similar things. But that means if you’re writing the wrong abstraction in one place, its impact may not be limited to that single area.

And when a new case arises, the next developer has to decide (without the context of the person who originally wrote it) whether to modify the existing abstraction, or create a new one. But the old one is “correct” (again, in that exists), so it’s safer to just use that one. Or, worst case, use it as a template to create a new abstraction. In either case, a new paradigm is being created that needs to be tested and raises the overhead on maintenance, because now we have a separate logic branch.

Those are the big topics I wanted to hit. The rest of these recommendations are important, but lack an overall theme. The biggest of these I want to discuss is commenting.

Naming should be used so we know the “what” of code, comments should be used so we know the “why.” I am not referring to automated comments here (e.g., explanations for input parameters in the like in JSDoc), but rather qualitative comments. I would argue that, currently, most existing comments I see would be superfluous if proper naming conventions were used.

What I want to see in a comment is why a particular variable is a particular value, when it’s not clear from the existing context.

⛔ Don't explain the what

const SOCIAL_MEDIA_CHARACTER_COUNT = 116; // shortens title for social media sharing export const getSocialShareText = (post: BlogPost) => { if (post.title.length =< SOCIAL_MEDIA_CHARACTER_COUNT) { return post.title; } else { return post.title.substr(0,SOCIAL_MEDIA_CHARACTER_COUNT); } }

This a pretty typical example of what I see comments used for. We’ve used naming properly (the method gets the social share text, the constant is the character count we use for social media posts)j, so the comment “shortens title for social media sharing” is superfluous.

This method provides the social media content. The piece of information I don’t have about this code that I would like, both for comprehensibility and maintainability, is why the character count is 116.

The answer is that Twitter used to be the social media service with the shortest content length, 140 characters. Except that since we’re developing an app, we’re always including a URL, for which Twitter automatically generates a shortlink that takes up 23 characters (+ 1 for the space between content and link). 140-23-1 = 116.

That context does not exist within the application, and it’s not under our control. So we should include it in a comment, so that if that number changes (or something else becomes popular but has a shorter length limit, or we stop worrying about Twitter entirely), we know both from reading the code what this does, and it puts a signpost with the word “Twitter” in the comment so it can be found if we just do a search.

✅ Explain the "why"

// Twitter has shortest character limit (140); URL shortener is always 23 + space const SOCIAL_MEDIA_CHARACTER_COUNT = 116; export const getSocialShareText = (post: BlogPost) => { if (post.title.length =< SOCIAL_MEDIA_CHARACTER_COUNT) { return post.title + ' ' + post.url; } else { return post.title.substr(0,SOCIAL_MEDIA_CHARACTER_COUNT) + ' ' + post.url; } }

The other thing to keep in mind about comments is that they’re a dependency just as much as code. If we do update that character count, we also need to update the comment explaining it, otherwise we’ve actively corrupted the context for the next person who has to make a change.

I used to say “never use ternaries,” but I’ve come around a bit. I now believe ternaries should be used only declaratively, with proper formatting.

✅ Use declarative ternaries, with formatting

`const title = (postRequest['title']) ? postRequest['title'] : '';

const title = postRequest['title'] || '';`

Ternaries are short, concise, and difficult to reason about if they’re too complicated. When I say “declarative” ternaries, I mean “the value of a variable is one of two options, dependent upon a single condition.”

If you need to test multiple conditions, or if you have more than one variable changing as a result of a condition or set of conditions, don’t use ternaries. Use regular if-else statements. It’s easier to read and comprehend, and it’s easier to make changes down the road (more likely if already have multiple conditions or states).

And never nest ternaries.

The last bit is around testing, specifically standard library functions. A standard library function is one that comes packaged in with the programming language you’re using - think Math.round() for Javascript, or the above substring method on strings str.substr(0,3).

As a rule, you should not test the functionality of code you have no control over - if Chrome is shipping a bad Math.round(), there isn’t anything you can do about it (plus, if you go down that rabbit hole long enough you’ll eventually have to test that the heat death of the universe hasn’t yet happened). Standard library functions fit that description.

But sometimes you do you want to test a method that only uses standard library functionality - the reason is not that you’re testing that functionality, but rather that you’re arriving at the desired result.

We’ll use the social media text as the example. I will always assume substring is working properly until I get user reports, and even then the most I would do is forward them along. What I want to test for is the length of the string that is returned - does it meet my requirements (under 116)? I’m not testing the functionality, I’m including a flag to myself and future developers that this is the maximum length and, if someone modifies the functionality of the method, it should be flagged.

describe('getSocialMediaText restricts to Twitter length', ()=> { it('when title is less than length', () => { expect(getSocialMediaText(MockPostShortTitle).length =< 116) }), it('when the title is more than length', () => { expect(getSocialMediaText(MockPostLongTitle).length =< 116) }) });

If we were testing functionality, I would call the same constant in my test, because that’s what’s being used internally. But because I’m testing outcomes, I use an independent value. If someone changes the length without changing the test, they’ll get notified. They can at that point change the value used in the test, too, but the test has served its purpose - it notified someone when they violated my “why.”

TL;DR

• Focus on specific aspects of code quality

• Comprehensible, Predictable, Maintainable
• Name stuff properly
  • Clarity over concision and wit
• Keep things simple
  • One module, one class, one method, one variable: One job
• Write tests
  • The only way to confidently modify or reuse code is be assured of what it does
• Remember the damp dyad

  • Don’t repeat yourself repeating yourself

  • The wrong abstraction costs more than repetition

• Comments should explain "why"
  • Provide context for the next person (let naming focus on “what”)

Solutions come in all sizes. The problem in tech (and many other industries, I presume) is that our processes and workflows are structured in such a way that the solutions for a given problem tend to be clustered around the smaller side of the scale.

Consider any given bug. Reported (hopefully) by your QA team or, worst-case, by a customer in production, it points out a specific issue. You, the developer, are tasked with devising a solution. Now, in most shops you’ll be given the opportunity to work out the root cause, ensuring that whatever change you make will a) actually fix the problem, and b) not cause any other immediate problems.

And that makes sense, for the most part. Small issues have small solutions. The problem is when you don’t step back and take a bigger-picture view of the situation - do all of these disparate problems actually stem from a particular source? Very often, developers are not only encouraged but actually mandated to stick to whatever story they’re on, for fear of going out of scope.

While that might make sense from a top-down control perspective, that style of thinking tends to permeate a lot of the other work that gets done, even up to larger-scale issues. Diversity is left to HR, or to a diversity committee, to take care of. In many cases, how and where to include AI in an application is left up to individual departments or teams. Remote work, a topic extremely divisive of late, is being eliminated or limited left up to “manager discretion” rather than actually looking at the benefits and harms that are associated with it. A cause extremely close to my heart, accessibility, is frequently treated as an add-on or left up to a handful of specialists to implement (or, worse, a third-party plugin).

These things not only don’t have to, they shouldn’t be left up to small groups to implement or reason through. They should be baked-in to how your organization makes decisions, builds software and interacts with its people.

You need a holistic approach. I want to break these concepts out of silos. If we're looking at a RACI chart, everyone is responsible for DEIB and accessibility. Everyone should be consulted and accountable for decisions about AI and remote work.

Now, I have a confession. I'm pretty sure it's Steve Jobs’ Second Law of Product that any time you think you have an insight, you have to give it a fancy name. I am guilty of this as well. 

I use the term “holistic tech” to talk about the convergence of these ideas. A lot of the specific things I'm talking about can be found in other systems or methodologies; I'm just trying to pull all the threads together so we can hopefully weave something useful about it. In the same way that responsive design was concerned with making sure you could use a product across all screen sizes, I want to make sure that (and here's the subtitle) tech works for everybody.

I'm also gonna borrow some concepts from universal design. Universal design is the concept that, "the design and composition of an environment so that it can be accessed, understood and used to the greatest extent possible by all people regardless of their age, size, ability or disability."

And last, we'll also fold in some concepts of human-centered design. This, in a nutshell, is thinking beyond your optimal user story. Eric Meyer calls them "stress cases,"  as opposed to edge cases, where you consider the emotional, physical and mental state of your user, rather just concerning yourself with the state of your application. 

But all of these, as implied with the word "design," are focused primarily on product creation. And while I do want to incorporate that, it's a part of how we work.

Basically, this whole idea boils down to a single word

EMPATHY

It's about seeing other people as, well, people.  

And it's applicable up and down your company stack. It applies to your employees, your boss, your monetization strategy (specifically, not using dark patterns), and it's especially about your communication, both within your organization and with your users.

As for product design, we'll start with accessibility.

Very broadly, accessibility is concerned with making sure that everyone can access your content and product. On the web side of things, this typically is accomplished by trying to adhere to the Web Content Access Guidelines, or WCAG. 

WCAG has four basic principles:

• The first is that content should be perceivable, which relates to multi-sensory content and interfaces. Essentially, you should still be able to access the fundamental value of the content even if you cannot engage with its primary medium; the common examples here are alt text for images or captions for videos.

• The second principle is operable: Users must be able to operate user interface controls in multiple modalities. The most common example of this is keyboard navigability; there are several requirements around people being able to video controls or manipulate modals without using the mouse (or touch).

• The third principle is understandable: Text needs to be readable and understandable, and user interface elements should behave in predictable ways. Headers should always act like headers.

• The last principle is robustness, which amounts to future-proofing. Make sure you adhere to the specs so that future products that are trying to parse your content know they can do so in a coherent manner.

Now the interesting thing is, I don't think many people would object to those principles in, well, principle. They seem pretty common-sensical? "I want people to be able to access my content" is a fairly unobjectionable statement. The problem is that most organizations don't have a good sense for accessibility yet, so the projects are designed and budgeted without the specific accessibility implementations. Then, when it gets brought up, making the change would be "too expensive," or it would "take too long."

"And besides, it's an insignificant part of our market anyway."**** I cannot tell you how many times I've heard this argument. Whether it's an intranet ("we don't have that many disabled people working here") or an internal training video (“there aren’t that many blind workers”) or a consumer-facing product ("we're willing to live without that tiny part of the market"), there's a sense that accessibility is only for a very small subset of the population.

My favorite group accessibility experiment is to ask people to raise their hand if they use an accommodation.

Then, I ask them to raise a hand if they wear glasses, contacts, or a hearing aid.Or if they don't keep your monitor at full resolution ("less space," on Macs). Or if they ever change their  browser's or IDE's zoom level.

Those are all accessibility accommodations.

Because the truth of the matter is, we're all just temporarily abled. I don’t ask for hands on this one don't, but I’ll often ask if anyone’s ever bought something on eBay while drunk. Formally speaking, you are technically operating with a cognitive impairment when you bought that giant taco blanket on Amazon. And I'm willing to bet your fine motor skills weren't quite up their usual par, either.

Or maybe you sprained your wrist, or broke a finger. That's a loss of fine motor control that's going to make it more difficult to operate the mouse, even if only for a few weeks. Or how about any kind of injury or chronic pain that makes it painful to sit in a chair for long periods? Willing to bet after 4 hours you're not thinking as clearly or as quickly as you were during hour 1.

Some of these things, like neurodivergence or vision impairment or being paralyzed, can be permanent conditions. But just as many of them aren't. And it's important to keep that in mind, because even if your ideal user story is a 34-year-old soccer mom, chances are she's going to have some sort of cognitive impairment (lack of sleep, stress about kids) or processing difference (trying to juggle multiple things at the same time) or fine motor skills (trying to use your mobile app on the sidelines during December) at some point. So ignoring accessibility doesn’t just disenfranchise the “small” portion of your users who are visibly permanently disabled, it's making things more difficult for potentially all of your users at some point or another.

And as it turns out, adding accessibility features can actually grow your overall market share.

Imagine your first day at NewTube, the hottest new video app on the market. We're looking to change the world … by letting people upload and watch videos. I don’t know, venture capital! Anyway, the number of humans on the internet is 5.19 billion, so that’s our addressable market. We don’t need the microscopic share that would come from adding accessibility features.

Or do we?

Standard accessibility features for videos include text transcripts of the words spoken aloud in the video. The primary intention behind these is to ensure that those with hearing impairments can still understand what’s going on in the video. In a past job, proper captions cost somewhere in the range of $20+ per minute of video, though some products such as YouTube now have AI autocaptioning that’s getting pretty good. 

Another standard feature is an audio description track (and transcript). This is sort of like alt text for video –  it describes the images that are being shown on the screen, in order to make that information comprehensible to someone with visual impairments. 

My favorite example of this is the end scene from the movie Titanic. As a transcript, it looks like this:

[ music swells ]

ROSE: Uh!

[ splash ]

Audio description, on the other hand, would look something like this:

Present-day Rose walks to the bow of the research ship, which is deserted. Deep in thought, she climbs the railing and stares down at the water where the Titanic rests below. She opens one hand to reveal the Heart of the Ocean diamond. We flash back to 17-year-old Rose standing on the deck of the Carpathia, digging her hands into Cal’s overcoat and finding the diamond. Present day Rose shakes her head and, with a small gasp, sends the diamond to rest where it should have been some 80 years earlier.

I took some poetic license there, but that’s kind of the point of audio description – you’re not a court reporter transcribing what’s being said, you’re trying to convey the emotion and the story for those who can’t see the pictures. The transcript part isn’t technically a requirement, but since you typically have to write down the script for the AD track anyway, it tends to be included. To my knowledge, no one’s managed to get AI to do this work for them in any usable fashion.

Lastly, we have keyboard navigability. Being able to interact with and control the site just using a keyboard makes it easy for those without fine motor control (or who use screen readers) to easily find their way around.

Three features/feature sets. The first two are pretty expensive - we’ve either got to pay for or develop an AI service to write the transcriptions, or we have to make sure they’re available. Audio Descriptions are going to be a cost to us, regardless, and not a cheap one. Keyboard navigability could be built-in to the product, but it would be faster if we could just throw everything together in React and not have to worry about it. 

How much of an impact could it have on our audience?

Well, though only 2-3 children out of 1000 are born with hearing impairment, by age 18 the percentage of Americans who complain of at least partial hearing loss rises to about 15%. So if we don’t have captions, we’d better hope all our videos are Fail compilations, or we’re going to see some steep drop-offs.

When it comes to vision, it’s even worse. Approximately a billion people in the world have a vision impairment that was not prevented or has not been addressed. Even assuming significant overlap with the hearing impairment group, we’ll use 750,000,000, for a total of 10.8 percent.

And for inability to use a mouse, we’ll look at “overall prevalence of adults with a month of musculoskeletal pain related to a repetitive stress injury,” which isn’t nearly a large enough category to include everyone who might be navigating by keyboard, but is at 4%.

Which leaves us 70% of our addressable market, or 3.63 billion.

Now obviously these numbers are not exact. We’re very back-of-the-napkin here, but I would also argue that a real-world scenario could just as easily see our percentages of accommodation-seekers go up as down. The number of temporary cases of all of these items, the fact that first-world countries have higher prevalence of RSI (though much better numbers for vision impairment) mean that this 70% number is probably not as far away from reality as we think.

And even beyond people who need those accommodations, what about those who simply want them? 

My best friend watches TV with the captions on all the time because it’s easier for her to follow along, and she’s not alone. Netflix says 40% of global users watch with captions, to say nothing of public exhibitions like bars (where it’s often not legally permissible to have the sound on). 

Transcripts/audio descriptions are often HUGE boons to SEO, because you’re capturing all your content in a written, easily search-indexable format. 

And presumably you’ve used a video app on a TV. The app has already been designed to be used with directional arrows and an OK button - why not extend that to the desktop? You’ll notice the remote’s functionality is a subset of a keyboard, not a mouse. Boom, keyboard navigation.

So, to recap accessibility: Good for disabled users. Good for abled users. Good for business.**** And that’s the thing, taking a holistic approach to how we do tech should actually make everyone better off. It is the rising tide.

But let’s talk about the looming wave that overshadows us all. I speak, of course, of artificial intelligence. In the same way that software ate the world 15 years ago, and Bitcoin was going to replace all our dollars, artificial intelligence is going to eat all our software and all the dollars we software developers used to get paid. 

I want to make clear up front that I am not an AI doomsayer. I don’t think we’re (necessarily) going to get Skynetted, and if we are it’s certainly not going to be ChatGPT. Artificial intelligence in its current form is not going to enslave us, but I do think large swaths of the population will become beholden to it – just not in the same way. 

Similar to how algorithms were used in the 90s and 2000s to replace human decision-making, I think AI is going to be (ab)used in the same manner. We’ve all called in to a customer support line only to find that the human on the other end is little more than a conduit between “the system” and us, and the person can’t do anything more to affect the outcome than we can. 

With AI, we’re just going to skip the pretense of the human and have the AI decipher what it thinks you said, attempt remedies within the limits of what it’s been programmed to allow, and then disconnect you. No humans (or, likely, actual support) involved.

Is that the worst? Maybe not in all cases. But it’s also, in a lot of cases, going to allow these organizations to skip what should be important work and just let the AI make decisions. I’m much less concerned about SkyNet than I am the Paperclip Maximizer.

The paperclip maximizer is a thought experiment proffered by Nick Bostrom in 2003. He postulated that an AI given a single instruction, “Make as many paperclips as possible,” would/should end with the destruction of the entire earth and all human life. The AI is not given any boundaries, and humans might switch the machine off (thus limiting the number of paperclips), so the AI will eventually eliminate humans. But even if the AI thinks us benign, at some point the AI consumes all matter on the earth aside from humans, and we are just so full of wonderfully bendable atoms that could be used for more paperclips.

The “thought processes” of generative AIs, as currently constructed, are inherently unknowable. We know the inputs, and we can see the outputs when we put in a prompt, but we can’t know what they’re going to say - that’s where the special sauce “thinking” comes in. We try to control this by introducing parameters, or guidelines, to those prompts to keep them in line.

And I know you might think, “Well, we’ll tell it not to harm humans. Or animals. Or disrupt the existing socio-political order. Or …” And that’s actually a separate angle to attack this problem - humans not giving the proper parameters. At a certain point though, if you have to control for the entire world and its infinite varieties of issues, isn’t it easier to just do the work yourself? We’ve already got a lackluster track record in regard to putting reliable guardrails around AI, as the Bing Image Generator’s output so thoughtfully proves. 

One of the things computer nerds love to do more than anything is break new tech, and image generators are no exception. When it introduced a new image generation tool a while back, though Bing did restrict uses of the phrase “9/11" or “September 11,” it still allowed for image generations of “Spongebob flying an airliner into New York in 2000.” And of course, the most prominent image of New York in 2000 is likely going to include the World Trade Center.

Sure, Spongebob doing 9/11 is a brand hit to Nickelodeon and insulting to the victims’ families. But this is showing both failures - despite Bing’s overwhelming image consciousness that should have been baked into a model, the model thought it more important to generate this image than to not. And, separately, Bing failed to put proper safeguards into the system. 

So yes, the paperclips are a hyperbolic hypothetical, but if there’s one thing that capitalism has taught us it’s that there are companies out there who care more about the next dollar than anything else.

Businesses large and small make decisions based on weighing costs versus expected benefits of a given option all the time. Famously, with the Ford Pinto, one of the analyses Ford conducted cited the overall cost of redesigning fuel safety systems vs. the general cost to society of the fatal car crashes that might be spared. Because, to Ford, individual deaths were not thought of as particularly tragic. They were just numbers. It does not seem unreasonable to assume AI systems will be misused by those who are unscrupulous in addition to those who are just oblivious. 

In accessibility, most people think the cost of not being accessible is “well, how likely are we to get sued?” ignoring the benefits of people using the product more. With AI, this short-sighted calculus can come into play where “Oh, we’ll let the AI do it, and not have to pay a person!” Except, as we’ve pointed out, the AI probably isn’t very good and the cost comes in consumer goodwill.

And this doesn’t even touch things like source data bias, which is a huge issue in resume-reviewing AIs (whose datasets will cause the AI to be more likely to select for existing employees, exacerbating skewed hiring trends) and predictive policing algorithms (which exacerbate existing crime biases). 

Don’t forget you can now convincingly generate human-sounding responses in astroturfing campaigns or review spoofing, or empower scammers previously held back by non-native English suddenly sounding like every corporate communication (because AI’s probably writing those communiques, too).

Remember the part where I said I’m not an AI doomsayer? I’m really not! I think AI can be used in a lot of unique and interesting applications to make things better. We just need to be more judicious about how we employ it, is all. 

For example, in the medical field, there are numerous AI experiments around trying to find tumors from body scans; the AI is not notifying these people on its own, there are doctors who review flagged scans for closer examination. Or in drug trials, companies are using AI to imagine new shapes of proteins that will then have lots of trials and study before they’re ever put in a test subject. 

Using AI to generate advice that is then examined by humans for robustness is a great application of the tool. And sure, if Amazon wants to use AI to suggest product recommendations, I guess go ahead. It can’t be any worse than its current system of, “Oh, you bought a refrigerator? I bet you also want to buy several more.”

But that “generation” word is a sticking point for me. To the point of the job applicant winnowing, I have no problem with using quantitative questions to weed out applicants (do you have x years of experience, boolean can you work in the US), but I would hesitate to let a black-box system make decisions even as small as who should be considered for hiring based on inherently unknowable qualifications (as would be the case with the application of actual AI versus just algorithmic sifting).

And finally, just limit the use of generated content in general. Reaching back into my accessibility bag for a minute, there’s a class of images that per spec don’t need alt text: Images that are “purely” decorative and not conveying information. The question I always ask in such cases is: If the image is really providing no value to the user, do you really need it?

The same would go for unedited generated content. If you’re sending a communication that can be wholly generated by a computer, do you really need to send it? We’re taking the idea of “this meeting could have been an email” even further down the stack: Could that email in fact just be a Slack message, or better yet a reaction emoji? Just because you can expand your one-sentence idea more easily with AI doesn’t you have to or even should.

There’s likely a place for generated content, but it’s not anywhere near where we’re using it for now, with AI-generated “news” articles or advertising campaigns. It’s like when we just tried to add accessibility “with a button” - you cannot just throw this stuff out there and hope it’s good enough.

And I would hope it would go without saying, but please don’t replace therapists or lawyers or any other human who considers ethics, empathy, common sense or other essentially human traits with AI.

This is along the same lines as “generate advice not decisions,” - if you need to talk to the AI in order to be comfortable sharing things with a live person, that makes total sense. But don’t use the AI as a 1:1 replacement for talking to a person, or getting legal advice.

AI recap: Good for advice, not decisions. Good for assisting people, not replacing them (it’s a tool, not the mechanic). It can be good for business.

Now, I think at this point you can pretty much guess what I’m gonna say about remote work. And that’s good! Both because this is already long enough and because “holistic tech” is supposed to be a framework, not just specific actionable items. 

Remote work, of course, is the idea that you need not be physically present in a building in order to perform a job. Hybrid work is a mix of remote work with some time spent in the office. I’m not gonna try sell you hard on either option - but I will note that employees prefer flexibility and employers tend to enjoy the larger talent pool. But mostly, I want to talk about how to set up your organization for success in the event you choose one of them.

One of the issues when you have some people in the office and others who aren’t is the sense that employees in the office are prioritized above those who are remote. Some of this is understandable – if the company wants to incentivize people to come into the office by offering, for example, catered lunches once a week or something, I wouldn’t see that as something that those who aren’t attending are missing out on …. Unless they were hired as fully remote.

In my case, for example, company HQ is in Chicago; I live in Phoenix, Arizona. I was hired fully remote, and it would feel to me like I were a lesser class of employee if those in the Chicago area were regularly incentivized with free lunches when there’s no pragmatic way for me to partake. Luckily, our office uses a system where everyone gets the same amount of delivery credit when we have whole-office lunches, which allows all of us to feel included. 

Beyond incentives, though, is the actual work being done, and this is where I think some teams struggle. Especially when it comes to meetings, the experience of the remote attendee is often an afterthought. This can take the forms of whiteboarding (literally writing on a whiteboard off-camera in the room), crosstalk or side discussions that aren’t in listening range of the microphone, or showing something on a screen physically that’s not present virtually.

It’s not just “don’t punish your remote team members for being remote,” you’re actually hurting the organization as a whole. Presumably every member of the team was hired with an eye to what they can bring to the table; excluding them, or not giving them the full information, hurts everyone involved. 

And technological solutions for remote workers will benefit in-person workers as well! Talking into the microphone during meetings can help someone with cochlear implants hear better in the room just as much as it’ll help me sitting in my garage office 1200 miles away. Same goes for whiteboarding - having a Google Jam (is that where they’re called anymore? Bring back the Wave!) on their screen means my wife can actually follow along; if they have to read a whiteboard from even 14 feet away, they’ll lose track of what’s going on in the meeting.

Taking the time to plan for how the remote attendee’s experience helps everyone, and it’s not terribly difficult to do. You can even see it for yourself by simply attending the meeting from another room to give you perspective and help troubleshoot any issues. Part and parcel of this, of course, is investing in the tools necessary to make sure everyone can interact and collaborate on the same level.

It’s not all about managers/employers, though! Remote employees tend to think that remote work is just like being in the office, only they don’t have the commute. And while that’s true to some extent, there’s another crucial aspect that many of them are missing: Communication. 

You have to communicate early and often when you’re remote for the simple reason that no one can come check up on you. No one can look over your shoulder to see if you’re struggling, no one knows intuitively what your workload looks like if you’re overloaded. Similarly, you don’t know what impacts your coworker’s commit is going to have unless you ask them. There are any number of tools and video sharing apps and all that, but the upshot is you actually have to make focused efforts to use them to make sure everyone’s rowing in the same direction.

Remote work: good for employees, good for employers. Good for business.

Finally, let’s talk diversity. Commonly abbreviated DEI, or DEIB, diversity, equity, inclusion and belonging has sort of morphed from “let’s make sure our workforce looks diverse” to “let’s make sure people of different backgrounds feel like they have a place here.”

And that’s because DEIB should be a culture, not an initiative. At the start, we talked about silos vs. intersectionality. This might start with a one-off committee, or an exec hire, but true DEIB is about your entire culture. Just like remote work can’t just be HR’s problem, and AI decisions shouldn’t be made solely by the finance team, DEIB needs to come from the entire organization.

I actually like the addition of the B to DEI because Belonging is a pretty good shorthand for what we’ve been discussing throughout. People who are temporarily or permanently disabled are provided the accommodations they need to succeed and thrive; programmers aren’t worried AI is going to be used to replace them, but instead given to them as a tool to increase their productivity. Remote workers feel like the company values them even in a different state.

DEIB should encompass all those things, but it can’t be left up to just a committee or an exec or even a department to account for it. It all falls on all of us.

And I specifically don’t want to leave out the traditional aspects of diversity, especially in tech culture. Minorities of all kinds – women, nonbinary folks, other gender identities, those of different sexual orientations, non-white racial backgrounds – are underrepresented in our industry, and it’s important that we keep up the work required to make sure that everyone is given the same access and opportunities. 

It’s good for business, too! Having a diverse array of perspectives as you develop products will give you ideas or user stories or parameters a non-diverse group might never have thought of. We keep hearing stories about VR headsets that clearly weren’t designed for people with long hair, or facial recognition algorithms that only work for those with lighter skin tones. If your product serves everybody, your product will be used by more people. That’s basic math!

Recent court rulings have put a damper on what used to be the standard for diversity, a “quota” of either applicants or hires meeting certain criteria. And look, if your organization was hiring just to meet a metric, you didn’t have true diversity. Quotas don’t create a culture of inclusion, so them going away shouldn’t cause that culture to dissipate, either. Seek out diverse upstreams for your hiring pipeline, ensure you’re not just tapping the same sources. I promise you, that investment will provide a return.

Say it with me: DEIB is good for employees, good for employers, and it’s good for business.

TLDR: Have empathy. Make sure you consider all aspects of decisions before you make them, because very often taking the personhood of the other party into account is actually the best business move as well. 

And with all of these, please note that when I say these are good for employees, good for employers and, especially, “good for business” requires these ideas to be executed well. Doing it right means taking as many of these factors into account as you can. This is where holistic tech comes in as our overarching concept.

• When it comes to accessibility, the more you have, the more customers you can reach and the more options you give them. With a long lens, that tends to mean you wind up with more money. 

• When you’re considering applications for artificial intelligence, try to keep its influence to advice rather than making decisions, and consider the work that would need to be done in order to implement the solution without AI – if it’s not work you’re willing to do, is it worth doing at all, AI or no? 

• With remote work, you need to invest the time and resources to ensure your remote employees have the tools to communicate, while employees need to invest the time and energy to actually communicate.

• Finally, diversity and belonging are about your culture, not a committee or a quota. Invest in it, and you’ll reap rewards.

Today I want to talk about data transfer objects, a software pattern you can use to keep your code better structured and metaphorically coherent.

I’ll define those terms a little better, but first I want to start with a conceptual analogy.

It is a simple truth that, no matter whether you focus on the frontend, the backend or the whole stack, everyone hates CSS.

I kid, but also, I don’t.

CSS is probably among the most reviled of technologies we have to use all the time. The syntax and structure of CSS seems almost intentionally designed to make it difficult to translate from concept to “code,” even simple things. Ask anyone who’s tried to center a div.

And there are all sorts of good historical reasons why CSS is the way it is, but most developers find it extremely frustrating to work with. It’s why we have libraries and frameworks like Tailwind. And Bulma. And Bootstrap. And Material. And all the other tools we use that try their hardest to make sure you never have to write actual while still reaping the benefits of a presentation layer separate from content.

And we welcome these tools, because it means you don’t need to understand the vagaries of CSS in order to get what you want. It’s about developer experience, making it easier on developers to translate their ideas into code.

And in the same way we have tools that cajole CSS into giving us what we want, I want to talk about a pattern that allows you to not worry about anything other than your end goal when you’re building out the internals of your application. It’s a tool that can help you stay in the logical flow of your application, making it easier to puzzle through and communicate about the code you’re writing, both to yourself and others. I’m talking about DTOs.

DTOs

So what is a DTO? Very simply, a data transfer object is a pure, structured data object - that is, an object with properties but no methods. The entire point of the DTO is to make sure that you’re only sending or receiving exactly the data you need to accomplish a given function or task - no more, no less. And you can be assured that your data is exactly the right shape, because it adheres to a specific schema.

And as the “transfer” part of the name implies, a DTO is most useful when you’re transferring data between two points. The title refers to one of the more common exchanges, when you’re sending data between front- and back-end nodes, but there are lots of other scenarios where DTOs come in handy.

Sending just the right amount of data between modules within your application, or consuming data from different sources that use different schemas, are just some of those.

I will note there is literature that suggests the person who coined the term, Martin Fowler, believes that you should not have DTOs except when making remote calls. He’s entitled to his opinion (of which he has many), but I like to reuse concepts where appropriate for consistency and maintainability.

The DTO is one of my go-to patterns, and I regularly implement it for both internal and external use. I’m also aware most people already know what pure data objects are. I’m not pretending we’re inventing the wheel here - the value comes in how they’re applied, systematically.

Advantages

1. For DTOs are a systematic approach to managing how your data flows through and between different parts of your application as well as external data stores.

2. Properly and consistently applied, DTOs can help you maintain what I call metaphorical coherence in your app. This is the idea that the names of objects in your code are the same names exposed on the user-facing side of your application.

Most often, this comes up when we’re discussing domain language - that is, your subject-matter-specific terms (or jargon, as the case may be).

I can’t tell you the number of times I’ve had to actively work out whether a class with the name of “post” refers to a blog entry, or the action of publishing an entry, or a location where someone is stationed. Or whether “class” refers to a template for object creation, a group of children, or one’s social credibility. DTOs can help you keep things organized in your head, and establish a common vernacular between engineering and sales and support and even end-users.

It may not seem like much, but that level of clarity makes talking and reasoning about your application so much easier because you don’t have to jump through mental hoops to understand the specific concept you’re trying to reference.

3. DTOs also help increase type clarity. If you’re at a shop that writes Typescript with “any” as the type for everything, you have my sympathies, and also stop it. DTOs might be the tool you can wield to get your project to start to use proper typing, because you can define exactly what data’s coming into your application, as well as morphing it into whatever shape you need it to be on the other end.

4. Finally, DTOs can help you keep your code as modular as possible by narrowing down the data each section needs to work with. By avoiding tight coupling, we can both minimize side effects and better set up the code for potential reuse.

And, as a bonus mix of points two and four, when you integrated with an external source, DTOs can help you maintain your internal metaphors while still taking advantage of code or data external to your system.

To finish off our quick definition of terms, a reminder that PascalCase is where all the words are jammed together with the first letter of each word capitalized; camelCase is the same except the very first letter is lowercase; and snake case is all lowercase letters joined by underscores.

This is important for our first example.

Use-case 1: FE/BE naming conflicts

The first real-world use-case we’ll look at is what was printed on the box when you bought this talk. That is, when your backend and frontend don’t speak the same language, and have different customs they expect the other to adhere to.

Trying to jam them together is about as effective as when an American has trouble ordering food at a restaurant in Paris and compensates by yelling louder.

In this example, we have a PHP backend talking to a Typescript frontend.

I apologize for those who don’t know one or both languages. For what it’s worth, we’ll try to keep the code as simple as possible to follow, with little-to-no language-specific knowledge required. In good news, DTOs are entirely language agnostic, as we’ll see as we go along.

Backend

class User { public function __construct( public int $id, public string $full_name, public string $email_address, public string $avatar_url ){} }

Per PSR-12, which is the coding standard for PHP, class cases must be in PascalCase, method names must be implemented in camelCase. However, the guide “intentionally avoids any recommendation” as to styling for property names, instead just choosing “consistency.”

Very useful for a style guide!

As you can see, the project we’re working with uses snake case for its property names, to be consistent with its database structure.

Frontend

`class User { userId: number; fullName: string; emailAddress: string; avatarImageUrl: string;

load: (userId: number) => {/* load from DB /}; save: () => {/ persist */}; }`

But Typescript (for the most part, there’s not really an “official” style guide in the same manner but most your Google, your Microsofts, your Facebooks tend to agree) that you should be using camelCase for your variable names.

I realize this may sound nit-picky or like small potatoes to those of used to working as solo devs or on smaller teams, but as organizations scales up consistency and parallelism in your code is vital to making sure both that your code and data have good interoperability, as well as ensuring devs can be moved around without losing significant chunks of time simply to reteach themselves style.

Now, you can just choose one of those naming schemes to be consistent across the frontend and backend, and outright ignore one of the style standards.

Because now your project asking one set of your developers to context-switch specifically for this application. It also makes your code harder to share (unless you adopt this convention-breaking in your extended cinematic code universe). You’ve also probably killed a big rule in your linter, which you now have to customize in all implementations.

OR, we can just use DTOs.

Now, I don’t have a generic preference whether the DTO is implemented on the front- or the back-end — that determination has more to do with your architecture and organizational structure than anything else.

Who owns the contracts in your backend/frontend exchange is probably going to be the biggest determiner - whichever side controls it, the other is probably writing the DTO. Though if you’re consuming an external data source, you’re going to be writing that DTO on the frontend.

Where possible, I prefer to send the cleanest, least amount of data required from my backend, so for our first example we’ll start there. Because we’re writing the DTO in the backend, the data we send needs to conform to the schema the frontend expects - in this instance, Typescript’s camel case.

Backend

class UserDTO { public function __construct( public int $userId, public string $fullName, public string $emailAddress, public string $avatarImageUrl ) {} }

That was easy, right? We just create a data object that uses the naming conventions we’re adopting for sharing data. But of course, we have to get our User model into the DTO. This brings me to the second aspect of DTOs, the secret sauce - the translators.

Translators

function user_to_user_dto(User $user): UserDTO { return new UserDTO( $user->id, $user->full_name, $user->email_address, $user->avatar_url ); }

Very simply, a translator is the function (and it should be no more than one function per level of DTO) that takes your original, nonstandard data and jams it into the DTO format.

Translators get called (and DTOs are created) at points of ingress and egress. Whether that’s internal or external, the point at which a data exchange is made is when a translator is run and a DTO appears – which side of the exchange is up to your implementation. You may also, as the next example shows, just want to include the translator as part of the DTO.

Using a static create method allows us to keep everything nice and contained, with a single call to the class.

`class UserDTO { public function __construct( public int $userId, public string $fullName, public string $emailAddress, public string $avatarImageUrl ) {}

public static function from_user(User $user): UserDTO { return new self( $user->id, $user->full_name, $user->email_address, $user->avatar_url ); } }

$userDto = UserDTO::from_user($user);`

I should note we’re using extremely simplistic base models in these examples. Often, something as essential as the user model is going to have a number of different methods and properties that should never get exposed to the frontend.

While you could do all of this through customizing the serialization method for your object. I would consider that to be a distinction in implementation rather than strategy.

An additional benefit of going the separate DTO route is you now have an explicitly defined model for what the frontend should expect. Now, your FE/BE contract testing can use the definition rather than exposing or digging out the results of your serialization method.

So that’s a basic backend DTO - great for when you control the data that’s being exposed to one or potentially multiple clients, using a different data schema.

Please bear with me - I know this probably seems simplistic, but we’re about to get into the really useful stuff. We gotta lay the groundwork first.

Frontend

Let’s back up and talk about another case - when you don’t control the backend. Now, we need to write the DTO on the frontend.

First we have our original frontend user model.

`class User { userId: number; fullName: string; emailAddress: string; avatarImageUrl: string;

load: (userId: number) => {/* load from DB /}; save: () => {/ persist */}; }`

Here is the data we get from the backend, which I classify as a Response, for organizational purposes. This is to differentiate it from a Payload, which data you send to the API (which we’ll get into those later).

interface UserResponse { id: number; full_name: string; email_address: string; avatar_url: string; }

You’ll note, again, because we don’t control the structure used by the backend, this response uses snake case.

So we need to define our DTO, and then translate from the response.

Translators

You’ll notice the DTO looks basically the same as when we did it on the backend.

interface UserDTO { userId: number; fullName: string; emailAddress: string; avatarImageUrl: string; }

But it's in the translator you can now see some of the extra utility this pattern offers.

const translateUserResponseToUserDTO = (response: UserResponse): UserDTO => ({ userId: response.id, fullName: response.full_name, emailAddress: response.email_address, avatarImageUrl: response.avatar_url });

When we translate the response, we can change the names of the parameters before they ever enter the frontend system. This allows us to maintain our metaphorical coherence within the application, and shield our frontend developers from old/bad/outdated/legacy code on the backend.

Another nice thing about using DTOs in the frontend, regardless of where they come from, is they provide us with a narrow data object we can use to pass to other areas of the application that don’t need to care about the methods of our user object.

DTOs work great in these cases because they allow you to remove the possibility of other modules causing unintended consequences.

Notice that while the User object has load and save methods, our DTO just has the properties. Any modules we pass our data object are literally incapable of propagating manipulations they might make, inadvertently or otherwise. Can’t make a save call if the object doesn’t have a save method.

Use-case 2: Metaphorically incompatible systems

For our second use-case, let’s talk real-world implementation. In this scenario, we want to join up two systems that, metaphorically, do not understand one another.

Magazine publisher

• Has custom backend system (magazines)

• Wants to explore new segment (books)

• Doesn’t want to build a whole new system

I worked with a client, let’s say they’re a magazine publisher. Magazines are a dying art, you understand, so they want to test the waters of publishing books.

But you can’t just build a whole new app and infrastructure for an untested new business model. Their custom backend system was set up to store data for magazines, but they wanted to explore the world of novels. I was asked them build out that Minimum Viable Product.

Existing structure

`interface Author { name: string; bio: string; }

interface Article { title: string; author: Author; content: string; }

interface MagazineIssue { title: string; issueNo: number; month: number; year: number; articles: Article[]; }`

This is the structure of the data expected by both the existing front- and back-ends. Because everything’s one word, we don’t even need to worry about incompatible casing.

Naive implementation This new product requires performing a complete overhaul of the metaphor.

`interface Author { name: string; bio: string; }

interface Chapter { title: string; author: Author; content: string; }

interface Book { title: string; issueNo: number; month: number; year: number; articles: Chapter[]; }`

But we are necessarily limited by the backend structure as to how we can persist data.

If we just try to use the existing system as-is, but change the name of the interfaces, it’s going to present a huge mental overhead challenge for everyone in the product stack.

As a developer, you have to remember how all of these structures map together. Each chapter needs to have an author, because that’s the only place we have to store that data. Every book needs to have a month, and a number. But no authors - only chapters have authors.

So we could just use the data structures of the backend and remember what everything maps to. But that’s just asking for trouble down the road, especially when it comes time to onboard new developers. Now, instead of them just learning the system they’re working on, they essentially have to learn the old system as well.

Plus, if (as is certainly the goal) the transition is successful, now their frontend is written in the wrong metaphor, because it’s the wrong domain entirely. When the new backend gets written, we’re going to have to the exact same problem in the opposite direction.

I do want to take a moment to address what is probably obvious – yes, the correct decision would be to build out a small backend that can handle this, but I trust you’ll all believe me when I say that sometimes decisions get made for reasons other than “what makes the most sense for the application’s health or development team’s morale.”

And while you might think that find-and-replace (or IDE-assisted refactoring) will allow you to skirt this issue, please trust me that you’re going to catch 80-90% of cases and spend twice as much time fixing the rest as it would have to write the DTOs in the first place.

Plus, as in this case, your hierarchies don’t always match up properly.

What we ended up building was a DTO-based structure that allowed us to keep metaphorical coherence with books but still use the magazine schema.

Proper implementation

You’ll notice that while our DTO uses the same basic structures (Author, Parts of Work [chapter or article], Work as a Whole [book or magazine]), our hierarchies diverge. Whereas Books have one author, Magazines have none; only Articles do.

The author object is identical from response to DTO.

You’ll also notice we completely ignore properties we don’t care about in our system, like IssueNo.

How do we do this? Translators!

Translating the response

We pass the MagazineResponse in to the BookDTO translator, which then calls the Chapter and Author DTO translators as necessary.

`export const translateMagazineResponseToAnthologyBookDTO = (response: MagazineResponse): AnthologyBookDTO => { const chapters = (response.articles.length > 0) ? response.articles.forEach((article) => translateArticleResponseToChapterDTO(article)) : []; const authors = [ ...new Set( chapters .filter((chapter) => chapter.author) .map((chapter) => chapter.author) ) ]; return {title: response.title, chapters, authors}; };

export const translateArticleResponseToChapterDTO = (response: ArticleResponse): ChapterDTO => ({ title: response.title, content: response.content, author: response.author });`

This is also the first time we’re using one of the really neat features of translators, which is the application of logic. Our first use is really basic, just checking if the Articles response is empty so we don’t try to run our translator against null. This is especially useful if your backend has optional properties, as using logic will be necessary to properly model your data.

But logic can also be used to (wait for it) transform your data when we need to.

Remember, in the magazine metaphor, articles have authors but magazine issues don’t. So when we’re storing book data, we’re going to use their schema by grabbing the author of the first article, if it exists, and assign it as the book’s author. Then, our chapters ignore the author entirely, because it’s not relevant in our domain of fiction books with a single author.

Because the author response is the same as the DTO, we don’t need a translation function. But we do have proper typing so that if either of them changes in the future, it should throw an error and we’ll know we have to go back and add a translation function.

The payload

Of course, this doesn’t do us any good unless we can persist the data to our backend. That’s where our payload translators come in - think of Payloads as DTOs for the anything external to the application.

`interface AuthorPayload name: string; bio: string; }

interface ArticlePayload { title: string; author: Author; content: string; }

interface MagazineIssuePayload { title: string; issueNo: number; month: number; year: number; articles: ArticlePayload[]; }`

For simplicity’s sake we’ll assume our payload structure is the same as our response structure. In the real world, you’d likely have some differences, but even if you don’t it’s important to keep them as separate types. No one wants to prematurely optimize, but keeping the response and payload types separate means a change to one of them will throw a type error if they’re no longer parallel, which you might not notice with a single type.

Translating the payload

`export const translateBookDTOToMagazinePayload = (book: BookDTO): MagazinePayload => ({ title: book.title, articles: (book.chapters.length > 0) ? book.chapters.forEach((chapter) => translateChapterDTOToArticlePayload(chapter, book) : [], issueNo: 0, month: 0, year: 0, });

export const translateChapterDTOToArticlePayload = (chapter: ChapterDTO, book: BookDTO): ArticlePayload => ({ title: chapter.title, author: book.author, content: chapter.content });`

Our translators can be flexible (because we’re the ones writing them), allowing us to pass objects up and down the stack as needed in order to supply the proper data.

Note that we’re just applying the author to every article, because a) there’s no harm in doing so, and b) the system like expects there be an author associated with every article, so we provide one. When we pull it into the frontend, though, we only care about the first article.

We also make sure to fill out the rest of the data structure we don’t care about so the backend accepts our request. There may be actual checks on those numbers, so we might have to use more realistic data, but since we don’t use it in our process, it’s just a question of specific implementation.

So, through the application of ingress and egress translators, we can successfully keep our metaphorical coherence on our frontend while persisting data properly to a backend not configured to the task. All while maintaining type safety. That’s pretty cool.

The single biggest thing I want to impart from this is the flexibility that DTOs offer us.

Use-case 3: Using the smallest amount of data required

When working with legacy systems, I often run into a mismatch of what the frontend expects and what the backend provides; typically, this results in the frontend being flooded an overabundance of data.

These huge data objects wind up getting passed around and used on the frontend because, for example, that’s what represents the user, even if you only need a few properties for any given use-case.

Or, conversely, we have the tiny amount of data we want to change, but the interface is set up expecting the entirety of the gigantic user object. So we wind up creating a big blob of nonsense data, complete with a bunch of null properties and only the specific ones we need filled in. It’s cumbersome and, worse, has to be maintained so that whenever any changes to the user model need to be propagated to your garbage ball, even if those changes don’t touch the data points you care about.

One way to eliminate the data blob is to use DTOs to narrowly define which data points a component or class needs in order to function. This is what I call minimizing touchpoints, referring to places in the codebase that need to be modified when the data structure changes.

In this scenario, we’re building a basic app and we want to display an avatar for a user. We need their name, a picture and a color for their frame.

const george = { id: 303; username: 'georgehernandez'; groups: ['users', 'editor'], sites: ['https://site1.com'], imageLocation: '/assets/uploads/users/gh-133133.jpg'; profile: { firstName: 'George'; lastName: 'Hernandez'; address1: '738 Evergreen Terrace'; address2: ''; city: 'Springfield'; state: 'AX'; country: 'USA'; favoriteColor: '#1a325e'; } }

What we have is their user object, which contains a profile and groups and sites the user is assigned to, in addition to their address and other various info.

Quite obviously, this is a lot more data than we really need - all we care about are three data points.

`class Avatar { private imageUrl: string; private hexColor: string; private name: string;

constructor(user: User) { this.hexColor = user.profile.favoriteColor: this.name = user.profile.firstName

• ' '
• user.profile.lastName; this.imageUrl = user.imageLocation; } }`

  This Avatar class works, technically speaking, but if I’m creating a fake user (say it’s a dating app and we need to make it look like more people are using than actually is the case), I now have to create a bunch of noise to accomplish my goal.

const lucy = { id: 0; username: ''; groups: []; sites: []; profile: { firstName: 'Lucy'; lastName: 'Evans'; address1: ''; address2: ''; city: ''; state: ''; country: ''; } favoriteColor: '#027D01' }

Even if I’m calling from a completely separate database and class, in order to instantiate an avatar I still need to provide the stubs for the User class.

Or we can use DTOs.

`class Avatar { private imageUrl: string; private hexColor: string; private name: string;

constructor(dto: AvatarDTO) { this.hexColor = dto.hexColor: this.name = dto.name; this.imageUrl = dto.imageUrl; } }

interface AvatarDTO { imageUrl: string; hexColor: string; name: string; }

const translateUserToAvatarDTO = (user: User): AvatarDTO => ({ name: [user.profile.firstName, user.profile.lastName].join(' '), imageUrl: user.imageLocation, hexColor: user.profile.favoriteColor });`

By now, the code should look pretty familiar to you. This pattern is really not that difficult once you start to use it - and, I’ll wager, a lot of you are already using it, just not overtly or systematically. The bonus to doing it in a thorough fashion is that refactoring becomes much easier - if the frontend or the backend changes, we have a single point from where the changes emanate, making them much easier to keep track of.

Flexibility

But there’s also flexibility. I got some pushback from implementing the AvatarDTO; after all, there were a bunch of cases already extant where people were passing the user profile, and they didn’t want to go find them. As much as I love clean data, I am a consultant; to assuage them, I modified the code so as to not require extra work (at least, at this juncture).

`class Avatar { private avatarData: AvatarDTO;

constructor(user: User|null, dto?: AvatarDTO) { if (user) { this.avatarData = translateUserToAvatarDTO(user); } else if (dto) { this.avatarData = dto; } } }

new Avatar(george); new Avatar(null, { name: 'Lucy Evans', imageUrl: '/assets/uploads/users/le-319391.jpg', hexColor: '#fc0006' });`

Instead of requiring the AvatarDTO, we still accept the user as the default argument, but you can also pass it null. That way I can pass my avatar DTO where I want to use it, but we take care of the conversion for them where the existing user data is passed in.

Use-case 4: Security

The last use-case I want to talk about is security. I assume some to most of you already get where I’m going with this, but DTOs can provide you with a rock-solid way to ensure you’re only sending data you’re intending to.

Somewhat in the news this month is the Spoutible API breach; if you’ve never heard of it, I’m not surprised. Spoutible a Twitter competitor, notable mostly for its appalling approach to API security.

I do encourage all of you to look this article up on troyhunt.com, as the specifics of what they were exposing are literally unbelievable.

{ err_code: 0, status: 200, user: { id: 23333, username: "badwebsite", fname: "Brad", lname: "Website", about: "The collector of bad website security data", email: 'fake@account.com', ip_address: '0.0.0.0', verified_phone: '333-331-1233', gender: 'X', password: '$2y$10$r1/t9ckASGIXtRDeHPrH/e5bz5YIFabGAVpWYwIYDCsbmpxDZudYG' } }

But for the sake of not spoiling all the good parts, I’ll just show you the first horrifying section of data. For authenticated users, the API appeared to be returning the entire user model - mundane stuff like id, username, a short user description, but also the password hash, verified phone number and gender.

Now, I hope it goes without saying that you should never be sending anything related to user passwords, whether plaintext or hash, from the server to the client. It’s very apparent when Spoutible was building its API that they didn’t consider what data was being returned for requests, merely that the data needed to do whatever task was required. So they were just returning the whole model.

If only they’d used DTOs! I’m not going to dig into the nitty-gritty of what it should have looked like, but I think you can imagine a much more secure response that could have been sent back to the client.

Summing up

If you get in the practice of building DTOs, it’s much easier to keep control of precisely what data is being sent. DTOs not only help keep things uniform and unsurprising on the frontend, they can also help you avoid nasty backend surprises as well.

To sum up our little chat today: DTOs are a great pattern to make sure you’re maintaining structured data as it passes between endpoints.

Different components only have to worry about exactly the data they need, which helps both decrease unintended consequences and decrease the amount of touchpoints in your code you need to deal with when your data structure changes. This, in turn, will help you maintain modular independence for your own code.

It also allows you to confidently write your frontend code in a metaphorically coherent fashion, making it easier to communicate and reason about.

And, you only need to conform your data structure to the backend’s requirements at the points of ingress and egress - Leaving you free to only concern your frontend code with your frontend requirements. You don’t have to be limited by the rigid confines backend’s data schema.

Finally, the regular use of DTOs can help put you in the mindset of vigilance in regard to what data you’re passing between services, without needing to worry that you’re exposing sensitive data due to the careless conjoining of model to API controller.