Luis Orduz - Engineering

Domains of engineers and users

2022-04-30T00:00:00-05:00

I finished my last post by mentioning how refactoring can help us achieve a code base where discussing the code and discussing the domain model can be analogous or even entail similar discussions. That's a topic that deserves a deeper looking into.

Experts on different domains

Imagine this conversation:

End user/Product owner: "We'd like for the URL of the Foo to have the provided identifier instead of these random characters."

Engineer: "Oh, by default the framework adds a UUID as the primary key, which is also used for URLs; we'll update it."

End user/Product owner: "Is that a front-end change or a back-end change?"

Now, now, that's an exaggeration as I've worked with plenty of users and product people who were well versed enough on technical details—often because they had to pick up the terminology—to understand what the developer was talking about in that exchange, but I hope this gets the idea across: Too often, there's a gap in shared terminology that hinders the communication between engineers and users.

And here's an example in the other direction:

End user/Product owner: "We need the aggregated results of the 'Foo' to be calculated from the monthly 'bar' instead of the weekly 'bar' from now on."

Engineer: "Understood, I'll add it to the backlog and change it as soon as possible."

Engineer (later, to another engineer or the tech lead): "Where the hell do the aggregated results value come from? Is that in the database? How is it called there? And what even is bar?"

Now that one is no exaggeration; I've had that exact conversation over the years, multiple times. And I've been on both sides of that last question too. Of course, I've also been in projects where there was enough rapport between the engineering and product teams for the engineer to simply ask the product owner right away... But the problem still persists, only in the form of periodic repetitive conversations instead of the latency caused by the developer looking for someone who understands.

The point is that engineers are experts on the domain of software engineering, while users are experts on their own domain, which they're hoping it gets easier to do with the application/platform/system they bought/subscribed to, or hired the engineers to create/maintain.

Code and stories

Let's imagine that the user/product owner comes at us with the following requirement:

As a user, I want to store my "Foo" with the "jon", "doe", "bar" and "baz" values. With the understanding that "baz" is seven times "bar".

After the appropriate back and forth, we come to the following user story:

Store Foo: Create a new model in cool_framework used to store the periodic Foo of the user. The fields are jon (unique string), doe (integer, 100 by default), and bar (float, cannot be empty); baz will be dynamically generated from bar. A UUID field will be included as primary key by the framework.

This user story is of course lacking a way for users to actually submit the data they want to store¹, but it already has more than enough to show how the mismatch in communication starts: The user/product owner could very well ask "what the hell is a UUID?"²

I should point out that I am in no way condemning this approach. It's perfectly fine and how around ninety percent of the projects I've worked on have looked.

But now let's look at the code that fulfills the user story:

from cool_framework import models


class Foo(models.Model):
  jon = models.StringField(unique=True)
  doe = models.IntegerField(default=100)
  bar = models.FloatField(nullable=False)

  @models.computed_property
  def baz(self):
    return self.bar * 7

Again, this is perfectly fine; I currently work in projects whose code looks like this (with a billion more fields and methods, of course), and there aren't problems, usually.

The thing is while this code can trivially be discused among engineers, it can't be discussed by or with users. By this point, both groups are essentially speaking different languages: The users talk about whatever Foo is, how the "jon" group performed this week, when the "baz" could change, etc.; the engineers, conversely, are talking about fields, properties, tables, migrations, etc.

And this works! Most of my experience has had this kind of separation and the teams work and the product is delivered. But what if the language gap could be narrowed?

A shared language

Let's rephrase that story:

Store Foo: Create a class/type/struct³ Foo where the user can store multiple instances of the following values: jon (type ProvidedIdentifier), doe (type UserEstimation), and bar (type PeriodicResults); baz (type AggregatedResults) is calculated from bar.

I just made up those types, but what's important here is that we're to assume that those types mean something to the users. Whatever is in "jon", the users usually call it the "provided identifier" of whatever Foo is; they know that "bar" is the "periodic results" of whatever it is that the users do. Ditto for the other fields.

With that understanding, let's rewrite the code to start illustrating why representing the domain more explicitly is important:

from decimal import Decimal


class ProvidedIdentifier(string):
  class Meta:
    unique = True

class UserEstimation(int):
  class Meta:
    default = 100

class PeriodicResults(Decimal):
  class Meta:
    nullable = False

class AggregatedResults(Decimal):
  FACTOR: int = 7

  def __new__(self, pr: PeriodicResults):
    return super().__new__(self, pr * self.FACTOR)


class Foo:
  jon: ProvidedIdentifier
  doe: UserEstimation
  bar: PeriodicResults

  @property
  def baz(self) -> AggregatedResults:
    return AggregatedResults(self.bar)

Is the code more verbose? In total, absolutely⁴ but it too became clearer about the what we're dealing with: We can assume that the signatures of those classes came from discussions with the users, where they described what that data means in their business and how it's supposed to behave. Sure, doe is still a string, bar a floating point value, etcetera, but now the developers can talk about the code in terms similar to what the users talk about, which also means they'll talk to the users in those terms too.

Let's reimage the conversations from the start, but now with the assumption that all the code is like this:

Well, for starters the first conversation no longer makes sense as another win from these changes is that we decoupled our business logic from whatever framework we're using⁵... As long as the framework treats the business code as source of truth anyway.

As for the second,

End user/Product owner: "We need the aggregated results of the 'Foo' to be calculated from the monthly 'bar' instead of the weekly 'bar' from now on."

Engineer: "Just to make sure, baz is the aggregated results and bar is the periodic results, correct?"

End user/Product owner: "Correct!"

Engineer: "Cool, I'll get it done."

Let's be real, it's likely that the engineer still doesn't know what the hell these "results" are or why the users care about them, but now at least it might be easier to pinpoint what needs to change without having to find someone who might be more familiar with the domain.

Finally, it bears mentioning that this approach doesn't solve all communication problems: The engineers will never be domain experts on the users' domain, so there will always be questions, specially when creating new features as the developers will need to ask what the new classes/types will need for fields and metadata⁶; as for the users/product owners, the decoupling with the framework and the increasing reliance on native language constructs means that the number of technical details they need to have an idea about becomes smaller, but not zero.

Regardless, giving the team (both involved users and developers) a way to more fluidly discuss the product is a huge step forward in my book.

A caveat

All of that is pretty nice; however, the boilerplate the users don't care about still needs to exist somewhere. Maybe abstracted and hidden away in specific modules or even internal libraries that have their own repos, but finding and correcting leaky abstractions is a neverending battle, so those discussions that ideally should be among engineers only might find their way in conversations with the users.

Imagine a world where we didn't need to abstract the boilerplate for every project; that it simply didn't exist. In that ideal world, we could just drop some native classes/structs/etc that contain all the business logic and only the business logic to some tool(s), and the tool would automagically take care of all the wiring needed for the application to reach the end user.

I've given a lot of thought to such an idea, and maybe it's a pipe dream, but stranger things happen in this industry all the time.

Let's say a form in some web site/application. ↩
Or, for that matter "what are strings and floats?" You never know! ↩
Or whatever construct your favorite language uses to group data, if any! ↩
But hey, the code of the Foo class itself got simpler. ↩
Yay for clean architecture! ↩
In other words, some conversations for the developers to make the business rules clearer, which they then can express in the code! ↩

Practical refactoring: Abstractions

2022-03-31T00:00:00-05:00

In my last post, we did a basic rundown of a very convoluted short algorithm to make more explicit what was actually happening in it. That by itself goes a long way in improving how readable the code is, and thus makes it easier to maintain. I've seen small improvements like that be welcome enthusiastically among different teams, but we can go further.

I remember a project I worked on where there was basically no separation of concerns between request handling boilerplate, database connection boilerplate and actual business logic; everything was handled within the same functions. It was a nightmare. I was hired to create some new APIs, but it took me just a week of trying to create new handlers like that to decide that such was no way to live. I took it upon myself to refactor that code. It bears reasserting that when refactoring, it's good—and often enough—to first reach for the lowest hanging fruit and in that code the easiest improvement was, of course, separating code that dealt with different things into different functions, and calling those new functions from the old ones.

Now, the small script we're going through does really only one thing, but that doesn't mean we can't divide some responsibilities by way of abstracting away some code not directly related to that one thing. If we do this, we improve the code in at least three ways:

The algorithm itself becomes more immediately obvious; it's easier to understand what the code does.
By abstracting the supporting/boilerplate code, we make it possible to reuse those same structures somewhere else.
This gives us a opportunity to bring the code in step with the domain: using names specific to what we're doing instead of talking exclusively about basic data types.

The abstractions

Something I didn't mention in my last post is that the small method we're refactoring is actually part of a basic genetic algorith I implemented for fun. Well, genetic algorithms deal with populations, so what if instead dealing with "lists" of "strings", we create an actual Population class/data type that does what we need our populations to do:

class Population(list):
    def __iter__(self) -> Iterable[Self]:
        return iter(self[x::2] for x in range(2))

Not complicated, now we know that a population is a list, but one that when iterated just produces two sublists: the two halves of the original. We can improve it further, but this is enough for what we need¹.

We have our population, but our genetic algorithm is not of random things; it specifically looks for individuals that match a given root individual, with the purpose of each generation to be closer to that individual.

Let's give it a try:

class Individual(str):
    def __and__(self, other: Self):
        return sum(ch_s == ch_o for ch_s, ch_o in zip(self, other))

For our "model"² we know that we need a specific type of string that when compared with another will return the number of identical characters. So we do just that: create a str subclass that can "intersect" with other strings of the same type, and give a numeric value representing how well they match.

With these two abstractions alone, our method improves considerably:

def get_best_matches(self) -> WinnerPair:
    # self.population: Population[Individual]
    # self.root: Individual
    winners = WinnerPair()
    for half in self.population:
        winner = max(half, key=lambda ind: ind & self.root)
        winners.append(winner)
    return winners

I like it. We get each member of the winner pair from half of the population, and notice how the code doesn't show things we don't need to know to understand the steps:

How do we get each half of the population? That doesn't matter for understanding the algorithm; we just need to know that we're getting a half. How that half is gathered is up to the implementtion in Population which we can check if we need to, or we could change it if we have to: Like instead of appending the odd-positioned items to a half and the even-positioned to the other half, we could just literally split it at the central index.
To get the winner we're comparing how well is the intersection between each individual and the root (self.root, renamed from self.word³, in this case). How is that intersection calculated/found? That's entirely an implementation detail, which again we can check and/or change in Individual if we have to. We could even make individuals a different type instead of strings, make a comparison appropriate for that type, and we wouldn't need to change anything in this method.

Notice that, at the end of my previous post, I said that we could use abstractions instead of relying on comments to tell us what the data types are supposed to represent. There are still comments in this piece of code... but that comment is just to note what abstractions we're using, and is there just for description purposes. In the full code it would be completely redundant, since the types of self.population and self.root would already be defined somewhere else. Likely at the top of the class this method belongs to.

Nonetheless, we could still make those comments explicit in the code⁴ by way of, say, making them arguments to the method. However, that would no longer be a refactoring⁵... But we can cheat a little bit:

def get_winners(
  population: Population[Individual], root: Individual
) -> WinnerPair:
    winners = WinnerPair()
    for half in population:
        winner = max(half, key=lambda ind: ind & root)
        winners.append(winner)
    return winners

def get_best_matches(self) -> WinnerPair:
    return get_winners(self.population, self.root)

Basically, we create a new function that isn't necessarily tied to a class (note the lack of self) and we call that function from our method. This gives us the potential advantage of reusing that very same winner finding logic somewhere else if the opportunity arises.

Hindsight

So there you have it, we improved the code, considerably, just by moving a few lines around and making what we're doing and using more explicit. It goes without saying that this would help us communicate between the developers and with the product team (if any) much more easily.

Ideally, in a good codebase, discussing the code and discussing the business model would entail very similar expressions, as the code would just be a literal (with caveats of course) representation of that business model. Hopefully this post showed how such a thing could be achieved.

This is a subject I really, really like and I'm hoping to keep writing about it.

When refactoring, it's easy to get lost making increasingly minute improvements. We always gotta remember that premature optimization is the root of all evil and that abstractions can easily become unnecessary indirections. This is a fine line, and sometimes I don't spot it, so it's important to remember that the goal is to make the code more maintainable instead of perfect from the get go (which is impossible anyway). In an actual product, it also helps to remember what I think as the rule zero of Software Engineering: "fight for the users". If something we're doing won't improve the users' experience in any meaningful way, it might be best left alone. ↩
This genetic algorithm in particular just iterates over populations that increasingly resemble the root word. ↩
In my opinion, the naming of the root individual as self.word was a code smell born out of the usage of basic data types instead of proper abstractions. It was a way to hint at the developer that the comparison was between strings. Using proper types/classes, we no longer need to do that. ↩
After all, "code is for what, tests are for why and comments are for jokes" (which is also a joke... or is it?) ↩
If "improve the easiest thing first" is rule one of refactoring, then "do not, under any circumstance, change the contract" is rule zero of refactoring. To put it another way: if we have a good test suite, a proper refactoring shouldn't change the result of any test. Anything other than that is an actual change in the behavior of the application/system, and it better have been agreed upon. ↩

Practical refactoring: 'clever' code

2022-02-28T00:00:00-05:00

Look at this code

def get_best_matches(self) -> WinnerPair:
    """
    Divide population in half.
    Pick the word closest to the matching word in each half.
    """
    return WinnerPair(*map(
        lambda population_half: max(
            population_half,
            default="",
            key=lambda word: sum(
                a == b for a, b in zip(word, self.word)
            )
        ),
        (self.population[::2], self.population[1::2])
    ))

I'm not gonna deny it, I liked writing it, I like that it is technically a single function call¹, the usage of lambdas and the built-in Python functions used for handling, well, functions and iterables. What can I say? It makes me feel "clever" because technically it's code that requires certain level of familiarity with the language.

It's also a total mess. I literally spent an entire afternoon explaining this "short" piece of code to an experienced engineer who had already invested a few months getting familiar with Python.

This code is me at my most self-indulgent and I'm well aware I would never have written this outside of a prototype meant only for me to play around. Code like this is not meant to live in a system worked at by more than one developer. It'd be a nightmare to maintain, as only the one who wrote it could possibly understand it. Hell, I wrote this and I had to struggle a bit puzzling what it actually did.

In short, this code is ripe for improvement, which is exactly what I'm gonna do.

Unclear iterations

The first thing that jumps at me upon seeing this code is that there are three nested iterations in it, but it's very difficult to tell which one is which or where each one ends. An easy first fix is then relying less on built-in functions and making the iterations more explicit via for statements.

def get_best_matches(self) -> WinnerPair:
    # Get the words closest to the target in each half of the population
    winners = WinnerPair()
    for population in (self.population[x::2] for x in range(2)):
        scores = []
        for word in population:
            similarity = 0
            for char_word, char_target in zip(word, self.word):
                similarity += chard_word == char_target
            scores.append((word, score))
        winner = max(scores, key=lambda score: score[1])
        winners.append(winner[0])
    return winners

Looks quite different, doesn't it? It would seem to someone completely unfamiliar with Python that I changed more than replacing the function calls with fors, but that's truly all I did:

The first function was map which is doing something to both halves of the population.
The second function was max which is picking the highest according to something in each word in the population.
The third function was sum which is actually calculating that previous "something": In this case, how similar is the current word with the target word.

I then reused max, but it's now clearer what maximum value of what it's being picked. I will not lie: I hesitated with leaving the sum as it was as I felt that with the other replacements it was clear enough, but then I saw the opportunity to further clarify that we were comparing the current word with the target word. On the other hand, I did leave zip as it was, as that one is clear enough to me.²

Aside: Someone with some familiarity with algorithm analysis might see three nested for loops and pale at the "cubic" complexity, but this function isn't iterating over the population input (let's call it "n") multiple times. It's instead iterating only once over the total characters input (let's say "m"). In short: This iteration only visits each character in the population once.

The word list (but not each word) is visited twice because of the max function, but since two is a constant, it remains of linear complexity.

Anyhow, those "straightfoward" changes are enough to at least being able to tell what the function is doing line by line, but it can be better.

As it is, we're doing a bunch of operations over basic data types with a comment explaining what those data types are supposed to represent. We could instead explicitly define our own abstractions over those data types and let those abstractions tell us what they can or can't do, or how they should be used.

But I feel like that is interesting enough for its own post, so see you in the next part!

Well, a function call wrapped in a class instantiation, but who's nitpicking? ↩
I firmly think that Software Engineering is engineering, and I have no problem calling myself "engineer" over, say, "craftsman", but there is a subjective factor to some decisions. ↩

Creating stories from requirements

2020-02-01T00:00:00-05:00

A few weeks ago, one of my best friends came to me with an idea for an application. Unlike most (if not all) of my ideas, this one I actually believe to have potential so I asked him to prepare a list of requirements we could use to at least have a rough goal to aim for with this application. He delivered, and I was presented a good enough list... And I did nothing with it, I've been letting it sit there, all unfulfilled potential. But that changes now!

I've been working as a software engineer for a while and that experience, the manifesto, what I remember from my college studies and what I've read regarding system design all point to the same conclusion: Having a fixed list of tasks and dedicating too much time to it is pointless because requirements change like waves in the sea.

However, I still feel it's important to have a set of stories to work on, both to have a general idea of what the system is and how it's supposed to work, as well as for documenting progress; nothing kills motivation like the feeling you're not actually advancing towards your goal. Besides, we control the requirements in this case and, even though waves are fickle, the global conveyor belt is still a thing. Much like with that overly elaborate metaphor, my goal is not to describe how every single detail in the application is supposed to work, but more a general description of each functionality, and general details on how to implement it.

With that in mind, I'll show how I approached turning that requirements list written in the "as a user" format into stories that developers can actually work on. I'll forgo things like "story points" or "acceptance criteria" because not only are those restrictive, I'm doing this as a developer for developers—all two of us—and what I care about is what to do and how to do it. And that's also the reason I won't be using stuff like certain project management tool that most software development teams are familiar with whose name I won't mention, at the risk of summoning giant lizards.¹

In fact, this is a good point to mention sourcehut, Drew DeVault's cool set of tools for creating software that's very reminiscent of how the biggest open source projects are maintained. I'm going to be creating these stories using sourcehut's todo, which is to say: simple issues.

Without further ado, let's take one of the requirements as example, one that is universal enough that I can use without risking IP: user accounts. This is an abbreviation of my friend's requirement:

As a user I can create an account in the application, with an email or using social networks (an email should be sent introducing the platform upon registration).

As mentioned, my goal with the stories is knowing what to do and how to do it, from a developer's perspective. Part by part, this is the architecture² that this requirement defines:

The application has a domain, and this domain includes the entity User because that's what we want to create.
The application has some sort of data layer that the application interfaces with to store this User data.
The application has adapters that interface with third parties to retrieve the user data, social networks in this case.
The application has the use case "create user" that is called in two different ways: using email or using the adapters mentioned above.
The application has an interface that the user employs to send the email and other data, or to trigger the retrieval from third parties.
The application has a second use case, which is to send an automated email to the, well, email provided by the user upon successful creation.

With that rough outline, we have an idea for two or three stories, because at this stage is better to restrict stories to the number of use cases, or to the number of times all use cases are instantiated across the application.³ Of course, stories ultimately can involve editing use cases too; the point is that we should make the stories about the application business rules whenever possible.

The stories mentioned here are deliberately vague on the tech stack because I want this to be applicable for as many developers as possible.

Create User from email: Create an Use Case that accepts raw data as well as a data repository⁴ and creates an instance of the User domain model using the raw data. It then passes this data to the repository for creation.; Create an interface adapter⁵ that receives data (including email) submitted by an user and passes it to the aforementioned Use Case, alongside the database repository.; Create a view⁶ that allows the user to submit this data to the adapter.⁷
Create User from third party: Create an interface adapter that gathers data (including email) about an user sent from third parties⁸. It passes this data to the create user Use Case alongside the database repository.⁹; Create a view that allows the user to go through the third party communication cycle.
Send email on user creation: Edit Create User Use case so that, upon successful storage, calls a new Use Case.; This new Use Case is in charge of sending the body and addressee of an email to an email sender interface adapter.¹⁰; Create an interface adapter that uses the data produced by the use case and sends it to an external queue¹¹, in the form of a message, that should send the email.

These stories are somewhat vague, and that is deliberate. I don't want to restrict—and not only because I would be restricting myself—and details of implementation are what code reviews and tests are for. The one exception I make on not specifying implementation is regarding the tech stack itself: it's such a big decision that all developers are benefitted if the stack is clear¹².

The stories should only define a rough end goal, and I believe the ones I wrote here achieve that. Anyhow, I feel that this is a good first step and a general description of how we'll be working on this project.

This is going to be part of a series, an idea born from a great article on blogging that I read the other day, which inspired me to write more and gave me the idea of how to actually write: I'm going to push myself into working on this application (and refactoring an older one, with another friend) so that I have material to write for this blog, and I can use the will to write for the blog as incentive to work on those projects, killing two birds with one stone.

IMPORTANT: Everything I'm writing in these series is my interpretation and general idea of a good process. Anything (or everything) I write might be entirely wrong and, in such event, I encourage you to correct me in the comments.

Just FYI, I don't hate it, it can be a great tool, as long as it isn't drowned in the bastardization of scrum, which I don't hate either... as long as it's used as a guideline instead of a forced two-week waterfall grind. ↩
I'm trying to describe this within the terms of the clean architecture. Results may vary. ↩
And debatably at any stage. I've seen projects where stories are created for everything, even changing the color of a button to a slightly differen shade of blue. YMMV on advantages and disadvantages of that practice. ↩
I'm a postgreSQL user. ↩
Probably a function that would retrieve the data in a given format, create a domain entity object from it that would then pass this object into the database adapter, probably a SQLAlchemy model. This function itself would be called from a flask endpoint. ↩
I've worked as a full stack web developer almost exclusively, so in here I'm thinking of a form, created either from a JS framework or just a simple web form. ↩
Of course, nobody is forced to use a client-server system, but it's what I'll use. ↩
Probably something like flask-dance. ↩
Remember not to pollute the business rules or the entities with data or steps specific to any third party. ↩
Remember not to break flow of control, these business rules don't care how the database signals success or how the sender adapter sends the message. But that's an implementation detail that should be discussed in review. ↩
Probably rabbitmq through Celery. ↩
Everything beyond that (meaning how the stack is used) is outside the scope of the story. That's for tests and reviews. ↩