Category: Uncategorized

A Trap For New Engineers

There is a common tendency when learning to program for people to move from language to language, or from tiny project to tiny project.

I think this is dangerous because oftentimes a particular concept in programming, say for instance orthogonality, doesn’t really sink in until after you’ve taken a project where you made a mistake around that concept far enough to see the problems surface naturally as a result of increased scale or complexity.

I mention orthogonality specifically, because I sat in a two separate programming interviews and explained what orthogonality meant in programming, and why it was important. I did a good enough job in the interviews that I was offered a job at that company (which I ultimately didn’t accept).

I then turned around a few weeks later in my day job and violated that principle by thinking that I could leverage an existing option being passed into a function to do something more than what it had been originally envisioned to do.

By the time the problem I’d created surfaced in my code a few weeks after that, I’d forgotten some of the context around what I’d done, and had a couple of days of tricky debugging before I realized my mistake.

Needless to say, I’m much less likely to make that mistake in the future, which illustrates one of the reasons that senior software engineers are paid so much more than junior software engineers.

There is real value in either having learned why not to do certain things, or having made the mistakes and learned from them. It it is hard to overstate how valuable it is not to make a mistake that sends an entire team of highly-paid engineers down a bad path that burns up weeks or months of effort.

It’s not just paying the salaries around that wasted effort that is costly. In some circumstances, a mistake like that can result in a competitor being first to market with a feature or product with a knock-on loss of opportunities that can value in the tens of millions of dollars.

All of which brings me to the point of this post. When working on a side project, I think that it’s important to pick something with enough complexity and scale that some of your design mistakes have a chance to surface. It’s better to learn the principles and never make the mistakes, but the more of those mistakes you surface and recognize, the more value you’ll ultimately be able to contribute to whatever organization you belong to.

The Hard Parts

I remember talking to a friend who worked at a company that did contract programming for people. The gist of the conversation was that they got a signficant amount of work from people who had some kind of application that had been programed by someone’s sibling/cousin/best friend. The app worked, and was starting to bring in money, but as the business continued to grow, the app wasn’t scaling the way that it needed to be.

The take away was that companies were engaging his firm because the writer of the original app had hit the limit of what they could do, either because they had other commitments, or because they had hit the limits of their expertise as a software engineer.

At the time, I was lucky if I could just get the right onto my screen when I was programming.

In the years since, that story has come to represent what I think of as the ‘hard’ parts of programming.

The initial roll out of a project is hard. You’re making dozens, hundreds even, of decisions about how to structure things. If you have deep subject matter expertise in the area that you’re trying to automate, then your decisions are more likely to be correct than if you don’t but even then, some of your assumptions may be drastically wrong once the app is in the hand of the users.

Those wrong assumptions can then result in needing to put in a lot of work to fundamentally re-structure the application.

Additionally, when first creating any kind of software application, there are a whole host of things that you need to get working together which aren’t a factor for someone coming along later in the process. It’s one thing to access the database via code in an app that’s been running for a year or two using methods that work hundreds of times over in other spots in the code base. It’s a whole different kettle of fish setting that database up and writing those methods which will allow those who come along later to access the database.

So, setting things up is hard. You can get around that to some extent by using open source tools that packing different components together, but I still have mad respect for those people who get the initial version of a project up and running.

The other thing that I’m realizing is really, really, hard is extending an existing project in ways that minimize the complexity so that you don’t cut your throat making future changes harder and harder to bring about.

Props to all of the engineers out there who are doing that or who are removing unnecessary complexity from an existing system.

I suspect it’s not the end-all-be-all, but in this area, I got a lot out of Code Simplicity by Max Kanat-Alexander (affiliate link).

On Linting

I still remember the first time I heard about the concept of linting. My initial response was unfavorable to the concept.

That was partially because the person speaking up in favor of linting didn’t have the best reputation at the organization where I worked. It was partially because my mentor at the time seemed to be anti linting, and it was partially because the concept of something telling me that I was doing a ‘bad’ job writing code sounded as pleasant as going in for unnecessary dental work.

Fast forward several years, and now I’ve been working for a few months in a codebase that has linting enabled.

My takeaway now? Why wouldn’t you want an automated tool that automatically highlights stuff that you forgot to address while you were working?

Sure, in any big organization, there are likely going to be some linting rules that you think take things too far, but the value to having something help debug your work even before it has a chance to break is incredible.

I guess, for me, it’s a lot like Typescript. There’s a reason that big organizations tend to use linting. It’s not because they are dinosaurs who can’t change and who like to move slowly. It’s because they’ve realized that it’s a great way to increase developer efficiency by reducing the number of bugs that get out the door.

Postgres ‘pg’ NPM Connection Pools

I’m currently in the middle of re-working my side project to use the pg npm package. As part of that, I now have to worry about managing the connection pool, which is something that was handled behind the scenes with the ORM that I was using previously, the ORM that I was having problems getting to play nicely with Typescript.

Online, I saw a recommendation to create a pool like so:

import { Pool } from 'pg';

const pool = new Pool({
  host: 'localhost',
  user: 'database-user',
  max: 20,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
})

And then to execute database operations like so:

const wrapperFunc = async (): Promise<void> => {
  const client = await pool.connect(); // create a connection

  try {

    const query = `
    CREATE TABLE users (
        email varchar,
        firstName varchar,
        lastName varchar,
        age int
    );
    `;

    await client.query(query); // use the connection
  } catch (error) {
    // Deal with your catch error
  } finally {
    client.release(); // release the connection
  }
}

Releasing the connection is important because otherwise you end up with a bunch of zombie connections that your database is waiting on, but which aren’t live anymore. Eventually, if you don’t release the connections, then your database hits its max connection count and it starts refusing new connections.

I had the bright idea of changing up my code to look more like this, which I thought would work just as well:

const wrapperFunc = async (): Promise<void> => {
  const client = await pool.connect(); // create a connection

  try {

    const query = `
    CREATE TABLE users (
        email varchar,
        firstName varchar,
        lastName varchar,
        age int
    );
    `;

    await client.query(query); // use the connection
    client.release(); // release the connection
    return
  } catch (error) {
    client.release(); // release the connection
    // Deal with your catch error
  }
}

I thought that would still result in the client being released regardless of whether we threw an error or not, but quickly started getting errors while trying to run my unit tests.

I can’t explain why yet, but for some reason, a client.release() position like I’ve got above doesn’t release the clients like it should, so I’ll be sticking with putting my client.release() statements in my finally block going forward.

Thoughts on Typescript

I once worked at a company where there was an ongoing rivalry among the developers between Java and Javascript/NodeJS.

To be honest, I didn’t know enough back in those days to have anything remotely resembling an informed opinion. I’m not sure that much has changed there, but one of the things that surprised me a bit was the way that one of the Java proponents would dismiss Typescript as just being a way to try and make Javascript more like Java.

I ended up ultimately choosing to focus my learning efforts more an Javascript/NodeJS simply because I liked the idea of not having to master Java for the back end and Javascript for the front end, but I always figured that someone who was in favor of strong typing would be in favor of bringing stronger typing to Javascript.

I’ve now been working with Typescript for a few months. There is still obviously a ton that I need to get my arms around with Typescript, but my initial impression has largely been favorable for all of the reasons that people typically praise it for.

I do occasionally miss being able to do crazy things with my variables, but by and large those types of things are likely to end up bitting me at a future point. By giving that up, I get instant feedback from my code editor when I mis-remember the type of a variable and try to use it in a way that is going to get me in trouble the first time I do a test run of my new code.

Getting things setup with some of the packages I was previously using on my side project proved to be beyond by current abilities, but once the setup is done, I seem to be developing at a faster rate simply because I’m chasing down fewer typing errors.

Typescript doesn’t magically make NodeJS something it isn’t. You still have to worry about blocking the event loop, but all in all (for what it’s worth), I think it’s a helpful extension to the base language.

Logging Adventures

Is ServiceNow you don’t really have a debugger that you can use to trace what’s happening in your code. Instead, most people spend a lot of time using logging statements.

I’ve gotten into the habit of using JSON.stringify(object, null, 2) to convert objects to something that can be logged out. I ran into an occasion recently where my logging statements were logging out an object that was missing a bunch of information on it.

Pro tip: (/sarcasm) Maps don’t convert and print via a JSON.stringfy, so if you have an object that contains a map, you’ll have to try something else :).

On Staffing

This is a potentially touchy subject.

Every time I’ve ever ‘settled’ when hiring someone onto my team, I’ve ended up wishing that I hadn’t settled.

This has held true both when acting as the hiring manager when I was a controller, and when I’ve had input into hiring decisions not as the hiring manager.

Obviously, no candidate is perfect because no person is perfect. Beyond that, you’re unlikely to get a near-perfect senior developer for what the market is paying a junior developer, and if you did, that person is going to leave if they ever figure out just how far under market you’ve got them, so you’re depending on them continuing to be uninformed, which isn’t a great place to put yourself.

Setting aside all of that, my strategy is that you should always do your very best to hire someone whom you’re really excited about, someone who is bright, driven, and conscientious.

If your budget is on the lower side, then that often means that you’re hiring someone with a bit less experience than you would have ideally wanted to bring in. That can be a tough combination because it means that you’re training someone to do a job that they are probably going to be outgrowing in a year or two.

Ideally, your department or team is growing too, and you can move them on to bigger and better things in your organization, but even if that isn’t the case, someone with the right mindset generally seems to still find a way to make it worth having trained them before they end up moving on, and just because they move on doesn’t mean that they won’t be back, or that you won’t work with them again at some other point.

Coming at things from the job seeker side, being turned down for a position that you’re sure you would have been great at is frustrating and incredibly disheartening. I get that–I’ve lived that and the struggle is real.

Coming at things from the hiring manager side, I get that the cost of a bad hire is really high. You’re probably going to spend a lot of time trying to figure out if they are hitting the performance levels they should or not (the more complicated their job and skill set, generally the harder it is to determine that), then you’ve got to spend time coaching them, then you’ve got to go through the painful process of firing them, and then you’ve got to interview candidates again.

Given all of that, most organizations and hiring managers tend to be willing to pass on candidates that would ‘probably’ be awesome at the role and wait until they find one that they are ‘certain’ will be awesome at the role.

It’s definitely not a perfect system, and I understand why people (on both sides) get frustrated.

Sometimes it helps me to imagine a world where hiring managers don’t do that. Instead, they say “I’m going to fill this role in the next 10 days. I don’t care how many applicants I do or don’t interview during that time. I’ll give it 10 days and then just hire the best of the bunch…or the one applicant who is minimally qualified.”

Having seen some bad hires in my day, that sentiment would give me the creepy crawlies.

As I described above. You have to be reasonable in your expectations. You have to calibrate your expectations against the market you’re in (the pool of talent may be deep and vast, or a bit lacking), and what you can afford to pay.

That calibrating process can take some time, but I think you have to respect that process of calibrating yourself, making your exceptions reasonable, and then finding someone who (inside the limitations you’re working with) makes you really excited to be bringing them onto your team.

Obviously, emergencies arise, and we don’t work in a perfect world, but not respecting that process would generally represent a failure of leadership.

Getters and Setters

I’m in the middle of a Typescript class. It’s been very informative, however in a recent video the teacher created a class and then proceeded to reference one of the attributes of the class rather than creating a getter in order to get the value.

I proceeded to adjust my notes so that the class had a getter method and then updated the calling code to use the getter method.

All of which made me laugh a bit. It’s a minor point, and I get the fact that the teacher is balancing how far to get into any one given topic, but it’s amusing to me that just a few years ago I was watching another developer program and I couldn’t see the point to the getters and setters he was building into his class.

For anyone that isn’t convinced, getters and setters give you a layer of virtualization which means that you’re buying yourself the ability to change the inner workings of your class if necessary without blowing up the rest of your code.

Given some of my recent experiences, I’m a big fan of adding virtualization layers like that anywhere it makes sense. It saves a lot of time and effort in the long run.

Database Denormalization

I’ve been working on a side project for quite a while now. It’s been a great learning experience, and I hope to eventually have something that I can monetize.

One of the initial decisions resolved around what to use for my database solution.

I ended up in a normalized, SQL database solution for various reasons, but I flirted quite heavily with the idea of a No-SQL database solution.

As I was building out my SQL tables, though I came to the conclusion that a fully normalized approach was the wrong answer.

I have certain tables which will see record creates take place, and some edits to those records across certain fields, but there are some fields that will never be edited.

Under a very common use-case, I need data from what would be two different tables under a fully-normalized approach. That would mean that every time a request was made to that endpoint, the database would have to make a join on those two tables.

I’m still getting my feet under me as it relates to the correct way to scale a system, but my sense is that your database generally ends up being the constraint that is the hardest to scale, so I’ve tried to build with an aim towards minimizing database load whenever possible.

A de-normalized scheme generally seems to trade additional space consumption for reduced load as a result of fewer joins, so that’s exactly what I ended up doing.

I took a few fields from one table and am also storing them on the table that my endpoint is primarily reading from. The fields in question aren’t being updated after the initial write, so I’m not introducing load to keep the two tables in sync, and the elimination of those joins should drastically push back the point where I have to worry about database load being a problem.