Queue Creation Options

I should be sleeping right now, but I really wanted to figure out what the best option is for creating a queue in Javascript.

It’s not super relevant since it can’t be used as a queue, but for reference purposes, I ran a loop for 100,000,000 (100 million) iterations, pushing values 0-99,999,999 into an array and then popping the values back out. On my system that took ~4.5 seconds.

Trying to place those same values into a linked list that had both a head and a tail attribute resulted in the process crashing after 32 seconds. Dropping the items being stored down by an order of magnitude to 10 million resulted in the process taking ~2 seconds.

I tried storing just the value 1 in each of the nodes, hoping that would stop the thread from crashing, but that didn’t make any kind of difference–we still crashed at around 32 seconds.

Also for comparison purposes, I pushed a million records (0-999,999) into and array and then used shift to pull them all back out. This is an order of magnitude less than the 10 million records that I put into my linked list and pulled back out in ~2 seconds. I let the algorithm run for more 200 seconds before canceling it.

That’s indicative that using push and shift is at least 3 orders of magnitude slower than a linked list at those kinds of sizes.

If you push and item and then immediately shift it back off of an array 100,000,000 times, the process takes less than 2 seconds, so the issue is with shift having to run through the existing items in the array.

If your queue is always going to be fairly small, then it’s entirely possible that push and shift is the way to go, however, if your queue is going to get very large, then it appears that a linked list would be a better option–as long as it doesn’t get so large that you run into other problems.

It might be worth trying out a map. A map can be iterated through, and it stores the order that things were added into it, but my best guess is that the fact that it is storing the order keys were inserted is going to work against it and we’ll have the same kind of issue that we have with shift and unshift on arrays.

Array.Shift speed

I got to thinking tonight that before blindly assuming that unshift in JavaScript was slow, but shift was fast, that I should double check that.

As it turns out, shift is roughly as slow as unshift on arrays, which means I need to re-think how I would implement a queue in JavasScript. Just using array.push() and array.shift() works, but is unnecessarily slow.

My initial thought is to use a linked list that has both a head and a tail pointer. I’ll need to test out the speed of that kind of construct against the speed of array.push() and array.shift(). It’s possible that there are some other factors that would slow a linked list down in comparison for use as a queue.

More on that once I’m done testing.

Blank Rows In Excel & Transform Maps

I recently had occasion to do some more work with transform maps in ServiceNow, and I couldn’t remember if I needed to worry about blank cells from my Excel document over-writing data in ServiceNow, so I did some experimenting.

I can confirm that blank cells in the Excel doc being imported and transformed do not overwrite existing data in the relevant field in ServiceNow.

Now it’s documented and I can refer back here the next time I have this particular question.

Unshift in Javascript

For a very long time now, I had been under the understanding that since Javascript arrays were just a kind of object, that array.unshift() wasn’t any more costly that array.push().

Today I finally did some testing and found out that I was very much wrong. Now you know and won’t experience the same performance issues that I ran into.

Life Getting Hectic

Apologies all around for missing my normal posting day. Things have gotten especially hectic, and something (actually, many things) had to give.

I’ll attempt to get some additional posts up to document some of what I’ve been learning recently, but my posting schedule will probably be off for a little while.

Maybe only check in once every couple of weeks. I’ll let you all know when I’m back to posting more regularly.

Merge Sort

As discussed in my last post, I’ve been spending time learning new algorithms, and reminding myself how to use ones that I haven’t used in years.

Here is the evolution of my Merge Sort algorithm:

I did some searching online looking for a high-level discussion of what a merge sort was, and it’s pros and cons. tldr: It trades the usage of more space for better performance than Quick Sort.

Then I found this example by Tim Han.

I scanned through that quickly, and then worked on other stuff for several days. When I came back to take my first shot at writing my merge sort (crazily enough, this isn’t one that I encountered during my year and a half of computer science classes many, many years ago), all I remembered was that you divide the array, and then you return a merge call of those array halves being recursively divided. Oh, and something about there being a single leftover data point that needed tacked onto something else at the end of the merge.

Here is my first compilable attempt:

//Mergesort. Works by splitting array into half until getting arrays of size 1
//Then merges the arrays back together, sorting as they are merged.

const mergeSort = (array) => {
    const length = array.length;

    if(length == 1) {
        return array;
    }

    const midpoint = Math.floor(length / 2);

    let left = array.slice(0,midpoint);
    let right = array.slice(midpoint);

    return merge(mergeSort(left), mergeSort(right));
}

const merge = (array1, array2) => {
    if(!array2.length) {
        return array1;
    }

    let mergedArray = [];

    while(array1.length && array2.length) {
        if(array1[0] <= array2[0]) {
            mergedArray.push(array1.shift());
        } else {
            mergedArray.push(array2.shift());
        }
    }

    if(!array1.length) {
        while(array2.length) {
            mergedArray.push(array2.shift());
        }
    }

    if(!array2.length) {
        while(array1.length) {
            mergedArray.push(array1.shift());
        }
    }

    return mergedArray;
}

const rawArray = [9,1,2,3,4,5,7,88,77,5,5,66,3,8,7,49,59,62,73,24,81,67,32,5,3,2];

let sorted = mergeSort(rawArray);
console.log(sorted);

As a side note, the reason that the version I wrote on paper didn't work was two-fold.
1. I just merged left and right inside of "mergeSort" rather than merging what was returned from calling mergeSort on each of those two array halves.
2. I used a Math.ceil rather than a Math.floor to generate the midpoint, which resulted in an array of length two not being properly split.

While writing my initial algorithm, the merge is the part that gave me pause, primarily, because I couldn't see how we could consistently guarantee that once we were done with the while loop that there would just be one item in one of the arrays. I could envision a scenario (for instance in an already-sorted array), where array1 gets emptied before any items are taken from array2

After a few minutes, I just went with my gut, and created something that would deal with emptying an unknown number of items out of whichever of the arrays still had items in it.

Once I'd confirmed that my code was compiling and properly sorting the array, I went back to Tim Han's algorithm and compared it to what I'd come up with.

The biggest thing that stuck out to me was the fact that he was using pointers rather than shifting items off of the two arrays passed into the merge function.

The more experienced of you are already nodding your heads, and don't need me to tell you that his approach is superior, but in my defense, I'm used to thinking in terms of JavaScript arrays, which are different than how most languages define an array.

In Java, an array is created with a pre-determined size and you can't grow or shrink that array after it is created. Instead, you have to create a new array of the desired size and copy the desired items from the first array into the new array.

The fact that Java arrays are all a single contiguous block of memory likely makes them faster than JavaScript arrays, which as I understand it are just objects that can be added to and removed from without creating a new object and destroying the old object.

Still, even running this in JavaScript, there isn't any reason that we have to shorten array1 and array2 while growing the mergedArray--unless maybe memory was becoming a constraint and we needed to free up the space being occupied by array1 and array2 simultaneously to growing the mergedArray.

Still, though, that seems unlikely. Generally, you'd just run things on a machine with more memory, or use a streaming type approach to the algorithm.

The next thing that I noticed was that I didn't need my if(!array2.length) statement. Ultimately, if array2 (or array1 for that matter) were empty, it's just going to skip the while loop and tack the other array onto the mergedArray.

Lastly, the use of the concat function rather than a while loop was nice and slick. It's another reminder that I need to spend some time just reading through JavaScript's built-in methods and arrays. I know a bunch of them by memory, but there are still tons of them that I don't use because I don't realize that they are out there.

Here's my updated version:

//Mergesort. Works by splitting array into half until getting arrays of size 1
//Then merges the arrays back together, sorting as they are merged.
const mergeSort = (array) => {
    const length = array.length;

    if(length == 1) {
        return array; //split down as far as we can go. Time to return so we can merge.
    }

    const midpoint = Math.floor(length / 2);

    let left = array.slice(0,midpoint);
    let right = array.slice(midpoint);

    return merge(mergeSort(left), mergeSort(right)); //split left and right again and the merge what comes back
}

const merge = (left, right) => { //merge the two arrays together into a sorted array
    let mergedArray = [];
    let leftIndex = 0;
    let rightIndex = 0;

    while(leftIndex < left.length && rightIndex < right.length) {
        //compare the two and put the smallest into mergedArray until we get to the end of one of them
        if(left[leftIndex] <= right[rightIndex]) {
            mergedArray.push(left[leftIndex]);
            leftIndex++;
        } else {
            mergedArray.push(right[rightIndex]);
            rightIndex++;
        }
    }

    //either left or right will still have values that can be tacked onto the end of
    //our sorted array
    let leftRemainder = left.slice(leftIndex);
    let rightRemainder = right.slice(rightIndex);

    return mergedArray.concat(leftRemainder).concat(rightRemainder);
}

const rawArray = [9,1,2,3,4,5,7,88,77,5,5,66,3,8,7,49,59,62,73,24,81,67,32,5,3,2];

let sorted = mergeSort(rawArray);
console.log(sorted);

Performance wise, my algorithm is the same now as what Tim did (which makes sense given that I read his algorithm several days before sitting down to do my own). I explicitely broke out the leftRemainder and rightRemainder because I think it's easier to follow what's going on that way.

I did confirm that while Tim's algorithm is great, he was wrong about there being just a single item left over after the while loop in the merge function. If you log leftRemainder and rightRemainder out, you'll see that they can be more than 1 item.

A big thanks to Tim for putting such a clear, well-written algorithm out there!

Route Forward

During my day job, there is rarely any question regarding what I’ll work on next. There may be several stories in a given sprint that haven’t been claimed yet, but my team has experienced some significant turnover over the last 18 months, so I generally just pick the most technically challenging story and move forward with that so that one of the more junior people on the team don’t get stuck with something that they aren’t ready to work yet.

Unfortunately (or fortunately, depending on how you look at things), my choice of what to do during my after hours pursuits are much more open-ended.

In theory, I could do anything. Practically speaking, there is usually a subset of options which are likely to have a higher return on the effort than other options.

In the last year, apart from learning done during work hours, I’ve completed a NodeJS course, done some consulting work, done some work on an email platform (which involved reading a large number of RFC’s in addition to the programming) and completed an Angular course.

If you’d asked me what I was going to work on when I completed my consulting engagement, I would have said that I was going to be working on my email platform for the next several years. Even a few months ago, I would have said that I was quite enjoying my work on the email platform, and that I was going to continue working on it.

That plan was sidelined by a couple of shocks at work, including a story being pulled into a sprint where it looked to me like I was going to need to create a widget from scratch, something that is heavily Angular intensive. Given that I knew very little Angular at the time, that was more than a little alarming, so I decided to take a break from my email platform and get through an Angular class so that I would be prepared the next time that we had something come down the pipe that needed Angular expertise.

Once I finished up with the Angular class, I debated between several different options. I could go back to my email solution. I could go take another class, to either learn a new framework, or even one to just improve my front-end abilities generally. I could go learn more ServiceNow-specific things, or I could spend some time pretending that I was a computer science student again.

I don’t enjoy front-end work as much as I do back-end work, but it was still tempting to sharpen my skills there. Ultimately though, at my current role I don’t do much traditional front-end work with CSS and the like. Additionally, it’s not something that I’m wanting to move into, so it didn’t feel like a great use of my time. I could easily spend a bunch of time getting better with CSS or learning another framework, and then not use what I was learning frequently enough to retain it.

ServiceNow is a great tool, but I’m already familiar with the areas where we spend most of our time at my day job, and I don’t want to spend dozens of hours learning a new module that may or may not get used at my job.

Studying the kinds of algorithms that get studied while pursuing a traditional computer science degree on the other hand feels like the sort of thing that will pay dividends for years to come.

I’m unlikely to ever write a quick sort and use it in production. Frankly, the sorting algorithms built into JavaScript as core parts of the language are more robust and optimized than anything I’m likely to write for production purposes.

The concepts that were used to solve the various problems represented by the algorithms on the other hand are things that I can use over and over again. As I’ve started working through some initial algorithms and data structures, it’s felt a little bit like sitting at the feet of some of the greatest minds of the last several decades, and learning directly from them.

I’m learning a new set of tools, new approaches to solving problems, and blowing the rust off of other tools and concepts that I’ve used in the past, but not recently.

All of which I suppose comes to the point of this post. One of the things that I really love about switching from Accounting to Software Development is the fact that it’s so much easier to make a solid investment in technical skills that make me more valuable than I was before the investment.

When deciding what skill investments to make, it can be tempting to keep chasing shiny new languages or frameworks. There is definitely some benefit to both of those things. Learning Go, which is a synchronous, blocking language help me understand NodeJS (an asynchronous, non-blocking language) at a much deeper level.

Taking an Angular class is useful because using a framework generally allows you to work more quickly than you can without one. Plus it’s something that I use from time to time at my day job.

Paste a certain point though, I think you’re better off working on core competencies. It’s important not to focus all of your time on a dying language or a shrinking platform, but I’m convinced that working on algorithms and data structures–and deepening my understanding of JavaScript/NodeJS is going to be way more valuable to me than learning a second framework or a 5th programming language.

So, when you see more data structure and algorithm type posts in coming weeks, now you know why. And maybe, if you’re just starting out, this post will help you go deeper with regards to your own studies rather than just chasing another shiny new framework or language.

Count Triplets Problem

Every so often, I grab a HackerRank problem in an attempt to learn something new. I recently tried the Count Triplets challenge. As is normal for me with HackerRank problems, the biggest issue was figuring out what the author was after.

Here was my initial code:

function countTriplets(arr, r) {
    let numbers = new Map();

    for(let i=0; i<arr.length; i++) {
        addToHash(numbers, arr, i);  
    }

    let triplets = 0;
    for(let i=0; i<arr.length; i++) {
        let number = arr[i];
        let secondNumber = (number * r);
        let thirdNumber = (number * r * r);

        let secondNumberLocations = numbers.get(secondNumber);
        let thirdNumberLocations = numbers.get(thirdNumber);

        if(secondNumberLocations && thirdNumberLocations) {
            triplets += secondNumberLocations.length * thirdNumberLocations.length;
        }
    }

    return triplets;
}

function addToHash(numbers, arr, index) {
    let locations = numbers.get(arr[index])
        if(!locations) {
            numbers.set(arr[index], [index]);
        } else {
            locations.push(index);
            numbers.set(arr[index], locations);
        }
}

It did exactly what I wanted it to, and returned the number of unique trios of indices that when up by a multiple of ‘r’. Unfortunately, it only passed 8/13 test cases.

As it turned out, they only wanted a match when the numbers in the indices were in order from smallest to largest.

This was my next attempt, which worked, but was too slow to complete one of the tests in the allotted time:

function countTriplets(arr, r) {
    let numbers = new Map();

    let triplets = 0;

    for(let i=0; i<arr.length; i++) {
        const number = arr[i];
        let numberArray = numbers.get(number.toString()); //Number in Map already?

        //Need to get all of the next smallest in sequence already in the Map
        //Lets us count matches and update Map with correct info for number
        let numberFraction = number / r; /*Next smallest in sequence*/
        let numberFractionArray = numbers.get(numberFraction.toString());

        //Do the counting
        if(numberFractionArray) { //potential exists for triplet
            for(let j=0; j<numberFractionArray.length; j++) {
                triplets += numberFractionArray[j].fractionCount;
            }
        }

        //Generate a numberArray object for addition to Map
        let numberArrayObject;
        if(numberFractionArray) {
            numberArrayObject = {
                index: i,
                fractionCount: numberFractionArray.length
            }
        } else {
            numberArrayObject = {
                index: i,
                fractionCount: 0
            }
        }

        //Add current number info to Map so that it can be used by a higher multiple
        if(!numberArray) { //Add for first time
            numbers.set(number.toString(), [numberArrayObject]);
        } else { //Add the new item to the existing array
            numberArray.push(numberArrayObject)
            numbers.set(number.toString(), numberArray);
        }
    }

    return triplets;
}

The above, second version only goes through the initial array one time, but when it finds a number that works as the largest or middle value in the triplet, the execution has to go through the entire array stored on the next number down in the sequence.

This was more or less instinctive for me. I have a tendency to grab hold of any data that comes through under the theory that I’ll probably need it later, but strictly speaking, I don’t need it to comply with the requirements of this particular exercise.

My third version of the code doesn’t try to story any kind of detailed data, instead grabbing summary totals and putting them into the map at each step.

function countTriplets(arr, r) {
    let numbers = new Map();

    let triplets = 0;

    for(let i=0; i<arr.length; i++) {
        const number = arr[i];
        let numberObject = numbers.get(number.toString()); //Number in Map already?

        /* {
            count: x,
            subcount: y
        } */

        //Need to get the value of the next smallest in sequence already in the Map
        //Lets us count matches and update Map with correct info for number
        let numberFraction = number / r; /*Next smallest in sequence*/
        let numberFractionObject = numbers.get(numberFraction.toString());

        //Subcount semi problematic. Need to increment numberObject
        //But only if numberFractionObject exsits
        let additionalSubcount = 0; //Assume 0. Change if numberFractionObject exists

        //Increase Triplets where appropriate
        if(numberFractionObject) {
            triplets += numberFractionObject.subcount;
            additionalSubcount = numberFractionObject.count;
        }

        //Add current number info to Map so that it can be used by a higher multiple
        if(!numberObject) { //Add for first time
            numbers.set(number.toString(), {
                count: 1,
                subcount: additionalSubcount
            });
        } else { //Add the new item to the existing array
            numberObject.count++;
            numberObject.subcount+= additionalSubcount
            numbers.set(number.toString(), numberObject);
        }
    }

    return triplets;
}

My solution is O(n), and the space required is likely to come in around 2n. If there are no duplicates, and no triplets, then it would just be a hash table of size n. If there were no duplicates and everything came back as a triplet, then the size would be 3n because you’d store each item and have two additional data points next to it.

If you have a bunch of duplicates that all come down to the same triplet, then you’d need space of 9 (3 triplets with 3 data points each).

In looking at the editorial on HackerRank and the solution there, the solution requires space of 3n, and they are claiming complexity of O(nlogn), but it looks to me like they are going through the list twice, which would in theory mean that it’s O(2n) which would reduce down to O(n), but my algorithm should be roughly twice as fast.

Table Fields that don’t show in system dictionary (ServiceNow)

I can’t think of a reason why this would or should matter, but it was not at all what I was expecting, so I thought I’d document it in case I need it at some point in the future. Maybe it will prove helpful to someone else.

I recently needed to confirm that iterating through all of the fields from within a server script would really iterate through all of the fields. Much to my surprise, I found out that there are a subset of fields that are on the glide record object (for project tasks), but which don’t show up if I go and look at the table definition in the GUI.

They are:
sys_tags
variable_pool
sys_meta
hierarchical_variables

I just used a for loop to do the iteration. Something like:

for (var field in record) {
   gs.log(field + " : " + record[field]);
}

Finding sys_audit Entries (ServiceNow)

We recently had an issue with something that required running a script to fix some values that had been incorrectly changed. The correct data we needed was in the sys_audit table, but getting to the table to verify that the data really was there turned out to be a miniature adventure all on its own.

The table checkpoints changes to nearly any field an any given form, so it’s huge. My mistake initially was in not thinking about how such a huge table can be so responsive when viewed from a form (in the history/activities section), but so slow when I’m trying to get at the records from a list view.

I’m used to tables in ServiceNow being indexed by one or more date/time fields, but obviously, that’s not how the data is being accessed from the forms.

All indications are that the sys_audit table is indexed by the documentkey field (the sys_id of the record being checkpointed or audited).

In keeping with my theme of also blogging about stuff that I’ll need to lookup at some point in the future, here is the url to get at all of the entries for a given record:

https://[instance].service-now.com/nav_to.do?uri=%2Fsys_audit_list.do%3Fsysparm_view%3D%26sysparm_first_row%3D1%26sysparm_query%3Ddocumentkey%253D[sys_id]%26sysparm_list_mode%3Dgrid%26sysparm_offset%3D

You’ll need to update the [instance] and [sys_id] with the correct fields