Go slow

Assigning too many people to a project is a pernicious way of slowing it down. The number of people on a project increase communication and Conway's Law tells us this directly affect complexity.

There are many products being developed and maintained by huge teams, but this requires huge investments in processes and architecture.

Processes like tests and reviews optimize the flow of communication. Anyone who would be misaligned with the vision that is encoded into processes will be stopped and realigned before they can introduce significant chaos.

Software architecture tries to maintain the relation between complexity and time as linear as possible. It attempts to isolate complexity behind boundaries and abstractions to benefit from information hiding.

Break things

The original "Go fast and break things" motto was meant not to let that one customer still using Windows XP from tying you down to .NET 4.0 (I still know of a case like that still in 2022!). It was never meant to let the software degrade into a big ball of mud.

But I'm subverting it for the catchy title. I mean "break things" as in the bugs that come from the increase in complexity. Even the most experienced developers with great processes and architecture will produce more bugs as complexity increases.

Ask developers on a 30 people or more project where the bugs come from and most will tell you that their services are well designed. It's the upstream team that is misusing their API and the downstream team that is putting constraints on their API that are preventing value delivery.

This is perfectly normal, especially when developing interdependent green field components in parallel. The contracts on those APIs need to be negotiated and no amount of overthinking can bypass that experimentation phase for any significantly complex problem. That negotiation builds a common vocabulary and understanding that lets people develop the systems-level thinking required to solve real world problems. I hope to write about how I believe linguistic determinism applies to software design someday...

For the functional aspects of the APIs, the negotiation is quite straightforward. It's the non-functional aspects that slip under the radar and into production issues. Performance/Scalability, error handling, what fields are sensitive data and should not be written in trace logs, etc.

Team sizing

Resource allocation is a complex topic affected by hiring/retention constraints from below as well as strategic initiatives and ROI vs cost of opportunity considerations from above. It is an incredibly complex problem, and every situation is unique.

Yet there are patterns to discern. I've seen many teams of a 1 to 3 people start or take over a project and scale up gradually as they gained new customers and maintain their velocity with great success. I've also seen a few instances of a new 10 or more people team being formed to work on a project and stand still for over a year, hardly delivering any value.

Then a few months into the project, it's time to implement authentication. It takes a few weeks to sync all services and clients.

Are we all good then?

Many months later, the monitoring still looks like this...

You'd think 400 class status codes are client side request problems so why are we monitoring them? The answer is that this is still pre-release. Those would be pretty valuable to find client side errors and report them to the correct team if the monitoring wasn't completely flooded...

Those failed requests are coming from 5-6 clients running on tester VMs or IoT devices and no one on the cloud services team can locate them... An intern that left an application running on some shared infrastructure and went back to school? A tester who forgot about that one off test he did on one of his physical machines?

The request IP is the corporate public address or that of a data center and there is no identifiable information in the payload because privacy protection good practices justly mandate it. And without authentication, there is no way to trace it back to anyone...

Every now and then, one of those is found and shutdown so the number of failed requests dips a little as we see in this graph. It's the little victories...

In this case, it's much worse since the old Proof of Concept client side application had a bug where the backoff mechanism didn't work and it just retried those errors in a tight loop...

Lessons learned

When building an API, before even the first client connects...

• Make sure you can identify your clients even if authentication isn't a requirement.
• Add the client application version in the request header. It will make it so much easier to retire deprecated functionalities.

In any case, next time someone says "We'll do authentication after proving the design", I'll have a war story to link them!

Everything else is transient on this machine, from unpublished commits to investigation files for an on-site issue.

So what makes changing machine complicated is the machine configuration and installed applications. Those need to be backed up somehow and I'll humbly share how I do it. Your mileage may vary.

Changing machines the painful way

I am always surprised to see programmers who will manually install everything they need whenever they get a new machine. I've seen many people postpone machine upgrades for months because of the overhead of transferring to the new machine.

And then when they do switch machines, there is a problem. It usually sounds like "one component is missing in my Visual Studio installation and an F# library won't build on my machine".

With a sufficiently large software, these issues amount to a list that ends up in a knowledge base and tribal knowledge gets built around the solutions. Countless hours are spent investigating these issues multiple times within each silo in the organization. I've even seen a case where the generally accepted solution was to install the company's software, then uninstall it so the dependencies will be present for the local build to succeed!

Simpler microservices and especially containerized development environments will change the game by adding the dependencies of the software under development as part of a source controlled container definition. Yet there will remain a certain number of applications which need to be installed, even if it's just git and some way to run the container. I think solving the problem of actually configuring a development machine in a repeatable manner will remain worthwhile for the foreseeable future.

Yearly is a pain, monthly is a breeze

The problem actually is quite similar to that very common backup strategy where we focus entirely the backup procedure instead of the restore procedure . We're sure we documented everything, but the problems emerge when we try to consume this documentation.

If we all reimaged our development machines every few months, any sane developer would have automated their machine setup. Next thing you'd know, teams would have put the common bits of that automation together in a shared starter package.

And as with automated tests, infrastructure as code and other routine automation, a big part of the value of automating the task doesn't come from the time saved not doing it manually, but from sharing the knowledge in the most unambiguous way possible : Code.

The team's setup script

Part of our machine setup is shared and maintained by the team in a shared version controlled location.

We use Chocolatey to install a few common packages, then install Visual Studio. The script also clones the mono-repo and builds the monolith application, but I'll leave that out. If you're using a more modern containerized multi-repo architecture, the bits of script below are only dependent on the .NET technology stack and git workflow.

Chocolatey and packages

I'll leave the actual installation of Chocolatey to the official documentation since they have changed it over the last year and it will keep up to date longer : https://docs.chocolatey.org/en-us/choco/setup

And what follows is a subset of our agreed upon choice of tools.

choco install git.install --params "/NoAutoCrlf /SChannel /NoShellIntegration /GitOnlyOnPath"
choco install linqpad -y
choco install sysinternals -y
choco install everything -y
choco install sql-server-express -y
choco install sql-server-management-studio -y

Visual Studio

This next script installs Visual Studio 2022.

# Visual Studio 2022
(New-Object System.Net.WebClient).DownloadFile("https://aka.ms/vs/17/release/vs_enterprise.exe", (Get-Item .).FullName + "/vs_enterprise.exe")
Start-Process "vs_enterprise.exe" -ArgumentList "--update --passive --wait" -Wait # Updates the Installer
Start-Process "vs_enterprise.exe" -ArgumentList "--passive --wait" -Wait # Installs Visual Studio 2022 if not present
Start-Process "vs_enterprise.exe" -ArgumentList "update --passive --wait" -Wait # Updates Visual Studio 2022
Remove-Item "vs_enterprise.exe"

And then apply the common .vsconfig file. In our case, the file is stored on a git repository which is cloned by the script.

If Visual Studio was not initially installed, it usually requires a reboot to proceed at this point, but the script is idempotent and can be executed again after the reboot to continue with the setup.

$VSInstallPath = & "${Env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe" -version 17.0 -property installationPath
Start-Process "${Env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vs_installer.exe" -ArgumentList "modify --config ""PathToVsConfig\.vsconfig"" --installPath ""$VSInstallPath""" -Wait

Visual Studio 2022 may have turned 64 bits and moved to "C:/Program Files", but Microsoft guaranteed to maintain the path for vswhere.exe and vs_installer.exe back in Visual Studio 2017, so we can safely hardcode the "Program Files(x86)" path for the foreseeable future.

Personal customization

Now this is where the $PROFILE file comes in for those parts of the setup that are custom to me.

I maintain an idempotent "setup-machine" Powershell function with the following.

Rider

First off, I am a big fan of Jetbrains' Rider. I use it for all of my main C# development and only fallback to Visual Studio for C++ and memory dump investigations.

Rider has a Chocolatey package that is regularly updated, but it doesn't add Rider to the PATH. Also, its installation path changes every minor version, so maintaining any sort of alias is painful. And their direct download link changes every minor version as well.

I currently install Rider using chocolatey and maintain an alias to it manually. I would be grateful for any better solution!

choco install jetbrains-rider -y

Git and pull requests

I use the following script to create Pull Requests for Azure DevOps directly from the command line.

function gitlogin
{
    echo SECRET-PRIVATE-ACCESS-TOKEN | az devops login --organization https://AzureDevopsServerUrl.com/DefaultCollection
}

choco install azure-cli -y
refreshenv

az extension add --name azure-devops

Replace AzureDevOpsServerUrl with the URL to your DevOps Server SECRET-PRIVATE-ACCESS-TOKEN with your own PAT .

Using a PAT is only necessary when using an on-premise Azure DevOps Server. It will also show messages about not being supported, but if you're running a relatively recent version of Azure DevOps Server, it should work fine for the simple use cases.

This line sets up an alias for az repos pr as git pr ( documentation ).

az devops configure --use-git-alias true

The following also makes git push a little easier to use by automatically creating the missing remote branch when pushing a new local branch.

git config --global push.default current

Combining these lets the normal git flow look like this :

gitlogin
git checkout -b user/kcoulombe/MyNewBranch
git add .
git commit -m "A very informative commit message"
git push
git pr create --open

The last command will open a browser tab with the PullRequest open to adjust the description.

More apps

This is my personal preference of tools.

Install-Module -Name posh-git
choco install powershell-core -y
choco install microsoft-windows-terminal -y
choco install paint.net -y
choco install powertoys -y

In order, these were...

1. posh-git : Adds current branch information within the Powershell command prompt.
2. Powershell Core : The next version of Powershell
3. Windows Terminal : Making command line on Windows easier
4. Paint.Net : Mostly just for editing screenshots
5. Windows PowerToys : Power User tools for Windows

I have excluded a few other more niche applications like HxD , PerfView and quite a few internal tools.

I recently had to build an optimized Sudoku puzzle solver for a homework which turned into a contest between students. I was part of a team of three and we competed again six similar teams… And we won! Here, I will discuss the algorithm we used.

The problem context

First, the Sudoku grid is provided as a simple text file like this example.

030605000
600090002
070100006
090000000
810050069
000000080
400003020
900020005
000908030

The position of each digits is just the same as on the Sudoku grid and a zero represents an unknown value (which the algorithm must figure out). The files always ends with an extra line break after the last line and contains no unnecessary white-space. This is important, otherwise the program return an error message. This example maps to the following Sudoku grid.

The time spent to read and parse the problem file is not counted towards the algorithm’s total resolution time.

The algorithm must be implemented in Java and was to be run on a Quad core 3.2 GHz with 4 GB RAM, but we did not take advantage of multithreading because of the thread creation overhead. For example, on my machine, resolving the example Sudoku takes anywhere between 1 and 3 milliseconds while creating a thread and switching to it takes at least 0.5 millisecond. Here is the code used to test the overhead.

final long time = System.nanoTime();

Thread thread = new Thread() {
    @Override
    public void run() {
        System.out.println("end: " + (System.nanoTime() - time));
    }
};

thread.run();

try {
    thread.join();
}
catch (InterruptedException ex) {}

Another aspect of the problem context is very important as it allowed us to forget about an edge case in which our algorithm does not perform well. The assumption is that the Sudoku grid is always set to have only one solution. This makes a common backtracking only algorithms take a lot more time than our solution. Such an algorithm would be better suited to calculate one possible solution for an empty or almost empty board. On the other hand, our algorithm which makes a lot of rules based decisions is unable to efficiently apply rules in such an open-ended scenario and thus spends considerably more time on an empty board than a naive backtracking algorithm does.

Explaining the basic rules of Sudoku is quite outside the scope of this post, but here I will define a few terms for the sake of clarity.

“Sections” are the 3×3 sub-grids (often separated by darker lines on Sudoku boards) “Numbers”, “digits” or “values” are the numerals from 1 through 9 that are placed on the board. On our board, zero means an unknown value. “Square” or “position” represents the location of a digit determined by its x and y coordinates.

Representing the board

We have chosen to represent the board as an 9×9 integer matrix. As we discovered later, this is not the most efficient solution. Java treats a two dimensional array as an array of arrays. This means accessing a cell in a matrix costs twice the time of accessing an element in an array, once for the column and once for the row. Unlike C++ and C#, Java does not support real multidimensional arrays. In light of this, we should have used a single dimension array.

In any case, we used a 9×9 integer matrix to represent the board as in the file, thus zero means the digits at this position is unknown. Accompanying the board is the “allowedValues” matrix. This matrix contains all the possible values for each position in a 9×9 matrix of integer bit fields. For each of these bit field, it is possible to determine if a certain “value” is legal using this calculation.

boolean isValueLegal = ((allowedValues[x][y] & (1 << (value - 1))) != 0);

In practice, we determined that it was more efficient to cache the possible masks into a static array called “allowedBitFields”. I would have thought accessing the value in the array would have taken longer than these mathematical operations, but profiling proved me wrong (likely due to the JIT compiler optimizing behind the scenes since the cache array is final).

Code structure

For the sake of efficiency, we created a single SudokuSolver class with only static private final methods and properties so there is no state for the program to manage (except for “main”, obviously). We also made heavy use of final method arguments and variables whenever possible.

Some loops even declare a final variable to hold the current iterated value if it was used enough for a performance gain to be measurable. Consider this method which applies the correct value for every square which have only one possible value left according to the “allowedValues” matrix.

private final static int moveNothingElseAllowed(final int[][] board,
        final int[][] allowedValues) {

    int moveCount = 0;

    for (int x = 0; x < 9; x++) {
        final int[] allowedValuesRow = allowedValues[x];

        for (int y = 0; y < 9; y++) {
            final int currentAllowedValues = allowedValuesRow[y];

            if (countSetBits(currentAllowedValues) == 1) {
                setValue(board, allowedValues, getLastSetBitIndex(
                        currentAllowedValues), x, y);

                moveCount++;
            }
        }
    }

    return moveCount;
}

The algorithm

The whole magic starts with the “solveBoard” method. It has two types of tools at its disposal to solve a Sudoku puzzle: rules and hypothesis (or brute force). An hypothesis is the equivalent of the common backtracking solution. We set a hypothesis such as the value of a square among the available values and then try to solve the modified board. If we cannot, then that was not a valid move, so we try another instead. If the program tries all the possible values for a square and none succeeds, then it declares the puzzle impossible to crack.

Using hypothesis only is very inefficient. We already know from the way people solve Sudoku puzzles that many moves can be logically deduced from the position. Thus, we implemented four rules that were shown to reduce the time taken to solve most puzzles. These are as follow:

1. No other value is allowed according to the allowed values matrix.
2. A certain value is allowed in no other square in the same section.
3. A certain value is allowed in no other square in the same row or column.
4. A certain value is allowed only on one column or row inside a section, thus we can eliminate this value from that row or column in the other sections.

Rules 2 and 3 are very similar and may benefit from being merged together as a single rule. However, they are not the same as rule 1 which only checks the “allowedValues” matrix to see if a position allows only one value and sets it. It is very fast. Rules 2 and 3 instead goes on all positions on the board, and for every legal value for these positions, checks if any other positions on the same row, column or section allow that value. If no other position the row (for example) allow a certain value, then we can be certain that the position we are testing has that value.

This diagram illustrates rule 2. The small numbers in each square represent the legal values for each position.

In the top left square, there were multiple possible values, but since “2” was allowed in no other square, then we can safely say that “2” belongs to the top left square. The same logic applies to rows and columns and is implemented as rule 3.

As for rule 4, it is the most complex of our rules and has the lowest benefits. Yet, in most Sudoku puzzles, it still improves performance, even if by a lower margin than the other rules. This diagram shows an application of this rule.

As we can see, in the top left section, the value “1” is allowed only in the second column. This means “1” is forced to be in that column, even if we do not know what row it is in. Using this property, we can remove “1” from the possible values in the same column in other sections. This rule does not let us set values directly. It simply supports the first three rules.

The program runs these four rules in a loop. We have witnessed that, whenever an iteration places less than four values, then it is more efficient to attempt a hypothesis than to continue using the rules. After doing a hypothesis, the program calls the “solveBoard” method recursively and runs the rules on the “test” board.

The legal stuff and acknowledgements

As everything provided on this site, you are free to do whatever you want with it. You don’t even have to mention me (although I’ll appreciate it). However, I’ll ask for one thing in return… If it doesn’t work, don’t sue me! Really, I provide no implied warranty, guarantee of fitness for any purpose or whatsoever. If you have any issue using it, post a comment and I’ll help you out because I’m a nice guy (!), but unless you’re willing to pay me for it, I won’t work for you… probably (I’m a really nice guy).

Also, thanks to my teammates on this project: Mathieu Lacasse-Potvin and Michael Badeau.

The program was part of an assignment for the class LOG320 at École de Technologie Supérieure in Montréal during summer 2010.

The code

The original version of the code is now available on Github : SudokuSolverV1

An updated version of this algorithm is upcoming...

The main blogging platform even back then was WordPress and I never had much interest in Front-End development, so the barest minimum was good enough with me.

The not so good old days

Back in 2010, bare minimum still meant stitching DNS with a WebHost with a WordPress installation with a Plugin selection... SSL/TLS certificates meant spending as much as the hosting cost and having to manually renew certificates using an obscure command line every two years, so... I didn't need no encryption!

Also, Virtual Private Servers (basically Virtual Machines) were expensive, so the cheapest solution was actually Shared Hosting : hundreds of user accounts on a Linux box each separated by only permissions and eventually cgroups against bad neighbors problems.

So I wrote a few blog posts, a few of which kept getting traffic even 10 years later, but...

What killed it

Then I went from consultant to full time dev with a teaching side gig and a now 2 years old daughter. Life happened. Blogging happened no longer. And all along, there was this very annoying WordPress installation and its cocktail of plugins that I had to go and update regularly.

So when my web host WebFaction got bought by GoDaddy and told me they couldn't migrate my account, I started looking around. But then, that WordPress backup format kind of locks me in to WordPress, doesn't it?

What now

These days there are plenty of SaaS offerings for blogging platforms. But if I'm going to convert those old blog posts to a new format, I'd rather it was the last time. So I want a standard format.

My perfect blogging platform would be a SaaS offering that dynamically loads up LaTex files from a Github repository. However, I figured building my own blogging engine just to recover those old blog posts was turning into a pretty long yak shaving chain...

So I settled with HashNode. It has great reviews, uses Markdown and free is a price you can't beat.

I'll be reposting a bunch of my old blog posts here in the coming weeks.