When you first start setting up the compute nodes you may have no meaningful infrastructure in place. There are steps to the sequence of bootstrapping from nothing and ending up with the nodes in a consistent state. You won’t be able to use PXE to do network-based installations. You also won’t have anything like an LDAP server to integrate via PAM to handle server authentication and authorization. Down the road maybe, but on day one, no.

Prev: Phase 1: Just a Bunch of Storage

When you begin the process you’ll have placed a bootable ISO image on a USB drive for the operating system you’ve chosen. I would recommend preparing a second USB drive with any firmware updates your particular compute node hardware model may be due. Update the firmware first, and then install the O/S via the ISO image. If you want to be extra careful, make sure you have a copy of the existing firmware version before performing the update, in case you have to revert to your starting point.

Make life easier on yourself and put some visible external labels on those USB sticks. If you have multiple compute nodes to set up, you’ll be following the same process over and over. Start taking notes as you go through the setup the first time.

As the installer runs you’ll be asked a number of questions. Some will relate to disk partitioning, for which the previous article will give you guidance. A few will relate to specifying a non-root user, to complement the root user which is always installed. Once the installation completes, you have the starting point from which you need to bootstrap a little security.

The initial security work won’t be all you’ll ever want to do, not by a long shot, but you need a baseline that will support routine remote access and running Infrastructure as Code (IaC) jobs that will progressively establish the full functionality of your platform. The first pass is only what you can’t or shouldn’t avoid. It is not enough to consider the environment as being secure. You’re establishing connectivity, and as part of that connectivity you will need credentials to support authentication.

Managing Credentials

Credentials like SSH keys, TLS certificates, and GPG keys become your first intellectual property assets that you don’t want to lose. The point of them is that you should be unable to perform operations or connect to systems that use those credentials to control access or certify identity. The more secure your platform, the harder it would be to recover functionality if those credentials were lost.

Preserving credentials is one area of functionality for which you should consider using a remote service vendor. Any passwords, keys, and certificates should be immediately preserved in the remote service. Assuming you are doing your cluster work as a solo experimenter, you won’t need to care about establishing any kind of-group permission structure, which will make things easier.

Credential automation is much tougher, and for “Phase 1” you probably need to be realistic about how much of that you can pull off. As an example I only know of one remote credential manager that has substantial Linux Pluggable Authentication Module (PAM) support — Keeper Security — and largely that is to use their Privileged Access Management product. Yes, I know, two entirely different uses of the acronym PAM, as if computer security wasn’t a strange enough world. I haven’t worked with Keeper myself, but from what I can tell the functionality somewhat overlaps on LDAP.

The Bootstrapping

My initial “good enough for now” bootstrapping is to use Bash scripts that I run on each compute node after the O/S install finishes. The complete process looks like:

1. Attach a monitor and keyboard to the uninitialized box.

2. Install the firmware USB, boot the box, then install any firmware.

3. Install the ISO image USB, boot the box again, then install the O/S (which will request the password for the non-root user it creates).

4. On whatever host you use for other work, such as your laptop, create an SSH key pair that will be used for authenticating as that user. Since “you” are “you” all the time, you use the same key pair for all the compute nodes. NOTE AGAIN, this is not a final security posture for the cluster, we’re just bootstrapping.

5. Save both halves of the SSH key to the remote vendor service you selected for preserving credentials, so you don’t lose them.

6. Configure your network router to recognize the MAC address for the new box and assign a static IP. You aren’t running a big company, I wouldn’t sweat DHCP for the experimentation platform until you find you need it, particularly since ssh would complain on every change to an IP address.

7. Using your working (e.g. laptop) host, connect remotely to the compute node via that static IP to supply the user and password. These are the non-root credentials you specified when the O/S was installing.

8. Copy-paste the bootstrapping Bash script that configures the user account.

9. Run the script. You should have designed it to ask you for the public half of the SSH key pair you created, which it will install in ~/.ssh and update ~/.ssh/authorized_keys to know about the new key.

10. If you have root-level changes to make (e.g. to configure sshd and sudo), copy-paste the Bash script for that.

11. Run the second script via sudo.

12. Don’t logout yet, but start a second terminal session and verify that you can login. If not, you have a mistake somewhere and you want to get that fixed before you lose viable access (else you may have to install the node from scratch).

13. As a final step I would recommend completely power-cycling the node. If you’re going to have any problems, you may as well find out now.

There are assorted refinements you can make to the instructions above, like:

• Remove password access to an account once the public SSH key is in place.

• Have separate accounts for human-user access versus automation-user access (a need that will become clearer in a future article).

• Ensure the ~/.ssh directory and file permissions meet the requirements that sshd will enforce.

• Inform the compute node of its intended host name, as it probably won’t pick that up from the router.

• Make the script behaviors idempotent so that re-running them isn’t harmful. Until you’ve going through the process a couple of times you’ll keep adding tweaks as you figure out exactly what bootstrap steps you need.

The options are only constrained by your Bash skills. Later on we’ll be doing proper IaC automation, so the primary purpose of any bootstrapping is to get us just to that stage.

Script Samples

My own bootstrapping is split into files for particular features, which currently are all for Ubuntu 22.04. Here is an example for ensuring git is installed:

and to install pip for Python 3 (which isn’t automatically installed on Ubuntu 22.04):

and to upgrade the Ansible installation (the default for Ubuntu 22.04 is ancient):

I like putting each feature function in a separate script, because then within a user or root bootstrapping script all I need is:

and then I can invoke the functions that make sense for either a non-root or a root user. The function files are just functions so they don’t take action until they are explicitly called.

The Experimentalist : Phase 1: Credentials Management and Bootstrapping © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Planning for storage in your newly-constructed lab has several elements to it:

• Hardware considerations.

• Filesystem considerations.

• Use-case considerations.

Prev: Phase 1: Just a Bunch of Compute | Next: Phase 1: Credentials Management and Bootstrapping

First a little terminology:

• Drive: a physical device you buy and mount in or attach to a computing node.

• Partition: a subdivision of the space available on the drive. There are two partitioning approaches, with GUID Partition Table (GPT) being what you’ll usually see, but Master Boot Record (MBR) partitions may exist on older systems. When picking compute nodes to buy, I would suggest not getting anything so old it doesn’t support UEFI, and any UEFI system should allow GPT. GPT gives you more flexibility on partition creation and supports larger drives.

• Volume: a logical construct that presents available storage space to the operating system. Historically a volume usually fit within a partition, but Logical Volume Management (LVM) can present a volume that spans partitions and drives. It does this in terms of “extents,” which it uses to associate physical to logical storage.

There is some hand-waving simplification in those explanations as partitions can be primary, extended, or logical, but those distinctions apply to using MBR.

Local Physical Storage Use Cases

Each compute node will have storage needs to account for. For simplicity I’ll assume a Linux variant is your goal.

• EFI System Partition (ESP): This provides space for boot loaders and kernel images. I plan a 1/4gb for this, but if you knew you intended to do a lot of experimenting with different kernels you could make it bigger to house more kernel images.

• Swap Partition: You may read online content or hear discussions that advocate against swap due to performance concerns. As a database guy there are reasons why I disagree with the position. You don’t have to allocate a physical partition as you can add swap files later. I prefer to allocate the partition and size it according to the amount of RAM I installed on the compute node. You could use a smaller amount, but the less you allocate the more you need monitoring on swap activity, which is a discussion for future articles. Obviously if you have more than 64gb of RAM, you won’t want to size swap as the entire memory footprint unless you have a strong reason to support a virtual memory space that large.

• Root Partition: This will contain the volume that the boot loader establishes as your running root filesystem. When you size this allow for the fact that package updates, run-time log files, applications, and the runtime needs of those applications will all require storage.

• Other Linux Partitions: While not required, you may decide that for system stability you want some of the application, app data, and logging activity go to a separate physical partition. That would ensure the O/S itself doesn’t fall over due to the root partition filling up. The likely mount points for a volume would be /var or /opt. If you want both in a common partition, LVM can achieve that.

• Maintenance Partition: Depending on the BIOS features, I’ve found it handy to add a small additional partition to write to. Sometimes when working in the BIOS you can record configuration snapshot images to remember what you changed. This relates to the “Make notes and take pictures“ advice from the previous article. When you set up many nodes, sometimes one or two will not behave consistently with the others, and knowing your BIOS setup without having to boot into it can be useful. Maybe give it 1-2 gig if your BIOS saves config snapshots as actual image-formatted (e.g. BMP, PNG) files. You may find that this needs to be formatted as FAT32, which will be reported by “fsdisk -f” on a UEFI/GPT setup as “Microsoft basic data.”

• Data Partitions: If you know you’ll be working with large volumes of data, it is worth planning data partitions. By having separate data partitions you protect the O/S from data filling up the root filesystem. It’s also the usage pattern that may later motivate you to add additional or replace with larger drives. It’s less disruptive to do that with clearly-segregated data than it usually is for a large filesystem muddling everything together.

An obvious question is whether all these partitions should be on the same drive or across multiple drives. This mostly relates to the root filesystem versus everything else, although for database purists there are other scenarios.

The most direct reason for multiple drives is protecting the ability to boot from the root filesystem. There can be performance benefits to multiple drives, although it is easy to overstate the potential for non-RAID performance parallelism unless you are spec’ing out compute nodes with 2 or more CPUs, 2 or more RAID cards, and NUMA memory configuration to channel memory transfers separately. Write traffic across multiple SSDs may hold up better as a performance story without that level of hardware due to how IOPS could get batch flushed to on-SSD SRAM cache.

In the case of some database architectures there are fault-tolerance scenarios around directing Write-Ahead Log (WAL) to separate disks. Again this story holds up better when you have multiple RAID cards, preferably cards with battery backup for ensuring completed flushes on a power outage. The goal here would be to approximate the theoretical concept “stable storage,” which forms the backbone for how we reason about database transactions and database recovery.

Remote Storage Use Cases

It is important to keep in mind that you’re creating an experimental lab platform. You should be able to set up and tear down compute nodes at will, or alter some aspect of their current application provisioning and configuration.

This will push you towards having externalized storage so that any data you care about will live across those major system alterations.

You are also going to run into cases where you want the same data available across multiple compute nodes. Having multiple nodes mount the same remote volume is useful for allowing tasks to run anywhere and be matched up to the data required.

The straightforward solution to remote storage is Network Attached Storage (NAS). I want to dig into NAS solutions more in later articles, but for now the outline is:

• You can either build your own, or buy a NAS. If you’re thinking about lab creation like a small startup business, you probably want to just buy something so you can focus your time elsewhere. Pick a reliable, established vendor and product line.

• You want your NAS to support the protocols you intend to use for file sharing. The two most likely are Server Message Block (SMB) and Network File System (NFS). There are others, but those are what you’ll typically use. Use the most recent stable version of a protocol available.

• The bulk of the storage should be in a RAID configuration that helps preserve your data. I prefer RAID 6 plus a hot spare, because I’m willing to trade the cost of some storage for increased protection against disk failure. Investigate RAID options and pick the configuration that suits your situation.

There are distributed filesystem technologies like Ceph, but as the discussion here is about “phase 1” planning for the lab platform, I don’t think that worth getting into. It is family of options can be pretty heavy-weight as a learning curve and maintenance responsibility. You may only want to head down that path if one of your goals from your cluster was hands-on experience with distributed filesystems.

Economic Considerations

You have three possible optimization outcomes:

• Low cost.

• High reliability.

• High performance.

You can only pick at most 2 out of the 3 if you push to the extremes, but sometimes you can balance a little in the middle. When it comes to storage, no matter the use case, I see absolutely no win in giving up on high reliability. You’re trying to accomplish things, and having storage fail underneath you does not accomplish things.

That leaves you to decide when to aim for low cost, versus when to emphasize high performance.

I lean towards the following:

• The NAS box itself should be capable of high performance. You already have the handicap of data traveling over the network. Just get a good box in the first place.

• The drives in the NAS are where you have have more flexibility. If you want a lot of storage, you really can’t beat the price point of most good-quality HDDs. If you prefer fast storage then you’ll want to get good-quality SSDs. Your bill and final storage capacity will differ between those choices, just don’t buy junk. This does not mean that you must buy drives marketed as “enterprise grade.” The price points on most of those are stupid. If you have a decent RAID setup, the entire point was to allow RAID to be your source of resilience. While I’m personally not a user of hybrid drives, hybrids could be a reasonable option for a NAS.

• For local storage your choice is between HDD and SSD. I use one of each in my compute nodes. The HDD is used for the basic O/S and for the activity that experiences the most churn like swap and logging traffic. The SSD is used for application data. This setup protects the operating system itself from the risk of lockup due to NAND burn on the SSD, or SSDs failing due to bad firmware (several SSD vendors have had issues). My SSD selection criteria is definitely performance and reliability. HDD purchases have emphasized quality, and sometimes you can luck out with finding unused batches of older product lines where you hit a nice sweet spot that balances good quality, very decent (but not best-of-breed) performance, and fair price.

• We won’t get into the meat of disaster recovery during phase 1 — there isn’t any intellectual property to protect yet — but for initial economic planning it is worth considering if you want a spare drive of each kind you use. Both my NAS and the compute nodes use the same model of SSD so I keep one idle SSD in the NAS as a hot spare for a RAID rebuild, but in a pinch I could swap that into a compute node for a failed data drive. I also have a couple of spares of the HDD models used in the compute nodes in case a root volume fails. You’re trying to run a business, and downtime means impaired operations. When you are bootstrapping your lab as a side project, parts delays interleaved with life scheduling may interrupt your plans for weeks.

The Experimentalist : Phase 1: Just a Bunch of Storage © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Time to get down to business. Phase 1 of the plan is the rudimentary layer to building your own home-brew “mad scientist” lab for software engineering experimentation. JBoC — Just a Bunch of Compute — is about deciding the physical portion of the platform.

Prev: Building a SWE Experimentation Platform | Next: Phase 1: Just a Bunch of Storage

I’ll be describing this in terms of what’s most typical for backend and data engineers, but don’t view that as limiting your options. If you’re interested more in the front-end, you might make slightly different choices. If you’ve been dreaming of building a thousand-node Raspberry PI supercluster, have at it! Honestly if some board manufacturer would underwrite the work, I’d do a supercluster in a heartbeat.

The Outline

Money only goes so far. There is benefit to planning purchases much the same way a cash-strapped company might. The heuristic I’m using is:

• 75% of purchases will go to lower-cost hardware.

• 20% will be the mid-tier cost range.

• 5% go to the very infrequent budget-busters you have decided you must say yes to.

The percentages are ballparks in number of units purchased, not in dollars. That 5% top-end units may very well end up 75% of your spend if you’ve decided you absolutely must have that Nvidia H100 card, for example. More commonly it’ll be the monster laptop rig that you talk yourself into once every 5 years.

The goal in experimentation with distributed computing is to have hardware to distribute work over. The per-node computational muscle may not be that important. It’s the distribution of function that you are exercising, not whether each and every node can supply a water-cooled 100 FPS while playing Cyberpunk 2077.

Lower-Tier Nodes

Used, open-box, and unused-but-discontinued hardware is a great way to build out a cluster. Product lines in the range of 3-6 years old can have provide a lot of bang for the buck. We pay an insane premium in order to get the newest thing with a 50% performance bump when operated flat out. Most of the time we aren’t using server compute anywhere near full load, particularly in a home lab.

Think of your lab like you’re setting up a business. Fund the work you are going to do, not the work that somebody, somewhere, in a big wide world might conceive of doing. Someday.

Having headroom for maximum performance is great… if you’re using it. Save that for the high-end 5% of purchases. If it turns out to have been a vanity purchase, at least you’ll understand where the motivation came from.

Try to identify a single hardware model that you will stick to. There are two reasons for this:

1. When you have many different hardware variations, it becomes more challenging to reason about behavior like performance and fault rates. The differences become primary variables instead of your chosen computational activity.

2. Maintenance is harder because you’ll have different issues arise with every kind of different hardware. Firmware patch issues will differ, BIOS configuration will differ, supported memory chips or SSDs can differ. By sticking to a single model you gain leverage on any learning curve: solve a problem once, then copy-paste to all the other nodes of the same model.

Here are the characteristics I look for in lower-cost nodes.

• The BIOS and motherboard features must support whatever operating system and O/S features I intend to use, and supply the connectivity I’ve decided I need for any hardware that will be mounted directly on or cabled to the motherboard. More than anything else, this draws a line for “too old to use”.

• The model must be easily available. There’s no point in picking something you rarely see available on the major reseller web sites.

• There should be a decent amount of internet history on experiences with hardware quality for that model. You want hardware that has been reliable, both in initial assembly and in use over time. The model might not house the fastest and hottest-running components as of when it was made, because heat means thermal stress. Find a reliable workhorse, not an older Ferrari.

• The physical size of the model must suit the location where it is going to live. If you have an unused (and dry!) basement with ample space for racks and 1U “pizza boxes” then the width and depth of those nodes might be fine. If you have a half dozen square feet on a couple of home-office shelves, then small form-factor PCs may be what you’re looking for: I use SFFs myself for exactly that reason.

• You need to be able to support the power requirements of that model. You don’t want the circuit breaker tripping all the time because you tried to run too many nodes with 1200w power supplies.

• The models must run cool enough for the environment where they are going to live. Obviously you may be able to improve on the situation with various cooling solutions, but the end result establishes your thermal constraints.

• CPUs won’t be the best possible but should be acceptable in terms of their performance, their heat generation, and any history related to reliability. As an example, you may decide to take a pass on Intel Raptor Lake 13900K and 14900K CPUs because of their voltage-related stability problems, but on the other hand I wouldn’t rush to anything in the i3 product line either because it’s just too low-powered.

• RAM capacity matters for almost everything. Unless you’re doing the Raspberry PI supercluster, I wouldn’t bother with anything that doesn’t have at least 32gb of RAM per node, and preferably at least 64gb. Note that sometimes the manufacturer information will suggest 32gb when 64gb was possible if you had the right RAM sticks, so do your homework online by tracking down posts from modders and happy owners of a particular node model. Be aware that some motherboards with 4 sockets for RAM may not operate with maximum memory performance in that configuration; if speed is your goal you may be limited to a single pair of DIMMS.

• I personally prefer to buy bare-bone boxes that have any CPU, GPU, Wi-Fi, Bluetooth, and wired network support components mounted… but not the RAM or SSD. That lets me pick a consistent model for RAM and SSD hardware without feeling like I wasted any money. If you aren’t super-fussy about performance testing, then I wouldn’t worry as much about that. Keeping the Wi-Fi / Bluetooth / Network cards consistent though is likely to save you a lot of headaches. There’s more than enough room for configuration-related ick around those as it is.

Middle-Tier Nodes

The process here is much the same as that for the low-cost tier. The difference is you may know you want to perform work that requires physical capabilities your lower-end nodes will lack.

An obvious example would be if the lower-tier nodes don’t have any Nvidia GPU cards in them, and you need something that would let you work with CUDA. The node models will be a bit newer and components slightly higher-end, but you don’t have to break the bank. What you likely care about are which versions of CUDA are supported by a particular GPU card model. The Wikipedia page on Nvidia card features can be a handy resource for looking at API support.

Avoiding the RTX cards that end in “80” or “90” can provide you with some affordable options. Obviously lower-end GPUs wouldn’t cut it for somebody training a 100B-parameter base LLM, but we’re not trying to spec out hardware for an activity that runs with budgets in the 10’s of millions USD. I’m not going to get into AI details further now, they’ll be “Cheapskate AI” material in future articles.

Another kind of node warrants consideration as a possible mid-tier expense, and that is a good-quality NAS (Network Attached Storage). That’ll be discussed more in the next article on JBoS (Just a Bunch of Storage).

Higher-Tier Nodes

Not much to explicitly plan here except to note that this can be a disproportionate hunk of your budget. Avoid it if you can, but make the money count if you do it.

My personal inclination is towards these being the periodic upgrade to a really good laptop, since even when you have a compute cluster you’ll behave like any developer and work on it remotely.

If you have big plans in the ML and GenAI space, you might instead opt for a desktop box with whatever you can afford in the way of GPU cards and the most VRAM you decide you can fund.

Power Planning

This is a big area and I’m not the person to do full justice to it, but as you have more money in hardware you’ll want to learn about:

• Surge protectors

• Surge arrestors/limiters

• Power/line conditioners

• Uninterruptible Power Supplies (UPS) and their common architectural variants: standby, line-interactive, and online

I have seen a rack taken out by a lightning strike, so learning about the relevant power-handling hardware and how to set it up properly is something to have on your radar for some stage of the evolution of your lab.

Until then, at least consider surge protection. To be extra safe, power down and unplug everything whenever you hear about an incoming electrical storm, or when notified that the utility company is going to be doing work in your area.

One point of this lab-building exercise is to treat it as though you are running a business, not playing computer games, so safeguard that investment.

Other Parts

Buying compute nodes will not be the only purchase. You’ll need to allow for:

• RAM sticks of the appropriate size and part numbers.

• Drives (whether HDD, SSD, or hybrid).

• Network cables with shielding appropriate to the speed and length until you can connect to a router or switch.

• A network switch.

• Some USB sticks for doing initial O/S installations or for firmware updates.

• Anything you need for interacting with nodes, such as a monitor and keyboard.

• If the motherboard or node case have vendor-specific parts, like fan housings or heat shield armor, consider picking up a few spares while you can get them.

• Add any cooling supplies or spare fan parts you want quickly available.

• Add any tools you want for assembling hardware like Torx screwdrivers, anti-static mats and wrist straps, magnifier and lighting for viewing small parts in tight quarters, etc.

• If you went the route of rack-mount hardware, then depending on your plan either full-sized or table-top sized racks and all their supporting hardware pieces; alternatively if you need any office furniture like shelving then allow for that.

• Storage bins for bits and pieces, because you’ll definitely collect bits and pieces.

Add or remove as appropriate. It’s your lab for your experiments.

The Process

I would suggest some habits around how you acquire and integrate hardware.

1. Learn which vendor sources you find reliable, and mostly stick to them. You’ll have enough going wrong as it is. Sometimes your compatibility homework on a RAM part number won’t turn out as you had hoped. Sometimes the wrong part will get shipped. The universe introduces enough chaos so knowing which vendors are more reliable can save you a lot of grief.

2. Don’t rush to buy multiple units of a compute node or expensive part until you’ve bought and tried the very first one. It’s not hard to trip over something that will be show-stopping for your plans, and you have to send that item back and start again on your homework for the alternative. Buy one, put it to some use, and then if your plan was to buy more you can proceed.

3. Don’t buy until you have time to install and test, particularly when it isn’t a brand-new shrink-wrapped product. Schedule the time for the work. Not only will that help you catch bad parts fast enough to return them, it will also help you spot when somebody sends you a part that is not what was advertised. Even the shrink-wrapped products are better to be able to test quickly, but anything used or open-box you want to make sure you didn’t waste your money or were scammed.

4. Make notes and take pictures if those might be helpful later. Setting up a compute cluster has a lot of repeat work, and you will find yourself routinely wishing you remembered exactly what steps you took on a previous node. Capture the information while it is fresh. Even a day later you’ll forget something.

5. Budget for a cost 10% above what you planned. You’ll discover you need something that you didn’t realize would be important until opening up a case, or after realizing half your network cables aren’t being recognized by the switch.

6. Expect to do operating system installations frequently at first. Not only are there many flavors to choose from — even just in the Unix/Linux world — but you’ll find yourself grappling with questions on specific O/S version numbers, desktop vs server variants, disk layout for the installation, and sometimes BIOS configuration. You will change your mind, and discover why keeping notes comes in handy. Don’t be emotionally wedded to any initial set-up. Learn to feel comfortable tearing down anything that isn’t exactly as you want it to be.

A Living Example

Here are the low-tier nodes for my SFF cluster.

• Intel NUCs, all but one are the same model; the exception is my Kubernetes control plane. It has a small keyboard and monitor if for some reason I can’t get to the cluster remotely, e.g. due to a flubbed network configuration experiment.

• Each node has 64gb of RAM, an HDD, and an SSD.

• Cooling isn’t too much of a concern as Intel made these from CPUs intended for the mobile market, but I keep a fan at one end to circulate the air just to be safe.

• The nodes connect to a network switch positioned on the shelf above.

All of these were bought after Intel had moved on to the next couple of series in the NUC product line. Fortunately I avoided the Raptor Lake debacle. At this point I think about 3/4ths of the Intel NUC lines are aging out as an option for those starting in 2025 on their own cluster. There are many other SFF vendors that have taken up the baton, so I expect SFFs to remain a viable approach for those not wanting to go in the direction of rack-mount hardware.

The Experimentalist : Phase 1: Just a Bunch of Compute © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

My "Karl Marx Would Buy GPUs" article concluded how, with ever-concentrating corporate and personal wealth, GenAI vendors will push that disparity further. Our finances whether as employee, solo entrepreneur, or small-company founder, are facing similar forces. The choices on strategy to overcome the slow squeeze boil down to combinations of:

1. Find a way to be more capable than the big GenAI vendors and identify opportunities to beat them at some piece of their own game. Challenging, but not impossible as big companies aren’t as nimble as individuals or small companies. Expect to change direction rapidly as LLM capabilities evolve.

2. Find something your market wants that is grounded in novelty or uniqueness or deep insight, or at least with a provenance that communicates value. If you were part of the reason for the transaction, that edge can’t be instantly prompted away.

Next: Phase 1: Just a Bunch of Compute

Skill Building

My own focus is the world of software engineering so I’ve been trying to navigate a path that is about 5% of the first option, and 95% of the second. A day may come when technical domain expertise ceases to be a thing, but I believe we’re a long way from that. The strongest reason I have was explored in On the Turn of a Phrase:

We cannot think thoughts for which we lack the language.

Within software engineering — or frankly any deep-knowledge career path — that suggests a couple of observations:

• If you aren’t invested in the GenAI world then you need a deep understanding of the detailed workings of a domain. You’re competing against an ever-changing LLM knowledgebase. Surface knowledge won’t cut it.

• If you are invested in the GenAI world then your ability to generate quality outcomes improves greatly if you have deep knowledge of a domain. Surface knowledge has too little selective power in prompts to generate the strongest results out of an infinite universe of choices. You need to know what to ask for.

Either way you land in a similar place as a SWE. Deep skill has moved from being an eventual nice-to-have to now being a key factor in your relevance.

You can’t just read a book for deep skill:

• You have to be in a position to actively experiment as you drill down.

• We learn from a mix of positive and negative experiences.

• These shape both what connects in our memory and the boundary on what is appropriate to generalize.

Those of us who have already been in software engineering (or adjacent field) for a couple of decades already had opportunities to do all that. We’ve built layers of foundation skills. I’m concerned that more recent graduates, junior, and even senior engineers are going to find themselves denied a similar learning process.

The Strategy

To help fill the gap I’m rolling out a series of articles on the approach I have used myself for many years. I’m in the midst of updating it for life in an LLM world.

The idea is simple. Nothing teaches you how things work, like having to actually make them work by yourself. The effort to create comprehensive infrastructure from scratch will teach you far more than a hundred LeetCode exercises.

If you follow along and attempt any of this, in essence you’ll be building the scaffolding for your own startup. Whether you have a great business idea or not, at a minimum you’ll be faced with how pre-seed tech startups start to stitch themselves together and you’ll see why they have so many rough edges. There is only so much time, only so much money, only so much energy, and you learn that the only way you say “yes” to important tasks is by saying “no” or “not yet” to unimportant ones.

Even if you aren’t in an early phase of your career perhaps you’ve experienced what many of us do: an industry of ever-broadening technology choices we often bounce between at increasingly shallow levels of understanding. If you haven’t built a proper experimental playground before, this may increase your motivation so that you can dig into all the high-leverage details that the day job may not enable.

There is another theme to this. Cloud compute and GenAI vendor APIs have something in common. Both have ways they tax you for experimentation, and can punish you badly for expensive mistakes in establishing budget guardrails.

The Experimentalist Laboratory

The approach here is to mostly run on hardware you’ve purchased for yourself. Obviously many things can be moved to the cloud, and some of them should, but there is nothing quite like the freedom to make any purchase choice that makes sense to you, apply any configuration that helps you, and throw away anything you later discover does not server you.

Technology should be about YOU, and not the other way around. The process here will very much be one of making, and revising as necessary, decisions that place YOU at the center of what happens, and keeps YOU as the primary decider of pretty much everything.

A side benefit is that, other than the initial hardware purchases, you aren’t constantly feeding mega-corps more money just to have permission to learn and push the envelope. Stop acting like their favorite flavor of catnip. They have an ample supply.

The essence of building your own laboratory for experimentation is straightforward:

• The economics of the platform should be approachable.

• Hardware, software, and some external services will be involved.

• It likely takes more than a laptop; you’ll want something that surfaces real challenges and provides the growth potential of actual distributed systems.

• There will be many moving parts, so tackle the assembly in bite-sized pieces.

• Optionally include LLMs. If you do, the same hardware you use for other experiments can be used for self-hosting some LLMs.

Including LLMs opens up more avenues for learning and — if this happens to be your goal — you were never going to avoid self-hosting experiments with the strategy “beat GenAI vendors at their own game” as your objective.

Assembling the laboratory is broken into two major sections and an optional third.

• Phase 1: the basic foundation of hardware plus minimal software tooling for rolling out system configuration and application installation. It also covers ensuring that your work gets protected somewhere remote.

• Phase 2: an ever-growing collection of technology artifacts or progress on procedural efforts so that, in time, you can build pretty much anything you wish.

• Cheapskate AI: optionally, run LLM tooling locally to experiment with, tune, and control the behavior of the AI tools you use. The tools should behave as you wish, not as an outside vendor decides for you.

The to-do list can be as long as your interests drive it, but for those earlier in their career you should come out of this with a gut-level understanding of what drives many decisions at the intersection of technology and business. That awareness can become one of your selling points in a career role or a new business.

In future articles I’ll start breaking down the laboratory assembly in detail.

The Experimentalist : Building a SWE Experimentation Platform © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0