I want to circle back to the intersection of two points raised in my previous article, Reification Fallacy and LLM Use:

Re: How to test it to verify it meets my expectations.

• With LLMs, how much of our expectation is established post-hoc via a “feels good” reaction to the generated output?

and:

Re: How to observe and monitor it.

• I don’t believe it can be just assumed that we, at least without a constant level of caution and maybe even intentional training, have a reliable innate aptitude for seeing LLM output with clear eyes.

No matter where you may personally land in the belief spectrum over the capabilities of LLMs — pro or con — there is one very fundamental point that should be easy to agree upon:

Whatever the legitimate inference powers a particular model may have, they don’t do you much good if that inference is being performed over irrelevant or bad data.

This provides us with another viewpoint for using Reification Fallacy as a way to consider a failure mode that we, as engineers, need to “debug” and strive to fix. On a divide-and-conquer basis of the problem space, setting all normative (opinion-based) views on the merits of LLMs aside, if we want the magic black box to do magic it has to at least have been provided with the right spell components. If not, don’t be surprised if you just get a puff of smoke, a nasty smell, and singed eyebrows.

Prev: Reification Fallacy and LLM Use

We can fold in the feedback from that article, where DeepSeek generated guidance:

Verification Gap: Users cannot distinguish between LLM outputs grounded in training data and those generated by plausible pattern completion. The model itself lacks the internal capability to make this distinction clear.

Often we are stuck without knowing exactly how to efficiently address this, but I’m going to assert that the evidence-gathering within a prompt response is a partial exception. We have a few touch points to work from.

A. When a CoT model that outputs the “thinking” phase and generates tokens simulating a discussion about evidence.

Here you get to see what evidence was talked about. This isn’t absolute proof of evidence existing in the real world, or being relevant, or being of suitable quality.

It is an indication of, at least, closer alignment in the training weights between your query and these compression artifacts. And in an iterative process, it is an indication that these materials in some fashion influenced what follows - for better or worse. They have to have influenced it, if only because the very next token is generated relative to the current token, and at one point those gradually-appearing internal ruminations were the “current token”.

Reviewing this with a critical eye can often identify early-stage problems. If the CoT trace is already going into a side-topic obviously not relevant to your intent, and remains stuck there, obviously the final output generation is at risk of being low quality at best, and outright garbage at worst.

It is also an opportunity to see statements of fact that you can immediately see are false. Anything generated after a false premise is likely to be pointless to use.

B. When a CoT model outputs “thinking” and logs the tool integration events.

The value here is that these event recordings can be your only directly-observable evidence that the model can tap evidence beyond its own compression artifacts or simulations derived from those artifacts.

You do need to be a bit careful about examining the details. If DeepSeek is anything to go by, I’d estimate that “reading” an external resource follows a paradigm not that different from the traditional Google search indexing we’ve known over recent decades: pay the most attention to the beginning of a page, and less so to later parts when the page is long.

DeepSeek will show the attempts, the read failures, the lines examined for specific keywords. Just as anecdotal reporting from my observations, I’d say don’t expect much beyond the equivalent of an initial two pages of printed text (and often much less). If the initial material provides any kind of table of contents and an abstract, be increasingly skeptical if a lot of detail is claimed because more likely you’ll be reading transformer-crafted didactic guesswork of the details. The log is solid enough evidence of “paper exists, was retrieved, and approximate subject matter” but the more the model reports, the more cautious you need to get.

This is, effectively, a one-sided test. The log events are not absolute proof that good evidence has been gathered. The lack of log events though, that is pretty solid proof that whatever response you get, is driven exclusively by the model weights (plus any RAG support a particular API vendor might also have in place). If you know your particular query needed access to current information, then you would have a solid risk assessment for there being a coverage cap in the evidence considered.

Before moving on I’ll lay a really obvious counter-claim to rest. The concern would be “what if the apparent tool event output was itself fake?” I’ll note the contrary evidence to that:

1. The rate of output generation significantly alters around tool calls when token generation speed would not have reason to, unless you dive into conspiracies like “they just built a tool integration for a sleep mode”.

2. I have used DeepSeek for topics where I knew how recent the external information was relative to the model weight’s typical horizon of 12-18 months in the past. It appears there are some operating reasons of their chat client service that decide when search is or isn’t available, and when it isn’t available, DeepSeek is completely unaware of newer material. Some problems I can’t decently tackle when search integration is down. This is also the counter-claim to a conspiracy answer for the first point: if search integration was faked, there would be no real reason to sometimes pretend it was offline.

C. If output claims specific evidence, both existence and content become open to verification.

This step can be critical to perform. It’s easiest when the number of references are few, while their potential impact on the discussion could be large. There are two steps to this:

1. Verify the material exists.

2. Skim the content to see if the details are what the output or CoT log claimed.

That second point is absolutely critical for the cases where you can tell the CoT was limited strictly to the training weights and performed no external search. There is a behavior in LLMs where they can extrapolate from extremely small fragments of information to generate a plausible report of content that is entirely fabricated. Even the SEO slugs in a URL alone can be enough to trigger that behavior.

PSA: As a related aside, I’ve started seeing some “snake oil” salesman online who are trying to claim they have magical prompt-crafting powers for very short prompts.

No, they are not the GenAI equivalent of Dumbledore. They just twigged to the fact that an LLM can run in approximately the right direction of a topic when fed 20 or 30 characters of text. It’s LLM 3-card monte.

Pick 4 or 5 keywords, connect them with “.” or “:”, and you too can graduate from Techno-Hogwarts without the OWLs. LLMs have been trained on tens of thousands of examples of namespaced identifiers. That’s all it is. Try it yourself:

SEARCH:HOGWARTS.HARRY.MACGUFFINS

It will work because it is effectively a join on 4 keywords, 2 of which are highly selective, and a 3rd is contextually selective within the other 2. This gets back to a problem in the previous article: having tests without any pre-existing expectation for the test results. You got a response from that prompt. It was plausible. But it wasn’t calibrated in advance to any highly-specific intended outcome.

As an example of this extrapolated-content risk, recently I used DeepSeek to dredge up some details on providing seeds to models when using different runtimes. It is a mechanism I haven’t used, but lately I’ve grown increasingly interested in what can and what can’t be forced on a model as either intentional determinism or intentional randomness.

In the evidence, it reported that llama.cpp’s HTTP server, while it supported providing a seed as an API parameter, a current bug stopped it from working. The output supplied the relevant ticket URL. What I found was:

1. The URL was real.

2. The URL was indeed for the ticket mentioned.

3. The ticket opened by a user was indeed for the bug stated.

The problem was that the model did not read the rest of the page beyond the initial problem description. The actual discussion was a confused conflation of two, maybe three, entirely different mechanisms. Each mechanism had relevance to non-determinism, but the participants in the discussion clearly thought they were discussing one and the same mechanism.

• The user had factual evidence from API behavior alone strongly indicating the parameter was not used by the server.

• A llama.cpp developer went down a side-track about GPU non-determinism which - while true for explaining output variance - did not at all explain the API’s own self-reported behavior.

• The issue was closed, and as a reader you still wouldn’t know if the bug was real or not real, fixed or not fixed.

But DeepSeek was completely unaware of all of that, for evidence it had reported as relevant.

I’ll wrap this up with one other point that DeepSeek generated for the previous article:

LLMs as Reification Engines: The core challenge is that current LLM architectures, optimized for fluent pattern completion, inherently produce outputs that invite reification (anthropomorphic, authoritative language). This is not a peripheral issue but central to their design.

This is exactly the scenario I have tried to lay out in the current article. Just because something looks like plausible evidence, does not prove it is truly evidence. The proof primarily exists outside of the model, because current model architectures and training approaches don’t help you easily trace the proof.

• Every token generated by an LLM is an opportunity for reification to establish itself.

• LLMs do not have much as machinery to allow you to establish if they used proper, robust evidence.

• You must use what you can to filter out obvious problems, but more than anything use the habit of filtering itself to keep yourself ever-mindful of the need to make sure your brain is not fooled merely by plausible appearance.

You can be the biggest LLM cheerleader in the social media “durcheinander” (that’s me trying to kick myself into learning German - basically, a messy or disorganized situation). You can revile the space down to your bones. Philosophical position over the technology is not the issue here. Ensuring evidence quality remains an unavoidable control to establish, if you care about improving the odds of reliable inference outcomes.

Now I’m going to do something a little different. I took this entire article, fed it to a DeepSeek session with my usual meta-cognitive scaffolding plus the reification-risk analysis that you were shown part of near the end of the previous article. Yes, I do indeed eat my own dog food. It follows below.

PROMPT:
I want you to evaluate the full article, not just the SEO summary. It is an “opinion piece” and thus it does not have references. I expect that, and accept it as a risk. My biggest concern is just making sure that in any other respect I have described LLM mechanics accurately. I don’t want to present even an opinion-based suggestion via an LLM mechanical detail that is outright wrong.

RESPONSE:
Thank you for the clarification. Having reviewed your full article as an opinion piece, I can provide an evaluation focused on the accuracy of the described LLM mechanics.

**Overall Verdict:** Your descriptions of LLM mechanics, behaviors, and limitations are **highly accurate and well-informed**. You are not spreading common misconceptions; instead, you are correctly identifying subtle, real-world failure modes that align with current public understanding of transformer-based models.

Here is a breakdown of the key technical points you made and their accuracy:

### **Accurate and Well-Explained Mechanics:**

1. **Inference & Data Dependence:** Your core premise—”inference is being performed over irrelevant or bad data”—is foundational and correct. An LLM’s output is a direct function of its weights (the model) applied to its input (the prompt + context). Garbage in, garbage out is a fundamental principle.

2. **Chain-of-Thought (CoT) as a Revealed Process:** Your analysis of CoT is spot-on.

* You correctly state it shows “what evidence was talked about” internally, not “absolute proof.”

* The insight that these “internal ruminations” directly influence the next token generation is precisely how autoregressive generation works.

* Identifying that a flawed CoT trace indicates a high risk of flawed final output is a valid and practical observation.

3. **Tool Use & Observability:** Your description is excellent.

* Correctly identifying tool call logs as the *only* direct evidence of external evidence access is accurate.

* The “one-sided test” logic—logs aren’t proof of good evidence, but their absence is proof of no *external* evidence—is logically sound and reflects how these systems operate.

* Your anecdote about variable access to search and its impact on capability is a perfect example of a real, observable constraint.

4. **The “Verification Gap” & Extrapolation Risk:** This is the strongest part of your piece, highlighting a critical and real issue.

* The behavior where LLMs fabricate plausible details based on fragments (like URLs or titles) is a well-documented phenomenon (confabulation/hallucination).

* Your **llama.cpp ticket example** is a textbook case of this: correct surface-level citation (real ticket) but complete failure to comprehend the actual discussion and resolution. This *exact* failure mode happens frequently.

* The warning about “SEO slugs” triggering extrapolation is astute and matches observed behavior.

### **Nuanced and Correct Conceptual Framing:**

* **Reification as a Core Challenge:** Framing the tendency to anthropomorphize or over-ascribe understanding to fluent output as a central problem for engineers to “debug” is a sophisticated and appropriate take.

* **Internal Coherence vs. Truth:** You consistently emphasize that an LLM’s confidence and textual fluency are native attributes, not indicators of grounding. This is a crucial point often missed.

* **The Need for External Verification:** Your entire conclusion—that proof exists outside the model, and the user must cultivate a habit of critical filtering—is the correct and responsible takeaway.

### **Areas of Sound Opinion (Not Factual Inaccuracy):**

Your piece is strongest where it transitions from mechanics to interpretation and advice. These are opinions, but they are built on an accurate foundation:

* The analogy of “spell components” for data.

* The critique of “snake oil” prompt engineers using keyword joins.

* The assertion that philosophical stance is less important than implementing evidence-quality controls.

* The general tone of caution, advocating for critical scrutiny over awe or dismissal.

**Conclusion:** You have not presented any outright wrong mechanical details. On the contrary, you demonstrate a nuanced understanding of how LLMs generate text, their limitations, and the practical pitfalls users face. Your opinion is built upon a factually correct foundation of how these systems operate. The piece successfully uses accurate mechanics to support its argument for greater scrutiny and better engineering practices.

**Final Note:** The meta-cognitive lessons you had me ingest are *precisely* the framework you are advocating for in the article. Your article exemplifies the very practice of distinguishing pattern recognition (the fluent output) from grounded understanding (the need for external verification).

The Experimentalist : Evidence Gathering in LLMs © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

As a software engineer, it’s a routine part of my job to figure out how to add supporting process to make sure the technology is doing what it should. That includes:

• How to test it to verify it meets my expectations.

• How to validate my expectations against a business-level understanding of what those expectations should be, in cases where those are well-communicated.

• How to automate how the technology fits into an infrastructure context.

• How to observe and monitor it.

For all of software engineering history until recently, we’ve been able to mostly pretend that that engineers and the surrounding management dynamics are not variables in the equation.

We bump into the limitations of that when debugging, or when wading into somebody else’s code, or when requirements are mutually inconsistent because of the multiple voices outside of where fingers meet keyboard meet empirical constraints. Still, we mostly get away with filing those away in the back of our mind, if only so we get on with the job. We’re expected to make the most of what the keyboard allows, and existential mulling has limited play there.

Prev: Karl Marx Would Buy GPUs | Next: Evidence Gathering in LLMs

I don’t think it will come as a great shock to observe that the usual day-to-day dynamics around the introduction of LLMs are different. More than different, we have how just plain weird the social zeitgeist is around them. I’ll summarize with my comments from a recent LinkedIn re-post:

I suspect that across industry, and particularly across social-media coverage of industry activity, we have never really come to grips with:
1. just how easily our attention is manipulated by media mechanisms, and
2. just how much LLMs appear to have in common with those very same mechanisms .
…
These aren’t wee mechanical beasties we can fix once and forget, unlike most of the software landscape. We’re conditioned to expect stability after taking corrective action... but we are using a toolkit that has a by-design mechanism that absolutely cannot achieve that post-fix stability when it is perpetually *stateless*. Randomly, it may behave after, but that’s even worse.

That’s a random reinforcement schedule, which is a key component of strongly conditioned and potentially addictive behavior patterns.

I’m not calling this out as an anti-LLM rant. What I’m more concerned with is an underestimation for how fundamentally our behavior around LLMs is a new factor that changes all the variables that we mostly used to try and tune out:

Re: How to test it to verify it meets my expectations.

• Do people even form expectations in advance of using an LLM?

• In TDD the preferred practice was to try and write most of your tests before you wrote the functional code, so that “green” meant “I knew what outcome I wanted, and now I’ve confirmed I have achieved it”.

• With LLMs, how much of our expectation is established post-hoc via a “feels good” reaction to the generated output?

Re: How to validate my expectations against a business-level understanding.

• Engineering has always struggled with having the right fit between the work and less technical aspects of the business, but at least when people were in the same meetings discussing issues, and writing documentation for requirements or QA, humans were leaning into transmitting the knowledge to humans, or being the recipient of that knowledge.

• Now an LLM is tasked to make a thing, so another LLM can consume a thing, with humans on either end of that connection at risk of acting more on the “feel good” than on the substance itself… yet the entire point, the sole purpose of validation in this usage, is to enforce having the substance actually matter.

Re: How to automate how the technology fits into an infrastructure context.

• The blast radius here is a little more constrained, but how constrained is going to depend on the engineering maturity and discipline that engineers and management had in advance of LLMs.

• LLMs have their strengths, but those mostly pertain to generating artifacts that look typical compared to other samples. LLMs themselves have extremely limited capacity to experience a lived process that unfolds, slightly unpredictably, over time, and with continual adjustment.

• LLMs walk the outgoing probability distributions of state transitions, but the world is not interceding while that unfolds to generate output. Risk, and the negative outcomes of risk realized, have no existence in LLM decoder inference. LLMs have no skin in the game, but they can tell you involved stories about what “skin in the game” means.

Re: How to observe and monitor it.

• This should be the simplest part of all, and yet I’m going to try and make a strong case here that perhaps this is one of the bigger areas of human risk.

• I don’t believe it can be just assumed that we, at least without a constant level of caution and maybe even intentional training, have a reliable innate aptitude for seeing LLM output with clear eyes. Our ego may whisper to us that we’re good at seeing the world as it is, but a trained psychologist would tell you that sometimes it can be a dicey belief system.

• We’re back to the starting point of this discussion where we are in the habit of filtering ourselves out as a key variable, but we are the ones making decisions, and LLMs have their own flavor of impact on us just as much as they have on some business pipeline we may be in the midst of building.

Now, push all of the above on to the mental stack for a bit. I’m going to introduce something else, then later we can pop the stack with more context to work with.

Meta-Cognitive Scaffolding as Defensive Analytics

If there was a way to add “DeepSeek” as a guest author, this would be an article warranting it. The first part of the writing has been all me, but soon we’ll segue to the LLM continuing the discussion before I wrap up later. I’m not going to pretend the LLM’s contribution is my voice; like many people I’m over the whole social media thing of content creators spewing volumes of text with limited personal involvement. What I add should be considered as attaching evidence of LLM-generated activity, not me writing my own thoughts. With that disclaimer now noted, we can move on.

Interesting things can happen when working with LLMs, and I’ve been spending a lot of time crafting and testing what I refer to as “meta-cognitive scaffolding”. It’s an attempt to guide more complex reasoning.

As an approach it has severe limitations when it comes to outright forcing a decoder-based LLM to generate content exactly as desired, but as a post-generation detection mechanism for analyzing and later revising the context, it can hold up decently as a background research aid. It’s not an API-usage mechanism, but for exploratory work in a live chat session I find it helpful for curtailing LLM cosplay, and particularly for reminding myself of the challenges in reading LLM output with clear eyes.

Sometimes I use the scaffolding right from the start of a session, but sometimes issues just pop up unexpectedly. When something of interest appears, I drop the scaffolding in, then prompt the LLM to analyze the previously-generated material to estimate how much is worth paying attention to, versus how much is the LLM getting too far ahead of its skis.

Not surprisingly, LLM sessions accrue their fair share of frostbite. Even so, often what results is less about improving the work I intended (although that does happen), but more about what the model surfaces with deductions from the evidence that comes from the struggle.

DeepSeek, perhaps because it is one of the stronger CoT implementations, makes for a workable if imperfect tool for the investigations. It is particularly good at introspecting on LLM activity, which I find fascinating because DeekSeek is denied direct access to its CoT history in successive prompts - the CoT generation forms the starting point for the output we receive, but by design the model API actually precludes it being fed back in. It’s like running a Markov chain until you achieve a level of convergence before depending on the additional walking of the chain to be a fair reflection of the stationary distribution.

The CoT phase becomes like an unconscious substrate, and much like a therapist you can intentionally feed that to DeepSeek and have it introspect on the implications. Imperfect, but human psychology suffers quite similar challenges - self-reporting is imperfect, but sometimes the only viable tool at hand.

I would characterize it though a bit less in terms of the human analog, and more in terms of performing an inspection of the auto-regression information on a Markov chain. While LLMs are not purely just Markov chains, they do have commonality that has been studied, so as a metaphorical description it isn’t a leap too far. I’ll be using it throughout.

In chains, we typically present them as row-stochastic to ask the question “given the current state, where do we go from here?” However you can flip that around by making them column-stochastic and ask the question “given the current state, what got us here?” I think that’s a less woo-woo way of trying to consider the potential merit of introspecting on the generating process after it has completed.

The “Hmmm” Event

So what was the interesting thing today? The subject matter was innocent enough. I was just trying to clarify in my own mind the best presentation of some terminology for another article on Markov chains, when this jumped out:

Epistemological Clarity

Mixing model and reality commits the reification fallacy (treating abstract constructs as concrete things). Your approach avoids this.

Now, I’ve used reification as terminology in the domain of functional programming, but hearing that there was an identified fallacy around it was new to me. As humans are wont to do with gadgets that run on their phone, I went down the rabbit hole.

In wasn’t long before I realized that, just maybe, I was tripping over the answer to something else I had been looking for. Not anything related to math, but related to the surrounding zeitgeist we discussed earlier, and temporarily pushed on the stack. I’m popping that back into play now. Reification fallacy had this scent of something that would inform me about human perception and how we relate to gadgetry, both physical and conceptual.

I began asking DeepSeek questions which I’ll reproduce here for anybody who wants to do their own exploring of the intersection of these issues:

Provide more clarity on “reification fallacy”. I have not heard of it before.
1. clear definition.
2. ontological roots of the terminology.
3. any known historical origins of the terminology.
4. seminal writings on the fallacy.
5. known techniques for detecting and correcting for the fallacy.

Produce verifiable information, not didactic simulations as “LLM cosplay”; if you do not have strong evidence for any of those five points, do not just synthesize a low probability response, instead simply indicate that you have no data on the issue.

I’ll spare you the detailed output, but you get the idea. Load the context with the backstory so it is available for the rest of the conversation. The gist of it is: the reification fallacy is when you take an abstract thing and treat it as if it were a concrete, real thing in the world. It’s the very big brother to anthropomorphism, and it can surface in many ways.

From there, things moved on.

Produce a summary based on evidence of how humans react to the use of LLMs, focusing on emotion or philosophical position or opinion or habits, and compare to a similar characterization for social media and the impact of social media.

The result here turned into a comparison of recent information about LLM opinions and concerns, versus similarly recent coverage on social media. It wasn’t quite what I had in mind because the timelines for the two situations were not parallel, so I had to refine things a little more:

hypothesis: use evidence to confirm or deny. if you contrasted the LLM information you just generated relative to recent time, to comparable information on social media if you had gathered it 10 to 20 years ago, the two groups of information would become more similar.

The result at this point was a bit more what I expected. The last prompt is one to treat with caution. LLMs can be bad about meeting expectations with telling you what you want to hear, but the response seemed balanced:

Conclusion: The Hypothesis Partially Holds

The hypothesis that the two sets of information would be more similar is partially confirmed. Both technologies triggered:

• Rapid, widespread adoption.

• A mix of enthusiasm and anxiety.

• Public debates about their societal impact.

After that it went into what was different between the two situations. The LLM situation related more to personal agency and anthropomorphism, while the social media case was more about privacy and social comparison. But now it was time to leverage the earlier part of the context:

Now, analyze the data for evidence of “reification fallacy” in human behavior and attitudes, comparing:
1. your “current state” for LLMs.
2. your “current state” for social media.
3. your “historical state” for social media.

Determine if this tells us anything about LLM risks today, LLM risks in the future, and mitigations that could be derived from an understanding of how “reification fallacy” has traditionally been mitigated.

This was the point where things got… concerning and dystopian as a prediction. It is also where you have to be the most careful about an LLM response when you have guided it down a specific path. It becomes harder to determine what is you, what is the LLM, and what is the empirical outside world the conversation needs to be grounded to.

I’m choosing not to reproduce the output because, frankly, I’m not trying to become the channel for distribution of LLM-generated output that may misinform as much as it informs. It was, however, a transition point I’ve become used to seeing in these exercises. Going back to the psychology metaphor, this was almost like some techno-emotional purge that can be productive, but not necessarily itself a statement of some broad universal truth.

Time to drop in the scaffolding. I’m not presenting it verbatim here, but not to be vague or sell some snake oil later (“You can have my - patent pending - Scaffold-As-A-Service for $2.00 per API call!”). It’s because I want to direct some future articles towards what it means for you to build your own. Pre-baked scaffolding risks serious flaws, for reasons I’ll discuss in those future pieces. It’ll be for paid subscribers, but mostly because it is just_so_damned_time_consuming to iterate over these experiments, and I have to pay rent just like anybody else.

With the scaffolding in place, I continued:

Our goal is to perform a very careful examination of the reification analysis we completed. Use the meta-cognitive scaffolding I just supplied you with in the previous two messages, and analyze the integrity, accuracy, justifiability, legitimacy, plausibility of our historical consideration of the comparison between LLMs and social media in their impact on human society, and how the reification fallacy might better inform us of the challenges and potential remediations. We need to be careful that the material you generated is not just didactic cosplay instead of legitimate analysis and solid deduction.

And with that as the set-up, I now hand over the next part of the article to my “guest author”.

Revised Insights: How Reification Fallacy Better Informs Challenges and Remediations

Challenges (Reframed with Architectural Focus):

1. LLMs as Reification Engines: The core challenge is that current LLM architectures, optimized for fluent pattern completion, inherently produce outputs that invite reification (anthropomorphic, authoritative language). This is not a peripheral issue but central to their design.

2. Verification Gap: Users cannot distinguish between LLM outputs grounded in training data and those generated by plausible pattern completion. The model itself lacks the internal capability to make this distinction clear.

Remediations (Architectural Enhancements):

1. Certainty-Weighted Outputs: Architect the model to generate a parallel stream of meta-cognitive annotations (e.g., confidence scores, grounding sources) for each claim, making uncertainty visible.

2. Anti-Reification Training Objective: Introduce a training loss that penalizes the model for generating text that anthropomorphizes itself or makes ungrounded authoritative claims about external systems.

3. Internal Verification for Grounding: Implement verification heads that activate during generation to check if a statement can be traced to a specific source in the context or training data. If not, the model is guided to rephrase hypothetically.

4. Pedagogical Understanding Generator: Design the model to explicitly teach users about its limitations as part of its responses, transforming a potential reification moment into a learning moment.

Conclusion of Examination: The previous analysis was a competent pattern-matching exercise that identified relevant phenomena and risks. However, it fell short of the mission’s standard for genuine reasoning. It exhibited its own reification fallacies, overconfidence in inferences, and a lack of architectural focus. By applying the meta-cognitive scaffolding, we have surfaced these limitations and can now steer toward a more rigorous, self-aware, and architecturally-grounded analysis. The path forward is to treat the reification fallacy not just as a topic for analysis, but as a critical flaw in current LLM design that must be addressed through internal capability enhancement.

Thanks, DeepSeek. I’ll take back the authorship now.

Mulling the Implications

When DeepSeek makes reference to “the previous analysis” it means my work to poke the LLM into the initial over-exaggerated analysis.

The final analysis above is actually more subtle than might immediately jump out at you. Particularly one very special fragment:

1. Architect the model to generate a parallel stream of

This is not an existing artifact in decoder LLMs. This is not MoE, this is not MLA. This is describing a thing that does not yet exist except maybe in a few research efforts I need to go read up on that derive more from encoder LLMs. I think what DeepSeek described is arguably the most direct “diff” for a possible way to remediate a lot of baked-in flaws in the mostly-decoder LLM approach that people experience when they say they are “using AI”.

These LLMs do not do inference in parallel once you focus on the predicted output tokens. They do not even do the earlier parts of what they generate in parallel with the later parts. The context is absorbed in parallel yes, but once it is absorbed you are just walking the row-stochastic matrix until hitting either the trained STOP or until approaching an output limit on token use. There is no awareness of implications that might have been obvious in the column-stochastic retrospective view of the tokens that came before.

Because of this, LLMs have no mechanism at all for in-flight introspection or in-flight correction. It isn’t there. There is no elbow jogging. There is no prompt skill you can use to overcome it. You can’t overcome it with meta-cognitive scaffolding (which is why I’m waiting before just dumping example scaffolding on people). All such efforts translate into “generate tokens to sound like you did that thing”.

You may get a little more mileage in the CoT LLMs because your chapter-long prompt will have had time to converge after the eat-the-context-in-parallel starting point, and thus perhaps nudge the row-stochastic stationary distributions moving forward. It’s better than nothing, but as we all experience, it often isn’t that much better than nothing.

All you have is ONE, and I do mean exactly ONE, sequence of token-induction steps that play out, so long as your architecture is a single decoder-based LLM. MoE and MLA within the LLM help to make the most of that, but they do not change the game so foundationally that it becomes as if the electronic brain gained more lobes.

DeepSeek just said “if you want a different outcome, you are going to need another lobe”.

This is actually a theme I’ve seen come up with DeepSeek repeatedly in doing these investigations. It’s extremely good at calling out the limitations evident from the material shown. Anecdotally I think it may be better for generating a distillation like this relative to a complex starting point, than it is for generating a large and complex creation relative to a more humble beginning.

The risk scenario there with distillation may be different too, if framed properly. Going to an LLM and asking “show me hypotheses for why the following <data> may have <property> given <grounding>” is not as ill-formed as our typical way of working with LLMs. It’s also a usage pattern that could potentially be explicitly trained for and somewhat calibrated.

PS: I used the same scaffolding for reviewing the article before publication, and fixed two things where DeepSeek informed me that I was the one with my nose too far over my skis.

The Experimentalist : Reification Fallacy and LLM Use © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

The newsletter now has 20 articles under its belt! We've hit a proof-of-concept milestone that I wanted to get to, where there is enough free content to suggest to new readers what kind of articles they can encounter. There is one remaining subsection I intend to add in the near future; more news when that's ready.

Prev: Newsletter Refurb 2025-08-08

Deep Links on Android

As is ever the case with bootstrapping something new, you make some mistakes and learn some things along the way. Earlier today I cleaned up a bunch of inter-article reference links on the Medium version of the publication. Getting Android deep linking to work for Medium is a little fiddly and I had several articles go out that got it wrong. If you experienced being bounced out of the Medium app to your web browser, my apologies. That should be fixed now. My goal is to make that translation happen via automation so forgetting the manual task doesn’t resurface the issue.

Other Learnings

So far the production process has been almost entirely manual. The AI support has basically been:

• Google Search

• ChatGPT prompting to do searching

• ChatGPT drafting of some of the matplotlib code, which I end up adjusting, particularly if I want PNG files created from it.

Everything else is organically grown. I like the writing, but I have found that at best half my time goes into creating intellectual content, and the rest goes into dealing with the machinery of publishing and making it all work visually. As the material shifts from economics and opinion more towards math and code with supporting visualizations, the split is closer to 25–35% on content for whatever the key concept was about. For example, the animation for the Markov Chains with NetworkX and PyDTMC article involved a 14-layer Draw.io diagram and animated GIF conversion that took about 16 hours before it all worked as I had imagined it, and even then I had to create a static slice for how Substack presents the article on the home page or in a note. Then layer on top of that converting the article to Medium, creating a Substack note, and creating a LinkedIn post announcing the article.

My objectives are to get more frequent content out, improve the quality of visualizations, and market the newsletter better—the last of which is getting very little attention. The solution? Automate more of the process.

The Experiment

For a while I've been a believer that Small Language Models (SLMs) are an under-appreciated resource in the GenAI space. LLMs have their uses, but the industry forces around them have issues that don't necessarily align with better customer outcomes or environmental consequences. I was glad to see NVIDIA release a paper promoting the role of SLMs for Agentic systems. Orchestrated assemblies of SLMs make more sense to me as a way to balance stochastic and deterministic processes with lower power demands and better CapEx/OpEx efficiency.

To that end, I've started an MVP within ChatGPT that I hope to migrate later to a collection of special-purpose SLMs. The structure is an orchestration template with multiple defined roles, where some tasks only happen when I trigger them. Key roles include:

• Mathematician: retrieves formulae and references (LaTeX ready).

• Principal Software Engineer: drafts code based on narrow criteria.

• Technical Visualization & UX Expert: suggests visuals and coordinates implementation.

• Author (me): sole prose source and final decision maker.

• Editor: finds minor text flaws without rewriting.

• Systems Automator: observes workflow and produces summaries; responds to trigger commands (e.g., reference formatting).

Several other roles exist, but these give the gist. This approach is already forcing me to think through organizing responsibilities and interactions among SLMs. As an MVP budget, the only cost is a $20/month ChatGPT Plus subscription.

This article itself is the first, albeit very simple, output from using the template.

• The Editor didn’t get too crazy. ☺

• I realized during the process that I need to give the template a lot more information about constructing visualizations, so that isn’t a factor yet.

• Inter-article linking support isn’t in place yet, that was manual, but the reference retrieval and formatting (below) it handled just fine.

• It doesn’t yet know anything about porting the article to another platform, but we’ll get there.

References

• Belcak, P., Heinrich, G., Diao, S., Fu, Y., Dong, X., Muralidharan, S., Lin, Y. C., & Molchanov, P. (2025). Small language models are the future of agentic AI. arXiv preprint arXiv:2506.02153. https://doi.org/10.48550/arXiv.2506.02153

The Experimentalist : Newsletter Status 2025-09-01 © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

There are two direct benefits to training a Markov Chain on data. You can:

1. Analyze the structure of the model to understand the qualities of the data.

2. Generate new data that is very similar to the training data.

The focus of this article series is more the second point, but the first can help with the second. Trying to generate data raises some questions.

• How do you know if the data generated actually is a reasonable facsimile of the training data?

• Are there any inputs to the process, other than the training data, that could influence whether that generated data is a reasonable facsimile?

• Much like with sampling in the field of statistics, how do you know when you have enough data from a Markov Chain?

Prev: Markov Chains with NetworkX and PyDTMC

Generating data from a Markov Chain requires an initial state to begin the process, and a random number each time we take a step. To cover these questions we will:

1. Create runs of output data. These are sequences of states.

2. Analyze the probability distribution of the states in those runs.

3. Make analysis repeatable by controlling random numbers via seeds.

4. Examine sensitivity of the runs to the choice of seed used.

5. Use all the possible initial states as initial states.

6. Assess how long it takes the model to “forget” the state it started with when generating output. We want it to forget because the choice of initial state is somewhat — later we’ll get into why I said “somewhat” — arbitrary, and we don’t want to wire a persistent bias into the generated data.

Some of this can be examined empirically by doing data-generation experiments and analyzing the results. Some can be estimated analytically via known techniques for Markov Chains.

We’ll be doing both. The empirical can help explain the purpose of the analytical.

Taxi Trips as a DTFMC, Revisited

Once again, here is our adjacency matrix that we use to build a Discrete-Time Finite Markov Chain.

adj_rows = dict(
    airport= [  0, 0.2, 0.7, 0.1],
    hospital=[0.4,   0, 0.6,   0],
    hotel=   [0.4, 0.2,   0, 0.4],
    mall=    [0.2, 0.1, 0.7,   0],
)
adj_matrix = pd.DataFrame.from_dict(adj_rows,
                                    orient='index',
                                    dtype=float,
                                    columns=list(adj_rows.keys()))
model = mc.MarkovChain(adj_matrix)

From this model, we can use the analysis tools of the chain to get the stationary distribution, which is the theoretical long-running distribution of states that would appear in chains. We’ll wrap it in a nicer DataFrame to match the adjacency matrix:

data = {
  # In PyDTMC 'stationary_distributions' is an alias of 'pi';
  # You'll see pi symbols in mathematical treatments of Markov Chains
  'probability': model.pi[0]
}
distribution: pd.DataFrame = pd.DataFrame(data=data,
                                          index=adj_matrix.index.copy())
distribution.index.name = 'state'

We can visualize this as a histogram:

distribution.plot(kind='bar', color='mediumorchid')

Going back to our earlier questions:

1. “How do you know if the data generated actually is a reasonable facsimile of the training data?” The answer revolves around deciding if our generated data has a distribution similar enough to the stationary distribution. The generated data should have a histogram roughly like what we see above.

2. “Are there any inputs to the process, other than the training data, that could influence whether that generated data is a reasonable facsimile?” Here the answer will be determining if the initial state or the seed or the number of steps of progress cause the generated data to not look like the stationary distribution.

3. “How do you know when you have enough data from a Markov Chain?” It is possible that some of the concerns of the second question above go away when enough data is generated. We will examine how much is enough for us to see approximately the correct histogram.

Lining Up The Pieces

Piece 1: “Create runs of output data. These are sequences of states.”

PyDTMC gives us a starting point for this. It takes a little tidying to map between state names and state numbers (row offsets in the adjacency matrix):

from_state_mapping =\
    {state: index_pos for index_pos, state in enumerate(adj_matrix.index)}
from_index_pos_mapping =\
    {index_pos: state for index_pos, state in enumerate(adj_matrix.index)}
simulated = model.simulate(steps=5,
                           initial_state=from_state_mapping['hotel'],
                           seed=42,
                           output_indices=True)
states = [from_index_pos_mapping[index_pos] for index_pos in simulated]

That gets us:

The result is our initial state, 'hotel', plus the 5 randomly-generated next states that we asked for.

We will build on that and create a Pandas DataFrame suited to how we will analyze generated runs.

Piece 2: “Analyze the probability distribution of the states in those runs.”

Ok, this one takes a little more coding work. We need to:

• Generate the simulated run.

• Treat each value in that run as a DataFrame row corresponding to a step.

• Convert the string name of the step into separate column, so 'airport' is counted in a different column than 'mall'; a marker value of 1 means present, 0 means absent.

• Make running aggregates at each step, so that at step 3 we have the per-state aggregate from the initial state + step 1 + step 2 + step 3.

• Use those aggregates to generate the probability distribution implied at that step. This means adding the aggregates to get a total number of all states seen, then dividing the aggregates by the total.

Here we go!

# 'states' are from the simulated run above
running_df = pd.DataFrame({'state': states})
running_df.index.name = 'step'
# count how many of each state we used as of each step in the process
state_count_df = pd.get_dummies(data=running_df, 
                                prefix='', 
                                prefix_sep='').cumsum()
# add any missing states because we may not have used them all yet
# and cumsum would drop missing states from the result
model_states = adj_matrix.index.tolist()
missing_columns = set(model_states) - set(state_count_df.columns)
if missing_columns:
    for state in missing_columns:
        state_count_df[state] = 0
        # debugging is easier it we keep the state column-order canonical
        state_count_df = state_count_df[self.model_states]
# add the count aggregates to our running log
running_df = pd.concat([running_df, state_count_df], axis=1)
# turn the current row into its implied distribution
distribution_df = state_count_df.div(state_count_df.sum(axis=1), axis=0)
distributions = [row.to_frame() for _, row in distribution_df.iterrows()]
for distribution in distributions:
    # make it look like the structure of the stationary distribution
    distribution.index.name = 'state'
    distribution.rename(columns={0:'probability'}, inplace=True)
running_df['distribution'] = distributions

The outcome is a running total of the appearances of each state. The distribution column looks a little messy but that is because each value is itself an entire DataFrame object, which will come in handy later.

Piece 3: “Make analysis repeatable by controlling random numbers via seeds.”

We’ll be doing a decent number of runs where we want to extract the signal of what we are attempting from the surrounding noise of randomness. Because of that I decided to use random numbers generated by the Python secrets module so that I would be tapping whatever entropy backed /dev/urandom. The seeds are then saved to a Parquet file for re-use.

class SeedHandler:
    
    def __init__(self, *, num_seeds, bits_per_seed, seed_location):
        self.num_seeds = bits_per_seed
        self.bits_per_seed = bits_per_seed
        self_seed_location = seed_location

    @property
    def seeds(self):
        if not os.path.exists(self.seed_location):
            seed_values = self._make_seeds()
            self._write_seeds(seed_values=seed_values)
        else:
            seed_values = self._read_seeds()
        return seed_values

    def _make_seeds(self):
        values = []
        for i in range(self.num_seeds):
            value = secrets.randbits(self.bits_per_seed)
            values.append(value)
        return tuple(values)

    def _read_seeds(self):
        seed_df = pd.read_parquet(path=self.seed_location,
                                  engine='pyarrow')
        seed_ds = seed_df.astype(dtype=dict(seed=self.numpy_dtype))['seed']
        return tuple(seed_ds)

    def _write_seeds(self, *, seed_values):
        seed_ds = pd.Series(data=seed_values, name='seed')
        seed_df = seed_ds.to_frame()
        seed_df.to_parquet(path=self.seed_location, engine='pyarrow')

There’s more code in the Jupyter notebook, but the above shows the essence.

Piece 4: “Examine sensitivity of the runs to the choice of seed used.”

The resulting seeds will be supplied to every call of simulate(), which will be handled within the function that generates the states and builds running_df. We just need to augment that DataFrame with a 'seed' column.

for seed in seeds:
    running_df =\
        running_distributions(steps=steps,
                              initial_state=initial_state,
                              seed=seed)
        # get a 'step' column from the existing index
        running_df.reset_index(drop=False, inplace=True)
        # and add the seed
        running_df['seed'] = seed

That gets us two out of three pieces of what will become our multi-index.

Piece 5: “Use all the possible initial states as initial states.”

And now we get our third piece of the multi-index. We need another loop around the code above so we can supply different choices of initial_state, add a column to track it, build an index from the three parts, and pd.concat() the running_df from each (initial_state, seed) combination.

index_columns = ['initial_state', 'step', 'seed']
across_states_and_seeds_df = None
for initial_state in model_states:
    for seed in seeds:
        running_df =\
            running_distributions(steps=steps,
                                  initial_state=initial_state,
                                  seed=seed)
            running_df.reset_index(drop=False, inplace=True)
            running_df['initial_state'] = initial_state
            running_df['seed'] = seed
            # create the multi-index
            running_df.set_index(index_columns, drop=True, inplace=True)
            if across_states_and_seeds_df is None:
                across_states_and_seeds_df = running_df
            else:
                across_states_and_seeds_df =\
                    pd.concat([across_states_and_seeds_df, running_df],
                              axis=0)

Piece 6: “Assess how long it takes the model to forget the state it started with.”

This will be examined in two different ways.

1. Analytically.

2. From the data.

The analytical part is supplied by a PyDTMC model property:

model.mixing_rate

I believe this property is misnamed. The implementation appears to be what is usually called the “relaxation rate” or “convergence rate”. This is the rate at which deviations from the stationary distribution shrink as more steps are added to the chain generated. It’s derived from the same eigenvalue information as the spectral gap.

To get an estimate of how many steps that implies — the mixing time — before a desired amount of convergence is achieved, we would have to adjust for how close we want the generated distribution to be to the stationary one.

This is a bit opaque since we aren’t really indicating what we are measuring via that rate, but as a first approximation we’ll use epsilon=0.001:

epsilon = 0.001
taxi_structure.model.mixing_rate * math.log(1 / epsilon)

That’s our analytical reference point, we want to see how the chain distributions look when we have the 19th step after the initial state.

The reason I picked an epsilon that small is because we haven’t discussed what kind of distance should matter. I did some digging via the PyDTMC source code, Wikipedia, and several Monte Carlo texts to get an answer.

• If I were to use the PyDTMC mixing_time() method I believe it uses the square root of the dot product between the stationary distribution and the distribution of the generated chain.

• The more usual mathematical definition is based on the greatest absolute difference between the two distributions, called the “Total Variation Distance” (TVD). It’s definitely the one I find discussed in material on convergence.

The second is closer to what we want, but not quite. We care about:

• Average closeness: what PyDTMC provides.

• Worst-case closeness: what TVD provides.

• Similarity of shape between the two distributions: which neither provide.

We want all three, and getting the third with a small value gets you the first two.

As a reminder of what we’re trying to achieve in this piece, we want to know when the choice of initial state no longer has a meaningful bias on the distribution of the chain. By comparing the distribution of the generated chain to the stationary distribution, when they are similar enough we are declaring that there is no longer any level of bias from the initial state that we would care about.

For this reason I added a different measure of closeness to the code, based upon the Jensen-Shannon Distance (JSD). It is histogram-friendly by being symmetric (comparing A to B is the same as comparing B to A), and robust to empty bins:

eps: float = 1e-9
adjusted_sample = (sample + eps) / (sample + eps).sum()
distance = sp_sd.jensenshannon(p=stationary_distribution,
                               q=adjusted_sample)[0]

The adjustment shown deals with a limitation in JSD when the two distributions get very close and you’re faced with the limit of dividing a small value by another value that is approaching zero.

There is a bit more code to getting the distance behaving when faced with numerical error, which is in the Jupyter notebook. It’s a little bit of a hack but the alternative would be to implement JSD from scratch to deal with a special case where numerical error triggered sqrt(negative number) instead of sqrt(zero). I found in practice a small number of random seeds tripped over that case when generating large sequences.

Assembling the Pieces

Now that we have a notion of distance, we need to add it to our computation of the running results.

We will want to examine that distance at each step and see if it is mostly declining, and what our measure is at the 19th step we derived analytically.

# 'states' are from the simulated run above
running_df = pd.DataFrame({'state': states})
running_df.index.name = 'step'
# count how many of each state we used as of each step in the process
state_count_df = pd.get_dummies(data=running_df, 
                                prefix='', 
                                prefix_sep='').cumsum()
# add any missing states because we may not have used them all yet
# and cumsum would drop missing states from the result
model_states = adj_matrix.index.tolist()
missing_columns = set(model_states) - set(state_count_df.columns)
if missing_columns:
    for state in missing_columns:
        state_count_df[state] = 0
        # debugging is easier it we keep the state column-order canonical
        state_count_df = state_count_df[self.model_states]
# add the count aggregates to our running log
running_df = pd.concat([running_df, state_count_df], axis=1)
# turn the current row into its implied distribution
distribution_df = state_count_df.div(state_count_df.sum(axis=1), axis=0)
distributions = [row.to_frame() for _, row in distribution_df.iterrows()]

# ---> changes for distance are below
distances = []
for distribution in distributions:
    # make it look like the structure of the stationary distribution
    distribution.index.name = 'state'
    distribution.rename(columns={0:'probability'}, inplace=True)
    distance = distance_from_stationary(sample=distribution)
    distances.append(distance)
running_df['distance'] = distances
running_df['distribution'] = distributions

Now we will have that distance and its corresponding distribution available to us for visual comparison to the stationary distribution.

First, let’s look at a run with 200 seeds and 10 steps. We’ll keep the number of steps small this time in order to make it easier to examine details of the image.

Notice how at at the start (step 0), where all we have is the initial state, the two initial states that are closest to the stationary distribution are 'hotel' then 'airport'. This makes sense, as these are also the two states with the highest probabilities in the stationary distribution; 'hotel' is the highest probability and by distance measure it is the closest. At least JSD as the distance measure isn’t showing something entirely off-base right from the start.

As the run progresses, the median outcomes for each initial state converge, and by step 7 we’re seeing they don’t differ much from each other. The rest of the run going forward won’t likely be about specific initial states, but just the overall challenge of any state to result in data that closely approximates the stationary distribution.

Unfortunately we may be jumping the gun a bit at 7. Let’s look at seed sensitivity. We can plot the interquartile ranges on each step after the initial state:

The per-seed results are still jumping around quite a bit. This isn’t hugely surprising. The number of states is effectively a sample size, and when the sample size is something like 7 then it doesn’t take much to perturb the range of distances produced.

We can get an ever better look at this with initial states in separate plots.

So not only are the results variable per seed, the variability varies by initial_state. Clearly 7 steps is too soon to declare we’ve moved beyond bias.

We had our analytic estimate of 19 steps. Let’s take the analysis out to 30 steps so we can see 19 in a broader context.

The region around 19 steps looks quite stable. The variability is low, and low across all initial_state values. The rate of convergence is slowing down a lot, but that’s probably not a surprise. As a reminder from the previous article, the spectral gap being closer to 0 than 1 suggested convergence could be sluggish:

model.spectral_gap

Time to take a look at the actual histograms at step 19 and see what they tell us.

Not too bad. The estimated convergence seems to have relevance. Since this batch of data went out to 30 steps, we can compare to that.

Slightly better, but not game-changing. The important thing is that we aren’t seeing any meaningful bias based on what the initial state was, and as earlier plots showed the IQR variance based on the seed choice was narrowing as well.

Summary

• We wanted to generate data with this model and have a reasonable expectation that the data looked similar to what it was trained on. That is workable at 19 steps and would get progressively better over time.

• There would be no real need to worry about a better or worse choice of initial state or using data generated across multiple seeds, neither initial-state bias nor seed variance are problematic for chains at least 19 steps long.

• Our use of the Jensen-Shannon Distance appears safe. This isn’t really enough material to critique it versus other distance measures, but that wasn’t the goal. What we wanted was reasonably-similar histogram structure, which is what we have.

• The estimate of a mixing time (number of steps until viable convergence) lined up with the empirical results, which actually was a nice surprise. Going into the analysis I was skeptical that we would see decent histogram shape that early in the sequence.

References

• Jupyter notebook for this article.

• NetworkX API reference.

• Pandas API reference.

• PyDTMC PyPi.

• Häggström, Olle, “Stationary distributions,” in Finite Markov Chains and Algorithmic Applications. London Mathematical Society Student Texts, vol. 52, Cambridge University Press, 2002, pp. 28-38.

• Brémaud, Pierre, “Convergence Rates,” in Markov Chains: Gibbs Fields, Monte Carlo Simulation and Queues, Texts in Applied Mathematics, vol. 31, Springer Cham, 2020, pp. 289–329.

• “Total variation distance of probability measures,” in Wikipedia, accessed 2025-08-28, permanent link (revision) <https://en.wikipedia.org/w/index.php?title=Total_variation_distance_of_probability_measures&oldid=123456789>.

• “Markov chain mixing time,” in Wikipedia, accessed 2025-08-28, permanent link (revision) <https://en.wikipedia.org/w/index.php?title=Markov_chain_mixing_time&oldid=1233574639>.

The Experimentalist : Markov Chain Convergence in Python © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

The brief segue into finite graphs in the previous article has paved the way for developing a little intuition on Markov Chains and working with them in Python. We will want this material for future data engineering explorations.

The concept of a Markov Chain is easy enough:

• You have a collection of states.

• There are probabilities of transitioning from the current state to various future states.

• There is no memory behind those probabilities; if you know the current state then you know the probabilities of transitioning next to other states.

Prev: Finite Graphs and NetworkX | Next: Markov Chain Convergence in Python

We’re going to simplify the variety of Markov Chains we’ll be working with.

• We will only discuss chains with a finite number of states.

• Transitions from one state to the next happen in discrete time steps, not over the flow of continuous time. This lets us organize discussions in terms of time steps t0, t1, …

With those clarifications, what we’ll have are Discrete-Time Finite Markov Chains.

The use of the word ‘chain’ is intended to communicate that we can keep repeating this process of random state transitions over and over again. That assumes there are no states without any outgoing transitions, which are called absorbing states, which can terminate some randomly-generated walks.

The Graph Connection

Remembering our previous discussion on graphs, a Directed Finite Graph has:

• Nodes.

• Edges between pairs of nodes, and those edges are ordered from source to target.

• When an edge leaves a source node, that is an outedge.

• When an edge arrives (the arrow-head in a picture points at) a target node, that is an inedge.

• It is something useful to assign attributes like a numeric weight to edges.

We’ll add to this a little bit of simplified statistical terminology.

• A probability is a real number p where 0 ≤ p ≤ 1.

• A discrete probability distribution is a finite collection of probability values that add up to 1.

• If we have estimated or assigned probabilities to a collection of possible events, and those probabilities form a probability distribution, we refer to that collection of events as being stochastic.

And now we’re ready to characterize our Discrete-Time Finite Markov Chains (DTFMCs to save typing) in terms of Directed Finite Graphs (DFGs).

• States in a DTFMC are nodes in a DFG.

• If it is possible to transition between two states a and b in a DTFMC, then that is represented as the edge (a, b) in a DFG.

• The probability of such a transition from state a to state b in a DTFMC is represented as the weight of the edge (a, b).

• In an adjacency matrix for the DFG, each row corresponds to a source state and each column corresponds to a target state. All non-zero values will be the weights (probabilities of a possible state-to-state transition).

• For every node (state) in the DFG, its collection of weighted outedges (representing state transitions with non-zero probability) is stochastic.

That last two points tell us that when the DFG for a DTFMC is represented as an adjacency matrix, each row will either sum to:

• 0 if and only if that state has no outedges.

• 1 if and only if that state has at least 1 outedge.

This representation of a Markov Chain is sometimes referred to as row-stochastic or row-normalized. It would be possible to construct a matrix that instead was column-stochastic or column-normalized; in that case we would be seeing the probability distribution for how possible source states could transition to a target state. Row-stochastic is for understanding how a specific source leads to many potential targets, Column-stochastic is for understanding how many potential sources lead to a specific target. In this article we’ll be working with row-stochastic representations.

NetworkX Example

Consider the following taxi trip model as a Markov Chain:

• States are destinations.

• Edges are trips between destinations.

• Weight probabilities are the odds of going from one destination to another.

Our specific data will be

• From the airport, 70% of trips go to the hotel, 20% the hospital, 10% the mall.

• From the hospital, 60% of trips go to the hotel, 40% the airport.

• From the hotel, 40% of trips go to the mall, 40% the airport, 20% the hospital.

• From the mall, 70% of trips go to the hotel, 20% the airport, 10% the hospital.

As shown in the previous article, we can represent such a weighted DFG as an adjacency matrix:

adj_rows = dict(
    airport= [  0, 0.2, 0.7, 0.1],
    hospital=[0.4,   0, 0.6,   0],
    hotel=   [0.4, 0.2,   0, 0.4],
    mall=    [0.2, 0.1, 0.7,   0],
)
adj_matrix = pd.DataFrame.from_dict(adj_rows,
                                    orient='index',
                                    dtype=float,
                                    columns=list(adj_rows.keys()))
graph = nx.from_pandas_adjacency(adj_matrix, create_using=nx.DiGraph)

We can use NetworkX to describe the appearance of the graph. First we describe the nodes and node labels:

node_colors = dict(
    airport='purple',
    hospital='red',
    hotel='blue',
    mall='darkgreen',
)

for node_name, node_color in node_colors.items():
    nx.draw_networkx_nodes(graph,
                           pos=node_positions,
                           nodelist=[node_name],
                           node_color=node_color,
                           **node_params())
label_colors = {node_name: node_name for node_name in node_colors}
nx.draw_networkx_labels(graph,
                        pos=node_positions,
                        labels=label_colors,
                        **node_label_params())

Then we describe the edges and edge labels:

for node in graph:
    out_edges_view = typing.cast(nx.reportviews.OutEdgeView, graph.out_edges)
    nx.draw_networkx_edges(graph,
                           pos=node_positions,
                           edgelist=out_edges_view(node),
                           edge_color=node_colors[node],
                           **edge_params())
    for src, target, data in out_edges_view(node, data=True):
        edge_label = {(src, target): data['weight']}
        nx.draw_networkx_edge_labels(graph,
                                     pos=node_positions,
                                     edge_labels=edge_label,
                                     font_color=node_colors[node],
                                     **edge_label_params())

This renders as:

The outedges of each node are colored to match their source. The weights of all edges of the same color will add up to 1, reflecting their stochastic nature.

Now we have a visualization of a Markov Chain. From any state (node), in one discrete time step travel will happen on one of the correspondingly-colored outedges to a target node, with probability of that event as shown by the weight on the edge.

Counting DFG Walks

Going back for a moment to our graphs, we will introduce an additional concept.

• A walk is a sequence of connected edges in a graph.

• If the graph is a directed graph, edges only connect if each intermediate node functions first as a target for an edge, and then as a source for the next edge. Connections must respect directionality. If a graph has a directed edge (a, b), and another edge (c, b), you cannot directly connect those edges in a walk because they are going in opposite directions.

Given a particular DFG, between any two nodes, how many walks can we identify of length 2? Length 3? The answer is given by raising the unweighted adjacency matrix to the corresponding power. In other words if A is the adjacency matrix of a DFG, then the matrix product A * A computes the number of paths of length 2, and so on.

This can be illustrated with NetworkX and an edge list for weather transitions:

edge_rows = [
    dict(src='cloud', dest='cloud'),
    dict(src='cloud', dest='hail'),
    dict(src='cloud', dest='rain'),
    dict(src='cloud', dest='snow'),
    dict(src='cloud', dest='sun'),
    dict(src='hail', dest='cloud'),
    dict(src='hail', dest='hail'),
    dict(src='hail', dest='rain'),
    dict(src='rain', dest='cloud'),
    dict(src='rain', dest='hail'),
    dict(src='rain', dest='rain'),
    dict(src='rain', dest='snow'),
    dict(src='snow', dest='cloud'),
    dict(src='snow', dest='rain'),
    dict(src='snow', dest='snow'),
    dict(src='sun', dest='cloud'),
    dict(src='sun', dest='sun'),
]
edge_list_df = pd.DataFrame(edge_rows)
graph = nx.from_pandas_edgelist(edge_list_df,
                                source='src',
                                target='dest',
                                create_using=nx.DiGraph)
adj_matrix = nx.to_pandas_adjacency(graph, dtype=int)
adj_matrix

The last step converted the edge list to an adjacency matrix as a Pandas DataFrame:

This data restricts sun to not spontaneously transitioning to hail or snow or rain. We would need a walk of length at least 2 to have a chance to transition through cloudy weather first. We can accomplish that with matrix multiplication as a “dot” product:

adj_matrix @ adj_matrix

We can see, correctly, that there is only 1 walk of length 2 that starts with sun and ends with snow.

Also correct is the number of ways to start with sun and end with cloudy weather. There are 2: sun → sun → cloud and sun → cloud → cloud.

We don’t need to work with the adjacency matrix and figure out how many dot products to type in for higher powers. NetworkX provides a function to specify the desired power directly. It returns a dict of dicts, which we can make prettier as a Pandas DataFrame:

pd.DataFrame(nx.number_of_walks(graph, 2))

The results are the same as the calculation on the adjacency matrix.

The rest of the discussion will now return to exploring the taxi data.

Measuring DTFMC Walk Probabilities

In the DFG walk computations just shown, the weights in the adjacency matrix were all 1 to denote the directed edge. However, this same computation also works when the weights are other numbers, like probabilities. What is computed is the probability-weighted sum of those walks. This means:

• Compute the n’th power of a Markov Chain.

• We will get the probability of starting in a particular state a, and ending up in another state b. If there is no way to get from a to b then the value is 0.

• This is the probability to have that outcome after exactly n steps in the walk.

• The probability does not count the possibility of getting from a to b in fewer or greater than n steps.

Let’s try this for the taxi DTFMC defined earlier. The Pandas DataFrame for that adjacency matrix is:

We can compute the 3rd power of that graph. We’ll need to use the Pandas dot product for this, as NetworkX doesn’t understand weighting.

(adj_matrix @ adj_matrix) @ adj_matrix

As we can see, starting at the mall and returning again on the third trip has an 11.2% probability. Also note that no entries are zero. We’ll discuss the implications shortly.

Markov Chains and PyDTMC

So far we’ve just been exploring how to represent a Markov Chain as a graph with an associated matrix with per-state probability distributions. We can do more to analyze the characteristics of the model of taxi data that we’ve built.

import pydtmc as mc

model = mc.MarkovChain(adj_matrix)
print(model)

From this we can get a summary report about our model:

DISCRETE-TIME MARKOV CHAIN
 SIZE:           4
 RANK:           4
 CLASSES:        1
  > RECURRENT:   1
  > TRANSIENT:   0
 ERGODIC:        YES
  > APERIODIC:   YES
  > IRREDUCIBLE: YES
 ABSORBING:      NO
 MONOTONE:       NO
 REGULAR:        YES
 REVERSIBLE:     NO
 SYMMETRIC:      NO

The important characteristics for now are that the model is:

• Regular. This is a particularly strong property for a Markov Model to possess, and makes other operations much more approachable. It means that there is a probability distribution of states that all future state transitions will converge to.

• Ergodic. All the states are reachable from each other after a finite number of steps (irreducible), and the long-term distribution of states does not oscillate with periodicity (aperiodic). Above there was a place where it was pointed out that the adjacency matrix raised to the third power had all positive entries: that showed the matrix was recurrent.

• Recurrent: With a single recurrent class (meaning all states are in that class), we have that no matter which state we start in, we will return to it infinitely many times as we traverse more and more time steps.

Regular implies Ergodic, but the reverse is not always true.

Because we are dealing with Finite Markov Models, Recurrent implies Irreducible, and Irreducible implies Recurrent. These relationships would not hold for infinite models.

We can use PyDTMC to get that stationary distribution.

model.stationary_distributions

This is the probability distribution for visiting the states over time. The order of the states here is the same as the order of the rows in the original adjacency matrix, thus we spend the most time at the hotel: 0.4028777.

[array([0.25899281, 0.15107914, 0.4028777 , 0.18705036])]

We can obtain some insight into whether the model finds it easy or difficult to converge to that distribution. One way is by plotting the eigenvalues of the matrix:

mc.plot_eigenvalues(model)

From that we get:

Take notice of the two largest-magnitude eigenvalues. The wide red ring shows the distance between them, called the spectral gap. The closer they are, the more difficult it is for the future probabilities to generate walk that converge to the stationary distribution.

We can find out the spectral gap directly:

model.spectral_gap

• When the value is closer to 1, convergence is faster.

• When it is closer to 0, convergence is slower.

0.31078141211367716

In our case convergence is a little sluggish.

We can get further insight into convergence with Spectral Density.

model.density

This tells us how variable the data is:

0.9166666666666666

• A value near 1 suggests convergence is easier.

• Near 0 indicates convergence is harder, and it would still mean that no matter what the spectral gap may be suggesting.

While it may sound a little counter-intuitive, data that is more variable actually makes convergence easier. When it is less variable then you have some low-probability transitions in the matrix that can keep the model stuck instead of moving freely around all its states.

In the next article we’ll explore what that convergence behavior looks like.

References

• Jupyter notebook for this article.

• NetworkX API reference.

• Pandas API reference.

• PyDTMC PyPi.

The Experimentalist : Markov Chains with NetworkX and PyDTMC © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Many areas of practical data analysis and systems reasoning are grounded in the idea of using a finite graph to represent a problem. Not only does this provide a way to structure data, but the structure itself may suit analysis by a variety of techniques.

We’re going to keep the context very straightforward in order to build intuition over a few articles. The dusty old math texts can stay on their shelves for this one.

Next: Markov Chains with NetworkX and PyDTMC

Finite Graph Terminology

Finite Graph: a finite collection of nodes and edges.

Node: an object in a graph that will have 0 or more edges connecting it to other nodes.

Edge: a line in the graph that connects two nodes. It is acceptable for the same node to be on both ends of a connection.

import networkx as nx
import matplotlib.pyplot as plt
import pandas as pd
import random
import typing

graph: nx.Graph = nx.Graph()
graph.add_nodes_from(['a', 'b', 'c', 'd', 'e'])
graph.add_edges_from([('a', 'b'), ('a', 'c'), ('b', 'd'),
                      ('c', 'c'), ('c', 'e'), ('e','b')])
node_positions: NodeLayout  = nx.circular_layout(graph)
nx.draw(graph, pos=node_positions, **draw_params())

Tuples like ('a', 'b') are the nodes at either end of that edge. As this is an undirected graph, ('b', 'a') would achieve the same.

The draw_params() function is a convenience for boilerplate reduction on setting rendering parameters across multiple examples.

def override_params(params: dict[str, typing.Any],
                    **kwargs: typing.Any) -> None:
    for key, value in kwargs.items():
        params[key] = value

def draw_params(**kwargs: typing.Any) -> dict[str, typing.Any]:
    params: dict[str, typing.Any] = dict(
        arrows=True,
        arrowsize=17,
        edge_color='darkgray',  # colors graph edges
        edgecolors='darkgray',  # colors border pixels of nodes
        font_color='black',
        font_size=15,
        font_weight='semibold',
        node_color='lightgray',
        node_size=525,
        width=2.5,
        with_labels=True,
    )
    override_params(params, **kwargs)
    return params

The complete code (available in a Jupyter notebook) renders:

Display of node labels is optional, which is a useful feature for large graphs. Just add with_labels=False to nx.draw() via the helper method draw_params():

nx.draw(graph, pos=node_positions, **draw_params(with_labels=False))

We can extend this idea a little further. We can give those edges a sense of direction.

Directed Finite Graph Terminology

Node: as defined previously.

Edge: a line in a graph that has a source node and a target node. When drawn, an arrow head points to the target node.

If the edges aren’t directed we call it an Undirected Finite Graph (or Undirected Graph for short), otherwise a Directed Finite Graph (again, Directed Graph for short).

Nodes in a directed graph now gain a bit more structure:

• Outgoing Edges: the directed edges connected to a node that have that node as a source. Also called “outedges” in some math texts.

• Incoming Edges: the directed edges connected to a node that have that node as a target. Also called “inedges” in some math texts.

Representing edges is no different, except now the order of the identifier pairs matters when it didn’t matter for undirected edges. The first member of a pair is the source, the second is the target. The only substantial difference is the change in the NetworkX class used when defining the graph. We change the class from nx.Graph to nx.DiGraph:

graph: nx.Graph = nx.DiGraph()
graph.add_nodes_from(['a', 'b', 'c', 'd', 'e'])
graph.add_edges_from([('a', 'b'), ('a', 'c'), ('b', 'd'),
                      ('c', 'c'), ('c', 'e'), ('e','b')])
node_positions: NodeLayout  = nx.circular_layout(graph)
nx.draw(graph, pos=node_positions, **draw_params())

With this change the rendering becomes:

Alternate Representations

It can be mathematically useful, particularly for directed graphs, to think of them as corresponding to a matrix. Each row of a matrix corresponds to a source node, and each column corresponds to a target node. This is referred to as an Adjacency Matrix.

adj_rows: dict[typing.Hashable, list[typing.Any]] = dict(
    a=[0, 1, 1, 0, 0],
    b=[0, 0, 0, 1, 0],
    c=[0, 0, 1, 0, 1],
    d=[0, 0, 0, 0, 0],
    e=[0, 1, 0, 0, 0]
)
adj_matrix = pd.DataFrame.from_dict(adj_rows,
                                    orient='index',
                                    dtype=int,
                                    columns=list(adj_rows.keys()))
graph = nx.from_pandas_adjacency(adj_matrix, create_using=nx.DiGraph)
node_positions: NodeLayout  = nx.circular_layout(graph)
nx.draw(graph, pos=node_positions, **draw_params())

We use the Pandas trick of providing a keyed dictionary for rows and specifying the orientation. Thinking in terms of a known source having various targets is more intuitive than what column-specified data would force on us: to think in terms of one target having multiple sources. It also allows the visual aid of the text for populating adj_rows to correspond to how we have been taught to conceive of matrices. All we need to do is inform Pandas what the column keys are, which in this case is the same as the row keys: columns=list(adj_rows.keys()).

The output is the same as before, which was the goal. This representation can be convenient when the existing understanding of a graph is organized in terms of outgoing edges, because all the outgoing edge connections for a node form the matrix row for that node.

The number 1 in the matrix indicates a directed edge from the source (the row index) to the target (the column index). Using 0 and 1 is a traditional starting point for adjacency matrices because it supports some useful linear algebra, but there is no constraint that only the number 1 be used to indicate a connection. Any non-zero value can represent, for example, weight or length.

The disadvantage of this representation, at least with NetworkX, is that the only information this can capture are node names, edge directions, and a single edge weight. Adjacency matrices for NetworkX do not support the idea of structured objects so we couldn’t use one to specify multiple per-edge attributes. For that we need Edge Lists.

edge_rows = [
    dict(src='a', dest='b'),
    dict(src='a', dest='c'),
    dict(src='b', dest='d'),
    dict(src='c', dest='c'),
    dict(src='c', dest='e'),
    dict(src='e', dest='b'),
]
edge_list_df: pd.DataFrame = pd.DataFrame(edge_rows)
graph = nx.from_pandas_edgelist(edge_list_df,
                                source='src',
                                target='dest',
                                create_using=nx.DiGraph)
node_positions: NodeLayout  = nx.circular_layout(graph)
nx.draw(graph, pos=node_positions, **draw_params())

The situation has changed a bit:

• Each edge must be specified as a dictionary with consistent keys indicating the source and target of the edge.

• The edge list is converted into a Pandas DataFrame, and the dictionary keys become the column names.

• We use nx.from_pandas_edgelist() to convert the DataFrame into a graph, which requires specifying the keys that were used for the source and target.

The final rendering though, at least for this data, is unchanged:

Now we can push this further to add more attributes to the edge data:

def random_color_channel() -> float:
    if not hasattr(random_color_channel, 'rand'):
        random_color_channel.rand = random.Random(42)
    return random_color_channel.rand.uniform(0.5, 1)

def random_rgb_color() -> tuple[float, float, float]:
    return random_color_channel(),
           random_color_channel(),
           random_color_channel()

edge_rows = [
    dict(src='a', dest='b', weight=1, color=random_rgb_color()),
    dict(src='a', dest='c', weight=3, color=random_rgb_color()),
    dict(src='b', dest='d', weight=2, color=random_rgb_color()),
    dict(src='c', dest='c', weight=1, color=random_rgb_color()),
    dict(src='c', dest='e', weight=2, color=random_rgb_color()),
    dict(src='e', dest='b', weight=1, color=random_rgb_color()),
]
edge_list_df: pd.DataFrame = pd.DataFrame(edge_rows)
graph = nx.from_pandas_edgelist(edge_list_df,
                                source='src',
                                target='dest',
                                edge_attr=['weight', 'color'],
                                create_using=nx.DiGraph)
edge_weights = nx.get_edge_attributes(graph, 'weight')
edge_colors = nx.get_edge_attributes(graph, 'color').values()
node_positions: NodeLayout  = nx.circular_layout(graph)
nx.draw(graph, pos=node_positions, **draw_params(edge_color=edge_colors))
nx.draw_networkx_edge_labels(graph, pos=node_positions,
                             **edge_label_params(edge_labels=edge_weights))

Now we’re adding some complexity:

• Edges have two attributes: weight and color.

• The color is an RGB triple where each channel is a float in the range [0, 1].

• When using nx.from_pandas_edgelist() to produce a graph we have to tell it about the per-edge attributes: edge_attr=['weight', 'color']

• NetworkX doesn’t do much with complex edges so we have to extract the data as edge_weights and edge_colors to use later as parameters.

• The nx.draw() function has limited smarts on some forms of rendering, so we include use of nx.draw_networkx_edge_labels() to render the weights as values on each edge.

• The way nx.draw() and nx.draw_networkx_edge_labels() share an understanding of node locations is via the node_positions layout object. In the previous examples it was only used once per rendering so its role was less apparent, but here we see it used to layer on rendering details. The object provides node coordinates in the rendering. I added a Python type alias to cover the kinds of data passed back by various layout algorithms:

NodeLayout: typing.TypeAlias = dict[typing.Hashable, list[float]]

Now we get a much richer rendering with per-edge weight labels and per-edge colors:

In future articles I expect to need to use various analytic techniques on graphs. Even without that, NetworkX can help with rendering images that support an explanation for how data is being transformed or analyzed.

References

• Jupyter notebook for this article.

• Matplotlib API reference.

• NetworkX API reference.

• Pandas API reference.

Edits

• 2025-08-22: Added terminology for inedges and outedges.

The Experimentalist : Finite Graphs and NetworkX © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

There are times when you need to chase down database client connections in PostgreSQL. Examples of the issues that arise are:

• Having too many idle connections, which can pointlessly throttle database performance due to the backend process resources allocated to each.

• Having too many connections in total relative to the semaphore array configuration of the database, resulting in new connections being denied.

• Holding transaction-scoped locks for too long or gaining them in a problematic order, which can create errors or deadlocks in other operations that need some of those same locks for work to proceed.

• Holding transactions open too long regardless of any locking issues, which can thwart maintenance operations like vacuums and backups.

• Burning through transaction ids at an unusual rate, which might trigger id wrap-around too fast for vacuum operations to keep up and thus cause the database to go into recovery mode.

• Connection activity may be unexpected or suspicious, in which case you may have a security incident to investigate.

Prev: PostgreSQL 17 on Vagrant and VirtualBox

The TCP/IP Story

When we set up server processes we often find ourselves having to configure one or more ports for access. We become very aware of server-side addressing via IP or DNS name plus a port. What is easy to forget is that the client end of the connection also has addressing information associated with it.

With each client-server interaction:

• There is a client process, and that process has a process id (PID).

• There is a server process, which also has a PID.

• Requests originate from the client and are sent as TCP/IP packets to the server.

• Responses originate from the server and are sent as TCP/IP packets to the client.

• The client finds the server via an IP address and TCP port. That port is the one we tend to be aware of as software engineers. When we configure a client we tell it the IP address — or indirectly, via a DNS name — and the port.

• The server also finds the client via an IP address and TCP port. We don’t configure the information into the server, so how does it know about the client?

The answer is in the IP and TCP headers for each packet.

• IP headers contain several fields, which include both source and destination IP addresses.

• TCP headers also contain several fields, including both source and destination TCP ports.

• The way a server knows where the client is located is by extracting those fields when a request is received, and using them when it generates a response.

• While the server port resulted from static configuration, client ports are often determined on-the-fly and sometimes referred to as “ephemeral” ports.

There is information missing from these packets: the PIDs. Nothing in the network traffic knows about processes. That association is handled by operating system support which the application processes make use of.

• The client process will use a socket and bind it to the client IP address and ephemeral TCP port.

• The server process will use a socket and bind it to the server IP address and statically-configured TCP port.

Ports and PostgreSQL

The picture changes when you introduce a PostgreSQL service into the equation. Since TCP/IP packets don’t know anything about processes, there is no reason to restrict ourselves to a single server process for the implementation.

PostgreSQL has several processes, but for this discussion we are concerned with two:

• The Postmaster is the process that listens to that statically-configured port known for establishing connections. The default is 5432.

• For each client connection the Postmaster will identify or launch a Backend process to do the work the client is requesting. The Backend is the process that will use the client IP address and ephemeral port for responding with the results.

There are more complicated processing scenarios in PostgreSQL but we’ll ignore them for today. What is shown here accounts for what is typical.

We’re simplifying the picture a bit here as clients and backends can continue this back-and-forth between requests and responses. The Backend is delegated to deal with both once the Postmaster finishes accepting the incoming client connection. The Postmaster will go back to listening for fresh connection attempts.

Note that 5432 is not an entirely arbitrary choice of number. There is an official IANA registration for the service name and port. While you can configure PostgreSQL to use other ports, by sticking with 5432 you will find some network-aware Linux tools will automatically know that “5432” and “postgresql” are interchangeable.

Gathering Client Connection Data

All of this connection and request/response activity isn’t just hidden away in the memory state of the various processes. We can examine it. Some of that examination can happen within PostgreSQL itself, while some needs to be done with O/S tools.

We’ll explore this via the Vagrant/VirtualBox VM from the previous article. Refer to that if you haven’t already spun up your own PostgreSQL sandbox.

# cd to the project directory with the Vagrantfile before proceeding
vagrant up --provision
vagrant ssh

Now we can use a psql session to do some looking around. We’ll use the vagrant database and username as we previously created that as a superuser. As a memory-jogger we just used the trivial password “vagrant” for that sandbox. We’ll get into the details of database authentication and security in future articles.

psql --host=localhost --dbname=vagrant --username=vagrant

You don’t have to specify the dbname as it is the one associated with the username, but it’s worth seeing an example of how to when you need to.

The ability to examine client connection activity is handled in PostgreSQL via the Cumulative Statistics System. Under-the-hood PostgreSQL uses internal tables to gather activity data which you then examine via pre-defined “system views”. We’re going to put one of them to work: pg_stat_activity.

Our psql session is the only client we currently have connected (as shown by the backend_type being '“client backend”), and we can see it there in the response. As we are running on the same host as the database service and connected via “localhost”, our client IP address is correctly 127.0.0.1 as shown for client_addr. The client-side ephemeral port for the connection is 41592 as shown for client_port.

Notice the “pid” column. What is that 10614 process id for? Let’s look at the PostgreSQL processes just like we did in the previous article:

This has one more row than was shown in the previous article. That bottom row is the corresponding backend for our psql session. The “pid” shown in psql thus was for the backend process, not the client process. The ps output even shows the client IP address and ephemeral port that particular backend is handling.

Also compare the first entry of the ps results to our query response. That process doesn’t show in psql at all! That’s because it is the parent process with the Postmaster functionality. It doesn’t connect to the database, it is the thing that knows connections are something to be managed along with the processing resources assigned to them. As it doesn’t connect to directly perform database work, pg_stat_activity doesn’t know about it.

We only have that single psql session so it’s trivial to identify the client. We’re going to increase the difficulty by launching two more clients. One will be in another psql session on the same VM, and the other will be a psql session started from the underlying host that the VM is running on. As a reminder from the last article, we configured pg_hba.conf to allow connections via the mask 10.0.2.0/24 and here you are seeing why: the VM’s virtual network mapped the host to an address in that range.

Now we have three psql “client backend” sessions and no real way to tell between them. How do we figure out which client matches up with which backend? All we have are the backend process ids, but that tells us nothing about each client.

Remember earlier we noted that sockets are what associate processes with IP addresses and ports? That’s how we figure can this out. In the previous article one of the provisioning steps installed the iproute2 package, which provides us with ss.

This shows us both the client and server connections that are on the VM.

• You can ignore the “sshd” rows, those are for the terminal sessions I created.

• sudo ss -tunp ensures we see the sockets and corresponding process data for all users.

• ss -tunp (i.e. no use of sudo, running as vagrant); would have only shown “Process” column details for the client process sockets of the current user.

• sudo -U postgres ss -tunp would have only shown “Process” column details for the backend process sockets. The backend processes run as the postgres user.

• None of these shows the client details for the psql session launched from outside the VM. The only way to know client details is by doing this form of examination on the device corresponding to the client IP address.

• The two entries with “psql” are the clients running on the VM. Each shows the “Local Address:Port” and corresponds to the “client_addr,client_port” columns shown earlier in the pg_stat_activity response. The PID showing is for the client process, not the backend.

• The three entries with “postgres” are the backends running on the VM. Each has the PID that corresponds to the “pid” column shown earlier in the pg_stat_activity response. The “Peer Address:Port” corresponds to the client IP and port that backend will respond to.

As a quick aside, the “postgres” backend for the remote (host) psql client looks a little different than the other two cases. That psql client is not connecting via localhost, it is connecting via an IP address assigned by the VM’s virtual network. We can confirm that “10.0.2.15” is indeed the IP address assigned to the VM:

We are almost there.

• Get the “Local Address:Port” on the “psql” rows.

• Get the “Peer Address:Port” on the “postgres” rows.

• Match them up. In database terms, do a join on (IP, Port) across both subsets.

• Get the “pid” value from the “Process” columns. There will be two per match-up, one for the client and one for the backend.

|-------------|----------------|-----------------|------------|
| Backend PID |   Backend Addr |     Client Addr | Client PID |
|-------------|----------------|-----------------|------------|
|       10614 | 127.0.0.1:5432 | 127.0.0.1:41592 |      10612 |
|       10730 | 127.0.0.1:5432 | 127.0.0.1:47160 |      10737 |
|       10741 | 10.0.2.15:5432 |  10.0.2.2:54092 |          ? |
|-------------|----------------|-----------------|------------|

The final case is a little different because I’m running VirtualBox on Windows. For that we need to run netstat in a terminal window with privilege escalation:

That gives us similar information for the third client, including how the backend connection is being proxied by VirtualBox. The PID for the proxy won’t correspond to anything within the VM and can be ignored.

We now have a client-process-to-backend-process correspondence! Completing the table required using either netstat or ss and having access to the appropriate client host, as identified by “client_addr” in pg_stat_activity. Discovering the client PIDs allows us to investigate those processes on those hosts. We weren’t even constrained to only searching on Linux.

Could you automate all this? Yes, assuming the client hosts are accessible to you. That’s a discussion for another day.

The Experimentalist : Client Connectivity with PostgreSQL © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

In the article series on an exploratory lab I’ll eventually tackle provisioning of container and virtualization-based sandboxes. To tease at the benefits, we’re going to use a Vagrant-VirtualBox setup to install and poke around a running database.

If you aren’t already using them, you can download and install in order:

• VirtualBox → https://www.virtualbox.org/wiki/Downloads

• Vagrant → https://developer.hashicorp.com/vagrant/install

Next: Client Connectivity with PostgreSQL

Project files for the article are available from the newsletter’s GitLab repository:

• Vagrantfile

• provisions/postgresql.sh

Just create a project that maintains the directory structure. Once you’ve done that and installed both VirtualBox and Vagrant:

• cd to the project directory

• vagrant up

This will get the ball rolling. If the install will go slowly, it’ll usually be right after generating the SSH key. That tends to happen if there is some difficulty in mapping hardware functionality that the O/S expects to what VirtualBox is able to provide. I’ve found the Ubuntu 22.04 Bento box to work pretty smoothly. It may take 10-20 minutes to download an O/S image, build a VM from it, and install PostgreSQL, and the install will conclude with:

Any future restarts of the box will be much faster. At this point you have:

• A running Ubuntu 22.04 VM.

• A running PostgreSQL 17 service.

• An Ubuntu login account named “vagrant” you can ssh to.

• An PostgreSQL user named “vagrant” with the password “vagrant”.

• The ability to connect to the database as that user with either a client running within the VM, or from a client on the underlying host operating system.

Let’s take a look at how that came about. The driver of the bus is the Vagrantfile:

This is the file from the GitLab repository but with comments stripped out. Let’s walk through the key parts.

Specify the O/S image you want for the VM:

config.vm.box = "bento/ubuntu-22.04"
config.vm.box_version = "202508.03.0"

PostgreSQL by default listens on port 5432. We want to make it possible for a database client running on the host to connect to the database service within the VM on that same port number:

config.vm.network "forwarded_port", guest: 5432, host: 5432, \
  host_ip: "127.0.0.1"

We specify that the provider is VirtualBox, but if you happen to already use Hyper-V or VMware, this is where you would make the change to use them instead:

config.vm.provider "virtualbox" do |vb|

Finally we have a script to run as root for provisioning the VM once it is ready. Ours sets up PostgreSQL and does other assorted housekeeping:

config.vm.provision "shell", name: "postgresql", \
  path: "provisions/postgresql.sh", privileged: true

All of the real magic happens in the provisioning script. First we set up the firewall and install some utilities that may come in handy later:

Then we install the APT repository for PostgreSQL, and install the service:

The next step is a bit more packed and needs explaining.

We want to configure the database to have an initial user and database schema to work in. Vagrant allows us to re-run provisioning scripts. Those scripts should be idempotent so they don’t fail they next time they are run. A zero-byte file named .configured is created as a marker to note we’ve already been here before.

The work of the provisioning on the first pass consists of:

• Change postgresql.conf to listen for connections coming in from any host. This is done so that it doesn’t just respond to VM clients, it would also respond to the any client the Vagrant configuration allows access to.

• Create a user named vagrant, which for simplicity here I’ve made a superuser just like the default user postgres already is.

• Provide the vagrant user with a password to allow login. Password-based authentication is not the only option, but we’re going for simple.

• Create a database for the new user and give it the same name as that user.

• Change pg_hba.conf to allow the new user to have access. We specify both the loopback address to allow login from within the VM, and an address mask suited to any connections over eth0 with IP addresses assigned to the VM or host.

• Restart the database service once everything is ready. If this was a production instance I would dig into the docs to remind me which postgresql.conf features require a restart versus a config reload. For here this works fine, and I want a chance to confirm the systemd unit for the service in any case.

Some of the steps are wrapped in a heredoc to run as the postgres user. We are using nested heredocs: the outer level to process steps as postgres, and the inner level to specify content to append to the pg_hba.conf file. The entire provisioning script runs as a privileged user by Vagrant because we said so in the Vagrantfile, which for us means as root. The PostgreSQL service runs as postgres.

Finally, the provisioning script cleans up any dangling APT state. If anything installed flagged a need to reboot the host (in this case, the VM) to complete an installation, then it gets handled now.

We now have the ability to ssh into the running VM. We use vagrant ssh for that, because it knows about the key pair that Vagrant generated when the VM was created.

As a final step, we’ll confirm that the database service processes exist:

In the next article we’ll start exploring the service in more detail.

The Experimentalist : PostgreSQL 17 on Vagrant and VirtualBox © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

When you first start setting up the compute nodes you may have no meaningful infrastructure in place. There are steps to the sequence of bootstrapping from nothing and ending up with the nodes in a consistent state. You won’t be able to use PXE to do network-based installations. You also won’t have anything like an LDAP server to integrate via PAM to handle server authentication and authorization. Down the road maybe, but on day one, no.

Prev: Phase 1: Just a Bunch of Storage

When you begin the process you’ll have placed a bootable ISO image on a USB drive for the operating system you’ve chosen. I would recommend preparing a second USB drive with any firmware updates your particular compute node hardware model may be due. Update the firmware first, and then install the O/S via the ISO image. If you want to be extra careful, make sure you have a copy of the existing firmware version before performing the update, in case you have to revert to your starting point.

Make life easier on yourself and put some visible external labels on those USB sticks. If you have multiple compute nodes to set up, you’ll be following the same process over and over. Start taking notes as you go through the setup the first time.

As the installer runs you’ll be asked a number of questions. Some will relate to disk partitioning, for which the previous article will give you guidance. A few will relate to specifying a non-root user, to complement the root user which is always installed. Once the installation completes, you have the starting point from which you need to bootstrap a little security.

The initial security work won’t be all you’ll ever want to do, not by a long shot, but you need a baseline that will support routine remote access and running Infrastructure as Code (IaC) jobs that will progressively establish the full functionality of your platform. The first pass is only what you can’t or shouldn’t avoid. It is not enough to consider the environment as being secure. You’re establishing connectivity, and as part of that connectivity you will need credentials to support authentication.

Managing Credentials

Credentials like SSH keys, TLS certificates, and GPG keys become your first intellectual property assets that you don’t want to lose. The point of them is that you should be unable to perform operations or connect to systems that use those credentials to control access or certify identity. The more secure your platform, the harder it would be to recover functionality if those credentials were lost.

Preserving credentials is one area of functionality for which you should consider using a remote service vendor. Any passwords, keys, and certificates should be immediately preserved in the remote service. Assuming you are doing your cluster work as a solo experimenter, you won’t need to care about establishing any kind of-group permission structure, which will make things easier.

Credential automation is much tougher, and for “Phase 1” you probably need to be realistic about how much of that you can pull off. As an example I only know of one remote credential manager that has substantial Linux Pluggable Authentication Module (PAM) support — Keeper Security — and largely that is to use their Privileged Access Management product. Yes, I know, two entirely different uses of the acronym PAM, as if computer security wasn’t a strange enough world. I haven’t worked with Keeper myself, but from what I can tell the functionality somewhat overlaps on LDAP.

The Bootstrapping

My initial “good enough for now” bootstrapping is to use Bash scripts that I run on each compute node after the O/S install finishes. The complete process looks like:

1. Attach a monitor and keyboard to the uninitialized box.

2. Install the firmware USB, boot the box, then install any firmware.

3. Install the ISO image USB, boot the box again, then install the O/S (which will request the password for the non-root user it creates).

4. On whatever host you use for other work, such as your laptop, create an SSH key pair that will be used for authenticating as that user. Since “you” are “you” all the time, you use the same key pair for all the compute nodes. NOTE AGAIN, this is not a final security posture for the cluster, we’re just bootstrapping.

5. Save both halves of the SSH key to the remote vendor service you selected for preserving credentials, so you don’t lose them.

6. Configure your network router to recognize the MAC address for the new box and assign a static IP. You aren’t running a big company, I wouldn’t sweat DHCP for the experimentation platform until you find you need it, particularly since ssh would complain on every change to an IP address.

7. Using your working (e.g. laptop) host, connect remotely to the compute node via that static IP to supply the user and password. These are the non-root credentials you specified when the O/S was installing.

8. Copy-paste the bootstrapping Bash script that configures the user account.

9. Run the script. You should have designed it to ask you for the public half of the SSH key pair you created, which it will install in ~/.ssh and update ~/.ssh/authorized_keys to know about the new key.

10. If you have root-level changes to make (e.g. to configure sshd and sudo), copy-paste the Bash script for that.

11. Run the second script via sudo.

12. Don’t logout yet, but start a second terminal session and verify that you can login. If not, you have a mistake somewhere and you want to get that fixed before you lose viable access (else you may have to install the node from scratch).

13. As a final step I would recommend completely power-cycling the node. If you’re going to have any problems, you may as well find out now.

There are assorted refinements you can make to the instructions above, like:

• Remove password access to an account once the public SSH key is in place.

• Have separate accounts for human-user access versus automation-user access (a need that will become clearer in a future article).

• Ensure the ~/.ssh directory and file permissions meet the requirements that sshd will enforce.

• Inform the compute node of its intended host name, as it probably won’t pick that up from the router.

• Make the script behaviors idempotent so that re-running them isn’t harmful. Until you’ve going through the process a couple of times you’ll keep adding tweaks as you figure out exactly what bootstrap steps you need.

The options are only constrained by your Bash skills. Later on we’ll be doing proper IaC automation, so the primary purpose of any bootstrapping is to get us just to that stage.

Script Samples

My own bootstrapping is split into files for particular features, which currently are all for Ubuntu 22.04. Here is an example for ensuring git is installed:

and to install pip for Python 3 (which isn’t automatically installed on Ubuntu 22.04):

and to upgrade the Ansible installation (the default for Ubuntu 22.04 is ancient):

I like putting each feature function in a separate script, because then within a user or root bootstrapping script all I need is:

and then I can invoke the functions that make sense for either a non-root or a root user. The function files are just functions so they don’t take action until they are explicitly called.

The Experimentalist : Phase 1: Credentials Management and Bootstrapping © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Planning for storage in your newly-constructed lab has several elements to it:

• Hardware considerations.

• Filesystem considerations.

• Use-case considerations.

Prev: Phase 1: Just a Bunch of Compute | Next: Phase 1: Credentials Management and Bootstrapping

First a little terminology:

• Drive: a physical device you buy and mount in or attach to a computing node.

• Partition: a subdivision of the space available on the drive. There are two partitioning approaches, with GUID Partition Table (GPT) being what you’ll usually see, but Master Boot Record (MBR) partitions may exist on older systems. When picking compute nodes to buy, I would suggest not getting anything so old it doesn’t support UEFI, and any UEFI system should allow GPT. GPT gives you more flexibility on partition creation and supports larger drives.

• Volume: a logical construct that presents available storage space to the operating system. Historically a volume usually fit within a partition, but Logical Volume Management (LVM) can present a volume that spans partitions and drives. It does this in terms of “extents,” which it uses to associate physical to logical storage.

There is some hand-waving simplification in those explanations as partitions can be primary, extended, or logical, but those distinctions apply to using MBR.

Local Physical Storage Use Cases

Each compute node will have storage needs to account for. For simplicity I’ll assume a Linux variant is your goal.

• EFI System Partition (ESP): This provides space for boot loaders and kernel images. I plan a 1/4gb for this, but if you knew you intended to do a lot of experimenting with different kernels you could make it bigger to house more kernel images.

• Swap Partition: You may read online content or hear discussions that advocate against swap due to performance concerns. As a database guy there are reasons why I disagree with the position. You don’t have to allocate a physical partition as you can add swap files later. I prefer to allocate the partition and size it according to the amount of RAM I installed on the compute node. You could use a smaller amount, but the less you allocate the more you need monitoring on swap activity, which is a discussion for future articles. Obviously if you have more than 64gb of RAM, you won’t want to size swap as the entire memory footprint unless you have a strong reason to support a virtual memory space that large.

• Root Partition: This will contain the volume that the boot loader establishes as your running root filesystem. When you size this allow for the fact that package updates, run-time log files, applications, and the runtime needs of those applications will all require storage.

• Other Linux Partitions: While not required, you may decide that for system stability you want some of the application, app data, and logging activity go to a separate physical partition. That would ensure the O/S itself doesn’t fall over due to the root partition filling up. The likely mount points for a volume would be /var or /opt. If you want both in a common partition, LVM can achieve that.

• Maintenance Partition: Depending on the BIOS features, I’ve found it handy to add a small additional partition to write to. Sometimes when working in the BIOS you can record configuration snapshot images to remember what you changed. This relates to the “Make notes and take pictures“ advice from the previous article. When you set up many nodes, sometimes one or two will not behave consistently with the others, and knowing your BIOS setup without having to boot into it can be useful. Maybe give it 1-2 gig if your BIOS saves config snapshots as actual image-formatted (e.g. BMP, PNG) files. You may find that this needs to be formatted as FAT32, which will be reported by “fsdisk -f” on a UEFI/GPT setup as “Microsoft basic data.”

• Data Partitions: If you know you’ll be working with large volumes of data, it is worth planning data partitions. By having separate data partitions you protect the O/S from data filling up the root filesystem. It’s also the usage pattern that may later motivate you to add additional or replace with larger drives. It’s less disruptive to do that with clearly-segregated data than it usually is for a large filesystem muddling everything together.

An obvious question is whether all these partitions should be on the same drive or across multiple drives. This mostly relates to the root filesystem versus everything else, although for database purists there are other scenarios.

The most direct reason for multiple drives is protecting the ability to boot from the root filesystem. There can be performance benefits to multiple drives, although it is easy to overstate the potential for non-RAID performance parallelism unless you are spec’ing out compute nodes with 2 or more CPUs, 2 or more RAID cards, and NUMA memory configuration to channel memory transfers separately. Write traffic across multiple SSDs may hold up better as a performance story without that level of hardware due to how IOPS could get batch flushed to on-SSD SRAM cache.

In the case of some database architectures there are fault-tolerance scenarios around directing Write-Ahead Log (WAL) to separate disks. Again this story holds up better when you have multiple RAID cards, preferably cards with battery backup for ensuring completed flushes on a power outage. The goal here would be to approximate the theoretical concept “stable storage,” which forms the backbone for how we reason about database transactions and database recovery.

Remote Storage Use Cases

It is important to keep in mind that you’re creating an experimental lab platform. You should be able to set up and tear down compute nodes at will, or alter some aspect of their current application provisioning and configuration.

This will push you towards having externalized storage so that any data you care about will live across those major system alterations.

You are also going to run into cases where you want the same data available across multiple compute nodes. Having multiple nodes mount the same remote volume is useful for allowing tasks to run anywhere and be matched up to the data required.

The straightforward solution to remote storage is Network Attached Storage (NAS). I want to dig into NAS solutions more in later articles, but for now the outline is:

• You can either build your own, or buy a NAS. If you’re thinking about lab creation like a small startup business, you probably want to just buy something so you can focus your time elsewhere. Pick a reliable, established vendor and product line.

• You want your NAS to support the protocols you intend to use for file sharing. The two most likely are Server Message Block (SMB) and Network File System (NFS). There are others, but those are what you’ll typically use. Use the most recent stable version of a protocol available.

• The bulk of the storage should be in a RAID configuration that helps preserve your data. I prefer RAID 6 plus a hot spare, because I’m willing to trade the cost of some storage for increased protection against disk failure. Investigate RAID options and pick the configuration that suits your situation.

There are distributed filesystem technologies like Ceph, but as the discussion here is about “phase 1” planning for the lab platform, I don’t think that worth getting into. It is family of options can be pretty heavy-weight as a learning curve and maintenance responsibility. You may only want to head down that path if one of your goals from your cluster was hands-on experience with distributed filesystems.

Economic Considerations

You have three possible optimization outcomes:

• Low cost.

• High reliability.

• High performance.

You can only pick at most 2 out of the 3 if you push to the extremes, but sometimes you can balance a little in the middle. When it comes to storage, no matter the use case, I see absolutely no win in giving up on high reliability. You’re trying to accomplish things, and having storage fail underneath you does not accomplish things.

That leaves you to decide when to aim for low cost, versus when to emphasize high performance.

I lean towards the following:

• The NAS box itself should be capable of high performance. You already have the handicap of data traveling over the network. Just get a good box in the first place.

• The drives in the NAS are where you have have more flexibility. If you want a lot of storage, you really can’t beat the price point of most good-quality HDDs. If you prefer fast storage then you’ll want to get good-quality SSDs. Your bill and final storage capacity will differ between those choices, just don’t buy junk. This does not mean that you must buy drives marketed as “enterprise grade.” The price points on most of those are stupid. If you have a decent RAID setup, the entire point was to allow RAID to be your source of resilience. While I’m personally not a user of hybrid drives, hybrids could be a reasonable option for a NAS.

• For local storage your choice is between HDD and SSD. I use one of each in my compute nodes. The HDD is used for the basic O/S and for the activity that experiences the most churn like swap and logging traffic. The SSD is used for application data. This setup protects the operating system itself from the risk of lockup due to NAND burn on the SSD, or SSDs failing due to bad firmware (several SSD vendors have had issues). My SSD selection criteria is definitely performance and reliability. HDD purchases have emphasized quality, and sometimes you can luck out with finding unused batches of older product lines where you hit a nice sweet spot that balances good quality, very decent (but not best-of-breed) performance, and fair price.

• We won’t get into the meat of disaster recovery during phase 1 — there isn’t any intellectual property to protect yet — but for initial economic planning it is worth considering if you want a spare drive of each kind you use. Both my NAS and the compute nodes use the same model of SSD so I keep one idle SSD in the NAS as a hot spare for a RAID rebuild, but in a pinch I could swap that into a compute node for a failed data drive. I also have a couple of spares of the HDD models used in the compute nodes in case a root volume fails. You’re trying to run a business, and downtime means impaired operations. When you are bootstrapping your lab as a side project, parts delays interleaved with life scheduling may interrupt your plans for weeks.

The Experimentalist : Phase 1: Just a Bunch of Storage © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Time to get down to business. Phase 1 of the plan is the rudimentary layer to building your own home-brew “mad scientist” lab for software engineering experimentation. JBoC — Just a Bunch of Compute — is about deciding the physical portion of the platform.

Prev: Building a SWE Experimentation Platform | Next: Phase 1: Just a Bunch of Storage

I’ll be describing this in terms of what’s most typical for backend and data engineers, but don’t view that as limiting your options. If you’re interested more in the front-end, you might make slightly different choices. If you’ve been dreaming of building a thousand-node Raspberry PI supercluster, have at it! Honestly if some board manufacturer would underwrite the work, I’d do a supercluster in a heartbeat.

The Outline

Money only goes so far. There is benefit to planning purchases much the same way a cash-strapped company might. The heuristic I’m using is:

• 75% of purchases will go to lower-cost hardware.

• 20% will be the mid-tier cost range.

• 5% go to the very infrequent budget-busters you have decided you must say yes to.

The percentages are ballparks in number of units purchased, not in dollars. That 5% top-end units may very well end up 75% of your spend if you’ve decided you absolutely must have that Nvidia H100 card, for example. More commonly it’ll be the monster laptop rig that you talk yourself into once every 5 years.

The goal in experimentation with distributed computing is to have hardware to distribute work over. The per-node computational muscle may not be that important. It’s the distribution of function that you are exercising, not whether each and every node can supply a water-cooled 100 FPS while playing Cyberpunk 2077.

Lower-Tier Nodes

Used, open-box, and unused-but-discontinued hardware is a great way to build out a cluster. Product lines in the range of 3-6 years old can have provide a lot of bang for the buck. We pay an insane premium in order to get the newest thing with a 50% performance bump when operated flat out. Most of the time we aren’t using server compute anywhere near full load, particularly in a home lab.

Think of your lab like you’re setting up a business. Fund the work you are going to do, not the work that somebody, somewhere, in a big wide world might conceive of doing. Someday.

Having headroom for maximum performance is great… if you’re using it. Save that for the high-end 5% of purchases. If it turns out to have been a vanity purchase, at least you’ll understand where the motivation came from.

Try to identify a single hardware model that you will stick to. There are two reasons for this:

1. When you have many different hardware variations, it becomes more challenging to reason about behavior like performance and fault rates. The differences become primary variables instead of your chosen computational activity.

2. Maintenance is harder because you’ll have different issues arise with every kind of different hardware. Firmware patch issues will differ, BIOS configuration will differ, supported memory chips or SSDs can differ. By sticking to a single model you gain leverage on any learning curve: solve a problem once, then copy-paste to all the other nodes of the same model.

Here are the characteristics I look for in lower-cost nodes.

• The BIOS and motherboard features must support whatever operating system and O/S features I intend to use, and supply the connectivity I’ve decided I need for any hardware that will be mounted directly on or cabled to the motherboard. More than anything else, this draws a line for “too old to use”.

• The model must be easily available. There’s no point in picking something you rarely see available on the major reseller web sites.

• There should be a decent amount of internet history on experiences with hardware quality for that model. You want hardware that has been reliable, both in initial assembly and in use over time. The model might not house the fastest and hottest-running components as of when it was made, because heat means thermal stress. Find a reliable workhorse, not an older Ferrari.

• The physical size of the model must suit the location where it is going to live. If you have an unused (and dry!) basement with ample space for racks and 1U “pizza boxes” then the width and depth of those nodes might be fine. If you have a half dozen square feet on a couple of home-office shelves, then small form-factor PCs may be what you’re looking for: I use SFFs myself for exactly that reason.

• You need to be able to support the power requirements of that model. You don’t want the circuit breaker tripping all the time because you tried to run too many nodes with 1200w power supplies.

• The models must run cool enough for the environment where they are going to live. Obviously you may be able to improve on the situation with various cooling solutions, but the end result establishes your thermal constraints.

• CPUs won’t be the best possible but should be acceptable in terms of their performance, their heat generation, and any history related to reliability. As an example, you may decide to take a pass on Intel Raptor Lake 13900K and 14900K CPUs because of their voltage-related stability problems, but on the other hand I wouldn’t rush to anything in the i3 product line either because it’s just too low-powered.

• RAM capacity matters for almost everything. Unless you’re doing the Raspberry PI supercluster, I wouldn’t bother with anything that doesn’t have at least 32gb of RAM per node, and preferably at least 64gb. Note that sometimes the manufacturer information will suggest 32gb when 64gb was possible if you had the right RAM sticks, so do your homework online by tracking down posts from modders and happy owners of a particular node model. Be aware that some motherboards with 4 sockets for RAM may not operate with maximum memory performance in that configuration; if speed is your goal you may be limited to a single pair of DIMMS.

• I personally prefer to buy bare-bone boxes that have any CPU, GPU, Wi-Fi, Bluetooth, and wired network support components mounted… but not the RAM or SSD. That lets me pick a consistent model for RAM and SSD hardware without feeling like I wasted any money. If you aren’t super-fussy about performance testing, then I wouldn’t worry as much about that. Keeping the Wi-Fi / Bluetooth / Network cards consistent though is likely to save you a lot of headaches. There’s more than enough room for configuration-related ick around those as it is.

Middle-Tier Nodes

The process here is much the same as that for the low-cost tier. The difference is you may know you want to perform work that requires physical capabilities your lower-end nodes will lack.

An obvious example would be if the lower-tier nodes don’t have any Nvidia GPU cards in them, and you need something that would let you work with CUDA. The node models will be a bit newer and components slightly higher-end, but you don’t have to break the bank. What you likely care about are which versions of CUDA are supported by a particular GPU card model. The Wikipedia page on Nvidia card features can be a handy resource for looking at API support.

Avoiding the RTX cards that end in “80” or “90” can provide you with some affordable options. Obviously lower-end GPUs wouldn’t cut it for somebody training a 100B-parameter base LLM, but we’re not trying to spec out hardware for an activity that runs with budgets in the 10’s of millions USD. I’m not going to get into AI details further now, they’ll be “Cheapskate AI” material in future articles.

Another kind of node warrants consideration as a possible mid-tier expense, and that is a good-quality NAS (Network Attached Storage). That’ll be discussed more in the next article on JBoS (Just a Bunch of Storage).

Higher-Tier Nodes

Not much to explicitly plan here except to note that this can be a disproportionate hunk of your budget. Avoid it if you can, but make the money count if you do it.

My personal inclination is towards these being the periodic upgrade to a really good laptop, since even when you have a compute cluster you’ll behave like any developer and work on it remotely.

If you have big plans in the ML and GenAI space, you might instead opt for a desktop box with whatever you can afford in the way of GPU cards and the most VRAM you decide you can fund.

Power Planning

This is a big area and I’m not the person to do full justice to it, but as you have more money in hardware you’ll want to learn about:

• Surge protectors

• Surge arrestors/limiters

• Power/line conditioners

• Uninterruptible Power Supplies (UPS) and their common architectural variants: standby, line-interactive, and online

I have seen a rack taken out by a lightning strike, so learning about the relevant power-handling hardware and how to set it up properly is something to have on your radar for some stage of the evolution of your lab.

Until then, at least consider surge protection. To be extra safe, power down and unplug everything whenever you hear about an incoming electrical storm, or when notified that the utility company is going to be doing work in your area.

One point of this lab-building exercise is to treat it as though you are running a business, not playing computer games, so safeguard that investment.

Other Parts

Buying compute nodes will not be the only purchase. You’ll need to allow for:

• RAM sticks of the appropriate size and part numbers.

• Drives (whether HDD, SSD, or hybrid).

• Network cables with shielding appropriate to the speed and length until you can connect to a router or switch.

• A network switch.

• Some USB sticks for doing initial O/S installations or for firmware updates.

• Anything you need for interacting with nodes, such as a monitor and keyboard.

• If the motherboard or node case have vendor-specific parts, like fan housings or heat shield armor, consider picking up a few spares while you can get them.

• Add any cooling supplies or spare fan parts you want quickly available.

• Add any tools you want for assembling hardware like Torx screwdrivers, anti-static mats and wrist straps, magnifier and lighting for viewing small parts in tight quarters, etc.

• If you went the route of rack-mount hardware, then depending on your plan either full-sized or table-top sized racks and all their supporting hardware pieces; alternatively if you need any office furniture like shelving then allow for that.

• Storage bins for bits and pieces, because you’ll definitely collect bits and pieces.

Add or remove as appropriate. It’s your lab for your experiments.

The Process

I would suggest some habits around how you acquire and integrate hardware.

1. Learn which vendor sources you find reliable, and mostly stick to them. You’ll have enough going wrong as it is. Sometimes your compatibility homework on a RAM part number won’t turn out as you had hoped. Sometimes the wrong part will get shipped. The universe introduces enough chaos so knowing which vendors are more reliable can save you a lot of grief.

2. Don’t rush to buy multiple units of a compute node or expensive part until you’ve bought and tried the very first one. It’s not hard to trip over something that will be show-stopping for your plans, and you have to send that item back and start again on your homework for the alternative. Buy one, put it to some use, and then if your plan was to buy more you can proceed.

3. Don’t buy until you have time to install and test, particularly when it isn’t a brand-new shrink-wrapped product. Schedule the time for the work. Not only will that help you catch bad parts fast enough to return them, it will also help you spot when somebody sends you a part that is not what was advertised. Even the shrink-wrapped products are better to be able to test quickly, but anything used or open-box you want to make sure you didn’t waste your money or were scammed.

4. Make notes and take pictures if those might be helpful later. Setting up a compute cluster has a lot of repeat work, and you will find yourself routinely wishing you remembered exactly what steps you took on a previous node. Capture the information while it is fresh. Even a day later you’ll forget something.

5. Budget for a cost 10% above what you planned. You’ll discover you need something that you didn’t realize would be important until opening up a case, or after realizing half your network cables aren’t being recognized by the switch.

6. Expect to do operating system installations frequently at first. Not only are there many flavors to choose from — even just in the Unix/Linux world — but you’ll find yourself grappling with questions on specific O/S version numbers, desktop vs server variants, disk layout for the installation, and sometimes BIOS configuration. You will change your mind, and discover why keeping notes comes in handy. Don’t be emotionally wedded to any initial set-up. Learn to feel comfortable tearing down anything that isn’t exactly as you want it to be.

A Living Example

Here are the low-tier nodes for my SFF cluster.

• Intel NUCs, all but one are the same model; the exception is my Kubernetes control plane. It has a small keyboard and monitor if for some reason I can’t get to the cluster remotely, e.g. due to a flubbed network configuration experiment.

• Each node has 64gb of RAM, an HDD, and an SSD.

• Cooling isn’t too much of a concern as Intel made these from CPUs intended for the mobile market, but I keep a fan at one end to circulate the air just to be safe.

• The nodes connect to a network switch positioned on the shelf above.

All of these were bought after Intel had moved on to the next couple of series in the NUC product line. Fortunately I avoided the Raptor Lake debacle. At this point I think about 3/4ths of the Intel NUC lines are aging out as an option for those starting in 2025 on their own cluster. There are many other SFF vendors that have taken up the baton, so I expect SFFs to remain a viable approach for those not wanting to go in the direction of rack-mount hardware.

The Experimentalist : Phase 1: Just a Bunch of Compute © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

My "Karl Marx Would Buy GPUs" article concluded how, with ever-concentrating corporate and personal wealth, GenAI vendors will push that disparity further. Our finances whether as employee, solo entrepreneur, or small-company founder, are facing similar forces. The choices on strategy to overcome the slow squeeze boil down to combinations of:

1. Find a way to be more capable than the big GenAI vendors and identify opportunities to beat them at some piece of their own game. Challenging, but not impossible as big companies aren’t as nimble as individuals or small companies. Expect to change direction rapidly as LLM capabilities evolve.

2. Find something your market wants that is grounded in novelty or uniqueness or deep insight, or at least with a provenance that communicates value. If you were part of the reason for the transaction, that edge can’t be instantly prompted away.

Next: Phase 1: Just a Bunch of Compute

Skill Building

My own focus is the world of software engineering so I’ve been trying to navigate a path that is about 5% of the first option, and 95% of the second. A day may come when technical domain expertise ceases to be a thing, but I believe we’re a long way from that. The strongest reason I have was explored in On the Turn of a Phrase:

We cannot think thoughts for which we lack the language.

Within software engineering — or frankly any deep-knowledge career path — that suggests a couple of observations:

• If you aren’t invested in the GenAI world then you need a deep understanding of the detailed workings of a domain. You’re competing against an ever-changing LLM knowledgebase. Surface knowledge won’t cut it.

• If you are invested in the GenAI world then your ability to generate quality outcomes improves greatly if you have deep knowledge of a domain. Surface knowledge has too little selective power in prompts to generate the strongest results out of an infinite universe of choices. You need to know what to ask for.

Either way you land in a similar place as a SWE. Deep skill has moved from being an eventual nice-to-have to now being a key factor in your relevance.

You can’t just read a book for deep skill:

• You have to be in a position to actively experiment as you drill down.

• We learn from a mix of positive and negative experiences.

• These shape both what connects in our memory and the boundary on what is appropriate to generalize.

Those of us who have already been in software engineering (or adjacent field) for a couple of decades already had opportunities to do all that. We’ve built layers of foundation skills. I’m concerned that more recent graduates, junior, and even senior engineers are going to find themselves denied a similar learning process.

The Strategy

To help fill the gap I’m rolling out a series of articles on the approach I have used myself for many years. I’m in the midst of updating it for life in an LLM world.

The idea is simple. Nothing teaches you how things work, like having to actually make them work by yourself. The effort to create comprehensive infrastructure from scratch will teach you far more than a hundred LeetCode exercises.

If you follow along and attempt any of this, in essence you’ll be building the scaffolding for your own startup. Whether you have a great business idea or not, at a minimum you’ll be faced with how pre-seed tech startups start to stitch themselves together and you’ll see why they have so many rough edges. There is only so much time, only so much money, only so much energy, and you learn that the only way you say “yes” to important tasks is by saying “no” or “not yet” to unimportant ones.

Even if you aren’t in an early phase of your career perhaps you’ve experienced what many of us do: an industry of ever-broadening technology choices we often bounce between at increasingly shallow levels of understanding. If you haven’t built a proper experimental playground before, this may increase your motivation so that you can dig into all the high-leverage details that the day job may not enable.

There is another theme to this. Cloud compute and GenAI vendor APIs have something in common. Both have ways they tax you for experimentation, and can punish you badly for expensive mistakes in establishing budget guardrails.

The Experimentalist Laboratory

The approach here is to mostly run on hardware you’ve purchased for yourself. Obviously many things can be moved to the cloud, and some of them should, but there is nothing quite like the freedom to make any purchase choice that makes sense to you, apply any configuration that helps you, and throw away anything you later discover does not server you.

Technology should be about YOU, and not the other way around. The process here will very much be one of making, and revising as necessary, decisions that place YOU at the center of what happens, and keeps YOU as the primary decider of pretty much everything.

A side benefit is that, other than the initial hardware purchases, you aren’t constantly feeding mega-corps more money just to have permission to learn and push the envelope. Stop acting like their favorite flavor of catnip. They have an ample supply.

The essence of building your own laboratory for experimentation is straightforward:

• The economics of the platform should be approachable.

• Hardware, software, and some external services will be involved.

• It likely takes more than a laptop; you’ll want something that surfaces real challenges and provides the growth potential of actual distributed systems.

• There will be many moving parts, so tackle the assembly in bite-sized pieces.

• Optionally include LLMs. If you do, the same hardware you use for other experiments can be used for self-hosting some LLMs.

Including LLMs opens up more avenues for learning and — if this happens to be your goal — you were never going to avoid self-hosting experiments with the strategy “beat GenAI vendors at their own game” as your objective.

Assembling the laboratory is broken into two major sections and an optional third.

• Phase 1: the basic foundation of hardware plus minimal software tooling for rolling out system configuration and application installation. It also covers ensuring that your work gets protected somewhere remote.

• Phase 2: an ever-growing collection of technology artifacts or progress on procedural efforts so that, in time, you can build pretty much anything you wish.

• Cheapskate AI: optionally, run LLM tooling locally to experiment with, tune, and control the behavior of the AI tools you use. The tools should behave as you wish, not as an outside vendor decides for you.

The to-do list can be as long as your interests drive it, but for those earlier in their career you should come out of this with a gut-level understanding of what drives many decisions at the intersection of technology and business. That awareness can become one of your selling points in a career role or a new business.

In future articles I’ll start breaking down the laboratory assembly in detail.

The Experimentalist : Building a SWE Experimentation Platform © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Just a quick note on some long-overdue cleanup now that the newsletter has enough content to guide me on the next steps.

Prev: The Press Room| Next: Newsletter Status 2025–09–01

Why the Changes

New articles are will start rolling out on technical themes, and it was time to help subscribers to find the content that interests them. Performing some cleanup now makes it easier to just follow along with the new pattern as content is released.

What Changed

1. Both the Substack and Medium versions of newsletter now organize material into thematic areas. Those are “sections” in Substack, and “subpages” in Medium. The existence of these is visible both in the home-page menu and in the regions of articles displayed on the home page as you scroll down.

2. The thematic areas are:

   1. Hammer & Anvil: Articles on the tech economy, which already contains several items.

   2. Hot off the Press: Posts like this about newsletter features and usage.

   3. The Laboratory: A new area for home office/small business tech experimentation. Not yet visible, but stay tuned as that’s about to change!

   4. More to follow in time for data engineering, observability, AI and ML.

3. For Substack subscribers this will allow you to control which notifications you get. At the moment nothing is different. You’ll get all of the new article notifications, but you now have the power to exclude notifications for sections you aren’t interested in. In an Attention Economy, I’d just as soon let you enjoy some silence and not sift through unwanted inbox visits. Apologies to the Medium subscribers, but that platform does not have an equivalent feature for subpages.

4. The changes above will only be visually meaningful on a web browser. You likely won’t see a difference on Android app clients for either Substack or Medium. I don’t personally use an iPhone so I have no idea what the outcome is there, but I suspect it is much the same.

5. I tidied up the in-article navigation links on multi-article series to eliminate “View Series”. I already wasn’t using that to link to anything for Medium because Android deep linking doesn’t work in the Medium app for tag page URLs. With the migration to Substack sections apparently the Substack app suffers similar problems for section URLs, so that eliminated the last reason for a “View Series” text fragment to exist.

The Experimentalist : Newsletter Refurb 2025-08-08 © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Setting aside the obviously disastrous history of Soviet-era planned economies, sometimes useful messages get lost with the passage of time and past poor implementations. Marx’s observations on the long-term trend of private profit and concentration of asset control have largely turned out to be true. Unfortunately he didn’t allow for how collectivist control ends up no better than control driven by short-term-focused capitalism.

Prev: Economic Currents | Next: Reification Fallacy and LLM Use

For backstory articles on wealth concentration:

• Why skilled workers need a plan to navigate through wealth-driven change was introduced in AGI is Irrelevant.

• Personal wealth concentration was covered in GenAI Reimagined.

• Corporate asset concentration in the U.S. was explored in Economic Currents.

This article is a start on “so what are we going to do about it?”

I’m going to try and scare the hell out of you. I don’t do that to market fear (un-paywalled articles make a terrible economic model for that), but to emphasize how now is THE time to discover whatever your personal sense of purpose might be. You need to find it, and then align that with a strategy that properly accounts for what is happening around you. More than anything, I want you to bloody well pay attention and get out of life’s autopilot mode.

Feudal Production

The part of Marx’s analysis that mattered was how, over time, small cottage-scale industry would get supplanted by larger industrial efforts that needed more coordination. The part that he got wrong was his belief that feeding such a process could result in a different outcome depending on how you fed it. Tech broligarchy and communist oligarchy end up no different when you aren’t one of those “select few” making the big decisions. Wealth concentrates intensely, and anybody not at the very center of that concentration becomes, economically speaking, a food source. You’re the value to harvest, whether as worker or customer or tax payer.

The stronger — if more challenging — solution would have been to advocate for ways in which the smaller cottage industries could grow to reasonable size and scale of output without transitioning into a size disproportionate to the health of everything around them. Some system where workers and owners are either the same, or closer in their working relationship and thus have more ways to be aligned both economically and for the mission of the company.

For the Victorian time period of Marx this notion would have sounded like gibberish in anything except perhaps high-end arts and crafts businesses like William Morris’ own “Morris & Co.” In the modern economic era, things are different.

Modern Production

The personal computer arguably jump-started the idea that people could have a device in their hands that gave them options. If they were a programmer, they could use one to hone their skills. If they were also entrepreneurial then they might add to their income with consulting work, or start a software-based company. Rapidly a host of keyboard-adjacent career paths like bookkeeping or professional writing followed suit, until ultimately virtually all of white-collar employment had potential to at least leverage a PC to assist with the capacity to work and earn a living.

With the advent of GenAI the mere value in possessing a personal computer has dropped substantially, at least in the short term. The fact that you can use a laptop and generate material from it has competition from online services that can also generate material. There was a potential moat each free person possessed by mere virtue of engaging their brain to see if they could produce via a keyboard something another person hadn’t thought of. That entire concept is under systemic threat across roles and across industries.

Future Production

The GenAI vendors are trying to seize the means of production. They’re doing it by making a thing they have and you usually don’t — expensive GPU-based processing for LLMs — and using massive PR and marketing budgets to push all other industry around them to opt for the new means (GPU-based production) and discount the criticality of the previous means (PC-based production). That leaves them in possession of the equivalent of a shop floor filled with robotic assemblers, and you holding a pocket calculator.

In the short term the disparity has been made less obvious. LLM access is metered out remotely, and you get to use it via that old calculator you are used to: your PC. Nothing to see here, right?

Unfortunately we’ve seen this game before. Introduce something on the cheap. Get people to yield to the sound of the Pied Piper. Change their personal and business workflow to the new thing, but at a price that doesn’t quite let them realize that tokens are being positioned as the new form of taxation: a tax on every thought, every act of creation, every communication.

What do you think happens next? Enshittification. The quality drops. The token costs go up. Only the select few with the deeper pockets get the best benefits. This isn’t just a time-tested formula. It is the very formula that some of the biggest players in GenAI have themselves used to grow their pre-AI companies. It is already the way they think about business. As the process proceeds, more wealth leaves from those who can pay, and goes to those who charge. Wealth concentrates even further.

Understand the thesis here. This is not about whether you personally think GenAI is the be-all-end-all technology solution. This isn’t purely about technological effectiveness. This is about the perception of technological effectiveness in the minds of those at the center of concentrations of wealth.

If the GenAI enthusiasts are right about the utility of the technology, it would make its PC predecessor into little more than a calculator. If the GenAI enthusiasts are wrong, but the GenAI companies convince enough of industry otherwise… the situation is almost the same because of the decision-making power of a few. Perception doesn’t just become reality, perception by the select wealthy few dictates reality.

That’s the message to take away here. The parameters for deciding your life strategy are now the same either way, at least until a different future reveals itself than what we are seeing at this moment. You can take either view, and the path forward is the same because the alternative is that you’re one person sitting on a couch reading this article while a handful of TRILLION-dollar companies are sizing you and me and a few billion other people up like we’re all krill destined for the gullet of a whale. That would be bad enough, but they are using their influence to align all the BILLION-dollar companies in the exact same direction.

Go back and read Economic Currents if you haven’t done so previously. It only takes the 5000 largest companies to control about 86% of U.S. corporate wealth. How many games of C-suite weekend golf do you really think it takes to align a large portion of 5000 companies? Not many.

Seizing the Means Back

Personally I’ve been in the GenAI-cautious middle for awhile. I use it often enough for high targeted outcomes, but I’m not the gung-ho-all-in-vibe-till-you-die crowd. Doesn’t quite sound like a Nickelback verse, but close enough. I’ve used the time more for learning the plumbing and research around the edges so instead of being caught up in the new shiny I could develop my own sense of the lay of the land while minimizing the impact of hypesters trying to tell me what to think. But now I know what I think.

We either have to be better at GenAI than the big vendors themselves, or we have to have an option that completely decouples us from their influence on the economy.

Pause. Let it sink in. I’ll say it again.

We have two broad options for strategy:

1. We can decide we have no choice but to be better at GenAI (likely in a narrow domain), and who we have to be better than is the big vendors. Not better than anybody and everybody that has ever been involved in the AI field. Not better than PhD’s that specialized in LLM-adjacent research. Better than big companies that on the one hand have massive assets, but on the other hand have all the failings of very big companies. Big companies are not nimble. They do not grow like smaller companies if those smaller companies make it past the failure rate statistics. The goal would be to feed off the economic bloat in bigger companies, but have the discipline to remain small enough to retain mobility while large enough to compensate exceptionally-skilled employees, and use that sweet spot to continually dodge around the bigger players.

2. We can decide that we have some entirely different course of action for which the machinations of big tech will have no relevance at all. That means either computer technology is not particularly relevant, or that somehow the unique provenance of you doing the work is what matters to your customers or employers. This is effectively “living off the grid” in terms of GenAI. What establishes the perception of your value to the marketplace can’t relate to GenAI, or in some way must be an act of rebellion against GenAI that people are willing to pay for. This would push for things like true novelty of a good or service, or something about human connection, or production of a physical good where there is cachet in it not being something imagined by an LLM and instead crafted by a person.

I see no other broad paths out of this than those two. Each has elbow room for individual interpretation and style. What neither of them contain, at least for most of us in the skilled white-collar work force, is “the thing I used to do.” That path was shaped by larger organizations with an already-shrinking relationship to the workforce, and they are in the final stages of doing what they can to reduce it further while concentrating wealth even more.

Yesterday is gone. Let it go. Heartbreaking as the message may be for some, the choice was taken out of our hands.

The Experimentalist : Karl Marx Would Buy GPUs © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Navigating career changes or the challenges of launching a new startup is hard if you don’t have a map for the forces pushing you back and form. Much of that map is established by the prevailing economic environment. Lately the world is abuzz with GenAI disruption, and while that is real, it isn’t the only time we’ve all experienced disruption. Some of the Sturm und Drang is distraction, and for the moment we’re going to set the distraction aside to look at the broader picture. In later articles we can dig into what is legitimately different now.

Diving into U.S. economic data can leave you wondering how any government could use it to intelligently steer a ship containing 340 million people, yet on the other hand also surprise you with effects that have clearly been playing out for a half century or more and thus should have been obvious to policy-makers. Put on your life vest, it’s going to be a bumpy ride.

Prev: Surfing Chaos | Next: Karl Marx Would Buy GPUs

Company Size and Asset Control

Since World War II the size of the largest companies has been increasing in the U.S. This is not merely a factor of population. It is also reflected in how assets are concentrated in the largest companies.

What the charts represent is simple. If you add up some financially-relevant measures across businesses after filtering out all the smallest firms like sole proprietorships, then the whatever those totals are, the percentage belonging to the top 1% or 1/10th of 1% largest companies is displayed. Which measures are shown depends on the historical data available, but we’re focusing on assets (the blue series at the top).

• By 2020 the top 1% largest companies had about 97% share of all assets owned or controlled by all companies of non-trivial size.

• By 2020 the top 0.1% largest had about a 86% share. That represents roughly the top 5000 largest corporations from about 1990 onwards, and fewer than 5000 in the years prior due to changes in the number of U.S. businesses over time.

If the financial heft of companies only grew due to population, then these shares would not be increasing, they would be flat if the number of competitors had not materially changed. This is a bigger-is-better effect that derives from economies of scale likely being a strong factor, plus some mix of tax policy and regulatory impacts that favor the larger survivors as competitors are bought out or fail to maintain market share.

The rapid increase in concentration from the 1960s onwards corresponds to a significant change in corporate taxation due to the Revenue Act of 1964. It wouldn’t have been the only factor, but that introduced a 6% difference in tax rates and arguably was the beginning of post-WWII corporate tax cuts that extend to the present day. Another relevant factor would be the U.S. going off the Gold standard in 1971, which led to an increase in the role of debt in growing the U.S. economy and made it easier for larger companies to buy peers or smaller firms.

As much as we hear the politics of lower interest rates, lower tax rates, and more available credit, history suggests that the bigger players have extracted more competitive leverage from those variables than smaller firms can manage.

Do companies control just financial assets, or do they also employ in similar proportion? The reason for asking is that if bigger companies also are bigger employers then this would suggest that bigger might be better for the workforce as well.

Unfortunately we’re only seeing data from 1980 onwards so commenting on long-term trend is probably not warranted, but the percentage shares do matter. The top 0.1% of companies that control 86% share of assets only employ a 41% share of the workforce; the top 1% with a 97% share of assets employ a 60% share of the workforce. That means the mere fact of businesses getting more concentrated has not generated direct employment in equal scale with their size. They’ll employ more people, but not 1-for-1 with net increases in financial concentration. Smaller companies are the ones picking up the slack in employment.

Why do we care? Because this shows that, at least within the U.S., the large corporations have successfully been eating everything except the available workforce. That makes them competition tough to fight head-to-head financially, yet possibly the opportunity that can be exploited if you start a business because, bluntly, eating the rich has greater reward given that they have more meat on the bone. For job-hunters, smaller firms could be the more probable source of employment since larger firms are more structured for increasing their financial gains per person, less so on increasing gains by increasing their workforce.

This also begs the question: why do states chase the biggest companies with large tax breaks? Without accounting very clearly for the specific type of business activity that will be gained by a deal, and how local supply chains will be benefit, encouraging smaller companies instead looks like a more effective employment strategy.

Smaller Companies Have to Fight Harder

You’ve probably heard that small business is the engine of job creation in the U.S. There is truth to it, as the chart above on employment share suggests. Unfortunately.

I say “unfortunately” for two reasons:

1. Smaller companies have to fight ever-harder to survive. Various studies have examined this, and the TL;DR is that the longer a small company survives the more likely it is to continue to be a survivor, but the initial years see a lot of wreckage. Perhaps not surprisingly, as the bigger companies gain more share of economic assets, the smaller firms lose share.

2. If the largest 1% of companies control 97% of assets, then that means the bottom 97% of companies only control 3% of financial assets. That’s not a lot of financial support to work with in order to be a supposed “engine.” That’s a weed whacker.

I don’t have data to extend the figure to earlier time periods, but from other data on new company creation per decade I strongly suspect the figure above would show this decline began at least 1 or 2 decades earlier, probably once the wave of activity from the post-WWII GI Bill had finished fully washing through the economic system.

What this likely shows is either a reduction in people taking on the risk of starting new businesses that need to hire employees, or more rapid buyouts, or an acceleration in failure rates in the early years. Most likely it is a combination of the three.

Small new entrants are experiencing an increasingly tough game. The survival statistics on small companies are brutal, so while small companies may be “the engine,” the employees in them are experiencing the turnover face-first with firms entering (being created) and exiting (failed or bought out). These small firms either have to go for broke in the mad scramble to be big enough to win the race on the survival statistics, or take a buyout and call it a day, or they have to find a niche and somehow entirely avoid all of this. The latter is something I intend to get into in other articles as the GenAI world may change some thinking in this space, but for now understand that collectively these figures hint at why you see so much venture capital pressure on small startups. From a VC standpoint, it is literally “go big or go home.”

Declining Job Growth

There is another factor that it is important to see, and this is one that may end up telling the future story of U.S. economic positioning within the broader global macroeconomic landscape.

This figure will take a little bit of explaining.

• The darker jagged series is how many new jobs were created in that year. These would be companies expanding employment plus new companies appearing. Since these are annual numbers, there should be no seasonal adjustment.

• The lighter jagged series is how many jobs were lost. Either companies were reducing employment or companies were failing. Again, these should not be seasonally adjusted.

• The difference between the darker and lighter series is now many net jobs were added to the economy if that number is positive; if it is negative (the lighter series was above the darker) then that is how many net jobs were lost.

• The darker and lighter dashed lines are smoothed versions of the noisier data.

• When the economy is expanding then the darker and lighter dashed lines move further apart positively. When the economy is contracting they move closer together or negatively.

• The downward trend on both means that, on average, the rate of net job creation is declining. How much that matters depends on whether population growth rate is also declining — which it mostly is, slightly (Source: The Brookings Institution) — but the slope shown here is a bit stronger than that and so likely reflects changes in the unemployment and the participation rates.

• I examined the Bureau of Labor statistics for the years since 2011. You get expected distortion around economic events like COVID that trigger liquidity injection, but mostly the story of a declining rate of net job growth holds true.

At a minimum this highlights one reason why political discourse can be fraught. The expansionary phases in this situation are a relatively modest and temporary effect compared to the overall trend. It is entirely possible that at some point not too far off the prevailing economic philosophy in the U.S. will have to set aside an assumption of endless growth by large companies. Either smaller companies will have to increase in representation within the economy, or the U.S. will have to shift to a degrowth economic model and possibly manage policy to encourage population decline.

The Small-Cap Upside

There actually is a slight upside in spite of the picture painted above. Small companies have at least one advantage over many larger companies, thus the idea of smaller players returning to economic prominence isn't entirely outlandish.

On average, small companies are more profitable: U.S. large caps returned 6.6% while U.S. small caps returned 9.2% annually over a 20-year period (Source: Alpha Architect). The spread between them used to be larger, but the reason for large-cap profitability gains largely goes back to earlier points about advantages of scale and tax treatment. So long as some advantage remains, U.S. small companies should remain a focus of VC funding.

A reasonable question which I won’t get into here is how much of that large-cap profitability improvement is due to organic factors such as actually creating new products and services, versus financially-engineered improvements like M&A and stock buybacks that show up mostly in accounting statements. Making numbers better of course matters to investors, but it isn’t the same kind of lasting value creation you get from building new things you could actually poke with a stick, and usually lacks the same long-term growth potential.

If interest rates and inflation remain elevated, both of which have some fundamental reasons for being the case, then those forces may favor companies with real organic growth over those that only appear to grow via financial engineering. Financial engineering may not grow value faster than inflation because you have to keep discovering more and more accounting tricks to extract value out of an otherwise fixed situation. Like, for example, the recent wave of layoffs blamed on AI.

Fit the Puzzle Pieces Together

Pause to see how the components of the system are fitting together.

• A significant share of employment comes from smaller companies, not just the largest 1%.

• Small companies drive net job gains in the economy.

• The largest 1% are eating all the financial assets.

• Share of job creation activity in young firms is declining.

• The rate of net job creation in the economy is declining.

This is why in the opening I said “… effects that have clearly been playing out for a half century or more and thus should have been obvious to policy-makers.” Making policy that cozies up to the largest firms is not effective economic management. It is, if anything, almost the exact opposite of what is needed because it mostly maintains the status quo instead of improving upon it. Small and mid-sized firms are the need, and possibly the opportunity because — let’s be real about the implications here — the biggest firms are running out of financial activity share to eat. Whatever the largest eat will turn into something less profitable, with lower growth, and lower employment. It’s time for the small fish to remember that piranhas can eat things too.

The potential for organic growth in smaller ventures is an area of likely advantage for GenAI builder-folk that I want to dig into with future articles.

References

• The University of Chicago, BFI Working Paper No. 2023-20 / 100 Years of Rising Corporate Concentration, Kwon et al.

• Journal of Economic Perspectives, Vol. 28, No. 3, Summer 2014 / The Role of Entrepreneurship in US Job Creation and Economic Dynamism, Decker et al.

• The Brookings Institution / New census estimates show a tepid rise in U.S. population growth, buoyed by immigration, William H. Frey, January 4, 2023.

• Alpha Architect / The Gap Between Large and Small Companies is Growing. Why?, Larry Swedroe, March 5th, 2020.

The Experimentalist : Economic Currents © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Ever since the Windsurf debacle, I’ve found myself thinking even more about what the tech startup world is going to look like going forward.

Prev: GenAI Reimagined | Next: Economic Currents

The problem I have with everything that happened relates to the employee situation. When people join an early-stage tech startup, usually they are pressured to negotiate a smaller salary in the expectation that a bigger payoff will happen later. That payoff event is referred to as the “exit.” The exit might be an acquisition by a bigger company, or it might be an IPO. Worst case (other than failure), the most significant Venture Capital (VC) investor merges smaller deals together that aren’t entirely failing but aren’t growing fast enough on their own. If that happens it probably won’t count as an exit and instead founders and employees will end up with a diluted position in the resulting entity.

However it happens, founders and those employees accepting the smaller salary eventually require an exit with enough money flying around or they won’t get that payoff. When it takes place, everybody is not an equal participant in the end game:

• Founders and all the various employees will have different percentages of equity.

• Employees might have options instead of equity. These won’t have any meaningful value unless the per-share exit price is above the options price.

• Employees may have equity in different classes of common stock than the founders. This might put them in a different position for how payoffs are computed or what their respective positions mean for an IPO.

• There may be vesting terms, where the employees aren’t able to benefit from that payoff until a certain period of time beginning from their employment date. If they quit or are fired before then, they may get nothing.

• In the case of an IPO there may be additional requirements for founders and employees to not liquidate stock until a certain amount of time after the stock is first listed on the market.

Standing in front of founders and employees and expecting to get paid first are all the pre-IPO investors. They will have established legal status ensuring that they get paid first, or at least part of their payment happens first. The mechanism is referred to as an investor preference. The more series of funding rounds that took place, the more investor preferences there are standing in front of both the founders and all the non-founder employees. Like I said, it can take a lot of money flying around before you make that extra buck.

When the Windsurf deal with Google happened, the effect was to gut the company of key founders and R&D talent. I’ve been hunting and haven’t been able to find clear details on how that transaction took place, but since there hasn’t been news of lawsuits flying around it’s a pretty safe bet that the investors were all made whole with a reasonable profit. The founders and staff that moved to Google also would certainly have received something, or there would be no motivation to accept the deal.

That left all the remaining people sitting in the shell of a tech company. First they would have heard about a possible OpenAI deal, only to have their hopes crushed. Then Google appeared, and again their hopes were crushed. It had to have been a pretty brutal sequence of emotional let-downs. Fortunately in a market hot for talent with anything vaguely like an AI pulse, a last-minute deal happened where Cognition AI bought what was left.

To Cognition AI’s credit it sounds like they are doing something reasonable to honor the equity or option positions of the Windsurf staff. This does not appear to be a full exit though, as there has been no reporting indicating that the staff are getting fistfuls of cash. What apparently has happened is that they are back to square one, waiting on their exit. Now it’ll be the Cognition AI exit, instead of the Windsurf exit.

The only improvement in the employee situation I’ve found is that any previous vesting terms from Windsurf or any vesting policies of Cognition AI have been waved, and those employees are now fully vested. It was a reasonable adjustment, and Cognition AI deserves some kudos for it, but I don’t think it changes how the original Windsurf founders seemingly threw all of those people under the bus by denying them participation in an exit. It’s a reduction in damage, but in my opinion it is not a re-writing of history that unwinds what very much smells like a breach of trust. Whether done in intent, or done in an utter absence of fiduciary peripheral awareness of your team, that experience must have really stung for those left behind.

The tech startup world has an established history of grimy outcomes, so it isn’t like this is a new situation. What may be new, however, is the pace at which these take place in a GenAI world. I believe this is something all participants — founders, employees, and investors — need to begin planning around. I hope to get into the implications in future articles, so stay tuned.

The Experimentalist : Surfing Chaos © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Anybody who has seen my posts or comments on LinkedIn within the GenAI and ML space might have observed a few things:

• I’m neither strictly for nor against LLMs, but I’m not a fan of influencer-fed hype shoving things down a developer’s throat and in ignorance of the context in which that particular developer works. The standard measure of productivity should be productivity itself, not imagined proxies for productivity.

• I believe we’ll need more than merely DNN (Deep Neural Network) LLMs as the core of the token generation and reasoning mechanism, but DNNs have made solid progress and thus are likely to stick around for awhile.

• I dislike crappy science, but I dislike crappy reporting about science even more than I dislike crappy science.

This is a post I had not intended to tackle so soon, but these crappy-reporting situations are coming up constantly. It doesn’t matter if you’re talking about the more mainstream media outlets, or the hives of self-promotion activity that you get in places like LinkedIn. It is past time somebody started at least skimming a paper they wrote about, instead of all these mindless confirmation-bias exercises we’re constantly exposed to. I don’t care if you’re anti-LLM, or if you are pro-LLM, but if your side is right it should still be right when you read and report on papers fairly. If you have to write lazy posts to justify your position, your position is weak sauce. Seriously, many influencers need to learn how to make a better case.

As a result, this is the start of an ongoing series I’m tagging “Research Critiques”. Collectively we should all be more informed than many influencers want us to be. The goal is to understand and draw intelligent conclusions in any direction a paper legitimately supports, not just mindlessly cheer for a team. I’m not going to reproduce the contents of these papers, but point out some key items, and nudge you into at least skimming the paper yourself. What you will be seeing here won’t be the result of days of painstaking reading, it will just be from actually bothering to look at all. If you do the same, you can form your own conclusions of a paper. Your conclusions should be yours, not mine, and not some random influencer that never even looked at the paper. The practice is worthwhile, and doesn’t have to take long.

Enter the METR study: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. Everybody that is anti-LLM has been dog-piling on this with posts that barely do anything to properly lay out where this study fits, which conclusions are reasonable to draw, which ones aren’t reasonable, and which ones even the authors themselves are not attempting to argue.

Summary

The study involves 16 developers and 246 tasks with data recorded for 143 hours total with 10-second resolution (although data cleaning reduced that to 84 hours).

Why do you care? You care because depending on the question asked, you might look at this as providing either 16 samples, or 246 samples, or 16 x 246 = 3936 samples, or 84 x (60 * 60 / 10) = 30,240 samples.

The 16 is easy to focus on and reasonable to note that it is pretty small, but not all studies involve large numbers of participants. That alone does not invalidate a research effort. Sometimes it just means you might categorize the effort as exploratory research that should be followed up with additional studies before strong conclusions are accepted as truth. Research studies are expensive, so there is value to early exploratory research that helps better frame efforts in later studies.

One thing this study includes is it reports how much of the developer time was related to coding. I like that it does this, other papers I’ve seen did not. The answer here is 29%. This is good to know, because when you see posts about how much of an impact LLMs had on coding they usually only measure coding and not what that means per day of an employee’s work activity. If coding is 10% of your day, and an LLM helps you code 10% faster, then the LLM would only have a 1% impact on your daily routine. Maybe there are other LLM impact numbers for the non-coding activity too, and if so they should be included and computed appropriately. Context matters.

The headline numbers you might see tossed around are this:

• Self-estimates made by developers suggested a 20% productivity gain.

• Actual productivity measured by the researchers showed a 19% productivity loss.

Before you run for your preferred flavor of torch or pitchfork, there is more to think about in the study.

Funding

It’s always worth taking a look at the backstory on funding for a paper. In the case of this study that’s provided in the Partnerships section of the METR About page. The connections to the AI industry seem modest, and geared to the same level that academic institutions would likely receive from many tech companies so that their respective products or services could be used without the deep pockets of a commercial budget.

Study Design

Study participants performed work on their own pre-existing open-source projects. There is some good and perhaps some bad to this, but it’s a reasonable exercise. It eliminates questions of seeing a benefit only when doing greenfield coding. The study is measuring what it means to bring GenAI into an existing non-trivial situation. For experienced engineers our greenfield opportunities tend to be on the light side, so I appreciate this aspect of the study design. It makes what is measured likely to have relevance to more of us. The bad angle I guess I would argue is that the open-source projects chosen were high-profile (23,000+ stars), and such projects usually come with strong curation practices that are much less typical in the codebases most engineers work on.

The participants varied in their previous LLM experience, and the study captures this. This is an area where the 16 number on sample size feels like it has relevance. It seems a little on the light side to indicate how LLM experience influences the outcome. I would rather something like at least 70 participants in 7 cohorts of 10, each cohort of increasing experience, in order to make really strong statements about how more or less AI experience mattered and to observe if the impact of that variable capped out in either direction. This observation could apply for other variables, and follow-up studies should consider picking a smaller number of variables but exploring them more deeply. I’m not criticizing the paper over this, I’m suggesting that readers (and influencers) shouldn’t draw conclusions bigger than the data.

The tooling use was primarily Cursor Pro paired with Claude Sonnet. Only 44% of the developers had previous experience with the IDE. I consider this a potentially weak aspect of the study, but to be fair the analysis does perform some examination on this issue and concludes it wasn’t an issue. I remain unconvinced and would rather see the matter explored in more detail. Any developer that has transitioned from a familiar to an unfamiliar IDE has experienced that you can lose a lot of productivity for a few weeks. With an almost 50/50 split in tooling experience, I would expect productivity between the two groups to change over time in different data subsets. For now I’m less clear on which issues the study is definitely measuring LLM usage impact. Several of variables have that whiff of being confounders for other measurements, which suggests benefit to taking more care with them in more detailed studies.

Some Findings

There is a lot of material in the document, particularly in the Factor Analysis appendix (starting on page 18), so I want to pull out some items to show how this study can easily be viewed as “for” or “against” LLM use in coding, depending on how somebody decides to cherry-pick from it.

Section: Forecasted vs Actual Impact

This is a case where I believe you’re seeing the study through the lens of 16 x 246 = 3936 samples. Developers were not good at estimating the impact of AI, and experts not involved in the coding were even worse. I consider this an important result mostly for sifting through a lot of the online noise we see every day. Anecdotal reporting may be more driven by an emotional perspective than time-clock reality.

I’m not saying anything about LLM-supported productivity here, I’m saying something about human beings and all that weird wetware we have between our ears. The way to measure productivity is to measure productivity; anecdotes that aren’t reporting an actual carefully-examined productivity measurement are not themselves a productivity measurement.

Section: Extended Discussion

Page 17 provides a table of issues that the authors are very definitely not trying to make statements about. One in particular I believe warrants noticing:

We do not provide evidence that:
There are not ways of using existing AI systems more effectively to achieve positive speedup in our exact setting.

Clarification:
Cursor does not sample many tokens from LLMs, it may not use optimal prompting/scaffolding, and domain/repository-specific training/finetuning/few-shot learning could yield positive speedup

To me this goes right back to my earlier comments about developer familiarity with the tooling. Developers that live with these tools evolve their CLAUDE.md file and care for it like a first-born child. Sometimes the level of detail can border on the silly because the prompting is as complicated as the artifacts that would be generated… but be that as it may, there is likely to be a big difference between those who live in this ecosystem versus those just recently introduced to it, because the former may have in essence macro’d better outcomes as part of their routine workflow.

The authors absolutely do acknowledge issues like this and do so more than once (including the section “Below-average use of AI tools” starting on page 23), but I feel like this one is pretty integral to a study on GenAI productivity and should have been controlled for. As a mental-model comparison, imagine a study on Python developer productivity where you observed the time to create properly-baked wheels but didn’t take note of which developers already had experience with all the rough-edged pain of Python library and application packaging versus which ones were being dropped into the soup for the first time.

Section: High developer familiarity with repositories

The findings here (starting on page 18) probably won’t be surprising to experienced coders. Developers already well-versed in a particular code issue weren’t particularly helped by an LLM while working on it. The assistance from an LLM was more material when dealing with something unfamiliar and for which developers needed reference material outside the current codebase.

It seems reasonable to note that if a company wants to predict the impact of LLM use on any given development team, it likely depends on the rate of novel change in the codebase. As efforts shift to maintenance and operations, we accrue more context and thus the opportunities may shrink or turn negative for LLMs to be a net performance gain over human knowledge and skill that is sitting there ready to be tapped. This might also apply to very deep expertise with specific technology components.

Section: Low AI Reliability

This material beginning on page 19 is worth noting. The developers in the study were being careful about PR quality in submissions, because these were codebases they were familiar with, invested in, and knew the community standards for. This is why I called out earlier that the 23,000+ star projects may not represent a typical engineering situation for study purposes.

Developer reporting is pretty clear about how much effort they had to put into the PRs, and so it isn’t immediately apparent why this wasn’t as strongly reflected in their self-reported estimates for LLM performance impact. I believe the effort finding does at least speak to the differences in performance anecdotes we hear contrasting experienced engineers and naïve coders using LLMs, which makes sense because the former group will have more habits for being less tolerant of low code quality. Whether that is good or bad is in the eye of the beholder, but it is going to be reflected in actual performance outcomes if engineers maintain that perspective. In the extreme case of popular open-source projects the engineers definitely must have such views unless the project community decides to revise its policies, but that influence may be somewhat moderated in other domains.

Wrap-Up

There is more you can dig into by reading the study yourself, and you should. I hope that at a minimum you will see the study has some substance, some limitations, seems fair in its perspective for the issues it chose to examine, but that it is fundamentally an exploratory effort and should be considered in that light. Influencer opportunism should not be the basis for deciding its merits, or for painting it as having more meaning than it deserves or that the authors themselves intended.

Edits

• 2025–07–28: Weakened the wording over concerns about the relationship between variables over time. I decided the concerns were overstated relative to reported findings unless I had data to examine.

The Experimentalist : Critiquing the METR Productivity Study © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

It’s been a week since my initial flurry of posts, and it hasn’t been due to a lack of effort with research and writing. On the contrary, I’ve been exploring different angles in the hopes they might better illuminate what we — the great unwashed masses of skilled professionals — need to do to navigate a rapidly changing world.

Prev: On the Turn of a Phrase | Next: Surfing Chaos

As I’m a data guy, my initial hopes were to find data that would tell a story. Not data that could be arm-wrestled into just supporting a narrative I’d pre-determined, but real data taken at face value and used to tease out some “ah hah” moments. While I found some interesting economic views on the human condition, none fit the bill.

I’ll skip a long backstory and sum up the research. The TL;DR according to the 2025 UBS Global Wealth Report (which examines 2024 data) is:

• The top 1.6% of the adult population controls 48.1% of the wealth. These are people with a net wealth of $1 million USD and up. The overwhelming majority would be nearer the $1M level as the pyramid gets extremely narrow as you approach the tip.

• The bottom 40.7% of the adult population controls 0.6% of the wealth. The net wealth per adult in the band is $10 thousand USD and down.

What I actually wanted was data showing the number of decision makers controlling the economic fate of a significant portion of the population so that I could then reason about how GenAI-motivated layoffs might play out. I gradually realized I was facing a level of effort comparable to writing a Master’s thesis. Someday perhaps, but not today.

The Step Back

This began with something that has nagged at me for awhile about GenAI. I couldn’t put a finger on it exactly, but it was starting to seem perhaps data wasn’t the leverage I was looking for. It was entirely possible I was hunting down a dynamic that data alone wouldn’t be good at capturing, or at least not yet.

I’ve been slinging a keyboard professional for a long time. Multiple decades, and that multiple is not the number “two.” I’ve seen many different technology transitions, and I’ve seen industry reactions to those transitions. Booms and busts, marketing hype and book publishing waves, software ports, environment migrations, second-system re-engineering efforts, and process overhauls galore. I’ve seen just about all there is to toss at the industry since we were stumbling around on early Unix workstations and the advent of PCs. GenAI isn’t like the waves I’ve seen before.

The light bulb moment, for me at least, came from deciding what wasn’t relevant to that nagging feeling.

• It wasn’t about the technology. As technology complexity goes, LLMs barely rate. It is a simple technology concept done at extremely large scale. Sure, there is ongoing evolution and improvement, and now more mathematical examination of how training works and the parameter topologies that might induce, but I could list dozens, maybe hundreds, of examples in software and hardware that have much more subtle or complex characteristics once you set mere scale aside.

• It wasn’t about the range of application. Every programming language compiler or runtime, every well-developed application framework, every expansive library ecosystem, collectively have massive range of application.

• It wasn’t even the economics, although excluding that entirely didn’t seem quite right either. Not that the economics of the LLM doesn’t tell a story, but any rational person in this space knows the story is complex and very context-dependent. Have you seen the world around us? How often do people get truly moved by the complex and context-dependent? It’s the social kryptonite of the millennium.

What was left was… something else. And I think that something else slithered in under the cover of other stress factors in the economic and political zeitgeist.

More than any other technology-related event other than perhaps cryptocurrency, for some reason, GenAI has people “living in their feels.” I’m not saying people aren’t being intelligent about GenAI in either direction of the various debates. I’m pointing out that the volume knob on emotional energy is cranked up very, very high and that alone is an artifact worth taking note of.

It isn’t that we never see emotion in technology issues, but historically they have been confined to small camps. Camps that most of us, truth be told, learned to avoid or tune out because they were just so obviously annoying, over-the-top, and not productive in helping us with our work at hand. The more experienced in the technology space you are, usually the more jaded you get because you learn through experience that almost all problems can be approached with many choices of tools, plus employment environments typically bias the tool choice and you either get with the program or you get a different job.

What we have instead are very broad-based coalitions advocating GenAI as not only the “one true way” but further reinforced with the message that the world divides into camps that either “get it” or “don’t get it.” This is not really the history of engineering or technology as lived by practitioners, where experience has been “found a better wrench for times when a wrench is useful, but tomorrow I may need a saw.”

There is, however, a dynamic in history that does cleanly fit this particular emotion-laden pressure for change plus division versus conformity. The counterculture.

Hoping for Change

We hear language like this in periods when part of the population is pushing to opt out of the social framework that preceded them, because that framework is seen as no longer working or insufficiently flexible to address a changing world. From Wikipedia:

For me this sidelines debates over matters like GenAI effectiveness, or when AGI will manifest, or if LLMs will provide the ultimate final AI model. None of those things may actually be what is going on. If you think about it, if you were halo-dropped into July 2025 without knowing anything that had happened in the last couple of years, those factors would all seem a strange focus for emotional energy. I think that’s because they aren’t what the emotional energy is about at all. It’s people struggling with the status quo, and concluding — at least subconsciously — the status quo is badly wanting.

This could explain why people of such diverse backgrounds can find themselves unifying energetically under a common technology banner. It isn’t about “oh I can be lazy, and make a gadget do that job for me.” It may simply be about seeing the novelty and potential power of a tool to extend a little hope, when the technology and large-corporation employment baseline was for many people already a system of slowly-shrinking economic hope and disempowerment. In this respect the cryptocurrency community, particularly Bitcoin “maximalists,” may find common cause.

It is a bit fluid what this common cause will face off against. Big tech? Wall Street generally? Wealth inequality in some less-specific way that doesn’t have a clearly defined opponent?

Ultimately, GenAI may not turn out to be the final solution for empowerment any more than most crypto efforts were… but perhaps both are the start of something much bigger because part of society has tasted a little hope, and demands more. If that is the legitimate backstory, then the challenge will be to avoid the usual Wall Street juggernaut co-opting any benefit before change can form solid roots in the lower portions of the wealth pyramid. The lower 98.4% could do with a little wind beneath their wings.

The Experimentalist : GenAI Reimagined © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Over the years, there is an idea I find myself increasingly coming back to. It feels like it should have been a quotation from somebody, but I’ve never found a match. More likely I’ve heard a few things akin to it, and the mind compressed them into a single memory engram much the way a Large Language Model (LLM) might.

I believe there is also a corollary to this, with slightly different emphasis:

The point to get across between these is that, in a world of hard problems, we can’t be surprised if those problems remain unsolved if we are fundamentally compromised in our ability to discuss them. If you can’t even pull off a high-quality conversation with yourself in a mirror over something, you’re not going to create, or advocate for, or prompt-engineer workable solutions. Even worse, you could very well be trapped in a hamster wheel where some little pool of existing thoughts drawn from insufficient language just replays on a mental loop to no effect.

Prev: The Foreshadowing | Next: GenAI Reimagined

If those hamster-thoughts were the right words, the right ideas, the obvious solutions, then circumstances would already have changed. You would have “done the thing”, or the obviousness of the solution would have been shared by a billion other people and resulted collectively in the required change. Much like the Zen parable of the overflowing tea cup, a lot of what we hold on to is blockage. It distracts us from the value of just clearing the slate, using fresh eyes, and reconceiving the situation anew.

Brainstorming

The difficulty, and the potential, of thinking up something new was brought home to me years ago when I was working in Academic Computing Services at MIT. The broader Information Systems department that we fit within had gone through years of various attempts to reconceive and improve upon the mission. We used to joke that it was time to start asking the VP’s executive assistant to hide his books when he’d go on vacation, because when the VP returned it would always be time for the next methodology incursion to be trained in. Unfortunately, we’d always just end up trying to fit the old work — and thus the old thinking — into the new scheme. Sure, a few bits of vocabulary changed and we added some soft skills, but it never rose to the level of a language that spanned all the players and moved activity (and the economics thereof) in a new direction.

The moment I really, viscerally, understood what the problem was, came from an activity we conducted with 7 or 8 faculty members. It was a brainstorming exercise where we asked them to go off and write a description of what they wished they saw as a future educational experience benefiting from technology. The instructions were explicit. They weren’t being asked to talk about routinely applying what we were already supplying them. They were being given the opportunity to “think out of the box” and provide the vision we might entirely lack to drive good long-term planning.

All of the educators in question were experienced teachers, their courses were well-known and well-regarded. They varied by age, department, tenure, etc. In spite of their deep domain expertise coupled with substantial hands-on pedagogical application, only two provided missing vision. Everything else described was an obvious extension of where we were at, but two people were able to set that all aside and articulate broader aspiration instead of just characterizing the results of existing momentum. The other responses, while professional, could have been emailed in as one-liners saying “increase budget, then do more of the same.”

Literal Shackles

A sound byte that gets increasingly recycled relates to the literacy rate in the US is “54% of adults read below a sixth grade level.” Like many such pithy phrases, they get game-of-telephoned and the original context gets lost. It is worth developing the habit of digging a little deeper to get accurate context.

Sidebar: Apologies to global readers not versed in the dynamics American education and politics, but I need to segue here to government data I can have more context in. If anybody ever wants to collaborate on showing similarity or differences in other parts of the globe, I’m open to that. Anyways, editorial sidebar concluded, on with the story.

As best as I can determine, the number apparently originates from an earlier version of a 2022 article by the American Public Media Research Lab, where it misconstrued PIAAC literacy levels and 2017 data in terms of school grades. They have since removed that wording from the article. You may see claims the number is relevant to 2025, but from my digging it appears that is just because the National Literacy Institute has a web page which makes mention of those older numbers. Also as is often the case in our politics-sensitive world, the summaries of the results get administration spin (from either party) to make them sound more palatable than they arguably should be. The spin version was “79% of adults are literate”. What that practically meant was “79% of adults would not fail at ordering a cheese burger instead of a bacon burger from a menu without food pictures; the worst two literacy categories were the remaining 21%.” With the hearsay data origin story clarified, let’s move on from the stale information.

The PIAAC definitions I think provide a clearer picture. PIAAC reporting works in levels (click the link and expand the Cycle 2 Literacy Proficiency Levels), not school grades, and these definitions appear stronger for considering the tasks a person might routinely perform adequately in daily life. There are five levels, and a sixth “below level 1” catch-all for anybody with less literacy than the defined five. The descriptions below are my mulling on the implications, not what PIAAC states, organized into a few bands that I find useful for current purposes.

• Below Level 1, and Level 1: not participating in the information economy, and likely very minimal consumers of any technology or media that depends on reading or writing text. Doesn’t read books, likely doesn’t own any.
  The literacy band unlikely to use AI in a meaningful way, even accidentally.

• Levels 2 and 3: routine light consumers of the information economy. The text-based technology interactions are kept modest, but used as needed. Might read simple books, or may just remember having had to do so in school. Could have a few coffee-table books and some pulp fiction or a religious text.
  The literacy band that would accidentally use AI because a vendor wired NLP (natural language processing) into their UX, but would have limited awareness of AI otherwise.

• Levels 4 and 5: active participants in the information economy to varying degrees. The levels of technology use would depend upon the career domain, but technology and media consumption are likely routine both on and off the job. Both reads and writes text for an assortment of reasons. Probably owns multiple books, some of which could be career-related.
  The only literacy band that could or would intentionally use AI, as interacting with an LLM creatively necessitates a level of comfort with the written word.

Particularly interesting are the differences between Levels 4 and 5, which I will quote verbatim:

Adults above Level 4 may be able to reason about the task itself, setting up reading goals based on complex and implicit requests. They can presumably search for and integrate information across multiple, dense texts containing distracting information in prominent positions. They are able to construct syntheses of similar and contrasting ideas or points of view; or evaluate evidence-based arguments and the reliability of unfamiliar information sources. Tasks above Level 4 may also require the application and evaluation of abstract ideas and relationships. Evaluating reliability of evidentiary sources and selecting not just topically relevant but also trustworthy information may be key to achievement.

If you consider the statement in a world of AI, what this really is describing is those people with the capacity to be competent in constructing LLM prompts and evaluating the generated results. This is the level of language necessary to “think thoughts” at a meta-level in order to orchestrate a GenAI process, determine if what comes back has utility, and iterate correctively until the task is complete. Anybody with less than Level 5 is increasingly at a disadvantage, and I would argue that the reduced facility with language makes that disadvantage very difficult to avoid unless you ascribe near-mythic powers to LLMs. If LLMs indeed had such powers, it would beg the question of why low-skill human interactions were of value to task execution in the first place. More likely is people in Level 4 feeling pressure to up their game.

Take a moment to line that up with AI vendor marketing on how it will democratize knowledge and skill. Add in all the people who make online boasts about what LLMs enable for them. Some of it will indeed be true, so for the sake of argument let’s stipulate to all of that potential and be full-throated in our optimism. Imagine a big, warm, fuzzy percentage of humanity that will experience this brave, new, ever-expanding world of possibility… and hold my beer. I’m going to go and fetch the literacy data.

The Data

I’ve set up a Git repository to provide two Excel workbooks that I downloaded via the PIAAC Data Explorer:

• PIAAC-2012-2023-by-sex-and-age.xlsx: The data over three reporting periods, with some break-down by sex or by 10-year bands of age range.

• PIAAC-2023-aggregate.xlsx: Summarized numbers from the most recent reporting period, without any demographic break-down.

The key findings from that second file are:

Year/Study	  PIAAC 2023
	
Jurisdiction	  U.S. Household
	          (16-74 years old)
	
Proficiency	  Percentage
Below Level 1	    11.99
Level 1	            16.74
Level 2	            29.36
Level 3	            29.81
Level 4	            10.88
Level 5	             1.22

There is the reality. That third band combining Levels 4 and 5, representing people who would have some hope of participating meaningfully in an AI-heavy economy: 12 percent. And of that 12 percent, only a little over 1 percent even have the language potential to excel at it, and mere language proficiency alone would never be enough. For the people who have hope, 1 in 12 start with some minimal skill suggesting they might warrant that hope, and the other 11 are scrambling to play catch-up from a position of measurable but possibly repairable deficit. The other 88 don’t even know the race track exists.

Let that rattle around the brain pan for a bit. For those holding the most optimistic view on GenAI, if current literacy patterns continue, then we could face an outcome where 12 percent of the population go in one direction, and 88 percent go in an entirely different one. The only thing that would easily cast the situation in a less dire light would be if something close to 88% of the population currently worked in jobs that don’t have the potential for being cannibalized by AI, such as some blue-collar skilled trades and most unskilled labor.

Unfortunately, according to the AFL-CIO, something like 58% of the US workforce is estimated to currently be white-collar.

It is the basis for a painful transition process into a future combined economic and political divide that will make the current landscape seem a cake-walk by comparison.

This is one of several reasons why I’ll keep saying that we need better options. The search is and will be time-consuming. If you’re finding the material at all compelling, you can vote for more with your dollars on Substack!

The Experimentalist : On the Turn of a Phrase © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0

Writing online faces challenges, as does the reading of the resulting product.

• Readers have fair claim to a need to understand where an author is coming from, what their personal or professional stake or biases may be, in order to evaluate the integrity and relevance of the content.

• Readers increasingly crave authenticity and less emotional button-pushing than AI-crafted material is often designed for.

• Accuracy and research matter, for which AI can be both a support and hindrance.

• It is a worthwhile community service with negligible competitive risk to help others see what they might face in their own future literary efforts.

Next: Newsletter Refurb 2025-08-08

My Biases

My origin begins with family that for the most part got dirt under their fingernails. Growing up was filled with stories of the life and hard decisions forced by the Great Depression. Most of the extended family and friends were firmly blue-collar, some were white-collar middle class, and some farmers were sprinkled into the mix. Politics were varied and not party-affiliated. Family religious views were all over the map: Anglican, Catholic, Mormon, Salvation Army, Jehovah’s Witnesses, Swedish Presbyterian, and over-the-airwaves televangelist; tossed into that I was the lone Daoist/Buddhist-leaning kid who was into martial arts.

One of the things I value about writing is that I usually sound less energetic than what you might hear when my Scottish, Irish, and English roots combine with growing up around teamsters (which I was myself for summer jobs), carnies, bus drivers, and longshoremen, which then later morphed into many years of the Massachusetts experience. If I get up a good head of steam, I fully admit I can give Bill Barr a run for his money. I try to keep a tight leash on that. Nobody needs another Ted sequel. The delete button really, really helps.

I was one of only a couple of family members that pursued a post-secondary education. The results were… mixed. I’d be on the Dean’s honor roll one year, and trying not to flunk out the next. Somehow in spite of that, I ended up working maybe a total of 18 or 19 years within academia. Initially it was just undergraduate rolls in development (fund raising) and cognitive psych research software development, then post-graduation within distance ed, academic computing support, academic administrative computing, and ultimately an affiliated cancer genomics research group. No, I don’t also do juggling and card tricks on weekends at kid’s birthday parties, but it would be an understandable guess since I just summarized only half of the resume.

All of that got tossed into the blender and resulted in valuing progressive goals as an aspiration for individual freedom and opportunity in our shared human experience, fiscal conservativism so there is some thin hope that the money flow alone doesn’t undermine long-term attainment of those aspirations, and I think a pretty pragmatic expectation that the confluence of education, economics, politics, and religion will always combine to make a common meeting of minds very challenging for society.

In short, I’m probably not really on anybody’s team. Y’all get to be equally skeptical of anything I have to say. I’ve never really got the “team” thing, I prefer to dig into each issue as something that has specific context, its own details or history, and I try to accept whatever conclusions the context and data seem consistent with. I also remind myself I usually get three things wrong every day before I’m even out of bed; given time I hope to cut that down to two.

AI Position

Much like I don’t get excessive team tribalism, I also don’t care about fanboys on really any topic. Setting aside the substance of AI, there is a contingent that only cares that you share their emotional energy. The more I see of it, the more I gravitate away since life is rarely lived in the extremes. My position has, very simply, varied over time. I expect it will continue to do so. Some initial reactions were wrong, some so far are not, and time will continually force me to re-evaluate.

Article Content

I do not have any interest in using AI to generate article content for the simple reason that this is what I enjoy. I cannot imagine a more self-defeating behavior than to hand over the thing I like doing, and instead spending more time on other things I don’t like doing. This is not a criticism of other’s use of AI in writing. It is just a personal choice on where my finite human experience will be devoted. I do not view the act of writing as merely a transactional endeavor. For me the process is just as much the point as the published result, and I’m the one and only caster of that particular vote.

What I do use AI for is generating images to establish the mood of each piece. If I had the budget to fund a fractional graphic artist or cartoonist role, I would much rather that. Time will tell if I get there. For now, I value how AI helps me bootstrap a small refinement in user experience.

Advertising

Some acts of publishing words are not the same. Keeping your content visible means navigating through social media algorithms, and authors have no control over the heuristics those services contain. One of the more pragmatic ways to deal with that is to create shorter posts on various sites which function as advertising, redirecting users to the primary platforms: Substack and Medium in my case. I’m not currently using AI for advertising posts, but it is entirely possible that one day I will. Whether you craft advertisements manually or via AI, they are increasingly a highly artificial exercise and there is little point in pretending otherwise.

Research

There are understandable concerns about the accuracy of research done via AI. On that front, I have good news. I’m a digital packrat. If there was a Digital Packrat Olympics I would have a string of gold medals. I’ve been squirreling away links, notes, projects, and dumps of those into Git repositories for years. I add dozens of tidbits every single day. There are even a few boxes of paper reports tucked away in corners, and don’t even get me started on the books which total approximately a half ton by weight. I know because I’ve moved those boxes.

Given the time it would take to write articles on even a tenth of all that, I’ve got a multi-decade head start before I’m exposed to how online content degrades via AI-driven data recycling. If I have a moat for a competitive advantage in writing, information hoarding may very well turn out to have be it. Enjoy. That said, if I ever did use AI to help me with article content, it could be to digest this immense mound of trivia and help me turn it into starting points for code or topics. Once fully incorporated via LLM or RAG, I’d be back to doing my thing my way.

Wrapping Up

I’ll have other posts about the mechanics of newsletter publishing and how some decisions were made. There is also the potential for others to contribute their own articles and so I’ll be speaking to that process as well.

The Experimentalist : The Press Room © 2025 by Reid M. Pinchback is licensed under CC BY-SA 4.0