Category: Technical

Posts related to programming or other highly technical stuff that can only be of interest to computer geeks.

  • My First Week Learning vLLM

    My First Week Learning vLLM

    When I first read about KV caching in AI Engineering by Chip Huyen, I remember wondering why I was reading this. I was never going to use it. This week I spent quite a lot of time looking at KV cache charts.

    I’ve been learning deep learning for a while now, mostly through books and by building things. I built and trained a small LLM, then implemented multi-head latent attention from the DeepSeek-V2 paper. I also built a non-LLM model at work for aesthetic evaluation of photos.

    More recently, I wanted to explore the other side: what happens when a model has been trained and there are requests waiting for it?

    I’m currently reading Inference Engineering by Philip Kiely, alongside Build a Reasoning Model (From Scratch) by Sebastian Raschka. I like learning from books. But I was ready to get hands-on, and I chose vLLM as the inference engine to play with.

    For the practical side, I’m using an LLM as a tutor. It put together a curriculum based on what I already knew and helps me turn the lessons into experiments and charts that I can question and explore. Sometimes that works very well; sometimes I have to challenge the explanation or the experiment. The theory from the books helps me spot when something doesn’t add up.

    What followed was roughly a week of experiments on my desktop GPU. I watched requests overlap, forced the engine to repeat work it would normally reuse, squeezed its memory, and changed how it scheduled competing requests. The most exciting part was watching the observations start to connect with the theory, forming a mental model I can reason about.

    This is my account of that process. I’m very new to vLLM, and some of my interpretations may be wrong. If you spot a mistake, please tell me. I’d like to learn.

    All of the experiments shown here were run with vLLM 0.33.0rc1.dev997 at commit c06ddca81c, using Qwen2.5-1.5B-Instruct on an RTX 3090 with 24 GB of VRAM. These are small, synthetic, text-only experiments, not production benchmarks. Timings include the host-side work of driving and observing the engine, not just GPU execution.

    Watching requests overlap

    I started by increasing concurrency and plotting the requests. I wanted a picture of what it meant for one GPU to be serving several requests at once.

    Every request in this experiment had 256 input tokens and generated 128 output tokens. I increased concurrency through powers of two, from one to 512. Each burst contained exactly that many requests. I recorded when each request was submitted, first scheduled, and produced its output tokens. The timelines show waiting, prefill, and decode.

    Prefill: processing the input prompt before generating the response.

    Decode: generating the output tokens one at a time.

    Request timelines at concurrency levels from one to 512, separating waiting, prefill, and decode on a shared time scale.
    Five runs. Each panel shows the run with the median completion time. Timings include host bookkeeping, not just GPU execution.

    This was where I started to understand concurrency in vLLM. It could work on several prompts in the same step, and it could combine that prefill work with decoding requests that were already underway. I began thinking of a step as one iteration of the engine, with a limited amount of token work to distribute across requests.

    Here, the token budget (max_num_batched_tokens in vLLM) was 2,048 token positions per step. That counted input positions being processed as well as positions used to generate the next output tokens. It wasn’t a limit of 2,048 output tokens. Once the budget was used, the remaining work had to wait for another step.

    The groups near the start of the larger bursts made sense once I connected that budget to the prompt length: 2,048 divided by 256 is eight. The first step could process eight complete prompts. Eight wasn’t a fixed limit on how many requests vLLM could prefill; it came from this workload and configuration.

    With chunked prefill, a prompt can be processed over several steps alongside requests that are decoding. In the next step, the eight requests already decoding each used one token position first. That left 2,040 positions for prefill: enough for seven complete prompts and 248 tokens of an eighth. That last prompt would need another step before producing its first output token. As more requests entered decode, their combined share left less room for incoming prompts.

    The shape of the timelines was the part I found especially interesting. In the bursts up to 256 requests, later requests waited longer for their first token, but then spent less time decoding. The early requests had to generate through more of the steps that were also processing incoming prompts. Later requests had fewer of those longer, prefill-heavy steps left to share. All requests still generated 128 tokens, but their blue decode bars had different lengths in elapsed time. Starts spread out more than finishes, giving the right-hand edge of the timelines a steeper slope.

    At 512 requests, I could see a second constraint: the concurrency limit of 256 running requests (max_num_seqs). Once those slots were full, no more prefills started, even though token budget was available. The trace showed steps processing just one decode position for each of the 256 running requests, with the rest of the 2,048-token budget unused. Waiting requests began entering as existing requests finished and freed slots. Their prefills added work to the shared steps again, so the earlier pattern of progressively shorter decode periods no longer held uniformly.

    Two limits explained much of that shape: the concurrency limit and the token budget. These were finite bursts submitted together. I wanted to try staggered arrivals too, but first I had a question about memory.

    KV caching felt much more fundamental than I expected

    KV cache: stored attention keys (K) and values (V) for previously processed tokens, reused when generating subsequent tokens.

    In my own small LLMs, KV caching had been an optimization I would add later. I expected to toggle it off in vLLM to measure its impact, but I couldn’t find an off switch for reuse within a request.

    I had underestimated how much repeated work it avoided. For ordinary full attention, generating n tokens from a fixed-length prompt involves O(n²) total attention work with KV caching, compared with O(n³) if every step recomputes the entire growing history. The official Transformers caching explanation describes the underlying per-step difference.

    To see the effect, I added a scheduler extension that discarded the request’s KV cache and reset the related bookkeeping after each output token except the last. The next pass had to process the prompt and all output tokens generated so far. Prefix caching was disabled throughout; I was comparing reuse between steps of a single response.

    Prefix caching: reusing the KV cache from an earlier request for matching tokens at the start of a new request.

    The engine still allocated and wrote the KV cache on each pass, and the timings included the reset overhead. I varied both input and output length to see how the cost grew:

    Two surfaces compare request duration with ordinary KV caching and forced history recomputation across input and output lengths from 128 to 1,024 tokens. Recomputation grows much more steeply.
    Five runs; medians shown. The surfaces connect measured points.

    At 1,024 input tokens and 1,024 output tokens, the median request duration was about 6.3 seconds with KV caching and 83.3 seconds with forced recomputation.

    I also wanted to separate the effects of input and output length, so I measured inter-token latency. First, I held the output at 128 tokens and increased the input length.

    Inter-token latency (ITL): the time between successive output tokens.

    Four panels show inter-token latency for 128 output tokens as input length increases. Forced recomputation becomes more expensive with longer inputs, while KV caching stays much lower over this range.
    Five runs; medians at each output-token position, with shading spanning the minimum and maximum. The first token is excluded.

    The first token still required processing the prompt in both modes. The difference opened up afterward, when one mode could reuse the previous work and the other kept doing it again.

    Then I held the input at 128 tokens and increased the output length:

    Four panels hold the input at 128 tokens and increase output length to 1,024 tokens, showing growing inter-token latency under forced recomputation.
    Five runs; medians at each output-token position, with shading spanning the minimum and maximum. The first token is excluded.

    As the response grew, recomputing its history made each subsequent pass more expensive.

    The KV caching curves looked approximately flat over this range. I don’t take that to mean decoding with KV caching has a constant cost for arbitrarily long contexts. They just look flat in comparison to how horrible the other ones are.

    From here on, ordinary KV caching within each request stayed enabled in all my experiments.

    Reusing work across requests

    Next, I wanted to understand prefix caching. I’d seen how much work I could save by keeping the KV cache while generating one answer. Now I wanted to see what could be reused when a new request arrived with some of the same input.

    I started each pair with a 1,024-token prompt, then tried four variations for the second request:

    • Repeating it exactly.
    • Appending 128 tokens.
    • Changing token 769.
    • Changing the first token.

    I ran each case with prefix caching enabled and disabled. I wanted to see both how quickly the second response began and whether reusing prompt work changed the pace of the tokens that followed.

    Paired request timelines compare prefix caching on and off for repeated, extended, and changed prompts, separating first-token waiting from later token intervals.
    Five request pairs per case and condition; medians at each token position. Each trace starts with TTFT, followed by ITL. Request 2 runs after request 1, but each request has its own clock starting at zero.

    Time to first token (TTFT): the wait from submitting a request to receiving its first output token.

    For an exact repeat of a 1,024-token prompt, median time to first token fell from about 49 ms to 9 ms. The subsequent token intervals were roughly unchanged, around 6 ms in both cases.

    That was the distinction I wanted to see: less work before the first output token, while the answer still had to be generated.

    There was a detail I hadn’t expected. The exact repeat reused 960 tokens, not all 1,024. In this experiment, each cache block held KV for 64 tokens, and prefix caching reused only complete blocks.

    Cache blocks: fixed-size units of KV storage, each covering a group of token positions.

    The 1,024-token prompt fits exactly into 16 blocks of 64. Even so, vLLM needed to recompute the final prompt token to obtain the logits, the scores used to choose the first output token. The KV cache held attention keys and values, not those logits. Since vLLM couldn’t reuse only the first 63 tokens of that last block, it recomputed the whole block.

    Changing just the first token prevented reuse of the entire remaining prefix in this experiment. Having the same later words wasn’t enough. Their preceding context had changed.

    A conversation that keeps growing

    This became more intuitive when I looked at a conversation over several turns. Each request included the accumulated history, followed by another 128 input tokens, and produced 64 output tokens. I replayed the same controlled transcript in both conditions to keep the inputs comparable.

    Twenty conversation turns compare first-token latency, prompt work, and occupied KV block capacity with prefix caching enabled and disabled.
    Five conversations per condition; medians shown, with TTFT shading spanning the minimum and maximum. Memory includes active and retained cache blocks. Each turn has equal width, regardless of elapsed time.

    By turn 20, median time to first token was about 202 ms without prefix caching and 20 ms with it.

    What surprised me was that peak KV occupancy was the same with and without prefix caching. In retrospect, both modes needed the same keys and values while processing a turn. Prefix caching kept reusable blocks between turns. The separate occupancy panels show that difference. They measure blocks holding active or retained KV data, not the GPU memory reserved for the cache pool. Freeing a block makes it available to vLLM again; it does not mean returning that memory to the GPU allocator.

    Why the reuse happened in steps

    I wanted to see how the 64-token blocks affected prefix caching as the shared prefix grew.

    I used pairs of 320-token prompts with prefix caching enabled. I changed one token in the second prompt to vary the shared prefix from zero to 256 tokens. For example, changing token 65 left a shared prefix of 64 tokens. Later tokens still matched, but their preceding context differed.

    I cleared the cache before each pair, then ran the two requests in order. For the second request, I plotted how many token positions reused cached keys and values, how many needed computation, and the time to first token.

    The number of token positions with reused KV rises at 64-token block boundaries, while first-token latency changes unevenly.
    Five runs. The timing line shows medians; dots show individual runs and bars show their range.

    The number of reused tokens increased in steps: a 63-token shared prefix allowed no blocks to be reused; 64 tokens allowed one. Another block became reusable at 128, 192, and 256 shared tokens.

    The timing curve was less tidy. Reusing another block didn’t always save the same amount of time. I hadn’t profiled why, so I left that as an observation.

    Running out of room

    I had seen how KV caching saved work. Next, I wanted to see what happened when requests competed for limited KV cache capacity.

    I scheduled 32 requests to arrive 500 ms apart. Each had 4,096 input tokens and generated 512 output tokens, with distinct prefixes. I kept the requests and arrival schedule fixed and varied the KV cache capacity.

    Request timelines at five KV capacities show admission waiting in the smallest pools, preemptions at intermediate capacities, and smoother execution with more room.
    Five runs per capacity; the same repetition is shown in every panel, selected by median batch duration in a separate 4 GiB reference condition. The display ends at 20 seconds; some requests finish later.

    I expected enough KV cache capacity to avoid preemptions, and less memory to cause more of them. What surprised me was that they only happened in the middle: at 1.5 GiB, but not at the smaller or larger capacities I tested.

    Preemption: interrupting a running request to free resources for other requests.

    At the smallest capacities, fewer requests could start, so the others waited while those already running finished. At 1.5 GiB, more requests could start, but their growing KV cache then exhausted the available memory. The scheduler interrupted requests to free blocks and later recomputed their discarded state. At the larger capacities, there was enough room for those requests to continue without interruption.

    Some runs had waiting requests without any preemptions. A zero preemption count didn’t mean requests could start promptly.

    These capacity thresholds depend on the workload and execution timing: KV grows as requests generate tokens, and completed requests free their blocks.

    I also plotted how many requests were running or waiting, alongside their KV cache occupancy:

    Paired panels show running requests, requests due but not yet started, and active KV memory use at each capacity.
    Five runs; the same selected runs as the timelines above. Vertical scales differ between capacities. “Offered backlog” counts requests due to arrive but not yet running, including those not yet submitted to the engine. Active KV excludes cached prefix blocks that can be reclaimed.

    I then looked at how long individual requests waited for their first token and how long they paused between output tokens:

    Per-request first-token waiting, maximum output-token gaps, and recomputed positions across five KV capacities, including variation over five repetitions.
    Five runs; medians shown, with shading spanning the minimum and maximum. Measurements cover complete responses, including portions beyond the timeline display. First-token waiting starts at actual submission.

    For request 13, increasing KV cache capacity from 1 GiB to 1.5 GiB reduced time to first token from 2.3 seconds to 258 ms (medians). But its longest pause between output tokens grew from 125 ms to 803 ms (medians). At 1.5 GiB, it also recomputed 1,024 token positions (median).

    Request 13 started sooner with more memory, then got interrupted. Looking only at time to first token would have missed the longer pause during its response.

    Sharing a prefix, leaving room to grow

    On my team at Canva, we have production workloads where the same prompt text is repeated across requests. I’ve seen this at many other companies too. I wanted to understand what happened when those requests shared KV cache blocks and competed for memory.

    I compared two batches of 32 requests: one with prompts about different topics, and one with a common 1,024-token context followed by a different 64-token question for each request. All prompts were 1,088 tokens long, and each request generated 256 output tokens.

    I used prefix caching with 168 MiB of KV cache capacity and a concurrency limit of 32.

    Different prompts make requests wait in groups. Sharing the prefix lets all 32 start generating early, but some responses are interrupted.
    Three runs; the second run is shown for each condition.

    I expected sharing to let more requests run at once and finish the batch sooner. What I learned was what happened after they started: all 32 could begin generating, but there wasn’t enough KV cache capacity for all their responses to grow to completion. Preemptions came in waves as the scheduler made room for requests to continue.

    More requests were preempted with the common prefix: half the responses were interrupted to make room for others to finish. The batch finished much sooner, but from the user’s perspective, an early first token could be followed by a long pause.

    That made me wonder whether using the memory savings to admit so many requests made their responses more vulnerable to interruptions. Would I want extra KV cache capacity for their growth? I hadn’t tested how much memory each workload would need for a chosen number of concurrent requests. Instead, I tried giving the running requests more room by admitting fewer of them.

    I then lowered the concurrency limit from 32 to 16 for the common-prefix workload. Of the 32 submitted requests, only 16 could run at once.

    With shared prefixes, a concurrency limit of 32 allows mid-response interruptions. A limit of 16 moves waiting before generation and avoids preemptions in these runs.
    Three runs; the second run is shown for each condition. All runs had 16 preemptions at the concurrency limit of 32 and none at 16.

    With the lower limit, the second group waited before starting and no requests were preempted. The batch took slightly longer to finish.

    The shared prefix needed to be stored only once, but each response needed its own growing KV cache. There was room to start all the requests, but not to let them all finish together. Admitting fewer left enough space for the running requests to grow without interruption.

    Here, lowering the concurrency limit moved the waiting before the first token and avoided interruptions during decode. The official vLLM tuning guidance also suggests reducing concurrency or increasing KV cache memory when preemptions are frequent; I could now connect that advice to the interruptions I was seeing.

    A new prompt interrupts the rhythm

    Next I wanted to see what happened when a new request needed prefill while another request was still decoding.

    I started one request, then introduced a second with a different prompt after the first had generated eight tokens. The engine had to prefill the new prompt while continuing to decode the first response. Each request had 4,096 input tokens and generated 32 output tokens, with a step budget of 256 tokens.

    The first request's inter-token latency rises while the second request processes its prompt. Request phase bars and step-completion marks align with the latency curve.
    Three runs; the first is shown.

    The step markers helped me see why the first response slowed down. It kept decoding one token per step, but those steps also included prefill for the new request and took longer. The first request was still making progress, yet its user had to wait longer between tokens.

    I then tried ten requests. I scheduled the first five 160 ms apart, giving each prefill time to finish before the next of those requests arrived. I scheduled the next five 60 ms apart, so new prompts arrived before the previous prefill had finished and prefills began to overlap. Each request had 1,536 input tokens and generated 32 output tokens, with a step budget of 256 tokens.

    Repeated incoming prefills line up with larger gaps between output tokens in ongoing requests, shown above a staircase of request timelines.
    Three runs; the first is shown. The latency curve takes the median across currently decoding requests at each step, not across runs. Arrival intervals are targets; requests were submitted between engine steps.

    In the first half, I could see inter-token latency spike during each prefill, then fall once it finished. In the second half, prefill work built up: inter-token latency stayed high, and new requests also waited before their prefill could start. Those initial waits are the gray sections of the timelines.

    This made me curious about disaggregated prefill: running prefill and decode in separate vLLM instances. Prefill felt like a batch job, while decode felt like an interactive job where every pause reached the user. Giving them separate capacity and settings might improve that experience, even with the cost of transferring the KV cache from the prefill nodes to the decode nodes. I’d like to experiment with that, but for now I only have one GPU in one computer.

    My next question was how much prefill work to allow into each step.

    The lesson I still don’t entirely trust

    The token-budget lesson was where the tutoring process became most complicated.

    The expected tradeoff sounded straightforward. Smaller prefill chunks could leave shorter gaps between output tokens for existing requests, while larger chunks could get the new prompt processed sooner. The official vLLM tuning guidance discusses that relationship between the step budget, first-token latency, and inter-token latency.

    But our early experiments didn’t make the story that simple.

    With a small finite workload, increasing the budget could get the prefill work out of the way sooner and improve average ITL, while causing a few much longer pauses. Even p95 could hide the interruptions I cared about if they made up less than 5% of the measurements.

    p95 (95th percentile): a value at or below which roughly 95% of the measurements fall.

    I challenged the lesson because the charts weren’t showing the principle as it had been explained. We changed the workloads and metrics several times. Eventually I wanted a simpler view of the actual interruptions, rather than another summary statistic.

    To see the individual steps clearly, I used eight requests, each with 16 input tokens and 64 output tokens. Once each had generated eight tokens, a new request arrived with a 4,096-token prompt and a 16-token output limit. I compared step budgets of 128, 256, 512, and 1,024 tokens. The new request is the bottom timeline in each panel, starting later than the others. All eight existing requests kept decoding while its prompt was being prefilled.

    Four step budgets process the same incoming long prompt while eight requests decode. Larger budgets produce fewer, longer token gaps in those streams.
    Three runs per budget; selected by the new request’s median time to first token. The ITL curve follows the first of the eight existing requests. Time axes match across panels; ITL scales differ.

    This finally gave me a picture I could follow. With a smaller token budget, the existing requests had lower inter-token latency, but prefill of the new prompt took longer. A larger budget finished that prefill sooner, but increased inter-token latency for the requests already decoding.

    I also compared time to first token with the longest pause between output tokens (maximum ITL) in a separate, longer-workload experiment. Here, eight requests each had 32 input tokens and 256 output tokens. After each had produced 64 tokens, I introduced a 16,384-token prompt with a 64-token output limit. The budgets ranged from 512 to 16,384 tokens per step.

    Connected points compare first-token waiting with the longest pause between output tokens at six token budgets. Larger budgets bring little further reduction in startup wait but much longer streaming pauses.
    Five runs per budget; the run with the median time to first token is shown. Both timings come from that run. Maximum ITL covers the full responses of the existing requests. Labels show the token budget; the line connects the tested budgets.

    The L-shaped curve made the tradeoff clearer to me. I could favor a shorter wait for the new request’s first token or shorter pauses for the requests already decoding. Around the bend, roughly 2,048 tokens per step, there seemed to be a sweet spot for this workload: most of the improvement in time to first token without the much longer pauses at higher budgets.

    I’m still uneasy about how we arrived at the lesson, though.

    I had encouraged the LLM to try experiments until it found conditions that made the principle visible. That can be useful for isolating a mechanism. It can also become a search for a chart that agrees with the explanation we started with.

    Playing with vLLM’s priority field

    The last group of experiments looked at priority scheduling. By then I had several different constraints in mind: which requests could run, how much work could fit in a step, and how much KV cache capacity was available.

    Scheduling priority: a number used to decide which requests get served first.

    In vLLM’s priority scheduling policy, smaller numbers mean higher priority; arrival time breaks ties.

    I wanted to see which of those priority affected.

    Choosing the next waiting request

    The first experiment submitted eight equal-length requests, 500 ms apart, with room for two running requests. Requests 4 and 6 had priority 0 and the others priority 1 in the priority-scheduling run. The first-come, first-served comparison used equal priorities.

    First-come, first-served and priority timelines show requests 4 and 6 moving ahead of older waiting requests, without interrupting the requests already running.
    Four paired runs; the pair closest to the median combined completion time is shown.

    The waiting order changed. The currently running requests weren’t immediately interrupted when a higher-priority request arrived.

    A higher priority didn’t buy a share of every step

    The next experiment made the token budget the constraint, rather than the number of running requests.

    One request began processing a 1,024-token prompt with a budget of 128 tokens per step. After two steps, a higher-priority request arrived with a 256-token prompt. There was room for a second running request, but the first request was still consuming the token budget.

    A step-by-step token-allocation matrix shows identical allocations under both policies: the existing prefill consumes the budget, then decode and the waiting prefill share later steps.
    Five runs, all with the same token allocations. Column widths are schematic, not elapsed time; counts are token positions, not GPU time.

    The higher-priority request waited. Once the first request was decoding, it needed one position per step, leaving 127 positions for the other request’s prefill. Both policies produced the same sequence of allocations in this experiment.

    There was room for another request, but no token budget left for its prefill. Raising its priority didn’t reserve a share of each step.

    Memory pressure changed which request was interrupted

    A separate two-request experiment reduced the KV cache capacity enough that both prompts fit initially, but their growing responses eventually exhausted the available blocks.

    Under KV-memory pressure, first-come, first-served preempts the later request, while priority scheduling preempts the lower-priority request. Both later rebuild KV and finish.
    Five paired runs; the pair closest to the median combined completion time is shown.

    Here, priority did affect a running request: it changed which one became the preemption victim when an allocation failed. In the first-come, first-served run, the later request was interrupted. In the priority run, the older, lower-priority request was interrupted.

    The event was triggered by memory pressure, not merely by the arrival of a higher-priority request. After the surviving request finished, the victim resumed, rebuilt its missing KV cache blocks, and continued generating. The output token sequences matched across the comparisons.

    Waiting can keep getting longer

    Finally, I gave one early request lower priority while a finite stream of higher-priority requests continued arriving.

    Seven later higher-priority requests begin before an earlier lower-priority request, extending its initial wait without any preemption.
    Five paired runs; the pair closest to the median combined completion time is shown.

    All seven later requests overtook request 3. In the pair shown, its initial queue wait grew from about 1.3 seconds under first-come, first-served to about 8.7 seconds under priority scheduling.

    There were no preemptions here. That long interval was waiting to start. The request eventually ran because the stream stopped; the experiment didn’t measure indefinite starvation.

    In these experiments, priority changed the waiting order and the choice of which request to preempt when memory ran out. It didn’t guarantee an immediate start or a share of every step.

    More questions, and a much better picture

    I started because I wanted some hands-on experience. What I enjoyed most was feeling things connect: ideas from books, behavior in the experiments, and questions I hadn’t known to ask.

    I’m still at the beginning, but I can already see how much there is to explore. I love the depth and complexity of these systems, and I’d like to spend more of my time working on them. If one GPU gives me this much to investigate, I can only imagine how interesting the problems become at scale.

    I’m excited to keep going.

  • Applying least privilege to end-to-end encrypted dashboards

    Applying least privilege to end-to-end encrypted dashboards

    Years ago I built an application called Dashman. Customers used it to show web dashboards on screens around an office, a factory, a classroom, or a public space: Google Analytics, internal Grafana, shop-floor counters, org announcements, whatever they cared about as long as it was on web pages. This post is a retrospective on how that system was put together: the architecture, the handling of rendering jobs, and the end-to-end encryption that applied least privilege to every machine in the system, so that a compromise of the server or the public computers would expose only the data that machine needed to do its job, and nothing else.

    The naive approach, and why it fails

    The naive version of the job is easy to state: log into the websites and display them. One machine with a browser could, in principle, do all of it.

    That machine would hold everything: URLs, cookies, and obviously the rendered pages. In many offices the display sat in a lobby where anyone could walk up to it. In plenty of deployments, all you needed to compromise that computer was a USB keyboard. Plug it in and you were inside a computer that still had the customer’s session cookies in memory or on disk. The attack was simple.

    So the first move is architectural, separating rendering from displaying. Rendering needs cookies and a real browser. Displaying only needs the pixels, the rendered image. The machine under the public screen should not be the machine that logged into Google Analytics.

    The second move is end-to-end encryption, making it mathematically impossible to read the cookies anywhere they weren’t needed. That meant excluding both the Displayer and me as the operator, so my customers didn’t have to trust me with their precious cookies. The encryption was the part that I had the most fun with.

    The architecture

    The hosted side had several components, and the customer side had three applications. Together, the whole stack was a heterogeneous distributed system that looked like this:

    Configurator, Renderer, and Displayer on the customer side; Server, PostgreSQL, S3, and PubNub on the hosted side. Screenshots flow directly between clients and S3; the Server hands out presigned URLs.

    • Configurator: a desktop app the customer ran on an administrator’s machine. They logged into dashboard sources here (cookies originated here), chose what to display and where, and approved Renderers and Displayers.
    • Renderer: ran on a customer-controlled machine inside their network. It loaded configured URLs in a built-in browser and snapshotted them. The customer chose where this ran, such as a machine in a data center.
    • Displayer: ran on machines in public places, connected to the screens. It asked for screenshots at its resolution and showed whatever it got back.
    • Server: a stateless REST API that all the client components talked to.
    • PostgreSQL: durable store for accounts, sites, screenshot metadata, and the render queue.
    • S3: encrypted screenshots at rest.
    • PubNub: push channel. The Server published; Renderers and Displayers subscribed.

    A small deployment could collapse Configurator, Renderer, and Displayer onto one physical machine. Larger ones spread many Displayers across screens, ran several Renderers for capacity and redundancy, and kept a few Configurators on the administrators’ laptops.

    Cookies and two keys

    Everything sensitive had to be end-to-end encrypted. Cookies were the most critical piece of information because with them you could just log into all the websites. So I needed one key, which we’ll call the master key, to encrypt them end-to-end. So far so good, not that hard. With that key in place, the cookies on my hosted database were encrypted blobs that meant nothing to me.

    The screenshots that the Renderers generated also needed to be encrypted so that I couldn’t see them. But I couldn’t use the master key for this task because the Displayers couldn’t ever have it. A compromised Displayer should not expose the key that could decrypt the cookies. That meant a second key, the displayer key.

    This table might help:

    ConfiguratorRendererDisplayerBackend
    master key✅ readable✅ readable🚫 absent🔒 encrypted
    cookies✅ readable✅ readable🚫 absent🔒 encrypted
    URLs✅ readable✅ readable🚫 absent🔒 encrypted
    displayer key✅ readable✅ readable✅ readable🔒 encrypted
    Screenshots✅ readable✅ readable✅ readable🔒 encrypted

    This is just the beginning, though. Keeping these two keys encrypted on the Backend but distributed to Renderers and Displayers is where things get really interesting. More on that after a tour of the render loop.

    The render loop

    Dashman’s day-to-day work was a loop. A Displayer on a wall asked the Server for the best screenshot at its resolution. The Server tried the cache first: it picked among recently successful renders for that tenant, scoring candidates by how closely area and aspect ratio matched the request, by their freshness, and with some randomness.

    The goal there was that if you had 10 sites to display and 10 Displayers, you didn’t want to run 100 jobs when most of those would be pixel identical. But also, you didn’t want the 10 Displayers to show exactly the same thing in sync; variety was valuable.

    If nothing in the freshness window qualified, the Server enqueued a render job, told the Displayer to wait, and woke up the Renderers to start working. PubNub handled the fast path: the Server published to the renderer channel when a job was enqueued and to the displayer channel when a screenshot was ready. Renderers also polled on a slower cadence as a fallback. The database was authoritative and PubNub was a hint about when to look.

    After being woken up by a PubNub notification, all Renderers would try to claim a job. Claiming was an atomic row lock, and only one Renderer won. The order of processing was LIFO, not FIFO, because for displaying screenshots, freshness mattered. That meant a render job could be missed and never processed (under heavy load), but that was acceptable. If a Renderer crashed mid-render, the claim aged out and the row became available again. The claiming Renderer loaded the page, snapshotted it, uploaded the PNG to S3, and reported back. To reduce latency, the Server could auto-claim the next job at that point and assign it when responding to the Renderer. That way rendering jobs were chained together.

    Each site had two knobs the Configurator set: how long to wait after load before snapshotting (charts and fonts need time to settle), and how long a Displayer showed that screenshot before rotating. Both were per-site because a quarterly KPI and a minute-by-minute load report required different amounts of freshness and possibly different amounts of time to render fully.

    After all that, the Server notified the Displayer; the Displayer pulled the file from S3 and showed it on screen.

    Sequence: at the top, the Configurator saves a configuration to the Server, which fans the change out through PubNub to Renderer and Displayer; below, the steady-state loop where the Displayer asks the Server, the Server enqueues and publishes, the Renderer claims and uploads to S3, and the Displayer fetches from S3.

    The sequence diagram above shows two flows: an admin saving a configuration at the top (initial setup and occasional changes), then one full pass of the steady-state render loop below.

    I had sketched an alternative push path using WebSockets and SQS. I shipped PubNub instead because it was less code and less operational complexity while finding product-market fit. At higher scale, migrating would have been worth the savings.

    The render loop assumed each component could decrypt what it received and encrypt what it sent. What follows is the cryptography, in the order a real deployment came up: register user and tenant together, log in, change passwords, approve Renderers and Displayers, render with keys, revoke a decommissioned or compromised machine.

    Cryptography overview

    The user’s password was the root of trust. It never left the Configurator in plaintext. For authentication the Configurator ran SCrypt locally and sent the result to the Server, which ran SCrypt again to store an over-hashed password (a hash of a hash). At login it was similar, first hashing on the Configurator and then over-hashing the password to compare against the database. That split is generally a better way to do authentication than just sending a plain text password, but for Dashman it was a must. You will see why below.

    ConfiguratorRendererDisplayerBackend
    password⏱️ briefly in memory🚫 absent🚫 absent➖ absent
    hashed password⏱️ briefly in memory🚫 absent🚫 absent⏱️ briefly in memory
    over-hashed password⏱️ briefly in memory🚫 absent🚫 absent✅ readable

    During registration the Configurator created the two symmetric keys named above: master and displayer key. They were never stored or sent in plaintext to any hosted server.

    ConfiguratorRendererDisplayerBackend
    master key✅ readable✅ readable🚫 absent🔒 encrypted
    displayer key✅ readable✅ readable✅ readable🔒 encrypted

    To pass those keys through the hosted stack, each user also generated a public/private key pair using elliptic-curve cryptography. The master and displayer keys were encrypted with that public key, and that encrypted payload was stored on the server. Only the matching private key could decrypt them.

    ConfiguratorRendererDisplayerBackend
    user’s private key✅ readable🚫 absent🚫 absent🔒 encrypted
    user’s public key✅ readable🚫 absent🚫 absent✅ readable

    But now we had the problem of storing the user’s private key. We needed to encrypt that key and for that, the Configurator used PBKDF2 to create the password-key, a key derived from the same password the user entered to register or log in. Only someone who knew the password could re-derive that key on a new machine and recover the private key.

    ConfiguratorRendererDisplayerBackend
    password-key⏱️ briefly in memory🚫 absent🚫 absent🚫 absent

    If you are following along, the master/displayer keys get encrypted with the public key of the user, and the private key gets encrypted with the password-key. If you feel there’s one step too many there, you’d be right, until you see how Displayers and Renderers are enrolled. Hold on tight.

    This meant that to successfully log in and start operating the system from the Configurator app all you needed was a single password. But at the API level the Configurator needed two pieces of information: the hashed password (SCrypt, for authentication) and the password-key (PBKDF2, for decryption). Both were derived from the same plaintext password, but only the hashed password ever reached the Server.

    Now picture an attacker with persistent access to the Server: they can dump the database and watch incoming login traffic. The dump only yields the over-hashed password, which can’t be replayed to log in. Watching logins over time, however, lets them collect hashed passwords as users sign in, and that is where most apps would be leaking the plain text password. In Dashman, the hashed password still couldn’t decrypt anything: that needed the password-key, which was derived from the plaintext password and never touched the Server. That is what kept a Server compromise from exposing cookies or URLs.

    Each Renderer and each Displayer also generated their own public/private key pair locally, same as the Configurator. Each also generated a random password, kept on that machine, for authenticating to the Server. Unlike with the Configurator, these private keys never left the machine; the Server received only the public key. The credential and the private key lived and died with the machine: unlike users who could log in from anywhere, identity for Renderers and Displayers was tied to a specific machine and not transferable.

    ConfiguratorRendererDisplayerBackend
    a renderer’s password🚫 absent✅ readable🚫 absent🚫 absent
    a renderer’s private key🚫 absent✅ readable🚫 absent🚫 absent
    a renderer’s public key✅ readable✅ readable🚫 absent✅ readable
    a displayer’s password🚫 absent🚫 absent✅ readable🚫 absent
    a displayer’s private key🚫 absent🚫 absent✅ readable🚫 absent
    a displayer’s public key✅ readable🚫 absent✅ readable✅ readable

    The mechanism by which the master key got to a Renderer was that during enrollment that key was encrypted with the Renderer’s public key and stored on the Server. The Renderer fetched that record and decrypted it with its own private key. That is the payoff: the user’s elliptic-curve key pair exists so the master and displayer keys can be re-wrapped for each new Renderer or Displayer using their public keys, without the user’s password ever touching another machine. The password-key alone couldn’t have done that, because it only exists on the Configurator.

    The master and displayer keys were each put in a key ring: the master key ring and the displayer key ring respectively. That was because whenever a Renderer or Displayer was removed, new keys were generated and then rotated. For some time after rotation, data encrypted with the previous keys still needed to be readable, so all parts of Dashman could decrypt both old and new information.

    With all of this in place, we achieved the goal of cookies and URLs only present in the Configurator and Renderer, and screenshots only readable by the customer but not me, completing the full table of all information:

    ConfiguratorRendererDisplayerBackend
    cookies and URLs✅ readable✅ readable🚫 absent✅ encrypted
    screenshots🚫 absent✅ readable✅ readable✅ encrypted
    password⏱️ briefly in memory🚫 absent🚫 absent🚫 absent
    hashed password⏱️ briefly in memory🚫 absent🚫 absent⏱️ briefly in memory
    over-hashed password⏱️ briefly in memory🚫 absent🚫 absent✅ readable
    master key✅ readable✅ readable🚫 absent🔒 encrypted
    displayer key✅ readable✅ readable✅ readable🔒 encrypted
    user’s private key✅ readable🚫 absent🚫 absent🔒 encrypted
    user’s public key✅ readable🚫 absent🚫 absent✅ readable
    password-key⏱️ briefly in memory🚫 absent🚫 absent🚫 absent
    a renderer’s password🚫 absent✅ readable🚫 absent🚫 absent
    a renderer’s private key🚫 absent✅ readable🚫 absent🚫 absent
    a renderer’s public key✅ readable✅ readable🚫 absent✅ readable
    a displayer’s password🚫 absent🚫 absent✅ readable🚫 absent
    a displayer’s private key🚫 absent🚫 absent✅ readable🚫 absent
    a displayer’s public key✅ readable🚫 absent✅ readable✅ readable

    Registration

    Registration is the first ceremony, the moment when everything the rest of Dashman depends on comes into existence: the authentication material the Server will compare against on every future login, the elliptic-curve key pair the user will carry across machines, the symmetric rings that protect cookies and screenshots, and a single root for all of it (the password). It is the longest of the ceremonies because it has to bootstrap every kind of material at once. After it, the Configurator can fetch and decrypt the user’s data from any machine that knows the password, and the Server is left holding only ciphertext.

    Registration: the user enters a password, the Configurator hashes it and posts it for authentication, the Server creates the account and over-hashes, then the Configurator generates the cryptographic estate locally and uploads only the encrypted artifacts.

    Hashing the password

    The user enters a name, an organization, an email, and a password. The password never leaves the Configurator in plaintext. Before it leaves the machine at all, the Configurator runs it through SCrypt configured with this spec:

    ParameterValue
    AlgorithmSCrypt
    Cost (N)2^14 (16,384)
    Block size (r)8
    Parallelization (p)1
    Output length256 bits
    Salt256-bit random, unique per user

    SCrypt is a memory-hard password hashing function: each guess costs not only CPU time but also a sizeable block of RAM, which makes GPU and ASIC acceleration uneconomical for a brute-force attacker. In 2018 it was the strongest password hashing function available in Bouncy Castle, the cryptography library Dashman shipped on, and the cost parameter is tunable: as hardware gets cheaper, the cost goes up, and because the Configurator stores the spec alongside the result, future generations of the same password are not stuck at today’s setting. The per-user 256-bit salt prevents precomputation across the user base.

    The hash and the spec that produced it travel together in the registration request. The plaintext password does not.

    Storing nothing the Server can replay

    The Server does not store what the Configurator just sent. It runs SCrypt one more time over the incoming hash, with a fresh server-side spec, and stores the result. Call that the over-hash: it is the SCrypt of an SCrypt.

    Hash-then-store on the Server is standard. The unusual half is client-side: by SCrypting before sending, the Configurator keeps the plaintext password off the wire entirely. An attacker watching login traffic only ever sees hashes, not the password itself, and in a system like Dashman, where the plaintext password is what unlocks the rest of the cryptographic material described below, that distinction is the whole game. The client’s spec is stored alongside the over-hash so any Configurator can reproduce the same inner hash from the password later.

    Once the Server has committed the over-hash and created the account and tenant rows, it returns enough about the new user for the Configurator to continue. The Configurator now has an account on the Server, but the Server is not yet holding anything sensitive: no cookies, no rings, not even a public key.

    Generating the cryptographic estate

    Everything that protects sensitive data is built next, and it all happens on the Configurator before anything is sent back.

    First, the Configurator generates an elliptic-curve key pair on secp521r1. The curve choice is deliberate. secp521r1 is the largest of the standard NIST curves, and at the volumes Dashman ever expected (small numbers of agents per tenant, low frequency of cryptographic operations) the runtime cost of the larger curve is invisible. The Server will hold the public key and use it later to wrap material for the user; the private key will be the only thing on the network capable of unwrapping that material, and the Server will never see it without a layer of encryption around it.

    Next, it derives a 256-bit symmetric key from the password using PBKDF2 with this spec:

    ParameterValue
    AlgorithmPBKDF2 with HMAC-SHA-256
    Iterations65,536
    Output length256 bits
    Salt256-bit random, per-derivation

    We called the output the password-key. Both SCrypt (used at registration and login) and PBKDF2 run once per login on the Configurator, and both serve the same underlying purpose: making each guess at the password expensive enough that brute force is impractical. They guard different ciphertexts. SCrypt-derived material is what the Server stores as the over-hash and what an attacker would have to crack to recover the password from a database dump. PBKDF2-derived material is what wraps the elliptic-curve private key, where an attacker who got hold of that wrapped blob would need to brute-force the password through AES-GCM to read it. Within the cross-platform Java stack Dashman ran on, SCrypt (via Bouncy Castle) was the strongest password hashing primitive available, and PBKDF2 with HMAC-SHA-256 (via the JCE’s SecretKeyFactory) was the standard option for deriving a symmetric key from a password.

    The Configurator then encrypts the elliptic-curve private key under the password-key with AES in GCM mode (AES/GCM/NoPadding). The IV, the PBKDF2 salt, and the iteration count travel alongside the ciphertext when it is stored on the Server. The password never does. AES-GCM gives authenticated encryption, so a wrong password produces a GCM tag failure rather than silently decrypting into garbage that might look like a private key.

    Two empty rings are created next, the master ring and the displayer ring, and one random 256-bit AES key is added to each. The keys are pulled directly from the platform CSPRNG: there is no derivation, no input, just bytes. A ring is an ordered list of AES keys where the last entry is the current key (used for fresh encryption) and earlier entries are retained so that older ciphertexts remain decryptable. At registration, each ring has exactly one key; rotations described later append more.

    Finally, each ring is wrapped with the user’s elliptic-curve public key using Bouncy Castle’s ECIESwithAES-CBC profile, with a 128-bit nonce and a 128-bit MAC. ECIES, as a family, glues an ECDH agreement to a symmetric cipher and a MAC; the BC profile chosen was the straightforward bundled option at the time, rather than assembling the construction from primitives by hand. Before encryption, each ring is placed inside a small verifier envelope identifying the user and account it belongs to; on decryption the verifier must match or the operation fails. The verifier is what makes a misrouted blob (say, one user’s ring pasted into another’s row) fail closed instead of silently decrypting into something the application would try to use.

    At this point the Configurator holds in memory: the password (still, very briefly), the password-key, the elliptic-curve key pair, both rings, and the SCrypt hash of the password.

    It sends to the Server, in one PUT request: the public key, the ciphertext of the private key, the ciphertext of each wrapped ring, and the parameters needed to rebuild the password-key on a future login. The Server persists all of that as JSON it cannot decrypt.

    By the time the request returns, the password is dropped from memory. From the Server’s vantage, it is holding pieces of mathematics whose meaning is gated on a password it has never seen and on a private key it cannot reconstruct. From the Configurator’s vantage, it now has everything it needs to manage cookies and sites for this account, and a path to recover those materials on any other machine that knows the password.

    Logging in

    Login does two things at once: it authenticates the user to the Server, and it rebuilds the same in-memory state that registration produced. Those are independent code paths that happen to share a single input (the password) and a single transport (the Configurator’s HTTPS connection). If the first one succeeds and the second one fails, the user is signed in but cannot read anything.

    Login: the Configurator fetches the user's SCrypt spec from the Server, hashes the entered password with it, authenticates with HTTP Basic, then receives the encrypted key pair, the two encrypted rings, and the encrypted cookies and decrypts them locally.

    Authenticating

    Each user has their own SCrypt salt, generated at registration. The Configurator does not yet know it on a fresh install, so it asks the Server: given this email, what spec should I use? The answer is not secret. Without the password, the spec is useless. With the spec, the Configurator can reproduce the same SCrypt hash registration computed, regardless of which machine it runs on.

    The Configurator runs SCrypt with the fetched spec and forms HTTP Basic credentials of the form email:base64(hash). The Server runs SCrypt once more over the incoming hash with the spec that sits next to the over-hash, compares byte for byte, and either accepts or rejects. There is no separate session token; subsequent requests reuse the same Basic credentials until the Configurator clears them.

    If the spec the Configurator received is older than current defaults (say, the cost parameter has been raised since the account was created), the Configurator re-hashes the password with the current spec and uploads the upgraded hash on the next save. Work factors rise over the lifetime of the system without forcing anyone to reset their password.

    Recovering the keys

    Authentication only proves the password matches the stored over-hash. The Configurator still has to decrypt everything that follows. The Server returns, in the response to the authenticated GET, the encrypted elliptic-curve key pair, both encrypted rings, and the tenant’s encrypted cookies. The Configurator then:

    1. Reads the PBKDF2 spec stored alongside the encrypted private key and re-derives the password-key from the entered password. PBKDF2 with the same inputs yields the same key it produced at registration.
    2. Decrypts the elliptic-curve private key under the password-key. AES-GCM verifies that the ciphertext has not been tampered with, and a wrong password produces a tag failure rather than a corrupt private key.
    3. Decrypts each ring with the elliptic-curve private key, using ECIES. The verifier inside each ring’s ciphertext must match the expected user and account, otherwise decryption is rejected even if the algebra would have succeeded.
    4. Decrypts cookies under the master ring. The ring is tried current-key first, then older keys, until one verifies; that is how cookies encrypted before the last rotation remain readable.

    By the time the Configurator’s log in process finishes, its memory looks identical to the state at the end of registration, except that the Configurator did not generate any of these materials, it derived and decrypted them. The password is dropped; the password-key has done its job and is discarded with it.

    A practical corner of this design: a wrong password and a tampered private-key blob both manifest as the same AES-GCM tag failure. In practice this rarely caused confusion because the authentication step ahead of decryption already filtered out the common case (a mistyped password).

    The reason the elliptic-curve private key is wrapped under the password rather than stored only in an OS keychain is that Dashman was designed to recover on a fresh install: type the password, get everything back, no preloaded secrets needed. A keychain-only design would have hurt that experience and would not have changed the security story, since whatever the keychain held would still need an unlock secret tied to the user.

    Changing the password

    A password change in Dashman is, deliberately, the cheapest cryptographic operation the system performs. The hard work is concentrated at registration; from that point on the password unwraps exactly one thing (the elliptic-curve private key) and nothing else.

    Password change: the Configurator hashes the current password to authenticate, hashes the new password with a fresh spec, derives a new password-key, re-wraps the elliptic-curve private key under it, and sends the bundle to the Server.

    The flow only runs while the user is already logged in, which means the Configurator already holds the decrypted elliptic-curve private key and both rings in memory. It does not hold the original password (login dropped it), so the user has to enter the current password again to prove identity to the Server, plus the new password to provide fresh derivation input.

    The Configurator does, in order:

    1. SCrypts the current password with the stored spec to construct authentication credentials, the same way login does. This proves to the Server that whoever is asking for the change still knows the password the account was created with.
    2. SCrypts the new password with a fresh spec (new salt, current defaults). This is the hash the Server will over-hash and store going forward.
    3. Re-derives a new password-key from the new password with PBKDF2 and a fresh spec. The old password-key was never persisted, so there is no need to invalidate it; it just stops being useful once the new one is in place.
    4. Encrypts the elliptic-curve private key, which is sitting in memory in plaintext, under the new password-key with AES-GCM. The public key does not change. The rings do not change.

    The Configurator sends, in a single request: the new hashed password and its spec, the new ciphertext of the elliptic-curve private key (with its new PBKDF2 salt embedded), and any other profile fields the user edited. The Server first verifies the request using the current password’s Basic credentials, then over-hashes the new hashed password, stores it in place of the old one, and replaces the encrypted key-pair blob.

    The master and displayer rings are untouched. Cookies, site configurations, and screenshots already in S3 do not need to be re-encrypted: their keys live in the rings, which are themselves wrapped under the unchanged elliptic-curve public key. Re-encrypting any of that on a password change would be a lot of work for no security benefit, since the password only ever protected one specific layer of the cryptographic onion.

    A password change does not retroactively invalidate any other Configurator that previously decrypted the private key. Cryptographic invalidation of in-flight or cached material is what ring rotation is for, and that runs in a different ceremony, described later.

    Approving a Renderer

    Setting up Dashman means enrolling at least one Renderer and at least one Displayer. The two ceremonies share almost the entire shape, both in the UI the customer touches and in the sequence of network calls underneath; what differs is what each agent ends up holding. The Renderer is the fuller case: it receives both rings (master to decrypt cookies and site URLs, displayer to encrypt screenshots). I’ll walk through it first.

    Renderer approval: the Renderer generates its own key pair and a random password and connects unapproved; the user approves it in the Configurator, which wraps both rings under the Renderer's public key; PubNub notifies the Renderer, which fetches and decrypts the rings and the cookies.

    Connecting without keys

    A Renderer is launched on a machine the customer controls. On first start it has no account, has nothing to render, and does not know anyone’s password. The only input it has is a target: a slug or email the user types in that identifies the tenant it wants to join.

    Locally, before contacting anyone, the Renderer generates two things. First, its own elliptic-curve key pair on secp521r1, the same curve the user picked at registration but generated independently and never derived from the user’s. Second, a random password drawn from the platform CSPRNG, which the user never sees. That password is purely a credential for talking to the Server later; it is not used to derive any encryption material. Unlike the Configurator, where a wrapped copy of the private key is uploaded so the user can log in from any machine, a Renderer’s private key never leaves: both the credential and the private key live and die with this specific machine, and onboarding another Renderer means generating fresh ones on the new machine with no way to transfer the original identity.

    It sends to the Server: the tenant identifier, a display name (typically the machine’s hostname, useful when the user is looking at a pending Renderer in the Configurator and has to recognize it), the public half of the key pair, and the random password.

    The Server hashes the password once with SCrypt and persists a Renderer record marked unapproved. A single hash, rather than the double SCrypt used for users, is appropriate here because the password was generated by a CSPRNG and is not a stretching target for an offline guesser; the hash exists only so the Server is not storing the credential in clear. The Renderer is now visible on the Server, but the only material attached to it is its public key, its name, and authentication state. The Server then publishes a PubNub notification on the tenant’s user channel, which is how the Configurator learns there is a pending Renderer.

    Approving

    In the Configurator, the user sees the pending Renderer appear in the list and clicks Approve. This is the cryptographic step: it is the moment the user, who has the only copy of the unwrapped rings, decides that a specific Renderer’s public key is allowed to decrypt them.

    The Configurator wraps both rings, master and displayer, under the Renderer’s public key with ECIES, using the same ECIESwithAES-CBC profile and the same verifier envelope as everywhere else. The verifier here ties the wrapped material to the Renderer’s identifier and the account identifier, so a copy of one Renderer’s ciphertext cannot be reused by a different Renderer even if both belonged to the same tenant. The Configurator then PUTs the wrapped rings, with the Renderer marked approved, to the Server.

    Setting approved without wrapping the rings would be inert: the Renderer can authenticate either way once it has a password, but it cannot decrypt anything until the encrypted rings exist. Approval and key wrapping are bundled in the same request to keep the two from drifting out of sync.

    A subtle point worth pulling out. The user’s rings were already wrapped, at registration, under the user’s elliptic-curve public key. Approval wraps them a second time, under the Renderer’s elliptic-curve public key. The Server ends up holding two ciphertexts that decrypt to the same plaintext: one for recovery on a fresh Configurator (login on a new laptop), one for use on the Renderer. The Configurator only ever sends material wrapped under public keys; the user’s password never reaches the Renderer machine, and the Renderer’s private key never reaches the Configurator.

    Coming online

    The Server publishes a second PubNub notification, this time on the Renderer’s channel, saying the Renderer is now approved. The Renderer fetches its own record (now with encrypted rings populated), decrypts the rings under its own private key, decrypts the tenant’s cookies under the master ring, and from that point on can render pages.

    Approving a Displayer

    The ceremony for adding a Displayer looks almost identical to the one for a Renderer. The Displayer generates an elliptic-curve key pair and a random password locally, sends a connection request with its public key, the Server hashes the password and creates an unapproved record, the Configurator gets a PubNub notification, the user clicks Approve, the Configurator wraps a ring under the Displayer’s public key, the Server stores it, PubNub tells the Displayer, the Displayer decrypts.

    Displayer approval: same shape as the Renderer ceremony, but only the displayer ring is wrapped and shipped.

    But did you notice the difference?

    The Configurator wraps a ring (singular), not the rings. The master ring is not part of the payload, and on the Displayer side, the accessor that would return the master ring deliberately returns nothing. The Displayer’s reality is: it receives only one ring, it only ever decrypts one kind of payload (screenshots), and there is no path in the codebase by which a master key could end up in its memory.

    That asymmetry is the entire reason the Displayer exists as a separate component. A Renderer is a trusted machine in the customer’s network (a data center, an admin’s desk) that needs cookies in order to do its job. A Displayer is the machine wired up to a screen in a lobby or a hallway. Even if a stranger walked up to the lobby machine with a USB keyboard and dumped everything in memory, all they would get is whatever screenshots had recently been displayed and the key that decrypts a few more from S3. The decryption authority that could log into Google Analytics never touches that machine, and the Configurator has no way to send it there.

    The flip side of that constraint is that the Displayer cannot do anything with a screenshot until it has been encrypted under its ring. Which is what the next section is about.

    Rendering a screenshot

    Up to this point every ceremony has been about provisioning. Once Renderers and Displayers are approved, Dashman spends the rest of its life in a loop where bytes flow through the system and end up on a screen. The architectural separation from earlier and the cryptographic plumbing from the last few sections finally pay off together in this loop.

    Screenshot rendering: a Displayer asks the Server for a screenshot, a render job goes through PubNub to a Renderer which fetches encrypted site config, renders the page, encrypts the screenshot for the displayer ring, uploads it to S3, and the Displayer downloads and decrypts it.

    A Displayer asks the Server for the best screenshot at its current resolution. The Server consults its cache. If the cache cannot satisfy the request, it queues a render job for a fresh screenshot and publishes a notification on the renderer channel. The earlier render-loop section covered the cache and the queue; here the focus is on what is encrypted at each step.

    A Renderer wakes up, claims the newest job (LIFO so that a Displayer waiting on the screen sees fresh pixels first), and gets back from the Server two things: the encrypted site configuration (URL, per-site delay, anything else specific to the site), and a pair of presigned S3 URLs (one for upload, one for download), both valid for a day. The site configuration is encrypted under the current master key; the Renderer decrypts it by trying every key in its master ring until one verifies, in practice the current one.

    With the URL in hand, the Renderer loads it in an embedded WebView with the tenant’s cookies attached. After the page has loaded, the Renderer waits the per-site delay (some sites take seconds for charts and fonts to settle), snapshots the WebView, and serializes the result as PNG bytes.

    It then encrypts those bytes with the current displayer key, using AES-GCM, and attaches a verifier identifying the screenshot and the site the bytes belong to. On the Displayer side, decryption will reject any blob whose verifier does not match the screenshot the Displayer asked for. The resulting ciphertext, along with its IV and the small envelope around it, is uploaded to S3 via the presigned PUT URL. The Server never sees the bytes; the upload goes directly from the Renderer to S3.

    The Renderer then PUTs a small marker back to the Server reporting the job done. The Server flips the screenshot to rendered, publishes a notification on the displayer channel, and, as an opportunistic latency improvement, returns the next claimable job in the same response so the Renderer can chain into it without round-tripping through PubNub.

    A Displayer subscribed to the displayer channel receives the notification, fetches the screenshot metadata (which includes the presigned GET URL), downloads the ciphertext from S3, and decrypts it with its own displayer ring. The verifier check rejects any blob whose screenshot or site identifiers do not match what the Displayer asked for: a misrouted file fails closed instead of decrypting into something unexpected. Decryption succeeds, the Displayer hands the PNG to the screen rendering code, and the public sees pixels.

    Three things land at once at this point. The Renderer briefly saw cookies and a fully rendered web page, but they existed only in memory while it painted. The Displayer never saw cookies or a URL; it received an opaque blob, verified it, decrypted it, and showed it. The Server stored a record of which screenshots existed and pointers to where in S3 they lived, but it stored no plaintext; an operator browsing the production database, or an attacker who dumped it, would find ciphertext indexed by tenant. S3 held the ciphertext but could not read it. The end-to-end story holds.

    Decommissioning a Renderer

    Renderers and Displayers come and go. A data center contract ends, a machine is replaced, a Displayer in a lobby is suspected of having been physically tampered with. From a cryptographic point of view, the question is whether the new state of the system can be reached without invalidating the user’s password. The answer is yes, by rotating rings rather than rotating roots.

    Renderer removal: the Server deletes the Renderer record and notifies it via PubNub; the Configurator generates new master and displayer keys, re-wraps both rings for every remaining agent, re-encrypts site data, and saves the whole thing in one transaction.

    When the user removes a Renderer in the Configurator, the Configurator first asks the Server to delete the Renderer row. The Server deletes it and publishes a notification on the Renderer’s channel. If the Renderer is online, it receives the notification, logs out, and stops; its credentials no longer exist on the Server, so even if it tried to keep polling, the Server would reject it.

    Deletion alone is not enough. The removed Renderer kept whatever plaintext it had already extracted, and it kept its copy of both rings on local disk. If it is offline at the moment of deletion (the machine was stolen, the operator only knows it is missing), it will never receive the notification at all. Anything encrypted with the current master or displayer key from this point onward (future cookies, future site configurations, future screenshots) must be unreadable to the Renderer that just left.

    So the Configurator, immediately after the delete completes, performs the rotation:

    1. Generates a new random 256-bit AES key and appends it to the master ring. The previous master key stays in the ring; it has to, because cookies and site configurations encrypted before this moment are still in the database and still need to decrypt.
    2. Generates another fresh 256-bit AES key and appends it to the displayer ring, for the same reason: screenshots already in S3 are encrypted under the old displayer key and still need to be readable by the Displayers that survived.
    3. Re-wraps both extended rings under the user’s elliptic-curve public key, since the user themselves needs to recover the new state on the next login.
    4. For each Renderer that remains approved, re-wraps both rings under that Renderer’s public key with ECIES, replacing the previous ciphertext on the Server.
    5. For each Displayer that remains approved, re-wraps the displayer ring under that Displayer’s public key, replacing the previous ciphertext.
    6. Re-encrypts every site’s configuration (URL and cookies) under the new master key. Because encryption always uses the last key in the ring, anything saved from now on is under the new key only; the old key stays in the ring solely to decrypt historical artifacts.

    All of that, plus the deletion that prompted it, is sent to the Server as one PUT and committed in one transaction. Either every agent is updated and every site is re-encrypted, or the rotation is aborted; there is no window in which some agents have the new ring and others do not.

    The removed Renderer is now in a peculiar but desirable position. It still has its old copy of both rings, so any old ciphertext it kept around (a screenshot that happened to be in transit, a site configuration it cached before deletion) would still decrypt locally. But it has no credentials to fetch anything new from the Server, and even if it had a backdoor channel, every new screenshot in S3 is encrypted under a key it never received, every site configuration the Server holds is encrypted under a key it never received, and any new cookie the Configurator saves is encrypted under a key it never received. The decryption capability the Renderer kept is bounded by what it already had, not by what the system will produce going forward.

    Removing a Displayer follows the same routine. The Displayer never held the master key, so the master rotation is, strictly speaking, more than the threat requires; the displayer rotation is what matters. The codebase rotates both anyway because it is cheap, the routine is shared with Renderer removal, and rotating both leaves no current key the removed agent ever held in either ring. Simpler to reason about, no real cost at the volumes Dashman handled.

    Verifiers and wire formats

    A note about a detail that has shown up in every ceremony so far without much explanation. Every encrypted payload, symmetric or asymmetric, is wrapped in a small typed envelope before it is encrypted. The envelope holds the payload itself plus a verifier: a small object identifying what the payload is supposed to be. For user blobs the verifier carries the user identifier and the account identifier; for renderer and displayer blobs, the agent and account identifiers; for screenshots, the screenshot identifier and the site identifier. The envelope is serialized to JSON, encrypted as one piece, and on decryption the verifier inside the plaintext must match the verifier the caller expected. If it does not, decryption raises a verification failure rather than returning the bytes.

    This is not a digital signature. It does not prove the payload came from a particular party. What it gives instead is type-level safety inside the ciphertext: a payload meant for one tenant cannot be misrouted to another and silently decrypt into something the application would try to use. Crossing tenants is the kind of bug a system like this should not tolerate as silent corruption; verifier mismatches make it loud.

    For asymmetric wrapping the system uses Bouncy Castle’s ECIESwithAES-CBC with a 128-bit nonce and a 128-bit MAC. The curve choice (secp521r1) was the largest NIST curve with a mature ECIES profile in Bouncy Castle. The bigger curve hedges against future cryptanalysis at a runtime cost invisible at Dashman’s volumes. Assembling ECIES from primitives by hand would have been more work for no real gain at the time.

    For symmetric encryption the system uses AES in GCM mode (AES/GCM/NoPadding) with a per-payload random IV. GCM gives authenticated encryption (a tag that fails closed on tampering), which composes cleanly with the verifier envelope: a wrong key produces a GCM tag failure, a right key over the wrong tenant’s payload produces a verifier mismatch, and both are surfaced as decryption failures with no plaintext returned.

    I would revisit the curve and the specific ECIES profile if I were starting again today; standards drift and library defaults have moved on. The compartmentalization story (which keys live where, who can wrap material for whom, and what each component is mathematically prevented from reading) would stay the same regardless.

    That was fun

    Building Dashman was a lot of fun. Thinking through how it could be attacked, from lobby USB keyboards to a malicious operator rifling through S3, was fun in the same way hard puzzles are. Putting it into production and watching real customers use it was satisfying: the design decisions held up under real configuration mistakes, flaky networks, and Renderers disappearing mid-job, not only under the load tests I ran myself.

    I do wish it had seen more sustained load in the wild. It did get real usage, just not the “millions of screens” scale that would have stress-tested every corner of the queue and cache behavior beyond what I could simulate. Still, for a retrospective on a system I built years ago, that is a good problem to have.

  • I got my hand in Apple’s cookie jar

    I got my hand in Apple’s cookie jar

    When I was building Dashman, I hit a wall that Apple’s paid developer support told me was unsolvable. Then I solved it by reading the binaries in assembly until I understood them well enough to swap them out.

    A note up front: this happened a long time ago and I’m writing it from memory. The shape of the work is right; small specifics, API names, and exact sequence of events may be slightly off.

    What was Dashman?

    Dashman was a B2B SaaS for displaying business dashboards (which are, technically, just web pages) on the screens around an office. An early version of it was a native macOS app with an embedded browser. That’s the version this post is about.

    The problem

    On macOS, an embedded browser meant Apple’s WKWebView, which shared a single, system-wide cookie jar with Safari.

    That was a non-starter. Dashman needed to be logged into the dashboards it displayed, typically with accounts different from the user’s: Google Analytics, AWS consoles, internal admin tools. Mixing Dashman’s and Safari’s cookies would have caused constant overwrites in both directions. The longer-term goal was stricter still: one cookie jar per dashboard, so the same install could display two accounts of the same web application side by side.

    Long story short, I needed full control of the cookie jar. Other forms of local storage existed by then but were still nascent; the cookies were where the leakage actually happened.

    I paid Apple’s developer support fee and asked. The reply was friendly, brief, and unambiguous:

    After investigating whether different WKWebViews can each have their own cookie jar the answer is no they cannot — they all share a single container. If you feel that this is something that you need for your application, please file an enhancement request here:

    Challenge accepted, Mr Apple.

    Reading the cookie jar in assembly

    The macOS cookie jar is NSHTTPCookieStorage, a Foundation class with a small public API: get cookies, set cookies, delete by URL, the usual. Underneath, it talks to CFNetwork, the C framework Foundation wraps. The implementation I’d need to replace lived in a binary I didn’t have source for.

    So I opened CFNetwork.framework in Hopper Disassembler and started reading.

    Disassembled CFNetwork in Hopper, showing the implementation of -[NSHTTPCookieStorageInternal initInternalWithCFStorage:] and the symbol list of every NSHTTPCookieStorage and NSHTTPCookieStorageInternal method.

    I sometimes wonder what this would look like on Apple Silicon today: ARM instead of x86, Swift sneaking into Apple’s frameworks. Different bytes, presumably the same archaeology.

    What the disassembly revealed was a three-layer onion. The public class, NSHTTPCookieStorage, was a thin façade. Most of its methods forwarded to an internal Objective-C class, NSHTTPCookieStorageInternal, that held the actual state behind an NSRecursiveLock. That class in turn called down into a family of plain C functions named _CFHTTPCookieStorageCreateDefault, _CFHTTPCookieStorageGetClass, _CFHTTPCookieStorageCreateInMemory, and friends. Foundation wrapper on top, private Objective-C in the middle, C core at the bottom.

    Three-layer diagram of Apple's cookie storage stack, with the public Foundation façade above the private internal class and the CFNetwork store shared with Safari.

    The disassembly told me five things at once. Which selectors did real work. Which were forwarded. Which were vestigial. Where cookies actually got persisted. And, as a bonus, the error strings the binary fell back to when something went wrong: literal byte sequences like Cannot get default cookie store - using a memory store for this process were sitting right there in the data segment, telling me which failure modes Apple’s engineers had anticipated and which they hadn’t.

    It was, honestly, fun. Hopper’s cross-references made it possible to climb up from any C function to every Objective-C selector that called it, and from any selector back down to the persistence calls. After a while I had a mental map of which methods I’d need to satisfy and which I could safely ignore.

    There’s no symbols in the source, which made this hard. Method names survive (Objective-C runtime needs them) but everything else is x86 instructions. What looks in the source like if (self.policy == NSHTTPCookieAcceptPolicyAlways) shows up as a cmp against an immediate, with a jne to a label that another four objc_msgSend calls eventually reveal as the rejection branch. You build the abstraction back up by hand.

    The replacement: a private cookie jar

    What I built was a single Objective-C class, CAHTTPCookieStorage, that implemented the subset of NSHTTPCookieStorage‘s interface that real callers actually used. Cookies live in a dictionary keyed by a composite of (domain, path, name), all lowercased, so duplicates collapse on insert:

    - (NSUInteger)hash {
    return self.domain.hash ^ self.path.hash ^ self.name.hash;
    }
    - (BOOL)isEqual:(id)other {
    if (![other isKindOfClass:[CookieKey class]]) {
    return NO;
    }
    CookieKey *otherCookieKey = (CookieKey *)other;
    return [self.domain isEqual: otherCookieKey.domain]
    && [self.path isEqual: otherCookieKey.path]
    && [self.name isEqual: otherCookieKey.name];
    }

    The lowercasing matters more than it looks. The HTTP cookie spec is case-insensitive on domains and paths in some places and case-sensitive in others, and NSHTTPCookie itself doesn’t normalise consistently. Without the lowercase hash, you can end up with two distinct entries for www.example.org and WWW.EXAMPLE.ORG and behaviour that diverges from Safari’s depending on which one you set last. The test suite (next section) caught this kind of thing immediately.

    Persistence is NSKeyedArchiver to a path under ~/Library/Application Support/<bundle>/Cookies, with the directory created lazily on first write. If the file is corrupted on read we surface it via a NSLog and start with an empty jar rather than crashing the host process. That seems obvious in retrospect; it wasn’t obvious the first time the archive deserialiser threw an exception three frames deep into application launch.

    Domain matching is the one piece that always trips people up. Cookies with a leading dot match subdomains; without a leading dot, they only match exactly. The whole rule collapses to a single ternary once you accept that:

    + (BOOL)match:(NSURL *)url toDomain:(NSString *)domain {
    NSString *host = [[url host] lowercaseString];
    if (host != nil) {
    return [domain hasPrefix:@"."]
    ? ([[@"." stringByAppendingString: host] hasSuffix: domain])
    : [host isEqualToString:domain];
    } else {
    return false;
    }
    }

    Expiry I made lazy. Every call to cookies walks the dictionary and evicts anything past its expiry. Cheaper than running a timer, and it always returns a clean view:

    - (NSArray *)cookies {
    NSMutableArray *cookiesToRemove = [NSMutableArray array];
    for (CookieKey *key in _cookies) {
    NSHTTPCookie *cookie = [_cookies objectForKey: key];
    if ([cookie expiresDate] != nil
    && [[cookie expiresDate] isLessThan:[NSDate date]]) {
    [cookiesToRemove addObject: key];
    }
    }
    for (CookieKey *key in cookiesToRemove) {
    [_cookies removeObjectForKey: key];
    }
    return [_cookies allValues];
    }

    About 300 lines of core logic across the storage class, the cookie key, and the swizzle category. Most of it is RFC 6265 plumbing.

    A test suite for someone else’s code

    Once I had a working mental model, I wrote a test suite. Not against my own code: against Apple’s. The suite exercised every observable behaviour I could find. Cookie expiry, including Max-Age versus Expires. Domain matching, including the leading-dot wildcard rules and the prefix trap (prefixexample.com must not match a cookie set on example.com). Path prefix matching. Secure flag handling. Set-Cookie parsing. The order in which cookies came back. Case sensitivity quirks across domain, path, and name. Cookie acceptance policies (Always, Never). Round-tripping through archive and unarchive. I deliberately included behaviours I suspected were abandoned, on the theory that if they still existed in the binary, something somewhere depended on them.

    The trick that paid off: every test was written so it could run against either the real NSHTTPCookieStorage or my CAHTTPCookieStorage. Same assertions, swap the class name at the top of the file, run again. When something passed Apple’s run and failed mine, I had a divergence to chase. When it passed both, I knew I’d matched behaviour.

    There was a charming downside, immortalised in a comment at the top of the test file:

    Be aware though that these tests are cookie destructive. If you run them against NSHTTPCookieStorage it will destroy all your Safari cookies and if you run them against CAHTTPCookieStorage it will destroy all the Ninja cookies.

    The tests cleared the jar before each run so the assertions had a known starting state. Validate them against Apple’s class to confirm the spec, and you’d lose every Safari cookie on the machine. Bank logins, session tokens, the lot.

    Once swizzling was in (more on that later), I went a step further. Every test got a twin with the suffix ThroughSharedStorage: same assertion, but reaching in through NSHTTPCookieStorage.sharedHTTPCookieStorage() rather than my class directly. This proved two things at once. First, the swizzle was alive: a call into the Foundation singleton was actually landing in my code. Second, my class behaved identically whether you called it directly or arrived through the Apple façade. Both halves had to be true for the integration to be safe, and the dual-path testing made any drift between them impossible to miss.

    This was the contract. Whatever I built next had to satisfy it.

    Method swizzling, the final step

    The last piece was making the rest of the system call my implementation instead of Apple’s. Objective-C method swizzling lets you swap the implementation of a selector with another at runtime: the class is still NSHTTPCookieStorage, the API surface is unchanged, but underneath it now calls into mine.

    I used JRSwizzle as the runtime helper, ran the swap from a +load method (which the Objective-C runtime calls before main), and guarded it with dispatch_once so a category accidentally loaded twice wouldn’t try to double-swap and undo itself:

    + (void)load {
    static dispatch_once_t swizzleMethodsToken;
    dispatch_once(&swizzleMethodsToken, ^{
    NSError *error = nil;
    [[self class] jr_swizzleMethod:@selector(deleteCookie:)
    withMethod:@selector(caDeleteCookie:) error:&error];
    [[self class] jr_swizzleMethod:@selector(cookieAcceptPolicy)
    withMethod:@selector(caCookieAcceptPolicy) error:&error];
    [[self class] jr_swizzleMethod:@selector(setCookieAcceptPolicy:)
    withMethod:@selector(caSetCookieAcceptPolicy:) error:&error];
    [[self class] jr_swizzleMethod:@selector(cookies)
    withMethod:@selector(caCookies) error:&error];
    [[self class] jr_swizzleMethod:@selector(cookiesForURL:)
    withMethod:@selector(caCookiesForURL:) error:&error];
    [[self class] jr_swizzleMethod:@selector(sortedCookiesUsingDescriptors:)
    withMethod:@selector(caSortedCookiesUsingDescriptors:) error:&error];
    [[self class] jr_swizzleMethod:@selector(setCookie:)
    withMethod:@selector(caSetCookie:) error:&error];
    [[self class] jr_swizzleMethod:@selector(setCookies:forURL:mainDocumentURL:)
    withMethod:@selector(caSetCookies:forURL:mainDocumentURL:) error:&error];
    });
    }

    Eight selectors. That was all the public API anyone called. The list maps one-to-one to the symbols I’d seen in Hopper’s left sidebar.

    There was one subtlety. My first instinct was to override +sharedHTTPCookieStorage so that anyone asking for the singleton would get my object instead of Apple’s. That didn’t work, and I wrote down the reason at the top of the swizzle header so I wouldn’t forget:

    We are not hijacking sharedHTTPCookieStorage because something from Safari calls it, gets a copy of NSHTTPCookieStorage and then calls private methods in it that CAHTTPCookieStorage doesn’t implement.

    The crash was deep in CFNetwork, with a stack trace pointing at internal selectors I’d seen but deliberately not bothered to ship. Swizzling on the class itself, on the real NSHTTPCookieStorage, sidestepped it. The shared instance stays the shared instance. Its public methods just do something different now. The private ones still work, because they’re still Apple’s.

    Two side-by-side process containers: a cyan Dashman process whose NSHTTPCookieStorage has the eight swizzled public selectors routed to a remotely synchronized cookie jar, and a red Safari process whose NSHTTPCookieStorage delegates unchanged through CFNetwork to the local cookie jar.

    There was a second subtlety, easy to miss. WebKit, by default, processes cookies for you on every request. If you only swizzle storage, you still get Safari-cookie behaviour because WebKit’s request pipeline reaches into its own cookie machinery before your storage runs. The fix is to disable WebKit’s cookie handling per-request and call the storage manually:

    request.HTTPShouldHandleCookies = false
    CAHTTPCookieStorage.sharedHTTPCookieStorage().handleCookiesInRequest(request)

    Same on the response side: receive the response yourself, extract Set-Cookie headers, hand them to the storage, then return. This had to be wired into WKWebView‘s request pipeline in two places: once on outgoing requests (to attach our cookies) and once on incoming responses (to harvest Set-Cookie). Redirects had to be handled twice, on both legs, because the redirect response itself can carry cookies that the next request needs.

    What shipped

    A single, fully isolated cookie jar separate from Safari, for the macOS build. That solved the immediate problem: nothing else on the user’s machine could be authenticated as Dashman just by being on the same machine, and Dashman couldn’t accidentally inherit the user’s personal Safari sessions either. The longer-term goal of one jar per site I designed for but didn’t ship before the architecture pivoted to a Java/JavaFX client where I owned the entire HTTP stack and the swizzling story became unnecessary.

    I’d call that a healthy outcome. The hack solved a real, present problem. The future problem got solved a different way, by changing the architecture rather than by deepening the hack.

    What I took from it

    A few things I’ve carried with me since:

    One more proof: “can’t be done” is, for me, a starting line. Long before Apple’s reply, treating “impossible” as a cue to dig in had become a reflex. When someone tells me a thing can’t be done, that’s the moment I commit to doing it, and I’ll go as deep as the problem needs. Apple’s reply was technically accurate from inside their boundaries. I went outside them. Here that meant x86 assembly. Next time it’ll be whatever the next problem demands.

    Behavioural equivalence is a real specification. When the source is unavailable, the binary is the spec, and a test suite written against the binary is the next best thing. The suite was as valuable as the replacement, because it told me, precisely, when I’d drifted. The dual-target trick (run the same tests against Apple’s class and mine) and the dual-path trick (run them direct and through the swizzled singleton) were what made the substitution trustworthy rather than hopeful.

  • When failure was an option

    When failure was an option

    A few years ago, at Wifinity, around 4 in the morning, I made one of the hardest decisions of my career: I rolled back a migration we’d spent months preparing for.

    We were migrating the production infrastructure of an ISP serving 200k+ customers. Traffic was already flowing through the new environment in read-only mode, the cutover was minutes from its point of no return, and we’d burned every minute of buffer trying to fix the issue that had surfaced. As we hit the edge of the rollback window, I made the call to bail.

    The rollback executed like the most rehearsed ballet production I’ve ever seen. We re-ran the migration cleanly a week later. Zero customer complaints.

    This post is about how rollback became the thing that made the migration work.

    The situation

    I’d inherited a production environment from a third-party agency that had been running our ISP’s software co-mingled with services for unrelated clients: one shared cluster, one database server hosting multiple databases, our services intermixed with theirs. There was no clean way to separate “our” infrastructure without fully migrating off.

    So we replicated the entire production stack into our own AWS environment, set up database replication between the two, and planned a single overnight cutover. The software was authenticating every Wi-Fi user across military bases, holiday parks, hotels and universities. Communication between the application and the network gear ran in both directions.

    And we had two overnight windows, a week apart, in which to do it. The expectation going in was that one window would be enough. I negotiated for two, because I like building wide margins of error into anything this consequential. If we missed both, the next opportunity wasn’t for another six months.

    Those two windows were both the earliest and the latest realistic options we had. The months on either side were full of problematic dates. It was tight enough that one of the windows fell on my wife’s birthday. She was understanding. Still, as an apology I bought her the Hogwarts LEGO castle. In our relationship it’s become a meme. She asks if work is making me miss her birthday again so she can pick out the next big apology LEGO set.

    The reverse runbook

    Every step of the forward runbook had a corresponding reverse step. About 120 steps in total. Some reverse steps were no-ops. Most weren’t. We wrote them in parallel with the forward steps. Writing the forward runbook and “thinking about rollback later” was never on the table.

    Treating rollback as a first-class artifact had design consequences.

    Engineer the point of no return to be as late as possible. The new infrastructure ran in read-only mode for as long as we could keep it there. Traffic flowed through it, customers got internet directly without going through the captive portal, and no new state was being written to the new system. Until we crossed the read-only-to-read-write boundary, the rollback was always available. We could test that things were working properly and still bail out.

    Keep users online by default. Our internal principle: it’s better to give internet access and not charge for it than to not give internet access at all. With one exception for a customer with specific security requirements, that principle simplified a lot of judgment calls during the cutover.

    Handle the legally-required logs inside the rollback envelope. UK law required us to keep connection logs for a defined retention period. During the read-only phase we captured them via a separate path, in a separate format, in a separate location. We made the deliberate choice to never integrate those logs back into the new system. They sat in storage for the retention period and that was that. Sometimes the right call is to accept a small permanent ugliness in exchange for never having to think about it again.

    The best workflow tool was…

    The migration was more than 100 individual steps, each with its reverse counterpart, forming a dependency graph that would split and merge, with different people in different teams performing different actions. The merges meant a person stopping and waiting for someone else to catch up. Not respecting these dependencies, in some cases, could have had catastrophic results.

    We tried workflow tools that could express all of this, but in the end, the best tool was a spreadsheet. It let us easily reorder steps, which we did a lot during planning, and it was flexible enough to capture everything else.

    Adversarial rehearsals

    We rehearsed the migration repeatedly over the months before the windows. The format was a mix: some live commands against the real environment, some test commands against a test environment that didn’t quite mirror production, and a lot of tabletop walkthroughs. We didn’t have a clean production analogue to play with, so we leaned on inspecting real systems and on injecting faults verbally: “assume this server is not responding from now on, what happens next, can we finish the migration?”

    Each rehearsal was more adversarial than the last. Late in the cycle we ran one where every engineer in the company was invited and asked to aggressively poke holes. They found issues. We kept rehearsing until nobody could think of any more.

    In the final rehearsal, I dropped from the call on purpose, without warning. My role was orchestrating the steps. I wanted to know whether the team could continue without me. Someone picked up the orchestration within a minute and the rehearsal carried on. By the time of the real migration I knew the team didn’t need me present for it to succeed. You know, in case my ISP boycotted me and decided to kill my internet that evening. We had contingency plans for other engineers too, but they were more complex, sometimes forcing us into rollback.

    A note on team shape. Most of my engineers had been at the company about six months. Almost nobody had deep tenure on the inherited system. The rehearsals were both a stress test of the plan and the team’s training course in how the system actually worked. The veterans we had (on the network side, plus a few from the original agency) brought essential know-how and surfaced questions we wouldn’t have thought to ask.

    The decision

    In the first window, a database incompatibility surfaced after traffic was already switched, with the new infrastructure running in read-only mode. We worked the problem. We burned every minute of buffer. As we hit the edge of the rollback window, I had to choose: keep trying to fix it, or bail.

    Even with all the preparation, the call was one of the hardest of my career. The competing voices in my head: maybe one more hour and we solve it. What if the rollback itself fails on something we missed? What’s more risky here: solving an unknown unknown under time pressure, or executing a rehearsed rollback? And underneath all of that, the knowledge that I’d planned the second window as a buffer to not be used, and consuming it meant I’d planned for two windows when I should have planned for three.

    I’d file that as a transferable lesson: buffer that gets used isn’t buffer. If your plan needs all N windows to succeed, you didn’t have margin, you had a fallback. Plan for one more than you think you need.

    I called the rollback. It ran beautifully. Smooth, quick, parallel across teams. We landed back on the original infrastructure right at the edge of our time budget. The rehearsals had paid off.

    The week between

    The week between the two windows looked like roughly one day to understand and mitigate the database issue (a calm second look at the problem, away from the pressure of an active cutover, made it tractable), and the rest of the week double-checking, searching for further unknown unknowns. We didn’t run more rehearsals.

    The mood I shielded my team from: I had dread going in. I was confident in the rollback if we needed it (we’d just proved it worked), but a second failure would no longer have been a tech problem. It would have been a business problem, with consequences I’d be answering for. I don’t think the team felt it. I made sure of that.

    The second window ran clean. We had a 5 hour window booked. We were done in under three. One of those hours was budget reserved for a rollback we didn’t need.

    We monitored customer support tickets live during the cutover. Customers complain about Wi-Fi immediately. At one point during the migration the support monitor interrupted the call to announce a ticket about no internet access. Everyone’s heart stopped. We started checking systems. Then someone re-read the ticket carefully: it was a customer of one of our competitors who had misdialled us. Zero real complaints.

    I’m now ready for my next big migration, but I’ll try to avoid my wife’s birthday. LEGO is expensive.

  • Constant Eval in Production, the idea

    Constant Eval in Production, the idea

    I’m sure this is not my idea, so I’m not claiming it to be. I’ve been wanting to do a sort of continuous AI eval in production for a while, but the situation never presented at work. It was a mixture of having the data to do the eval off line, and wanting to avoid the risks of doing it in prod. But now I’m going to do it for a side project.

    I don’t want to reveal what my side project is yet, so I’ll keep it vague. I’m very excited about this part, so I wanted to share it early. And I’m hoping that the Internet will tell me if, as it usually does, this is a bad idea.

    I have a task that will be done by an AI and I can measure how successful it was done but only 2 to 7 days after the task was completed and seeing it out there, in the world. I will gather some successful examples to use as part of the prompt, but I don’t have a good way to measure the AI’s output other than my personal vibes which is not good enough.

    My plan is to use OpenRouter and use most models in parallel, each doing a portion of the tasks (there are a lot of instances of these tasks). So if I go with 10 models, each model would be doing 10% of the tasks.

    After a while I’m going to calculate the score of each model and then assign the proportion of tasks according to that score. So the better scoring models will take most of the tasks. I’m going to let the system operate like that for a period of time and recalculate scores.

    After I see it become stable, I’m going to make it continuous, so that day by day (hour by hour?), the models are selected according to their performance.

    Why not just select the winning model? This task I’m performing benefits from diversity, so if there are two or more models maxing it out, I want to distribute the tasks.

    But also, I want to add, maybe even automatically, new models as they are released. I don’t want to have to come back to re-do an eval. The continuous eval should keep me on top of new releases. This will mean a fixed percentage for models with no wins.

    What about prompts? I will also do the same with prompts. Having a diversity of prompts is also useful, but having high performing prompts is the priority. This will allow me to throw prompts on the arena and see them perform. My ideal would be all prompts in all models. I think here I will have to watch out for the amount of combinations making it take too long to get statistically significant data about each combination’s score.

    What about cost? Good question! I’m still not sure if cost affects the score, as a sort of multiplier, or whether there’s a cut-off cost and if a model exceeds it, it just gets disqualified. At the moment, since I’m in the can-AI-even-do-this phase, I’m going to ignore cost.

  • Generating an all-files file in Windows, to verify backups

    Generating an all-files file in Windows, to verify backups

    When I was 16 years old or so, one day, my computer didn’t boot. I got a blue screen with some white text-mode error. Something was broken with the file system. This was after being utterly disappointed by Windows 95 and before I became a Linux person, so I was running Windows NT 4.0. I was the only person I knew running that operating system, and thus the only person I knew with an NTFS partition.

    What to do now? That was my only computer, thus I couldn’t get online, smartphones wouldn’t be invented for another decade, I had nobody to ask for help and no tools to run checks on an NTFS partition. That filesystem was quite new back then. I could just blank the hard drive and reinstall Windows NT and all the software. But what about my data? my data!!!

    At 16 years old I learned the lesson that the data is the most valuable and important thing inside my computer. Everything is replaceable. I’m sure if I could see that data now I would life, but for 16-year-old-me, that was my life. I started making backups and since that day I had a personal backup strategy that’s more robust than 90% of the companies I talk to. I have yet to lose a file and I hope to keep it that way. My ex-wife recently recovered from a complete computer failure because she’s following the backup strategy I set up for her.

    One of the things I wonder is, should I have to do a total restore of my data, how do I verify it? I have more than 2 million files. Big chunks could be missing and it might take me years to notice. Because I have so much data to backup, keeping my 3 backups all up to date is hard, so it’s possible that I may have to reconstruct my information piecing things together from the 3 of them. Technically my backup software should be able to do it. But… I’m skeptical.

    This is why every night I have an automatic script that generates a list of all of my files in a text file. That text file gets backed up and unless that files gets permanently and historically lost, I can use it to verify a backup restore. I think my friend Daniel Magliola gave me this idea.

    Since I use Windows (shocker, I know, but try building a Mac workstation with 6 screens and play video games and report back to me), I wrote the script in PowerShell, but since I couldn’t find anything like Linux’s find, the script invokes wsl. Here it is, normally I put it in c:\Users\pupeno\.bin\filelist.ps1:

    echo "Creating list of all files in C:"
    wsl find /mnt/c/Users/pupeno -type b,c,p,f,l,s > C:\Users\pupeno\.all-files.new.txt
    move -Force C:\Users\pupeno\.all-files.new.txt C:\Users\pupeno\.all-files.txt
    echo "Creating lists of all files in D:"
    wsl find /mnt/d -type b,c,p,f,l,s > D:\.all-files.new.txt
    move -Force D:\.all-files.new.txt D:\.all-files.txt
    echo "Creating lists of all files in E:"
    wsl find /mnt/e -type b,c,p,f,l,s > E:\.all-files.new.txt
    move -Force E:\.all-files.new.txt E:\.all-files.txt

    And this is how it’s configured in the Task Scheduler to run every night. First run Task Scheduler:

    Once it’s open, create a new task:

    I hope it helps.

  • Creating blog posts in WordPress.com programatically

    Creating blog posts in WordPress.com programatically

    One of my projects, Unbreach, has a database of more than 600 breaches. These come from haveibeenpwned and they are composed of some metadata, a one-paragraph description, and an image. I wanted to improve these with more content, links to articles, tweets, videos, and some content of my own.

    I decided that a good way to do it would be to move them from the app, which resides at app.unbrea.ch, to the marketing website, which is at unbrea.ch, essentially creating them as blog posts. That way after the blog post is automatically created (when haveibeenpwned ads the breach), I can go in and manually edit it in all the WordPress glory. I thought this was going to take me a few hours, not days.

    Hopefully, with this blog post, it’ll only take you hours. I’ll be using Ruby but it should be trivial to translate it to Python, JavaScript, or any other programming language. Writing the code wasn’t the hard part, understanding the WordPress.com world was.

    WordPress has two different APIs that should be able to accomplish this task, one is the XML-RPC API and the other is the REST API. The XML-RCP API depends on a file called xmlrpc.php and it’s strongly recommended you leave this deactivated because it had a lot of security issues. It’s also old, cumbersome, and possibly on the way out. I didn’t want to use it and I don’t think you should either.

    From what I can gather the REST API is what the admin tool uses, so using it sounds like a safe bet. If you are going to be creating blog posts from an unattended background process, as I do, you’ll find your first obstacle when you read about authentication because it just assumes there’s a browser sending cookies.

    Fear not! There are plug-ins that implement other authentication methods and one of those is the Application Passwords plug-in. Which is now discontinued because it’s been merged into WordPress itself in version 5.6. This sounds promising until you realize the feature seems to be missing in WordPress.com.

    If you search how to create an Application Password on WordPress.com you’ll land in the wrong place. WordPress.com users have an Application Password that’s hidden behind the Two-Step Authentication in Security. This is what it looks like:

    If you are here you are in the wrong place

    What’s going on here? Well, WordPress.com has its own API, which is a REST API, and if you talk to support and WordPress.com they’ll point you to that. I wasn’t a fan of that solution because although I want to use WordPress.com, I don’t want to be tied to it. I want to be able to move to WP Engine or something like that whenever I want.

    That API, similar to the REST API, assumes there’s a human interacting through a third-party application, so it’s not great for unattended processes. Authentication works using OAuth2 which for a background job that just needs an API key I find very annoying. It’s doable but annoying. Well… it’s doable until you enable 2FA and then it’s not doable anymore, and that’s why that specific Application Password exists.

    WordPress.com support also told me that the WordPress REST API is enabled only if you are on a business plan or above.

    So… where’s the Application Password for the REST API then? I don’t know if there’s a link to it anywhere, but you get to it by going to https://example.com/wp-admin/profile.php where example.com is the URL of your blog. That is, add /wp-admin/profile.php to it. On WordPress.com’s defense, it was their support that finally pointed me to it. When you go there you’ll see an old-style profile page:

    The correct place to set up an application password to use the WordPress REST API

    The previous Application Password was tied to the user, this one is tied to the user and the site, so if you have more than one site you’ll need to create one per site.

    And that was the hard part. Once I got that application password things just worked. It’s a straightforward and mostly well-documented API. I’ll share my messy code here anyway (sorry, didn’t have time to clean it up).

    In Ruby I’m using a library called Faraday to talk to APIs. The first thing is creating the Farady object that has the metadata that will be used in all the requests:

    auth_token = "#{Rails.application.credentials.wordpress&.username}:#{Rails.application.credentials.wordpress&.app_pass}"
    auth_token = Base64.strict_encode64(auth_token)
    conn = Faraday.new(url: ENV["WORDPRESS_URL"],
      headers: { "Authorization" => "Basic #{auth_token}" }) do |conn|
     conn.request :json
     conn.response :json
    end

    According to Faraday’s documentation, this should have worked as a better way of setting up the authentication details:

    conn.request :authorization,
                 :basic,
                 Rails.application.credentials.wordpress&.username,
                 Rails.application.credentials.wordpress&.app_pass

    but for me it didn’t. It was completely ignored.

    The first thing I need is the id of the category in which these posts will end up. This is very important because they appear on a separate page about breaches and not on the blog and that’s achieved with categories:

    response = conn.get("/wp-json/wp/v2/categories", {search: "Breach", _fields: %w[id name]})
    if response.status != 200
      raise "Unexpected response #{response.status}: #{response.body}"
    end
    category = response.body.find { |category| category["name"] == "Breach" }

    Now, if the category doesn’t exist, I want to create it:

    if category.nil?
      response = conn.post("/wp-json/wp/v2/categories") do |req|
        req.body = {name: "Breach"}
      end
      if response.status != 201
        raise "Unexpected response #{response.status}: #{response.body}"
      end
      category = response.body
    end

    Then I needed to do the same with tags. In my case, the tags were in a field called data_classes and the code for getting the id of the tag and creating it if it doesn’t exist is very similar:

    tags = data_classes.map do |data_class|
      response = conn.get("/wp-json/wp/v2/tags", {search: data_class, _fields: %w[id name]})
      if response.status != 200
        raise "Unexpected response #{response.status}: #{response.body}"
      end
      tag = response.body.find { |tag| tag["name"] == data_class }
      if tag.nil?
        response = conn.post("/wp-json/wp/v2/tags") do |req|
          req.body = {name: data_class}
        end
        if response.status != 201
          raise "Unexpected response #{response.status}: #{response.body}"
        end
        tag = response.body
      end
      tag
    end

    And finally, we can create the post. I create the content as an HTML snippet which causes WordPress to interpret it as classic content, not as blocks. But that’s fine because it renders well and the first time I edit one of those posts converting them to blocks is two clicks and works perfectly for this simple content.

    content = <<~CONTENT
      <p>#{description}</p>
      <p><!--more--></p>
      <p>Accounts breached: #{pwn_count}</p>
      <p>Breached on: #{breach_date&.strftime("%B %d, %Y")}
      <p>Exposed data: #{data_classes.to_sentence}</p>
      <p>Domain: #{domain}</p>
      <p>Added on: #{added_date.strftime("%B %d, %Y")}</p>
    CONTENT
    response = conn.post("/wp-json/wp/v2/posts", {
      title: title,
      content: content,
      excerpt: description,
      status: "publish",
      categories: [category["id"]],
      tags: tags.map { |tag| tag["id"] },
      date_gmt: (breach_date.to_time(:utc) + 12.hours).iso8601.to_s,
      template: "breach-template",
      ping_status: "closed"
    })
    if response.status != 201
      raise "Unexpected response #{response.status}: #{response.body}"
    end
    post = response.body

    At this point, I wasn’t done. I wanted these posts to have the image associated with the breach (the logo of the company breached). The first step was downloading it which was a trivial one-liner:

    logo_request = Faraday.new(url: logo_path).get("")

    In that code, logo_path is actually a full URL of the file.

    To create media items in WordPress, I needed to encode the post as multi-part, so I ended up creating a separate Faraday object for that:

    multipart_conn = Faraday.new(url: ENV["WORDPRESS_URL"],
      headers: {"Authorization" => "Basic #{auth_token}"}) do |conn|
      conn.request :multipart
      conn.response :json
    end

    It should have been possible to use a single Faraday object for all requests, but when you specify multipart, you need to take care of encoding the JSON requests yourself and adding them as one of the parts. This is where I got lazy and just moved on with my work.

    The code for creating the image in WordPress is this:

    extension = File.extname(logo_path)
    file_name = "#{name.underscore.tr("_", "-")}#{extension}"
    content_type = if extension == ".png"
      "image/png"
    else
      raise "Unexpected extension #{extension}"
    end
    media = multipart_conn.post("/wp-json/wp/v2/media", {
      date_gmt: (breach_date.to_time(:utc) + 12.hours).iso8601.to_s,
      status: "publish",
      title: title,
      comment_status: "closed",
      ping_status: "closed",
      alt_text: "Logo for #{title}",
      caption: "Logo for #{title}",
      description: "Logo for #{title}",
      post: post["id"],
      file: Faraday::Multipart::FilePart.new(StringIO.new(logo_request.body), content_type, file_name)
    })

    In reality, 100% of the images are PNG so I was ok with such a simplistic approach. When creating the FilePart I wrapped logo_request.body in a StringIO because it already contained the binary data of the image. If you have a local file you can just pass the path to FilePart.new and it just works.

    And now that I had the image, I could set it as the featured image for the post I created earlier:

    response = conn.post("/wp-json/wp/v2/posts/#{post["id"]}", {
      featured_media: media.body["id"]
    })
    if response.status != 200
      raise "Unexpected response #{response.status}: #{response.body}"
    end

    The reason why I didn’t create the image before creating the post was so that I could pass the post id to the image and thus the image would be connected to the post. I’m not sure how useful that is.

    And that’s all.

    I wonder if this code should be put in a gem and made reusable. WordPress points to the wp-api-client gem as the Ruby solution, which is read-only and abandoned. There’s also wordpress_v2_api, but I wasn’t a fan of the API (it’s almost like using HTTP directly), it hasn’t been touched in 6 years and I don’t believe it supports writing. I’m half tempted to fork wp-api-client, but does anybody else care, or is it just me? Please leave a comment if this is something you want to use.

  • This simple GitHub config change puts can put you at the top of my hiring list

    This simple GitHub config change puts can put you at the top of my hiring list

    I’ve hired about 20 developers in my career so far (and I’m looking forward to hire more). When job applications arrive I separate them in three piles: Yes, No, and Maybe. It’s better to do Yes and No piles, but it’s a luxury that I haven’t had (if curious, drop a comment and I’ll write another blog post about it).

    In the No-pile I put all those people that are obviously not a match, people that explicitly tell me they never wrote code before, people outside the time zone target, applications with grammar so bad I can’t understand them, etc.

    […] the most important bit: I find some evidence that they have written code before.

    In the Yes-pile I put all those that show promise. Their application looks good, their profile match and this is the most important bit: I find some evidence that they have written code before.

    The rest of the applicants go to the Maybe-pile. They are not discarded, but I’m going to focus on the ones in the Yes-pile first because I believe I’ll find more successful candidates there than in the Maybe-pile. This doesn’t mean that someone brilliant isn’t in the Maybe-pile. It only means that I couldn’t find any evidence about their potential brilliance.

    Landing on the Maybe-pile is almost as bad as landing on the No-pile

    Here’s the kicker: I never get to the Maybe-pile. I always find all the candidates I want from the Yes-pile. Landing on the Maybe-pile is almost as bad as landing on the No-pile.

    There are many ways in which you can make yourself go from the Maybe to the Yes pile. These are the best ways: blogging about code you write, writing tutorials, contributing to open source software. Having done all of that will not only put you in the Yes-pile, it’ll probably put you at the front.

    But those are things that require a lot of time an effort. There’s another thing that may only require a couple of minutes. If you’ve been using your GitHub account for work or for university, that account has a lot of activity. You may not be able to show code from work and you may not want to show code from university, but you may show the activity. Go to your Public Profile settings and tick “Include private contributions on my profile”:

    This, if you have any private activity, will turn your public GitHub profile from something that looks like a ghost town:

    into something like looks more lively:

    The latter looks like a developer that wrote some code. It will not send you to the top of the Yes-pile, but if the rest of your application looks good, it might be enough to send you to the Yes-pile and it’s a change that requires only a few seconds.

    Now, if you are in the know, you might object that this is completely fakable, and you are right. Yet, I don’t see people faking it. I see lots of empty profiles looking sad and empty, so it’s still a useful signal.

    I’m not trying to find a perfect way to evaluate candidates, I’m trying to find a heuristic to help me find which ones to evaluate first

    Even if some people fake it, I might still continue using the signal. It’s not like you don’t have to pass all the interviews after this anyway. Remember that I’m not trying to find a perfect way to evaluate candidates, I’m trying to find a heuristic to help me find which ones to evaluate first because it’s impossible to evaluate everyone.

    Now, if you decide to fake it, you have two paths. The first one is to just fake, essentially lying about it. That’s a deception. The second is to troll. You can spell out your name or draw something in that part of GitHub. I won’t know if you have real activity or not, your profile won’t look as empty, but at least I know you know enough about coding to pull off that.

    You could argue that because some people wrote the code to make that happen and uploaded to GitHub, all you have to do is find it and run it, and it doesn’t prove that you are above script kiddie. And then again, most applications I have don’t even have script kiddie level of a display of ability, so you may still be coming ahead.

  • One on One Meetings (a free chapter from How to Hire and Manage Remote Teams)

    One on One Meetings (a free chapter from How to Hire and Manage Remote Teams)

    This blog post is a sample chapter from my book:
    How to Hire and Manage Remote Teams

    The term one-on-one evolved to refer to a specific type of meeting and does not mean any general meeting that has two people on it. One-on-ones are the regular meetings between a manager and each of their reportees. One-on-ones are always important but in a distributed team they become critical – mainly because you are not going to be able to run into your reportees on the hallways and have a quick meaningful conversation opportunistically.

    A one-on-one has several goals and I recommend the relevant chapters on The Hard Thing About Hard Things (Ben horowitz) and High Output Management (Andrew Grove) in order to understand more about how to conduct them and their importance.

    The main and most critical role of the one-on-one is to identify problems, or potential problems, as early as possible. By the time one of your employees tells you they have another job offer, more often than not, it’s way too late to do anything about it. The issues leading to their moving posts will have started long before, perhaps months, and the opportunity to address them will have been when it first arose. If you’re not aware, and not providing the opportunity for your worker to make you aware, a problem can fester beyond the point of salvation. The one-on-one aims to reduce this possibility.

    A one-on-one is the time when an employee can bring up small issues:

    • I’m not happy with our project.
    • Work is boring.
    • I feel my ideas are ignored.
    • This person is being rude to me.
    • The way we work just sucks.

    And from that information it is your job to start fixing it. 

    However, it is unlikely your employee will feel empowered to say any of this unless they already trust you. For this reason, one-on-ones should not be reserved for only dealing with critical issues, or when you suspect there could be a problem. Routinely meeting is an exercise in building the rapport that will be required for the worker to bring up any real problem.

    For those uneventful one-on-ones, it is important that you keep a balance between listening and sharing. If you don’t share anything at all, you’ll end up coming across as an interrogator. It is through sharing that you show them that you are listening and empathizing with what they are saying.

    The opposite tends to be a bigger problem. If you as the manager go into long monologues during the one on ones most of your employees will respectfully listen and hate every minute of it. Many of us tend to have a bias that means that if we feel we spoke for 50% of the time, we actually spoke 70% of the time, so you might need to rein it in.

    Since you are doing this remotely, you could get a stopwatch and during a couple one-on-ones measure how much you talk or how much they talk. Don’t worry about getting too precise (for example, when switching back and forth quickly during clarifying questions). This is just a rough approximation. Your performance during those one-on-ones will be distracted, so this isn’t something to do regularly, but it can be a useful self-test to show yourself your baseline.

    To achieve a good balance, here’s a recipe you can follow:

    1. Start with the pleasantries: How are you? Fine, you? Fine. This is neither sincere nor useless. It’s a protocol to start communicating, it establishes cadence, tone of voice. Don’t ignore it, but also don’t take it at face value.
    2. Ask one or two questions to let the worker become comfortable talking. “How was this week?” or “That was a good release, wasn’t it?”
    3. Re-ask them how they are doing: “Now, really, how are you doing? All good in your life?”. Now is when you need to start practicing silence.
    4. Ask them if they have any questions for you. Continue practicing silence.
    5. Ask them what was the best or worst part of the week. Again… silence.

    When I say practice silence, what I mean is that you ask the question and then shut up. Different people take different amounts of time to start talking. Especially if they need to bring up a difficult subject, which can require mustering some courage. Give them time and space. This will feel uncomfortable, but it’s a skill you need to master. There are some stereotypes that introverts are more comfortable with silence than extroverts, but I’m not sure how true it is. If you are a manager, you are probably more comfortable talking than the worker (if they are developers for example), so you might need to put up with more discomfort.

    The initial “How are you?” question is part of the protocol of starting a conversation. There are some studies that show that it establishes speed cadence, tone of voice and other aspects of communication. What it’s not is a sincere question of how someone is doing. Don’t expect people to answer it truthfully; if they do, great! But most people need to be asked twice. But… avoid asking twice in a row. When we are asked the same question twice in quick succession, it engenders the feeling of being accused, as though we are lying or are being unreliable – so most of us will dig in our heels and avoid changing our answer. Most of us need to warm up to a conversation before we can answer truthfully. So ask some other benign questions, and then circle back round to it. 

    If at any point during that recipe your worker takes off on a tangent that is useful, as in it’s providing you with the information you need about their wellbeing, drop the recipe and follow their lead. The recipe is there for the cases where a worker is being more passive, which is likely to be true when you are just starting to work together.

    I recommend taking copious notes during the ones-on-ones and following up on things that were happening. These notes should be strictly private.

    I tend to run one-on-ones in two different schedules:

    • Weekly but optional.
    • Every other week but mandatory.

    Every other week but mandatory is the default way I use to schedule one-on-ones. Weekly but optional is a scheme that I use under many special circumstances:

    • The employee and I don’t know each other well.
    • The employee is new to the team.
    • I am new to the team.
    • There’s an ongoing issue or conflict.
    • They are a junior employee needing more guidance.

    The optionality has limitations: we can skip one, if they are being productive and have no pending issues to discuss in the one-on-one. Once it’s skipped one week, the next week it becomes mandatory.

    Normally I book one-on-ones to last 25 minutes with five minutes for me to finalize my notes. If someone reaches the end of the one-on-one and there are pending issues to resolve, book another meeting straight away to continue working on it (this rarely happens).

    It’s very easy for one on ones to become status reports. It’s something easy for the worker to say, and it’s something easy for the manager to consume, but the one on ones are not about performance or what got done. To drive that point: imagine the situation in which the line manager and the project manager are different people, the line manager does the one-on-one but the project manager cares about the status reports of what has been done. 

    Instead, I suggest you just say something along the lines of: “This is not about a progress report, but was there anything you enjoyed or that annoyed you this week?”

    That last part of the question allows the worker to do a retrospective and instead of talking about having achieved tasks A, B and C, they can talk about how whilst doing C they had a conflict with another worker; or how they loved doing B and wished they were doing more of that. Those are important signals for you.

    The one-on-ones cannot be extremely transactional. It takes time for someone to be comfortable to tell you about a problem they have. This is normal: people literally go to the doctor, where their confidentiality is protected by law, and still procrastinate on sharing something because it’s uncomfortable. So don’t expect that a worker will just show up and tell you there’s a problem because you have an “open door” policy. You need more than that. You really need to prove yourself approachable proactively, and that happens through repeated positive small interactions.

    If, like me, you have a terrible memory, write down the names of their spouses, children, pets, birthdays, what’s going on with their lives and ask them about it the next time (“how was your holiday to Mallorca?” or “How was [daughter’s name] school play?”). These are notes that I consider extremely private; not to be shared with employers or other managers. I don’t even write them on a medium they could gain access to (normally I use paper). For me, it’s the same as with any other friend: I have a calendar with their birthdays, because it’s important that I don’t miss them.

    In Summary: one-on-ones are often neglected or perceived as only necessary when something is already wrong. In fact they are a vital means of establishing rapport with your team and keeping on top of what’s going on with your employers. This is really the only way you’re going to be able to identify minor issues before they become big problems, and has knock-on effects on employee retention, team morale, and productivity.

    This blog post is a sample chapter from my book:
    How to Hire and Manage Remote Teams

  • Using Font Awesome 6 in a Rails 7 project that uses importmaps

    Using Font Awesome 6 in a Rails 7 project that uses importmaps

    I just figured out how to use Font Awesome 6 in a Rails 7 project that uses importmaps. I’m not entirely sure why this works and why some of the workarounds are needed, but my googling yielded no results when I was searching so hopefully here I’ll be saving the next person some time.

    If you search Rubygems for gems with the name “font awesome” you’ll find quite a few but I didn’t like any of them. They all use the font version of the icons, instead of the SVG, or they are very outdated, or they expect you to use SCSS, which I’m not using at the moment. But ultimately, the team at Font Awesome maintains the NPM packages and we should use those directly, not re-wrap packages that will always be out of date.

    For me, using NPM packages directly was higher priority than using importmaps. That’s how strongly I feel about it. I would have installed Webpacker to use Font Awesome’s main package.

    I managed to make this work, but if I’m frank, I’m not 100% sure why the workarounds are needed, so if you have any insights about it or how to improve this, please drop a comment.

    Font Awesome’s documentation says you should install the fontawesome-free package:

    npm install --save @fortawesome/fontawesome-free

    Instead we are going to pin that package, but also some of the dependencies we need later:

    ./bin/importmap pin @fortawesome/fontawesome-free \
                        @fortawesome/fontawesome-svg-core \
                        @fortawesome/free-brands-svg-icons \
                        @fortawesome/free-regular-svg-icons \
                        @fortawesome/free-solid-svg-icons

    This adds the following lines to your importmap.rb:

    pin "@fortawesome/fontawesome-free", to: "https://ga.jspm.io/npm:@fortawesome/fontawesome-free@6.0.0/js/fontawesome.js"
    pin "@fortawesome/fontawesome-svg-core", to: "https://ga.jspm.io/npm:@fortawesome/fontawesome-svg-core@1.3.0/index.es.js"
    pin "@fortawesome/free-brands-svg-icons", to: "https://ga.jspm.io/npm:@fortawesome/free-brands-svg-icons@6.0.0/index.es.js"
    pin "@fortawesome/free-regular-svg-icons", to: "https://ga.jspm.io/npm:@fortawesome/free-regular-svg-icons@6.0.0/index.es.js"
    pin "@fortawesome/free-solid-svg-icons", to: "https://ga.jspm.io/npm:@fortawesome/free-solid-svg-icons@6.0.0/index.es.js"

    Then Font Awesome’s documentation says you should add these lines to your code:

    <script defer src="/your-path-to-fontawesome/js/brands.js"></script>
    <script defer src="/your-path-to-fontawesome/js/solid.js"></script>
    <script defer src="/your-path-to-fontawesome/js/fontawesome.js"></script>

    Which might make you think this is a good idea:

    <script defer src="https://ga.jspm.io/npm:@fortawesome/fontawesome-free@6.0.0/js/brands.js"></script>
    <script defer src="https://ga.jspm.io/npm:@fortawesome/fontawesome-free@6.0.0/js/solid.js"></script>
    <script defer src="https://ga.jspm.io/npm:@fortawesome/fontawesome-free@6.0.0/js/fontawesome.js"></script>

    But it doesn’t work. It fails with this error:

    Here I’m a bit confused. How come it fails with that error? Any ideas?

    What did work was editing app/javascript/application.js and adding::

    import {far} from "@fortawesome/free-regular-svg-icons"
    import {fas} from "@fortawesome/free-solid-svg-icons"
    import {fab} from "@fortawesome/free-brands-svg-icons"
    import {library} from "@fortawesome/fontawesome-svg-core"
    import "@fortawesome/fontawesome-free"
    library.add(far, fas, fab)

    I can’t help but feel that there’s a function or method in fontawesome-free that I could call that would do all the setup automatically with less imports and less library building, but I couldn’t find it yet.