Engineering & Technical

The Engineer

The Signal

Kubernetes v1.35 kubelets refuse to start on cgroup v1 nodes.

Your next upgrade breaks on any node that isn't cgroup v2, and kubeadm preflight now fails instead of warning, Chris Short reports. Like Deno Deploy's shutdown and the crypto-agility debate, the bill lands on whatever you hardcoded, so sweep every node pool before you roll.

In Play

  1. Kubernetes v1.35 Ends cgroup v1 Nodes

    Kubernetes v1.35 defaults failCgroupV1 to true, so the kubelet will not start on a cgroup v1 node and kubeadm preflight fails instead of warning, Chris Short reports. For your clusters, an unchecked node pool now means a failed upgrade rather than a log warning. The reward for migrating is node swap, GA since v1.34, which took gVisor Python sandboxes from 80 to 240 pods per node on GKE NVMe Local SSDs.

  2. Cloudflare Buys Deno, Deploy Sunsets

    Cloudflare is acquiring Deno and will shut down Deno Deploy about six months after the October 10 announcement, around April 2027, Techpresso reports. The standalone runtime gets one year of security updates, to roughly October 2027, and no further development. Every service on Deploy now has a fixed exit date, and the promised self-hosted Celld/Workerd build has no release date to plan against.

  3. AI Hasn't Moved Crypto Assumptions

    Stanford's Dan Boneh and an a16z crypto panel found that OpenAI's recent math results contained no cryptography problems and weakened no assumption, elliptic curves included. Justin Drake, quoted in Exponential View, still warns AI could undermine digital signatures before quantum computers do, though no mechanism is cited. Both views point at the same exposure in your code: crypto bugs AI makes cheaper to find, and systems that cannot swap algorithms without a deploy.

  4. Agent PR Flood Hits Reviewers and Maintainers

    Agent-written pull requests are reportedly up 9x in eight months. Rahim Hirji's Box of Amazing names the failure: reviewers of a confident diff check correctness, not whether it solves the right problem. Upstream, curl ended its bug bounty and Sindre Sorhus and Hono closed external PRs, which James Ross attributes to AI making contributing cheap and maintaining expensive. Upstreaming a fix is no longer a dependable plan, and your own reviewers face the same flood.

  5. Production Automation Keeps Models Narrow

    Standard Bots told Latent.Space its models run on local edge GPUs and handle only perception, while conventional code owns motion and tasks stay short-horizon. WSJ reporting on Walmart, via Morning Brew, puts automated grocery facilities at over $800K a month in electricity versus $250K for a manual warehouse. Walmart's robots still can't lift big bags of dog food, so its manual path is permanent. For your agents, both argue for a narrow learned step and a staffed fallback.

Deep Dives

The cgroup v1 Nodes Your Next Kubernetes Upgrade Can't Boot

Four separate floors decide whether a node survives v1.35, and the reward for clearing them is a swap feature with a sharp cliff once working sets spill out of RAM.

Where the v1 nodes are hiding

cgroup v1 rarely survives on the nodes anyone looks at. Start with long-lived base images: Amazon Linux 2, Ubuntu 20.04, and CentOS/RHEL 7- and 8-era hosts. They shipped with v1 defaults, pre-5.8 kernels, or both. Next are custom AMIs and golden images nobody has rebuilt in two years. After those, check the pools that rarely get touched: GPU, batch and on-prem.

A node has to clear four separate floors. Each one can fail an upgrade on its own:

RequirementMinimumHow to check
cgroup modev2 unified hierarchystat -fc %T /sys/fs/cgroup/ returns cgroup2fs; v1 hosts return tmpfs
Linux kernel5.8 or laterkubectl get nodes -o wide, KERNEL-VERSION column
Container runtimecontainerd 1.4+ or CRI-O 1.20+Same command, CONTAINER-RUNTIME column
cgroup driverkubelet and runtime must matchCompare kubelet cgroupDriver with the runtime config, such as containerd's SystemdCgroup

The failCgroupV1: false override exists. Record each use as a dated exception with a named owner, with an expiry. Switching cgroup mode already needs a boot parameter change and a reboot. Rebuilding the image and rolling the pool costs about the same as patching hosts in place. The rebuild also leaves no override behind to forget.


What v2 changes on day one

  • Whole-container OOM kills. The OOM killer now kills the entire container, not one process. That ends the zombie pod, where a worker died and the supervisor kept the container looking healthy. Restart patterns in dashboards and on-call runbooks will change.
  • PSI metrics on by default. Pressure Stall Information records how long tasks stalled waiting on memory, CPU or I/O. It is a better saturation signal than utilization. It is the instrument to have in place before enabling swap.
  • In-place Pod-level resizing without recreating pods. Memory QoS with memory.min/memory.low tiering is still alpha in v1.36. Do not design around it yet.

Reading the swap numbers honestly

Node swap works only because cgroup v2 accounts for swap separately. Chris Short highlights GKE benchmarks run on NVMe Local SSDs. Next to the 3x sandbox result, they show the upside and the cliff:

  • gVisor headless Chrome went from 80 to 160 pods per node.
  • A Linux kernel build with its memory limit halved from 600 MB to 300 MB finished faster: 374s against 433s.
  • The same build at 200 MB ran more than 40% slower.

The middle result looks backwards. The source does not explain it. We think this is what happens, though we have not tested it. Without swap, the kernel can reclaim only file-backed pages. A memory-constrained build evicts the page cache it needs and rereads from disk. With swap, cold anonymous pages go out instead and the hot file cache stays resident. At 200 MB nothing cold is left, so the build pages its own working set.

Size node pools on RAM alone and treat swap as headroom for idle and bursty pages. Adding the two together is how a pool reaches the cliff.

The tests ran on local NVMe, the first of three caveats on the headline figure. Network-attached block storage will have much worse swap-in latency. The baseline configuration for the 433s run is not given, so treat the build gain as directional until it is reproduced. Anything in anonymous memory can land on the swap device. Swap encryption and local-disk lifecycle are therefore part of the design.


The smart move

The order is sweep, gate, then harvest. The sweep takes an afternoon with a DaemonSet or kubectl debug node/.... The gate is a pipeline precondition of zero v1 nodes, with one pool canaried through kubeadm preflight before the fleet. After that, prototype swap. Setup is a few lines of KubeletConfiguration with LimitedSwap, applied to Burstable pods on mostly-idle workloads such as agent sandboxes, headless browsers and CI runners. Latency-sensitive services stay on Guaranteed QoS. LimitedSwap does not apply there.

What to do

  1. Run a DaemonSet or kubectl debug node sweep across every cluster this week that reports cgroup mode (stat -fc %T /sys/fs/cgroup/ must return cgroup2fs), kernel 5.8+, containerd 1.4+ or CRI-O 1.20+, and matching cgroup drivers, including GPU, batch and on-prem pools.

  2. Add 'zero cgroup v1 nodes' as a hard precondition in your v1.35 upgrade pipeline this sprint, canary one pool through kubeadm preflight first, and rebuild failing base images instead of setting failCgroupV1: false.

  3. Prototype LimitedSwap this quarter on one local-NVMe pool of Burstable, mostly-idle pods such as agent sandboxes, headless browsers or CI runners, with PSI memory-pressure alerts wired in and rollout gated on p99 latency rather than pod count.

Picking a Deno Exit Before Deploy Goes Dark

The cheapest port depends on how much of your code uses runtime-specific APIs, and the self-hosted option on offer can't yet carry a hard deadline.

What the deal doesn't tell you yet

Techpresso gives dates but no deal terms. It does not cover Deno KV data hosted on Deploy, the JSR registry, or what the migration help offered to paying customers actually includes. Those gaps decide which exit is viable. We would get answers in writing from the vendor contact before building a plan on assumptions.

Ryan Dahl and Bert Belder will merge Deno's open-source Celld project into Cloudflare's Workerd runtime. The merged platform is meant to run on self-managed infrastructure, with a standard object storage bucket as its only coordination dependency. If that holds, it removes the consensus cluster, which is the part that makes self-hosted edge runtimes painful to operate. It is a good goal but unverified, and we'd want four answers first:

  • Which primitives it assumes, such as conditional writes, compare-and-swap or lease objects. S3-compatible stores do not all support the same set, so “standard bucket” may quietly mean S3 semantics.
  • Whether the request hot path ever touches the bucket.
  • What running isolates do when the bucket is slow or unavailable.
  • What coordination traffic costs in request volume at a given node count.

Four exits, compared

Exit pathCode changesDeadline fitOps burdenMain risk
Cloudflare WorkersModerate: swap Deno.* APIs for Workers bindingsMust be live before Deploy shuts (~April 2027)LowCoupling to Workers bindings and pricing
Self-hosted Celld/WorkerdUnknown; likely close to Workers (our inference)No release date, so it can't be your only exitMedium: your nodes plus a bucketBetting a hard deadline on unreleased software
Deno runtime on your own hostsMinimalBuys time to ~October 2027High: you own hosting, scaling, TLS and deploysUnpatched runtime once security updates end
Node.js or Bun on another hostModerate to high, by Deno-namespace usageSame Deploy cutoffVaries by hostLargest rewrite, least tied to this deal

The deciding variable is API surface. Handlers written to web standards (fetch, Request/Response, Web Crypto, Streams) are the cheapest port to Workers. Code that leans on Deno.serve, Deno.env, Deno KV, Deno.cron or deno compile costs roughly the same to port anywhere. For that code, operational fit is the better basis for choosing.


The rehearsal most teams skip

The WEF's Next-Generation Computing Framework, covered by Exponential View, names the gap this deal makes concrete: resilience planning that assumes “peacetime” access to suppliers. Most game days simulate availability failures, such as a region going down or a dependency timing out. Few simulate access failures, where a license server returns a valid 403, an IdP suspends the tenant, or a model API retires the pinned version. Access failures are cleaner and worse than outages because no timeout trips the circuit breaker. The caller gets an authoritative no.

We are running the Deploy sunset as an access-failure drill, because most failover plans have never rehearsed one.

Deno customers got roughly six months of notice. The next vendor may give less. We'd set the internal Deploy cutover for end of Q1 2027 rather than April, because six-month sunsets lose their last month to stragglers, DNS cutovers and data export. We would also stop greenfield work on a runtime with a twelve-month security horizon. The same rehearsal is worth running against dependencies that haven't announced anything yet.

What to do

  1. Inventory every Deno Deploy service and every use of Deno.serve, Deno.env, Deno KV, Deno.cron, deno compile and JSR imports this week, record a target platform for each, and freeze new greenfield Deno work.

  2. Request Deno's migration help in writing this sprint (tooling, credits, a Deno KV export path and a timeline), then set an internal cutover of end of Q1 2027 and port one web-standards handler to Workers or Node first.

  3. Schedule an access-revocation game day this quarter for your top five external dependencies (IdP, model APIs, package registries, license servers, managed control planes), injecting authoritative 401/403 and deprecation responses rather than timeouts.

AI Moved No Crypto Assumption, but Your Hardcoded Algorithms Are Exposed

Two camps disagree on whether models will break signatures, yet both point to the same fix, and it lives in your token formats and call sites rather than in an algorithm choice.

The argument both camps share

Georgetown's Justin Thaler gave the a16z crypto panel its cleanest line: AI “cannot find the nonexistent algorithm.” That protects the math only conditionally. It holds if no efficient algorithm exists. Nobody can prove that today. Dan Boneh ties the proof to P versus NP. Boneh also admits that model skill at cryptanalysis is unmeasured. Asked whether models are bad at cryptography, he says: “We don't know.” Justin Drake's “bunker mode” warning, relayed by Exponential View, sits at the other pole with no paper and no demonstrated break behind it. Neither camp claims an assumption has actually moved.

Both sources cite the same precedent. In 2022, SIKE, a NIST post-quantum candidate, fell to classical mathematics in roughly an hour on a single core. The Rainbow signature scheme fell the same year to an attack that ran for a weekend on a laptop. Neither break used a quantum computer or a model. Each took one insight against a young, structured assumption. That is why Boneh's sharpest warning matters: a model could “look at our post-quantum systems and potentially weaken them in some way, maybe break them altogether.” A post-quantum-only migration trades one bet for another.


Three attack surfaces, three answers

Attack surfaceExamplesEvidence todayDoes a PQC swap help?
Implementation bugsNonce reuse, missing point validation, non-constant-time comparisons, key material in logsMost plausible near-term threatNo: new PQC code has had less hardening
Protocol and config misuseJWT alg confusion, unpinned algorithms, key reuseKnown bug classNo: it's a configuration problem
Hardness-assumption breaksElliptic-curve discrete log, factoring, latticesSpeculative; no break citedPartially: only hybrid hedges both

We expect the first row to move first. Thaler concedes “the surface area for bugs just gets very large,” and cheaper model analysis helps attackers as much as defenders. Boneh puts the defender's edge before deployment. After ship is “the wrong time to attack a cryptosystem.” That argues for AI review before merge on crypto-touching code. A failed AI attack reads like a fuzzer reporting zero crashes. It describes what that model searched. It says nothing about what attacks exist. Re-run the review when model generations change.


Hybrid covers one failure; agility covers the next

Hybrid elliptic-curve-plus-lattice encryption already runs on some internet connections. Boneh's reasoning for it is sound: “If you break elliptic curves, it doesn't matter because it's also encrypted under a lattice-based system.” Now take the scenario he raises himself, where AI weakens the lattice half. Hybrid falls back to classical-only security and the algorithm has to be swapped. That swap is cryptographic agility, and it has its own failure mode. JWT alg-header confusion and TLS downgrade attacks show what happens when the peer picks the algorithm. Keep the allowed list server-side and set by policy.

The engineering cost is mostly bytes. ML-DSA-44 signatures run about 2.4KB. Ed25519 signatures are 64 bytes. Base64-encode one inside a JWT in a header and it collides with common ~8KB proxy header limits. Better to find that on an internal path than during a forced migration. Hybrid X25519MLKEM768 key exchange adds roughly 1.2KB to the TLS ClientHello. Current browsers, major CDNs and OpenSSL 3.5+ support it. On enable, watch for middlebox breakage and handshake p99.

One caveat on the panel: weigh its “FUD” framing against a16z's portfolio exposure to elliptic-curve-secured assets. Its technical claims still match mainstream cryptographic consensus.

What to do

  1. Inventory every signature and encryption path this sprint (JWT signing, mTLS and internal PKI, artifact signing, SSH, encrypted DB columns, long-lived tokens) and flag each one that lacks an algorithm ID or key version, or needs a code deploy to change algorithm.

  2. Prototype hybrid classical-plus-ML-DSA signatures on build-artifact signing and enable X25519MLKEM768 at one TLS edge this quarter, measuring signature size against your proxy header limits, middlebox breakage and handshake p99.

  3. Pilot AI security review before merge on crypto-touching code this quarter (nonce/IV generation, key derivation, signature verification, constant-time comparisons, key deserialization) and measure its false-positive rate before letting it block merges.

The bottom line

These forced changes share one property: none is costly in itself, and each gets costly in exact proportion to how deeply you hardcoded the thing being changed. That kills the comfortable idea that a deprecation notice or a vendor roadmap buys you time, because the time exists only if swapping was designed in. This week, rehearse one swap for each layer you'd least like to replace, whether a base image, a hosting platform or a signing algorithm. Record whether it took a config change or a rewrite, because that record is your real migration backlog.