xAI engineers got Grok running on 100,000 GPUs in just 23 minutes. This is not a show of skill—it's engineering.

The speaker, a former Berkeley PhD who now leads SGLang development at xAI, explained their approach in a video: they split prefill and decode, assign experts to different GPUs, route tokens to the corresponding experts, and overlap communication and computation. In the end, this stack brings inference costs 5 times lower than the DeepSeek API.

None of these techniques are new, but what made xAI's implementation successful is that they turned every step into a repeatable engineering loop. This same idea is broken down into a more general framework called Loop Engineering in another article shared by h100envy.

The article focuses on how to automatically tune RAG instead of tuning it manually. Manual RAG tuning is classic tedious work: adjust chunk size, run evaluation, check recall, swap embeddings, run again, add a reranker, run again. After a dozen rounds, you'll have long forgotten which result corresponds to which combination.

The solution the author proposes is a loop: define the search space, set an honest checkpoint (recall@k on the evaluation set), then let the loop search for configurations automatically until it meets the target or exhausts the budget.

The search space looks like this:

```python

SEARCH_SPACE = {

"chunk_size": [400, 600, 800, 1200],

"chunk_overlap":[0, 100, 200],

"embedding": ["text-embedding-3-small", "bge-large", "e5-large"],

"k": [5, 10, 20],

"reranker": [None, "bge-reranker", "cohere-rerank"],

"hybrid": [False, True],

}

```

There are 648 possible combinations, and brute-force full search is both expensive and foolish. So the loop uses coordinate descent: tune only one parameter at a time while keeping all others fixed, move on to the next parameter after finding the optimal value for the current one.

The key is there are two "brakes": max_evals limits the number of runs, and max_budget_usd caps the total cost. The system checks if the next round would exceed the budget before running, and stops early if it does. This is not an after-the-fact check—it's a pre-emptive stop.

Another common pitfall is overfitting to the evaluation set. The loop may find a set of parameters that happen to work well for 30 test questions but fail completely in production. The solution is to split the evaluation set into a training split and a held-out split: the loop only optimizes on the training split, and the held-out split is used for final validation. If the recall gap between the training split and held-out split exceeds 0.1, it indicates overfitting and the configuration is unreliable.

The article also emphasizes that the improvement threshold should be set to 0.02, otherwise the loop will chase random noise.

In the end, what you get is not just a tuned RAG system, but a reproducible process: a jsonl log records every step, the held-out check tells you if the result is reliable, and when your data or model changes next time, you can rerun the whole process with a single command.

This is what Loop Engineering is: delegate repetitive, measurable, clearly-defined tasks to loops, and humans are only responsible for defining the search space and setting the brakes. What xAI does on the 100K GPU cluster essentially follows the same idea—they orchestrate operations like prefill/decode splitting, expert routing, and communication overlap into a loop, then let it run automatically on the cluster.

Stop tuning parameters manually. Write a loop and let it do the search for you.