Man! the world moves fast!

I always wondered how much time I should spend learning agent frameworks when the landscape is changing so fast.

Today, I can ask a model like GPT-5.6-Sol to “write a checkpointed orchestrator with sub-agents for this task,” point it at a repository, and get a working starting point. Dependencies, worker assignments, recovery instructions, the scripts to check whether the whole thing makes sense. It can write those too.

I recently tried this on a project with a long release checklist. The app itself had been vibe-coded. Then the release work arrived, and suddenly I was supposed to become a very diligent project manager.

I had no mood of going through that sh*t manually.

So I asked GPT-5.6-Sol to build the orchestrator for me.

Case in point

I’ll use a fictional packing-list app, Packlight, throughout this post. It lets you create a list for a trip, tick things off, and use it offline. The app details, filenames, task IDs, and prompts below are adapted examples. The orchestration decisions and the problems I ran into come from the actual sessions.

The starting request was roughly: “Turn this small HTML and JavaScript app into an Android app.”

That sounds manageable. Wrap the web app, build it, install it on a phone. Done?

Then came the follow-up work. Check whether a saved list survives an update. Try launching with no network. Test the keyboard on a small screen. Fix a browser test that sometimes passes and sometimes fails. ReGPT-5.6-Solve a dependency conflict in the Android tests. Prepare signing, icons, screenshots, privacy information, and a store listing.

The checklist ran past a hundred lines of Markdown. Each line was reasonable. Collectively, it was a lot of boring work between “this runs on my phone” and “I can release this.”

Some of it could happen together. Someone investigating a browser test did not need to wait for someone reading a build dependency report. Other work had to wait: screenshots needed the final app, and testing an update needed an installable build. A few steps needed decisions from me.

I wanted an agent to keep track of all of that, assign the work, check the results, and carry on. And when the session ended, I wanted to continue without explaining the whole project again.

At this point I was thinking of sub-agents, but was too lazy to craft the setup myself.

So I asked GPT-5.6-Sol to do it

Here is an adapted version of the planning prompt:

Look into docs/exec-plans/pending/action-items.md and create an
agentic orchestrator implementation spec to execute the remaining
work one by one, or in parallel where possible.

Context:
Read the other pending plans, AGENTS.md, ARCHITECTURE.md, and the
existing test reports. Use them to understand what already works
and what remains.

Task:
Write a spec that I can give to an orchestrator agent. It should
break the work into scoped assignments, use sub-agents, review
their results, and continue through the checklist.

The orchestrator will run on GPT-5.6-Sol with xhigh reasoning and a 1M
context window. It may choose GPT-5.6-Sol, Terra, or Luna for workers.
Prefer a cheaper model for simpler work. Define when to escalate.

You can use the Codex CLI for persistent worker sessions and
configure a smaller context window for each worker.

IMPORTANT: Checkpointing
Keep checkpointing the state so a fresh session can resume from
the last checkpoint. Use local Git commits. Create as many as
needed; I can clean up the history later. Preserve useful helper
scripts and evidence. Keep credentials and private data out of Git.

Resources:
A USB-connected Android phone will be available. Inventory the
host, emulator support, and required software during planning.
Tell me about manual setup, firmware changes, or missing tools
before I start the orchestrator.

Save the implementation spec under docs/exec-plans/pending/.
Prepare the framework now; release execution starts separately.

The repository already had useful context: an AGENTS.md, architecture notes, pending plans, and test reports. Those files gave the model something concrete to work from.

I only had to make two refinements over what GPT-5.6-Sol did on its own.

• First: migrate the Markdown action items to JSON once, validate that nothing was lost, and use that JSON as the authoritative tracker.
• Second: Suggested to use ordinary git add, git commit, and git worktree commands. The first version asked for direct access to the repository’s .git directory. I asked it to remove that requirement. Git already manages its own metadata; any necessary tool permission should be scoped to the command being run.

That was enough to get the first version. I reviewed what it generated and corrected things as they surfaced. I did not hand-write the scheduler, tracker, worker contract, or recovery protocol.

What GPT-5.6-Sol generated

The resulting setup was a set of repository files and a few validation scripts. Codex read the spec and did the coordination. There was no separate scheduling service to deploy.

With the example names, the main pieces looked like this:

Only the orchestrator could update the tracker. Workers returned results for review. That avoided three agents independently deciding what the shared task list should say.

GPT-5.6-Sol worked out the full task dependency graph

Each task had explicit dependencies. For the packing-list example, part of the task graph could look like this:

• Selecting work: The orchestrator identifies tasks whose dependencies are satisfied and assigns up to three workers at a time. It prevents conflicts over shared files or resources and integrates results in rank order.
• Assigning a task: Each worker receives one task, its completion criteria, and a model suited to the work. Workers make changes in separate checkouts and can resume interrupted sessions. Difficult tasks can escalate to a more capable model.
• Reviewing results: Workers return their changes, test results, supporting evidence, blockers, and recommended next action. The orchestrator reviews the changes and reruns relevant tests before accepting the work and updating the tracker. It preserves the evidence needed to continue in a later session.

Checkpointing saves progress for the next session

Checkpointing uses local git commits to save the integrated changes, supporting evidence, and updated task tracker together. Each checkpoint records the work completed, test or diagnostic results, remaining blockers, and next actions so a fresh session can pick up from the saved progress.

The orchestrator creates a checkpoint after reviewing, retesting, and integrating each task. It also commits useful diagnostic findings while a task is still unfinished—for example, a reproduced build failure and the evidence needed to continue investigating it. Saving that progress does not mark the task complete.

Checkpoints also capture the reviewed baseline before implementation begins, activation of the release plan, and the final release archive when the program finishes. Before a planned handoff to a fresh session, the orchestrator finishes the current work and checkpoints its results. These commits stay local; pushing them is a separate action.

Using the orchestrator

To start, point the agent at the orchestrator spec:

Start the orchestrator in docs/exec-plans/pending/release-orchestrator.md.

To pause for a fresh session:

Finish the in-progress work, checkpoint, and pause.

Then, in a new session:

Resume the orchestrator in docs/exec-plans/active/release-orchestrator.md.

The spec tells the orchestrator how to find and validate the saved state and choose the next task.

In action

I’m using the orchestrator to work through a release checklist across multiple sessions. It assigns eligible tasks to workers, reviews and tests their results, integrates completed work, and checkpoints progress. When work is interrupted, it uses the saved state to continue. I review decisions and provide access or approval where needed; independent tasks can proceed while others are blocked.

The agent also resolved problems in the orchestration itself: incompatible output schemas, incorrect worker directories, brittle tracker tests, and a misleading checkpoint error caused by sandbox restrictions. It recovered unfinished work after a worker hit a usage limit, verified the results, and checkpointed them.

So far, this has produced integrated fixes, repeatable test evidence, and successful recovery across sessions. The release is still in progress.

What I took away

I skipped the deep LangChain agent stack. Understanding the concepts still mattered.

I needed to recognize why the tracker should have one writer, why a worker’s result needed review, and why a saved session could disagree with the files on disk. Those decisions shaped the prompt and helped me review what came back. The model wrote the machinery around them.

I expect to spend less time reviewing the routine parts as the models improve. In this run, the review caught things worth catching. I would be quite happy to stop finding working-directory mistakes in generated worker commands.

For the next long checklist, I will probably start the same way: give the model the repository context, ask it to design the execution and recovery rules, review those rules, and let it begin.

Llama2 is a popular open-source model from Facebook. In this article let’s go over how one can run Llama2 on a MacBook in CPU mode and create a simple AI chatbot like ChatGPT.

First, go over the steps given in Build and run Llama2 LLM locally.

The above article gives a detailed step-by-step guide for the following:

• Make sure you have the necessary prerequisites: Python 3, Git, etc.
• Download the Llama repository: git clone https://github.com/facebookresearch/llama.git
• Download the llama.cpp repository: git clone https://github.com/ggerganov/llama.cpp.git
• Download Llama2 model weights by filling in the request form at Meta’s Llama downloads page.
• Build llama.cpp. This generates an executable that can be used to interact with the downloaded models.
• Convert the downloaded models to f16 format and then quantize them to reduce their size. With quantization, the downloaded Llama2 13B chat model of 25 GB becomes about 7 GB.
• Run the sample chat-with-bob.txt example prompt.

The only thing in addition to the steps in the linked article is that the latest version of the code when you clone the llama.cpp repository may not work as is. To make it work, I had to check out a specific older commit (a113689) before making and building the llama.cpp repository.

Here are the steps and commands for easy reference.

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.CPP/
python3 -m venv llama2
source llama2/bin/activate

git checkout a113689
make
python3 -m pip install -r requirements.txt
mkdir -p models/13B/
# Convert the downloaded models to f16 format
python3 convert.py --outfile models/13B/ggml-model-f16.2.bin --outtype f16 ../llama/llama-2-13b-chat --vocab-dir ../llama
# Quantize them to reduce size.
./quantize ./models/13B/ggml-model-f16.bin ./models/13B/ggml-model-q4_0.2.bin q4_0

For details, please refer to the referenced article.

At this point, you should have a compiled version of the main executable in the llama.cpp app directory and a quantized version of the model, such as models/13B/ggml-model-q4_0.bin.

Let’s run Llama2.

In this article, I am using Llama2 13B chat, but feel free to use other models, such as 7B, if you find the performance too slow with 13B.

Our first prompt

[jd@15:16:26 llama.cpp]$ ./main --model ./models/13B/ggml-model-q4_0.bin --prompt "How to make mayo?" -i

main: build = 929 (a113689)
main: seed  = 1705226162

[...]

generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0

== Running in interactive mode. ==

 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMA.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\\'.

 How to make mayo?

To make mayonnaise, you will need:

* 2 egg yolks (save the whites for another use)
* 1/2 cup (120 ml) neutral-tasting oil, such as canola or grapeseed
* 1 tablespoon lemon juice or vinegar
* 1/2 teaspoon Dijon mustard (optional)
* Salt and pepper to taste

[...]

(A long and fairly detailed recipe follows.)

So far so good.

Let’s create a chatbot

Next, let’s explore the llama.cpp input parameters and see how we can use them to make the chatbot better.

I used the llama.cpp README as my reference.

--reverse-prompt, --in-prefix, and --in-suffix

Together, these options allow you to create a more chat-like experience. Even more, you can include a few examples in your prompt to better control the output format.

See the example below:

$ ./main --model ./models/13B/ggml-model-q4_0.bin --in-prefix " " --in-suffix "Assistant:" -r "User:" -i -p "User: Hi\"

Assistant: Hi! I am an AI assistant. Ask me anything you want and I will try to answer to the best of my ability

User: Tell me about the wall of China. Use at most 20 words.

Assistant:

[...]

User: Hi
Assistant: Hi! I am an AI assistant. Ask me anything you want and I will try to answer to the best of my ability

User: Tell me about the wall of China. Use at most 20 words.
Assistant: Great Wall of China stretches over 4,000 miles across China's mountains and valleys.

User: How old is it?
Assistant: The Great Wall of China was built over 2,500 years ago.

User: Is it older than the pyramids in Egypt?
Assistant: Yes, the Great Wall of China is older than the pyramids in Egypt.

User: When were the pyramids built?
Assistant: The pyramids in Egypt were built around 4,500 years ago.

User: Now tell me which one is older?
Assistant: The Great Wall of China is older than the pyramids in Egypt. It was built over 2,500 years ago, while the pyramids were built around 4,500 years ago.

As you can see, it is not very good at logical deduction or basic maths (not sure if the 70B model is better). But it can spit out facts pretty well. So there is that.

Here we found out how we can give the AI a few examples to tune the answers, similar to the methodology of few-shot inference.

$ ./main --model ./models/13B/ggml-model-q4_0.bin --in-prefix " " --in-suffix "Assistant:" -r "User:" -i -p "User: Cow\

Assistant: A cow is a farm animal.
User: Dog
Assistant: A dog is a faithful animal.
User: Chair
Assistant: "

[...]

User: Cow
Assistant: A cow is a farm animal.

User: Dog
Assistant: A dog is a faithful animal.

User: Chair
Assistant: A chair is a piece of furniture for sitting.

User: Pi
Assistant: Pi is a mathematical constant equal to the ratio of a circle's circumference to its diameter.

Please note that this is just an example and it is not recommended to use these responses as actual answers to questions. The goal of this exercise was to demonstrate the different types of responses that a language model assistant can provide based on the input given.

User: Pig
Assistant: A pig is a farm animal.

Please note that this is just an example and it is not recommended to use these responses as actual answers to questions. The goal of this exercise was to demonstrate the different types of responses that a language model assistant can provide based on the input given.

Text summarization

While the performance of Llama2 may not be satisfactory for logical inference or math, it is pretty good at text processing jobs; for example, summarization. Before we go, let’s see how we can use Llama2 for text summarization. Here let’s try to summarize the first section of the Wikipedia article on Earth. Let’s assume we have copied and kept it in a file earth.txt. Before summarization, it has 584 words.

$ wc -w earth.txt
     584 earth.txt

We can try to summarize this using Llama2 following a prompt structure given in Llama2 and text summarization.

Write a concise summary of the text, and return your responses with 5 lines that cover the key points of the text.

   ```{text}```
   SUMMARY:

But wait. Before we proceed, we need to increase the Llama2 default context window size. Otherwise it would fail with this error:

./main --model ./models/13B/ggml-model-q4_0.bin --prompt "\
Write a concise summary of the text, and return your responses with 5 lines that cover the key points of the text.

`cat ./earth.txt`

SUMMARY:
"

main: error: prompt is too long (914 tokens, max 508)

We can increase the context window using the --ctx_size option. Here is an example. Note that it is a one-shot summarization task, so we do not need the -i option.

$ wc -w earth.txt
     584 earth.txt

$ ./main --model ./models/13B/ggml-model-q4_0.bin --ctx_size 1024 --prompt "\
Write a concise summary of the text, and return your responses with 5 lines that cover the key points of the text.

`cat ./earth.txt`

SUMMARY:
"

[...]

SUMMARY:

Earth is a water world, with 71% of its surface covered in water and the remaining 29% being land. Earth's crust consists of slowly moving tectonic plates which create mountains, volcanoes, and earthquakes. The atmosphere sustains life by capturing energy from the Sun, creating a dynamic climate system with different weather phenomena. Earth is the densest planet in the Solar System and orbits the Sun at a distance of about 8 light-minutes. Humanity's impact on the planet has been unsustainable, threatening its livelihood and causing widespread extinctions. [end of text]

One thing to note is that it took about five minutes on my Mac to finish this summarization. So for production use cases, you either need a GPU or choose a hosted alternative, such as Amazon Bedrock’s Llama2 support or Azure AI’s Llama2 support. For summarizing larger documents that are much bigger than the context window, we can take a recursive approach where the input is split into chunks and the chunks are first summarized. Then the summaries generated in the previous step are stitched together and further summarized using the same LLM. This recursive, multi-pass approach can summarize text of any length. LangChain’s map-reduce can also be used.

Conclusion

In this article, we learned how to run Facebook’s Llama2 model on a MacBook using CPU mode. We also used Llama2 to create a chatbot and for a text summarization task.

Mirrored from: https://techgargle.blogspot.com/2024/01/in-this-post-lets-add-step-by-step.html

• GitHub gists: https://gist.github.com/JDatta
• Tech blog at blogspot: http://techgargle.blogspot.com/

Apart from these, my older blogs can be found here:

• Passing Thoughts: http://joydipdatta.blogspot.com

As memory management is done by jvm, identifying and debugging memory leaks in java can be tricky. In case of permgen leaks, more so. Reasons for this include:

• Permgen leaks are rare
• More often than not; out of memory errors due to permgen space is not a leak but a legitimate case of memory crunch. In which case you either need to allocate more memory to permgen using -XX:MaxPermSize jvm parameter or strip down your libraries and dependencies.
• permgen is not included in standard jvm heap dump generated using jmap -dump:file=file.hprof <pid> or with option -XX:+HeapDumpOnOutOfMemoryError. They are also typically not included in profiler outputs. You often need to take inferences indirectly.

What is a permgen leak

Permgen stands for permanent generation. It is the part of the jvm memory map which is used by java to store classes (i.e. codes; executable instructions), interned strings etc. Conceptually this is similar to the code area of a “C” process.

The default permgen size in jvm is rather small, the max permgen size is only 64 MB for 32bit jvm, 83.2 MB for 64bit jvm ^{[see here]}. When this space get full, you see exceptions like: Root Cause Exception: java.lang.OutOfMemoryError: PermGen space. If your dependency library size is more than this size, then you have a genuine issue and not a leak and you should increase the permgen size using jvm parameters ` -XX:PermSize and -XX:MaxPermSize. For example, this could be the java command to start your application: java -cp lib/* MyClass -XX:PermSize=128m -XX:MaxPermSize=256m`

This permgen area is also garbage collected by jvm. Problems arise when there are items in the permgen area, which are no longer in use but somehow there are still references to them. In which case this is a case of leak.

Case Study

Stale references to permgen objects are more complicated than references in java heap space. Consider the following case.

Suppose you are using a custom classloader (See: URLClassLoader) then all the classes loaded by this classloader would occupy a fresh area in the permgen space. You can create a plethora of java objects using this loader. As long as we hold a reference to even one of these objects the loader won’t be garbage collected. And till the loader is garbage collected all the classes loaded by it would occupy space in permgen area. For example, the culprit object could be a daemon thread started by one of your own code or some code in the dependency libraries that you call. Another example would be a Runnable object registered in application hooks like the JVMShutDownHook. Other cases include static variables and thread local variables.

This is confusing because the space occupied by the culprit java object in heap could be small, say in few bytes. The space occupied by the loader object in the heap would also be typically small, again within a few bytes. So the heap dump analysis won’t tell you anything suspicious. But the classes loaded by the loader still occupying permgen area could easily be in mega bytes. In general, if you see multiple custom loader instances alive in heap dump is a red flag and require more investigation.

Debugging steps

Generate and analyze permgen dump

Before analyzing heap dump, for permgen issues you should try to get the permgen dump. If the process for which you are getting permgen error is still live, run this command to extract the permgen dump and save it to a file

jmap -permstat <pid> > /tmp/permgen_dump

This is a simple file with the following format:

$ head /tmp/permgen_dump
class_loader    classes bytes   parent_loader   alive?  type

<bootstrap>     1989    11034488          null          live    <internal>
0x0000000691e83e40      1       3056    0x000000069000cda8      dead    sun/reflect/DelegatingClassLoader@0x000000068004f888
0x0000000691f93938      0       0       0x000000069000cda8      live    com/company/CustomClassLoader@0x00000006839f3198
0x00000006905393e8      1       3056    0x000000069000cda8      dead    sun/reflect/DelegatingClassLoader@0x000000068004f888
0x0000000691a42bb0      1       3040      null          dead    sun/reflect/DelegatingClassLoader@0x000000068004f888
[...]

First we would sum the bytes occupied by all alive loaders.

$ grep live /tmp/permgen_dump|tail -n +3 |awk '{s+=$3} END{print s}'
283469897

This must be more than the permgen size limit.

From this file we would prepare a histogram to find out loaders occupying most of the spaces. Following steps show how I created the histogram from above file. Skip ahead if you want as the scripts are pretty basic.

## Print the type information (last column i.e. col 6)
## and bytes column and save to a file
$ cat /tmp/permgen_dump|tail -n +3 |awk '{print $6" "$3}' > /tmp/hist1.txt

## Invoke a simple python script to print the histogram largest to smallest
$ .hist.py /tmp/hist1.txt
225629976       com/company/CustomClassLoader@0x00000006839f3198
57763728        sun/misc/Launcher$AppClassLoader@0x000000068020c380
11034488        <internal>
859720  org/apache/derby/impl/services/reflect/ReflectLoaderJava2@0x0000000681cf52f8
719832  sun/reflect/DelegatingClassLoader@0x000000068004f888
75904   sun/misc/Launcher$ExtClassLoader@0x00000006801a0998
289     N/A
0       org/apache/hadoop/hbase/util/DynamicClassLoader@0x000000068bb4b768
0       java/util/ResourceBundle$RBClassLoader@0x0000000680562600
[...]

The python script is pretty straight forward. It just sums the bytes against same type and prints the result in descending order. The script is added in appendix as reference.

Now, from the above histogram, we can clearly see unusually large amount of permgen space is occupied by a single loader: com/company/CustomClassLoader@0x00000006839f3198

At this step, you can check your code to see how this custom class loader is being used and find out obvious errors like object references kept in static variables in some of your classes etc. If not found you need to analyze heap dump.

How to generate heap dump

You can generate heap dump using following command

jmap -dump:file=/tmp/dump.hprof <pid>

Alternatively you can add the following java options in your startup command to automatically create heap dump whenever there is an OutOfMemoryError

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/dump/dir

Once you have the dump file, you should use a memory analyzer tool to analyze. One such tool is eclipse MAT Memory Analyzer. This comes both as a plugin of eclipse and a stand alone program.

Analyze the heap dump in Eclipse MAT

Open the hprof file generated above in eclipse. Once it is loaded, click on Dominator Tree option under the head Action Items in Overview page.

In the dominator tree page look for the loader class name which we noted to take unusually large amount of permgen space in permgen dump. For our case this is com/company/CustomClassLoader. Right click on any one of the instances of CustomClassLoader in the dominator tree and click Path to GC Roots > exclude weak references. This would give you a list of instances that is holding the reference of the loader and preventing it from getting garbage collected. Expand each of them individually and check the corresponding code. For example, when we checked the code we found one daemon thread was being started by one of our dependency API preventing the loader to be garbage collected.

Analyze the heap dump using jhat

jhat is another heap dump analyzing tool

Run the command to start jhat server

$ jhat -port 9091 /tmp/dump.hprof

Now open http://localhost:9091 on any browser. Scroll down to the end of the page and click Show heap histogram. In the page that opens, search for the suspicious class we found in permgen histogram (CustomClassLoader for our case). Follow the hyper link and come to the page describing class com.company.CustomClassLoader. At the end of the page there would be a section with name Other Queries. Under this section there is a hyper link Reference Chains from Rootset > Exclude weak refs. Click on it. In the page that opens, there would be comprehensive list of codes that is holding the loader reference and preventing it from getting garbage collected. This page corresponds to the Path to GC Root page of eclipse MAT.

Remember, the loader would not be garbage collected until all the classes/objects loaded by it directly or transitively are not garbage collected. From the path to GC root page you need to carefully analyze the list of object references which was directly or indirectly loaded by our custom loader and not garbage collected.

Appendix:

Python code to generate histogram

#!/usr/bin/python
import sys
import operator

# The histogram file path should be given in argument
# The file should have two columns, key and value 
# Each line in the histogram file should contain the key
# then a space and then the value
# The following code would sum the values against the same 
# key and print the result in descending order

f = open(sys.argv[1]) # Open the histogram file 

# Create histogram map
hist = {}
for line in f:
    line = line.strip()
    if line == "":
        continue
    fields = line.split()
    mtype = fields[0].strip()
    mval = fields[1].strip()

    if mtype in hist:
        hist[mtype] = hist[mtype] + int(mval)
    else:
        hist[mtype] = int(mval)

# Sort in descending order
sorted_hist = reversed(sorted(hist.items(), key=operator.itemgetter(1)))

# print
for i in sorted_hist:
    print i[1], "\t", i[0]

Mirrored from: https://gist.github.com/JDatta/7f82db25ccef7772a0e73fe9ceb329d7, http://techgargle.blogspot.com/2016/11/java-debugging-permgen-leak.html