Rasoul Kaljahi

Examples of the kind of work I do

I am starting my independent practice, so I do not have client case studies to show yet.

The four stories below are invented. They are not real clients, and the numbers are made up to show scale. They describe how I would approach each kind of problem. My real, past work is described on the home page.

1. A support chatbot that changes every week

Situation. A small software company has an LLM support chatbot. A developer edits the prompt almost weekly. Each change feels like an improvement, but two customers recently got wrong refund information, and nobody can say whether recent changes caused it.

What I would do. Pull a few hundred past tickets and turn them into a test set, including refund and billing cases that carry real cost. Write a short rating guide, have two people score a sample, and set up an LLM judge. Then check how often the judge agrees with the people.

What they would have. A test they can run in minutes before each prompt change. The first run might show that the newest prompt improved tone but lowered accuracy on billing questions. They would see this before release, not from a complaint.

2. An assistant launching in ten languages

Situation. A company is about to launch its AI assistant in ten languages. It has tested English thoroughly and the others only by spot checks from colleagues.

What I would do. Build a test set for each language from the company’s real content. Use native-speaking reviewers, with a shared guide so their scores can be compared. Classify errors as wrong meaning, missing information, terminology or style.

What they would have. A per-language report. It might show that seven languages are fine, two have a terminology problem that a glossary fixes, and one produces meaning errors often enough that it should launch later or with human review.

3. Keep the translation vendor or switch to an LLM?

Situation. A localization team pays a machine translation vendor and is under pressure to try an LLM instead. Vendor demos and internal opinions disagree.

What I would do. Take a sample of the team’s own content across its main content types, such as product UI, help articles and legal text. Translate it with both systems and compare them with human review and automatic measures, split by language and content type.

What they would have. A decision with evidence. A typical result is mixed: the LLM is better on help articles in some languages, the vendor is better on short UI strings and legal text. The recommendation then says which to use where, and what a switch would save.

4. Reviewing every document by hand

Situation. A company generates product descriptions with an LLM, and an editor reads every one before publishing. The queue grows with the catalog and review is now the main cost.

What I would do. Use the editors’ past accept and reject decisions as labels. Build a quality estimator and test it on descriptions it has not seen. Pick a threshold, and measure how many bad descriptions would pass without a human check.

What they would have. A system that sends the risky share to editors and lets the rest through, with a known miss rate. The company can then choose how much risk it accepts. A stricter threshold means less saving and fewer errors.

Have a problem like one of these?

Tell me about it and I will say how I would approach it.

Email me