What does an AI engineer actually build?

What does it take to build an AI assistant for a particular subject? You can try one here: this site's assistant answers questions about my work and experience.

A suggested question appears as you reach the assistant, and you can change it before sending. It searches this portfolio's knowledge base, asks for consent before storing the conversation, and doesn't display source links in the chat.

To explain how an application like this works, let's follow a hypothetical library building an assistant for its readers. We'll look at the information it needs, the actions it can take, and how its builders check the answers.

1. Start with a question

Imagine a reader asks, “Can I renew a book if someone else has reserved it?” A language model can produce a convincing answer, but it may not know this library's rules.

The assistant could search the library's borrowing guide and give the relevant passage to the model alongside the question. The model can then use that passage to answer.

This approach is called retrieval-augmented generation, or RAG: find useful information first, then include it in the request to the model. Martin Fowler and Bharani Subramaniam explain how retrieval supplies information beyond a model's training.

2. Prepare the guide for search

Start with the current borrowing guide and ask a librarian to confirm it is correct. Split it into passages that keep each rule and its exceptions together, and save each passage with its source and version.

Make the passages searchable, then try questions readers might ask. A search for “Can I renew a reserved book?” should find the renewal rule and its exception before a model is asked to answer.

3. Give the model context

Suppose the guide says books can be renewed twice, except when another reader has reserved them. With that passage and the question, the model can explain the exception and the application can show where the rule came from.

The team can instruct the model to answer from the guide and refer readers to a librarian when information is missing. An old guide or a poorly matched passage can still produce a wrong answer, so the search and the response both need checking.

The basic flow looks like this:

question = reader's question
passages = search the current library guide
 
if no useful passage is found:
    offer help from a librarian
else:
    ask the model to answer using the question and passages
    show the answer and its sources

This is a sketch, not runnable code. The team still has to decide how to identify useful passages and check whether an answer is supported by them.

4. Decide whether it can take action

Answering a question is different from changing a library account. To check a due date or renew a loan, the assistant would need a connection to the library system and a way to verify the reader's identity.

The team must decide which actions are allowed and make sure a failed renewal isn't reported as a success. Anthropic's guide to building agents recommends starting simply and adding complexity when it improves the result; for this library, answering questions may be enough for a first version.

5. Check the answers

I'd make a set of questions the library expects, including a reserved book and a situation the guide doesn't cover. For each question, the team can record the expected answer and the passage that supports it.

Run those questions through the assistant and review the results with a librarian. Repeat the checks when the guide, search, instructions, or model changes, and add failures discovered after launch.

These checks are called evaluations, or evals; Anthropic's guide describes combining code and human judgment to assess agent behavior. Recording which passage was retrieved can also help the team trace a wrong answer, without retaining unnecessary reader information.

6. Keep watching after launch

The assistant could answer hundreds of questions and still leave readers confused. I'd want the library to check whether answers are correct, how often readers need a librarian, and whether those handoffs work.

Amazon's recent engineering leadership update makes a related point about developers using AI to build software: customer outcomes matter, while targets based on token usage or code changes can encourage waste. Amazon is discussing a different use of AI, but the question carries across: did the technology help someone get the job done?

Where does the engineer fit?

An AI engineer may build applications around existing models, adapt models, or work on the systems that run and evaluate them; the boundaries between job titles vary by team. The library example focuses on the application: finding information, protecting accounts, checking answers, and deciding when a person needs to help.

When a reader asks about a reserved book, the work has paid off if the assistant can explain the current rule, show where it came from, and be honest when it needs help. That's the kind of result I'd want to measure.