AI & Agentic AIInterview questions and answers

A transformer is a neural network architecture introduced in 2017 that uses self-attention to model relationships between every pair of tokens in an input in parallel. It replaced recurrent networks (RNNs) and powers all modern LLMs — GPT, Claude, Gemini, Llama — because attention scales to long sequences and trains efficiently on GPUs.

Self-attention is a mechanism where each token in a sequence 'attends' to every other token via Query-Key-Value vectors. The model learns which other tokens are most relevant and weights its understanding of the current token by them. This is how transformers grasp context like 'it' referring to a noun earlier in the sentence.

An embedding is a fixed-length numeric vector that captures the semantic meaning of text, image, audio, or code. Similar items end up close in the vector space — 'king' is near 'queen', 'dog' near 'puppy'. Generated by models like OpenAI's text-embedding-3 or open-source models like bge and e5.

Tokenization splits text into smaller units (tokens) the model understands. Modern LLMs use subword tokenizers like BPE or SentencePiece — common words are one token, rare ones split into pieces. 'Hello world' might be 2 tokens; 'antidisestablishmentarianism' might be 7. This affects cost (you pay per token) and context limits.

The context window is the maximum number of tokens an LLM can attend to in one request — both input and output combined. GPT-4 has 128k, Claude has 200k-1M, Gemini up to 2M. Bigger windows allow more documents and longer conversations, but cost more, run slower, and models often pay less attention to the middle.