Editing Portal:Artificial Intelligence (section)

=== Retrieval, embeddings, and AI data pipelines ===

Many modern AI applications use retrieval-augmented generation, or RAG. In a RAG system, documents are split into chunks, embedded into vectors, stored in a search system or vector database, retrieved by similarity, and then supplied to an LLM as context.

Perl is well suited to the non-glamorous but essential parts of this workflow:

* crawling directories and websites;
* extracting text from files;
* cleaning OCR output;
* normalizing encodings;
* splitting documents into chunks;
* storing metadata in SQLite, PostgreSQL, MySQL, or MariaDB;
* calling embedding APIs;
* loading vectors into a vector store;
* generating citations or source links;
* producing audit logs;
* testing and validating outputs.

Perl does not need to be the model to be valuable in AI. In many systems, the model is only one component. The surrounding data pipeline, retrieval layer, monitoring, and integration code are equally important.