LLMs.txt Reader

LLMsTxtReader creates a document for an llms.txt overview, then fetches up to max_urls linked pages.

llms_txt_reader.py

from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.llms_txt_reader import LLMsTxtReader
from agno.models.openai import OpenAIResponses
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge = Knowledge(
    name="LLMs.txt Docs",
    vector_db=PgVector(table_name="llms_txt_docs", db_url=db_url),
)

agent = Agent(
    model=OpenAIResponses(id="gpt-5.2"),
    knowledge=knowledge,
    search_knowledge=True,
)

if __name__ == "__main__":
    knowledge.insert(
        url="https://docs.agno.com/llms.txt",
        reader=LLMsTxtReader(
            max_urls=10,
            allowed_hosts=["docs.agno.com"],
        ),
    )
    agent.print_response("What is Agno?", markdown=True)

Run the Agent

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

uv venv --python 3.12
.venv\Scripts\activate

Install dependencies

uv pip install -U agno beautifulsoup4 openai pgvector psycopg sqlalchemy

Export the API key

export OPENAI_API_KEY=your_openai_api_key_here

Run PgVector

docker run -d \
  -e POSTGRES_DB=ai \
  -e POSTGRES_USER=ai \
  -e POSTGRES_PASSWORD=ai \
  -e PGDATA=/var/lib/postgresql \
  -v pgvolume:/var/lib/postgresql \
  -p 5532:5432 \
  --name pgvector \
  agnohq/pgvector:18

docker run -d `
  -e POSTGRES_DB=ai `
  -e POSTGRES_USER=ai `
  -e POSTGRES_PASSWORD=ai `
  -e PGDATA=/var/lib/postgresql `
  -v pgvolume:/var/lib/postgresql `
  -p 5532:5432 `
  --name pgvector `
  agnohq/pgvector:18

Run the agent

python llms_txt_reader.py

Reader Parameters

Parameter	Type	Default	Description
`max_urls`	`int`	`20`	Maximum number of linked URLs to fetch from the file
`timeout`	`int`	`60`	HTTP timeout in seconds
`proxy`	`Optional[str]`	`None`	Optional HTTP proxy URL
`skip_optional`	`bool`	`False`	Skip entries under the `## Optional` section
`chunking_strategy`	`Optional[ChunkingStrategy]`	`FixedSizeChunking()`	Strategy for chunking content
`allowed_hosts`	`Optional[List[str]]`	`None`	Hostnames the reader is allowed to fetch from. See Restricting URL Fetches.

LLMsTxtReader also accepts the base Reader constructor parameters. allowed_hosts applies to the index, linked pages, and redirects. LLMsTxtReader.async_read() fetches the linked pages concurrently.

Next Steps

Task	Guide
Restrict outbound requests	Restricting URL Fetches
Crawl pages without an llms.txt index	Website Reader
Configure content chunking	Chunking Overview

Get Started

Basics

Advanced

Production

Providers

Other

Additional Resources

LLMs.txt Reader

Run the Agent

Reader Parameters

Next Steps

​Run the Agent

​Reader Parameters

​Next Steps

Run the Agent

Reader Parameters

Next Steps