Build a web-research agent with n8n and Crawl4AI

Define the research question and the request

Luke’s February 2025 lesson builds a low-code agent that searches for information and responds to questions using a scraper and vector storage. Its early chapters explain HTTP requests, tokens and memory before the n8n workflow. These concepts matter because fetching a page and retrieving previously stored text are different operations.

1. Open the retained n8n resource, https://n8n.partnerlinks.io/trxvmi48ria4, and follow the workflow and chat-model setup at 13:53. Use one permitted public page and a question you can answer from its content.
2. Write the source URL, relevant text and expected answer into the test record. Keep the request and returned page content visible before involving an agent or embedding model.

Test the scraper before adding stored memory

The tool section starts at 23:23 and the Python/Crawl4AI scraper at 24:40. A page request may return incomplete text or an error even when the workflow continues. Check what the agent receives rather than assuming a successful HTTP response contains the useful article.

1. Follow the demonstrated scraper configuration and run it on the known page. Compare the extracted title, body and source reference with the actual content, noting missing or unrelated sections.
2. Connect the scraper as a tool and ask the known question. Inspect the tool output and final answer separately, then test a page or query that produces no useful result.

Inspect the vector-storage handoff

The video covers vector-database integration at 34:48 and text splitting and embeddings at 40:59. Storage should make source material retrievable, not erase where it came from or confuse a previous page with a fresh fetch. Keep the question’s evidence traceable through the chunks and returned results.

1. Follow the database and embedding setup shown in the lesson. Inspect a small stored sample, keeping the original URL or document identity associated with the text used for retrieval.
2. Run a query with a known relevant passage and inspect the retrieved material before judging the answer. Distinguish retrieval of the wrong passage from a model misreading the right one.

Debug a complete question-to-answer run

Troubleshooting begins at 46:56 and the final demonstration at 57:44. The source supplies no full template URL, Python payload or embedding configuration in its description, so the original recording remains the reference for those exact implementation details.

1. Run the small test from question to scrape, storage or retrieval and final response. Compare each boundary with the expected evidence before changing unrelated nodes.
2. Record the working configuration and any unanswered cases. Retest with another small source and verify that the agent does not answer from stale text or claim knowledge absent from the retrieved material.

The takeaway

A useful research agent exposes the source behind its answer. Check fetching, extraction, storage, retrieval and interpretation independently before extending the workload.