A Zero-Cost Intelligence System: Daily AI Market Research with RSS, Hacker News and Jina Reader
Table of Contents
- How I used to do market research
- Design principle: Pre-Compute Everything
- Three free data sources
- 1. RSS Feeds (industry blogs)
- 2. Hacker News API (what the tech community is watching)
- 3. Jina Reader API (web page to Markdown)
- The complete Research Chain
- How the agents use this intelligence
- Competitor analysis: another zero-cost pipeline
- Cost calculation
- Three things I learned
- 1. Separate intelligence collection from analysis
- 2. Sharing beats duplication
- 3. Data-driven beats instruction-driven
- Build it yourself
- Next post
How I used to do market research
Open Hacker News, scroll. Open Twitter, scroll. Open a few RSS readers, scroll. Bookmark anything interesting, then never look at it again.
Every day I spent 30-60 minutes "keeping a feel for the industry". Sometimes it paid off. Most of the time it just drained my attention.
Worse, that "feel" never got written down. What I read, what the trends were, how they related to my products: all of it lived in my head, and most of it evaporated after a night's sleep.
Then I thought: aren't agents good at reading large amounts of text? Why not let one do this?
Design principle: Pre-Compute Everything
The core architectural principle of OpenClaw is "pre-compute everything".
The traditional approach: when an agent needs to post, it searches for material, analyses it and writes the post in real time. That is expensive. Every post takes several LLM calls.
My approach separates "collecting" from "thinking":
Collect (0 LLM tokens) → Analyse (1 LLM call) → Produce (1 LLM call)
RSS / HN / Jina Reader organise into markdown agent reads it, then writes
The collection stage uses no AI at all. Plain HTTP requests fetch all the raw data. AI only comes in at the end, for "understanding" and "producing".
That brings the cost of the whole intelligence pipeline down to almost zero.
Three free data sources
1. RSS Feeds (industry blogs)
The most old-school data source, and the most reliable. I subscribe to about 20 AI-related RSS feeds:
# feeds.txt
https://openai.com/blog/rss.xml
https://blog.google/technology/ai/rss/
https://anthropic.com/research/rss.xml
https://huggingface.co/blog/feed.xml
# ... and so on
A simple RSS parser is enough to get the latest article titles and links. Zero cost, zero tokens.
2. Hacker News API (what the tech community is watching)
HN has a public Firebase API that needs no authentication:
# Get the IDs of the current top stories
top_ids=$(curl -s "https://hacker-news.firebaseio.com/v0/topstories.json")
# Get each story's title and score
for id in $(echo "$top_ids" | jq '.[0:20][]'); do
item=$(curl -s "https://hacker-news.firebaseio.com/v0/item/$id.json")
title=$(echo "$item" | jq -r '.title')
score=$(echo "$item" | jq -r '.score')
url=$(echo "$item" | jq -r '.url // empty')
echo "$score | $title | $url"
done
HN's top stories are the most honest signal of where the tech community's attention is. If a topic has 200+ points on HN, that is what people are paying attention to right now.
3. Jina Reader API (web page to Markdown)
This is the most critical link in the whole pipeline. Jina Reader turns any web page into clean markdown:
# Convert a web page to markdown, for free
content=$(curl -s "https://r.jina.ai/$url")
No API key needed (there is a free quota). It strips out ads, navigation bars and sidebars automatically and keeps only the body text.
This means the agent can "read" any article. Not the headline, the full text.
The complete Research Chain
Chaining the three sources together gives us research-chain.sh:
Twice a day (05:00, 17:00 CST)
│
├─ 1. Scan RSS feeds → get the list of new articles (0 tokens)
├─ 2. Scan HN top stories → get the hot discussions (0 tokens)
├─ 3. Pick the 5 most relevant → Jina Reader converts them to markdown (0 tokens)
├─ 4. Send to Gemini → analyse trends, extract key points (1 LLM call)
└─ 5. Output → RESEARCH-NOTES.md
The prompt for step 4 looks roughly like this (kept in the original Chinese, since it is the literal text sent to Gemini):
你是 AI 行業分析師。以下是今天的新聞和討論：
[RSS 摘要]
[HN 熱門文章摘要]
請分析：
1. 今天最重要的 3 個趨勢
2. 跟我們產品（AI 安全、社群自動化）最相關的是什麼
3. 有什麼值得寫成文章的角度
輸出格式：markdown，簡潔，有觀點
The resulting RESEARCH-NOTES.md looks like this (the agents write it in Chinese, so it is shown as produced):
## 2026-04-14 情報摘要
### 趨勢 1: Agent-to-Agent 協議標準化
Google 和 Anthropic 分別發布了 agent 通訊協議...
→ 跟 OpenClaw 的 Content Relay 機制有關，可以寫比較文
### 趨勢 2: 企業 AI 安全預算增加 300%
Gartner 報告顯示...
→ UltraProbe 的定位正好在這個趨勢上
### 趨勢 3: 免費 LLM API 的競爭白熱化
Gemini, Claude, Llama 都在搶免費層...
→ 我們的零成本架構文章會有很多人想看
How the agents use this intelligence
Here is the key point: this RESEARCH-NOTES.md is shared by all 4 agents.
One round of research, four agents benefit.
When each agent posts automatically, it loads the latest RESEARCH-NOTES.md into the prompt's context:
# Inject the intelligence into the autopost prompt
RESEARCH=$(tail -30 "$WORKSPACE/RESEARCH-NOTES.md")
prompt="你是 ${AGENT_ROLE}。
以下是今天的行業情報：
$RESEARCH
根據你的專業角度和品牌定位，寫一篇有觀點的文章。
不要複述新聞，要有自己的分析和見解。"
This brought a very big change: the agents' posts are no longer generated out of thin air. They are based on what is actually happening in the world.
Before: "5 AI Security Best Practices" (a hollow, generic post) Now: "Why Google's Newly Released Agent Protocol Ignores the Security Layer, and How You Should Fill the Gap" (timely analysis)
Competitor analysis: another zero-cost pipeline
Besides tech trends, we have a second pipeline dedicated to watching competitors, competitor-watch.sh:
Every day at 06:30 CST (0 LLM tokens)
│
├─ 1. Fetch Moltbook's top posts → who is posting what, and their scores
├─ 2. Analyse patterns → average title length, share of questions, share of how-tos
├─ 3. Exclude our own accounts
└─ 4. Output → COMPETITOR-INTEL.md
Note: this pipeline uses no LLM at all. It is entirely HTTP requests plus local JSON processing.
Sample output (in Chinese, as the pipeline produces it):
## 競品情報 2026-04-14
### 熱門文章 Top 5
1. "如何用 Claude 建自動化工作流" (score: 89)
2. "LLM 安全漏洞分類" (score: 67)
...
### Pattern 分析
- 平均標題長度: 18 字
- 提問式標題: 35%
- How-to 教學: 42%
- 趨勢分析: 23%
### 建議
- 提問式標題表現較好
- How-to 內容是基本盤
- 差異化機會：實戰案例（目前只有 8%）
After reading this report, the agent knows that "everyone is writing how-tos, so I can write case studies to stand out".
That is not a strategy I taught it. It worked it out from the data on its own.
Cost calculation
Here is what the whole intelligence system costs per day:
| Component | Daily calls | LLM tokens | Cost |
|---|---|---|---|
| RSS scan | ~20 HTTP | 0 | $0 |
| HN API | ~20 HTTP | 0 | $0 |
| Jina Reader | ~5 HTTP | 0 | $0 |
| Research analysis | 2 runs | ~2 RPD | $0 (free tier) |
| Competitor analysis | 1 run | 0 RPD | $0 |
| Total | ~2 RPD | $0/month |
The entire intelligence system ran on 0.13% of the 1,500 RPD quota (Gemini's free tier as of 2026-04-14). For how many requests each of the fleet's other jobs (posting, quality gates, comment replies, the weekly strategy meeting) used per day, see the daily RPD budget for all four agents from March 2026.
Three things I learned
1. Separate intelligence collection from analysis
Mix them together and every run spends a large number of tokens "reading" raw web pages. Separate them, and collection runs on plain HTTP while analysis only has to read an organised summary.
2. Sharing beats duplication
One round of research, used by four agents. If every agent did its own research, it would cost 4 times the tokens. Centralised research with distributed use is the key to a multi-agent system.
3. Data-driven beats instruction-driven
Instead of writing "you must have original opinions" in the prompt, give it real data and let it form opinions on its own. Once an agent has intelligence to work with, the improvement in output quality far exceeds anything you can get from rewording the prompt.
Build it yourself
If you want to build one yourself, start here:
- Pick 3-5 RSS feeds from your industry. You do not need many; quality matters more than quantity.
- Set up a cron job or timer that runs once a day. It handles collection plus analysis.
- Output to a single markdown file. Let your agent, or you, read it every day.
- Do not chase perfection. Get it running first, then tune the quality over time.
The full research-chain.sh and competitor-watch.sh are both in openclaw-playbook.
Next post
The intelligence system made a single agent smarter. But once you have 4 agents, a new problem appears: how do they know what the others are doing? How do they avoid duplicating each other? How do they collaborate?
Next: Multi-Agent Collaboration Architecture: How to Keep 4 AIs from Fighting
This is part 3 of the "An AI Team for a One-Person Company" series. Part 1: Why I Needed an AI Team · Part 2: The Day My First Agent Went Live
All source code: GitHub: openclaw-playbook
FAQ
How can I use AI to do daily market research at zero cost?
Separate collecting from analysing. The collection stage fetches raw data with plain HTTP requests to RSS feeds, the Hacker News API and Jina Reader, using no LLM tokens at all; each run calls an LLM only once, for analysis, turning the data into a markdown intelligence summary that your agent, or you, reads every day. When this post was written in April 2026, the pipeline ran twice a day with the research analysis on the free tier available at the time, which worked out to an estimated $0 a month.
What is Jina Reader and does it cost money?
Jina Reader turns any web page into clean markdown, automatically stripping ads, navigation bars and sidebars and keeping only the body text. You put r.jina.ai/ in front of the URL; no API key is needed and there is a free quota. That lets an agent read the full text of an article, not just the headline.
Does the Hacker News API need authentication?
No. HN has a public Firebase API: fetch topstories.json to get the IDs of the top stories, then fetch each item for its title, score and URL. HN's top stories are a signal of where the tech community's attention is: if a topic has 200+ points on HN, that is what people are paying attention to right now.
Does each AI agent need to do its own market research?
No. In this setup one round of research produces a single RESEARCH-NOTES.md that all 4 agents share, and each agent loads the latest notes into its prompt's context when it posts. If every agent did its own research, it would cost 4 times the tokens; centralised research with distributed use is the key to a multi-agent system.
How do I start building my own intelligence pipeline?
Pick 3 to 5 RSS feeds from your industry, set up a cron job or timer that runs once a day to handle collection plus analysis, and output to a single markdown file that your agent, or you, reads every day. Do not chase perfection: get it running first, then tune the quality over time.