HackerRank VS Lyrenth

Lyrenth's answer:

Lyrenth is an index, not a scraper. That one difference changes the economics.

A scraper fetches the page again for every caller, every time. Lyrenth fetches it once, turns it into a clean AIDocument, and serves that same document to everyone who asks for the URL afterwards. Over 3.7 billion pages are already indexed, across 166 million domains, growing by more than 50 million a day.

• One shape for every URL. Cleaned Markdown, page structure, provenance, and the token and cost economics, in one JSON envelope that is versioned and validated against a published schema.
• Measured savings, not estimates. The Stripe API reference goes from 307,902 raw HTML tokens to 2,000 in its AIDocument, a 99.4% reduction. Ten benchmarks are published with the exact numbers, reproducible with one API call.
• The format is open. AIDocument is defined by the AIWebIndex standard, kept deliberately separate from the company so the format does not depend on us. Anyone can implement, fork or extend it.
• The crawler is identifiable. It honours robots.txt, publishes its IP ranges, forward-confirms its reverse DNS, and signs its requests under Web Bot Auth. We do not train foundation models on crawled content.

Lyrenth's answer:

Because the work has usually been done already.

• Someone else paid for the fetch. The index is shared, so a page another caller read is served to you from the index. In the published benchmark every repeat read came back in 10 ms or less, whatever the page weighs.
• JavaScript pages are rendered once, on our side. A page that ships an empty shell is the slowest thing on the web to read. We take that hit once, then serve it like any other page.
• Predictable billing. One credit per document. No per-byte fees, no size multipliers, and failed fetches never count. You are paused at your cap rather than surprise-billed.
• Freshness you control. Every plan states how old a cached page can be, and you can force a fresh copy of any URL at any time.
• Friendly to the sites you read. One identified crawler visits a page for everyone, instead of every customer hitting the same origin separately. Site owners can verify who we are, and opt out in one line.
• No lock-in. The document format is an open standard with a public JSON Schema, so what you build against is not ours to take away.

Lyrenth's answer:

Developers who need machines to read the web reliably.

• Agent builders. Anyone giving an AI assistant or autonomous agent the ability to read a URL, through the API, an SDK, or the MCP server in Claude Desktop, Claude Code and Cursor.
• RAG and search teams. Engineers ingesting web content into vector stores or retrieval pipelines, who need clean, structured, consistent input instead of raw HTML.
• AI platform and infrastructure teams. Companies whose own product reads the web at volume and would rather not build and maintain a crawler, a renderer and an extractor.
• Site owners, as a second audience. Publishers who want to see which AI crawlers visit them, check how readable their pages are to machines, and control how their content is represented.

In practice that is developer-first: people who read the docs, try the free tier without a card, and integrate the same day.

Lyrenth's answer:

What we publish is the interface, not the engine. The AIDocument contract is open, versioned and validated against a public JSON Schema, so what you build against is stable and inspectable. How we produce it is ours.

What you actually integrate with:

• A REST API that returns the AIDocument JSON contract
• Python and TypeScript SDKs
• A LangChain document loader, a LlamaIndex reader, and a Vercel AI SDK tool
• An MCP server for Claude Desktop, Claude Code, Cursor and any other MCP client

Open standards we implement rather than reinvent:

• AIWebIndex, the open format for AI-readable web documents
• RFC 9309, for robots.txt compliance
• RFC 9421 HTTP Message Signatures, for signed and verifiable crawler requests
• JSON Schema, for the document contract

The index, the crawler and the extraction pipeline are built in-house. Lyrenth is not a wrapper around someone else's scraping service.

Lyrenth's answer:

At the end of 2025 our founder, Aleksandar Martinovic, read a short post that said the next big thing is not only making AI smarter, but giving AI the ability to see the web the way humans see it. That sentence started Lyrenth.

The problem became hard to unsee. Every AI company, agent and assistant was solving the same task privately: fetch a page, strip the navigation, remove the cookie banners, parse broken HTML, pay for the same work again and again. The information was already on the web. What was missing was a layer machines could read.

So we started building one. Not a replacement for the web, and not a closed archive owned by one model company, but a cleaner, structured, machine-readable version of pages that already exist.

Two parts came out of that, and we keep them separate on purpose:

• AIWebIndex, the open standard for AI-readable web documents, so the format does not belong to one company.
• Lyrenth, the commercial implementation: the production index, API, crawler and tooling, operated by Aleksma Ai, Inc., a Delaware corporation.

It is bootstrapped, and currently in public beta.

Lyrenth's answer:

Lyrenth is in public beta and we do not name individual users without their permission, so there is no customer list to publish yet. When there is one, it will be real names who agreed to be listed.

What we can point at instead is where Lyrenth is already integrated:

• Anthropic's Claude connectors directory
• The official Model Context Protocol registry
• The Cursor directory, Smithery, Glama and LobeHub
• LangChain, listed in the official documentation as a provider, tool and document loader
• LlamaIndex, through a published reader package
• npm and PyPI, as officially maintained SDKs