There is no public Google Trends API for interest over time. The website, the Python library pytrends, and the actor I link below all hit the same unofficial endpoints the explore page uses. The five-keyword cap is Google's.

A 100 is the peak of that request

Interest over time is an index. Google looks at every keyword in the request, finds the highest point any of them reached in the date range, and calls that point 100. Everything else is an integer share of that peak.

Fetched alone, a quiet term and a huge term both peak at 100. Sent together, the quiet term falls toward 0. A 50 is not half the searches of another chart's 50, and it is not 50 searches. Two charts match only when they share a peak.

The lists under the chart use other scales. Top related queries are scored against the most common companion query, not against the interest line. Rising queries are growth figures. They do not belong in the same column.

Five keywords, then the scale resets

The explore page takes at most five terms in one comparison. A sixth term is a new request, and that request picks a new peak. Five quiet synonyms will still produce a 100. Put that 100 beside the 100 from a group that contained "coffee" and the sheet is wrong.

Groups of five answer a narrower question: is matcha rising relative to the drinks beside it? They do not answer "how do all 40 terms compare?" Coffee, tea, matcha, espresso, and cold brew fill one request. Oolong and chai need another. Without a shared reference, a 100 in that second request means "highest among whatever you paired together," not "highest next to coffee."

Keep one anchor in every request

Use one of the five slots for a term you already understand, and put that same term in every request. Four slots remain for the list. Each group is then scaled against a set that includes the anchor.

This lasts only while the anchor is the highest series in every group. If another keyword spikes past it, Google moves 100 to that spike, and the anchor's line in that request is no longer on the same scale as the anchor's line in a quieter group. "Coffee" is a sound anchor for tea styles. It is a bad anchor for a list that also contains "weather" or "youtube."

If the export repeats the anchor, compare those copies. A peak of 100 in one chunk and 40 in the next means the second chunk was scaled to something else. This run stores coffee once, from the first request, so it does not prove the second request used the same coffee series. For the week of 19 October 2025 it shows coffee at 65, tea at 33, and the second-group terms at 0 and 3. That fits an anchor that still dominated.

Dividing a single-keyword series by its own maximum does not create a shared scale. Each series was already 100 at its own peak.

Top queries and rising queries

Top queries are what people often type along with the term. The most common one scores 100 inside that list only. On this run, coffee's top two were "coffee near me" at 100 and "coffee shop" at 87. Tea's were "green tea" at 100 and "black tea" at 40. Tea's 100 is not coffee's interest value of 65. It means "green tea" led tea's companions.

Rising queries are the biggest increases. Google prints a percentage, or Breakout when growth is too large or the baseline too small for a tidy percent. Tea's first rising row was "how to make cold brew tea" at +2,000%. Coffee's first rising row was "sports scores today," labeled Breakout. Google did return that row. It is a bad keyword suggestion. Rising lists are noisy on broad head terms. Treat them as leads, and drop the ones that are not about the topic.

Related topics were empty on this run. Google often skips that widget for scripted clients and still returns queries. Empty topics are not a failed keyword.

A worked example

I built Google Trends Scraper (Batch, Reliable) to chunk a list and to keep an anchor in each request, then write one dataset row per keyword. You do not need it for the method. pytrends, below, calls the same endpoints from your own machine.

This input asks for six drinks, with coffee fixed as the anchor, for US web search over the past 12 months. Category 0 is all categories.

{
  "keywords": ["tea", "matcha", "espresso", "cold brew", "oolong", "chai"],
  "geo": "US",
  "timeframe": "today 12-m",
  "category": 0,
  "gprop": "web",
  "comparisonMode": "anchor",
  "anchorKeyword": "coffee"
}

I ran it on 20 October 2026. It took 48 seconds at 256 MB and cost $0.00083. Google answered HTTP 429 six times. The actor retried, opened a second datacenter session, and still saved 7 of 7 rows. Nothing was marked blocked.

Coffee is stored once, with isAnchor: true. The other rows are scale: "anchored". Tea, matcha, espresso, and cold brew share chunk 0 with coffee. Oolong and chai are chunk 1, a second Google request that also included coffee. Each series has 53 weekly points. The last week, 18–24 October 2026, is partial: true because that week was not finished on the run date.

First point, week of 19 October 2025:

Oolong at 0 does not mean nobody searched it. Next to coffee, that week rounded to nothing. Chai at 3 is the same kind of number, from the other request.

The real interestOverTime array has 53 objects. One week is enough to show the row:

{
  "keyword": "tea",
  "geo": "US",
  "timeframe": "today 12-m",
  "comparison": "anchor",
  "chunkIndex": 0,
  "anchorKeyword": "coffee",
  "isAnchor": false,
  "scale": "anchored",
  "status": "ok",
  "interestPoints": 53,
  "interestOverTime": [
    { "timestamp": 1760832000, "date": "Oct 19–25, 2025", "value": 33, "partial": false }
  ],
  "relatedQueries": {
    "top": [
      { "query": "green tea", "value": "100", "extractedValue": 100 },
      { "query": "black tea", "value": "40", "extractedValue": 40 }
    ],
    "rising": [
      { "query": "how to make cold brew tea", "value": "+2,000%", "extractedValue": 2000 }
    ]
  }
}

Without the actor

pytrends is the usual free Python client. It is unofficial, it runs on your machine, and it adds no per-keyword fee. A two-term comparison is:

from pytrends.request import TrendReq

pytrends = TrendReq(hl="en-US", tz=0)
pytrends.build_payload(["coffee", "tea"], timeframe="today 12-m", geo="US")
pytrends.interest_over_time()
pytrends.related_queries()

I did not use pytrends for the table. You still slice the list into fives, repeat the anchor, and handle HTTP 429 yourself. Releases go quiet when Google changes the endpoints. That is a fair deal if you already run Python and the list is small. The Trends site is the other free option: fine for five terms, slow for ten anchored groups copied by hand.

Trust the index inside one request, and only when the anchor stayed on top. The export is not search volume, and a Breakout row is not a recommendation.

Two files decide a lot of this before any page is parsed: robots.txt and, if you publish one, /llms.txt. After that, the usual on-page checks still matter, because an answer engine quotes a title, a heading, and a paragraph.

I built an Apify Actor that scores those checks from 0 to 100. The checklist below does not need it. The numbers later are from a real run.

Which bot you are blocking

These are not one crawler.

• GPTBot is OpenAI's training crawler. ChatGPT search can use a different agent, OAI-SearchBot. Blocking GPTBot does not, by itself, block that.
• ClaudeBot is Anthropic's crawler. Anthropic also documents other agents for user-triggered fetches. A Disallow for ClaudeBot does not name those.
• PerplexityBot is Perplexity's crawler. Perplexity has described a separate user-agent for fetches a person triggers.
• Google-Extended is Google's control for Gemini and Vertex AI use. It is not Googlebot. Disallowing Google-Extended does not remove a page from Google Search.

If you only check User-agent: *, you will misread sites that special-case one bot.

How a group is chosen

robots.txt is groups of User-agent lines, then Allow and Disallow. A named group beats * when the bot's name matches. An empty Disallow: means "allow everything" for that group. No matching group means the bot is allowed.

On 19 October 2026, https://jvns.ca/robots.txt was 212 bytes and said, in substance:

User-agent: *
Disallow:

Sitemap: https://jvns.ca/sitemap.xml

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

There is a comment, "NO LLM PLZ". The rules are narrower than the comment. GPTBot and Google-Extended are blocked at /. ClaudeBot and PerplexityBot have no group of their own, so they inherit *, whose empty Disallow allows them. A crawler that only reads the comment will get this site wrong.

https://www.python.org/robots.txt does not mention those four names. The * group blocks a few old paths, not /. The homepage is allowed for all four. https://apify.com/robots.txt is User-agent: * with Allow: /, plus a Sitemap: line. It also sends Content-Signal: search=yes, ai-input=yes, ai-train=yes. That line is not a robots rule. I would not treat it as one until the client you care about documents it.

llms.txt is a map, not a permission

/llms.txt is a proposal for a Markdown file at the site root that tells a model which URLs matter and what the site is. It is not a standard, and it does not override robots.txt. A missing file is not a penalty in Google Search. It does mean you skipped an easy place to describe the site in sentences.

I requested three:

• https://www.python.org/llms.txt returned 404.
• https://jvns.ca/llms.txt returned 404.
• https://apify.com/llms.txt returned 200, text/plain, about 16 KB. It opens with a one-paragraph description of the site and then points at concrete URLs.

If you add one, write it for a model that will not click around. Name the canonical docs, the changelog, and the pages you want quoted. Do not paste your whole site into it.

What the HTML itself needs

Fetch the URL with something that does not run JavaScript. Then check the response, not the painted page.

1. Final status is 200, with at most one redirect.
2. No noindex in the robots meta tag or X-Robots-Tag.
3. One title of about 30–60 characters, and one meta description of about 70–160.
4. One H1 that says what the page is. Extra H1s inside examples still count, because they are in the HTML.
5. One absolute rel=canonical.
6. html lang set.
7. JSON-LD that matches the page. Organization or Article helps a quoter more than an empty WebSite node, but WebSite is still real structured data. Do not add FAQPage unless the page is an FAQ.
8. The first paragraph under the H1 answers the page in plain text, before the next section. About 40–80 words is enough to see it. This is a writing check, not a proven ranking factor.
9. An author and a date in the HTML or in Article schema, where a date means something.
10. alt on images that are content. A coverage percentage cannot tell decoration from content.
11. Internal links in the HTML. A nav that appears only after hydration is invisible here.

Open Graph and Twitter cards are for sharing, not for crawl permission. I still note them.

A three-page run

The Actor I built, SEO Audit & AI Search Readiness, does the list above over plain HTTP. It does not launch a browser. On 19 October 2026 I audited three homepages with discoverFromSitemap off. Run 2ZsEXDHay0SOS27v7, 256 MB, about $0.00009. Three pages, no fetch errors, average score 76.

The Python.org H1 result is a lead, not a screenshot. The check counts H1 elements in document order. A homepage that embeds a tutorial can fail "one H1" even when the design looks fine.

jvns.ca is the useful AI example. The site is readable, the opening HTML answers who the site is, and the robots file blocks two of the four bots on purpose. A score of 64 is not "this blog is bad." It is "several checks I weighted did not pass," including a short title, no JSON-LD, and those two Disallow rules. The weights are mine. They are not Google's.

apify.com lost points for answer-first structure and for having neither an author nor a date on the homepage. Both are reasonable things for a marketing homepage to skip. I would not rewrite a homepage into a fake article to gain those points.

What this kind of audit cannot see

It cannot see text that appears only after JavaScript runs. If the word count is tiny and the HTML is a <div id="root">, the page is not ready for a crawler that does not render. It also cannot see bots you did not name. If the bot you care about is OAI-SearchBot or Googlebot, read that group yourself.

robots.txt is checked for the path that was fetched. A bot blocked only under /account can still be allowed on /.

For the URL list first, I use Sitemap URL Extractor, then audit a sample. Both are free apart from platform usage.

A short checklist

• Fetch /robots.txt and write down the group for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Confirm you meant that result.
• Fetch /llms.txt. If you want models to quote you, add a Markdown map. If you do not, a 404 is an honest answer.
• Fetch one important URL with JavaScript disabled. Read the title, the H1 count, the canonical, and the first paragraph.
• Validate JSON-LD. Keep types you can defend.
• Do not block a bot in a comment and allow it in the rules, or the reverse. jvns.ca shows why the file, not the banner, is the rule.

I built a small Apify Actor that does this for larger lists. You do not need it for the method below. The Python script uses only the standard library. The table later is from a real, capped run of the Actor.

Read robots.txt before you guess paths

The sitemap protocol asks sites to advertise files with Sitemap: lines. On 19 October 2026, https://docs.python.org/robots.txt contained:

Sitemap: https://docs.python.org/sitemap.xml

That is the file to fetch. Do not assume the homepage and the docs share one. https://www.python.org/robots.txt has no Sitemap: line. Requesting /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, and /sitemap.xml.gz on that host returned 404. The documentation sitemap is real. The homepage does not publish one.

A robots file can list several Sitemap: lines. Collect all of them. Ignore Allow and Disallow for this job. Those lines govern crawling, not the URL list.

An index is not a page list

There are two root elements.

A urlset holds pages. Each <url> has a <loc>, and sometimes <lastmod>, <changefreq>, and <priority>. https://docs.python.org/sitemap.xml is a urlset. When I fetched it while writing this, it had seven <loc> values: /3.11/ through /3.16/, plus /3/.

A sitemapindex holds other sitemap files. https://apify.com/sitemap.xml is an index. It listed 17 child files, from https://apify.com/sitemap/pages.xml through https://apify.com/sitemap/users.xml. Those <loc> values are not pages. If you save them as pages, your list is a list of XML files.

Follow child sitemaps, and remember URLs you already downloaded. Indexes can point at other indexes. A seen-set stops a loop.

Gzip, and locs that are not pages

Some files are gzip. Sometimes the URL ends in .gz. Sometimes the body is gzip and the URL does not. Check the bytes, not the name. A gzip payload starts with 1f 8b. Decompress once, then parse XML. The script below does that. A file gzipped twice is rare; this script does not peel a second layer.

Namespaces cause the other common bug. Google image, video, and news sitemaps add extra <loc> elements inside <image>, <video>, or <news>. Those are media URLs. Only a <loc> whose parent is <url> is a page. xhtml:link is different again. On the Python docs sitemap, https://docs.python.org/3.11/ carries 19 hreflang alternates, including de and el. Those are translations of the same page, not extra rows, unless you specifically want them.

lastmod is optional and often a date the CMS rewrote for the whole file. Treat it as a hint. priority and changefreq are hints too. Google has said it ignores them for ranking. They are still useful as fields to keep.

A script you can run without an account

Save this as extract_sitemaps.py. It reads Sitemap: lines, probes four common paths only when robots.txt lists none, follows indexes, decompresses gzip, and prints page URLs.

import gzip
import re
import sys
import urllib.request
import xml.etree.ElementTree as ET
from urllib.parse import urljoin, urlparse

UA = "sitemap-extract/1.0"
PATHS = ("/sitemap.xml", "/sitemap_index.xml", "/wp-sitemap.xml", "/sitemap.xml.gz")

def fetch(url: str) -> bytes:
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=20) as resp:
        data = resp.read()
    if data[:2] == b"\x1f\x8b":
        data = gzip.decompress(data)
    return data

def local_name(tag: str) -> str:
    return tag.rsplit("}", 1)[-1]

def sitemaps_from_robots(origin: str) -> list[str]:
    try:
        text = fetch(urljoin(origin, "/robots.txt")).decode("utf-8", "replace")
    except Exception:
        return []
    return re.findall(r"(?im)^[ \t]*Sitemap:[ \t]*(\S+)", text)

def parse(body: bytes):
    root = ET.fromstring(body.lstrip(b"\xef\xbb\xbf").lstrip())
    kind = local_name(root.tag)
    children, pages = [], []
    if kind == "sitemapindex":
        for node in root:
            if local_name(node.tag) != "sitemap":
                continue
            for child in node:
                if local_name(child.tag) == "loc" and child.text:
                    children.append(child.text.strip())
    elif kind == "urlset":
        for node in root:
            if local_name(node.tag) != "url":
                continue
            row = {}
            for child in node:
                key = local_name(child.tag)
                if key in {"loc", "lastmod", "changefreq", "priority"} and child.text:
                    row[key] = child.text.strip()
            if row.get("loc"):
                pages.append(row)
    return children, pages

def extract(start: str, limit: int = 1000) -> list[dict]:
    parsed = urlparse(start if "://" in start else "https://" + start)
    origin = f"{parsed.scheme}://{parsed.netloc}"
    seeds = sitemaps_from_robots(origin)
    if not seeds:
        seeds = [urljoin(origin, path) for path in PATHS]
    seen, out, queue = set(), [], list(dict.fromkeys(seeds))
    while queue and len(out) < limit:
        url = queue.pop(0)
        if url in seen:
            continue
        seen.add(url)
        try:
            body = fetch(url)
        except Exception as exc:
            print(f"skip {url}: {exc}", file=sys.stderr)
            continue
        if not body.lstrip().startswith(b"<"):
            continue
        children, pages = parse(body)
        for child in children:
            if child not in seen:
                queue.append(child)
        for page in pages:
            if len(out) >= limit:
                break
            page["source"] = url
            out.append(page)
    return out

if __name__ == "__main__":
    site = sys.argv[1] if len(sys.argv) > 1 else "https://docs.python.org/"
    limit = int(sys.argv[2]) if len(sys.argv) > 2 else 20
    for row in extract(site, limit):
        print(f"{row.get('lastmod','')}\t{row.get('changefreq','')}\t{row['loc']}")

I ran python3 extract_sitemaps.py https://docs.python.org/ 20. It printed seven rows, including:

	yearly	https://docs.python.org/3.11/
	daily	https://docs.python.org/3.14/
	daily	https://docs.python.org/3/

python3 extract_sitemaps.py https://apify.com/ 3 followed the index and printed https://apify.com/, /about, and /actors, not the XML file URLs. python3 extract_sitemaps.py https://www.python.org/ 5 printed nothing and logged four 404s, which matches the missing Sitemap: line.

The script is a teaching tool. It has no retries, no size cap, and no check that a URL stays on the public internet. Point it at sites you are allowed to fetch. A timeout or a bad XML file is skipped, and the rest continue.

What a larger run looks like

For a long list I use the Actor I built, Sitemap URL Extractor. It is the same idea with retries, a byte cap, nested indexes, and optional image, video, news, and hreflang fields. It uses plain HTTP at 256 MB. It does not open a browser. It is free: you pay Apify platform usage only.

On 19 October 2026 I ran it with docs.python.org and jvns.ca, maxUrls 8, and 256 MB of memory. Run 7Nt2owGKPyqvBpr87 succeeded in under a second and cost about $0.00009. It fetched two sitemaps, both via robots.txt, and saved 8 URLs because of the cap. These rows are from that dataset:

https://docs.python.org/3.11/ also had alternateCount 19. The first two alternates were de → https://docs.python.org/de/3.11/ and el → https://docs.python.org/el/3.11/. The run stopped at maxUrls, so it is not the full jvns.ca sitemap. The docs file had seven page URLs when I fetched it separately a few minutes later. A cap truncates. Raise maxUrls only after the first rows look right.

Use the script when you want a few hundred URLs on your own machine. Use the Actor when you want many sites, gzip indexes, hreflang, and a dataset you can hand to something else. One thing I hand it to is an SEO audit that scores the pages. That is a separate post.