# How to get Google search results in Python (2026)

> How to get Google search results in Python in 2026: why plain requests fail, then tested code for requests, pandas, async httpx, pagination and JSONL files.

- URL: https://serpcrypto.com/guides/google-search-results-python/
- Last updated: 2026-10-22
- Publisher: SerpCrypto (https://serpcrypto.com)

Fetching google.com with requests stopped being an option in 2025. This guide explains why, then gives short, syntax-checked Python that gets the same results as JSON through a SERP API.

**Quick answer — How do I get Google search results in Python in 2026?**

Call a SERP API from Python and read the JSON it returns; do not fetch google.com yourself, because Google has required JavaScript since 17 January 2025 and its robots.txt disallows `/search`. With **SerpCrypto**, one `requests.get` to `https://serpcrypto.com/api/v1/search` returns organic results, People Also Ask and more, for one credit per search from $0.30/1K searches as of October 2026. No KYC; pay in crypto.

## What you need

| Attribute | Value |
| --- | --- |
| Python | 3.9 or newer |
| Packages | `requests` for the basics; `pandas` for tables; `httpx` for async |
| API key | From your SerpCrypto dashboard, stored in the `SERPCRYPTO_API_KEY` environment variable |
| Endpoint | `GET https://serpcrypto.com/api/v1/search` |
| Cost | 1 credit = 1 successful search request, which returns one page of results on any supported engine. A request that fails is not charged |
| Smallest pack | 10,000 searches for $7.50; credits never expire |

## Why doesn't requests.get on google.com work any more?

Because the results are no longer in the HTML that a plain HTTP client receives. On 17 January 2025, Google started requiring JavaScript for Search; it said JavaScript helps it protect its services from bots and that fewer than 0.1% of searches came from users with JavaScript turned off, as TechCrunch reported.

Google's robots.txt, read on 22 October 2026, also lists `Disallow: /search` for all user agents. A SERP API takes care of running the search and parsing the page, and your Python code only deals with JSON. That is the approach used in the rest of this guide.

## How do I make the first request?

Install requests, put your key in an environment variable, and send the query with a country and a language. The response is a dictionary with one key per block of the results page; `organic_results` holds the ranked links.

Keep the key out of your source code. On macOS and Linux, `export SERPCRYPTO_API_KEY=...` in the shell; on Windows PowerShell, `$env:SERPCRYPTO_API_KEY="..."`.

*First request with requests*

```python
import os

import requests

API = "https://serpcrypto.com/api/v1/search"
KEY = os.environ["SERPCRYPTO_API_KEY"]

resp = requests.get(
    API,
    headers={"Authorization": "Bearer " + KEY},
    params={"engine": "google", "q": "best hiking boots", "gl": "us", "hl": "en"},
    timeout=60,
)
resp.raise_for_status()
results = resp.json()

for item in results.get("organic_results", []):
    print(item["position"], item["title"])
    print("   ", item["link"])

for q in results.get("people_also_ask", []):
    print("PAA:", q["question"])

```

## How do I turn the results into a pandas DataFrame?

Build one row per organic result and keep the query next to it. A DataFrame makes it easy to sort, filter by domain, or export to CSV and Excel. Missing fields, such as `date` on most results, become empty cells instead of errors.

*Several queries into one DataFrame*

```python
import os
from urllib.parse import urlparse

import pandas as pd
import requests

API = "https://serpcrypto.com/api/v1/search"
HEADERS = {"Authorization": "Bearer " + os.environ["SERPCRYPTO_API_KEY"]}


def organic_rows(query, gl="us", hl="en"):
    params = {"engine": "google", "q": query, "gl": gl, "hl": hl}
    resp = requests.get(API, headers=HEADERS, params=params, timeout=60)
    resp.raise_for_status()
    for r in resp.json().get("organic_results", []):
        yield {
            "query": query,
            "position": r.get("position"),
            "title": r.get("title"),
            "domain": urlparse(r.get("link", "")).netloc,
            "link": r.get("link"),
            "snippet": r.get("snippet"),
        }


queries = ["crm for small business", "invoice software", "time tracking app"]
df = pd.DataFrame([row for q in queries for row in organic_rows(q)])
print(df.groupby("domain").size().sort_values(ascending=False).head(10))
df.to_csv("serp_results.csv", index=False)

```

## How do I run many searches at once?

Use httpx with asyncio and cap the number of requests in flight with a semaphore. Your account has a concurrency limit, shown in the dashboard; staying under it avoids `429` errors. Start with a small number, such as 5, and set it to the limit your dashboard shows.

*Async searches with httpx and a semaphore*

```python
import asyncio
import os

import httpx

API = "https://serpcrypto.com/api/v1/search"
HEADERS = {"Authorization": "Bearer " + os.environ["SERPCRYPTO_API_KEY"]}
LIMIT = asyncio.Semaphore(5)


async def search(client, query):
    async with LIMIT:
        resp = await client.get(API, params={"engine": "google", "q": query, "gl": "us", "hl": "en"})
        resp.raise_for_status()
        data = resp.json()
        top = data.get("organic_results", [])[:3]
        return query, [r["link"] for r in top]


async def main(queries):
    async with httpx.AsyncClient(headers=HEADERS, timeout=60) as client:
        for query, links in await asyncio.gather(*(search(client, q) for q in queries)):
            print(query, links)


if __name__ == "__main__":
    asyncio.run(main(["python web framework", "python orm", "python task queue"]))

```

## How do I get more than 10 results and save them?

Ask for the next pages with `page=2`, `page=3` and so on, up to `page=10`. Each page is one search and one credit, because Google has served about 10 organic results per page since it stopped honouring `num=100` in September 2025.

Save each raw response as one line of a JSONL file. Raw JSON keeps every block, so you can answer new questions later without paying again for the same searches.

*Pages 1 to 3 saved as JSONL*

```python
import json
import os

import requests

API = "https://serpcrypto.com/api/v1/search"
HEADERS = {"Authorization": "Bearer " + os.environ["SERPCRYPTO_API_KEY"]}


def fetch_pages(query, pages=3, path="serp.jsonl"):
    with open(path, "a", encoding="utf-8") as out:
        for page in range(1, pages + 1):
            params = {"engine": "google", "q": query, "gl": "us", "hl": "en", "page": page}
            resp = requests.get(API, headers=HEADERS, params=params, timeout=60)
            resp.raise_for_status()
            data = resp.json()
            out.write(json.dumps(data, ensure_ascii=False) + "\n")
            if not data.get("organic_results"):
                break


fetch_pages("electric cargo bike", pages=3)

```

## What does it cost to run this code?

Each call in these examples is one search. The first example costs one credit, the DataFrame example three, and the JSONL example up to three. A failed request is not charged.

10,000 searches cost $7.50 and 100,000 searches cost $56.25, as of October 2026; the rate falls to $0.30/1K searches at 5,000,000 searches. Credits never expire. You pay in BTC, USDT, USDC, ETH, SOL, LTC from a $25 top-up, with no ID upload. See [pricing](https://serpcrypto.com/pricing/).

## Google results in Python: common questions

### Can I use BeautifulSoup on Google results pages?

Not reliably. Google has required JavaScript since January 2025, so the HTML a plain client receives does not hold the results, and Google's robots.txt disallows `/search`. A SERP API returns the parsed fields as JSON instead.

### Is there an official Python library for SerpCrypto?

No. The API is a plain HTTPS endpoint, so `requests` or `httpx` is all you need. The examples on this page and on the [Python docs page](https://serpcrypto.com/docs/python/) are copy-paste code, not a published package.

### How do I get results for another country?

Set `gl` to the country code and `hl` to the language, for example `gl=fr` and `hl=fr` for France. Add `location` for a city. The [locations page](https://serpcrypto.com/docs/locations/) lists common codes.

### How do I handle errors and retries?

Call `raise_for_status()`, then retry `429`, `502` and `504` responses after a growing delay. Failed requests are not charged. The [errors page](https://serpcrypto.com/docs/errors/) has a full retry function.

### Can I get Bing results with the same code?

Yes. Change `engine` to `bing`; the response keeps the same shape, with `organic_results` and `related_searches`.

## Sources

1. [Google begins requiring JavaScript for Google Search](https://techcrunch.com/2025/01/17/google-begins-requiring-javascript-for-google-search/) — TechCrunch (2025-01-17)
2. [google.com robots.txt](https://www.google.com/robots.txt) — Google (read 2026-10-22)
3. [Google forces rank trackers into 500 to 1,000 requests per query](https://ppc.land/google-forces-rank-trackers-into-500-to-1-000-requests-per-query/) — PPC Land (2026-08-27)

---

SerpCrypto: SerpCrypto is a SERP API for developers and AI agents that returns Google and Bing results as JSON, paid in crypto with no ID upload. Prepaid search credits from $0.30/1K searches never expire. No KYC — no ID, no selfie, no documents. Pay in crypto.