Absolutely, this is a great example where Hister can shine.

I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.

Also very interested in this. I was playing around with doing the same thing with YaCY. I want the proxy aspect so that I can proxy my phone traffic through it as well.

Would you mind sharing these links? Or a subset? I want to grow my collection which is tiny because I started way too late

And the number of times I've searched for something that I saw a while ago but is now gone is way too damned high.

Hister stores only the text content of HTML/pdf pages. 1000 documents require around 80-100MB of storage and there is still plenty of room to optimize for storage space.

I'm using it for 6-7 months and my index size is below 1GB with almost 10k pages.

Also, a downside of the proxy approach: it does not handle properly JS based websites and cannot identify dynamic content changes. Our extension periodically checks if the browser tabs' content has been changed and automatically updates the index when change detected.

Multiple users can use a shared instance and collect their indexed content in a central place. Hister has user handling and a "public mode" as well: https://hister.org/posts/public-search

> What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful?

There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation.

> Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?

My long term goal is exactly this. I'd like to add federation/P2P feature [2][3] to evolve from being a private search companion. I'd appreciate any help designing the system.

[1] https://blevesearch.com/docs/Home/ [2] https://github.com/asciimoo/hister/discussions/432 [3] https://hister.org/posts/public-search

It is possible. The automatic website indexing can be turned off in the extension and manual indexing can be triggered via the command line tool, the extension popoup or by hotkeys.

In its current form it cannot handle this amount of data efficiently (and doesn't support powerpoint/excel/word yet), but this is a valid use-case, I've added a TODO item to experiment with it.

Both are search engines, but that's all the similarity. Hister has a traditional crawler, but its biggest strength is automatically indexing browser tabs as those are rendered. This way it bypasses authentication, CloudFlare, captchas and most of the annoying limitations of traditional crawlers. Hister also provides full offline result previews. Check out the small read-only demo: https://demo.hister.org/

I accept donations ;)

Hister has a built in crawler with standard HTTP lib and browser based backends, you can feed your link collection to it. Also, Hister supports importing your existing browser history automatically using either of the mentioned backends.

Sure, it cannot fully replace web search engines (yet), but it can reduce the dependence on these services more and more as your index grows. Hister is designed to support quickly falling back to traditional search engines with a single hotkey if no results found.

I agree, we should add more extractors [1]. Can you recommend extractors you missed?

[1] https://github.com/asciimoo/hister/issues/305

I ran my own YaCY instances. Three of them to be specific, because they are "super fast" and "reboot" often. I crawl with them the smallweb, smallcomic and smallyt sites and also all feeds from my miniflux instance; getting them via the miniflux api. Beside that i have other static entries that i crawl. For wikibooks and wikipedia i tried and use also YaCY, but it use a lot of resources. So its only in one instance. I suggest >16GB RAM and 300GB+ HDD if you want to do this. To access wikimedia, gutemberg, archwiki or media.ccc.de directly, I use also SearXNG. Usually it takes 1-3 Seconds to get search results from YaCY in my setup. I run them in docker on aarch64 with ~6GB of RAM and 200GB HDD. The VPS it-self has 8GB RAM, 6 arm cores and 250GB HDD. If YaCY hang, i just restart it. This are my pretty good working docker deploy and java settings I use currently:

    environment:
      JAVA_OPTS: >-
        -XX:+UseG1GC
        -XX:MaxGCPauseMillis=200
        -XX:+ParallelRefProcEnabled
        -XX:+UseStringDeduplication
        -XX:InitiatingHeapOccupancyPercent=45
        -XX:G1ReservePercent=15
        -Xms1024m
        -Xmx3072m
        -XX:MaxMetaspaceSize=256m
        -XX:MaxDirectMemorySize=256m
        -XX:+ExitOnOutOfMemoryError
        -XX:G1HeapWastePercent=10
        -XX:G1MixedGCCountTarget=4
    deploy:
      resources:
        limits:
          cpus: "4.2"
          memory: 5.2G
        reservations:
          cpus: "2"
          memory: 2.5G
    healthcheck:
      test: |
        /bin/bash -c '
        if ! timeout 55s wget --spider --no-verbose http://127.0.0.1:8090/yacysearch.html?query=exiguus; then
          exit 1
        fi
        if ! timeout 55s yacy_search_server/bin/checkalive.sh; then
          exit 1
        fi
        exit 0
        '
      interval: 120s
      timeout: 60s
      retries: 3
      start_period: 240s

That's the smallest I got it running mostly stable and self-healing with a index size of +100GB. I also avoid to use crawling by the build in tasks and use the API and cron jobs for weekly feed importing, because I found out, that kind of crawling eats up less resources then the usual. All-Over, to much running crawlers, make retrieving search results slow. For production use, I suggest to min. double the resources. If you do this, it becomes very stable.

Thanks to pointing out kiwix. I'll give it a try.

I use Searxng through Onyx, both as regular search and Onyx's Deep Research mode. I also have https://github.com/ihor-sokoliuk/mcp-searxng to add search to coding agents. Haven't really had many issues with it.

Yep -

`gemma4-26b-a4b-it-qat.gguf`

https://huggingface.co/lmstudio-community/gemma-4-26B-A4B-it...

It is really great to use. As the poster above mentioned, my setup with Sear is the following, all through `llama.cpp`, which has a built-in webui with an MCP client:

* SearXNG in Docker — enable the JSON API (`search.formats: [html, json]`; off by default).

* `searxng-mcp` (FastMCP, native streamable-HTTP): `TRANSPORT=streamable-http HOST=127.0.0.1 PORT=8100` `SEARXNG_URL=http://localhost:8888 uvx --from searxng-mcp --with fastmcp searxng-mcp`

* `llama-server` with `--webui-mcp-proxy`, then add the server in the webui.

Some gotchas:

* `searxng-mcp` forgets to declare its own dep → `--with fastmcp`.

* Endpoint is `/mcp`, not the `/searxng-mcp/mcp` the docs claim.

* `--webui-mcp-proxy` only enables the CORS proxy; each MCP server entry still needs its "Use llama-server proxy" checkbox ticked, or the browser fetches direct and CORS-fails.

* Terminal clients (OpenCode etc.) skip the proxy — point them straight at `:8100/mcp`.

A couple interesting tidbits:

* There are temporal issues with search-related tool calls. The model trips out. 2026 results read to it a "future-dated hallucination" because it doesn't know the date. There's an additional `--tools get_datetime` function that will allow it to ground via the real date.

* Snippets-only is enough for most "what's current" questions and keeps context tiny.

Let me know if you have any questions!

https://www.zarl.dev/posts/hal-by-any-other-name Here is my write up on my local model setup also have https://zarldev.github.io/zarlmono/ as my local 1st coding agent

It does have json response though so pretty trivial to get an LLM to make you a mcp