Gosoft Connect - AI chat grounded in customer documents

What it is

Gosoft Connect is a multi-tenant SaaS I designed and built solo at GoSoft, my company: embeddable website widgets - AI chat, live chat, forms, popups - and an admin portal.

Each customer workspace has a knowledge base filled four ways: paste text, fetch a URL, upload files, or crawl a whole sitemap. The AI chat on the customer’s site answers visitor questions from that knowledge base, streamed token by token.

The assistant works through internal tool calls: search tools, a calculator, and tools for looking up company details, capturing leads and handing over to a human. The first round of every substantive message is forced to call a tool, so answers come from the customer’s data, not from the model’s assumptions.

Around the chat sit digest emails, AI summaries of form submissions, conversation categorization, nightly reports of questions the bot couldn’t answer, and moderation on both input and output.

The retrieval pipeline

Answers are grounded by hybrid search: dense vectors in pgvector and full-text ranking fused with reciprocal rank fusion, then reranked by a cross-encoder, with a per-document diversity cap and language-aware retrieval across Finnish, Swedish and English content. Chunking is paragraph-aware with contextual headers, and content hashing means unchanged documents are never re-embedded.

Stack & infrastructure

Deliberately small: everything runs on one Linux node, CPU only. The embedding model (bge-m3) and the reranker (bge-reranker-v2-m3) run locally via ONNX - no GPU, no embedding API. Customer documents leave the box for exactly one thing: the LLM call itself (OpenAI, model selectable per workspace), covered by a data processing agreement with the provider. The vector store is plain PostgreSQL with pgvector - one database for everything. Bun and Hono on the backend, a Solid.js portal, nginx in front.

The engineering

The numbers I can publish are engineering numbers. Quantizing the reranker took retrieval from 14 seconds to 4.1 on CPU, the rerank step itself from 5.6 to 1.1. An early load test dropped 2 of 10 concurrent chats and killed the process at 10.4 GB of memory; after an inference semaphore and batch chunking, the same test served 10 of 10 at a 6.8 GB peak with zero restarts. In production since June 2026.

Models are chosen by an automated evaluation suite that runs through the real chat path, the same code a visitor’s question takes, and stays in the codebase as a regression test. At GoSoft we also use AI agents internally, in software development and day-to-day work.