Building an Internal AI Assistant on Your Company Docs
An internal AI assistant answers your team's questions from your own documents: policies, runbooks, product specs, past proposals, support macros. When it works, new hires stop pinging senior staff for the same answers. When it does not, it confidently quotes last year's pricing to a customer.
The difference is not the model. It is retrieval quality, permissions, freshness, and testing. This guide covers how to get each right and where to put the assistant so people actually use it.
RAG, done right
Retrieval-augmented generation (RAG) means the assistant searches your documents for relevant passages, then answers using only those passages. The search step decides most of the quality. If it retrieves the wrong paragraph, the best model in the world will still give a wrong answer.
Good retrieval combines keyword search (for exact terms like product codes and policy names) with semantic search (for questions phrased differently from the docs). Documents should be split on their natural structure (headings, sections, table rows) rather than arbitrary chunks, and each chunk should carry its title, source, owner, and last-updated date.
Every answer should cite its sources with links. Citations let people check, and they let you see which documents the assistant relies on. If the assistant cannot find support for an answer, it should say so instead of guessing.
Permissions are not optional
The assistant must never show someone a document they could not open themselves. That means carrying permissions from the source system (Google Drive, SharePoint, Confluence, Notion) into the search index and filtering results by the asking user's identity.
The shortcut of indexing only "public" docs works for a pilot, but it usually means HR, finance, and leadership content is either excluded (so the assistant is less useful) or included unsafely. Plan for permission-aware retrieval from the start, and test it with accounts at different access levels.
Keeping answers fresh
Stale answers kill trust faster than wrong ones. Sync sources on a schedule, and re-index documents when they change rather than rebuilding everything. Remove deleted documents from the index promptly.
Prefer the newest version when two documents disagree, and show the last-updated date next to each citation. Give document owners a simple way to flag that an answer is out of date, and route those flags to the person who owns the source.
Want one assistant that knows your business?
Answers from your docs and tools, with citations and permissions. From $7,500.
Evals: how you know it works
Before launch, write a test set of real questions your team asks, with the correct answer and the document it should come from. Fifty well-chosen questions is a good start. Run the assistant against them after every change to prompts, retrieval, or sources.
Score two things separately: did it retrieve the right source, and did it answer correctly from that source. When an answer is wrong, this split tells you whether to fix search or the prompt. After launch, add the questions people mark as unhelpful to the test set.
What it costs to build and run
Running cost has three parts: model usage per question, embedding and storage for the index, and hosting. For most small and mid-size teams, model usage per question is modest, but it grows with long documents and long conversations. Measure cost per answer during the pilot and set a monthly budget alert.
Our Internal AI Assistant starts from $7,500 and takes 2 to 3 weeks. That covers connecting your sources, permission-aware retrieval, the evaluation set, tool actions where they help (for example, looking up an order or creating a ticket), and deployment in Slack. Model and hosting costs run on your accounts.
Slack or Teams: meet people where they work
A separate web app gets used for a week. An assistant in Slack or Microsoft Teams gets used every day, because people already ask questions there. Support direct messages and mentions in channels, and keep answers short with citations.
Start with one team that has a clear pain (support, sales, or operations), measure how often answers are marked helpful, then expand. If your company is under 15 people and your docs fit in one shared folder, a built-in AI search in your existing tools may be enough. Try that first.
Key takeaways
- Retrieval quality decides answer quality. Combine keyword and semantic search, and always cite sources.
- Carry document permissions into the index. Never show users content they cannot open.
- Freshness and a written eval set keep trust high after launch.
- Deploy in Slack or Teams, start with one team, and measure helpfulness.
Frequently asked questions
Ours starts from $7,500 and takes 2 to 3 weeks. Model and hosting costs run on your accounts.
Not if retrieval is permission-aware. Results are filtered by what the asking user can access in the source system.
Yes. Slack is our default deployment. Teams is supported with the same retrieval and permission model.
Answer only from retrieved passages, cite sources, refuse when nothing relevant is found, and run an eval set after every change.
The 15-Day AI MVP Playbook
Day-by-day plan, a starter CLAUDE.md, the Lovable-to-production checklist, eval and cost worksheets, and 10 prompts. Pure value, no pitch.
Book 20 minutes, or send a brief
A former CTO replies. You leave with a written plan and a fixed estimate.
Prefer LinkedIn? Connect with Harish.