Log inSign up freeBook a demo
Tines Stories
  1. Docs
  2. Self-Hosted
  3. Configuring Tines
  1. Docs
  2. Self-Hosted
  3. Configuring Tines

Local AI Models

Last updated

This page is for running Tines Stories with a local model: a model you host on your own hardware and serve with a tool such as llama.cpp, vLLM, or Ollama. Teams often do this to keep AI traffic inside their own network, or when an air-gapped deployment can't reach a hosted provider.

Tines AI features, including AI Agent actions, automatic mode, and formula generation, rely on the model you connect. With a local model, you control the settings that decide how well those features work. This page covers what to check before you connect one.

If you use a hosted provider such as OpenAI, Anthropic, or Amazon Bedrock, see AI provider configuration instead.

Check these five settings for a local model:

  • Model capability. On the Artificial Analysis coding index, aim for about 25 or higher for the smart model.

  • Context size. Set at least 200,000 tokens in the server launch command.

  • Reasoning. Turn thinking on for the smart model. Some servers leave it off.

  • Effort. Higher effort takes longer. none turns thinking off.

  • Quantization. Q4_K_XL or Q4_K_M is the smallest size we recommend. Use Q8_0 when the GPU has room.

Model capability

We compare models with the Artificial Analysis coding index. The open-weights view lists models you can run yourself. Read the score on the day you choose. The index is updated, and older cutoffs stop matching the chart.

The smart model is used for AI Agent actions with tools, formula generation, and automatic mode. On Tines Cloud it also powers Workbench. Set it in AI settings. A model selected on a Workbench conversation or an AI Agent action overrides that default.

The fast model is used for automatic transform, simple AI Agent runs, and generated names. It can score lower. If you run one model, use it as the smart model.

If the model is not on the chart, test it on a story you know before you roll it out.

Context size

Set the server context length to at least 200,000 tokens. Use the value in the launch command. A model started at 8,000 or 16,000 tokens runs at that smaller size.

Confirm the context fits in GPU memory at the quantization you serve. The context cache uses more memory than the model weights.

Reasoning and effort

Turn thinking on for the smart model. On llama.cpp, set enable_thinking in --chat-template-kwargs when the model uses that switch.

An effort of none turns thinking off. If the server rejects reasoning_effort, check Omit reasoning effort under Extra options and leave thinking on at the server.

Quantization

We recommend Q4_K_XL or Q4_K_M at a minimum. Use Q8_0 when it fits. Below 4-bit, the coding-index score no longer describes the model you are running. A 4-bit copy on your own hardware scores a little below the published number.

Connect the server

Point Tines Stories at an OpenAI-compatible endpoint that supports streaming and tool use. Set API type under Extra options. See AI for providers, custom models, and credentials.

  • The server must complete a chain of tool calls. If Tines Stories does not recognize the format, set Function calling mode to Hermes.

  • The model id must match the name the server advertises. Add it as a custom model if the server has no models list.

  • Set the context length in the launch command.

Check whether a model fits on your GPU at llmrun.dev or with the APXML VRAM calculator. Run the strongest model that fits at Q4_K_XL or better with a 200,000-token context.

Was this helpful?