Add a custom server

Overview

The AI chat provides your own server for the chat to interact with. It allows for both streaming and non-streamed results, or a mixture of both.

Creating your custom messaging server

The AI chat provides a MessageRequest when someone sends a message or when you use instance.send. The AI chat expects a MessageResponse to be returned. You can stream the MessageResponse. See instance.messaging.addMessageChunk for an explanation of the streaming format.

Inside the MessageResponse the AI chat can accept response_types. You can navigate to the properties for each response_type by visiting the base GenericItem type.

The AI chat takes custom messaging server configuration part of its configuration options. You are required to provide a messaging.customSendMessage function that the AI chat calls any time the user sends a message. It also gets called if you make use of instance.send.

In this function, the AI chat passes three parameters:

1. MessageRequest: The message being sent.
2. MessageRequestOptions: Options about that message. It includes a signal property by using AbortController to allow you to cancel or stop generating a response.
3. instance: The AI chat instance object.

This function can return nothing or it can return a promise object. If you return a promise object, the AI chat does the following actions:

1. Set up a message queue and only pass the next message to your function when the message completes.
2. Show a loading indicator if the message is taking a while to return (or return its first chunk if streaming).
3. Throw a visible error and pass an abort signal if the waiting for the message crosses the messaging.messageTimeoutSecs timeout. The messageTimeoutSecs property is only set to 40 seconds by default. For your use case you can raise this number to any value.

If you do not return a promise object, the AI chat does not queue messages for you. You must use instance.updateIsTypingCounter to manage if a loading indicator is shown and the AI chat also does not do any timeout handling.

By default, the AI chat sends a MessageRequest with input.text set to a blank string and history.is_welcome_request set to true when a user first opens the chat. It is to allow you to inject a hard coded greeting response to the user. If you do not wish to use this functionality, you can set messaging.skipWelcome to false.

Finally, the MessageRequest includes a context object. By default, the AI chat keeps that in state and manually merges any new properties received in a MessageResponse with that local state. If you want to handle statefulness of context on your own, you can set messaging.copyPreviousContextOnRequest to false.

Once you call messaging.customSendMessage, you need to feed responses back into the chat. Use of the instance passed into the function for this purpose.

Additional resources

For streaming operations see instance.messaging.addMessageChunk. For non-streaming responses see instance.messaging.addMessage. Your assistant can return responses in either format.

You can also call instance.messaging.removeMessages to remove messages you have added.

Creating your custom history store

Your history store returns an array of HistoryItem members. There is currently no recommended strategy for storing your LLM friendly history, if that is part of your use case.

The AI chat allows you to define a messaging.customLoadHistory function in your configuration options. This method returns a promise that resolves when you have finished loading messages into the chat. During the AI chat's hydration process, you can call this function with the AI chat instance object as its only parameter. You can then call instance.messaging.insertHistory to load a conversation into the chat.

Some use cases can have more the one conversation attached to the chat. You can optionally forgo by using messaging.customLoadHistory and directly call instance.messaging.insertHistory as needed. In a use case where a user can change between different conversations, you can to call instance.messaging.clearConversation to clear out the previous conversation before calling instance.messaging.insertHistory.

The AI chat also supports the ability to save state on your user_defined response types in your history store if you implement UpdateHistoryEvent in your custom messaging server and use it to update your custom history store. See instance.updateHistoryUserDefined.