diff --git a/content/en/docs/genai/mendix-cloud-genai/_index.md b/content/en/docs/genai/mendix-cloud-genai/_index.md index 7054d8b315e..9cf554b59ee 100644 --- a/content/en/docs/genai/mendix-cloud-genai/_index.md +++ b/content/en/docs/genai/mendix-cloud-genai/_index.md @@ -26,7 +26,7 @@ There are three types of resources: ## Getting Started 1. Learn about GenAI Resource Packs and how to acquire them in the [Mendix Cloud GenAI Resource Packs](/agents/mx-cloud-genai/resource-packs/) document. -2. Once you have access to GenAI resources, log in to the [Mendix Cloud GenAI portal](https://genai.home.mendix.com/) to generate access keys for your resources. This portal provides an overview of all resources you have access to and enables you to request new GenAI resources. For more information, see [Navigate through the Mendix Cloud GenAI Portal](/agents/mx-cloud-genai/Navigate-MxGenAI/). +2. Once you have access to GenAI resources, sign in to the [Mendix Cloud GenAI portal](https://genai.home.mendix.com/) to generate access keys for your resources. This portal provides an overview of all resources you have access to and enables you to request new GenAI resources. For more information, see [Navigate Through the Mendix Cloud GenAI Portal](/agents/mx-cloud-genai/Navigate-MxGenAI/). 3. Use a starter app containing the [Mendix Cloud GenAI Connector](https://marketplace.mendix.com/link/component/239449) (for example, the [BlankGenAI starter app](https://marketplace.mendix.com/link/component/227934)) or implement the connector in your Mendix app. For more information on configuration and usage, see [Mendix Cloud GenAI Connector](/agents/mx-cloud-genai/mxgenai-connector/). Once you have imported an access key in the configuration overview, you are connected to Mendix Cloud GenAI and can access available resources within your application. ## Relevant Sources diff --git a/content/en/docs/genai/mendix-cloud-genai/mendix-cloud-grp.md b/content/en/docs/genai/mendix-cloud-genai/mendix-cloud-grp.md index 2f31ffcfd76..063f49e16e2 100644 --- a/content/en/docs/genai/mendix-cloud-genai/mendix-cloud-grp.md +++ b/content/en/docs/genai/mendix-cloud-genai/mendix-cloud-grp.md @@ -2,7 +2,7 @@ title: "Mendix Cloud GenAI Resource Packs" url: /agents/mx-cloud-genai/resource-packs linktitle: "Mendix Cloud GenAI Resource Packs" -description: "Provides an overview of Mendix Cloud GenAI Resource Packs, including their capabilities, limitations, and frequently asked questions (FAQ)" +description: "Provides an overview of Mendix Cloud GenAI Resource Packs, including their capabilities, limitations, and frequently asked questions." weight: 10 aliases: - /appstore/modules/genai/mx-cloud-genai/resource-packs @@ -21,6 +21,7 @@ Developers can use the Mendix Portal to manage their Mendix Cloud GenAI resource ### General Availability Mendix Cloud GenAI Resource Packs is a premium Mendix product that requires an additional purchase. To start using GenAI Resource Packs or inquire about pricing, contact your Customer Success Manager (CSM). For more information, you can also reach out to [genai-resource-packs@mendix.com](mailto:genai-resource-packs@mendix.com). + GenAI Resource Packs can be purchased using Mendix Cloud Tokens. For details around costs, refer to [Cloud Tokens](/control-center/cloud-tokens/). ## Models @@ -44,7 +45,7 @@ The models are available through the Mendix Cloud, leveraging AWS's highly secur Customers looking to leverage other models in addition to the above can also take advantage of Mendix's [(Azure) OpenAI Connector](/agents/reference-guide/external-connectors/openai/), Amazon [Bedrock Connector](/agents/reference-guide/external-connectors/bedrock/), and [Mistral Connector](/agents/reference-guide/external-connectors/mistral/) to integrate numerous other models into their apps. {{% alert color="info" %}} -Additional regions will be available in the future. If you have questions about upcoming regions, or would like to explore making models available in your specific region, reach out to `genai-resource-packs@mendix.com`. +Additional regions will be available in the future. If you have questions about upcoming regions or want to explore making models available in your specific region, reach out to `genai-resource-packs@mendix.com`. {{% /alert %}} ### Technical Details for Model Resource Packs @@ -58,8 +59,11 @@ Additional regions will be available in the future. If you have questions about ## Accessing GenAI Resources Developers can easily obtain access to GenAI resources through a self-service capability, enabling them to access and manage GenAI resources independently. + Developers with the required prerequisites can use the self-service capability to provision, deprovision, and manage GenAI resources directly from the Control Center. This enables faster provisioning and reduces manual dependency. -For developers who do not have self-service capabilities, GenAI resources can still be provisioned or deprovisioned by contacting sales representatives or customer success manager (CSM) to order an existing stock keeping unit (SKU). + +For developers who do not have self-service capabilities, GenAI resources can still be provisioned or deprovisioned by contacting sales representatives or a Customer Success Manager (CSM) to order an existing stock keeping unit (SKU). + Both approaches allow users to scale GenAI resources efficiently and explore more generative AI solutions with Mendix. ### Provisioning GenAI Resources Using the Self-Service Capability @@ -75,11 +79,11 @@ For further details, refer to the [Prerequisites](/control-center/genai-resource ### Provisioning GenAI Resources Without Using the Self-Service Capability -If the self-service capability is not available in your environment, you can still provision your GenAI resources by ordering the existing SKU associated to your Mendix subscription. To do so, you can contact your sales representative or CSM. +If the self-service capability is not available in your environment, you can still provision your GenAI resources by ordering the existing SKU associated with your Mendix subscription. To do so, you can contact your sales representative or CSM. ## Knowledge Bases -Mendix Cloud Knowledge Base Resource Packs provide customers with an elastic, logically isolated vector database, to use for standard Generative AI architectural patterns such as Retrieval-Augmented Generation (RAG), semantic similarity search, and other Generative AI use cases. The Knowledge Bases on Mendix Cloud are based on AWS's highly secure Amazon Bedrock Knowledge Bases capability, combined with AWS' OpenSearch Serverless database— a widely adopted standard infrastructure for Generative AI Knowledge Bases on AWS, ensuring fast & accurate information retrieval. +Mendix Cloud Knowledge Base Resource Packs provide customers with an elastic, logically isolated vector database to use for standard Generative AI architectural patterns such as Retrieval-Augmented Generation (RAG), semantic similarity search, and other Generative AI use cases. The Knowledge Bases on Mendix Cloud are based on AWS's highly secure Amazon Bedrock Knowledge Bases capability, combined with AWS' OpenSearch Serverless database—a widely adopted standard infrastructure for Generative AI Knowledge Bases on AWS, ensuring fast and accurate information retrieval. Knowledge bases enable you to bring your own data for RAG, semantic similarity search, and other generative AI use cases: @@ -116,7 +120,7 @@ Save these links for future reference. Always review the terms before starting d Compliance with these terms is mandatory to maintain access to the services. {{% /alert %}} -## More resources +## More Resources ### Mendix Cloud GenAI Portal @@ -128,7 +132,7 @@ The [Mendix Cloud GenAI Portal](https://genai.home.mendix.com/) allows easy acce * Create and manage connection keys to connect your apps with all resources. * Track activity logs for team access and connection key management. -For more information, see [Navigate through the Mendix Cloud GenAI Portal](/agents/mx-cloud-genai/Navigate-MxGenAI/). +For more information, see [Navigate Through the Mendix Cloud GenAI Portal](/agents/mx-cloud-genai/Navigate-MxGenAI/). ### Mendix Cloud GenAI Connector @@ -146,7 +150,7 @@ For Mendix Cloud GenAI Model Resources using Anthropic’s Claude and Cohere’s Data stored in GenAI Knowledge Base Resources resides in a logically isolated database, accessible only to you—the customer—via keys you can generate in the Portal. -### How does Mendix Cloud GenAI service Store and Use Data Sent to It? +### How Does the Mendix Cloud GenAI Service Store and Use Data Sent to It? Requests (prompts) sent to and responses (answers, embeddings) received from the models are not stored and not used for training. Only metadata—such as token input/output counts—is collected for logging, monitoring, metering, billing, product improvement, and maintenance purposes. diff --git a/content/en/docs/genai/mendix-cloud-genai/navigate_mxgenai.md b/content/en/docs/genai/mendix-cloud-genai/navigate_mxgenai.md index 4d5914e3284..66419c0a2dd 100644 --- a/content/en/docs/genai/mendix-cloud-genai/navigate_mxgenai.md +++ b/content/en/docs/genai/mendix-cloud-genai/navigate_mxgenai.md @@ -1,8 +1,8 @@ --- -title: "Navigate through the Mendix Cloud GenAI Portal" +title: "Navigate Through the Mendix Cloud GenAI Portal" url: /agents/mx-cloud-genai/Navigate-MxGenAI/ linktitle: "Mendix Cloud GenAI Portal" -description: "Describes how to navigate through the Mendix Cloud GenAI Portal." +description: "Describes how to navigate through the Mendix Cloud GenAI portal." weight: 30 aliases: - /appstore/modules/genai/mx-cloud-genai/Navigate-MxGenAI/ @@ -10,11 +10,11 @@ aliases: ## Introduction -The [Mendix Cloud GenAI portal](https://genai.home.mendix.com/) is the part of the Mendix portal that provides access to [Mendix Cloud GenAI Resource Packs](/agents/mx-cloud-genai/resource-packs/). After logging in, you can navigate to the overview of all resources. You can see all resources, that you are a team member of and access their details. +The [Mendix Cloud GenAI portal](https://genai.home.mendix.com/) is the part of the Mendix portal that provides access to [Mendix Cloud GenAI Resource Packs](/agents/mx-cloud-genai/resource-packs/). After logging in, you can navigate to the overview of all resources. You can see all resources that you are a team member of and access their details. ## Resource Details -After clicking on a specific resource, you land on its details page, offering shortcut to consumption insights, key generation, team management, and helpful documentation. +After clicking a specific resource, you land on its details page, offering a shortcut to consumption insights, key generation, team management, and helpful documentation. {{< figure src="/attachments/genai/navigate_mxgenai/GenAIResource_Details.png" alt="" >}} @@ -32,7 +32,7 @@ The **Settings** tab contains the details of a GenAI resource. It shows the foll * **Plan**: indicates the subscription plan used for compute resources (for example, embedding or text generation resources). * **Environment**: shows which environment is used, for example, test, acceptance, or production. -¹ Cross-region inference (CRI) allows a model to redirect requests to another region, helping to distribute the load across multiple regions within the same area. So, EU requests always stay within EU regions. Connecting to a cross-region inference profile does not change how the request is sent; the redirection happens on the server side, determining the region to handle the request to get the fastest response. For more information, see [Increase throughput with cross-Region inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html). If applicable, CRI profiles are selected during provisioning of a model resource. New models are available under the CRI inferencing type by default. +¹ Cross-region inference (CRI) allows a model to redirect requests to another region, helping to distribute the load across multiple regions within the same area. EU requests always stay within EU regions. Connecting to a cross-region inference profile does not change how the request is sent; the redirection happens on the server side, determining the region to handle the request to get the fastest response. For more information, see [Increase throughput with cross-region inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html). If applicable, CRI profiles are selected during provisioning of a model resource. New models are available under the CRI inferencing type by default. #### Additional Details for Knowledge Base Resources @@ -70,7 +70,7 @@ You can invite members from outside your organization to access your GenAI resou You can track invitations in the **Pending Invites** tab. Invited users will receive an email with a link to accept or decline the invitation. If they do not yet have a Mendix account, the link redirects them to create one. Once the invitation is accepted, the resource will appear in their GenAI portal overview. -Pending invitations can be withdrawn at any time and will automatically expire after two weeks. External members can create and delete keys, export consumption data, manage knowledge base content and collections, and change the model. However, they can not modify the display name or environment, or manage team membership. +Pending invitations can be withdrawn at any time and will automatically expire after two weeks. External members can create and delete keys, export consumption data, manage knowledge base content and collections, and change the model. However, they cannot modify the display name or environment, or manage team membership. ### Keys @@ -86,7 +86,7 @@ Make sure to store it securely, as it will only be shown once. {{< figure src="/attachments/genai/navigate_mxgenai/GenAIResource_KeyGeneration.png" alt="" >}} -Once created, the key can be used in the Mendix application via the Mendix Cloud GenAI Connector. +Once created, the key can be used in the Mendix application via the Mendix Cloud GenAI connector. #### Additional Information for Knowledge Base Resource Keys @@ -143,7 +143,7 @@ Instead of relying solely on similarity-based searches of ticket descriptions, u #### Add Data from a Mendix Application -You can upload data directly from Mendix to the Knowledge Base. To do so, several operations of the Mendix Cloud GenAI Connector are required. For a detailed guide on this process, see the [Add Data Chunks to Your Knowledge Base](/agents/mx-cloud-genai/mxgenai-connector/#add-data-chunks-to-your-knowledge-base) section of **Mendix Cloud GenAI Connector**. +You can upload data directly from Mendix to the Knowledge Base. To do so, several operations of the Mendix Cloud GenAI connector are required. For a detailed guide on this process, see the [Add Data Chunks to Your Knowledge Base](/agents/mx-cloud-genai/mxgenai-connector/#add-data-chunks-to-your-knowledge-base) section of *Mendix Cloud GenAI Connector*. ### Consumption (Only for Text and Embeddings Generation Resources) diff --git a/content/en/docs/genai/reference-guide/agent-editor.md b/content/en/docs/genai/reference-guide/agent-editor.md index 2c7f0b51242..a32b2db5432 100644 --- a/content/en/docs/genai/reference-guide/agent-editor.md +++ b/content/en/docs/genai/reference-guide/agent-editor.md @@ -14,7 +14,7 @@ The [Agent Editor](https://marketplace.mendix.com/link/component/257918) module With the Agent Editor module, you can define agents at design time in Studio Pro (11.9.0 and above) and manage their lifecycle as part of your app by leveraging existing platform capabilities such as Model documents, version control, and deployment capabilities. Define and develop agents locally, then deploy them directly to cloud environments using the app model. -Agent Editor is compatible with the Agent Commons module. Using this module, you can define and manage prompts, microflows (as tools), external MCP servers, knowledge bases, and large language models to build agentic patterns that support your business logic. Additionally, you can define variables that act as placeholders for data from the app session context. These placeholders are replaced with actual values when the end-user interacts with the app. +Agent Editor is compatible with the Agent Commons module. Using this module, you can define and manage prompts, microflows (as tools), external Model Context Protocol (MCP) servers, knowledge bases, and large language models to build agentic patterns that support your business logic. Additionally, you can define variables that act as placeholders for data from the app session context. These placeholders are replaced with actual values when the end-user interacts with the app. The Agent Editor module includes a Studio Pro extension that you can use to define GenAI agents as documents in the app model. The Agent Editor Commons module, which is installed as part of the same package, includes logic and activities to call these agents from microflows in a running app. @@ -50,7 +50,7 @@ Agent Editor provides the following features: The Agent Editor module requires Mendix Studio Pro version 11.9.0 or above. -The following module are required dependencies for the supported capabilities of Agent Editor and need to be installed: +The following modules are required dependencies for the supported capabilities of Agent Editor and need to be installed: * [Administration](https://marketplace.mendix.com/link/component/23513) * [Agent Commons](https://marketplace.mendix.com/link/component/240371) @@ -196,7 +196,7 @@ In the knowledge base entry: * Select the configured knowledge base document in the **Knowledge base**. * In **Collection**, select one of the available collections from the dropdown. Alternatively, type or paste a collection name to reference a collection that does not exist yet. -* Provide **Name** and **Description** so the LLM can determine when this knowledge base should be used. This serves the same purpose as naming tools. +* Provide **Name** and **Description** so the LLM can determine when to use this knowledge base. This serves the same purpose as naming tools. * Optionally configure retrieval settings: * **Max results** controls the maximum number of chunks returned in a single retrieval. * **Min similarity** sets the cosine-similarity threshold between 0 and 1. Higher values (for example, 0.8) are stricter than lower values (for example, 0.2). @@ -237,9 +237,7 @@ For **Call Agent without History**, you can optionally pass a `Request` object t The output is a `GenAICommons.Response` object, aligned with the GenAI Commons and Agent Commons domain models and actions. You can use this object for further logic. Additionally, all agents created via the Agent Editor extension are integrated with other Mendix offerings, such as the [Token consumption monitor](/agents/genai-for-mx/conversational-ui/#snippet-token-monitor) or the [Traceability](/agents/genai-for-mx/conversational-ui/#traceability) feature from [ConversationalUI](/agents/genai-for-mx/conversational-ui/). -### Including the Agent in Workflows - -Agents can also be invoked from workflows using the [AI Agent Task](/refguide/ai-agent-task/) element. The AI Agent Task calls a microflow that you configure. Inside that microflow, use one of the **Call Agent** toolbox actions described above to execute the agent and return the result to the workflow. +You can also invoke agents from workflows using the [AI Agent Task](/refguide/ai-agent-task/) element. The AI Agent Task calls a microflow that you configure. Inside that microflow, use one of the **Call Agent** toolbox actions described above to call the agent and return the result to the workflow. ### Including the Agent in a Conversational User Interface {#conversational-ui} @@ -262,7 +260,7 @@ Use version control to view and restore previous agent versions. This lets you i ## Known Limitations {#limitations} * Currently, Agent Editor supports only Mendix Cloud GenAI as a provider for text generation models and knowledge bases. Support for other providers, such as (Azure) OpenAI and Amazon Bedrock, is planned for a future release. -* Support for Mac users is limited. Some functionalities might not work, such as doing a test call for Model documents. Mendix recommends using Studio Pro on Windows to use all features of Agent Editor smoothly. +* Support for Mac users is limited. Some functionalities might not work, such as doing a test call for Model documents. Mendix recommends using Studio Pro on Windows to use all features of Agent Editor. * MCP tool support is limited to whole-server integration. Selecting individual tools from a consumed MCP service to be added to an agent is not yet supported. That also means that the tool choice option `Tool` can only refer to a microflow tool currently. * If a document referenced by an Agent document is excluded, Studio Pro shows a consistency error. These consistency errors may not be resolved automatically when you include the excluded document again. Resolve this by synchronizing the app directory (F4) or by making a small change in any agent-related document (for example, add a character to a system prompt and remove it again). * The extension creates a `/agenteditor` log folder in the app directory. This folder is not excluded from version control automatically when you include the module from Marketplace. Add this folder to `.gitignore` manually, as described in the [First-time setup](#setup) section. @@ -283,7 +281,7 @@ This is often caused by validations executed in the after-startup logic. Ensure ### Errors Pane Shows “Extension Agent-Editor Failed To Complete Its Consistency Checks” -This is a known issue caused by internal timeouts. It is more likely to occur if there are many Agent documents as part of the project. Resolve this by synchronizing the project directory (F4), running the project locally, or by making a small change in any agent-related document (for example, add a character to a system prompt and remove it again). If it happens very frequently, contact Mendix Support. +This is a known issue caused by internal timeouts. It is more likely to occur if there are many Agent documents as part of the project. Resolve this by synchronizing the project directory (F4), running the project locally, or by making a small change in any agent-related document (for example, add a character to a system prompt and remove it again). If it happens frequently, contact Mendix Support. ### Agent Documents Are Not Visible in Agent Commons UI diff --git a/content/en/docs/genai/reference-guide/external-platforms/mistral.md b/content/en/docs/genai/reference-guide/external-platforms/mistral.md index 6d9d1edb52c..b2df85b802a 100644 --- a/content/en/docs/genai/reference-guide/external-platforms/mistral.md +++ b/content/en/docs/genai/reference-guide/external-platforms/mistral.md @@ -2,7 +2,7 @@ title: "Mistral" url: /agents/reference-guide/external-connectors/mistral/ linktitle: "Mistral" -description: "Describes the configuration and usage of the Mistral Connector, which allows you to integrate generative AI into your Mendix app." +description: "Describes how to configure and use the Mistral connector to integrate generative AI capabilities into Mendix apps." weight: 20 aliases: - /appstore/modules/genai/reference-guide/external-connectors/mistral/ @@ -10,21 +10,21 @@ aliases: ## Introduction -The [Mistral Connector](https://marketplace.mendix.com/link/component/248276) allows you to integrate generative AI capabilities into your Mendix application. Since the Mistral API is compatible with [OpenAI API](https://platform.openai.com/), this module mainly focuses on Mistral specific UI while reusing the operations inside of the OpenAI connector. +The [Mistral connector](https://marketplace.mendix.com/link/component/248276) lets you integrate generative AI capabilities into Mendix apps. Because the Mistral API is compatible with the [OpenAI API](https://platform.openai.com/), this module focuses on Mistral-specific UI while reusing operations from the OpenAI connector. ### Features {#features} -The Mistral Connector is commonly used for text generation based on the [Chat Completions API](https://docs.mistral.ai/api/endpoint/chat) and embeddings generation with the [Embeddings API](https://docs.mistral.ai/api/endpoint/embeddings). Typical use cases for generative AI are described in the [Typical LLM Use Cases](/agents/get-started/#llm-use-cases). +The Mistral connector is commonly used for text generation based on the [Chat Completions API](https://docs.mistral.ai/api/endpoint/chat) and embeddings generation with the [Embeddings API](https://docs.mistral.ai/api/endpoint/embeddings). -For more information about the models, see [Mistral models](https://docs.mistral.ai/getting-started/models). +For more information about the available models, see [Mistral models](https://docs.mistral.ai/getting-started/models). #### Image Generation {#use-cases-images} -Mistral does not currently offer image generation models out of the box. It is possible to equip a Mistral agent with an image generation tool (see [Image generation](https://docs.mistral.ai/agents/connectors/image_generation/)), however, this functionality is not supported by the Mistral Connector. +Mistral does not offer image generation models out of the box. You can equip a Mistral agent with an image generation tool (see [Image generation](https://docs.mistral.ai/agents/connectors/image_generation/)), but the Mistral connector does not support this functionality. #### Knowledge Base -The Mistral connector supports Knowledge bases from providers such as pgVector, Mendix Cloud, Amazon Bedrock, and Azure AI Search to be added to a conversation. +The Mistral connector supports knowledge bases from providers such as pgVector, Mendix Cloud, Amazon Bedrock, and Azure AI Search. ### Prerequisites @@ -32,7 +32,7 @@ To use this connector, you need to sign up for a Mistral account and create an A ### Dependencies {#dependencies} -* Mendix Studio Pro version 10.24.0 or above +* Mendix Studio Pro 10.24.0 and above * [GenAI Commons module](/agents/genai-for-mx/commons/) * [Encryption module](/appstore/modules/encryption/) * [Community Commons module](/appstore/modules/community-commons-function-library/) @@ -40,20 +40,20 @@ To use this connector, you need to sign up for a Mistral account and create an A ## Installation -Install all required modules from the Mendix Marketplace as listed in the [Dependencies](#dependencies) section above. +Install all required modules from Mendix Marketplace as listed in the [Dependencies](#dependencies) section above. -To import the [Mistral Connector](https://marketplace.mendix.com/link/component/248276) and the other modules into your app, follow the instructions in [Using Marketplace Content](/appstore/use-content/). +To import the [Mistral connector](https://marketplace.mendix.com/link/component/248276) and the other modules into your app, follow the instructions in [Using Marketplace Content](/appstore/use-content/). ## Configuration {#configuration} -After you install the Mistral and OpenAI connector, you can find them in the **Marketplace Modules** section of the **App Explorer**. The Mistral connector provides a domain model and several pages. You can reuse all activities to connect your app to Mistral from the OpenAI connector. To implement an activity, use it in a microflow. Configure the [Encryption module](/appstore/modules/encryption/#configuration) to ensure the connection of your app to Mistral is secure. +After you install the Mistral and OpenAI connectors, you can find them in the **Marketplace Modules** section of the **App Explorer**. The Mistral connector provides a domain model and several pages. You can reuse all activities to connect your app to Mistral from the OpenAI connector. To implement an activity, use it in a microflow. Configure the [Encryption module](/appstore/modules/encryption/#configuration) to secure the connection between your app and Mistral. ### General Configuration {#general-configuration} -1. Add the module roles `OpenAIConnector.Administrator` and `MistralConnector.Administrator` to your Administrator **User roles** in the **Security** settings of your app. -2. Add the **MistralConfiguration_Overview** page from the Mistral connector module (**USE_ME > MistralConfiguration**) to your navigation, or add the `Snippet_MistralConfigurations` to a page that is already part of your navigation. -3. Continue setting up your Mistral configuration at runtime. For more information, follow the instructions in the [Mistral Configuration](#mistral-configuration) section below. -4. Configure the models you need for your use case. +1. Add the module roles `OpenAIConnector.Administrator` and `MistralConnector.Administrator` to your Administrator user role in the **Security** settings of your app. +2. Add the **MistralConfiguration_Overview** page from the Mistral connector module (**USE_ME > MistralConfiguration**) to your navigation, or add `Snippet_MistralConfigurations` to a page that is already part of your navigation. +3. Continue setting up your Mistral configuration at runtime. For more information, see the [Mistral Configuration](#mistral-configuration) section below. +4. Configure the models for your use case. #### Mistral Configuration {#mistral-configuration} @@ -61,138 +61,139 @@ The following inputs are required for the Mistral configuration: | Parameter | Value | | ----------- | ------------------------------------------------------------ | -| Display name | This is the name identifier of a configuration (for example, *MyConfiguration*). | -| Endpoint | This is the API endpoint (for example, `https://api.mistral.ai/v1/`) | -| Token | This is the access token to authorize your API call.
To get an API key, follow the steps mentioned in the [Quickstart](https://docs.mistral.ai/getting-started/quickstart). | +| Display name | The name identifier of a configuration (for example, *MyConfiguration*). | +| Endpoint | The API endpoint (for example, `https://api.mistral.ai/v1/`). | +| Token | The access token to authorize your API call.
To get an API key, see the [Quickstart](https://docs.mistral.ai/getting-started/quickstart). | #### Configuring the Mistral Deployed Models -A [Deployed Model](/agents/genai-for-mx/commons/#deployed-model) represents a GenAI model instance that can be used by the app to generate text, embeddings, or images. For every model you want to invoke from your app, you need to create a `MistralDeployedModel` record, a specialization of `DeployedModel` (and also a specialization of `OpenAIDeployedModel`). In addition to the model display name and a technical name or identifier, a Mistral deployed model contains a reference to the additional connection details as configured in the previous step. +A [deployed model](/agents/genai-for-mx/commons/#deployed-model) represents a GenAI model instance that the app can use to generate text, embeddings, or images. For each model you want to invoke from your app, create a `MistralDeployedModel` record—a specialization of `DeployedModel` (and also a specialization of `OpenAIDeployedModel`). In addition to the model display name and a technical name or identifier, a Mistral deployed model contains a reference to the connection details configured in the previous step. -1. Click the three dots ({{% icon name="three-dots-menu-horizontal" %}}) icon for a Mistral configuration and open **Manage Deployed Models**. It is possible to use a predefined syncing method, where all available models are retrieved for the specified API key and then filtered according to their capabilities. If you want to use additional models that are made available by Mistral you can add them manually by clicking the **New** button instead. -2. For every additional model, add a record. The following fields are required: +1. Click the three dots ({{% icon name="three-dots-menu-horizontal" %}}) icon for a Mistral configuration and open **Manage Deployed Models**. You can use a predefined syncing method that retrieves all available models for the specified API key and filters them according to their capabilities. To use additional models made available by Mistral, add them manually by clicking **New**. +2. For each additional model, add a record. The following fields are required: | Field | Description | | -------------- | ------------------------------------------------------------ | - | Display name | This is the reference for app users when selecting the appropriate model to use. | - | Model name | This is the technical reference of the model. For Mistral, this is equal to the [model IDs](https://docs.mistral.ai/getting-started/models), for example `mistral-medium-2508`. | - | Output modality | Describes the output of the model. This connector currently supports text, embedding, and image. | - | Input modality| Describes the input modalities accepted by the model. This connector currently supports text and image. | + | Display name | The reference for app users when selecting the model. | + | Model name | The technical reference of the model. For Mistral, this equals the [model ID](https://docs.mistral.ai/getting-started/models) (for example, `mistral-medium-2508`). | + | Output modality | The output of the model. This connector currently supports text, embedding, and image. | + | Input modality| The input modalities accepted by the model. This connector currently supports text and image. | -3. Close the **Manage Deployed Models** popup and test the configuration with the newly created deployed models. +3. Close the **Manage Deployed Models** dialog box and test the configuration with the newly created deployed models. ### Using GenAI Commons Operations {#genai-commons-operations} -After following the general setup above, you are all set to use the microflow actions under the **GenAI (Generate)** category from the toolbox. These operations are part of GenAI Commons. Since OpenAI (and therefor Mistral) is compatible with the principles of GenAI Commons, you can pass a `MistralDeployedModel` to all GenAI Commons operations that expect the generalization of `DeployedModel`. All actions under **GenAI (Generate)** will take care of executing the right provider-specific logic, based on the type of specialization passed, in this case, Mistral. From an implementation perspective, it is not needed to required the inner workings of this operation. The input, output, and behavior are described in the [GenAICommons](/agents/genai-for-mx/commons/#microflows) documentation. Applicable operations and some Mistral-specific aspects are listed in the sections below. +After completing the general setup above, you can use the microflow actions under the **GenAI (Generate)** category in the toolbox. These operations are part of GenAI Commons. Because OpenAI (and therefore Mistral) is compatible with the principles of GenAI Commons, you can pass a `MistralDeployedModel` to all GenAI Commons operations that expect the generalization of `DeployedModel`. All actions under **GenAI (Generate)** execute the appropriate provider-specific logic based on the specialization type passed (in this case, Mistral). From an implementation perspective, understanding the inner workings of this operation is not required. The [GenAI Commons](/agents/genai-for-mx/commons/#microflows) documentation describes the input, output, and behavior. The sections below list applicable operations and Mistral-specific aspects. -For more inspiration or guidance on how to use the microflow actions in your logic, Mendix recommends downloading [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of examples that cover all the operations mentioned. +For more inspiration or guidance on how to use the microflow actions in your logic, download the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of examples covering all the operations mentioned. -You can use the GenAI Commons toolbox actions to [create the required Request](/agents/genai-for-mx/commons/#genai-request-building) and [handle the Response](/agents/genai-for-mx/commons/#genai-response-handling) for your use case. +Use the GenAI Commons toolbox actions to [create the required request](/agents/genai-for-mx/commons/#genai-request-building) and [handle the response](/agents/genai-for-mx/commons/#genai-response-handling) for your use case. -The internal chat completion logic supports [JSON mode](#chatcompletions-json-mode), [function calling](#chatcompletions-functioncalling), and [vision](#chatcompletions-vision) for Mistral. Make sure to check the actual compatibility of the available models with these functionalities, as this changes over time. The following sections list toolbox actions which are specifically for OpenAI compatible APIs (especially Mistral). +The internal chat completion logic supports [JSON mode](#chatcompletions-json-mode), [function calling](#chatcompletions-functioncalling), and [vision](#chatcompletions-vision) for Mistral. Check the compatibility of available models with these functionalities, as this changes over time. The following sections list toolbox actions specifically for OpenAI-compatible APIs (especially Mistral). #### Chat Completions -Operations for chat completions focus on the generation of text based on a certain input. In this context, system prompts and user prompts are two key components that help guide the language model in generating relevant and contextually appropriate responses. For more information on the type of prompts and message roles, see the [ENUM_MessageRole](/agents/genai-for-mx/commons/#enum-messagerole) enumeration. To learn more about how to create the right prompts for your use case, see the [Read More](#read-more) section below +Operations for chat completions focus on generating text based on input. In this context, system prompts and user prompts are two key components that guide the language model in generating relevant and contextually appropriate responses. For more information on prompt types and message roles, see the [ENUM_MessageRole](/agents/genai-for-mx/commons/#enum-messagerole) enumeration. To learn more about how to create the right prompts for your use case, see the [Read More](#read-more) section below. -The `MistralDeployedModel` is compatible with the two [Chat Completions operations from GenAI Commons](/agents/genai-for-mx/commons/#genai-generate). While developing your custom microflow, you can drag and drop the following operations from the toolbox in Studio Pro. See category **GenAI (Generate)**: +The `MistralDeployedModel` is compatible with the two [chat completions operations from GenAI Commons](/agents/genai-for-mx/commons/#genai-generate). While developing your custom microflow, drag and drop the following operations from the toolbox in Studio Pro under the **GenAI (Generate)** category: * Chat Completions (with history) * Chat Completions (without history) #### JSON Mode {#chatcompletions-json-mode} -When JSON mode is used, the model is programmatically instructed to return valid JSON. For Mistral connector, you have to explicitly mention the necessity of a JSON structure in a message in the conversation, e.g. the system prompt. Additionally, after creating the request, but before passing it to the chat completions operation, use the toolbox action `Set Response Format` to set the required response format to JSON. +JSON mode instructs the model to return valid JSON. To use JSON mode with the Mistral connector, explicitly specify that a JSON structure is required in a conversation message (for example, in the system prompt). After creating the request but before passing it to the chat completions operation, use the `Set Response Format` toolbox action to set the response format to JSON. #### Function Calling {#chatcompletions-functioncalling} -Function calling enables LLMs to connect with external tools to gather information, execute actions, convert natural language into structured data, and much more. Function calling thus enables the model to intelligently decide when to let the Mendix app call one or more predefined function microflows to gather additional information to include in the assistant's response. +Function calling enables LLMs to connect with external tools to gather information, run actions, convert natural language into structured data, and more. Function calling enables the model to intelligently decide when to let the Mendix app call one or more predefined function microflows to gather additional information for the assistant's response. -Mistral does not call the function. The model returns a tool called JSON structure that is used to build the input of the function (or functions) so that they can be executed as part of the chat completions operation. Functions in Mendix are essentially microflows that can be registered within the request to the LLM​. The OpenAI connector takes care of handling the tool call response as well as executing the function microflows until the API returns the assistant's final response for Mistral. +Mistral does not call the function. The model returns a tool called JSON structure that is used to build the input of the function (or functions) so they can run as part of the chat completions operation. Functions in Mendix are essentially microflows that can be registered within the request to the LLM. The OpenAI connector handles the tool call response and runs the function microflows until the API returns the assistant's final response for Mistral. -This is all part of the implementation that is executed by the GenAI Commons chat completions operations mentioned before. As a developer, you have to make the system aware of your functions and what these do by registering the function(s) to the request. This is done using the GenAI Commons operation [Tools: Add Function to Request](/agents/genai-for-mx/commons/#add-function-to-request) once per function before passing the request to the chat completions operation. +The GenAI Commons chat completions operations mentioned earlier run this implementation. As a developer, you must make the system aware of your functions and their purposes by registering the functions to the request. To do so, use the GenAI Commons operation [Tools: Add Function to Request](/agents/genai-for-mx/commons/#add-function-to-request) once per function before passing the request to the chat completions operation. -Function microflows can have none, a single, or multiple primitive input parameters such as Boolean, Datetime, Decimal, Enumeration, Integer or String. Additionally, they may accept the [Request](/agents/genai-for-mx/commons/#request) or [Tool](/agents/genai-for-mx/commons/#tool) objects as inputs. The function microflow must return a String value. +Function microflows can have none, one, or multiple primitive input parameters such as Boolean, Datetime, Decimal, Enumeration, Integer, or String. Additionally, they may accept the [Request](/agents/genai-for-mx/commons/#request) or [Tool](/agents/genai-for-mx/commons/#tool) objects as inputs. The function microflow must return a string value. {{% alert color="warning" %}} -Function calling is a very powerful capability and should be used with caution. Note that function microflows run in the context of the current user without enforcing entity-access. You can use `$currentUser` in XPath queries to ensure you retrieve and return only information that the end-user is allowed to view; otherwise confidential information may become visible to the current end-user in the assistant's response. +Function calling is a very powerful capability and should be used with caution. Note that function microflows run in the context of the current user without enforcing entity-access. You can use `$currentUser` in XPath queries to ensure you retrieve and return only information that the end-user is allowed to view; otherwise, confidential information may become visible to the end-user in the assistant's response. -Mendix also strongly advises that you build user confirmation logic into function microflows that have a potential impact on the world on behalf of the end-user. Some examples of such microflows include sending an email, posting online, or making a purchase. +Mendix recommends building user confirmation logic into function microflows that potentially impact the world on behalf of the end-user. Examples of such microflows include sending an email, posting online, or making a purchase. {{% /alert %}} For more information, see [Function Calling](/agents/function-calling/). #### Adding Knowledge Bases {#chatcompletions-add-knowledge-base} -Adding knowledge bases to a call enables LLMs to retrieve information when a related topics are mentioned. Including knowledge bases in the request object along with a name and description, enables the model to intelligently decide when to let the Mendix app call one or more predefined knowledge bases. This allows the assistant to include the additional information in its response. +Adding knowledge bases to a call enables LLMs to retrieve information when related topics are mentioned. Including knowledge bases in the request object along with a name and description enables the model to intelligently decide when to let the Mendix app call one or more predefined knowledge bases, allowing the assistant to include additional information in its response. -Mistral does not directly connect to the knowledge resources. The model returns a tool call JSON structure that is used to build the input of the retrievals so that they can be executed as part of the chat completions operation. The OpenAI connector takes care of handling the tool call response for Mistral as well as executing the function microflows until the API returns the assistant's final response. +Mistral does not directly connect to knowledge resources. The model returns a tool call JSON structure that builds the input of the retrievals so they can run as part of the chat completions operation. The OpenAI connector handles the tool call response for Mistral and runs the function microflows until the API returns the assistant's final response. -This functionality is part of the implementation executed by the GenAI Commons Chat Completions operations mentioned earlier. As a developer, you need to make the system aware of your indexes and their purpose by registering them with the request. This is done using the GenAI Commons operation [Tools: Add Knowledge Base](/agents/genai-for-mx/commons/#add-knowledge-base-to-request), which must be called once per knowledge resource before passing the request to the Chat Completions operation. +The GenAI Commons chat completions operations mentioned earlier run this functionality. As a developer, make the system aware of your indexes and their purpose by registering them with the request. Use the GenAI Commons operation [Tools: Add Knowledge Base](/agents/genai-for-mx/commons/#add-knowledge-base-to-request), which must be called once per knowledge resource before passing the request to the chat completions operation. Note that the retrieval process is independent of the model provider and can be used with any model that supports function calling, as it relies on the generalized `GenAICommons.ConsumedKnowledgeBase` input parameter. #### Vision {#chatcompletions-vision} -Vision enables models like Mistral Medium 3.1 and Mistral Small 3.2 to interpret and analyze images, allowing them to answer questions and perform tasks related to visual content. This integration of computer vision and language processing enhances the model's comprehension and makes it valuable for tasks involving visual information. To make use of vision with Mistral connector, an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or multiple images must be sent along with a single message. +Vision enables models like Mistral Medium 3.1 and Mistral Small 3.2 to interpret and analyze images, allowing them to answer questions and perform tasks related to visual content. This integration of computer vision and language processing enhances the model's comprehension and makes it valuable for tasks involving visual information. To use vision with the Mistral connector, send an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or multiple images along with a single message. For `Chat Completions without History`, `FileCollection` is an optional input parameter. For `Chat Completions with History`, `FileCollection` can optionally be added to individual user messages using [Chat: Add Message to Request](/agents/genai-for-mx/commons/#chat-add-message-to-request). -Use the two microflow actions from the OpenAI specific toolbox [Files: Initialize Collection with OpenAI File](#initialize-filecollection) and [Files: Add OpenAIFile to Collection](#add-file) to construct the input with either `FileDocuments` (for vision, it needs to be of type `Image`) or `URLs`. There are similar file operations exposed by the GenAI commons module that can be used for vision requests with the OpenAIConnector for Mistral. However, these generic operations do not support the optional OpenAI API-specific `Detail` attribute. +Use the two microflow actions from the OpenAI-specific toolbox—[Files: Initialize Collection with OpenAI File](#initialize-filecollection) and [Files: Add OpenAIFile to Collection](#add-file)—to construct the input with either `FileDocuments` (for vision, this must be of type `Image`) or `URLs`. The GenAI Commons module exposes similar file operations that can be used for vision requests with the OpenAI connector for Mistral. However, these generic operations do not support the optional OpenAI API-specific `Detail` attribute. -For more information on vision, see [Mistral](https://docs.mistral.ai/capabilities/vision) documentation. +For more information on vision, see the [Mistral documentation](https://docs.mistral.ai/capabilities/vision). #### Document Chat {#chatcompletions-document} -Document chat enables the model to interpret and analyze PDF documents, allowing it to answer questions and perform tasks based on the document content. Document chat is currently not supported by the Mistral connector as it requires its own API. Check out [Document AI](https://docs.mistral.ai/capabilities/document_ai) documentation if you want to learn about Mistral's OCR capabilities. +Document chat enables the model to interpret and analyze PDF documents, allowing it to answer questions and perform tasks based on the document content. The Mistral connector does not support document chat because it requires its own API. To learn about Mistral's optical character recognition (OCR) capabilities, see the [Document AI documentation](https://docs.mistral.ai/capabilities/document_ai). #### Image Generations {#image-generations-configuration} -Image generation is currently not supported by the Mistral connector. You can learn more about image generation with Mistral in the [Image Generation](https://docs.mistral.ai/agents/connectors/image_generation/) section. +The Mistral connector does not support image generation. To learn more about image generation with Mistral, see [Image Generation](https://docs.mistral.ai/agents/connectors/image_generation/) in the Mistral documentation. #### Embeddings Generation {#embeddings-configuration} -Mistral also provides vector embedding generation capabilities which can be invoked using this connector module. The `MistralDeployedModel` entity is compatible with the [knowledge base operations](/agents/genai-for-mx/commons/#genai-knowledgebase-content) from the GenAI Commons. +Mistral provides vector embedding generation capabilities that can be invoked using this connector module. The `MistralDeployedModel` entity is compatible with the [knowledge base operations](/agents/genai-for-mx/commons/#genai-knowledgebase-content) from GenAI Commons. -In order to implement embeddings generation into your Mendix application, you can use the Embedding generation microflow actions from GenAI Commons directly. When developing your microflow, you can drag and drop the one you need from the toolbox: find it under the **GenAI (Generate)** category in the **Toolbox** in Mendix Studio Pro: +To implement embeddings generation into your Mendix application, use the embeddings generation microflow actions from GenAI Commons. When developing your microflow, drag and drop the action you need from the **GenAI (Generate)** category in the **Toolbox** in Studio Pro: * Generate Embeddings (String) * Generate Embeddings (Chunk Collection) -Depending on the operation you use in the microflow, an `InputText` String or a [ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection) needs to be provided. The current version of this operation only supports the float representation of the resulting vector. +Depending on the operation you use in the microflow, provide an `InputText` string or a [ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection). The current version of this operation only supports the float representation of the resulting vector. {{% alert color="info" %}} -The Mistral API limits the amount of chunks that can be embedded within the single API call. To embed a larger amount of chunks, it is recommended to process them in batches. You can find the example of this use case in the Clustering example of the [GenAI showcase](https://marketplace.mendix.com/link/component/220475) application. +The Mistral API limits the number of chunks that can be embedded within a single API call. To embed a larger number of chunks, process them in batches. You can find an example of this use case in the Clustering example of the [GenAI showcase](https://marketplace.mendix.com/link/component/220475) application. {{% /alert %}} -The microflow action `Generate Embeddings (String)` supports scenarios where the vector embedding of a single string must be generated, e.g. to use for a nearest neighbor search across an existing knowledge base. This input string can be passed directly as the `InputText` parameter of this microflow. Additionally, [EmbeddingsOptions](/agents/genai-for-mx/commons/#embeddingsoptions-entity) is optional and can be instantiated using [Embeddings: Create EmbeddingsOptions](/agents/genai-for-mx/commons/#embeddingsoptions-create) from GenAI Commons. Use the GenAI Commons toolbox action [Embeddings: Get First Vector from Response](/agents/genai-for-mx/commons/#embeddings-get-first-vector) to retrieve the generated embeddings vector. Both mentioned operations can be found under **GenAI Knowledge Base (Content)** in the **Toolbox** in Mendix Studio Pro. +The `Generate Embeddings (String)` microflow action supports scenarios where the vector embedding of a single string must be generated (for example, to use for a nearest neighbor search across an existing knowledge base). Pass this input string directly as the `InputText` parameter of this microflow. Additionally, [EmbeddingsOptions](/agents/genai-for-mx/commons/#embeddingsoptions-entity) is optional and can be instantiated using [Embeddings: Create EmbeddingsOptions](/agents/genai-for-mx/commons/#embeddingsoptions-create) from GenAI Commons. Use the GenAI Commons toolbox action [Embeddings: Get First Vector from Response](/agents/genai-for-mx/commons/#embeddings-get-first-vector) to retrieve the generated embeddings vector. Both operations can be found under **GenAI Knowledge Base (Content)** in the **Toolbox** in Studio Pro. -The microflow action `Generate Embeddings (Chunk Collection)` supports the more complex scenario where a collection of string inputs is vectorized in a single API call, such as when converting a collection of texts (chunks) into embeddings to be inserted into a knowledge base. Instead of calling the API for each string, executing a single call for a list of strings can significantly reduce HTTP overhead. Use the exposed microflows of GenAI Commons [Chunks: Initialize ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-create) to create the wrapper and [Chunks: Add Chunk to ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-add-chunk), or [Chunks: Add KnowledgeBaseChunk to ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) to construct the input. The resulting embedding vectors returned after a successful API call will be stored in the `EmbeddingVector` attribute in the same `Chunk` object. \ -Purely to generate embeddings, it does not matter whether the ChunkCollection contains Chunks or its specialization KnowledgeBaseChunks. However, if the end goal is to store the generated embedding vectors in a knowledge base (e.g. using the [PgVector Knowledge Base](/appstore/modules/pgvector-knowledge-base/) module), then Mendix recommends adding `KnowledgeBaseChunks` to the `ChunkCollection` and using these as an input for the embeddings operations, so they can later be used directly to populate the knowledge base. +The `Generate Embeddings (Chunk Collection)` microflow action supports the more complex scenario where a collection of string inputs is vectorized in a single API call, such as when converting a collection of texts (chunks) into embeddings to be inserted into a knowledge base. Instead of calling the API for each string, executing a single call for a list of strings can significantly reduce HTTP overhead. Use the exposed microflows from GenAI Commons [Chunks: Initialize ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-create) to create the wrapper and [Chunks: Add Chunk to ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-add-chunk), or [Chunks: Add KnowledgeBaseChunk to ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) to construct the input. After a successful API call, the resulting embedding vectors are stored in the `EmbeddingVector` attribute in the same `Chunk` object. -Note that, currently, the knowledge base interaction (e.g. inserting or retrieving chunks) is not supported for OpenAI compatible APIs. For more information on possible ways to work with knowledge bases for embedding generation, see [PgVector Knowledge Base](/appstore/modules/pgvector-knowledge-base/) and [setting up a Vector Database](/agents/reference-guide/external-connectors/pgvector-setup/). +To generate embeddings, it does not matter whether the ChunkCollection contains Chunks or its specialization KnowledgeBaseChunks. However, if the goal is to store the generated embedding vectors in a knowledge base (such as using the [PgVector Knowledge Base](/appstore/modules/pgvector-knowledge-base/) module), Mendix recommends adding `KnowledgeBaseChunks` to the `ChunkCollection` and using these as input for the embeddings operations so they can later be used directly to populate the knowledge base. + +OpenAI-compatible APIs do not support knowledge base interaction (inserting or retrieving chunks). For more information on ways to work with knowledge bases for embedding generation, see [PgVector Knowledge Base](/appstore/modules/pgvector-knowledge-base/) and [Setting Up a Vector Database](/agents/reference-guide/external-connectors/pgvector-setup/). ### Exposed Microflow Actions for OpenAI-compatible APIs {#exposed-microflows} -T exposed microflow actions used to construct requests via drag-and-drop specifically for OpenAI-compatible APIs are listed below. You can find these microflows in the **Toolbox** of Studio Pro. Note that these flows are only required if you need to add Mistral-specific options to your requests. For generic functionality, can use the GenAI Commons toolbox actions to [create the required Request](/agents/genai-for-mx/commons/#genai-request-building) and [handle the Response](/agents/genai-for-mx/commons/#genai-response-handling). These actions are available under the **GenAI (Request Building)** and **GenAI (Response Handling)** categories in the Toolbox. +The exposed microflow actions used to construct requests via drag-and-drop for OpenAI-compatible APIs are listed below. You can find these microflows in the **Toolbox** in Studio Pro. These actions are only required if you need to add Mistral-specific options to your requests. For generic functionality, use the GenAI Commons toolbox actions to [create the required Request](/agents/genai-for-mx/commons/#genai-request-building) and [handle the Response](/agents/genai-for-mx/commons/#genai-response-handling). These actions are available under the **GenAI (Request Building)** and **GenAI (Response Handling)** categories in the **Toolbox**. #### Set Response Format {#set-responseformat-chat} -This microflow changes the `ResponseFormat` of the `OpenAIRequest_Extension` object, which will be created for a `Request` if not already present. This describes the format that the chat completions model must output. By default, models compatible with the OpenAI API return `Text`. To enable JSON mode, you must set the input value as *JSONObject*. +This microflow changes the `ResponseFormat` of the `OpenAIRequest_Extension` object, which is created for a `Request` if not already present. This describes the format that the chat completions model must output. By default, models compatible with the OpenAI API return `Text`. To enable JSON mode, set the input value as *JSONObject*. #### Files: Initialize Collection with OpenAI Image {#initialize-filecollection} -This operation is currently not relevant for Mistral connector. +This operation is currently not relevant for the Mistral connector. #### Files: Add OpenAI Image to Collection {#add-file} -This operation is currently not relevant for Mistral connector. +This operation is currently not relevant for the Mistral connector. #### Image Generation: Set ImageOptions Extension {#set-imageoptions-extension} -This operation is currently not relevant for Mistral connector. +This operation is currently not relevant for the Mistral connector. ## Technical Reference {#technical-reference} @@ -201,13 +202,13 @@ The module includes technical reference documentation for the available entities The **Documentation** pane displays the documentation for the currently selected element. To view it, perform the following steps: 1. In the [View menu](/refguide/view-menu/) of Studio Pro, select **Documentation**. -2. Click the element for which you want to view the documentation. +2. Click the element you want to view documentation for. {{< figure src="/attachments/appstore/platform-supported-content/modules/technical-reference/doc-pane.png" alt="" >}} ### Tool Choice -Mistral supports the following [tool choice types](/agents/genai-for-mx/commons/#enum-toolchoice) of GenAI Commons for the [Tools: Set Tool Choice](/agents/genai-for-mx/commons/#set-toolchoice) action are supported. For API mapping reference, see the table below: +Mistral supports the following [tool choice types](/agents/genai-for-mx/commons/#enum-toolchoice) from GenAI Commons for the [Tools: Set Tool Choice](/agents/genai-for-mx/commons/#set-toolchoice) action. For API mapping reference, see the table below: | GenAI Commons (Mendix) | Mistral | | -----------------------| ------- | @@ -215,23 +216,23 @@ Mistral supports the following [tool choice types](/agents/genai-for-mx/commons/ | any | any | | none | none | -## GenAI Showcase Application {#showcase-application} +## GenAI Showcase App {#showcase-application} -For more inspiration or guidance on how to use those microflows in your logic, Mendix recommends downloading the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of example use cases. +For more inspiration or guidance on how to use these microflows in your logic, download the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of example use cases. {{% alert color="info" %}} -Some examples demonstrate knowledge base interaction and require a connection to a vector database. For more information on these concepts, see [Retrieval Augmented Generation (RAG)](/agents/rag/) +Some examples demonstrate knowledge base interaction and require a connection to a vector database. For more information on these concepts, see [Retrieval Augmented Generation (RAG)](/agents/rag/). {{% /alert %}} ## Troubleshooting {#troubleshooting} ### Attribute or Reference Required Error Message After Upgrade -If you encounter an error stating that an attribute or a reference is required after an upgrade, first upgrade all modules by right-clicking the error, then upgrade Data Widgets. +If you encounter an error stating that an attribute or reference is required after an upgrade, upgrade all modules by right-clicking the error. Then upgrade Data Widgets. ### Conflicted Lib Error After Module Import -If you encounter an error caused by conflicting Java libraries, such as `java.lang.NoSuchMethodError: 'com.fasterxml.jackson.annotation.OptBoolean com.fasterxml.jackson.annotation.JsonProperty.isRequired()'`, try synchronizing all dependencies (**App** > **Synchronize dependencies**) and then restart your application. +If you encounter an error caused by conflicting Java libraries, such as `java.lang.NoSuchMethodError: 'com.fasterxml.jackson.annotation.OptBoolean com.fasterxml.jackson.annotation.JsonProperty.isRequired()'`, synchronize all dependencies (**App** > **Synchronize dependencies**) and restart your application. ## Read More {#read-more} diff --git a/content/en/docs/genai/reference-guide/external-platforms/mx-genai-connector.md b/content/en/docs/genai/reference-guide/external-platforms/mx-genai-connector.md index 7376a94dc18..393518f1490 100644 --- a/content/en/docs/genai/reference-guide/external-platforms/mx-genai-connector.md +++ b/content/en/docs/genai/reference-guide/external-platforms/mx-genai-connector.md @@ -2,7 +2,7 @@ title: "Mendix Cloud GenAI Connector" url: /agents/mx-cloud-genai/mxgenai-connector/ linktitle: "Mendix Cloud GenAI Connector" -description: "Describes the configuration and usage of the Mendix Cloud GenAI Connector, enabling you to integrate Mendix Cloud GenAI Resource Packs directly into your Mendix application." +description: "Describes how to configure and use the Mendix Cloud GenAI Connector, enabling you to integrate Mendix Cloud GenAI Resource Packs directly into your Mendix application." weight: 20 aliases: - /appstore/modules/genai/MxGenAI/ @@ -11,7 +11,7 @@ aliases: ## Introduction -The [Mendix Cloud GenAI connector](https://marketplace.mendix.com/link/component/239449) lets you use [Mendix Cloud GenAI Resource Packs](/agents/mx-cloud-genai/resource-packs/) directly within your Mendix application. It allows you to integrate generative AI by dragging and dropping common operations from its toolbox. +The [Mendix Cloud GenAI connector](https://marketplace.mendix.com/link/component/239449) lets you use [Mendix Cloud GenAI Resource Packs](/agents/mx-cloud-genai/resource-packs/) directly within your Mendix application. You can integrate generative AI by dragging and dropping common operations from its toolbox. ### Features @@ -21,12 +21,10 @@ Typical use cases for generative AI are described in more detail in the [Typical ### Prerequisites -To use this connector, you need configuration keys to authenticate to the Mendix Cloud GenAI services. You can generate keys in the [Mendix Cloud GenAI Portal](https://genai.home.mendix.com) or ask someone with access to either generate them for you or add you to their team so that you can generate keys yourself. +To use this connector, you need configuration keys to authenticate to the Mendix Cloud GenAI services. You can generate keys in the [Mendix Cloud GenAI Portal](https://genai.home.mendix.com). Alternatively, ask someone with access to generate keys for you or add you to their team so you can generate keys yourself. {{% alert color="info" %}} - -The Mendix Cloud GenAI Connector module generates embeddings internally when interacting with a knowledge base. This means that you do not need to create embedding keys yourself when interacting with a Mendix Cloud knowledge base. Direct embedding operations are only required if additional processes, such as using the generated vectors instead of text, are needed. For example, a similar search algorithm could use vector distances to calculate relatedness. - +The Mendix Cloud GenAI Connector module generates embeddings internally when interacting with a knowledge base. You do not need to create embedding keys yourself when interacting with a Mendix Cloud knowledge base. Direct embedding operations are only required if additional processes are needed, such as using the generated vectors instead of text. For example, a similar search algorithm could use vector distances to calculate relatedness. {{% /alert %}} ### Dependencies {#dependencies} @@ -41,14 +39,14 @@ Add the [dependencies](#dependencies) listed above from the Marketplace. To impo ## Configuration {#configuration} -After installing the Mendix Cloud GenAI connector, you can find it in the **App Explorer** inside of the **Marketplace modules** section. The connector includes a domain model and several activities to help integrate your app with the Mendix Cloud GenAI service. To implement the connector, simply use its actions in a microflow. You can find the Mendix GenAI actions in the microflow toolbox. +After installing the Mendix Cloud GenAI connector, you can find it in the **App Explorer** under the **Marketplace modules** section. The connector includes a domain model and several activities to integrate your app with the Mendix Cloud GenAI service. To implement the connector, use its actions in a microflow. You can find the Mendix GenAI actions in the microflow toolbox. -Follow the steps below to get started: +To get started, follow these steps: -* Make sure to configure the [Encryption module](/appstore/modules/encryption/#configuration) before you connect your app to Mendix Cloud GenAI. -* Add the module role `MxGenAIConnector.Administrator` to your Administrator **User roles** in the **Security** settings of your app. -* Add the `Configuration_Overview` page (**USE_ME** > **Configuration**) to your navigation, or add the `Snippet_Configuration` to a page that is already part of your navigation. Alternatively, you can register your key by using the `Configuration_RegisterByString` microflow. -* Complete the runtime setup of the Mendix Cloud GenAI configuration by navigating to the page mentioned above. Import a key generated in the [Mendix Cloud GenAI Portal](https://genai.home.mendix.com) or provided to you and click **Test Key** to validate its functionality. Note that this key establishes a connection between the Mendix Cloud resources and your application. It contains all the information required to set up the connection. +* Configure the [Encryption module](/appstore/modules/encryption/#configuration) before you connect your app to Mendix Cloud GenAI. +* Add the module role `MxGenAIConnector.Administrator` to your Administrator user roles in the **Security** settings of your app. +* Add the `Configuration_Overview` page (**USE_ME** > **Configuration**) to your navigation, or add the `Snippet_Configuration` to a page that is already part of your navigation. Alternatively, register your key by using the `Configuration_RegisterByString` microflow. +* Complete the runtime setup of the Mendix Cloud GenAI configuration by navigating to the page mentioned above. Import a key generated in the [Mendix Cloud GenAI Portal](https://genai.home.mendix.com) or provided to you and click **Test Key** to validate its functionality. This key establishes a connection between the Mendix Cloud resources and your application and contains all the information required to set up the connection. {{% alert color="info" %}} When using an Embeddings Model Resource together with a Knowledge Base Resource, you do not need to import both keys. Importing the Knowledge Base Resource key automatically generates the connection details for the embeddings generation model. @@ -58,84 +56,83 @@ When using an Embeddings Model Resource together with a Knowledge Base Resource, {{< figure src="/attachments/genai/mxgenAI-connector/mxgenaiconnector-configuration.png" alt="" >}} -Configuration keys are stored persistently after they are imported (either via the UI or the exposed microflow). There are three different types of configurations that reflect the use cases this service supports. The specific operations are described below. +Configuration keys are stored persistently after import, either via the UI or the exposed microflow. There are three different types of configurations that reflect the use cases this service supports. The specific operations are described below. To use the operations, either a `DeployedModel` (text, embeddings) or a `DeployedKnowledgeBase` must always be passed as input. -### How to get the `DeployedModel` in scope +### How to Get the `DeployedModel` in Scope -The `DeployedModel` object will be created automatically when importing keys at runtime and needs to be retrieved from the database. +The `DeployedModel` object is created automatically when importing keys at runtime and must be retrieved from the database. -### How to get the `DeployedKnowledgeBase` in scope +### How to Get the `DeployedKnowledgeBase` in Scope -In Mendix Cloud GenAI, a single knowledge base resource (`MxCloudKnowledgeBaseResource`) can contain multiple `DeployedKnowledgeBase` objects (tables, referred to as 'collections'). As a result, several collections may belong to the same resource. You can use the `DeployedKnowledgeBase: Get` toolbox action to retrieve the right collection and initialize a knowledge base operation. It requires the `Collection.Name` (string) as input (which is usually different from the `Collection.DisplayName` attribute). +In Mendix Cloud GenAI, a single knowledge base resource (`MxCloudKnowledgeBaseResource`) can contain multiple `DeployedKnowledgeBase` objects (tables, referred to as collections). Several collections may belong to the same resource. Use the `DeployedKnowledgeBase: Get` toolbox action to retrieve the right collection and initialize a knowledge base operation. It requires the `Collection.Name` (string) as input, which is usually different from the `Collection.DisplayName` attribute. ### Chat Completions Operation -After following the general setup above, you are ready to use the chat completions microflows in the GenAICommons and MxGenAIConnector modules. You can find `Chat Completions (without history)` and `Chat Completions (with history)` in the **Text & Files** folder of the GenAICommons. The chat completions microflows are also exposed as microflow actions under the **GenAI (Generate)** category inside of the **Toolbox**. +After following the general setup above, you are ready to use the chat completions microflows in the GenAICommons and MxGenAIConnector modules. You can find `Chat Completions (without history)` and `Chat Completions (with history)` in the **Text & Files** folder of GenAICommons. The chat completions microflows are also exposed as microflow actions under the **GenAI (Generate)** category in the **Toolbox**. These microflows expect a `DeployedModel` as input to determine the connection details. -In chat completions, system prompts and user prompts are two key components that help guide the language model in generating relevant and contextually appropriate responses. For more information on prompt engineering, see the [Read More](#readmore) section. Different exposed microflow activities may require different prompts and logic for how the prompts must be passed, as described in the following sections. For more information on message roles, see the [ENUM_MessageRole](/agents/genai-for-mx/commons/#enum-messagerole) enumeration in *GenAI Commons*. +In chat completions, system prompts and user prompts are two key components that guide the language model in generating relevant and contextually appropriate responses. For more information on prompt engineering, see the [Read More](#readmore) section. Different exposed microflow activities may require different prompts and logic for how prompts must be passed, as described in the following sections. For more information on message roles, see the [ENUM_MessageRole](/agents/genai-for-mx/commons/#enum-messagerole) enumeration in *GenAI Commons*. The chat completion operations support [Function Calling](#function-calling), [Vision](#vision), and [Document Chat](#document-chat). For more inspiration or guidance on how to use the above-mentioned microflows in your logic, Mendix recommends downloading the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of examples. -#### Chat Completions (without History) +#### Chat Completions (Without History) The microflow activity [Chat Completions (without history)](/agents/genai-for-mx/commons/#chat-completions-without-history) supports scenarios where there is no need to send a list of (historic) messages comprising the conversation so far as part of the request. -#### Chat Completions (with History) +#### Chat Completions (With History) The microflow activity [Chat completions (with history)](/agents/genai-for-mx/commons/#chat-completions-with-history) supports more complex use cases where a list of (historical) messages (for example, the conversation or context so far) is sent as part of the request to the LLM. #### Retrieve & Generate {#retrieve-and-generate} -To use retrieval and generation in a single operation, an internally predefined tool can be added to the [Request](/agents/genai-for-mx/commons/#request) via the `Tools: Add Knowledge Base` action. The model can then decide whether to use the [knowledge base retrieval](/agents/genai-for-mx/commons/#knowledge-base-retrieval) tool when handling the request. This functionality is supported in both with-history and without-history operations. The (optional) `Description` helps the model to understand the knowledge base content and decide whether it should be called in the current chat context. Additionally, you may apply optional filters, such as `MaxNumberOfResults` or `MinimumSimilarity`, or pass a [MetadataCollection](/agents/genai-for-mx/commons/#metadatacollection-entity). +To use retrieval and generation in a single operation, add an internally predefined tool to the [Request](/agents/genai-for-mx/commons/#request) via the `Tools: Add Knowledge Base` action. The model can then decide whether to use the [knowledge base retrieval](/agents/genai-for-mx/commons/#knowledge-base-retrieval) tool when handling the request. This functionality is supported in both with-history and without-history operations. The optional `Description` parameter helps the model understand the knowledge base content and decide whether it should be called in the current chat context. You can also apply optional filters, such as `MaxNumberOfResults` or `MinimumSimilarity`, or pass a [MetadataCollection](/agents/genai-for-mx/commons/#metadatacollection-entity). {{< figure src="/attachments/genai/mxgenAI-connector/mxgenaiconnector-rag.png" alt="" >}} The returned `Response` includes [References](/agents/genai-for-mx/commons/#reference) for each retrieved chunk from the knowledge base. -Optionally, you can control both reference creation and the output returned for the model during the insertion step: +You can optionally control both reference creation and the output returned for the model during the insertion step: -* The `HumanReadableId` of a chunk is used for the reference title in the response, which is shown to the end user in the [ConversationalUI](/agents/genai-for-mx/conversational-ui/). -* To utilize the `Source` attribute of the references, include `MetaData` with the key `sourceUrl`. In [ConversationalUI](/agents/genai-for-mx/conversational-ui/), this will appear as a clickable link for the end user. -* In some cases, a knowledge chunk consists of two texts: one for the semantic search (retrieval) step, and another for the generation step. For example, when solving a problem based on historical solutions, semantic search identifies similar problems using their descriptions, while the generation step produces a solution based on the corresponding historical solutions. In such cases, you can add [MetaData](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) with the key `knowledge` to each chunk during insertion. This allows the model to generate its response using the specified metadata instead of the input text (only the value of `knowledge` is passed to the model). +* The `HumanReadableId` of a chunk is used for the reference title in the response, shown to the end-user in the [ConversationalUI](/agents/genai-for-mx/conversational-ui/). +* To utilize the `Source` attribute of the references, include `MetaData` with the key `sourceUrl`. In [ConversationalUI](/agents/genai-for-mx/conversational-ui/), this appears as a clickable link for the end-user. +* In some cases, a knowledge chunk consists of two texts: one for the semantic search (retrieval) step and another for the generation step. For example, when solving a problem based on historical solutions, semantic search identifies similar problems using their descriptions, while the generation step produces a solution based on the corresponding historical solutions. In such cases, add [MetaData](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) with the key `knowledge` to each chunk during insertion. This allows the model to generate its response using the specified metadata instead of the input text. Only the value of `knowledge` is passed to the model. -#### Function Calling{#function-calling} +#### Function Calling {#function-calling} -Function calling enables LLMs to connect with external tools to gather information, execute actions, convert natural language into structured data, and much more. Function calling thus enables the model to intelligently decide when to let the Mendix app call one or more predefined function microflows to gather additional information to include in the assistant's response. +Function calling enables LLMs to connect with external tools to gather information, execute actions, convert natural language into structured data, and more. Function calling thus enables the model to intelligently decide when to let the Mendix app call one or more predefined function microflows to gather additional information to include in the assistant's response. -The model does not call the function but rather returns a tool called JSON structure that is used to build the input of the function (or functions) so that they can be executed as part of the chat completions operation. Functions in Mendix are essentially microflows that can be registered within the request to the LLM​. The connector takes care of handling the tool call response and executing the function microflows until the API returns the assistant's final response. +The model does not call the function. Instead, it returns a tool called JSON structure that builds the input of the function (or functions) so they can be executed as part of the chat completions operation. Functions in Mendix are microflows that can be registered within the request to the LLM. The connector handles the tool call response and executes the function microflows until the API returns the assistant's final response. Function microflows can have none, a single, or multiple primitive input parameters such as Boolean, Datetime, Decimal, Enumeration, Integer or String. Additionally, they may accept the [Request](/agents/genai-for-mx/commons/#request) or [Tool](/agents/genai-for-mx/commons/#tool) objects as inputs. The function microflow must return a String value. {{% alert color="warning" %}} -Function calling is a very powerful capability and should be used with caution. Note that function microflows run in the context of the current user without enforcing entity-access. You can use `$currentUser` in XPath queries to ensure you retrieve and return only information that the end-user is allowed to view; otherwise confidential information may become visible to the current end-user in the assistant's response. +Function calling is a powerful capability and should be used with caution. Function microflows run in the context of the current user without enforcing entity access. Use `$currentUser` in XPath queries to ensure you retrieve and return only information that the end-user is allowed to view. Otherwise, confidential information may become visible to the current end-user in the assistant's response. -Mendix also strongly advises that you build user confirmation logic into function microflows that have a potential impact on the world on behalf of the end-user. Some examples of such microflows include sending an email, posting online, or making a purchase. +Mendix recommends building user confirmation logic into function microflows that potentially impact the world on behalf of the end-user. Examples of such microflows include sending an email, posting online, or making a purchase. {{% /alert %}} -You can use function calling in all chat completions operations by adding a `ToolCollection` with a `Function` via the [Tools: Add Function to Request](/agents/genai-for-mx/commons/#add-function-to-request) operation. -For more information, see [Function Calling](/agents/function-calling/). +Use function calling in all chat completions operations by adding a `ToolCollection` with a `Function` via the [Tools: Add Function to Request](/agents/genai-for-mx/commons/#add-function-to-request) operation. For more information, see [Function Calling](/agents/function-calling/). -#### Vision{#vision} +#### Vision {#vision} -Vision enables the model to interpret and analyze images, allowing them to answer questions and perform tasks related to visual content. This integration of computer vision and language processing enhances the model's comprehension and makes it valuable for tasks involving visual information. To ensure vision inside the connector, an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or multiple images must be sent with a single message. +Vision enables the model to interpret and analyze images, allowing it to answer questions and perform tasks related to visual content. This integration of computer vision and language processing enhances the model's comprehension and makes it valuable for tasks involving visual information. To use vision with the connector, send an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or multiple images with a single message. For [Chat Completions (without history)](/agents/genai-for-mx/commons/#chat-completions-without-history), `OptionalFileCollection` is an optional input parameter. For [Chat completions (with history)](/agents/genai-for-mx/commons/#chat-completions-with-history), a `FileCollection` can optionally be added to individual user messages using [Add Message to Request](/agents/genai-for-mx/commons/#chat-add-message-to-request). -In the entire conversation, you can pass up to 20 images that are smaller than 3.75 MB each and with a height and width of a maximum 8000 pixels. The following types are accepted: PNG, JPEG, JPG, GIF, and WebP. +In the entire conversation, you can pass up to 20 images that are smaller than 3.75 MB each and with a height and width of a maximum of 8000 pixels. The following types are accepted: PNG, JPEG, JPG, GIF, and WebP. -#### Document Chat{#document-chat} +#### Document Chat {#document-chat} -Document chat enables the model to interpret and analyze documents, such as PDFs or Excel files, allowing them to answer questions and perform tasks related to the content. To use document chat, an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or multiple documents must be sent along with a single message. +Document chat enables the model to interpret and analyze documents, such as PDFs or Excel files, allowing it to answer questions and perform tasks related to the content. To use document chat, send an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or multiple documents along with a single message. For [Chat Completions (without history)](/agents/genai-for-mx/commons/#chat-completions-without-history), `OptionalFileCollection` is an optional input parameter. For [Chat completions (with history)](/agents/genai-for-mx/commons/#chat-completions-with-history), a `FileCollection` can optionally be added to individual user messages using [Add Message to Request](/agents/genai-for-mx/commons/#chat-add-message-to-request). -In the entire conversation, you can pass up to five documents that are smaller than 4.5 MB each. Note that there is also a practical, model-dependent limit on the number of pages a document can contain - typically around 100 pages, but this is not fixed and can vary with the selected model and the complexity of the file (for example, images, heavy formatting, or embedded content can reduce the effective page limit). If you expect to work with very large documents, consider splitting them into smaller files or providing summarized extracts to improve reliability. +In the entire conversation, you can pass up to five documents that are smaller than 4.5 MB each. There is also a practical, model-dependent limit on the number of pages a document can contain, typically around 100 pages. This is not fixed and can vary with the selected model and the complexity of the file. For example, images, heavy formatting, or embedded content can reduce the effective page limit. If you expect to work with very large documents, consider splitting them into smaller files or providing summarized extracts to improve reliability. The following file types are accepted: PDF, CSV, DOC, DOCX, XLS, XLSX, HTML, TXT, and MD. @@ -149,19 +146,19 @@ The model uses the file name when analyzing documents, which may introduce a pot ##### Collections -A Knowledge Base resource can comprise several collections. Each collection is specifically designed to hold numerous documents, serving as a logical grouping for related information based on its shared domain, purpose, or thematic focus. +A knowledge base resource can comprise several collections. Each collection is tdesigned to hold numerous documents and serves as a logical grouping for related information based on its shared domain, purpose, or thematic focus. Below is a diagram showing how resources are organized into separate collections. This approach allows multiple use cases to share a common resource while the option to only add the required collections to the conversation context is preserved. For example, both employee onboarding and IT ticket support require information about IT setup and equipment. However, only onboarding needs knowledge about the company culture and values, while only IT support requires access to historical support ticket data. {{< figure src="/attachments/genai/navigate_mxgenai/GenAIKnowledgeBaseResource.png" alt="" >}} -While collections provide a mechanism for data separation, it is not best practice to create a large number of collections within a single Knowledge Base resource. A more performant and practical approach for achieving fine-grained data separation is through the strategic use of metadata. +While collections provide a mechanism for data separation, it is not best practice to create a large number of collections within a single knowledge base resource. A more performant and practical approach for achieving fine-grained data separation is through the strategic use of metadata. ##### Metadata -Metadata is additional information that can be attached to data in a GenAI knowledge base. Unlike the actual content, metadata provides structured details that help in organizing, searching, and filtering information more efficiently. It helps manage large datasets by allowing the retrieval of relevant data based on specific attributes rather than relying solely on similarity-based searches. +Metadata is additional information that can be attached to data in a GenAI knowledge base. Unlike the actual content, metadata provides structured details that help organize, search, and filter information more efficiently. It helps manage large datasets by allowing the retrieval of relevant data based on specific attributes rather than relying solely on similarity-based searches. -Metadata consists of key-value pairs and serves as additional information connected to the data, though it is not part of the vectorization itself. +Metadata consists of key-value pairs and serves as additional information connected to the data, though it is not part of the vectorization. In the employee onboarding and IT ticket support example, instead of having two different collections, such as IT setup, and equipment and historical support tickets, there could be one named 'Company IT'. To retrieve tickets only and no other information from this collection, add the metadata below during insertion. @@ -173,21 +170,21 @@ The model then generates its response using the specified metadata instead of so {{< figure src="/attachments/genai/navigate_mxgenai/GenAIKBMetadataSeparation.png" alt="" >}} -Using metadata, even more fine-grained filtering becomes feasible. Each ticket may have associated metadata, such as +Using metadata, even more fine-grained filtering becomes feasible. Each ticket may have associated metadata, such as the following: * key: `Ticket Type`, value: `Bug` * key: `Status`, value: `Solved` * key: `Priority`, value: `High` -Instead of relying solely on similarity-based searches of ticket descriptions, users can then filter for specific tickets, such as 'Bug' tickets with the status set to 'Solved'. You can add [MetaData](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) with the respective key to each chunk during insertion. +Instead of relying solely on similarity-based searches of ticket descriptions, users can filter for specific tickets, such as Bug tickets with the status set to Solved. Add [MetaData](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) with the respective key to each chunk during insertion. #### How to Get Data Into a Knowledge Base -If you are looking for a step-by-step guide on how to get your application data into a collection inside a Mendix Cloud Knowledge Base Resource, refer to [Grounding Your Large Language Model in Data – Mendix Cloud GenAI](/agents/how-to/howto-groundllm/). Note that the Mendix Portal also provides options for importing data into your knowledge base, such as file uploads. For more information, see [Navigate through the Mendix Cloud GenAI Portal](/agents/mx-cloud-genai/Navigate-MxGenAI/). This documentation focuses solely on adding data from inside a Mendix application and using the connector. +For a step-by-step guide on how to get your application data into a collection inside a Mendix Cloud knowledge base resource, see [Grounding Your Large Language Model in Data – Mendix Cloud GenAI](/agents/how-to/howto-groundllm/). The Mendix Portal also provides options for importing data into your knowledge base, such as file uploads. For more information, see [Navigate through the Mendix Cloud GenAI Portal](/agents/mx-cloud-genai/Navigate-MxGenAI/). This documentation focuses solely on adding data from inside a Mendix application and using the connector. ### Knowledge Base Operations -To implement knowledge base logic into your Mendix application, you can use the actions in the **USE_ME** > **Knowledge Base** folder or under the **GenAI Knowledge Base (Content)** or **Mendix Cloud Knowledge Base** categories in the **Toolbox**. These actions require a specialized [DeployedKnowledgeBase](/agents/genai-for-mx/commons/#deployed-knowledge-base) of type `Collection` that determines the model and endpoint to use. Additionally, the collection name must be passed when creating the object, and it must be associated with a `Configuration` object. Note that for Mendix Cloud GenAI, a knowledge base resource may contain several collections (tables). +To implement knowledge base logic into your Mendix application, use the actions in the **USE_ME** > **Knowledge Base** folder or under the **GenAI Knowledge Base (Content)** or **Mendix Cloud Knowledge Base** categories in the **Toolbox**. These actions require a specialized [DeployedKnowledgeBase](/agents/genai-for-mx/commons/#deployed-knowledge-base) of type `Collection` that determines the model and endpoint to use. The collection name must be passed when creating the object, and the object must be associated with a `Configuration` object. For Mendix Cloud GenAI, a knowledge base resource may contain several collections (tables). Dealing with knowledge bases involves two main stages: @@ -197,18 +194,18 @@ Dealing with knowledge bases involves two main stages: You do not need to manually add embeddings to a chunk, as the connector handles this internally. To see all existing collections for a knowledge base configuration, go to the **Knowledge Base** tab on the [Mendix Cloud GenAI Configuration](#configuration) page and refresh the view on the right. Alternatively, use the `Get Collections` action to retrieve a synchronized list of collections inside your knowledge base resource to include in your module. Lastly, you can delete a collection using the `Delete Collection` action. {{% alert color="warning" %}} -The knowledge chunks are stored in an AWS OpenSearch Serverless database to ensure scalable and high-performance vector calculations—for example, retrieving the nearest neighbors of a given input. Inserted or modified chunks are only available for read operations (retrieval) in the knowledge base within 60-120 seconds. For more information, see [AWS documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-vector-search.html#serverless-vector-limitations). +Knowledge chunks are stored in an AWS OpenSearch Serverless database to ensure scalable and high-performance vector calculations, such as retrieving the nearest neighbors of a given input. Inserted or modified chunks are only available for read operations (retrieval) in the knowledge base within 60-120 seconds. For more information, see [AWS documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-vector-search.html#serverless-vector-limitations). {{% /alert %}} -#### Knowledge Base Insertion{#knowledge-base-insertion} +#### Knowledge Base Insertion {#knowledge-base-insertion} ##### Data Chunks -To add data to the knowledge base, you need discrete pieces of information and create knowledge base chunks for each one. Use the GenAICommons operations to first [initialize a ChunkCollection object](/agents/genai-for-mx/commons/#chunkcollection-create), and then [add a KnowledgeBaseChunk](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) object to it for each piece of information. Both can be found in the **Toolbox** inside of the **GenAI Knowledge Base (Content)** category. +To add data to the knowledge base, you need discrete pieces of information and create knowledge base chunks for each one. Use the GenAICommons operations to first [initialize a ChunkCollection object](/agents/genai-for-mx/commons/#chunkcollection-create), and then [add a KnowledgeBaseChunk](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) object to it for each piece of information. Both can be found in the **Toolbox** under the **GenAI Knowledge Base (Content)** category. ##### Chunking Strategy -Dividing data into chunks is crucial for model accuracy, as it helps optimize the relevance of the content. The best chunking strategy is to keep a balance between reducing noise by keeping chunks small and retaining enough content within a chunk to get relevant results. Creating overlapping chunks can help preserve more context while maintaining a fixed chunk size. It is recommended to experiment with different chunking strategies to decide the best strategy for your data. In general, if chunks are logical and meaningful to humans, they will also make sense to the model. A chunk size of approximately 1500 characters with overlapping chunks has been proven to be effective for longer texts in the past. +Dividing data into chunks is crucial for model accuracy, as it helps optimize the relevance of the content. The best chunking strategy balances reducing noise by keeping chunks small with retaining enough content within a chunk to get relevant results. Creating overlapping chunks can help preserve more context while maintaining a fixed chunk size. Experiment with different chunking strategies to decide the best strategy for your data. In general, if chunks are logical and meaningful to humans, they will also make sense to the model. A chunk size of approximately 1500 characters with overlapping chunks has been proven effective for longer texts. Because embeddings operations have a maximum character limit of 2048 characters per chunk, you must ensure that your chunks do not exceed this limit before submitting them for embedding. Chunks exceeding this limit will cause the embedding operation to fail, so validate your input data accordingly. @@ -216,7 +213,7 @@ The chunk collection can then be stored in the knowledge base using one of the f ##### Add Data Chunks to Your Knowledge Base -Use the following toolbox actions inside the **Mendix Cloud Knowledge Base** toolbox category to populate knowledge data into a collection: +Use the following toolbox actions in the **Mendix Cloud Knowledge Base** toolbox category to populate knowledge data into a collection: 1. `Embed & Insert` embeds a list of chunks (passed via a [ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection)) and inserts them into the knowledge base. 2. `Embed & Repopulate KB` is similar to `Embed & Insert`, but deletes all existing chunks from the knowledge base before inserting the new chunks. @@ -229,9 +226,9 @@ Additionally, use the following toolbox actions to delete chunks: When data in your Mendix app that is relevant to the knowledge base changes, it is usually necessary to keep the knowledge base chunks in sync. Whenever a Mendix object changes, the affected chunks must be updated. Depending on your use case, `Embed & Replace` and `Delete for Objects` can be used in event handler microflows. -##### Knowledge Base Retrieval{#knowledge-base-retrieval} +##### Knowledge Base Retrieval {#knowledge-base-retrieval} -The following toolbox actions can be used to retrieve knowledge data from a collection (and associate it with your Mendix data): +Use the following toolbox actions to retrieve knowledge data from a collection and associate it with your Mendix data: 1. `Retrieve` retrieves knowledge base chunks from the knowledge base. You can use pagination via the `Offset` and `MaxNumberOfResults` parameters or apply filtering via a `MetadataCollection` or `MxObject`. 2. `Retrieve & Associate` is similar to `Retrieve` but associates the returned chunks with a Mendix object if they were linked at the insertion stage. @@ -244,9 +241,9 @@ The following toolbox actions can be used to retrieve knowledge data from a coll ### Embedding Operations -If you are working directly with embedding vectors for specific use cases that do not include knowledge base interaction (for example, clustering or classification), the below operations are relevant. For practical examples and guidance, consider referring to the [GenAI Showcase Application](https://marketplace.mendix.com/link/component/220475) showcase to see how these embedding-only operations can be used. +If you are working directly with embedding vectors for specific use cases that do not include knowledge base interaction, such as clustering or classification, the operations below are relevant. For practical examples and guidance, refer to the [GenAI Showcase Application](https://marketplace.mendix.com/link/component/220475) to see how these embedding-only operations can be used. -To implement embeddings into your Mendix application, you can use the microflows in the **Knowledge Bases & Embeddings** folder inside of the GenAICommons module. Both microflows for embeddings are exposed as microflow actions under the **GenAI (Generate)** category in the **Toolbox** in Mendix Studio Pro. +To implement embeddings into your Mendix application, use the microflows in the **Knowledge Bases & Embeddings** folder in the GenAICommons module. Both microflows for embeddings are exposed as microflow actions under the **GenAI (Generate)** category in the **Toolbox** in Studio Pro. These microflows require a [DeployedModel](/agents/genai-for-mx/commons/#deployed-model) that determines the model and endpoint to use. Depending on the selected operation, an `InputText` String or a [ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection) needs to be provided. Note that embedding operations enforce a maximum character limit of 2048 characters per chunk; input exceeding this limit will cause the embedding operation to fail, so validate your input before submitting it for embedding. @@ -288,9 +285,9 @@ For more guidance on how to use microflows in your logic, Mendix recommends down ## Troubleshooting {#troubleshooting} -### Outdated JDK Version Causing Errors while Calling a REST API {#outdated-jdk-version} +### Outdated JDK Version Causing Errors While Calling a REST API {#outdated-jdk-version} -The Java Development Kit (JDK) is a framework needed by Mendix Studio Pro to deploy and run applications. For more information, see [Studio Pro System Requirements](/refguide/system-requirements/). Usually, the correct JDK version is installed during the installation of Studio Pro, but in some cases, it may be outdated. An outdated version can cause exceptions when calling REST-based services with large data volumes, like for example embeddings operations or chat completions with vision. +The Java Development Kit (JDK) is a framework needed by Studio Pro to deploy and run applications. For more information, see [Studio Pro System Requirements](/refguide/system-requirements/). Usually, the correct JDK version is installed during the installation of Studio Pro, but in some cases, it may be outdated. An outdated version can cause exceptions when calling REST-based services with large data volumes, such as embeddings operations or chat completions with vision. Mendix has seen the following two exceptions when using JDK versions below `jdk-11.0.5.0-hotspot`: `java.net.SocketException - Connection reset` or @@ -298,7 +295,7 @@ Mendix has seen the following two exceptions when using JDK versions below `jdk- To check your JDK version and update it if necessary, follow these steps: -1. Check your JDK version – In Studio Pro, go to **Edit** > **Preferences** > **Deployment** > **JDK directory**. If the path points to a version below `jdk-11.0.5.0-hotspot`, you need to update the JDK by following the next steps. +1. Check your JDK version. In Studio Pro, go to **Edit** > **Preferences** > **Deployment** > **JDK directory**. If the path points to a version below `jdk-11.0.5.0-hotspot`, update the JDK by following the next steps. 2. Go to [Eclipse Temurin JDK 11](https://adoptium.net/en-GB/temurin/releases/?variant=openjdk11&os=windows&package=jdk) and download the `.msi` file of the latest release of **JDK 11**. 3. Open the downloaded file and follow the installation steps. Remember the installation path. Usually, this should be something like `C:/Program Files/Eclipse Adoptium/jdk-11.0.22.7-hotspot`. 4. After the installation has finished, restart your computer if prompted. @@ -319,12 +316,11 @@ To resolve this issue, download and install [GenAI Commons 6.2.2](https://market ### Migrating From Add-on Module to App Module -Since the module has been changed with version 3.0.0 from an add-on to an app module, updating it via the marketplace will require a migration to ensure it works properly with your application. - -To do this, follow the steps below: +Because the module has been changed with version 3.0.0 from an add-on to an app module, updating it via Marketplace requires a migration to ensure it works properly with your app. -1. Back up your data — either as a full database backup or by exporting individual components: +To do this, follow these steps: +1. Back up your data, either as a full database backup or by exporting individual components: * Keys for the Mendix Cloud GenAI Resource Packs can be reimported later. * Incoming associations to the protected module’s entities will be deleted. 2. Delete the add-on module: MxGenAIConnector. diff --git a/content/en/docs/genai/reference-guide/external-platforms/openai.md b/content/en/docs/genai/reference-guide/external-platforms/openai.md index 572727c2405..82aab3672e5 100644 --- a/content/en/docs/genai/reference-guide/external-platforms/openai.md +++ b/content/en/docs/genai/reference-guide/external-platforms/openai.md @@ -2,7 +2,7 @@ title: "OpenAI" url: /agents/reference-guide/external-connectors/openai/ linktitle: "OpenAI" -description: "Describes the configuration and usage of the OpenAI Connector, which allows you to integrate generative AI into your Mendix app." +description: "Describes how to configure and use the OpenAI connector to integrate generative AI capabilities into Mendix apps." weight: 20 aliases: - /appstore/connectors/openai-connector/ @@ -12,25 +12,23 @@ aliases: ## Introduction {#introduction} -The [OpenAI Connector](https://marketplace.mendix.com/link/component/220472) allows you to integrate generative AI into your Mendix app. It is compatible with [OpenAI's platform](https://platform.openai.com/) and [Microsoft Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-ai-foundry), where you can access OpenAI models. +The [OpenAI connector](https://marketplace.mendix.com/link/component/220472) lets you generative AI capabilities into Mendix apps. It is compatible with [OpenAI's platform](https://platform.openai.com/) and [Microsoft Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-ai-foundry), where you can access OpenAI models. ### Features {#features} OpenAI provides market-leading LLM capabilities with GPT-4: -* Advanced reasoning – Follow complex instructions in natural language and solve difficult problems with accuracy. -* Creativity – Generate, edit, and iterate with end-users on creative and technical writing tasks, such as composing songs, writing screenplays, or learning an end-user’s writing style. -* Longer context – GPT-4 can handle over 25,000 words of text, allowing for use cases like long-form content creation, extended conversations, and document search and analysis. +* **Advanced reasoning** – Follows complex instructions in natural language and solves difficult problems with accuracy. +* **Creativity** – Generates, edits, and iterates with end-users on creative and technical writing tasks, such as composing songs, writing screenplays, or learning an end-user’s writing style. +* **Longer context** – GPT-4 can handle over 25,000 words of text, supporting use cases like long-form content creation, extended conversations, and document search and analysis. -Mendix provides support for [OpenAI](https://platform.openai.com/) and [Microsoft Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-ai-foundry) (formerly known as Azure OpenAI or Cognitive Services). Microsoft Foundry is Microsoft's unified AI platform that streamlines the creation and management of AI agents and models, including the OpenAI models. +Mendix supports [OpenAI](https://platform.openai.com/) and [Microsoft Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-ai-foundry) (formerly known as Azure OpenAI or Cognitive Services). Microsoft Foundry is Microsoft's unified AI platform that streamlines the creation and management of AI agents and models, including the OpenAI models. With the current version, Mendix supports the Chat Completions API for [text generation](https://platform.openai.com/docs/guides/text-generation), the Image Generations API for [images](https://platform.openai.com/docs/guides/images), the Embeddings API for [vector embeddings](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings), and indexes via [Azure AI Search](https://learn.microsoft.com/en-us/azure/search/) for knowledge base retrieval. -Typical use cases for generative AI are described in the [Typical LLM Use Cases](/agents/get-started/#llm-use-cases). - #### Knowledge Base -By integrating Azure AI Search, the OpenAI Connector enables knowledge base retrieval from Azure data sources. For Retrieval Augmented Generation (RAG) scenarios, chat completions with (Azure) OpenAI can also be combined with knowledge bases by other provider such as Mendix Cloud. +By integrating Azure AI Search, the OpenAI connector enables knowledge base retrieval from Azure data sources. For Retrieval Augmented Generation (RAG) scenarios, chat completions with (Azure) OpenAI can also be combined with knowledge bases from other providers such as Mendix Cloud. ### Prerequisites {#prerequisites} @@ -38,77 +36,77 @@ To use this connector, you need to either sign up for an [OpenAI account](https: ### Dependencies {#dependencies} -* Mendix Studio Pro version 10.24.0 or above +* Mendix Studio Pro 10.24.0 and above * [GenAI Commons module](/agents/genai-for-mx/commons/) * [Encryption module](/appstore/modules/encryption/) * [Community Commons module](/appstore/modules/community-commons-function-library/) ## Installation {#installation} - The following modules from the Marketplace need to be installed: +Install the following modules from Mendix Marketplace: -* [GenAI Commons](https://marketplace.mendix.com/link/component/239448) module -* [Encryption](https://marketplace.mendix.com/link/component/1011) module -* [Community Commons](https://marketplace.mendix.com/link/component/170) module +* [GenAI Commons](https://marketplace.mendix.com/link/component/239448) +* [Encryption](https://marketplace.mendix.com/link/component/1011) +* [Community Commons](https://marketplace.mendix.com/link/component/170) To import the OpenAI Connector into your app, follow the instructions in [Using Marketplace Content](/appstore/use-content/). ## Configuration {#configuration} -After you install the OpenAI Connector, you can find it in the **App Explorer**, in the **Marketplace Modules** section. The connector provides a domain model and several activities that you can use to connect your app to OpenAI. To implement an activity, use it in a microflow. To ensure that your app can connect to OpenAI, you must also [configure the Encryption module](/appstore/modules/encryption/#configuration). +After installing the OpenAI connector, you can find it in the **Marketplace Modules** section of **App Explorer**. The connector provides a domain model and several activities that you can use to connect your app to OpenAI. To implement an activity, use it in a microflow. Configure the [Encryption module](/appstore/modules/encryption/#configuration) to secure the connection between your app and OpenAI. ### General Configuration {#general-configuration} -1. Add the module role **OpenAIConnector.Administrator** to your Administrator user role in the security settings of your app. -2. Add the **Configuration_Overview** page (**USE_ME > Configuration**) to your navigation, or add the **Snippet_Configurations** to a page that is already part of your navigation. -3. Continue setting up your OpenAI configuration at runtime. Follow the instructions in either [OpenAI Configuration](#openai-configuration) or [Microsoft Foundry Configuration](#azure-openai-configuration), depending on which platform you are using. -4. Configure the models you need to use for your use case. +1. Add the module role **OpenAIConnector.Administrator** to your Administrator user role in the **Security** settings of your app. +2. Add the **Configuration_Overview** page (**USE_ME > Configuration**) to your navigation, or add **Snippet_Configurations** to a page that is already part of your navigation. +3. Continue setting up your OpenAI configuration at runtime. For more information, see [OpenAI Configuration](#openai-configuration) or [Microsoft Foundry Configuration](#azure-openai-configuration), depending on which platform you use. +4. Configure the models for your use case. -#### OpenAI Configuration {#openai-configuration} +#### OpenAI Configuration {#openai-configuration} -The following inputs are required for the OpenAI configuration: +OpenAI configuration requires the following inputs: | Parameter | Value | | ----------- | ------------------------------------------------------------ | -| Display name | This is the name identifier of a configuration (for example, *MyConfiguration*). | +| Display name | The name identifier of a configuration (for example, *MyConfiguration*). | | API type | Select `OpenAI`. | -| Endpoint | This is the API endpoint (for example, `https://api.openai.com/v1`) | -| Token | This is the access token to authorize your API call.
To get an API, follow these steps:
  1. Create an account and sign in at [OpenAI](https://platform.openai.com/).
  2. Go to the [API key page](https://platform.openai.com/account/api-keys) to create a new secret key.
  3. Copy the API key and save this somewhere safe.
| +| Endpoint | The API endpoint (for example, `https://api.openai.com/v1`). | +| Token | The access token to authorize your API call.
To get an API key, follow these steps:
  1. Create an account and sign in at [OpenAI](https://platform.openai.com/).
  2. Go to the [API key page](https://platform.openai.com/account/api-keys) to create a new secret key.
  3. Copy the API key and save this somewhere safe.
| -#### Microsoft Foundry Configuration {#azure-openai-configuration} +#### Microsoft Foundry Configuration {#azure-openai-configuration} -The following inputs are required for the Microsoft Foundry configuration: +Microsoft Foundry configuration requires the following inputs: | Parameter | Value | | -------------- | ------------------------------------------------------------ | -| Display name | This is the name identifier of a configuration (for example, *MyConfiguration*). | +| Display name | The name identifier of a configuration (for example, *MyConfiguration*). | | API type | Select `AzureOpenAI` for Microsoft Foundry deployments. | -| Endpoint | This is the API endpoint (for example, `https://your-resource-name.openai.azure.com/openai/deployments/`).
For details on how to obtain `your-resource-name`, see the [Obtaining Resource Name](#azure-resource-name) section below. | -| Azure key type | This is the type of token that is entered in the API key field. For Azure OpenAI, two types of keys are currently supported: Microsoft Entra token and API key.
For details on how to generate a Microsoft Entra access token, see [How to Configure Azure OpenAI Service with Managed Identities](https://learn.microsoft.com/en-gb/azure/ai-services/openai/how-to/managed-identity). Alternatively, if your organization allows it, you could use the Azure `api-key` authentication mechanism. For details on how to obtain an API key, see the [Obtaining API keys](#azure-api-keys) section below. For more information, see the [Technical Reference](#technical-reference) section. | -| Token / API key | This is the access token to authorize your API call. | +| Endpoint | The API endpoint (for example, `https://your-resource-name.openai.azure.com/openai/deployments/`).
For details on how to obtain `your-resource-name`, see the [Obtaining Resource Name](#azure-resource-name) section below. | +| Azure key type | The type of token entered in the API key field. For Azure OpenAI, two types of keys are currently supported: Microsoft Entra token and API key.
For details on generating a Microsoft Entra access token, see [How to Configure Azure OpenAI Service with Managed Identities](https://learn.microsoft.com/en-gb/azure/ai-services/openai/how-to/managed-identity). Alternatively, if your organization allows it, you can use the Azure `api-key` authentication mechanism. For details on obtaining an API key, see the [Obtaining API Keys](#azure-api-keys) section below. For more information, see the [Technical Reference](#technical-reference) section. | +| Token / API key | The access token to authorize your API call. | ##### Obtaining the Resource Name {#azure-resource-name} -1. Go to the [Microsoft Foundry portal](https://ai.azure.com/) and sign in. -2. Select the right resource in the upper right corner. -3. The home page should show **Resource configuration** where you can find the **Microsoft Foundry endpoint**. -4. Use the copy icon ({{% icon name="copy" %}}) and use it as your resource name in the endpoint URL. +1. Sign in to the [Microsoft Foundry portal](https://ai.azure.com/). +2. In the upper-right corner, select the resource. +3. On the home page, go to **Resource configuration** to find the **Microsoft Foundry endpoint**. +4. Click **Copy** ({{% icon name="copy" %}}) and use it as your resource name in the endpoint URL. ##### Obtaining API Keys {#azure-api-keys} -1. On the same page where the resource name is located, you can find your API key information. -2. You can now view ({{% icon name="view" %}}) and copy ({{% icon name="copy" %}}) the value of the **key1** or **key2** field as your API key while setting up the configuration. Note that these keys might not be visible for everyone in the portal, depending on your organization's security settings. +1. On the same page where the resource name is located, find your API key information. +2. View ({{% icon name="view" %}}) and copy ({{% icon name="copy" %}}) the value of the **key1** or **key2** field as your API key while setting up the configuration. Note that these keys might not be visible for everyone in the portal, depending on your organization's security settings. ##### Adding Azure AI Search Resources {#azure-ai-search} | Parameter | Value | | -------------- | ------------------------------------------------------------ | -| Display name | This is the name identifier of a Azure AI Search Resource (for example, *MySearchResource*). | -| Endpoint URL | This is the API endpoint (for example, `https://your-resource-name.search.windows.net`).
For details on how to obtain `your-resource-name`, see [Azure AI Search service in the Azure portal](https://learn.microsoft.com/en-us/azure/search/search-create-service-portal). | -| API version | This is the version of the REST API. | -| API key | This is the access token to authorize your API call. | +| Display name | The name identifier of an Azure AI Search Resource (for example, *MySearchResource*). | +| Endpoint URL | The API endpoint (for example, `https://your-resource-name.search.windows.net`).
For details on how to obtain `your-resource-name`, see [Azure AI Search service in the Azure portal](https://learn.microsoft.com/en-us/azure/search/search-create-service-portal). | +| API version | The version of the REST API. | +| API key | The access token to authorize your API call. | -After saving, the indexes in this resource will be automatically synced and displayed in the configuration page. They will all be separate indexes that can be added to the request when using Chat completions. +After saving, the indexes in this resource are automatically synced and displayed in the configuration page. All indexes can be added to the request when using Chat completions. {{% alert color="warning" %}} Currently, the only supported authorization method for Azure AI Search resources is the API key. @@ -116,81 +114,81 @@ Currently, the only supported authorization method for Azure AI Search resources #### Configuring the OpenAI Deployed Models -A [Deployed Model](/agents/genai-for-mx/commons/#deployed-model) represents a GenAI model instance that can be used by the app to generate text, embeddings, or images. For every model you want to invoke from your app, you need to create a `OpenAIDeployedModel` record, a specialization of `DeployedModel`. In addition to the model display name and a technical name/identifier, an OpenAI deployed model contains a reference to the additional connection details as configured in the previous step. For OpenAI, a set of common models can be created automatically using the designated button. If you want to use additional models that are made available by OpenAI you need to configure additional OpenAI deployed models in your Mendix app. For Microsoft Foundry, the model names can be different. The technical model names depend on the deployment names that were chosen while deploying the models in the [Microsoft Foundry portal](https://ai.azure.com/). Therefore in this case you always need to configure the deployed models manually in your Mendix app. +A [deployed model](/agents/genai-for-mx/commons/#deployed-model) represents a GenAI model instance that the app can use to generate text, embeddings, or images. For each model you want to invoke from your app, create an `OpenAIDeployedModel` record (a specialization of `DeployedModel`). In addition to the model display name and a technical name or identifier, an OpenAI deployed model contains a reference to the connection details configured in the previous step. For OpenAI, a set of common models can be created automatically using the designated button. To use additional models made available by OpenAI, configure additional OpenAI deployed models in your Mendix app. For Microsoft Foundry, the model names can be different. The technical model names depend on the deployment names chosen while deploying the models in the [Microsoft Foundry portal](https://ai.azure.com/). In this case, always configure the deployed models manually in your Mendix app. -1. If needed, click the three dots ({{% icon name="three-dots-menu-horizontal" %}}) icon for an OpenAI configuration to open the **Manage Deployed Models** pop-up. -2. For every additional model, add a record. The following fields are required: +1. If needed, click the three dots ({{% icon name="three-dots-menu-horizontal" %}}) icon for an OpenAI configuration to open the **Manage Deployed Models** dialog box. +2. For each additional model, add a record. The following fields are required: | Field | Description | | -------------- | ------------------------------------------------------------ | - | Display name | This is the reference to the model for app users in case they have to select which one is to be used. | - | Deployment name / Model name | This is the technical reference for the model. For OpenAI this is equal to the [model aliases](https://platform.openai.com/docs/models#current-model-aliases). For Microsoft Foundry this is the deployment name from the [Microsoft Foundry portal](https://ai.azure.com/). - | Output modality | Describes what the output of the model is. This connector currently supports Text, Embedding, and Image. - | Input modality | Describes what input modalities are accepted by the model. This connector currently supports Text and Image. - | Azure API version | Azure OpenAI only. This is the API version to use for this operation. It follows the `yyyy-MM-dd` format. For supported versions, see [Azure OpenAI documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). The supported versions can vary depending on the type of model, so make sure to look for the right section (such as Chat Completions, Image Generation, or Embeddings) on that page. | + | Display name | The reference for app users when selecting the model. | + | Deployment name / Model name | The technical reference of the model. For OpenAI, this equals the [model alias](https://platform.openai.com/docs/models#current-model-aliases). For Microsoft Foundry, this is the deployment name from the [Microsoft Foundry portal](https://ai.azure.com/). + | Output modality | The output of the model. This connector currently supports text, embeddings, and images. + | Input modality | The input modalities accepted by the model. This connector currently supports text and images. + | Azure API version | Azure OpenAI only. The API version to use for this operation. It follows the `yyyy-MM-dd` format. For supported versions, see [Azure OpenAI documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). The supported versions can vary depending on the type of model, so make sure to look for the right section (such as Chat Completions, Image Generation, or Embeddings) on that page. | -3. Close the popup and test the configuration with the newly created deployed models. +3. Close the dialog box and test the configuration with the newly created deployed models. -### Using GenAI Commons Operations {#genai-commons-operations} +### Using GenAI Commons Operations {#genai-commons-operations} -After following the general setup above, you are all set to use the microflow actions under the **GenAI (Generate)** category from the toolbox. These operations are part of GenAI Commons. Since OpenAI is compatible with the principles of GenAI Commons, you can pass an `OpenAIDeployedModel` to all GenAI Commons operations that expect the generalization `DeployedModel`. All actions under **GenAI (Generate)** will take care of executing the right provider-specific logic, based on the type of specialization passed, in this case OpenAI. From an implementation perspective, it is not needed to inspect the inner workings of this operation. The input, output, and behavior are as described in the [GenAICommons documentation](/agents/genai-for-mx/commons/#microflows). Applicable operations and some OpenAI-specific aspects are listed below. +After completing the general setup, you can use the microflow actions under **GenAI (Generate)** in the toolbox. These operations are part of GenAI Commons. Because OpenAI is compatible with the principles of GenAI Commons, you can pass an `OpenAIDeployedModel` to all GenAI Commons operations that expect the generalization `DeployedModel`. All actions under **GenAI (Generate)** execute the appropriate provider-specific logic based on the specialization type passed (in this case, OpenAI). From an implementation perspective, understanding the inner workings of this operation is not required. The [GenAI Commons documentation](/agents/genai-for-mx/commons/#microflows) describes the input, output, and behavior. Applicable operations and some OpenAI-specific aspects are listed below. -For more inspiration or guidance on how to use the microflow actions in your logic, Mendix recommends downloading our [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of examples that cover all the operations mentioned. +For more inspiration or guidance on how to use the microflow actions in your logic, download the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of examples covering all the operations mentioned. #### Chat Completions -Operations for chat completions focus on the generation of text based on a certain input. In this context, system prompts and user prompts are two key components that help guide the language model in generating relevant and contextually appropriate responses. For more information on the type of prompts and message roles, see the [ENUM_MessageRole](/agents/genai-for-mx/commons/#enum-messagerole) enumeration. To learn more about how to create the right prompts for your use case, see the prompt engineering links in the [Read More](#read-more) section. +Operations for chat completions focus on generating text based on input. In this context, system prompts and user prompts are two key components that guide the language model in generating relevant and contextually appropriate responses. For more information on prompt types and message roles, see the [ENUM_MessageRole](/agents/genai-for-mx/commons/#enum-messagerole) enumeration. To learn more about how to create the right prompts for your use case, see the prompt engineering links in the [Read More](#read-more) section. -The `OpenAIDeployedModel` is compatible with the two [Chat Completions operations from GenAI Commons](/agents/genai-for-mx/commons/#genai-generate). While developing your custom microflow, you can drag and drop the following operations from the toolbox in Studio Pro, see category **GenAI (Generate)**: +The `OpenAIDeployedModel` is compatible with the two [chat completions operations from GenAI Commons](/agents/genai-for-mx/commons/#genai-generate). While developing your microflow, drag and drop the following operations from the toolbox in Studio Pro under **GenAI (Generate)**: -* Chat Completions (with history) +* Chat Completions (with history) * Chat Completions (without history) -You can use the GenAI Commons toolbox actions to [create the required Request](/agents/genai-for-mx/commons/#genai-request-building) and [handle the Response](/agents/genai-for-mx/commons/#genai-response-handling) for your use case. +Use the GenAI Commons toolbox actions to [create the required request](/agents/genai-for-mx/commons/#genai-request-building) and [handle the response](/agents/genai-for-mx/commons/#genai-response-handling) for your use case. -The internal chat completion logic within the OpenAI connector supports [JSON mode](#chatcompletions-json-mode), [function calling](#chatcompletions-functioncalling), and [vision](#chatcompletions-vision). Make sure to check the actual compatibility of the available models with these functionalities, as this changes over time. Any specific OpenAI microflow actions from the toolbox are listed below. +The internal chat completion logic within the OpenAI connector supports [JSON mode](#chatcompletions-json-mode), [function calling](#chatcompletions-functioncalling), and [vision](#chatcompletions-vision). Check the compatibility of available models with these functionalities, as compatibility changes over time. Specific OpenAI microflow actions from the toolbox are listed below. #### JSON Mode {#chatcompletions-json-mode} -When JSON mode is used, the model is programmatically instructed to return valid JSON. For OpenAI you have to explicitly mention the necessity of a JSON structure in a message in the conversation, e.g. the system prompt. Additionally after creating the request, but before passing it to the chat completions operation, use the toolbox action `Set Response Format` to set the required response format to JSON. +JSON mode instructs the model to return valid JSON. To use JSON mode with OpenAI, explicitly specify that a JSON structure is required in a conversation message (for example, in the system prompt). After creating the request but before passing it to the chat completions operation, use the [Set Response Format](#set-responseformat-chat) toolbox action to set the response format to JSON. #### Function Calling {#chatcompletions-functioncalling} -Function calling enables LLMs to connect with external tools to gather information, execute actions, convert natural language into structured data, and much more. Function calling thus enables the model to intelligently decide when to let the Mendix app call one or more predefined function microflows to gather additional information to include in the assistant's response. +Function calling enables LLMs to connect with external tools to gather information, run actions, convert natural language into structured data, and more. Function calling enables the model to intelligently decide when to let the Mendix app call one or more predefined function microflows to gather additional information for the assistant's response. -OpenAI does not call the function. The model returns a tool called JSON structure that is used to build the input of the function (or functions) so that they can be executed as part of the chat completions operation. Functions in Mendix are essentially microflows that can be registered within the request to the LLM​. The OpenAI connector takes care of handling the tool call response as well as executing the function microflows until the API returns the assistant's final response. +OpenAI does not call the function. The model returns a tool called JSON structure that builds the input of the function (or functions) so they can run as part of the chat completions operation. Functions in Mendix are essentially microflows that can be registered within the request to the LLM. The OpenAI connector handles the tool call response and runs the function microflows until the API returns the assistant's final response. -This is all part of the implementation that is executed by the GenAI Commons chat completions operations mentioned before. As a developer, you have to make the system aware of your functions and what these do by registering the function(s) to the request. This is done using the GenAI Commons operation [Tools: Add Function to Request](/agents/genai-for-mx/commons/#add-function-to-request) once per function before passing the request to the chat completions operation. +The GenAI Commons chat completions operations mentioned earlier run this implementation. As a developer, you must make the system aware of your functions and their purposes by registering the functions to the request. To do so, use the GenAI Commons operation [Tools: Add Function to Request](/agents/genai-for-mx/commons/#add-function-to-request) once per function before passing the request to the chat completions operation. -Function microflows can have none, a single, or multiple primitive input parameters such as Boolean, Datetime, Decimal, Enumeration, Integer or String. Additionally, they may accept the [Request](/agents/genai-for-mx/commons/#request) or [Tool](/agents/genai-for-mx/commons/#tool) objects as inputs. The function microflow must return a String value. +Function microflows can have none, one, or multiple primitive input parameters such as Boolean, Datetime, Decimal, Enumeration, Integer, or String. They may also accept the [Request](/agents/genai-for-mx/commons/#request) or [Tool](/agents/genai-for-mx/commons/#tool) objects as inputs. The function microflow must return a string value. {{% alert color="warning" %}} -Function calling is a very powerful capability and should be used with caution. Note that function microflows run in the context of the current user without enforcing entity-access. You can use `$currentUser` in XPath queries to ensure you retrieve and return only information that the end-user is allowed to view; otherwise confidential information may become visible to the current end-user in the assistant's response. +Function calling is a powerful capability and should be used with caution. Function microflows run in the context of the current user without enforcing entity access. Use `$currentUser` in XPath queries to ensure you retrieve and return only information that the end-user is allowed to view; otherwise, confidential information may become visible to the end-user in the assistant's response. -Mendix also strongly advises that you build user confirmation logic into function microflows that have a potential impact on the world on behalf of the end-user. Some examples of such microflows include sending an email, posting online, or making a purchase. +Mendix recommends building user confirmation logic into function microflows that potentially impact the world on behalf of the end-user. Examples of such microflows include sending an email, posting online, or making a purchase. {{% /alert %}} For more information, see [Function Calling](/agents/function-calling/). #### Index {#chatcompletions-index} -Adding Azure indexes to a call enables LLMs to retrieve information when a related topics are mentioned. By including these indexes in the request object along with a name and description, enables the model to intelligently decide when to let the Mendix app call one or more predefined indexes. This allows the assistant to include the additional information in its response. +Adding Azure indexes to a call enables LLMs to retrieve information when related topics are mentioned. By including these indexes in the request object along with a name and description, the model can intelligently decide when to let the Mendix app call one or more predefined indexes, allowing the assistant to include additional information in its response. -OpenAI does not directly connect to the Azure AI Search resource. The model returns a tool called JSON structure that is used to build the input of the retrievals so that they can be executed as part of the chat completions operation. The OpenAI connector takes care of handling the tool call response as well as executing the function microflows until the API returns the assistant's final response. +OpenAI does not directly connect to the Azure AI Search resource. The model returns a tool called JSON structure that builds the input of the retrievals so they can run as part of the chat completions operation. The OpenAI connector handles the tool call response and runs the function microflows until the API returns the assistant's final response. -This functionality is part of the implementation executed by the GenAI Commons Chat Completions operations mentioned earlier. As a developer, you need to make the system aware of your indexes and their purpose by registering them with the request. This is done using the GenAI Commons operation [Tools: Add Knowledge Base](/agents/genai-for-mx/commons/#add-knowledge-base-to-request), which must be called once per index before passing the request to the Chat Completions operation. +The GenAI Commons chat completions operations mentioned earlier run this functionality. As a developer, make the system aware of your indexes and their purpose by registering them with the request. Use the GenAI Commons operation [Tools: Add Knowledge Base](/agents/genai-for-mx/commons/#add-knowledge-base-to-request), which must be called once per index before passing the request to the chat completions operation. -Note that the retrieval process is independent of the model provider and can be used with any model that supports function calling, as it relies on the generalized `GenAICommons.ConsumedKnowledgeBase`entity. For Azure indexes specifically, as part of this module, when collection identifiers needs to be passed to operations, the `Name` of the `Index` should be used. +Note that the retrieval process is independent of the model provider and can be used with any model that supports function calling, as it relies on the generalized `GenAICommons.ConsumedKnowledgeBase`entity. For Azure indexes specifically, as part of this module, when collection identifiers need to be passed to operations, the `Name` of the `Index` should be used. #### Vision {#chatcompletions-vision} -Vision enables models like GPT-4o and GPT-4 Turbo to interpret and analyze images, allowing them to answer questions and perform tasks related to visual content. This integration of computer vision and language processing enhances the model's comprehension and makes it valuable for tasks involving visual information. To make use of vision inside the OpenAI connector, an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or multiple images must be sent along with a single message. +Vision enables models like GPT-4o and GPT-4 Turbo to interpret and analyze images, allowing them to answer questions and perform tasks related to visual content. This integration of computer vision and language processing enhances the model's comprehension and makes it valuable for tasks involving visual information. To use vision inside the OpenAI connector, an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or multiple images must be sent along with a single message. -For `Chat Completions without History`, `FileCollection` is an optional input parameter. +For `Chat Completions without History`, `FileCollection` is an optional input parameter. For `Chat Completions with History`, `FileCollection` can optionally be added to individual user messages using [Chat: Add Message to Request](/agents/genai-for-mx/commons/#chat-add-message-to-request). -Use the two OpenAI-specific microflow actions from the toolbox [Files: Initialize Collection with OpenAI File](#initialize-filecollection) and [Files: Add OpenAIFile to Collection](#add-file) to construct the input with either `FileDocuments` (for vision, it needs to be of type `Image`) or `URLs`. There are similar file operations exposed by the GenAI commons module that can be used for vision requests with the OpenAIConnector; however, these generic operations do not support the optional OpenAI-specific `Detail` attribute. +Use the two OpenAI-specific microflow actions from the toolbox [Files: Initialize Collection with OpenAI File](#initialize-filecollection) and [Files: Add OpenAIFile to Collection](#add-file) to construct the input with either `FileDocuments` (for vision, it needs to be of type `Image`) or `URLs`. There are similar file operations exposed by the GenAI Commons module that can be used for vision requests with the OpenAI Connector; however, these generic operations do not support the optional OpenAI-specific `Detail` attribute. {{% alert color="info" %}} OpenAI and Microsoft Foundry do not necessarily provide feature parity across all models when it comes to combining functionalities. In other words, Microsoft Foundry does not support the use of JSON mode and function calling in combination with image (vision) input for certain models, so make sure to check the [Azure Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models). @@ -202,11 +200,11 @@ For more information on vision, see [OpenAI](https://platform.openai.com/docs/gu #### Document Chat {#chatcompletions-document} -Document chat enables the model to interpret and analyze PDF documents, allowing it to answer questions and perform tasks based on the document content. To use document chat, you can send an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or more documents along with a single message. +Document chat enables the model to interpret and analyze PDF documents, allowing it to answer questions and perform tasks based on the document content. To use document chat, send an optional [FileCollection](/agents/genai-for-mx/commons/#filecollection) containing one or more documents along with a single message. -For [Chat Completions (without history)](/agents/genai-for-mx/commons/#chat-completions-without-history), `OptionalFileCollection` is an optional input parameter. For [Chat completions (with history)](/agents/genai-for-mx/commons/#chat-completions-with-history), a `FileCollection` can optionally be added to individual user messages using [Add Message to Request](/agents/genai-for-mx/commons/#chat-add-message-to-request). +For [Chat Completions (without history)](/agents/genai-for-mx/commons/#chat-completions-without-history), `OptionalFileCollection` is an optional input parameter. For [Chat Completions (with history)](/agents/genai-for-mx/commons/#chat-completions-with-history), a `FileCollection` can optionally be added to individual user messages using [Add Message to Request](/agents/genai-for-mx/commons/#chat-add-message-to-request). -You can send up to 100 pages across multiple files, with a maximum combined size of 32 MB per conversation. Currently, processing multiple files with OpenAI is not always guaranteed and can lead to unexpected behavior (for example, only one file being processed). +You can send up to 100 pages across multiple files, with a maximum combined size of 32 MB per conversation. Processing multiple files with OpenAI is not always guaranteed and can lead to unexpected behavior (for example, only one file being processed). {{% alert color="info" %}} Microsoft Foundry does not currently support file input. @@ -216,37 +214,38 @@ Note that the model uses the file name when analyzing documents, which may intro #### Image Generations {#image-generations-configuration} -OpenAI also provides image generation capabilities which can be invoked using this connector module. The `OpenAIDeployedModel` entity is compatible with the [image generation operation from GenAI Commons](/agents/genai-for-mx/commons/#generate-image). +OpenAI provides image generation capabilities that can be invoked using this connector module. The `OpenAIDeployedModel` entity is compatible with the [image generation operation from GenAI Commons](/agents/genai-for-mx/commons/#generate-image). -To implement image generation into your Mendix application, you can use the Image generation microflow action from GenAI Commons directly. When developing your microflow, you can drag and drop it from the toolbox: find it under the **GenAI (Generate)** category in the **Toolbox** in Mendix Studio Pro: +To implement image generation into your Mendix application, use the Image generation microflow action from GenAI Commons directly. When developing your microflow, drag and drop it from the toolbox under **GenAI (Generate)** in **Toolbox** in Studio Pro: * Generate Image -When you drag this operation into your app microflow logic, use the `user prompt` to describe the desired image, and for the `DeployedModel` pass the relevant `OpenAIDeployedModel` that supports image generation. Additional parameters like the height and the width can be configured using [Image Generation: Create ImageOptions](/agents/genai-for-mx/commons/#imageoptions-create). To configure OpenAI-specific options, like quality and style an extension to the ImageOptions can be added using [Image Generation: Set ImageOptions Extension](#set-imageoptions-extension). +When you drag this operation into your app microflow logic, use the `user prompt` to describe the desired image, and for the `DeployedModel` pass the relevant `OpenAIDeployedModel` that supports image generation. Additional parameters like height and width can be configured using [Image Generation: Create ImageOptions](/agents/genai-for-mx/commons/#imageoptions-create). To configure OpenAI-specific options like quality and style, an extension to the ImageOptions can be added using [Image Generation: Set ImageOptions Extension](#set-imageoptions-extension). -A generated image needs to be stored in a custom entity that inherits from the `System.Image` entity. The `Response` from the single image operation can be processed using [Get Generated Image (Single)](/agents/genai-for-mx/commons/#image-get-single) to store the image in your custom `Image` entity. +A generated image must be stored in a custom entity that inherits from the `System.Image` entity. The `Response` from the single image operation can be processed using [Get Generated Image (Single)](/agents/genai-for-mx/commons/#image-get-single) to store the image in your custom `Image` entity. #### Embeddings Generation {#embeddings-configuration} -OpenAI also provides vector embedding generation capabilities which can be invoked using this connector module. The `OpenAIDeployedModel` entity is compatible with the [knowledge base operations](/agents/genai-for-mx/commons/#genai-knowledgebase-content) from the GenAI Commons. +OpenAI provides vector embedding generation capabilities that can be invoked using this connector module. The `OpenAIDeployedModel` entity is compatible with the [knowledge base operations](/agents/genai-for-mx/commons/#genai-knowledgebase-content) from GenAI Commons. -In order to implement embeddings generation into your Mendix application, you can use the Embedding generation microflow actions from GenAI Commons directly. When developing your microflow, you can drag and drop the one you need from the toolbox: find it under the **GenAI (Generate)** category in the **Toolbox** in Mendix Studio Pro: +To implement embeddings generation into your Mendix application, use the Embedding generation microflow actions from GenAI Commons directly. When developing your microflow, drag and drop the one you need from the toolbox under **GenAI (Generate)** in **Toolbox** in Studio Pro: * Generate Embeddings (String) * Generate Embeddings (Chunk Collection) -Depending on the operation you use in the microflow, an `InputText` String or a [ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection) needs to be provided. The current version of this operation only supports the float representation of the resulting vector. +Depending on the operation you use in the microflow, an `InputText` String or a [ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection) must be provided. The current version of this operation only supports the float representation of the resulting vector. + +The microflow action `Generate Embeddings (String)` supports scenarios where the vector embedding of a single string must be generated (for example, to use for a nearest neighbor search across an existing knowledge base). This input string can be passed directly as the `InputText` parameter of this microflow. [EmbeddingsOptions](/agents/genai-for-mx/commons/#embeddingsoptions-entity) is optional and can be instantiated using [Embeddings: Create EmbeddingsOptions](/agents/genai-for-mx/commons/#embeddingsoptions-create) from GenAI Commons. Use the GenAI Commons toolbox action [Embeddings: Get First Vector from Response](/agents/genai-for-mx/commons/#embeddings-get-first-vector) to retrieve the generated embeddings vector. Both operations can be found under **GenAI Knowledge Base (Content)** in **Toolbox** in Studio Pro. -The microflow action `Generate Embeddings (String)` supports scenarios where the vector embedding of a single string must be generated, e.g. to use for a nearest neighbor search across an existing knowledge base. This input string can be passed directly as the `InputText` parameter of this microflow. Additionally, [EmbeddingsOptions](/agents/genai-for-mx/commons/#embeddingsoptions-entity) is optional and can be instantiated using [Embeddings: Create EmbeddingsOptions](/agents/genai-for-mx/commons/#embeddingsoptions-create) from GenAI Commons. Use the GenAI Commons toolbox action [Embeddings: Get First Vector from Response](/agents/genai-for-mx/commons/#embeddings-get-first-vector) to retrieve the generated embeddings vector. Both mentioned operations can be found under **GenAI Knowledge Base (Content)** in the **Toolbox** in Mendix Studio Pro. +The microflow action `Generate Embeddings (Chunk Collection)` supports the more complex scenario where a collection of string inputs is vectorized in a single API call, such as when converting a collection of texts (chunks) into embeddings to be inserted into a knowledge base. Instead of calling the API for each string, executing a single call for a list of strings can significantly reduce HTTP overhead. Use the exposed microflows of GenAI Commons [Chunks: Initialize ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-create) to create the wrapper and [Chunks: Add Chunk to ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-add-chunk) or [Chunks: Add KnowledgeBaseChunk to ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) to construct the input. The resulting embedding vectors returned after a successful API call are stored in the `EmbeddingVector` attribute in the same `Chunk` object. -The microflow action `Generate Embeddings (Chunk Collection)` supports the more complex scenario where a collection of string inputs is vectorized in a single API call, such as when converting a collection of texts (chunks) into embeddings to be inserted into a knowledge base. Instead of calling the API for each string, executing a single call for a list of strings can significantly reduce HTTP overhead. Use the exposed microflows of GenAI Commons [Chunks: Initialize ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-create) to create the wrapper and [Chunks: Add Chunk to ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-add-chunk), or [Chunks: Add KnowledgeBaseChunk to ChunkCollection](/agents/genai-for-mx/commons/#chunkcollection-add-knowledgebasechunk) to construct the input. The resulting embedding vectors returned after a successful API call will be stored in the `EmbeddingVector` attribute in the same `Chunk` object. \ -Purely to generate embeddings, it does not matter whether the ChunkCollection contains Chunks or its specialization KnowledgeBaseChunks. However, if the end goal is to store the generated embedding vectors in a knowledge base (e.g. using the [PgVector Knowledge Base](/appstore/modules/pgvector-knowledge-base/) module), then Mendix recommends adding `KnowledgeBaseChunks` to the `ChunkCollection` and using these as an input for the embeddings operations, so they can afterward directly be used to populate the knowledge base with. +To generate embeddings, it does not matter whether the ChunkCollection contains Chunks or its specialization KnowledgeBaseChunks. However, if the goal is to store the generated embedding vectors in a knowledge base (for example, using the [PgVector Knowledge Base](/appstore/modules/pgvector-knowledge-base/) module), Mendix recommends adding `KnowledgeBaseChunks` to the `ChunkCollection` and using these as an input for the embeddings operations, so they can afterward be used directly to populate the knowledge base. -Note that currently, the OpenAI connector does not support knowledge base interaction (e.g. inserting or retrieving chunks). For more information on possible ways to work with knowledge bases when using the OpenAI Connector for embedding generation, read more about [PgVector Knowledge Base](/appstore/modules/pgvector-knowledge-base/) and [setting up a Vector Database](/agents/reference-guide/external-connectors/pgvector-setup/). +Currently, the OpenAI connector does not support knowledge base interaction (for example, inserting or retrieving chunks). For more information on possible ways to work with knowledge bases when using the OpenAI Connector for embedding generation, see [PgVector Knowledge Base](/appstore/modules/pgvector-knowledge-base/) and [Setting Up a Vector Database](/agents/reference-guide/external-connectors/pgvector-setup/). -### Exposed Microflow Actions for (Azure) OpenAI {#exposed-microflows} +### Exposed Microflow Actions for OpenAI {#exposed-microflows} -OpenAI-specific exposed microflow actions to construct requests via drag-and-drop are listed below. These microflows can be found in the **Toolbox** in Studio Pro. Note that using these flows is only required if you need to add options to the request that are specific to OpenAI. For the generic part can use the GenAI Commons toolbox actions to [create the required Request](/agents/genai-for-mx/commons/#genai-request-building) and [handle the Response](/agents/genai-for-mx/commons/#genai-response-handling), which can be found under the **GenAI (Request Building)** and **GenAI (Response Handling)** categories in the Toolbox. +OpenAI-specific exposed microflow actions to construct requests via drag-and-drop are listed below. These microflows can be found in **Toolbox** in Studio Pro. Using these flows is only required if you need to add options to the request that are specific to OpenAI. For the generic part, use the GenAI Commons toolbox actions to [create the required Request](/agents/genai-for-mx/commons/#genai-request-building) and [handle the Response](/agents/genai-for-mx/commons/#genai-response-handling), which can be found under **GenAI (Request Building)** and **GenAI (Response Handling)** in the Toolbox. #### Set Response Format {#set-responseformat-chat} @@ -271,13 +270,13 @@ The module includes technical reference documentation for the available entities The **Documentation** pane displays the documentation for the currently selected element. To view it, perform the following steps: 1. In the [View menu](/refguide/view-menu/) of Studio Pro, select **Documentation**. -2. Click the element for which you want to view the documentation. +2. Click the element you want to view documentation for. {{< figure src="/attachments/appstore/platform-supported-content/modules/technical-reference/doc-pane.png" alt="" >}} ### Tool Choice -All [tool choice types](/agents/genai-for-mx/commons/#enum-toolchoice) of GenAI Commons for the [Tools: Set Tool Choice](/agents/genai-for-mx/commons/#set-toolchoice) action are supported. For API mapping reference, see the table below: +All [tool choice types](/agents/genai-for-mx/commons/#enum-toolchoice) from GenAI Commons for the [Tools: Set Tool Choice](/agents/genai-for-mx/commons/#set-toolchoice) action are supported. For API mapping reference, see the table below: | GenAI Commons (Mendix) | OpenAI | | -----------------------| ------- | @@ -288,21 +287,21 @@ All [tool choice types](/agents/genai-for-mx/commons/#enum-toolchoice) of GenAI ### Knowledge Base Retrieval -When adding a [KnowledgeBaseRetrieval](/agents/genai-for-mx/commons/#add-knowledge-base-to-request) object to your request, there are some optional parameters. Currently, only the MaxNumberOfResults parameter can be added to the search call and the others (`MinimumSimilarity` and `MetadataCollection`) are not compatible with the OpenAI Connector. +When adding a [KnowledgeBaseRetrieval](/agents/genai-for-mx/commons/#add-knowledge-base-to-request) object to your request, there are some optional parameters. Currently, only the `MaxNumberOfResults` parameter can be added to the search call. The others (`MinimumSimilarity` and `MetadataCollection`) are not compatible with the OpenAI Connector. -## GenAI showcase Application {#showcase-application} +## GenAI Showcase App {#showcase-application} -For more inspiration or guidance on how to use those microflows in your logic, Mendix recommends downloading the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of example use cases. +For more inspiration or guidance on how to use these microflows in your logic, download the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475), which demonstrates a variety of example use cases. {{% alert color="info" %}} -Some examples demonstrate knowledge base interaction and require a connection to a vector database. For more information on these concepts, see [Retrieval Augmented Generation (RAG)](/agents/rag/) +Some examples demonstrate knowledge base interaction and require a connection to a vector database. For more information on these concepts, see [Retrieval Augmented Generation (RAG)](/agents/rag/). {{% /alert %}} ## Troubleshooting {#troubleshooting} ### Outdated JDK Version Causing Errors while Calling a REST API {#outdated-jdk-version} -The Java Development Kit (JDK) is a framework needed by Mendix Studio Pro to deploy and run applications. For more information, see [Studio Pro System Requirements](/refguide/system-requirements/). Usually, the correct JDK version is installed during the installation of Studio Pro, but in some cases, it may be outdated. An outdated version can cause exceptions when calling REST-based services with large data volumes, like for example embeddings operations or chat completions with vision. +The Java Development Kit (JDK) is a framework needed by Studio Pro to deploy and run applications. For more information, see [Studio Pro System Requirements](/refguide/system-requirements/). Usually, the correct JDK version is installed during the installation of Studio Pro, but in some cases, it may be outdated. An outdated version can cause exceptions when calling REST-based services with large data volumes, such as embeddings operations or chat completions with vision. Mendix has seen the following two exceptions when using JDK versions below `jdk-11.0.5.0-hotspot`: `java.net.SocketException - Connection reset` or @@ -314,10 +313,10 @@ To check your JDK version and update it if necessary, follow these steps: 2. Go to [Eclipse Temurin JDK 11](https://adoptium.net/en-GB/temurin/releases/?variant=openjdk11&os=windows&package=jdk) and download the `.msi` file of the latest release of **JDK 11**. 3. Open the downloaded file and follow the installation steps. Remember the installation path. Usually, this should be something like `C:/Program Files/Eclipse Adoptium/jdk-11.0.22.7-hotspot`. 4. After the installation has finished, restart your computer if prompted. -5. Open Studio Pro and go to **Edit** > **Preferences** > **Deployment** > **JDK directory**. Click **Browse** and select the folder with the new JDK version you just installed. This should be the folder containing the *bin* folder. Save your settings by clicking **OK**. -6. Run the project and execute the action that threw the above-mentioned exception earlier. - 1. You might get an error saying `FAILURE: Build failed with an exception. The supplied javaHome seems to be invalid. I cannot find the java executable.` In this case, verify that you have selected the correct JDK directory containing the updated JDK version. - 2. You may also need to update Gradle. To do this, go to **Edit** > **Preferences** > **Deployment** > **Gradle directory**. Click **Browse** and select the appropriate Gradle version from the Mendix folder. For Mendix 10.10 and above, use Gradle 8.5. For Mendix 10 versions below 10.10, use Gradle 7.6.3. Then save your settings by clicking **OK**. +5. Open Studio Pro and go to **Edit** > **Preferences** > **Deployment** > **JDK directory**. Click **Browse** and select the folder with the new JDK version you just installed. This should be the folder containing the *bin* folder. Click **OK** to save your settings. +6. Run the project and execute the action that threw the exception. + 1. If you get an error saying `FAILURE: Build failed with an exception. The supplied javaHome seems to be invalid. I cannot find the java executable.`, verify that you have selected the correct JDK directory containing the updated JDK version. + 2. You may also need to update Gradle. To do this, go to **Edit** > **Preferences** > **Deployment** > **Gradle directory**. Click **Browse** and select the appropriate Gradle version from the Mendix folder. For Studio Pro 10.10 and above, use Gradle 8.5. For Studio Pro 10 versions below 10.10, use Gradle 7.6.3. Click **OK** to save your settings. 3. Rerun the project. @@ -335,15 +334,15 @@ Microsoft Foundry does not support the use of JSON mode and function calling in ### Chat Completions with Vision Response is Cut Off (Microsoft Foundry) -When you use Microsoft Foundry, it is recommended to set the optional `MaxTokens` input parameter; otherwise, the response may be cut off. For more details, see the [Microsoft Foundry Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision?tabs=rest%2Csystem-assigned%2Cresource#call-the-chat-completion-apis). +When using Microsoft Foundry, Mendix recommends setting the optional `MaxTokens` input parameter; otherwise, the response may be cut off. For more details, see the [Microsoft Foundry Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision?tabs=rest%2Csystem-assigned%2Cresource#call-the-chat-completion-apis). -### Attribute or Reference Required Error Message After Upgrade +### Attribute or Reference Required Error Message After Upgrade -If you encounter an error stating that an attribute or a reference is required after an upgrade, first upgrade all modules by right-clicking the error, then upgrade Data Widgets. +If you encounter an error stating that an attribute or reference is required after an upgrade, right-click the error to upgrade all modules. Then upgrade Data Widgets. ### Conflicted Lib Error After Module Import -If you encounter an error caused by conflicting Java libraries, such as `java.lang.NoSuchMethodError: 'com.fasterxml.jackson.annotation.OptBoolean com.fasterxml.jackson.annotation.JsonProperty.isRequired()'`, try synchronizing all dependencies (**App** > **Synchronize dependencies**) and then restart your application. +If you encounter an error caused by conflicting Java libraries, such as `java.lang.NoSuchMethodError: 'com.fasterxml.jackson.annotation.OptBoolean com.fasterxml.jackson.annotation.JsonProperty.isRequired()'`, synchronize all dependencies (**App** > **Synchronize dependencies**) and restart your application. ## Read More {#read-more} diff --git a/content/en/docs/genai/reference-guide/external-platforms/pg-vector-knowledge-base/_index.md b/content/en/docs/genai/reference-guide/external-platforms/pg-vector-knowledge-base/_index.md index a41bef919df..b733fb584a2 100644 --- a/content/en/docs/genai/reference-guide/external-platforms/pg-vector-knowledge-base/_index.md +++ b/content/en/docs/genai/reference-guide/external-platforms/pg-vector-knowledge-base/_index.md @@ -2,7 +2,7 @@ title: "PgVector Knowledge Base" url: /agents/reference-guide/external-connectors/pgvector/ linktitle: "PgVector Knowledge Base" -description: "Describes the configuration and usage of the PgVector Knowledge Base module from the Mendix Marketplace. This module allows developers to integrate PostgreSQL databases with pgvector installed as knowledge bases into their Mendix app." +description: "Describes how to configure and use the PgVector Knowledge Base module to integrate PostgreSQL databases with pgvector installed as knowledge bases." weight: 70 aliases: - /appstore/modules/pgvector-knowledge-base/ @@ -12,13 +12,13 @@ aliases: ## Introduction {#introduction} -The [PgVector Knowledge Base](https://marketplace.mendix.com/link/component/225063) module contains operations to interact with a PostgreSQL database that has the [pgvector](https://github.com/pgvector/pgvector?tab=readme-ov-file#pgvector) extension installed. It lets you easily store vectors and perform cosine similarity calculations from your Mendix app. This way, you can leverage knowledge bases to enhance your app functionality by performing operations based on (embedding) vectors and vector similarity. In the context of generative AI, large language models (LLMs), and embeddings, this is a key component in natural language processing (NLP) patterns such as retrieval augmented generation (RAG), recommendation algorithms, and similarity search operations. +The [PgVector Knowledge Base](https://marketplace.mendix.com/link/component/225063) module contains operations to interact with a PostgreSQL database that has the [pgvector](https://github.com/pgvector/pgvector?tab=readme-ov-file#pgvector) extension installed. It lets you easily store vectors and perform cosine similarity calculations from your Mendix app. You can use knowledge bases to enhance your app functionality by performing operations based on (embedding) vectors and vector similarity. In the context of generative AI, large language models (LLMs), and embeddings, this is a key component in natural language processing (NLP) patterns such as retrieval augmented generation (RAG), recommendation algorithms, and similarity search operations. ### Typical Use Cases {#use-cases} This module is particularly powerful for Mendix apps that use large language models in generative AI contexts. The PgVector Knowledge Base module allows these apps to securely use private company data in the app logic. For example, this might be essential when constructing prompts. -When there is a need for a separate private knowledge base outside of the LLM infrastructure, this module provides a low-code way to store discrete pieces of data (commonly refered to as **chunks**) in the private knowledge base and retrieve relevant information for end-user actions or app processes. +When you need a separate private knowledge base outside of the LLM infrastructure, this module provides a low-code way to store discrete pieces of data (commonly referred to as chunks) in the private knowledge base and retrieve relevant information for end-user actions or app processes. {{% alert color="info" %}} Check out the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475) to see example implementations, including retrieval augmented generation and semantic search with knowledge bases. @@ -26,19 +26,19 @@ Check out the [GenAI Showcase App](https://marketplace.mendix.com/link/component #### Retrieval Augmented Generation {#use-cases-rag} -A common NLP pattern is retrieval augmented generation (RAG), where the goal is to have LLMs construct answers to questions or provide on-demand information about private knowledge base data. In order to make this work, discrete pieces of information from the knowledge base are sent along with user questions to the LLM. The retrieval operations from this module are designed for this step in such use cases. +A common NLP pattern is retrieval augmented generation (RAG), where the goal is to have LLMs construct answers to questions or provide on-demand information about private knowledge base data. To make this work, discrete pieces of information from the knowledge base are sent along with user questions to the LLM. The retrieval operations from this module are designed for this step in such use cases. #### Semantic Search {#use-cases-semantic-search} -Also without invoking LLMs directly with the retrieved information, the similarity search logic from the retrieval operation can be leveraged in combination with embedding models to create a semantic search in a Mendix app. This can be used for fuzzy search capabilities, suggestions, or simple recommendation systems. +Even without invoking LLMs directly with the retrieved information, the similarity search logic from the retrieval operation can be used in combination with embedding models to create a semantic search in a Mendix app. This can be used for fuzzy search capabilities, suggestions, or simple recommendation systems. ### Features {#features} -With the current version, Mendix supports inserting data chunks with their vectors into a knowledge base (population) and selecting those records from that moment onwards (retrieval). Apart from cosine similarity search, which is executed based on the vector only, custom filtering is possible using key-value labeling (metadata) to support an additional traditional search component. +The current version supports inserting data chunks with their vectors into a knowledge base (population) and selecting those records from that moment onwards (retrieval). In addition to cosine similarity search, which is executed based on the vector only, custom filtering is possible using key-value labeling (metadata) to support an additional traditional search component. ### Prerequisites {#prerequisites} -You should have access to your own (remote) PostgreSQL database server with the [pgvector](https://github.com/pgvector/pgvector) extension installed. For more information, see the [Setting up a Vector Database](/agents/reference-guide/external-connectors/pgvector-setup/) section. +You must have access to your own (remote) PostgreSQL database server with the [pgvector](https://github.com/pgvector/pgvector) extension installed. For more information, see [Setting up a Vector Database](/agents/reference-guide/external-connectors/pgvector-setup/). {{% alert color="info" %}}This module cannot be used with the Mendix Cloud app database. It only works if you are using your own database server or Amazon RDS.{{% /alert %}} @@ -60,7 +60,7 @@ After you install the PgVector Knowledge Base module, you can find it in the **A ### General Configuration {#general-configuration} -You must perform the following steps to integrate a Mendix app integrate a PgVector knowledge base: +To integrate a PgVector knowledge base into a Mendix app, perform the following steps: 1. Add the module role **PgVectorKnowledgeBase.Administrator** to your Administrator user role in the security settings of your app. Optionally, map **GenAICommons.User** to any user roles that need read access directly on retrieved entities. 2. Add the **DatabaseConfiguration_Overview** page (**USE_ME > Configuration**) to your navigation, or add the **Snippet_DatabaseConfigurations** to a page that is already part of your navigation. @@ -72,17 +72,17 @@ It is possible to have multiple knowledge bases in the same database in parallel ### General Operations {#general-operations-configuration} -After following the general setup above, you are all set to use the microflows and Java actions in the **USE_ME > Operations** folder in your logic. Currently, eleven operations (microflows and Java actions) are exposed as microflow actions under the **PgVector Knowledge Base** category in the **Toolbox** in Mendix Studio Pro. These can be split into three categories, corresponding to the main functionalities: managing data chunks in the knowledge base (for example, [(Re)populate](#repopulate-knowledge-base)), finding relevant data chunks in an existing knowledge base (for example, [Retrieve](#retrieve)), and deleting chunk data or a whole knowledge base (for exapmle, [Delete Knowledge Base](#delete-knowledge-base)). In many occasions, metadata in a [MetadataCollection](/agents/genai-for-mx/commons/#metadatacollection-entity) can be provided to enable additional filtering. +After completing the general setup above, you can use the microflows and Java actions in the **USE_ME > Operations** folder in your logic. Currently, 11 operations (microflows and Java actions) are exposed as microflow actions under the **PgVector Knowledge Base** category in the **Toolbox** in Studio Pro. These can be split into three categories corresponding to the main functionalities: managing data chunks in the knowledge base (for example, [(Re)populate](#repopulate-knowledge-base)), finding relevant data chunks in an existing knowledge base (for example, [Retrieve](#retrieve)), and deleting chunk data or a whole knowledge base (for example, [Delete Knowledge Base](#delete-knowledge-base)). In many occasions, metadata in a [MetadataCollection](/agents/genai-for-mx/commons/#metadatacollection-entity) can be provided to enable additional filtering. Additionally, there is one activity to prepare the connection input, which is a required input parameter for all operations and exposed separately in the **Toolbox** in Studio Pro. The following section describes this operation: #### `DeployedKnowledgeBase: Create` {#create-pgvectordeployedknowledgebase} -All operations that include knowledge base interaction need the connection details to the knowledge base. Adhering to the GenAI Commons standard, this information is conveyed in a specialization of the GenAI Commons [DeployedKnowledgeBase](/agents/genai-for-mx/commons/#deployed-knowledge-base) entity and the [ConsumedKnowledgeBase](/agents/genai-for-mx/commons/#consumed-knowledge-base) (see the [Technical Reference](#technical-reference) section). After instantiating the `PgVectorKnowledgeBase` based on custom logic and/or front-end logic, this object can be used for the actual knowledge base operations. For operations where collection identifiers are needed in combination with a `ConsumedKnowledgeBase` object, the `Name` of the KnowledgeBase (see the `PgVectorKnowledgeBase` entity) needs to be passed as string. +All operations that include knowledge base interaction need the connection details to the knowledge base. This information is conveyed in a specialization of the GenAI Commons [DeployedKnowledgeBase](/agents/genai-for-mx/commons/#deployed-knowledge-base) entity and the [ConsumedKnowledgeBase](/agents/genai-for-mx/commons/#consumed-knowledge-base) (see the [Technical Reference](#technical-reference) section). After instantiating the `PgVectorKnowledgeBase` based on custom logic or front-end logic, you can use this object for the actual knowledge base operations. For operations where collection identifiers are needed in combination with a `ConsumedKnowledgeBase` object, the `Name` of the KnowledgeBase (see the `PgVectorKnowledgeBase` entity) must be passed as string. ### (Re)populate Operations {#repopulate-operations-configuration} -In order to add data to the knowledge base, you need to have discrete pieces of information and create knowledge base chunks for those. You can use the [operations for Chunks and KnowledgeBaseChunks in the GenAI Commons module](/agents/genai-for-mx/commons/#genai-knowledgebase-content). After you create the knowledge base chunks and [generate embedding vectors for them](/agents/genai-for-mx/commons/#add-knowledge-base-to-request), the resulting `ChunkCollection` can be inserted into the knowledge base using an operation for insertion, for example the `(Re)populate Knowledge Base` operation. +To add data to the knowledge base, you need to have discrete pieces of information and create knowledge base chunks for those. You can use the [operations for Chunks and KnowledgeBaseChunks in the GenAI Commons module](/agents/genai-for-mx/commons/#genai-knowledgebase-content). After you create the knowledge base chunks and [generate embedding vectors for them](/agents/genai-for-mx/commons/#add-knowledge-base-to-request), the resulting `ChunkCollection` can be inserted into the knowledge base using an operation for insertion, for example, the `(Re)populate Knowledge Base` operation. A typical pattern for populating a knowledge base is as follows: @@ -107,15 +107,15 @@ The population handles a whole collection of chunks at once, and this `ChunkColl #### `Insert` {#insert} -In cases where additional records need to be added to an existing knowledge base, the `Insert` operation can be used. This operation handles a collection of chunks that need to be inserted into the knowledge base. It behaves similarly to the [(Re)populate](#repopulate-knowledge-base) operation, except that it does not delete any data. +When additional records need to be added to an existing knowledge base, you can use the `Insert` operation. This operation handles a collection of chunks that need to be inserted into the knowledge base. It behaves similarly to the [(Re)populate](#repopulate-knowledge-base) operation, except that it does not delete any data. #### `Replace` {#replace} -The `Replace` operation is intended to be used in scenarios in which the chunks in the knowledge base are related to Mendix objects (in other words, data in the Mendix database). It can be used to keep the knowledge base in sync when data in your Mendix app database changes, which needs to be reflected in the knowledge base. The operation handles a collection of chunks: it will remove the knowledge base data for the Mendix objects the chunks refer to, after which the new data is inserted. For example, this operation can be used before a Mendix object gets committed to keep the knowledgebase in sync with the change. +The `Replace` operation is intended to be used in scenarios where the chunks in the knowledge base are related to Mendix objects (that is, data in the Mendix database). It can be used to keep the knowledge base in sync when data in your Mendix app database changes, which needs to be reflected in the knowledge base. The operation handles a collection of chunks: it removes the knowledge base data for the Mendix objects the chunks refer to, after which the new data is inserted. For example, you can use this operation before a Mendix object gets committed to keep the knowledge base in sync with the change. ### Retrieve Operations {#retrieve-operations} -Currently, four operations are available for on-demand retrieval of data chunks from a knowledge base. All operations work on a single knowledge base (specified by the knowledge base name) on a single database server (specified by the `DatabaseConfiguration`). The details for this are captured in the `PgVectorKnowledgeBase`. Apart from a regular [Retrieve](#retrieve), an additional operation was exposed to [Retrieve Nearest Neighbors](#retrieve-nearest-neighbors), where the cosine similarity between the input vector and the vectors of the records in the knowledge base is calculated. In both cases it is possible to filter on metadata. +Currently, four operations are available for on-demand retrieval of data chunks from a knowledge base. All operations work on a single knowledge base (specified by the knowledge base name) on a single database server (specified by the `DatabaseConfiguration`). The details for this are captured in the `PgVectorKnowledgeBase`. Apart from a regular [Retrieve](#retrieve), an additional operation was exposed to [Retrieve Nearest Neighbors](#retrieve-nearest-neighbors), where the cosine similarity between the input vector and the vectors of the records in the knowledge base is calculated. In both cases, it is possible to filter on metadata. A typical pattern for retrieval from a knowledge base uses GenAI Commons operations and can be illustrated as follows: @@ -123,7 +123,7 @@ A typical pattern for retrieval from a knowledge base uses GenAI Commons operati 2. Use [Add Metadata to MetadataCollection](/agents/genai-for-mx/commons/) (iteratively) to create a collection of the necessary metadata. 3. Do the retrieval. For example, you could use [Retrieve Nearest Neighbors](#retrieve-nearest-neighbors) to find chunks based on vector similarity. -For scenarios in which the created chunks were based on Mendix objects at the time of population and these objects need to be used in logic after the retrieval step, two additional operations are available. The Java actions [Retrieve & Associate](#retrieve-associate) and [Retrieve Nearest Neighbors & Associate](#retrieve-nearest-neighbors-associate) take care of the chunk retrieval and set the association towards the original object, if applicable. +For scenarios where the created chunks were based on Mendix objects at the time of population and these objects need to be used in logic after the retrieval step, two additional operations are available. The Java actions [Retrieve & Associate](#retrieve-associate) and [Retrieve Nearest Neighbors & Associate](#retrieve-nearest-neighbors-associate) take care of the chunk retrieval and set the association toward the original object, if applicable. A typical pattern for this retrieval is as follows: @@ -134,23 +134,23 @@ A typical pattern for this retrieval is as follows: #### `Retrieve` {#retrieve} -Use this operation to retrieve knowledge base chunks from the knowledge base. Additional selection and filtering can be done by specifying the optional input parameters for offset and a maximum number of results, as well as a collection of metadata or a Mendix object. If a metadata collection is provided, this operation only returns chunks that conform with all of the metadata in the collection. If a Mendix object is passed, only knowledge base chunks that were related to this Mendix object during insertion will be retrieved. +Use this operation to retrieve knowledge base chunks from the knowledge base. You can do additional selection and filtering by specifying the optional input parameters for offset and a maximum number of results, as well as a collection of metadata or a Mendix object. If a metadata collection is provided, this operation only returns chunks that conform with all the metadata in the collection. If a Mendix object is passed, only knowledge base chunks that were related to this Mendix object during insertion are retrieved. #### `Retrieve & Associate` {#retrieve-associate} -Use this operation to retrieve knowledge base chunks from the knowledge base and set associations to the related Mendix objects (if applicable). Additional selection and filtering can be done by specifying the optional input parameters for offset and a maximum number of results, as well as a collection of metadata. If a metadata collection is provided, this operation only returns knowledge base chunks that are conform with all the metadata in the collection. +Use this operation to retrieve knowledge base chunks from the knowledge base and set associations to the related Mendix objects (if applicable). You can do additional selection and filtering by specifying the optional input parameters for offset and a maximum number of results, as well as a collection of metadata. If a metadata collection is provided, this operation only returns knowledge base chunks that conform with all the metadata in the collection. #### `Retrieve Nearest Neighbors` {#retrieve-nearest-neighbors} -Use this operation to retrieve knowledge base chunks from the knowledge base where the retrieval and sorting are based on vector similarity with regard to a given input vector. Additional selection and filtering can be done by specifying the optional input parameters: minimum (cosine) similarity (0–1.0), maximum number of results, and a collection of metadata. If a metadata collection is provided, this operation only returns chunks that conform with all of the metadata in the collection. +Use this operation to retrieve knowledge base chunks from the knowledge base where the retrieval and sorting are based on vector similarity with regard to a given input vector. You can do additional selection and filtering by specifying the optional input parameters: minimum (cosine) similarity (0–1.0), maximum number of results, and a collection of metadata. If a metadata collection is provided, this operation only returns chunks that conform with all the metadata in the collection. #### `Retrieve Nearest Neighbors & Associate` {#retrieve-nearest-neighbors-associate} -Use this operation to retrieve knowledge base chunks from the knowledge base and set associations to the related Mendix objects (if applicable). In this operation the retrieval and sorting are based on vector similarity with regard to a given input vector. Additional selection and filtering can be done by specifying the optional input parameters: minimum (cosine) similarity (0–1.0), maximum number of results, as well as a collection of metadata. If a metadata collection is provided, this operation only returns knowledge base chunks that are conform with all of the metadata in the collection. +Use this operation to retrieve knowledge base chunks from the knowledge base and set associations to the related Mendix objects (if applicable). In this operation, the retrieval and sorting are based on vector similarity with regard to a given input vector. You can do additional selection and filtering by specifying the optional input parameters: minimum (cosine) similarity (0–1.0), maximum number of results, and a collection of metadata. If a metadata collection is provided, this operation only returns knowledge base chunks that conform with all the metadata in the collection. ### Delete Operations {#delete-operations-configuration} -When a whole knowledge base, or part of its data, is no longer needed, this can be handled by using a delete operation. If, however, the knowledge base is still needed, but the data needs to be replaced, see [(Re)populate Operations](#repopulate-operations-configuration) or [Replace](#replace) operations instead. For cases where the chunks in the knowledge base were based on Mendix objects during insertion, chunks can be deleted using the original Mendix object as a starting point in two additional `Delete for List` operations. +When a whole knowledge base, or part of its data, is no longer needed, you can use a delete operation. However, if the knowledge base is still needed but the data needs to be replaced, see [(Re)populate Operations](#repopulate-operations-configuration) or [Replace](#replace) operations instead. For cases where the chunks in the knowledge base were based on Mendix objects during insertion, chunks can be deleted using the original Mendix object as a starting point in two additional `Delete for List` operations. #### `Delete Knowledge Base` {#delete-knowledge-base} @@ -158,11 +158,11 @@ Use this operation to delete a complete knowledge base at once. After execution, #### `Delete for Object` {#delete} -In scenarios where the chunks in the knowledge base are related to Mendix objects (in other words, data in the Mendix database), deletion of Mendix data typically needs to result in the removal of its related knowledge base chunks from the knowledge base. For this, the `Delete for Object` operation can be used. The `Delete for Object` operation accepts any kind of Mendix object, and it removes all the knowledge base chunks related to the provided Mendix object at the time of insertion. +In scenarios where the chunks in the knowledge base are related to Mendix objects (that is, data in the Mendix database), deletion of Mendix data typically needs to result in the removal of its related knowledge base chunks from the knowledge base. For this, you can use the `Delete for Object` operation. The `Delete for Object` operation accepts any kind of Mendix object, and it removes all the knowledge base chunks related to the provided Mendix object at the time of insertion. #### `Delete for List` {#delete-list} -This operation is meant to be used in a similar scenario to the one described for the [Delete for Object](#delete) operation, but handles a list of Mendix objects in a single operation. Executing this operation removes all the knowledge base chunks related to the provided Mendix objects at the time of insertion. +This operation is meant to be used in a similar scenario to the one described for the [Delete for Object](#delete) operation, but it handles a list of Mendix objects in a single operation. Executing this operation removes all the knowledge base chunks related to the provided Mendix objects at the time of insertion. ## Technical Reference {#technical-reference} @@ -177,10 +177,10 @@ The **Documentation** pane displays the documentation for the currently selected ## Showcase Application {#showcase-application} -For more inspiration and guidance on how to use these operations in your logic and how to combine it with use cases in the context of generative AI, Mendix highly recommends downloading the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475) from the Marketplace. This application contains various examples in the context of generative AI, some of which use the PgVector Knowledge Base module for storing embedding vectors. +For more inspiration and guidance on how to use these operations in your logic and how to combine them with use cases in the context of generative AI, Mendix recommends downloading the [GenAI Showcase App](https://marketplace.mendix.com/link/component/220475) from Marketplace. This application contains various examples in the context of generative AI, some of which use the PgVector Knowledge Base module for storing embedding vectors. {{% alert color="info" %}} -For more information on how to set up a vector database for retrieval augmented generation (RAG), see the [Setting up a Vector Database](/agents/reference-guide/external-connectors/pgvector-setup/) section and the [RAG Example Implementation in the GenAI Showcase App](/agents/rag/) section. +For more information on how to set up a vector database for retrieval augmented generation (RAG), see [Setting up a Vector Database](/agents/reference-guide/external-connectors/pgvector-setup/) and [RAG Example Implementation in the GenAI Showcase App](/agents/rag/). {{% /alert %}} ## Troubleshooting diff --git a/content/en/docs/genai/reference-guide/external-platforms/snowflake-cortex.md b/content/en/docs/genai/reference-guide/external-platforms/snowflake-cortex.md index dd3453d4667..57ab87e1111 100644 --- a/content/en/docs/genai/reference-guide/external-platforms/snowflake-cortex.md +++ b/content/en/docs/genai/reference-guide/external-platforms/snowflake-cortex.md @@ -9,7 +9,7 @@ aliases: ## Introduction -[Snowflake Cortex AI](https://docs.snowflake.com/en/guides-overview-ai-features) allows users to quickly analyze data and build generative AI applications using fully managed LLMs, vector search and fully managed text-to-SQL services. It also enables multiple users to use AI models with no-code, SQL and Python interfaces. +[Snowflake Cortex AI](https://docs.snowflake.com/en/guides-overview-ai-features) allows users to quickly analyze data and build generative AI applications using fully managed LLMs, vector search, and fully managed text-to-SQL services. It also enables multiple users to use AI models with no-code, SQL, and Python interfaces. ## Integrating Your Mendix App with Snowflake Cortex @@ -29,12 +29,12 @@ In addition to the above, the integration also supports the [ANOMALY DETECTION]( The showcase app has the following pages: -* **Introduction** - Information about Snowflake AI and the necessary prerequisites to use it. -* **Machine Learning** - Sample implementation of the ANOMALY DETECTION machine learning functionality. -* **Large Language Models** - Information about the available LLM functions, as well as a sample implementation of COMPLETE and TRANSLATE. -* **Cortex Analyst** - Sample implementation of Snowflake Cortex Analyst, including a chat feature that can take the SQL answer returned by Cortex Analyst, and convert it to natural language. +* **Introduction** – Information about Snowflake AI and the necessary prerequisites to use it. +* **Machine Learning** – Sample implementation of the ANOMALY DETECTION machine learning functionality. +* **Large Language Models** – Information about the available LLM functions, as well as a sample implementation of COMPLETE and TRANSLATE. +* **Cortex Analyst** – Sample implementation of Snowflake Cortex Analyst, including a chat feature that can take the SQL answer returned by Cortex Analyst and convert it to natural language. -Under the hood, the functionalities are implemented by calling them from microflows which you can use as examples to implement the functions in your own app. For more information, refer to the following sections. +The functionalities are implemented by calling them from microflows, which you can use as examples to implement the functions in your own app. For more information, see the following sections. ### Implementing the Analyst Functionality @@ -42,29 +42,27 @@ For more information about configuring the integration between Mendix and Snowfl ### Implementing Other Functionalities {#functionalities} -The [Snowflake showcase app](https://marketplace.mendix.com/link/component/225845) contains example implementations of the Analyst, ANOMALY DETECTION, COMPLETE and TRANSLATE functionalities. To examine these examples, perform the following steps: +The [Snowflake showcase app](https://marketplace.mendix.com/link/component/225845) contains example implementations of the Analyst, ANOMALY DETECTION, COMPLETE, and TRANSLATE functionalities. To examine these examples, perform the following steps: 1. Import the sample app into your Mendix Studio Pro. For more information, see [Using Marketplace Content](/appstore/use-content/). -2. In Studio Pro, in the [App Explorer](/refguide/app-explorer/), go to **Showcase_AI_RESTSQLAPI** > **Pages**. - - This section contains the following pages: +2. In Studio Pro, in the [App Explorer](/refguide/app-explorer/), go to **Showcase_AI_RESTSQLAPI** > **Pages**. This section contains the following pages: 1. Introduction 2. ML functions 3. Cortex LLM Functions 4. Cortex Analyst -3. To see how a Snowflake Cortex Analyst action is called, use the **Explorer** search box to find and open the *EXAMPLE_CortexAnalyst_GenerateResponseMessage** microflow. +3. To see how a Snowflake Cortex Analyst action is called, use the **Explorer** search box to find and open the **EXAMPLE_CortexAnalyst_GenerateResponseMessage** microflow: {{< figure src="/attachments/appstore/platform-supported-content/modules/snowflake-ai-data-connector/CortexAnalystRequestExample.png" alt="" >}} This microflow calls the Snowflake Cortex Analyst function. -4. To see how you can modify the statement, refer to the *DS_Statement_ML_CreateView_Analyze* example microflow and check how the parameters are set at the **Statement_SetUp** step. +4. To see how to modify the statement, refer to the *DS_Statement_ML_CreateView_Analyze* example microflow and check how the parameters are set at the **Statement_SetUp** step: {{< figure src="/attachments/genai/snowflake/StatementSetup.png" alt="" >}} - For information about the parameters required by each functionality, refer to Snowflake documentation. + For information about the parameters required by each functionality, see the Snowflake documentation.