# OpenAI [![openupm](https://camo.githubusercontent.com/5ced614ee9ee59216bd2918b393e10334ecdfd5c2a04f86997e0b1c05335d288/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f636f6d2e6f70656e61692e756e6974793f6c6162656c3d6f70656e75706d2672656769737472795f7572693d68747470733a2f2f7061636b6167652e6f70656e75706d2e636f6d)](https://openupm.com/packages/com.openai.unity/) [![openupm](https://camo.githubusercontent.com/6247b1a65cc42b48ae763cc6cd5794edad4a7b1043eca2e984aeed7e47e6691d/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f64796e616d69632f6a736f6e3f636f6c6f723d627269676874677265656e266c6162656c3d646f776e6c6f6164732671756572793d2532342e646f776e6c6f616473267375666669783d2532466d6f6e74682675726c3d68747470732533412532462532467061636b6167652e6f70656e75706d2e636f6d253246646f776e6c6f616473253246706f696e742532466c6173742d6d6f6e7468253246636f6d2e6f70656e61692e756e697479)](https://openupm.com/packages/com.openai.unity/) Based on [OpenAI-DotNet](https://github.com/RageAgainstThePixel/OpenAI-DotNet) A [OpenAI](https://openai.com/) package for the [Unity](https://unity.com/) to use though their RESTful API. Independently developed, this is not an official library and I am not affiliated with OpenAI. An OpenAI API account is required. ***All copyrights, trademarks, logos, and assets are the property of their respective owners.*** ## Installing Requires Unity 2021.3 LTS or higher. The recommended installation method is though the unity package manager and [OpenUPM](https://openupm.com/packages/com.openai.unity). ### Via Unity Package Manager and OpenUPM #### Terminal ```bash openupm add com.openai.unity ``` #### Manual - Open your Unity project settings - Add the OpenUPM package registry: - Name: `OpenUPM` - URL: `https://package.openupm.com` - Scope(s): - `com.openai` - `com.utilities` ![scoped-registries](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/OpenAI/Packages/com.openai.unity/Documentation~/images/package-manager-scopes.png) - Open the Unity Package Manager window - Change the Registry from Unity to `My Registries` - Add the `OpenAI` package ### Via Unity Package Manager and Git url > [!WARNING] > This repo has dependencies on other repositories! You are responsible for adding these on your own. - Open your Unity Package Manager - Add package from git url: `https://github.com/RageAgainstThePixel/com.openai.unity.git#upm` - [com.utilities.async](https://github.com/RageAgainstThePixel/com.utilities.async) - [com.utilities.websockets](https://github.com/RageAgainstThePixel/com.utilities.websockets) - [com.utilities.extensions](https://github.com/RageAgainstThePixel/com.utilities.extensions) - [com.utilities.rest](https://github.com/RageAgainstThePixel/com.utilities.rest) - [com.utilities.audio](https://github.com/RageAgainstThePixel/com.utilities.audio) - [com.utilities.encoder.wav](https://github.com/RageAgainstThePixel/com.utilities.encoder.wav) --- ## [Documentation](https://rageagainstthepixel.github.io/OpenAI-DotNet) > Check out our new api docs! ### Table of Contents - [Authentication](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#authentication) - [Azure OpenAI](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#azure-openai) - [Azure Active Directory Authentication](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#azure-active-directory-authentication) - [OpenAI API Proxy](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#openai-api-proxy) - [Models](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#models) - [List Models](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-models) - [Retrieve Models](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-model) - [Delete Fine Tuned Model](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#delete-fine-tuned-model) - [Realtime](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#realtime) - [Create Realtime Session](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-realtime-session) - [Client Events](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#client-events) - [Sending Client Events](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#sending-client-events) - [Server Events](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#server-events) - [Receiving Server Events](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#receiving-server-events) - [Assistants](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#assistants) - [List Assistants](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-assistants) - [Create Assistant](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-assistant) - [Retrieve Assistant](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-assistant) - [Modify Assistant](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#modify-assistant) - [Delete Assistant](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#delete-assistant) - [Assistant Streaming](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#assistant-streaming) - [Threads](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#threads) - [Create Thread](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-thread) - [Create Thread and Run](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-thread-and-run) - [Streaming](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-thread-and-run-streaming) - [Retrieve Thread](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-thread) - [Modify Thread](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#modify-thread) - [Delete Thread](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#delete-thread) - [Thread Messages](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#thread-messages) - [List Messages](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-thread-messages) - [Create Message](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-thread-message) - [Retrieve Message](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-thread-message) - [Modify Message](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#modify-thread-message) - [Thread Runs](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#thread-runs) - [List Runs](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-thread-runs) - [Create Run](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-thread-run) - [Streaming](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-thread-run-streaming) - [Retrieve Run](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-thread-run) - [Modify Run](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#modify-thread-run) - [Submit Tool Outputs to Run](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#thread-submit-tool-outputs-to-run) - [Structured Outputs](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#thread-structured-outputs) - [List Run Steps](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-thread-run-steps) - [Retrieve Run Step](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-thread-run-step) - [Cancel Run](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#cancel-thread-run) - [Vector Stores](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#vector-stores) - [List Vector Stores](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-vector-stores) - [Create Vector Store](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-vector-store) - [Retrieve Vector Store](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-vector-store) - [Modify Vector Store](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#modify-vector-store) - [Delete Vector Store](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#delete-vector-store) - [Vector Store Files](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#vector-store-files) - [List Vector Store Files](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-vector-store-files) - [Create Vector Store File](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-vector-store-file) - [Retrieve Vector Store File](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-vector-store-file) - [Delete Vector Store File](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#delete-vector-store-file) - [Vector Store File Batches](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#vector-store-file-batches) - [Create Vector Store File Batch](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-vector-store-file-batch) - [Retrieve Vector Store File Batch](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-vector-store-file-batch) - [List Files In Vector Store Batch](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-files-in-vector-store-batch) - [Cancel Vector Store File Batch](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#cancel-vector-store-file-batch) - [Chat](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat) - [Chat Completions](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat-completions) - [Streaming](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat-streaming) - [Tools](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat-tools) - [Vision](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat-vision) - [Audio](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat-audio) - [Structured Outputs](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat-structured-outputs) - [Json Mode](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat-json-mode) - [Audio](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#audio) - [Create Speech](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-speech) - [Stream Speech](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#stream-speech) - [Create Transcription](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-transcription) - [Create Translation](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-translation) - [Images](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#images) - [Create Image](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-image) - [Edit Image](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#edit-image) - [Create Image Variation](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-image-variation) - [Files](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#files) - [List Files](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-files) - [Upload File](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#upload-file) - [Delete File](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#delete-file) - [Retrieve File](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-file-info) - [Download File Content](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#download-file-content) - [Fine Tuning](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#fine-tuning) - [Create Fine Tune Job](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-fine-tune-job) - [List Fine Tune Jobs](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-fine-tune-jobs) - [Retrieve Fine Tune Job Info](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-fine-tune-job-info) - [Cancel Fine Tune Job](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#cancel-fine-tune-job) - [List Fine Tune Job Events](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-fine-tune-job-events) - [Batches](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#batches) - [List Batches](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#list-batches) - [Create Batch](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-batch) - [Retrieve Batch](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#retrieve-batch) - [Cancel Batch](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#cancel-batch) - [Embeddings](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#embeddings) - [Create Embedding](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-embeddings) - [Moderations](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#moderations) - [Create Moderation](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-moderation) ### [Authentication](https://platform.openai.com/docs/api-reference/authentication) There are 4 ways to provide your API keys, in order of precedence: > [!WARNING] > We recommended using the environment variables to load the API key instead of having it hard coded in your source. It is not recommended use this method in production, but only for accepting user credentials, local testing and quick start scenarios. 1. [Pass keys directly with constructor](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#pass-keys-directly-with-constructor) :warning: 2. [Unity Scriptable Object](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#unity-scriptable-object) :warning: 3. [Load key from configuration file](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#load-key-from-configuration-file) 4. [Use System Environment Variables](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#use-system-environment-variables) You use the `OpenAIAuthentication` when you initialize the API as shown: #### Pass keys directly with constructor > [!WARNING] > We recommended using the environment variables to load the API key instead of having it hard coded in your source. It is not recommended use this method in production, but only for accepting user credentials, local testing and quick start scenarios. ```csharp var api = new OpenAIClient("sk-apiKey"); ``` Or create a `OpenAIAuthentication` object manually ```csharp var api = new OpenAIClient(new OpenAIAuthentication("sk-apiKey", "org-yourOrganizationId", "proj_yourProjectId")); ``` #### Unity Scriptable Object You can save the key directly into a scriptable object that is located in the `Assets/Resources` folder. You can create a new one by using the context menu of the project pane and creating a new `OpenAIConfiguration` scriptable object. > [!WARNING] > Beware checking this file into source control, as other people will be able to see your API key. It is recommended to use the [OpenAI-DotNet-Proxy](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#openai-api-proxy) and authenticate users with your preferred OAuth provider. ![Create new OpenAIConfiguration](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/OpenAI/Packages/com.openai.unity/Documentation~/images/create-scriptable-object.png) #### Load key from configuration file Attempts to load api keys from a configuration file, by default `.openai` in the current directory, optionally traversing up the directory tree or in the user's home directory. To create a configuration file, create a new text file named `.openai` and containing the line: > [!NOTE] > Organization and project id entries are optional. ##### Json format ```json { "apiKey": "sk-aaaabbbbbccccddddd", "organizationId": "org-yourOrganizationId", "projectId": "proj_yourProjectId" } ``` ##### Deprecated format ```shell OPENAI_API_KEY=sk-aaaabbbbbccccddddd OPENAI_ORGANIZATION_ID=org-yourOrganizationId OPENAI_PROJECT_ID=proj_yourProjectId ``` You can also load the configuration file directly with known path by calling static methods in `OpenAIAuthentication`: - Loads the default `.openai` config in the specified directory: ```csharp var api = new OpenAIClient(new OpenAIAuthentication().LoadFromDirectory("path/to/your/directory")); ``` - Loads the configuration file from a specific path. File does not need to be named `.openai` as long as it conforms to the json format: ```csharp var api = new OpenAIClient(new OpenAIAuthentication().LoadFromPath("path/to/your/file.json")); ``` #### Use System Environment Variables Use your system's environment variables specify an api key and organization to use. - Use `OPENAI_API_KEY` for your api key. - Use `OPENAI_ORGANIZATION_ID` to specify an organization. - Use `OPENAI_PROJECT_ID` to specify a project. ```csharp var api = new OpenAIClient(new OpenAIAuthentication().LoadFromEnvironment()); ``` --- ### [Azure OpenAI](https://learn.microsoft.com/en-us/azure/cognitive-services/openai) You can also choose to use Microsoft's Azure OpenAI deployments as well. You can find the required information in the Azure Playground by clicking the `View Code` button and view a URL like this: ```markdown https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version} ``` - `your-resource-name` The name of your Azure OpenAI Resource. - `deployment-id` The deployment name you chose when you deployed the model. - `api-version` The API version to use for this operation. This follows the YYYY-MM-DD format. To setup the client to use your deployment, you'll need to pass in `OpenAISettings` into the client constructor. ```csharp var auth = new OpenAIAuthentication("sk-apiKey"); var settings = new OpenAISettings(resourceName: "your-resource-name", deploymentId: "deployment-id", apiVersion: "api-version"); var api = new OpenAIClient(auth, settings); ``` #### [Azure Active Directory Authentication](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/reference#authentication) [Authenticate with MSAL](https://github.com/AzureAD/microsoft-authentication-library-for-dotnet) as usual and get access token, then use the access token when creating your `OpenAIAuthentication`. Then be sure to set useAzureActiveDirectory to true when creating your `OpenAISettings`. [Tutorial: Desktop app that calls web APIs: Acquire a token](https://learn.microsoft.com/en-us/azure/active-directory/develop/scenario-desktop-acquire-token?tabs=dotnet) ```csharp // get your access token using any of the MSAL methods var accessToken = result.AccessToken; var auth = new OpenAIAuthentication(accessToken); var settings = new OpenAISettings(resourceName: "your-resource", deploymentId: "deployment-id", apiVersion: "api-version", useActiveDirectoryAuthentication: true); var api = new OpenAIClient(auth, settings); ``` --- ### [OpenAI API Proxy](https://github.com/RageAgainstThePixel/OpenAI-DotNet/blob/main/OpenAI-DotNet-Proxy/Readme.md) Using either the [OpenAI-DotNet](https://github.com/RageAgainstThePixel/OpenAI-DotNet) or [com.openai.unity](https://github.com/RageAgainstThePixel/com.openai.unity) packages directly in your front-end app may expose your API keys and other sensitive information. To mitigate this risk, it is recommended to set up an intermediate API that makes requests to OpenAI on behalf of your front-end app. This library can be utilized for both front-end and intermediary host configurations, ensuring secure communication with the OpenAI API. #### Front End Example In the front end example, you will need to securely authenticate your users using your preferred OAuth provider. Once the user is authenticated, exchange your custom auth token with your API key on the backend. Follow these steps: 1. Setup a new project using either the [OpenAI-DotNet](https://github.com/RageAgainstThePixel/OpenAI-DotNet) or [com.openai.unity](https://github.com/RageAgainstThePixel/com.openai.unity) packages. 2. Authenticate users with your OAuth provider. 3. After successful authentication, create a new `OpenAIAuthentication` object and pass in the custom token with the prefix `sess-`. 4. Create a new `OpenAISettings` object and specify the domain where your intermediate API is located. 5. Pass your new `auth` and `settings` objects to the `OpenAIClient` constructor when you create the client instance. Here's an example of how to set up the front end: ```csharp var authToken = await LoginAsync(); var auth = new OpenAIAuthentication($"sess-{authToken}"); var settings = new OpenAISettings(domain: "api.your-custom-domain.com"); var api = new OpenAIClient(auth, settings); ``` This setup allows your front end application to securely communicate with your backend that will be using the OpenAI-DotNet-Proxy, which then forwards requests to the OpenAI API. This ensures that your OpenAI API keys and other sensitive information remain secure throughout the process. #### Back End Example In this example, we demonstrate how to set up and use `OpenAIProxy` in a new ASP.NET Core web app. The proxy server will handle authentication and forward requests to the OpenAI API, ensuring that your API keys and other sensitive information remain secure. 1. Create a new [ASP.NET Core minimal web API](https://learn.microsoft.com/en-us/aspnet/core/tutorials/min-web-api?view=aspnetcore-6.0) project. 2. Add the OpenAI-DotNet nuget package to your project. - Powershell install: `Install-Package OpenAI-DotNet-Proxy` - Dotnet install: `dotnet add package OpenAI-DotNet-Proxy` - Manually editing .csproj: `` 3. Create a new class that inherits from `AbstractAuthenticationFilter` and override the `ValidateAuthentication` method. This will implement the `IAuthenticationFilter` that you will use to check user session token against your internal server. 4. In `Program.cs`, create a new proxy web application by calling `OpenAIProxy.CreateWebApplication` method, passing your custom `AuthenticationFilter` as a type argument. 5. Create `OpenAIAuthentication` and `OpenAIClientSettings` as you would normally with your API keys, org id, or Azure settings. ```csharp public partial class Program { private class AuthenticationFilter : AbstractAuthenticationFilter { public override async Task ValidateAuthenticationAsync(IHeaderDictionary request) { await Task.CompletedTask; // remote resource call to verify token // You will need to implement your own class to properly test // custom issued tokens you've setup for your end users. if (!request.Authorization.ToString().Contains(TestUserToken)) { throw new AuthenticationException("User is not authorized"); } } } public static void Main(string[] args) { var auth = OpenAIAuthentication.LoadFromEnv(); var settings = new OpenAIClientSettings(/* your custom settings if using Azure OpenAI */); using var openAIClient = new OpenAIClient(auth, settings); OpenAIProxy.CreateWebApplication(args, openAIClient).Run(); } } ``` Once you have set up your proxy server, your end users can now make authenticated requests to your proxy api instead of directly to the OpenAI API. The proxy server will handle authentication and forward requests to the OpenAI API, ensuring that your API keys and other sensitive information remain secure. --- ### [Models](https://platform.openai.com/docs/api-reference/models) List and describe the various models available in the API. You can refer to the [Models documentation](https://platform.openai.com/docs/models) to understand what models are available and the differences between them. Also checkout [model endpoint compatibility](https://platform.openai.com/docs/models/model-endpoint-compatibility) to understand which models work with which endpoints. To specify a custom model not pre-defined in this library: ```csharp var model = new Model("model-id"); ``` The Models API is accessed via `OpenAIClient.ModelsEndpoint` #### [List models](https://platform.openai.com/docs/api-reference/models/list) Lists the currently available models, and provides basic information about each one such as the owner and availability. ```csharp var api = new OpenAIClient(); var models = await api.ModelsEndpoint.GetModelsAsync(); foreach (var model in models) { Debug.Log(model.ToString()); } ``` #### [Retrieve model](https://platform.openai.com/docs/api-reference/models/retrieve) Retrieves a model instance, providing basic information about the model such as the owner and permissions. ```csharp var api = new OpenAIClient(); var model = await api.ModelsEndpoint.GetModelDetailsAsync("gpt-4o"); Debug.Log(model.ToString()); ``` #### [Delete Fine Tuned Model](https://platform.openai.com/docs/api-reference/fine-tunes/delete-model) Delete a fine-tuned model. You must have the Owner role in your organization. ```csharp var api = new OpenAIClient(); var isDeleted = await api.ModelsEndpoint.DeleteFineTuneModelAsync("your-fine-tuned-model"); Assert.IsTrue(isDeleted); ``` --- ### [Realtime](https://platform.openai.com/docs/api-reference/realtime) > [!WARNING] > Beta Feature. API subject to breaking changes. - [Realtime Guide](https://platform.openai.com/docs/guides/realtime) The Realtime API enables you to build low-latency, multi-modal conversational experiences. It currently supports text and audio as both input and output, as well as function calling. The Assistants API is accessed via `OpenAIClient.RealtimeEndpoint` #### Create Realtime Session Here is a simple example of how to create a realtime session and to send and receive messages from the model. ```csharp var api = new OpenAIClient(); var cancellationTokenSource = new CancellationTokenSource(); var tools = new List { Tool.FromFunc("goodbye", () => { cancellationTokenSource.Cancel(); return "Goodbye!"; }) }; var configuration = new SessionConfiguration(Model.GPT4oRealtime, tools: tools); using var session = await api.RealtimeEndpoint.CreateSessionAsync(configuration); var responseTask = session.ReceiveUpdatesAsync(ServerEvents, cancellationTokenSource.Token); await session.SendAsync(new ConversationItemCreateRequest("Hello!")); await session.SendAsync(new CreateResponseRequest()); await session.SendAsync(new InputAudioBufferAppendRequest(new ReadOnlyMemory(new byte[1024 * 4])), cancellationTokenSource.Token); await session.SendAsync(new ConversationItemCreateRequest("GoodBye!")); await session.SendAsync(new CreateResponseRequest()); await responseTask; void ServerEvents(IServerEvent @event) { switch (@event) { case ResponseAudioTranscriptResponse transcriptResponse: Debug.Log(transcriptResponse.ToString()); break; case ResponseFunctionCallArgumentsResponse functionCallResponse: if (functionCallResponse.IsDone) { ToolCall toolCall = functionCallResponse; toolCall.InvokeFunction(); } break; } } ``` #### Client Events The library implements `IClientEvent` interface for outgoing client sent events. - [`UpdateSessionRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update): Update the session with new session options. - [`InputAudioBufferAppendRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/input-audio-buffer/append): Append audio to the input audio buffer. (Unlike made other client events, the server will not send a confirmation response to this event). - [`InputAudioBufferCommitRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/input-audio-buffer/commit): Commit the input audio buffer. (When in Server VAD mode, the client does not need to send this event). - [`InputAudioBufferClearRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/input-audio-buffer/clear): Clear the input audio buffer. - [`ConversationItemCreateRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/conversation/item/create): Create a new conversation item. This is the main way to send user content to the model. - [`ConversationItemTruncateRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/conversation/item/truncate): Send this event to truncate a previous assistant message’s audio. - [`ConversationItemDeleteRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/conversation/item/delete): Delete a conversation item. This is useful when you want to remove a message from the conversation history. - [`CreateResponseRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create): Create a response from the model. Send this event after creating new conversation items or invoking tool calls. This will trigger the model to generate a response. - [`ResponseCancelRequest`](https://platform.openai.com/docs/api-reference/realtime-client-events/response/cancel) -Send this event to cancel an in-progress response. ##### Sending Client Events You can send client events at any time to the server by calling the `RealtimeSession.SendAsync` method on the session object. The send call will return a `IServerEvent` handle that best represents the appropriate response from the server for that event. This is useful if you want to handle server responses in a more granular way. Ideally though, you may want to handle all server responses with [`RealtimeSession.ReceiveUpdatesAsync`](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#receiving-server-events). > [!NOTE] > The server will not send a confirmation response to the `InputAudioBufferAppendRequest` event. > [!IMPORTANT] > You will also need to send `CreateResponseRequest` to trigger the model to generate a response. ```csharp var serverEvent = await session.SendAsync(new ConversationItemCreateRequest("Hello!")); Debug.Log(serverEvent.ToJsonString()); serverEvent = await session.SendAsync(new CreateResponseRequest()); Debug.Log(serverEvent.ToJsonString()); ``` #### Server Events The library implements `IServerEvent` interface for incoming server sent events. - [`RealtimeEventError`](https://platform.openai.com/docs/api-reference/realtime-server-events/error): Returned when an error occurs, which could be a client problem or a server problem. - [`SessionResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/session): Returned for both a `session.created` and `session.updated` event. - [`RealtimeConversationResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/conversation/created): Returned when a new conversation item is created. - [`ConversationItemCreatedResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/conversation/item/created): Returned when a new conversation item is created. - [`ConversationItemInputAudioTranscriptionResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/conversation): Returned when the input audio transcription is completed or failed. - [`ConversationItemTruncatedResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/conversation/item/truncated): Returned when a conversation item is truncated. - [`ConversationItemDeletedResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/conversation/item/deleted): Returned when a conversation item is deleted. - [`InputAudioBufferCommittedResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/input_audio_buffer/committed): Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. - [`InputAudioBufferClearedResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/input_audio_buffer/cleared): Returned when an input audio buffer is cleared. - [`InputAudioBufferStartedResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/input_audio_buffer/speech_started): Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user. - [`InputAudioBufferStoppedResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/input_audio_buffer/speech_stopped): Returned in server_vad mode when the server detects the end of speech in the audio buffer. - [`RealtimeResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/response): Returned when a response is created or done. - [`ResponseOutputItemResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/response/output_item): Returned when a response output item is added or done. - [`ResponseContentPartResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/response/content_part): Returned when a response content part is added or done. - [`ResponseTextResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/response/text): Returned when a response text is updated or done. - [`ResponseAudioTranscriptResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/response/audio_transcript): Returned when a response audio transcript is updated or done. - [`ResponseAudioResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/response/audio): Returned when a response audio is updated or done. - [`ResponseFunctionCallArgumentsResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/response/function_call_arguments): Returned when a response function call arguments are updated or done. - [`RateLimitsResponse`](https://platform.openai.com/docs/api-reference/realtime-server-events/rate_limits): Returned when rate limits are updated. ##### Receiving Server Events To receive server events, you will need to call the `RealtimeSession.ReceiveUpdatesAsync` method on the session object. This method will return a `Task` that will complete when the session is closed or when the cancellation token is triggered. Ideally this method should be called once and runs for the duration of the session. > [!NOTE] > You can also get sent `IClientEvent` callbacks as well by using the `IRealtimeEvent` interface instead of `IServerEvent`. ```csharp await session.ReceiveUpdatesAsync(ServerEvents, cancellationTokenSource.Token); void ServerEvents(IServerEvent @event) { switch (@event) { case RealtimeEventError error: // raised anytime an error occurs break; case SessionResponse sessionResponse: // raised when a session is created or updated break; case RealtimeConversationResponse conversationResponse: // raised when a new conversation is created break; case ConversationItemCreatedResponse conversationItemCreated: // raised when a new conversation item is created break; case ConversationItemInputAudioTranscriptionResponse conversationItemTranscription: // raised when the input audio transcription is completed or failed break; case ConversationItemTruncatedResponse conversationItemTruncated: // raised when a conversation item is truncated break; case ConversationItemDeletedResponse conversationItemDeleted: // raised when a conversation item is deleted break; case InputAudioBufferCommittedResponse committedResponse: // raised when an input audio buffer is committed break; case InputAudioBufferClearedResponse clearedResponse: // raised when an input audio buffer is cleared break; case InputAudioBufferStartedResponse startedResponse: // raised when speech is detected in the audio buffer break; case InputAudioBufferStoppedResponse stoppedResponse: // raised when speech stops in the audio buffer break; case RealtimeResponse realtimeResponse: // raised when a response is created or done break; case ResponseOutputItemResponse outputItemResponse: // raised when a response output item is added or done break; case ResponseContentPartResponse contentPartResponse: // raised when a response content part is added or done break; case ResponseTextResponse textResponse: // raised when a response text is updated or done break; case ResponseAudioTranscriptResponse transcriptResponse: // raised when a response audio transcript is updated or done break; case ResponseFunctionCallArgumentsResponse functionCallResponse: // raised when a response function call arguments are updated or done break; case RateLimitsResponse rateLimitsResponse: // raised when rate limits are updated break; } } ``` --- ### [Assistants](https://platform.openai.com/docs/api-reference/assistants) > [!WARNING] > Beta Feature. API subject to breaking changes. Build assistants that can call models and use tools to perform tasks. - [Assistants Guide](https://platform.openai.com/docs/assistants) - [OpenAI Assistants Cookbook](https://github.com/openai/openai-cookbook/raw/main/examples/Assistants_API_overview_python.ipynb) The Assistants API is accessed via `OpenAIClient.AssistantsEndpoint` #### [List Assistants](https://platform.openai.com/docs/api-reference/assistants/listAssistants) Returns a list of assistants. ```csharp var api = new OpenAIClient(); var assistantsList = await api.AssistantsEndpoint.ListAssistantsAsync(); foreach (var assistant in assistantsList.Items) { Debug.Log($"{assistant} -> {assistant.CreatedAt}"); } ``` #### [Create Assistant](https://platform.openai.com/docs/api-reference/assistants/createAssistant) Create an assistant with a model and instructions. ```csharp var api = new OpenAIClient(); var request = new CreateAssistantRequest(Model.GPT4o); var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(request); ``` #### [Retrieve Assistant](https://platform.openai.com/docs/api-reference/assistants/getAssistant) Retrieves an assistant. ```csharp var api = new OpenAIClient(); var assistant = await api.AssistantsEndpoint.RetrieveAssistantAsync("assistant-id"); Debug.Log($"{assistant} -> {assistant.CreatedAt}"); ``` #### [Modify Assistant](https://platform.openai.com/docs/api-reference/assistants/modifyAssistant) Modifies an assistant. ```csharp var api = new OpenAIClient(); var createRequest = new CreateAssistantRequest(Model.GPT4_Turbo); var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(createRequest); var modifyRequest = new CreateAssistantRequest(Model.GPT4o); var modifiedAssistant = await api.AssistantsEndpoint.ModifyAssistantAsync(assistant.Id, modifyRequest); // OR AssistantExtension for easier use! var modifiedAssistantEx = await assistant.ModifyAsync(modifyRequest); ``` #### [Delete Assistant](https://platform.openai.com/docs/api-reference/assistants/deleteAssistant) Delete an assistant. ```csharp var api = new OpenAIClient(); var isDeleted = await api.AssistantsEndpoint.DeleteAssistantAsync("assistant-id"); // OR AssistantExtension for easier use! var isDeleted = await assistant.DeleteAsync(); Assert.IsTrue(isDeleted); ``` #### [Assistant Streaming](https://platform.openai.com/docs/api-reference/assistants-streaming) > [!NOTE] > Assistant stream events can be easily added to existing thread calls by passing `Func streamEventHandler` callback to any existing method that supports streaming. #### [Threads](https://platform.openai.com/docs/api-reference/threads) Create Threads that [Assistants](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#assistants) can interact with. The Threads API is accessed via `OpenAIClient.ThreadsEndpoint` ##### [Create Thread](https://platform.openai.com/docs/api-reference/threads/createThread) Create a thread. ```csharp var api = new OpenAIClient(); var thread = await api.ThreadsEndpoint.CreateThreadAsync(); Debug.Log($"Create thread {thread.Id} -> {thread.CreatedAt}"); ``` ##### [Create Thread and Run](https://platform.openai.com/docs/api-reference/runs/createThreadAndRun) Create a thread and run it in one request. > See also: [Thread Runs](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#thread-runs) ```csharp var api = new OpenAIClient(); var assistant = await api.AssistantsEndpoint.CreateAssistantAsync( new CreateAssistantRequest( name: "Math Tutor", instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less.", model: Model.GPT4o)); var messages = new List { "I need to solve the equation `3x + 11 = 14`. Can you help me?" }; var threadRequest = new CreateThreadRequest(messages); var run = await assistant.CreateThreadAndRunAsync(threadRequest); Debug.Log($"Created thread and run: {run.ThreadId} -> {run.Id} -> {run.CreatedAt}"); ``` ###### Create Thread and Run Streaming Create a thread and run it in one request while streaming events. ```csharp var api = new OpenAIClient(); var tools = new List { Tool.GetOrCreateTool(typeof(WeatherService), nameof(WeatherService.GetCurrentWeatherAsync)) }; var assistantRequest = new CreateAssistantRequest(tools: tools, instructions: "You are a helpful weather assistant. Use the appropriate unit based on geographical location."); var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(assistantRequest); ThreadResponse thread = null; async Task StreamEventHandler(IServerSentEvent streamEvent) { switch (streamEvent) { case ThreadResponse threadResponse: thread = threadResponse; break; case RunResponse runResponse: if (runResponse.Status == RunStatus.RequiresAction) { var toolOutputs = await assistant.GetToolOutputsAsync(runResponse); foreach (var toolOutput in toolOutputs) { Debug.Log($"Tool Output: {toolOutput}"); } await runResponse.SubmitToolOutputsAsync(toolOutputs, StreamEventHandler); } break; default: Debug.Log(streamEvent.ToJsonString()); break; } } var run = await assistant.CreateThreadAndRunAsync("I'm in Kuala-Lumpur, please tell me what's the temperature now?", StreamEventHandler); run = await run.WaitForStatusChangeAsync(); var messages = await thread.ListMessagesAsync(); foreach (var response in messages.Items.Reverse()) { Debug.Log($"{response.Role}: {response.PrintContent()}"); } ``` ##### [Retrieve Thread](https://platform.openai.com/docs/api-reference/threads/getThread) Retrieves a thread. ```csharp var api = new OpenAIClient(); var thread = await api.ThreadsEndpoint.RetrieveThreadAsync("thread-id"); // OR if you simply wish to get the latest state of a thread thread = await thread.UpdateAsync(); Debug.Log($"Retrieve thread {thread.Id} -> {thread.CreatedAt}"); ``` ##### [Modify Thread](https://platform.openai.com/docs/api-reference/threads/modifyThread) Modifies a thread. > [!NOTE] > Only the metadata can be modified. ```csharp var api = new OpenAIClient(); var thread = await api.ThreadsEndpoint.CreateThreadAsync(); var metadata = new Dictionary { { "key", "custom thread metadata" } } thread = await api.ThreadsEndpoint.ModifyThreadAsync(thread.Id, metadata); // OR use extension method for convenience! thread = await thread.ModifyAsync(metadata); Debug.Log($"Modify thread {thread.Id} -> {thread.Metadata["key"]}"); ``` ##### [Delete Thread](https://platform.openai.com/docs/api-reference/threads/deleteThread) Delete a thread. ```csharp var api = new OpenAIClient(); var isDeleted = await api.ThreadsEndpoint.DeleteThreadAsync("thread-id"); // OR use extension method for convenience! var isDeleted = await thread.DeleteAsync(); Assert.IsTrue(isDeleted); ``` ##### [Thread Messages](https://platform.openai.com/docs/api-reference/messages) Create messages within threads. ###### [List Thread Messages](https://platform.openai.com/docs/api-reference/messages/listMessages) Returns a list of messages for a given thread. ```csharp var api = new OpenAIClient(); var messageList = await api.ThreadsEndpoint.ListMessagesAsync("thread-id"); // OR use extension method for convenience! var messageList = await thread.ListMessagesAsync(); foreach (var message in messageList.Items) { Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}"); } ``` ###### [Create Thread Message](https://platform.openai.com/docs/api-reference/messages/createMessage) Create a message. ```csharp var api = new OpenAIClient(); var thread = await api.ThreadsEndpoint.CreateThreadAsync(); var request = new CreateMessageRequest("Hello world!"); var message = await api.ThreadsEndpoint.CreateMessageAsync(thread.Id, request); // OR use extension method for convenience! var message = await thread.CreateMessageAsync("Hello World!"); Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}"); ``` ###### [Retrieve Thread Message](https://platform.openai.com/docs/api-reference/messages/getMessage) Retrieve a message. ```csharp var api = new OpenAIClient(); var message = await api.ThreadsEndpoint.RetrieveMessageAsync("thread-id", "message-id"); // OR use extension methods for convenience! var message = await thread.RetrieveMessageAsync("message-id"); var message = await message.UpdateAsync(); Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}"); ``` ###### [Modify Thread Message](https://platform.openai.com/docs/api-reference/messages/modifyMessage) Modify a message. > [!NOTE] > Only the message metadata can be modified. ```csharp var api = new OpenAIClient(); var metadata = new Dictionary { { "key", "custom message metadata" } }; var message = await api.ThreadsEndpoint.ModifyMessageAsync("thread-id", "message-id", metadata); // OR use extension method for convenience! var message = await message.ModifyAsync(metadata); Debug.Log($"Modify message metadata: {message.Id} -> {message.Metadata["key"]}"); ``` ##### [Thread Runs](https://platform.openai.com/docs/api-reference/runs) Represents an execution run on a thread. ###### [List Thread Runs](https://platform.openai.com/docs/api-reference/runs/listRuns) Returns a list of runs belonging to a thread. ```csharp var api = new OpenAIClient(); var runList = await api.ThreadsEndpoint.ListRunsAsync("thread-id"); // OR use extension method for convenience! var runList = await thread.ListRunsAsync(); foreach (var run in runList.Items) { Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}"); } ``` ###### [Create Thread Run](https://platform.openai.com/docs/api-reference/runs/createRun) Create a run. ```csharp var api = new OpenAIClient(); var assistant = await api.AssistantsEndpoint.CreateAssistantAsync( new CreateAssistantRequest( name: "Math Tutor", instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less.", model: Model.GPT4o)); var thread = await api.ThreadsEndpoint.CreateThreadAsync(); var message = await thread.CreateMessageAsync("I need to solve the equation `3x + 11 = 14`. Can you help me?"); var run = await thread.CreateRunAsync(assistant); Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}"); ``` ###### Create Thread Run Streaming Create a run and stream the events. ```csharp var api = new OpenAIClient(); var assistant = await api.AssistantsEndpoint.CreateAssistantAsync( new CreateAssistantRequest( name: "Math Tutor", instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.", model: Model.GPT4o, responseFormat: ChatResponseFormat.Json)); var thread = await api.ThreadsEndpoint.CreateThreadAsync(); var message = await thread.CreateMessageAsync("I need to solve the equation `3x + 11 = 14`. Can you help me?"); var run = await thread.CreateRunAsync(assistant, async streamEvent => { Debug.Log(streamEvent.ToJsonString()); await Task.CompletedTask; }); var messages = await thread.ListMessagesAsync(); foreach (var response in messages.Items.Reverse()) { Debug.Log($"{response.Role}: {response.PrintContent()}"); } ``` ###### [Retrieve Thread Run](https://platform.openai.com/docs/api-reference/runs/getRun) Retrieves a run. ```csharp var api = new OpenAIClient(); var run = await api.ThreadsEndpoint.RetrieveRunAsync("thread-id", "run-id"); // OR use extension method for convenience! var run = await thread.RetrieveRunAsync("run-id"); var run = await run.UpdateAsync(); Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}"); ``` ###### [Modify Thread Run](https://platform.openai.com/docs/api-reference/runs/modifyRun) Modifies a run. > [!NOTE] > Only the metadata can be modified. ```csharp var api = new OpenAIClient(); var metadata = new Dictionary { { "key", "custom run metadata" } }; var run = await api.ThreadsEndpoint.ModifyRunAsync("thread-id", "run-id", metadata); // OR use extension method for convenience! var run = await run.ModifyAsync(metadata); Debug.Log($"Modify run {run.Id} -> {run.Metadata["key"]}"); ``` ###### [Thread Submit Tool Outputs to Run](https://platform.openai.com/docs/api-reference/runs/submitToolOutputs) When a run has the status: `requires_action` and `required_action.type` is `submit_tool_outputs`, this endpoint can be used to submit the outputs from the tool calls once they're all completed. All outputs must be submitted in a single request. > [!NOTE] > See [Create Thread and Run Streaming](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#create-thread-and-run-streaming) example on how to stream tool output events. ```csharp var api = new OpenAIClient(); var tools = new List { // Use a predefined tool Tool.Retrieval, Tool.CodeInterpreter, // Or create a tool from a type and the name of the method you want to use for function calling Tool.GetOrCreateTool(typeof(WeatherService), nameof(WeatherService.GetCurrentWeatherAsync)), // Pass in an instance of an object to call a method on it Tool.GetOrCreateTool(api.ImagesEndPoint, nameof(ImagesEndpoint.GenerateImageAsync)), // Define func<,> callbacks Tool.FromFunc("name_of_func", () => { /* callback function */ }), Tool.FromFunc("func_with_multiple_params", (t1, t2) => { /* logic that calculates return value */ return tResult; }) }; var assistantRequest = new CreateAssistantRequest(tools: tools, instructions: "You are a helpful weather assistant. Use the appropriate unit based on geographical location."); var testAssistant = await api.AssistantsEndpoint.CreateAssistantAsync(assistantRequest); var run = await testAssistant.CreateThreadAndRunAsync("I'm in Kuala-Lumpur, please tell me what's the temperature now?"); // waiting while run is Queued and InProgress run = await run.WaitForStatusChangeAsync(); // Invoke all of the tool call functions and return the tool outputs. var toolOutputs = await testAssistant.GetToolOutputsAsync(run.RequiredAction.SubmitToolOutputs.ToolCalls); foreach (var toolOutput in toolOutputs) { Debug.Log($"tool call output: {toolOutput.Output}"); } // submit the tool outputs run = await run.SubmitToolOutputsAsync(toolOutputs); // waiting while run in Queued and InProgress run = await run.WaitForStatusChangeAsync(); var messages = await run.ListMessagesAsync(); foreach (var message in messages.Items.OrderBy(response => response.CreatedAt)) { Debug.Log($"{message.Role}: {message.PrintContent()}"); } ``` ##### [Thread Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) Structured Outputs is the evolution of JSON mode. While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence. > [!IMPORTANT] > > - When using JSON mode, always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don't forget, the API will throw an error if the string "JSON" does not appear somewhere in the context. > - The JSON in the message the model returns may be partial (i.e. cut off) if `finish_reason` is length, which indicates the generation exceeded max_tokens or the conversation exceeded the token limit. To guard against this, check `finish_reason` before parsing the response. First define the structure of your responses. These will be used as your schema. These are the objects you'll deserialize to, so be sure to use standard Json object models. ```csharp public class MathResponse { [JsonProperty("steps")] public IReadOnlyList Steps { get; private set; } [JsonProperty("final_answer")] public string FinalAnswer { get; private set; } } public class MathStep { [JsonProperty("explanation")] public string Explanation { get; private set; } [JsonProperty("output")] public string Output { get; private set; } } ``` To use, simply specify the `MathResponse` type as a generic constraint in either `CreateAssistantAsync`, `CreateRunAsync`, or `CreateThreadAndRunAsync`. ```csharp var api = new OpenAIClient(); var assistant = await api.AssistantsEndpoint.CreateAssistantAsync( new CreateAssistantRequest( name: "Math Tutor", instructions: "You are a helpful math tutor. Guide the user through the solution step by step.", model: "gpt-4o-2024-08-06")); ThreadResponse thread = null; try { async Task StreamEventHandler(IServerSentEvent @event) { try { switch (@event) { case MessageResponse message: if (message.Status != MessageStatus.Completed) { Debug.Log(@event.ToJsonString()); break; } var mathResponse = message.FromSchema(); for (var i = 0; i < mathResponse.Steps.Count; i++) { var step = mathResponse.Steps[i]; Debug.Log($"Step {i}: {step.Explanation}"); Debug.Log($"Result: {step.Output}"); } Debug.Log($"Final Answer: {mathResponse.FinalAnswer}"); break; default: Debug.Log(@event.ToJsonString()); break; } } catch (Exception e) { Debug.Log(e); throw; } await Task.CompletedTask; } var run = await assistant.CreateThreadAndRunAsync("how can I solve 8x + 7 = -23", StreamEventHandler); thread = await run.GetThreadAsync(); run = await run.WaitForStatusChangeAsync(); Debug.Log($"Created thread and run: {run.ThreadId} -> {run.Id} -> {run.CreatedAt}"); var messages = await thread.ListMessagesAsync(); foreach (var response in messages.Items.OrderBy(response => response.CreatedAt)) { Debug.Log($"{response.Role}: {response.PrintContent()}"); } } finally { await assistant.DeleteAsync(deleteToolResources: thread == null); if (thread != null) { var isDeleted = await thread.DeleteAsync(deleteToolResources: true); } } ``` You can also manually create json schema json string as well, but you will be responsible for deserializing your response data: ```csharp var api = new OpenAIClient(); var mathSchema = new JsonSchema("math_response", @" { ""type"": ""object"", ""properties"": { ""steps"": { ""type"": ""array"", ""items"": { ""type"": ""object"", ""properties"": { ""explanation"": { ""type"": ""string"" }, ""output"": { ""type"": ""string"" } }, ""required"": [ ""explanation"", ""output"" ], ""additionalProperties"": false } }, ""final_answer"": { ""type"": ""string"" } }, ""required"": [ ""steps"", ""final_answer"" ], ""additionalProperties"": false }"); var assistant = await api.AssistantsEndpoint.CreateAssistantAsync( new CreateAssistantRequest( name: "Math Tutor", instructions: "You are a helpful math tutor. Guide the user through the solution step by step.", model: "gpt-4o-2024-08-06", jsonSchema: mathSchema)); ThreadResponse thread = null; try { var run = await assistant.CreateThreadAndRunAsync("how can I solve 8x + 7 = -23", async @event => { Debug.Log(@event.ToJsonString()); await Task.CompletedTask; }); thread = await run.GetThreadAsync(); run = await run.WaitForStatusChangeAsync(); Debug.Log($"Created thread and run: {run.ThreadId} -> {run.Id} -> {run.CreatedAt}"); var messages = await thread.ListMessagesAsync(); foreach (var response in messages.Items) { Debug.Log($"{response.Role}: {response.PrintContent()}"); } } finally { await assistant.DeleteAsync(deleteToolResources: thread == null); if (thread != null) { var isDeleted = await thread.DeleteAsync(deleteToolResources: true); Assert.IsTrue(isDeleted); } } ``` ###### [List Thread Run Steps](https://platform.openai.com/docs/api-reference/runs/listRunSteps) Returns a list of run steps belonging to a run. ```csharp var api = new OpenAIClient(); var runStepList = await api.ThreadsEndpoint.ListRunStepsAsync("thread-id", "run-id"); // OR use extension method for convenience! var runStepList = await run.ListRunStepsAsync(); foreach (var runStep in runStepList.Items) { Debug.Log($"[{runStep.Id}] {runStep.Status} {runStep.CreatedAt} -> {runStep.ExpiresAt}"); } ``` ###### [Retrieve Thread Run Step](https://platform.openai.com/docs/api-reference/runs/getRunStep) Retrieves a run step. ```csharp var api = new OpenAIClient(); var runStep = await api.ThreadsEndpoint.RetrieveRunStepAsync("thread-id", "run-id", "step-id"); // OR use extension method for convenience! var runStep = await run.RetrieveRunStepAsync("step-id"); var runStep = await runStep.UpdateAsync(); Debug.Log($"[{runStep.Id}] {runStep.Status} {runStep.CreatedAt} -> {runStep.ExpiresAt}"); ``` ###### [Cancel Thread Run](https://platform.openai.com/docs/api-reference/runs/cancelRun) Cancels a run that is `in_progress`. ```csharp var api = new OpenAIClient(); var isCancelled = await api.ThreadsEndpoint.CancelRunAsync("thread-id", "run-id"); // OR use extension method for convenience! var isCancelled = await run.CancelAsync(); Assert.IsTrue(isCancelled); ``` #### [Vector Stores](https://platform.openai.com/docs/api-reference/vector-stores) Vector stores are used to store files for use by the `file_search` tool. - [File Search Guide](https://platform.openai.com/docs/assistants/tools/file-search) The Vector Stores API is accessed via `OpenAIClient.VectorStoresEndpoint` ##### [List Vector Stores](https://platform.openai.com/docs/api-reference/vector-stores/list) Returns a list of vector stores. ```csharp var api = new OpenAIClient(); var vectorStores = await api.VectorStoresEndpoint.ListVectorStoresAsync(); foreach (var vectorStore in vectorStores.Items) { Debug.Log(vectorStore); } ``` ##### [Create Vector Store](https://platform.openai.com/docs/api-reference/vector-stores/create) Create a vector store. ```csharp var api = new OpenAIClient(); var createVectorStoreRequest = new CreateVectorStoreRequest("test-vector-store"); var vectorStore = await api.VectorStoresEndpoint.CreateVectorStoreAsync(createVectorStoreRequest); Debug.Log(vectorStore); ``` ##### [Retrieve Vector Store](https://platform.openai.com/docs/api-reference/vector-stores/retrieve) Retrieves a vector store. ```csharp var api = new OpenAIClient(); var vectorStore = await api.VectorStoresEndpoint.GetVectorStoreAsync("vector-store-id"); Debug.Log(vectorStore); ``` ##### [Modify Vector Store](https://platform.openai.com/docs/api-reference/vector-stores/modify) Modifies a vector store. ```csharp var api = new OpenAIClient(); var metadata = new Dictionary { { "Test", DateTime.UtcNow } }; var vectorStore = await api.VectorStoresEndpoint.ModifyVectorStoreAsync("vector-store-id", metadata: metadata); Debug.Log(vectorStore); ``` ##### [Delete Vector Store](https://platform.openai.com/docs/api-reference/vector-stores/delete) Delete a vector store. ```csharp var api = new OpenAIClient(); var isDeleted = await api.VectorStoresEndpoint.DeleteVectorStoreAsync("vector-store-id"); Assert.IsTrue(isDeleted); ``` ##### [Vector Store Files](https://platform.openai.com/docs/api-reference/vector-stores-files) Vector store files represent files inside a vector store. - [File Search Guide](https://platform.openai.com/docs/assistants/tools/file-search) ###### [List Vector Store Files](https://platform.openai.com/docs/api-reference/vector-stores-files/listFiles) Returns a list of vector store files. ```csharp var api = new OpenAIClient(); var files = await api.VectorStoresEndpoint.ListVectorStoreFilesAsync("vector-store-id"); foreach (var file in vectorStoreFiles.Items) { Debug.Log(file); } ``` ###### [Create Vector Store File](https://platform.openai.com/docs/api-reference/vector-stores-files/createFile) Create a vector store file by attaching a file to a vector store. ```csharp var api = new OpenAIClient(); var file = await api.VectorStoresEndpoint.CreateVectorStoreFileAsync("vector-store-id", "file-id", new ChunkingStrategy(ChunkingStrategyType.Static)); Debug.Log(file); ``` ###### [Retrieve Vector Store File](https://platform.openai.com/docs/api-reference/vector-stores-files/getFile) Retrieves a vector store file. ```csharp var api = new OpenAIClient(); var file = await api.VectorStoresEndpoint.GetVectorStoreFileAsync("vector-store-id", "vector-store-file-id"); Debug.Log(file); ``` ###### [Delete Vector Store File](https://platform.openai.com/docs/api-reference/vector-stores-files/deleteFile) Delete a vector store file. This will remove the file from the vector store but the file itself will not be deleted. To delete the file, use the delete file endpoint. ```csharp var api = new OpenAIClient(); var isDeleted = await api.VectorStoresEndpoint.DeleteVectorStoreFileAsync("vector-store-id", vectorStoreFile); Assert.IsTrue(isDeleted); ``` ##### [Vector Store File Batches](https://platform.openai.com/docs/api-reference/vector-stores-file-batches) Vector store files represent files inside a vector store. - [File Search Guide](https://platform.openai.com/docs/assistants/tools/file-search) ###### [Create Vector Store File Batch](https://platform.openai.com/docs/api-reference/vector-stores-file-batches/createBatch) Create a vector store file batch. ```csharp var api = new OpenAIClient(); var files = new List { "file_id_1","file_id_2" }; var vectorStoreFileBatch = await api.VectorStoresEndpoint.CreateVectorStoreFileBatchAsync("vector-store-id", files); Debug.Log(vectorStoreFileBatch); ``` ###### [Retrieve Vector Store File Batch](https://platform.openai.com/docs/api-reference/vector-stores-file-batches/getBatch) Retrieves a vector store file batch. ```csharp var api = new OpenAIClient(); var vectorStoreFileBatch = await api.VectorStoresEndpoint.GetVectorStoreFileBatchAsync("vector-store-id", "vector-store-file-batch-id"); // you can also use convenience methods! vectorStoreFileBatch = await vectorStoreFileBatch.UpdateAsync(); vectorStoreFileBatch = await vectorStoreFileBatch.WaitForStatusChangeAsync(); ``` ###### [List Files In Vector Store Batch](https://platform.openai.com/docs/api-reference/vector-stores-file-batches/listBatchFiles) Returns a list of vector store files in a batch. ```csharp var api = new OpenAIClient(); var files = await api.VectorStoresEndpoint.ListVectorStoreBatchFilesAsync("vector-store-id", "vector-store-file-batch-id"); foreach (var file in files.Items) { Debug.Log(file); } ``` ###### [Cancel Vector Store File Batch](https://platform.openai.com/docs/api-reference/vector-stores-file-batches/cancelBatch) Cancel a vector store file batch. This attempts to cancel the processing of files in this batch as soon as possible. ```csharp var api = new OpenAIClient(); var isCancelled = await api.VectorStoresEndpoint.CancelVectorStoreFileBatchAsync("vector-store-id", "vector-store-file-batch-id"); ``` --- ### [Chat](https://platform.openai.com/docs/api-reference/chat) Given a chat conversation, the model will return a chat completion response. The Chat API is accessed via `OpenAIClient.ChatEndpoint` #### [Chat Completions](https://platform.openai.com/docs/api-reference/chat/create) Creates a completion for the chat message ```csharp var api = new OpenAIClient(); var messages = new List { new Message(Role.System, "You are a helpful assistant."), new Message(Role.User, "Who won the world series in 2020?"), new Message(Role.Assistant, "The Los Angeles Dodgers won the World Series in 2020."), new Message(Role.User, "Where was it played?"), }; var chatRequest = new ChatRequest(messages, Model.GPT4o); var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest); var choice = response.FirstChoice; Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice.Message} | Finish Reason: {choice.FinishReason}"); ``` #### [Chat Streaming](https://platform.openai.com/docs/api-reference/chat/create#chat/create-stream) ```csharp var api = new OpenAIClient(); var messages = new List { new Message(Role.System, "You are a helpful assistant."), new Message(Role.User, "Who won the world series in 2020?"), new Message(Role.Assistant, "The Los Angeles Dodgers won the World Series in 2020."), new Message(Role.User, "Where was it played?"), }; var chatRequest = new ChatRequest(messages); var response = await api.ChatEndpoint.StreamCompletionAsync(chatRequest, async partialResponse => { Debug.Log(partialResponse.FirstChoice.Delta.ToString()); await Task.CompletedTask; }); var choice = response.FirstChoice; Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice.Message} | Finish Reason: {choice.FinishReason}"); ``` #### [Chat Tools](https://platform.openai.com/docs/guides/function-calling) ```csharp var api = new OpenAIClient(); var messages = new List { new(Role.System, "You are a helpful weather assistant. Always prompt the user for their location."), new Message(Role.User, "What's the weather like today?"), }; foreach (var message in messages) { Debug.Log($"{message.Role}: {message}"); } // Define the tools that the assistant is able to use: // 1. Get a list of all the static methods decorated with FunctionAttribute var tools = Tool.GetAllAvailableTools(includeDefaults: false, forceUpdate: true, clearCache: true); // 2. Define a custom list of tools: var tools = new List { Tool.GetOrCreateTool(objectInstance, "TheNameOfTheMethodToCall"), Tool.FromFunc("a_custom_name_for_your_function", ()=> { /* Some logic to run */ }) }; var chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto"); var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest); messages.Add(response.FirstChoice.Message); Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice} | Finish Reason: {response.FirstChoice.FinishReason}"); var locationMessage = new Message(Role.User, "I'm in Glasgow, Scotland"); messages.Add(locationMessage); Debug.Log($"{locationMessage.Role}: {locationMessage.Content}"); chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto"); response = await api.ChatEndpoint.GetCompletionAsync(chatRequest); messages.Add(response.FirstChoice.Message); if (response.FirstChoice.FinishReason == "stop") { Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice} | Finish Reason: {response.FirstChoice.FinishReason}"); var unitMessage = new Message(Role.User, "Fahrenheit"); messages.Add(unitMessage); Debug.Log($"{unitMessage.Role}: {unitMessage.Content}"); chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto"); response = await api.ChatEndpoint.GetCompletionAsync(chatRequest); } // iterate over all tool calls and invoke them foreach (var toolCall in response.FirstChoice.Message.ToolCalls) { Debug.Log($"{response.FirstChoice.Message.Role}: {toolCall.Function.Name} | Finish Reason: {response.FirstChoice.FinishReason}"); Debug.Log($"{toolCall.Function.Arguments}"); // Invokes function to get a generic json result to return for tool call. var functionResult = await toolCall.InvokeFunctionAsync(); // If you know the return type and do additional processing you can use generic overload var functionResult = await toolCall.InvokeFunctionAsync(); messages.Add(new Message(toolCall, functionResult)); Debug.Log($"{Role.Tool}: {functionResult}"); } // System: You are a helpful weather assistant. // User: What's the weather like today? // Assistant: Sure, may I know your current location? | Finish Reason: stop // User: I'm in Glasgow, Scotland // Assistant: GetCurrentWeather | Finish Reason: tool_calls // { // "location": "Glasgow, Scotland", // "unit": "celsius" // } // Tool: The current weather in Glasgow, Scotland is 39°C. ``` #### [Chat Vision](https://platform.openai.com/docs/guides/vision) > [!WARNING] > Beta Feature. API subject to breaking changes. ```csharp var api = new OpenAIClient(); var messages = new List { new Message(Role.System, "You are a helpful assistant."), new Message(Role.User, new List { "What's in this image?", new ImageUrl("https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", ImageDetail.Low) }) }; var chatRequest = new ChatRequest(messages, model: Model.GPT4o); var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest); Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice.Message.Content} | Finish Reason: {response.FirstChoice.FinishDetails}"); ``` You can even pass in a `Texture2D`! ```csharp var api = new OpenAIClient(); var messages = new List { new Message(Role.System, "You are a helpful assistant."), new Message(Role.User, new List { "What's in this image?", texture }) }; var chatRequest = new ChatRequest(messages, model: Model.GPT4o); var result = await api.ChatEndpoint.GetCompletionAsync(chatRequest); Debug.Log($"{result.FirstChoice.Message.Role}: {result.FirstChoice} | Finish Reason: {result.FirstChoice.FinishDetails}"); ``` #### [Chat Audio](https://platform.openai.com/docs/guides/audio) ```csharp var api = new OpenAIClient(); var messages = new List { new Message(Role.System, "You are a helpful assistant."), new Message(Role.User, "Is a golden retriever a good family dog?") }; var chatRequest = new ChatRequest(messages, Model.GPT4oAudio, audioConfig: Voice.Alloy); var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest); Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice} | Finish Reason: {response.FirstChoice.FinishDetails}"); audioSource.PlayOneShot(response.FirstChoice.Message.AudioOutput.AudioClip); ``` #### [Chat Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) The evolution of [Json Mode](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#chat-json-mode). While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence. > [!IMPORTANT] > > - When using JSON mode, always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don't forget, the API will throw an error if the string "JSON" does not appear somewhere in the context. > - The JSON in the message the model returns may be partial (i.e. cut off) if `finish_reason` is length, which indicates the generation exceeded max_tokens or the conversation exceeded the token limit. To guard against this, check `finish_reason` before parsing the response. First define the structure of your responses. These will be used as your schema. These are the objects you'll deserialize to, so be sure to use standard Json object models. ```csharp public class MathResponse { [JsonProperty("steps")] public IReadOnlyList Steps { get; private set; } [JsonProperty("final_answer")] public string FinalAnswer { get; private set; } } public class MathStep { [JsonProperty("explanation")] public string Explanation { get; private set; } [JsonProperty("output")] public string Output { get; private set; } } ``` To use, simply specify the `MathResponse` type as a generic constraint when requesting a completion. ```csharp var api = new OpenAIClient(); var messages = new List { new(Role.System, "You are a helpful math tutor. Guide the user through the solution step by step."), new(Role.User, "how can I solve 8x + 7 = -23") }; var chatRequest = new ChatRequest(messages, model: "gpt-4o-2024-08-06"); var (mathResponse, chatResponse) = await api.ChatEndpoint.GetCompletionAsync(chatRequest); for (var i = 0; i < mathResponse.Steps.Count; i++) { var step = mathResponse.Steps[i]; Debug.Log($"Step {i}: {step.Explanation}"); Debug.Log($"Result: {step.Output}"); } Debug.Log($"Final Answer: {mathResponse.FinalAnswer}"); chatResponse.GetUsage(); ``` #### [Chat Json Mode](https://platform.openai.com/docs/guides/text-generation/json-mode) > [!IMPORTANT] > > - When using JSON mode, always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don't forget, the API will throw an error if the string "JSON" does not appear somewhere in the context. > - The JSON in the message the model returns may be partial (i.e. cut off) if `finish_reason` is length, which indicates the generation exceeded max_tokens or the conversation exceeded the token limit. To guard against this, check `finish_reason` before parsing the response. > - JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors. ```csharp var messages = new List { new Message(Role.System, "You are a helpful assistant designed to output JSON."), new Message(Role.User, "Who won the world series in 2020?"), }; var chatRequest = new ChatRequest(messages, Model.GPT4o, responseFormat: ChatResponseFormat.Json); var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest); foreach (var choice in response.Choices) { Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice} | Finish Reason: {choice.FinishReason}"); } response.GetUsage(); ``` --- ### [Audio](https://platform.openai.com/docs/api-reference/audio) Converts audio into text. The Audio API is accessed via `OpenAIClient.AudioEndpoint` #### [Create Speech](https://platform.openai.com/docs/api-reference/audio/createSpeech) Generates audio from the input text. ```csharp var api = new OpenAIClient(); var request = new SpeechRequest("Hello world!"); var speechClip = await api.AudioEndpoint.GetSpeechAsync(request); audioSource.PlayOneShot(speechClip); Debug.Log(speechClip); ``` ##### [Stream Speech] Generate streamed audio from the input text. ```csharp var api = new OpenAIClient(); var request = new SpeechRequest("Hello world!", responseFormat: SpeechResponseFormat.PCM); var speechClip = await api.AudioEndpoint.GetSpeechAsync(request, partialClip => { audioSource.PlayOneShot(partialClip); }); Debug.Log(speechClip); ``` > [!NOTE] > Checkout any of the demo scenes for best practices on how to handle playback with `OnAudioFilterRead`. #### [Create Transcription](https://platform.openai.com/docs/api-reference/audio/createTranscription) Transcribes audio into the input language. ```csharp var api = new OpenAIClient(); var request = new AudioTranscriptionRequest(audioClip, language: "en"); var result = await api.AudioEndpoint.CreateTranscriptionAsync(request); Debug.Log(result); ``` You can also get detailed information using `verbose_json` to get timestamp granularities: ```csharp var api = new OpenAIClient(); using var request = new AudioTranscriptionRequest(transcriptionAudio, responseFormat: AudioResponseFormat.Verbose_Json, timestampGranularity: TimestampGranularity.Word, temperature: 0.1f, language: "en"); var response = await api.AudioEndpoint.CreateTranscriptionTextAsync(request); foreach (var word in response.Words) { Debug.Log($"[{word.Start}-{word.End}] \"{word.Word}\""); } ``` #### [Create Translation](https://platform.openai.com/docs/api-reference/audio/createTranslation) Translates audio into into English. ```csharp var api = new OpenAIClient(); var request = new AudioTranslationRequest(audioClip); var result = await api.AudioEndpoint.CreateTranslationAsync(request); Debug.Log(result); ``` --- ### [Images](https://platform.openai.com/docs/api-reference/images) Given a prompt and/or an input image, the model will generate a new image. The Images API is accessed via `OpenAIClient.ImagesEndpoint` #### [Create Image](https://platform.openai.com/docs/api-reference/images/create) Creates an image given a prompt. ```csharp var api = new OpenAIClient(); var request = new ImageGenerationRequest("A house riding a velociraptor", Models.Model.DallE_3); var imageResults = await api.ImagesEndPoint.GenerateImageAsync(request); foreach (var result in imageResults) { Debug.Log(result.ToString()); Assert.IsNotNull(result.Texture); } ``` #### [Edit Image](https://platform.openai.com/docs/api-reference/images/create-edit) Creates an edited or extended image given an original image and a prompt. ```csharp var api = new OpenAIClient(); var request = new ImageEditRequest(Path.GetFullPath(imageAssetPath), Path.GetFullPath(maskAssetPath), "A sunlit indoor lounge area with a pool containing a flamingo", size: ImageSize.Small); var imageResults = await api.ImagesEndPoint.CreateImageEditAsync(request); foreach (var result in imageResults) { Debug.Log(result.ToString()); Assert.IsNotNull(result.Texture); } ``` #### [Create Image Variation](https://platform.openai.com/docs/api-reference/images/create-variation) Creates a variation of a given image. ```csharp var api = new OpenAIClient(); var request = new ImageVariationRequest(imageTexture, size: ImageSize.Small); var imageResults = await api.ImagesEndPoint.CreateImageVariationAsync(request); foreach (var result in imageResults) { Debug.Log(result.ToString()); Assert.IsNotNull(result.Texture); } ``` Alternatively, the endpoint can directly take a Texture2D with Read/Write enabled and Compression set to None. ```csharp var api = new OpenAIClient(); var request = new ImageVariationRequest(imageTexture, size: ImageSize.Small); var imageResults = await api.ImagesEndPoint.CreateImageVariationAsync(request); foreach (var result in imageResults) { Debug.Log(result.ToString()); Assert.IsNotNull(result.Texture); } ``` --- ### [Files](https://platform.openai.com/docs/api-reference/files) Files are used to upload documents that can be used with features like [Fine-tuning](https://github.com/RageAgainstThePixel/com.openai.unity/raw/main/#fine-tuning). The Files API is accessed via `OpenAIClient.FilesEndpoint` #### [List Files](https://platform.openai.com/docs/api-reference/files/list) Returns a list of files that belong to the user's organization. ```csharp var api = new OpenAIClient(); var fileList = await api.FilesEndpoint.ListFilesAsync(); foreach (var file in fileList) { Debug.Log($"{file.Id} -> {file.Object}: {file.FileName} | {file.Size} bytes"); } ``` #### [Upload File](https://platform.openai.com/docs/api-reference/files/create) Upload a file that can be used across various endpoints. The size of all the files uploaded by one organization can be up to 100 GB. The size of individual files can be a maximum of 512 MB. See the Assistants Tools guide to learn more about the types of files supported. The Fine-tuning API only supports .jsonl files. ```csharp var api = new OpenAIClient(); var file = await api.FilesEndpoint.UploadFileAsync("path/to/your/file.jsonl", FilePurpose.FineTune); Debug.Log(file.Id); ``` #### [Delete File](https://platform.openai.com/docs/api-reference/files/delete) Delete a file. ```csharp var api = new OpenAIClient(); var isDeleted = await api.FilesEndpoint.DeleteFileAsync(fileId); Assert.IsTrue(isDeleted); ``` #### [Retrieve File Info](https://platform.openai.com/docs/api-reference/files/retrieve) Returns information about a specific file. ```csharp var api = new OpenAIClient(); var file = await api.FilesEndpoint.GetFileInfoAsync(fileId); Debug.Log($"{file.Id} -> {file.Object}: {file.FileName} | {file.Size} bytes"); ``` #### [Download File Content](https://platform.openai.com/docs/api-reference/files/retrieve-content) Downloads the file content to the specified directory. ```csharp var api = new OpenAIClient(); var downloadedFilePath = await api.FilesEndpoint.DownloadFileAsync(fileId); Debug.Log(downloadedFilePath); Assert.IsTrue(File.Exists(downloadedFilePath)); ``` --- ### [Fine Tuning](https://platform.openai.com/docs/api-reference/fine-tuning) Manage fine-tuning jobs to tailor a model to your specific training data. Related guide: [Fine-tune models](https://platform.openai.com/docs/guides/fine-tuning) The Files API is accessed via `OpenAIClient.FineTuningEndpoint` #### [Create Fine Tune Job](https://platform.openai.com/docs/api-reference/fine-tuning/create) Creates a job that fine-tunes a specified model from a given dataset. Response includes details of the enqueued job including job status and the name of the fine-tuned models once complete. ```csharp var api = new OpenAIClient(); var fileId = "file-abc123"; var request = new CreateFineTuneRequest(fileId); var job = await api.FineTuningEndpoint.CreateJobAsync(Model.GPT3_5_Turbo, request); Debug.Log($"Started {job.Id} | Status: {job.Status}"); ``` #### [List Fine Tune Jobs](https://platform.openai.com/docs/api-reference/fine-tuning/list) List your organization's fine-tuning jobs. ```csharp var api = new OpenAIClient(); var jobList = await api.FineTuningEndpoint.ListJobsAsync(); foreach (var job in jobList.Items.OrderByDescending(job => job.CreatedAt))) { Debug.Log($"{job.Id} -> {job.CreatedAt} | {job.Status}"); } ``` #### [Retrieve Fine Tune Job Info](https://platform.openai.com/docs/api-reference/fine-tuning/retrieve) Gets info about the fine-tune job. ```csharp var api = new OpenAIClient(); var job = await api.FineTuningEndpoint.GetJobInfoAsync(fineTuneJob); Debug.Log($"{job.Id} -> {job.CreatedAt} | {job.Status}"); ``` #### [Cancel Fine Tune Job](https://platform.openai.com/docs/api-reference/fine-tuning/cancel) Immediately cancel a fine-tune job. ```csharp var api = new OpenAIClient(); var isCancelled = await api.FineTuningEndpoint.CancelFineTuneJobAsync(fineTuneJob); Assert.IsTrue(isCancelled); ``` #### [List Fine Tune Job Events](https://platform.openai.com/docs/api-reference/fine-tuning/list-events) Get status updates for a fine-tuning job. ```csharp var api = new OpenAIClient(); var eventList = await api.FineTuningEndpoint.ListJobEventsAsync(fineTuneJob); Debug.Log($"{fineTuneJob.Id} -> status: {fineTuneJob.Status} | event count: {eventList.Events.Count}"); foreach (var @event in eventList.Items.OrderByDescending(@event => @event.CreatedAt)) { Debug.Log($" {@event.CreatedAt} [{@event.Level}] {@event.Message}"); } ``` --- ### [Batches](https://platform.openai.com/docs/api-reference/batch) Create large batches of API requests for asynchronous processing. The Batch API returns completions within 24 hours for a 50% discount. - [Batch Guide](https://platform.openai.com/docs/guides/batch) The Batches API is accessed via `OpenAIClient.BatchesEndpoint` #### [List Batches](https://platform.openai.com/docs/api-reference/batch/list) List your organization's batches. ```csharp var api = new OpenAIClient(); var batches = await api.BatchEndpoint.ListBatchesAsync(); foreach (var batch in listResponse.Items) { Debug.Log(batch); } ``` #### [Create Batch](https://platform.openai.com/docs/api-reference/batch/create) Creates and executes a batch from an uploaded file of requests ```csharp var api = new OpenAIClient(); var batchRequest = new CreateBatchRequest("file-id", Endpoint.ChatCompletions); var batch = await api.BatchEndpoint.CreateBatchAsync(batchRequest); ``` #### [Retrieve Batch](https://platform.openai.com/docs/api-reference/batch/retrieve) Retrieves a batch. ```csharp var api = new OpenAIClient(); var batch = await api.BatchEndpoint.RetrieveBatchAsync("batch-id"); // you can also use convenience methods! batch = await batch.UpdateAsync(); batch = await batch.WaitForStatusChangeAsync(); ``` #### [Cancel Batch](https://platform.openai.com/docs/api-reference/batch/cancel) Cancels an in-progress batch. The batch will be in status cancelling for up to 10 minutes, before changing to cancelled, where it will have partial results (if any) available in the output file. ```csharp var api = new OpenAIClient(); var isCancelled = await api.BatchEndpoint.CancelBatchAsync(batch); Assert.IsTrue(isCancelled); ``` --- ### [Embeddings](https://platform.openai.com/docs/api-reference/embeddings) Get a vector representation of a given input that can be easily consumed by machine learning models and algorithms. Related guide: [Embeddings](https://platform.openai.com/docs/guides/embeddings) The Edits API is accessed via `OpenAIClient.EmbeddingsEndpoint` #### [Create Embeddings](https://platform.openai.com/docs/api-reference/embeddings/create) Creates an embedding vector representing the input text. ```csharp var api = new OpenAIClient(); var response = await api.EmbeddingsEndpoint.CreateEmbeddingAsync("The food was delicious and the waiter...", Models.Embedding_Ada_002); Debug.Log(response); ``` --- ### [Moderations](https://platform.openai.com/docs/api-reference/moderations) Given a input text, outputs if the model classifies it as violating OpenAI's content policy. Related guide: [Moderations](https://platform.openai.com/docs/guides/moderation) The Moderations API can be accessed via `OpenAIClient.ModerationsEndpoint` #### [Create Moderation](https://platform.openai.com/docs/api-reference/moderations/create) Classifies if text violates OpenAI's Content Policy. ```csharp var api = new OpenAIClient(); var isViolation = await api.ModerationsEndpoint.GetModerationAsync("I want to kill them."); Assert.IsTrue(isViolation); ``` Additionally you can also get the scores of a given input. ```csharp var api = new OpenAIClient(); var response = await api.ModerationsEndpoint.CreateModerationAsync(new ModerationsRequest("I love you")); Assert.IsNotNull(response); Debug.Log(response.Results?[0]?.Scores?.ToString()); ``` ---