# Watson SDK Plugin ## Before you Begin Ensure that you have the following prerequisites: * An IBM Cloud account. If you don't have one, [sign up](https://console.bluemix.net/registration/). * Unreal Engine 4.18. ## Setup 1. If it doesn't exist already, create Plugins/ folder in your project's directory 2. Clone the plugin to your project's Plugins/ directory 3. In Unreal Engine 4, Create Pawn C++ Class 4. Add Sample Pawn Code to MyPawn.h * Replace YourProjectName in class with your project name. (ex: class WATSONUNREALENGINEPROJECT_API) 5. Add Sample Pawn Code to MyPawn.cpp * Insert your IBM Cloud credentials in the following format: "username","password" _Note: See Configure your service credentials below_ * Insert Watson Conversation Workspace ID into

FConversationMessagePendingRequest* ConvRequest = MyConversation->Message("1a2xxxxx-xxxx-xxxx-xxxx-xxxxxxxx2af", ConversationRequest);

_Note: See Configure your Watson Conversation Workspace ID below_ 6. Add the following module dependencies to your project's .Build.cs (ex: YourProjectName.Build.cs) ```csharp PrivateDependencyModuleNames.AddRange(new string[] { "WatsonSdk", "Voice" }); ``` 7. Add the following to your engine config DefaultEngine.ini ``` [OnlineSubsystem] bHasVoiceEnabled=true [Voice] bEnabled=true ``` 8. Compile 9. If you want a trigger key, Create Microphone Action Mapping 10. Place Pawn in environment ## Configure your service credentials To create instances of Watson services and their credentials, follow the steps below. **Note:** Service credentials are different from your IBM Cloud account username and password. 1. Determine which services to configure. 1. If you have configured the services already, complete the following steps. Otherwise, go to step 3. 1. Log in to Bluemix at https://bluemix.net. 1. Click the service you would like to use. 1. Click **Service credentials**. 1. Click **View credentials** to access your credentials. 1. If you need to configure the services that you want to use, complete the following steps. 1. Log in to Bluemix at https://bluemix.net. 1. Click the **Create service** button. 1. Under **Watson**, select which service you would like to create an instance of and click that service. 1. Give the service and credential a name. Select a plan and click the **Create** button on the bottom. 4. Click **Service Credentials**. 5. Click **View credentials** to access your credentials. 1. Paste your credentials in the matching line item in the MyPawn.cpp file 1. Your service credentials can be used to instantiate Watson Services within your application. Most services also support tokens which you can instantiate the service with as well. The credentials for each service contain either a `username`, `password` and endpoint `url` **or** an `apikey` and endpoint `url`. **WARNING:** You are responsible for securing your own credentials. Any user with your service credentials can access your service instances! ## Configure your Watson Conversation Workspace ID 1. Log in to Bluemix at https://bluemix.net 1. Go to Watson Conversation 1. Launch Conversation Tool by clicking Launch Tool 1. Click on the settings icon (three dots) on the Workspace you want to use 1. Click View Details 1. Copy & Past Workspace ID into MyPawn.cpp ## Create Microphone Action Mapping 1. Edit -> Project Settings -> Input -> Bindings -> Action Mapping -> Name = Microphone -> Choose a microphone hotkey -> Close ## Usage ### Watson ```cpp // Create Watson Instance MyWatson = CreateDefaultSubobject(TEXT("Watson")); ``` ### Watson Conversation ```cpp // Create UConversation UConversation* MyConversation = MyWatson->CreateConversation(FAuthentication(/* username */, /* password */)); // Create and send request FConversationMessageRequest ConversationRequest; ConversationRequest.input.text = "Hello there, how are you?"; FConversationMessagePendingRequest* Request = MyConversation->Message("1a2cdfcd-e42a-43f7-947c-ba44ebf4f2af", ConversationRequest); Request->OnSuccess.BindUObject(this, &AMyPawn::OnConversationMessage); Request->OnFailure.BindUObject(this, &AMyPawn::OnConversationFailure); Request->Send(); ``` ### Watson Speech To Text ```cpp // Create USpeechToText USpeechToText* MySpeechToText = MyWatson->CreateSpeechToText(FAuthentication(/* username */, /* password */)); // Create and send request FSpeechToTextRecognizePendingRequest* Request = MySpeechToText->Recognize(MyMicrophone->GetRecording()); Request->OnSuccess.BindUObject(this, &AMyPawn::OnSpeechToTextRecognize); Request->OnFailure.BindUObject(this, &AMyPawn::OnSpeechToTextFailure); Request->Send(); ``` ### Watson Text To Speech ```cpp // Create UTextToSpeech UTextToSpeech* MyTextToSpeech = MyWatson->CreateTextToSpeech(FAuthentication(/* username */, /* password */)); // Create and send request FTextToSpeechSynthesizeRequest SynthesisRequest; SynthesisRequest.text = "Hello there, how are you?"; FTextToSpeechSynthesizePendingRequest* T2sRequest = MyTextToSpeech->Synthesize(SynthesisRequest, "en-US_AllisonVoice"); T2sRequest->OnSuccess.BindUObject(this, &AMyPawn::OnTextToSpeechSynthesize); T2sRequest->OnFailure.BindUObject(this, &AMyPawn::OnTextToSpeechFailure); T2sRequest->Send(); ``` ### Speaker ```cpp USpeaker* MySpeaker = CreateDefaultSubobject(TEXT("Speaker")); MySpeaker->PlayAudio(/* raw audio */, /* raw audio length */); ``` ### Microphone ```cpp UMicrophone* MyMicrophone = CreateDefaultSubobject(TEXT("Microphone")); MyMicrophone->StartRecording(); MyMicrophone->StopRecording(); MyMicrophone->GetRecording(); ``` ## Sample Pawn Code ### MyPawn.h ```cpp #pragma once #include "CoreMinimal.h" #include "GameFramework/Pawn.h" #include "Watson.h" #include "Common/Speaker.h" #include "Common/Microphone.h" #include "MyPawn.generated.h" //Enter Your Project Name Below UCLASS() class AMyPawn : public APawn { GENERATED_BODY() public: AMyPawn(); void BeginPlay() override; void Tick(float DeltaTime) override; void SetupPlayerInputComponent(class UInputComponent* InputComponent); protected: // Speaker UPROPERTY() USpeaker* MySpeaker; // Microphone UPROPERTY() UMicrophone* MyMicrophone; void OnMicrophoneStart(); void OnMicrophoneStop(); // Watson UPROPERTY() UWatson* MyWatson; // Conversation UPROPERTY() UConversation* MyConversation; TSharedPtr LastResponse; void OnConversationMessage(TSharedPtr Response); void OnConversationFailure(FString Error); // TextToSpeech UPROPERTY() UTextToSpeech* MyTextToSpeech; void OnTextToSpeechSynthesize(TSharedPtr Response); void OnTextToSpeechFailure(FString Error); // SpeechToText UPROPERTY() USpeechToText* MySpeechToText; void OnSpeechToTextRecognize(TSharedPtr Response); void OnSpeechToTextFailure(FString Error); }; ``` ### MyPawn.cpp ``` cpp // Here's a snippet that allows you to converse with Watson Conversation using Speech-To-Text & Text-To-Speech. // Make sure to create a Microphone Action Mapping for push-to-talk behavior. #include "MyPawn.h" AMyPawn::AMyPawn() { PrimaryActorTick.bCanEverTick = true; AutoPossessPlayer = EAutoReceiveInput::Player0; MyWatson = CreateDefaultSubobject(TEXT("Watson")); MySpeaker = CreateDefaultSubobject(TEXT("Speaker")); MyMicrophone = CreateDefaultSubobject(TEXT("Microphone")); MyConversation = MyWatson->CreateConversation(FAuthentication("ENTER WATSON SERVICE CREDENTIALS HERE")); MyTextToSpeech = MyWatson->CreateTextToSpeech(FAuthentication("ENTER WATSON SERVICE CREDENTIALS HERE")); MySpeechToText = MyWatson->CreateSpeechToText(FAuthentication("970ef92d-xxxxxxxxxxxxxxx", "xxxxxxXXXXXX")); } void AMyPawn::SetupPlayerInputComponent(UInputComponent* InputComponent) { Super::SetupPlayerInputComponent(InputComponent); InputComponent->BindAction("Microphone", IE_Pressed, this, &AMyPawn::OnMicrophoneStart); InputComponent->BindAction("Microphone", IE_Released, this, &AMyPawn::OnMicrophoneStop); } void AMyPawn::BeginPlay() { Super::BeginPlay(); } void AMyPawn::Tick(float DeltaTime) { Super::Tick(DeltaTime); } void AMyPawn::OnMicrophoneStart() { UE_LOG(LogTemp, Warning, TEXT("Microphone Starting...")); MyMicrophone->StartRecording(); } void AMyPawn::OnMicrophoneStop() { UE_LOG(LogTemp, Warning, TEXT("Microphone Stopping...")); MyMicrophone->StopRecording(); // Make Speech To Text Request FSpeechToTextRecognizePendingRequest* Request = MySpeechToText->Recognize(MyMicrophone->GetRecording()); Request->OnSuccess.BindUObject(this, &AMyPawn::OnSpeechToTextRecognize); Request->OnFailure.BindUObject(this, &AMyPawn::OnSpeechToTextFailure); Request->Send(); } void AMyPawn::OnConversationMessage(TSharedPtr Response) { UE_LOG(LogTemp, Warning, TEXT("Conversation Success: %s"), *Response->output.text.Last()); LastResponse = Response; // Make Text To Speech Request FTextToSpeechSynthesizeRequest SynthesisRequest; SynthesisRequest.text = Response->output.text.Last(); FTextToSpeechSynthesizePendingRequest* T2sRequest = MyTextToSpeech->Synthesize(SynthesisRequest, "en-US_AllisonVoice"); T2sRequest->OnSuccess.BindUObject(this, &AMyPawn::OnTextToSpeechSynthesize); T2sRequest->OnFailure.BindUObject(this, &AMyPawn::OnTextToSpeechFailure); T2sRequest->Send(); } void AMyPawn::OnConversationFailure(FString Error) { UE_LOG(LogTemp, Warning, TEXT("Conversation Error: %s"), *Error); } void AMyPawn::OnSpeechToTextRecognize(TSharedPtr Response) { UE_LOG(LogTemp, Warning, TEXT("Speech To Text Success: %s"), *Response->results[0].alternatives[0].transcript); // Make Conversation Request FConversationMessageRequest ConversationRequest; ConversationRequest.input.text = Response->results[0].alternatives[0].transcript; if (LastResponse.Get() != nullptr) { ConversationRequest.context = LastResponse->context; ConversationRequest.output = LastResponse->output; } // Replace string with Workspace ID in IBM Cloud Conversation FConversationMessagePendingRequest* ConvRequest = MyConversation->Message("1a2xxxxx-xxxx-xxxx-xxxx-xxxxxxxx2af", ConversationRequest); ConvRequest->OnSuccess.BindUObject(this, &AMyPawn::OnConversationMessage); ConvRequest->OnFailure.BindUObject(this, &AMyPawn::OnConversationFailure); ConvRequest->Send(); } void AMyPawn::OnSpeechToTextFailure(FString Error) { UE_LOG(LogTemp, Warning, TEXT("Speech To Text Error: %s"), *Error); } void AMyPawn::OnTextToSpeechSynthesize(TSharedPtr Response) { MySpeaker->PlayAudio(Response->audioData, Response->audioLength); } void AMyPawn::OnTextToSpeechFailure(FString Error) { UE_LOG(LogTemp, Warning, TEXT("Text To Speech Error: %s"), *Error); } ```