AIVA Loan Eligibility Survey: Create Speech based interface

Note:

• This project has a comparatively longer deployment process. However, once deployed, additional development is simple. Thus, you only need to overcome the pain of deployment once to have a smooth development experience.

• This contest has subjective review due to the nature of the project. It will still have Appeals and Appeals response phases though.

 

Challenge Objectives

• Update existing text based chat application to use speech as well

 

Project Background

• Our client is looking to create a chat bot that interacts with the user using a browser, but the chat is audio based - the bot will speak to the user while the user will speak back to the bot

• We have built the backend for the chat bot using Amazon Lex and AWS Lambda and the front end (which is currently text based) using Angular 6

• We have also created the AWS Cloudformation Template for the app, for easy deployment of the app in the AWS ecosystem.

 

Technology Stack

• AWS Lambda

• Amazon Lex

• Angular (version 6) - for the UI

 

Code access

• We will be sharing the code base for the app in the contest forum. Please go through the README.md file present to understand how you can deploy the app

 

Individual requirements

Update Angular UI to support Speech

• Update the existing UI written in Angular. Currently, it has a simple text based chat interface. Update the app to also have a speech based interface.

• You would have to capture the speech of the user. This reference might be helpful. Once captured, pass the audio stream to Amazon Lex.

• In addition to the text showing up on the screen, the text will also be read out to the user. Both, the text that shows up on the screen as well as the audio that is read out to the user will arrive from Amazon Lex. You can go through the documentation and understand how this can be carried out by reading the inputTranscript, message and audioStream attributes in the response.

• The process for speech based interaction would thus be as follows:

  • User is presented with an introductory text. At the same time, an audio is played back to the user, which is basically the text in speech form.

  • User speaks back to the app. The speech is captured and sent to Amazon Lex. Show a processing indicator (something that shows that the audio has been captured and is being processed or something that indicates that the bot is attempting to respond) until Amazon Lex returns with a response.

  • Since the response contains the text as interpreted by Lex, you would thus show the text corresponding the user’s speech only after it has arrived in the response from Lex and not immediately. Hence, from a user experience perspective, it is important that you indicate to the user in some way that they need to wait and that the app has listened to them.

  • Once the response arrives from Lex, the user’s text and the text of the bot’s response will show up and the audio from the bot will be played.

• The design for the speech interface is modelled on Apple’s Siri. We will share a document in the contest forum describing this in detail.

• It is important that you use the audio response from Lex itself, which uses Amazon Polly internally and NOT accept only text as response or process the response using any third party tools (no man-in-the-middle approach to process response).

• Ensure that the UI works well on Google Chrome for mobile devices. Only mobile device needs to be supported

• At the end of this task, the user should be able to carry out a complete conversation with the app, where internally, Amazon Lex and Lambda functions are responding to the user’s inputs. To put it in another way, update the existing app to have both text based interface and speech based interface. Everything else should work as it does before.

Update Backend to support speech (Major Requirement)

• You may have to update the backend to support speech based input / output. Please ensure that your updates are restricted to only supporting speech based input and output and no changes are carried out that affect the core aspect of the Chatbot (the brain of the chatbot that takes the decision on how to respond to the user)

• You may also have to update the AWS Cloudformation template as needed to ensure that the deployment process is still through cloudformation itself. The existing deployment step is aimed at providing a single-step deployment of the entire project in the AWS ecosystem and we expect the same with your submission as well.

 

Deployment guide and validation document

• Deployment guide is a major requirement here. It is critical that you update the existing deployment guide (README.md) with any steps necessary to ensure that we are able to deploy the app to the AWS ecosystem. You need to update both the Amazon CloudFormation based deployment as well as Individual components deployment sections as necessary.

Final Submission Guidelines

• The only change expected for this contest is to allow the end user to interact with the chatbot through speech. As such, you will not be making any changes to the core part of the application - the chatbot itself and the lambda function that is responsible for helping the chatbot come up with the appropriate response.

• You need to zip the code base and upload to topcoder. Alternatively, you can upload a git patch of the changes made.