Domc.Dev
Published on

Building an AI Avatar Chatbot (Part 1)

Authors

I wanted to build something fun and unique for my portfolio / blog home page that adds a level of interactivity the site. I decided to build an AI version of myself that users can interact with. In the future I'm planning on adding the following:

  • Blinking animation set at random intervals
  • Mouth moving speech animation for responses
  • Cyber/robot exposed skin to AI-ify the avatar (think Terminator 2)
  • Allow the chatbot to navigate users to specific pages

Creating the 3D model

Originally I planned on using my iPhone and the 3dscannerapp to export my head model into blender and use that as a starting point, however it turned out creepy and too lifelike. After a bit of research I stumbled upon Ready Player Me which creates a 3d model of your entire body using a selfie. As I'm only interested in the head and neck I removed everything but and aligned the head to the center of the scene for exporting.

blender

A nice feature of Ready Player Me is that it separates the assets and texture files for you making it much easier to apply the correct mesh textures later on.

Getting the model into my Next.js App

For getting the model loaded into my next app I used react-three-fibre and drei helpers for texture loading. I used gltfjsx to convert my exported glb file into a jsx component ready for importing. This creates the following JSX file:

/*
Auto-generated by: https://github.com/pmndrs/gltfjsx
*/

import React, { useRef } from 'react'
import { useGLTF } from '@react-three/drei'

export default function Model({ ...props }) {
  const group = useRef()
  const { nodes, materials } = useGLTF('/domcenter.glb')
  return (
    <group ref={group} {...props} dispose={null}>
      <mesh
        geometry={nodes.Wolf3D_Hair.geometry}
        material={materials.Wolf3D_Hair}
        position={[0, 0.1, -0.02]}
      />
      <mesh
        name="EyeLeft"
        geometry={nodes.EyeLeft.geometry}
        material={materials.Wolf3D_Eye}
        morphTargetDictionary={nodes.EyeLeft.morphTargetDictionary}
        morphTargetInfluences={nodes.EyeLeft.morphTargetInfluences}
        position={[0.03, 0.06, 0.07]}
      />
      <mesh
        name="EyeRight"
        geometry={nodes.EyeRight.geometry}
        material={materials.Wolf3D_Eye}
        morphTargetDictionary={nodes.EyeRight.morphTargetDictionary}
        morphTargetInfluences={nodes.EyeRight.morphTargetInfluences}
        position={[-0.03, 0.06, 0.07]}
      />
      <mesh
        name="Wolf3D_Head"
        geometry={nodes.Wolf3D_Head.geometry}
        material={materials.Wolf3D_Skin}
        morphTargetDictionary={nodes.Wolf3D_Head.morphTargetDictionary}
        morphTargetInfluences={nodes.Wolf3D_Head.morphTargetInfluences}
        position={[0, 0.03, 0.05]}
      />
      <mesh
        name="Wolf3D_Teeth"
        geometry={nodes.Wolf3D_Teeth.geometry}
        material={materials.Wolf3D_Teeth}
        morphTargetDictionary={nodes.Wolf3D_Teeth.morphTargetDictionary}
        morphTargetInfluences={nodes.Wolf3D_Teeth.morphTargetInfluences}
        position={[0, 0, 0.06]}
      />
    </group>
  )
}

useGLTF.preload('/domcenter.glb')

It's a fairly well documented process to load models into react, however using Nextjs adds a layer of complexity and there are a few things to watch out for to avoid errors.

  • Lazy import your models
  • Make sure you don't SSR any pages related to the threejs model
  • Use react suspense for managing loading states

Head following mouse

This function called getMouseDegrees is used to track where the mouse location inside the canvas and adjust the rotation of the head model in relation to mouse position in the canvas:

export function getMouseDegrees(x, y, degreeLimit) {
  let dx = 0,
    dy = 0,
    xdiff,
    xPercentage,
    ydiff,
    yPercentage

  let w = { x: window.innerWidth, y: window.innerHeight }

  // Left (Rotates neck left between 0 and -degreeLimit)
  // 1. If cursor is in the left half of screen
  if (x <= w.x / 2) {
    // 2. Get the difference between middle of screen and cursor position
    xdiff = w.x / 2 - x
    // 3. Find the percentage of that difference (percentage toward edge of screen)
    xPercentage = (xdiff / (w.x / 2)) * 100
    // 4. Convert that to a percentage of the maximum rotation we allow for the neck
    dx = ((degreeLimit * xPercentage) / 100) * -1
  }

  // Right (Rotates neck right between 0 and degreeLimit)
  if (x >= w.x / 2) {
    xdiff = x - w.x / 2
    xPercentage = (xdiff / (w.x / 2)) * 100
    dx = (degreeLimit * xPercentage) / 100
  }
  // Up (Rotates neck up between 0 and -degreeLimit)
  if (y <= w.y / 2) {
    ydiff = w.y / 2 - y
    yPercentage = (ydiff / (w.y / 2)) * 100
    // Note that I cut degreeLimit in half when she looks up
    dy = ((degreeLimit * 0.5 * yPercentage) / 100) * -1
  }
  // Down (Rotates neck down between 0 and degreeLimit)
  if (y >= w.y / 2) {
    ydiff = y - w.y / 2
    yPercentage = (ydiff / (w.y / 2)) * 100
    dy = (degreeLimit * yPercentage) / 100
  }
  return { x: dx, y: dy }
}

Then you import this function into the jsx file created by gltfjsx as a prerequisite to the head rotation / mouse movement tracking.

Declare two states for xrotation and yrotation:

  const [xrotation, setXRotation] = useState(0)
  const [yrotation, setYRotation] = useState(0)

Import useFrame from react-three-drei and once everything id glued together you'll end up with something like this:

import * as THREE from 'three'
import React, { useRef, useState, useEffect } from 'react'
import { useGLTF, useTexture } from '@react-three/drei'
import { useThree, useLoader, useFrame } from '@react-three/fiber'
import { getMouseDegrees } from './utils'

function moveJoint(mouse, joint, degreeLimit = 40) {
  let degrees = getMouseDegrees(mouse.x, mouse.y, degreeLimit)

  joint.rotation.xD = THREE.MathUtils.lerp(0, degrees.y, 0.1)
  joint.rotation.yD = THREE.MathUtils.lerp(0, degrees.x, 0.1)
  joint.rotation.x = THREE.Math.degToRad(joint.rotation.xD)
  joint.rotation.y = THREE.Math.degToRad(joint.rotation.yD)

  return joint.rotation
}

export default function Model({ ...props }) {
  const [xrotation, setXRotation] = useState(0)
  const [yrotation, setYRotation] = useState(0)
  const group = useRef()
  const { nodes } = useGLTF('/domneck.glb')
  const eyeTexture = useTexture('/Image_0.jpg')
  const headTexture = useTexture('/Image_5.jpg')
  const hairTexture = useTexture('/Image_3.jpg')
  const teethTexture = useTexture('/Image_14.jpg')

  const [mixer] = useState(() => new THREE.AnimationMixer())
  useFrame((state, delta) => mixer.update(delta))

  const { size } = useThree()
  useFrame((state, delta) => {
    const mouse = {
      x: size.width / 2 + (state.mouse.x * size.width) / 2,
      y: size.height / 2 + (-state.mouse.y * size.height) / 2,
    }
    mixer.update(delta)

    const x = moveJoint(mouse, nodes.Wolf3D_Head, 40).x
    const y = moveJoint(mouse, nodes.Wolf3D_Head, 40).y

    setXRotation(x)
    setYRotation(y)


  })

  return (
    <group
      ref={group}
      {...props}
      dispose={null}
      rotation={[0.1 * Math.PI, 0 * Math.PI, 0 * Math.PI]}
    >
      {/* <OrbitControls /> */}
      <group rotation={[xrotation * Math.PI, yrotation * Math.PI, 0 * Math.PI]}>
        <mesh
          geometry={nodes.Wolf3D_Hair.geometry}
          //material={materials.Wolf3D_Hair}
          scale={1}
        >
          <meshBasicMaterial color={'#ad8157'} map={hairTexture} map-flipY={false} />
        </mesh>
        <mesh
          name="EyeLeft"
          geometry={nodes.EyeLeft.geometry}
          //material={materials.Wolf3D_Eye}
          morphTargetDictionary={nodes.EyeLeft.morphTargetDictionary}
          morphTargetInfluences={nodes.EyeLeft.morphTargetInfluences}
          scale={1}
        >
          <meshBasicMaterial map={eyeTexture} map-flipY={false} />
        </mesh>
        <mesh
          name="EyeRight"
          geometry={nodes.EyeRight.geometry}
          //material={materials.Wolf3D_Eye}
          morphTargetDictionary={nodes.EyeRight.morphTargetDictionary}
          morphTargetInfluences={nodes.EyeRight.morphTargetInfluences}
          scale={1}
        >
          <meshBasicMaterial map={eyeTexture} map-flipY={false} />
        </mesh>

        <mesh
          name="Wolf3D_Head"
          geometry={nodes.Wolf3D_Head.geometry}
          // material={materials.Wolf3D_Skin}
          morphTargetDictionary={nodes.Wolf3D_Head.morphTargetDictionary}
          morphTargetInfluences={nodes.Wolf3D_Head.morphTargetInfluences}
          scale={1}
        >
          <meshBasicMaterial map={headTexture} map-flipY={false} />
        </mesh>

        <mesh
          name="Wolf3D_Teeth"
          geometry={nodes.Wolf3D_Teeth.geometry}
          //material={materials.Wolf3D_Teeth}
          morphTargetDictionary={nodes.Wolf3D_Teeth.morphTargetDictionary}
          morphTargetInfluences={nodes.Wolf3D_Teeth.morphTargetInfluences}
          scale={1}
        >
          <meshBasicMaterial map={teethTexture} map-flipY={false} />
        </mesh>
      </group>
      <mesh
        name="Wolf3D_Head001"
        geometry={nodes.Wolf3D_Head001.geometry}
        //material={materials.Wolf3D_Skin}
        morphTargetDictionary={nodes.Wolf3D_Head001.morphTargetDictionary}
        morphTargetInfluences={nodes.Wolf3D_Head001.morphTargetInfluences}
        scale={1}
      >
        <meshBasicMaterial map={headTexture} map-flipY={false} />
      </mesh>
    </group>
  )
}

useGLTF.preload('/domneck.glb')

Finally you'll want to import this into the required page/component:

const DomMod = lazy(() => import('./DomModel'))

Remember to do a lazy import or errors will throw one of the following errors:

  • error - SyntaxError: Unexpected token '?'
  • Webpack - Fix for ModuleNotFoundError: Module not found: Error: Can't resolve '...'

And other next related errors that I encountered along the way.

Finally you'll need to set up your canvas, add react suspense to the model for loading state, and pass the mouse, position, and scale props through to the model file.

 <DomMod mouse={mouse} position={[0, 0, 0]} scale={[60, 60, 60]} />

I'm using the react-loader-spinner package to display a nice spinner for loading state.

All together the file will look something like this:


import React, { Suspense, useRef, lazy } from 'react'
import { Canvas } from '@react-three/fiber'
import { Dna } from 'react-loader-spinner'

const DomMod = lazy(() => import('./DomModel'))

function Plane({ ...props }) {
  return (
    <mesh {...props} receiveShadow>
      <planeBufferGeometry args={[500, 500, 1, 1]} />
      <shadowMaterial transparent opacity={0.2} />
    </mesh>
  )
}

export default function Dom() {
  const d = 8.25
  const mouse = useRef({ x: 0, y: 0 })
  return (
    <Suspense
      fallback={
        <div className="flex h-full w-full items-center justify-center">
          <Dna
            visible={true}
            height="80"
            width="80"
            ariaLabel="dna-loading"
            wrapperStyle={{ display: 'inline' }}
            wrapperClass="dna-wrapper"
          />
        </div>
      }
    >
      <Canvas
        style={{ position: 'unset' }}
        shadows
        dpr={[1, 1.5]}
        camera={{ position: [0, -3, 18] }}
      >
        <hemisphereLight
          skyColor={'black'}
          groundColor={0xffffff}
          intensity={0.5}
          position={[0, 50, 0]}
        />
        <directionalLight
          position={[-8, 20, 8]}
          shadow-camera-left={d * -1}
          shadow-camera-bottom={d * -1}
          shadow-camera-right={d}
          shadow-camera-top={d}
          shadow-camera-near={0.1}
          shadow-camera-far={1500}
          castShadow
        />
        <mesh position={[0, 0, -10]}>
          <circleBufferGeometry args={[15, 64]} />
          <meshBasicMaterial opacity={0.2} transparent color="#60a5fa" />
        </mesh>
        <Plane rotation={[-0.5 * Math.PI, 0, 0]} position={[0, -12, 0]} />

        <DomMod mouse={mouse} position={[0, 0, 0]} scale={[60, 60, 60]} />
      </Canvas>
    </Suspense>
  )
}

Voila! Your head will be tracking your mouse position and the component can be used anywhere in your application.

AI chat

Now lets add some personality to lifeless bobble head.

To create the interactive chat bot element to this project I decided to use an OpenAI gpt3 model. By sending a request to the OpenAI api I can easily make the chatbot AI respond to user text input. React simply conditionally displays chat messages based on the state response of the API request.

The steps are as follows:

  1. Sign up for an Open AI API key
  2. Create your textbox and response message component
  3. Overlay these components using absolute positions over the model component
  4. Create the request (I'm using the fetch API)

Here is my div that holds everything

          <div className="relative flex h-full min-h-[350px] flex-col justify-between gap-10">
            <Suspense
              fallback={
                <div className="flex h-full w-full items-center justify-center">
                  <Dna
                    visible={true}
                    height="80"
                    width="80"
                    ariaLabel="dna-loading"
                    wrapperStyle={{ display: 'inline' }}
                    wrapperClass="dna-wrapper"
                  />
                </div>
              }
            >
              <motion.div
                initial={{ scale: 0 }}
                animate={{ rotate: 0, scale: 1 }}
                transition={{
                  type: 'spring',
                  stiffness: 160,
                  damping: 14,
                  delay: 1,
                }}
                className="absolute top-0 left-0 right-0 max-h-[260px] w-fit max-w-[176px] overflow-auto rounded-[20px] rounded-br-none bg-blue-100 bg-opacity-80 p-3 dark:bg-gray-700"
              >
                {awaitingResponse ? (
                  <Discuss
                    visible={true}
                    height="30"
                    width="30"
                    ariaLabel="ai-loading"
                    wrapperStyle={{}}
                    wrapperClass="comment-wrapper"
                    colors={['#2563eb', '#60a5fa']}
                  />
                ) : (
                  <div className="text-sm font-bold">
                    {response ? response : "Hi 👋 I'm AI Dom, Ask me anything."}
                  </div>
                )}
              </motion.div>

              <Dom />
            </Suspense>
            <div className="absolute bottom-0 left-0 right-0 ml-auto mr-auto w-full">
              <div className="focus-within:ring-bluw-600 rounded-md border  border-gray-700 px-3 py-2 opacity-80 shadow-sm focus-within:border-blue-600 focus-within:ring-1">
                <input
                  type={'text'}
                  onChange={onType}
                  value={miscState?.newMessage || ''}
                  onKeyDown={handleKeyDown}
                  placeholder={'Ask something'}
                  className="block w-full rounded-lg border-0 bg-gray-100 p-3 bg-blend-screen focus:ring-0
                dark:bg-gray-800 dark:text-gray-200 dark:placeholder-gray-400 sm:text-sm"
                />
              </div>
            </div>

            <motion.div
              initial={{ scale: 0 }}
              animate={{ rotate: 0, scale: 1 }}
              transition={{
                type: 'spring',
                stiffness: 160,
                damping: 14,
              }}
              className=" absolute top-0 right-0   cursor-pointer"
            >
              <Link href={'/blog/ai-dom'} className="text-xs font-light italic text-gray-400">
                What is this?
              </Link>
            </motion.div>
          </div>

I'm using tailwind for styling but the react logic is transferable to any project.

Here is the JavaScript logic entirely on the client side, just note that this will expose your API key publicly.


  const onType = (e) => updateMiscState({ ...miscState, newMessage: e.target.value })

  const handleKeyDown = (event) => {
    console.log(event.key)
    console.log(event.key)
    if (event.key === 'Enter') {
      sendMessage({ text: miscState?.newMessage })
      setAwaitingResponse(true)
    }
  }

  const getAIAnswer = async ({ statement = miscState.newMessage, currentMessages }) => {
    console.log('AI Answers')
    const headers = { 'Content-Type': 'application/json' }
    const body = JSON.stringify({
      configuration: { OPEN_AI_ORG, OPENAI_API_KEY },
      statement: miscState.newMessage,
      model: 'text-davinci-002',
    })
    const requestData = {
      method: 'POST',
      headers,
      body,
    }
    await fetch(NavigationService.getApiEndPointURL({ endPoint: 'chat' }), requestData)
      .then((response) => response.json())
      .then((result) => {
        console.log(result)
        const response = result?.data?.choices[0].text || null
        if (response) {
          //   const aiResponse = new Message({ id: 1, message: data[0].text })
          //  currentMessages.push(aiResponse)
          setResponse(response)
          setAwaitingResponse(false)
          updateMiscState({
            ...miscState,
            isTyping: false,
            newMessage: '',
            messages: currentMessages,
          })
        }
      })

      .catch((error) => console.log('error', error))
  }

  const sendMessage = async ({ text = '' }) => {
    if (text.length <= 0) {
      console.log('text length <=0')
      updateMiscState({ ...miscState, newMessage: '', isTyping: false })
    } else {
      const currentMessages = miscState?.messages || []
      console.log(currentMessages)
      // currentMessages.push(new Message({ id: 0, message: text }))
      updateMiscState({
        ...miscState,
        messages: currentMessages,
        newMessage: '',
        isTyping: true,
        aiResponse: null,
      })

      let context = ''
      if (currentMessages) {
        context =
          Object.keys(currentMessages)
            .map(
              (message) =>
                `${ID_WISE_USER[currentMessages[message].id]}: ${currentMessages[message].message}`
            )
            .join('\n') || ''
        await getAIAnswer({ statement: context, currentMessages })
        //  updateScroll()
      }
    }
  }

If you choose to use this unsafe way of hitting the OpenAI API (not recommended) remember to import your OPENAI_API_KEY from a .env file or it won't work.

Most of the previous code is borrowed from various projects I found on github, for the next part, I'm going to re-write almost everything so I work with Netlify functions and keep my API key secure.

Adding context

I want to make sure that the chat log is saved in state so that the AI will 'remember' things you tell it. So I'll se the initial prompt in the initial state and then once the new message is received I will update the state with the new user message and AI response appended.

The initial prompt will be something like this:

prompt: `The following is a conversation with an AI assistant called Dom. The assistant is helpful, creative, clever, and very friendly. He can also direct you to his blog www.domc.dev/blog\n\nHuman: Hello, who are you?\nAI: I am an AI chatbot made by my creator - Dom. My name is AI Dom. How can I help you today?\nHuman: ${newMessage}`,

I can add additional information as I get new ideas. In the future I want to make it possible to navigate my website by asking the bot "take me to x page".

Using Netlify functions to keep API key secure

So we have the chatbot working and responding sensibly to outgoing messages. Now we have to make it so that our API key isn't exposed on the client side. It's not a massive issue, but if anyone got hold of it they could start abusing it and we might be charged.

We're going to use the Netlify custom env variables that can be set in the user dashboard settings for our project.

We'll create a new file called .env.production in the base of our project and add the following key/value pairs:

API_KEY="yourApiKey"
API_URL="https://api.dialogflow.com/v1/"

We can now access these values from our code by using process.env. So for example, in src/App.js, we can do this:

const apiKey = process.env.API_KEY // "yourApiKey" will be returned here when the code is ran on the server at build time
const apiUrl = process.env.API_URL