Fency.ai

Document analysis

Upload a PDF, wait for the memory.updated webhook, then extract typed data points with MemorySearch.

This example uploads a PDF as a FILE memory, waits until Fency has indexed it, then extracts user-defined data points. Each data point is a MemorySearch query over that one document. A StructuredChatCompletion task then merges the excerpts into typed JSON (string, string list, boolean, or number).

Prerequisites

You need a publishable key for the React SDK and a secret key for your server routes. This example uses sessions to authenticate the client.

The example uses Clerk for user authentication. That is part of the app shell and not specific to Fency.

This example also requires a PostgreSQL database to persist uploaded documents and the SEMANTIC memory type id. After each upload, Fency indexes the PDF asynchronously and sends a memory.updated webhook. Register the webhook in the dashboard and store the generated secret as DOCUMENT_ANALYSIS_WEBHOOK_SECRET. Fency cannot reach localhost, so expose the app with a tunnel when you run locally.

API routes

Each route under api/ maps to one operation in the API reference, except the webhook and document-status routes which persist Fency events in Postgres.

app/document-analysis/api/
  create-stream-session/            POST /v1/sessions (createStream)
  create-memory-search-agent-task-session/ POST /v1/sessions (createAgentTask MEMORY_SEARCH)
  create-structured-agent-task-session/ POST /v1/sessions (createAgentTask STRUCTURED_CHAT_COMPLETION)
  upload-document/                  POST /v1/memories + POST /v1/memories/{id}/uploads + S3
  fency-webhook/                    user.test + memory.updated (signature verification)
  get-document-status/              app status check (Drizzle)
  list-documents/                   app document list (Drizzle)

Upload a PDF as a FILE memory

The upload route creates a SEMANTIC memory type if needed, creates an empty FILE memory, asks Fency for a presigned S3 POST, and uploads the PDF. See Uploading files for memories.

Create a SEMANTIC memory type at api/upload-document/createFencyMemoryType.ts:

import 'server-only'

import { z } from 'zod'
import { fencyJson } from './fencyJson'

const createdMemoryTypeSchema = z.object({
  id: z.string().optional(),
  error: z.object({ message: z.string().optional() }).optional(),
})

export async function createFencyMemoryType(body: {
  name: string
  description: string
  type: 'SEMANTIC'
}) {
  return fencyJson('/v1/memory-types', createdMemoryTypeSchema, {
    method: 'POST',
    body: JSON.stringify(body),
  })
}

Create the empty FILE memory at api/upload-document/createFencyFileMemory.ts. The response field is id:

import 'server-only'

import { z } from 'zod'
import { fencyJson } from './fencyJson'

const createdMemorySchema = z.object({
  id: z.string(),
})

export async function createFencyFileMemory(body: {
  memoryTypeId: string
  title: string
  metadata: Record<string, string>
}) {
  const { ok, status, data } = await fencyJson(
    '/v1/memories',
    createdMemorySchema,
    {
      method: 'POST',
      body: JSON.stringify({
        memoryTypeId: body.memoryTypeId,
        sourceType: 'FILE',
        title: body.title,
        metadata: body.metadata,
      }),
    },
  )

  if (!ok) {
    throw new Error(`Failed to create file memory (${status}).`)
  }

  return data
}

Get the presigned upload at api/upload-document/createFencyMemoryUpload.ts:

import 'server-only'

import { z } from 'zod'
import { fencyJson } from './fencyJson'

const awsS3PostRequestSchema = z.object({
  amzDate: z.string(),
  amzSignature: z.string(),
  amzAlgorithm: z.string(),
  amzCredential: z.string(),
  policy: z.string(),
  key: z.string(),
  uploadUrl: z.string(),
  sessionToken: z.string(),
})

const createdUploadSchema = z.object({
  awsS3PostRequest: awsS3PostRequestSchema,
})

export async function createFencyMemoryUpload(
  memoryId: string,
  body: {
    fileName: string
    fileSize: number
    mimeType: string
  },
) {
  const { ok, status, data } = await fencyJson(
    `/v1/memories/${memoryId}/uploads`,
    createdUploadSchema,
    {
      method: 'POST',
      body: JSON.stringify(body),
    },
  )

  if (!ok) {
    throw new Error(`Failed to create memory upload (${status}).`)
  }

  return data.awsS3PostRequest
}

Post the file to S3 at api/upload-document/uploadFileToS3.ts. The file field must be last:

import 'server-only'

export async function uploadFileToS3(
  awsS3PostRequest: {
    key: string
    policy: string
    amzAlgorithm: string
    amzCredential: string
    amzDate: string
    amzSignature: string
    sessionToken: string
    uploadUrl: string
  },
  file: File,
) {
  const formData = new FormData()
  formData.append('key', awsS3PostRequest.key)
  formData.append('policy', awsS3PostRequest.policy)
  formData.append('x-amz-algorithm', awsS3PostRequest.amzAlgorithm)
  formData.append('x-amz-credential', awsS3PostRequest.amzCredential)
  formData.append('x-amz-date', awsS3PostRequest.amzDate)
  formData.append('x-amz-signature', awsS3PostRequest.amzSignature)
  formData.append('x-amz-security-token', awsS3PostRequest.sessionToken)
  formData.append('file', file)

  const response = await fetch(awsS3PostRequest.uploadUrl, {
    method: 'POST',
    body: formData,
  })

  if (!response.ok) {
    throw new Error(`Failed to upload file to S3 (${response.status}).`)
  }
}

The upload route at api/upload-document/route.ts wires the three calls together and stores the document as EMPTY until the webhook arrives:

import { NextResponse } from 'next/server'
import { getAuthorizedUserId } from '../../../auth'
import {
  DOCUMENT_TAG_KEY,
  DOCUMENT_TAG_VALUE,
} from '../../documentAnalysisConstants'
import { documentRepository } from '../../db/documentRepository'
import { createFencyFileMemory } from './createFencyFileMemory'
import { createFencyMemoryUpload } from './createFencyMemoryUpload'
import { ensureDocumentMemoryType } from './ensureDocumentMemoryType'
import { uploadFileToS3 } from './uploadFileToS3'

export async function POST(request: Request) {
  const userId = await getAuthorizedUserId()
  if (!userId) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
  }

  const formData = await request.formData()
  const file = formData.get('file')
  if (!(file instanceof File)) {
    throw new Error('Expected a file field named file.')
  }
  if (file.type !== 'application/pdf') {
    return NextResponse.json(
      { error: 'Upload a PDF file.' },
      { status: 400 },
    )
  }

  const memoryTypeId = await ensureDocumentMemoryType()
  const memory = await createFencyFileMemory({
    memoryTypeId,
    title: file.name,
    metadata: {
      userId,
      [DOCUMENT_TAG_KEY]: DOCUMENT_TAG_VALUE,
    },
  })
  const awsS3PostRequest = await createFencyMemoryUpload(memory.id, {
    fileName: file.name,
    fileSize: file.size,
    mimeType: file.type,
  })
  await uploadFileToS3(awsS3PostRequest, file)

  const document = await documentRepository.insert({
    userId,
    fencyMemoryId: memory.id,
    fileName: file.name,
    contentStatus: 'EMPTY',
  })

  return NextResponse.json({
    id: document.id,
    fileName: document.fileName,
    contentStatus: document.contentStatus,
    contentParts: document.contentParts,
  })
}

Webhook: wait for SYNCHRONIZED

There is no memory.synchronized event. After Fency indexes the file, it sends memory.updated. The entity.contentStatus field is EMPTY, SYNCHRONIZING, SYNCHRONIZED, or SYNCHRONIZATION_ERROR.

Verify the x-fency-signature header as described in Webhooks. Read the raw body first. The helper lives at api/fency-webhook/verifyFencySignature.ts:

import { createHmac, timingSafeEqual } from 'node:crypto'

export function verifyFencySignature(payload: string, header: string | null) {
  const secret = process.env.DOCUMENT_ANALYSIS_WEBHOOK_SECRET
  if (!secret) {
    throw new Error('DOCUMENT_ANALYSIS_WEBHOOK_SECRET is not defined.')
  }
  if (!header) {
    throw new Error('Missing x-fency-signature header.')
  }

  const expected = Buffer.from(
    `sha256=${createHmac('sha256', secret).update(payload).digest('hex')}`,
    'utf8',
  )
  const received = Buffer.from(header, 'utf8')
  if (
    expected.length !== received.length ||
    !timingSafeEqual(expected, received)
  ) {
    throw new Error('Signature invalid')
  }
}

The webhook route at api/fency-webhook/route.ts is public. A dashboard test send is a user.test event (message: "Hello world!") with no entity; acknowledge it and return 200. On memory.updated it writes contentStatus for the matching memory. Unknown memory ids are ignored:

import { NextResponse } from 'next/server'
import { z } from 'zod'
import { contentStatusSchema } from '../../analyzedDocument'
import { documentRepository } from '../../db/documentRepository'
import { verifyFencySignature } from './verifyFencySignature'

const eventTypeSchema = z.object({
  type: z.string(),
})

const userTestEventSchema = z.object({
  type: z.literal('user.test'),
  message: z.string(),
})

const memoryUpdatedEventSchema = z.object({
  type: z.literal('memory.updated'),
  entity: z.object({
    id: z.string(),
    contentStatus: contentStatusSchema,
    contentParts: z.number().nullable().optional(),
  }),
})

export async function POST(request: Request) {
  const raw = await request.text()
  verifyFencySignature(raw, request.headers.get('x-fency-signature'))

  const body: unknown = JSON.parse(raw)
  const { type } = eventTypeSchema.parse(body)

  if (type === 'user.test') {
    const event = userTestEventSchema.parse(body)
    return NextResponse.json({ ok: true, message: event.message })
  }

  if (type !== 'memory.updated') {
    return NextResponse.json({ ok: true })
  }

  const event = memoryUpdatedEventSchema.parse(body)
  await documentRepository.updateContentStatus(
    event.entity.id,
    event.entity.contentStatus,
    event.entity.contentParts ?? null,
  )

  return NextResponse.json({ ok: true })
}

The browser polls api/get-document-status until contentStatus is SYNCHRONIZED. Analysis is blocked until then.

Server session routes

The example defines three session routes. Each route is standalone so you can read a single file and see the full POST /v1/sessions call.

Create the stream session route at api/create-stream-session/route.ts:

import { NextResponse } from 'next/server'
import { getAuthorizedUserId } from '../../../auth'

export async function POST() {
  const userId = await getAuthorizedUserId()
  if (!userId) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
  }

  const secretKey = process.env.FENCY_SECRET_KEY
  if (!secretKey) {
    throw new Error('FENCY_SECRET_KEY is not defined.')
  }

  const response = await fetch('https://api.fency.ai/v1/sessions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${secretKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ createStream: {} }),
  })
  const data = await response.json()
  return NextResponse.json(data, { status: response.status })
}

Create the MemorySearch session at api/create-memory-search-agent-task-session/route.ts. Guard rails pin the task to the one uploaded memory:

import { NextResponse } from 'next/server'
import { z } from 'zod'
import { getAuthorizedUserId } from '../../../auth'
import { DOCUMENT_MEMORY_TYPE_NAME } from '../../documentAnalysisConstants'
import { documentRepository } from '../../db/documentRepository'
import { memoryTypeRepository } from '../../db/memoryTypeRepository'
import { buildDocumentGuardRails } from './buildDocumentGuardRails'

const bodySchema = z.object({
  documentId: z.string(),
})

export async function POST(request: Request) {
  const userId = await getAuthorizedUserId()
  if (!userId) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
  }

  const { documentId } = bodySchema.parse(await request.json())
  const document = await documentRepository.findById(documentId)
  if (!document) {
    return NextResponse.json({ error: 'Document not found' }, { status: 404 })
  }
  if (document.userId !== userId) {
    return NextResponse.json(
      { error: 'Document does not belong to this user' },
      { status: 403 },
    )
  }
  if (document.contentStatus !== 'SYNCHRONIZED') {
    return NextResponse.json(
      { error: 'Document is not synchronized yet.' },
      { status: 409 },
    )
  }

  const memoryType = await memoryTypeRepository.findByName(
    DOCUMENT_MEMORY_TYPE_NAME,
  )
  if (!memoryType) {
    throw new Error('AnalyzedDocument memory type is not set up.')
  }

  const secretKey = process.env.FENCY_SECRET_KEY
  if (!secretKey) {
    throw new Error('FENCY_SECRET_KEY is not defined.')
  }

  const response = await fetch('https://api.fency.ai/v1/sessions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${secretKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      createAgentTask: {
        taskType: 'MEMORY_SEARCH',
        metadata: { userId },
        guardRails: buildDocumentGuardRails(
          memoryType.fencyMemoryTypeId,
          document.fencyMemoryId,
        ),
      },
    }),
  })
  const data = await response.json()
  return NextResponse.json(data, { status: response.status })
}

The guard rails helper at api/create-memory-search-agent-task-session/buildDocumentGuardRails.ts:

export function buildDocumentGuardRails(
  memoryTypeId: string,
  memoryId: string,
) {
  return {
    memoryTypes: [
      {
        memoryTypeId,
        memoryIds: [memoryId],
      },
    ],
  }
}

Create the structured extraction session at api/create-structured-agent-task-session/route.ts:

import { NextResponse } from 'next/server'
import { getAuthorizedUserId } from '../../../auth'

export async function POST() {
  const userId = await getAuthorizedUserId()
  if (!userId) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
  }

  const secretKey = process.env.FENCY_SECRET_KEY
  if (!secretKey) {
    throw new Error('FENCY_SECRET_KEY is not defined.')
  }

  const response = await fetch('https://api.fency.ai/v1/sessions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${secretKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      createAgentTask: {
        taskType: 'STRUCTURED_CHAT_COMPLETION',
        metadata: { userId },
      },
    }),
  })
  const data = await response.json()
  return NextResponse.json(data, { status: response.status })
}

Data points and JSON schema

MEMORY_SEARCH returns ranked chunks, not typed JSON. The UI lets the user add rows of label, type, and description. Those rows become a Zod object, then a JSON Schema string for StructuredChatCompletion.

The data point types live in dataPoint.ts:

export const dataPointTypes = [
  'string',
  'stringList',
  'boolean',
  'number',
] as const

export type DataPointType = (typeof dataPointTypes)[number]

export type DataPoint = {
  key: string
  label: string
  type: DataPointType
  description: string
}

export const defaultDataPoints: DataPoint[] = [
  {
    key: 'title',
    label: 'Title',
    type: 'string',
    description: 'The document title or heading',
  },
  {
    key: 'parties',
    label: 'Parties',
    type: 'stringList',
    description: 'Named people or organizations in the document',
  },
  {
    key: 'isSigned',
    label: 'Is signed',
    type: 'boolean',
    description: 'Whether the document appears signed',
  },
  {
    key: 'amount',
    label: 'Amount',
    type: 'number',
    description: 'A primary monetary amount if present',
  },
]

Build the schema at runtime in dataPointJsonSchema.ts:

import { z } from 'zod'
import type { DataPoint } from './dataPoint'

function fieldForDataPoint(dataPoint: DataPoint) {
  const description = dataPoint.description || dataPoint.label
  switch (dataPoint.type) {
    case 'string':
      return z.string().nullish().describe(description)
    case 'stringList':
      return z.array(z.string()).nullish().describe(description)
    case 'boolean':
      return z.boolean().nullish().describe(description)
    case 'number':
      return z.number().nullish().describe(description)
  }
}

export function dataPointZodSchema(dataPoints: DataPoint[]) {
  const shape: Record<string, ReturnType<typeof fieldForDataPoint>> = {}
  for (const dataPoint of dataPoints) {
    shape[dataPoint.key] = fieldForDataPoint(dataPoint)
  }
  return z.object(shape)
}

export function dataPointJsonSchema(dataPoints: DataPoint[]) {
  return JSON.stringify(z.toJSONSchema(dataPointZodSchema(dataPoints)))
}

Client provider and analysis pipeline

Both session routes respond with a client token. Parse that response with a Zod schema in sessionClientTokenSchema.ts so a bad shape throws:

import { z } from 'zod'

export const sessionClientTokenSchema = z.object({
  clientToken: z.string(),
})

Load the Fency client and wrap your component tree with FencyProvider in page.tsx:

'use client'

import { loadFency } from '@fencyai/js'
import { FencyProvider } from '@fencyai/react'
import { DocumentAnalyzer } from './components/DocumentAnalyzer'
import { sessionClientTokenSchema } from './sessionClientTokenSchema'

const fency = loadFency({
  publishableKey: process.env.NEXT_PUBLIC_FENCY_PUBLISHABLE_KEY!,
})

async function fetchCreateStreamClientToken() {
  const res = await fetch('/document-analysis/api/create-stream-session', {
    method: 'POST',
  })
  if (!res.ok) {
    throw new Error('Failed to create stream session')
  }
  const { clientToken } = sessionClientTokenSchema.parse(await res.json())
  return { clientToken }
}

export default function DocumentAnalysisPage() {
  return (
    <FencyProvider
      fency={fency}
      fetchCreateStreamClientToken={fetchCreateStreamClientToken}
    >
      <DocumentAnalyzer />
    </FencyProvider>
  )
}

components/DocumentUploader.tsx uses an Uppy drop field so you can drop or pick a PDF. Uppy posts file to api/upload-document.

hooks/useDocumentAnalysis.ts runs one MemorySearch per data point, then one StructuredChatCompletion. Each MemorySearch session is scoped to the selected document. The analysis list is one row per data point; click a row to open AgentTaskProgress for that search in components/DataPointSearchProgressModal.tsx. While StructuredChatCompletion runs, AgentTaskProgress also renders under the Analyze button.

const searchResult = await createAgentTask(
  {
    type: 'MemorySearch',
    query,
    model: ANALYSIS_MODEL,
    language: 'en',
    chunkLimit: 5,
    contextExpansion: { before: 1, after: 1 },
  },
  {
    fetchCreateAgentTaskClientToken: () =>
      fetchCreateMemorySearchClientToken(documentId),
    onTaskRegistered: (task) => {
      setDataPointSearches((prev) => [
        ...prev,
        {
          taskKey: task.taskKey,
          label: dataPoint.label,
          query,
        },
      ])
    },
  },
)
const structResult = await createAgentTask(
  {
    type: 'StructuredChatCompletion',
    messages: [
      {
        role: 'SYSTEM',
        content:
          'Merge facts from the labelled document search excerpts into the JSON schema. Use only information clearly supported by the excerpts. Leave a field null when it cannot be inferred.',
      },
      {
        role: 'USER',
        content: aggregated.join('\n\n---\n\n'),
      },
    ],
    jsonSchema,
    model: ANALYSIS_MODEL,
    temperature: 0.1,
  },
  {
    fetchCreateAgentTaskClientToken: fetchCreateStructuredClientToken,
    onTaskRegistered: (task) => {
      setStructuringTaskKey(task.taskKey)
    },
  },
)

The completed MemorySearch event includes items with matchingChunk and surrounding chunks (page numbers and MATCH / CONTEXT_BEFORE / CONTEXT_AFTER). Those excerpts become the user message for structured extraction. Parse event.response.jsonResponse with the same Zod schema.

Running locally

Clone the repository and install dependencies:

git clone https://github.com/fencyai/fency-react-examples.git
cd fency-react-examples
npm install

Copy .env.example to .env.local and add your Fency keys:

cp .env.example .env.local

Set the keys in .env.local:

FENCY_SECRET_KEY=sk_...
NEXT_PUBLIC_FENCY_PUBLISHABLE_KEY=pk_...
DATABASE_URL=postgresql://fency:fency@127.0.0.1:5433/fency_react_examples
DOCUMENT_ANALYSIS_WEBHOOK_SECRET=whsec_...

Start Postgres and apply migrations:

docker compose up -d
npm run db:migrate

Initialize Clerk authentication (provisions a development app):

npx -y clerk@latest init --keyless -y

Start the development server and a public tunnel so Fency can deliver webhooks:

npm run dev
cloudflared tunnel --url http://localhost:3000

Create a webhook in the Fency dashboard pointing at https://<tunnel-host>/document-analysis/api/fency-webhook. Copy the generated secret into DOCUMENT_ANALYSIS_WEBHOOK_SECRET.

Open http://localhost:3000/document-analysis. Sign up from the header, upload a PDF, wait until the document is ready, then run analysis.

Explore memories

Memory-grounded chat over a catalog with guard rails and conversations.

On this page

PrerequisitesAPI routesUpload a PDF as a FILE memoryWebhook: wait for SYNCHRONIZEDServer session routesData points and JSON schemaClient provider and analysis pipelineRunning locally