AI Generation
Introduction
This example demonstrates advanced AI data processing using Upstash Workflow. The following example workflow downloads a large dataset, processes it in chunks using OpenAI’s GPT-4 model, aggregates the results and generates a report.
Use Case
Our workflow will:
- Receive a request to process a dataset
- Download the dataset from a remote source
- Process the data in chunks using OpenAI
- Aggregate results
- Generate and send a final report
Code Example
Code Breakdown
1. Preparing our data
We start by retrieving the dataset URL and then downloading the dataset:
Note that we use context.call
for the download, a way to make HTTP requests that run for much longer than your serverless execution limit would normally allow.
2. Processing our data
We split the dataset into chunks and process each one using OpenAI’s GPT-4 model:
3. Aggregating our data
After processing our data in smaller chunks to avoid any function timeouts, we aggregate results every 10 chunks:
4. Sending a report
Finally, we generate a report based on the aggregated results and send it to the user:
Key Features
-
Non-blocking HTTP Calls: We use
context.call
for API requests so they don’t consume the endpoint’s execution time (great for optimizing serverless cost). -
Long-running tasks: The dataset download can take up to 2 hours, though is realistically limited by function memory.
Was this page helpful?