Caching OpenAI API Responses with Upstash Redis
If you used the OpenAI API, you might have noticed that it’s pretty slow and sometimes doesn’t even respond. Especially the GPT-4 model is prone to high latency responses. Also, you pay for every answer you get. Those are all reasons to avoid serving responses directly.
You can store your responses in Upstash Redis to get around these issues. If you’re serving the same response to many clients, this can save you quite some money, and with global deployments, you can be sure users get the info as quickly as possible.
This article will guide you through caching OpenAI API responses to an Upstash Redis database.
Features
We will build a web app showing visitors a list of history jokes.
- Query the OpenAI API daily to get a history joke related to the current date
- Cache the joke for future delivery
- Display all jokes
Technology
We will use the following technologies to build the app:
- Node.js as HTTP server
- The OpenAI API to generate history jokes
- Upstash QStash to trigger a new joke request from the HTTP server to the OpenAI API
- Upstash Redis to store the joke for later use
Prerequisites
- An OpenAI account with API access to get access to an AI
- An Upstash account to send the request to the OpenAI API and cache the AI responses in Upstash Redis
- A Node.js installation to call the OpenAI and Upstash APIs
Implementation
Let’s start by creating the HTTP server to fetch the jokes, store them in Upstash Redis, and display them to visitors.
Setting Up the Project
First, we create a new Node.js project and install the dependencies.
$ mkdir history-jokes & cd history-jokes
$ npm init -y
$ npm i dotenv express axios @upstash/redis @upstash/qstash
$ mkdir history-jokes & cd history-jokes
$ npm init -y
$ npm i dotenv express axios @upstash/redis @upstash/qstash
Implementing the Server
To implement the server, create an index.js
file and copy the following code into it:
require("dotenv").config();
const axios = require("axios");
const express = require("express");
const { Redis } = require("@upstash/redis");
const { Receiver } = require("@upstash/qstash");
const redisClient = new Redis({
url: process.env.UPSTASH_REDIS_URL,
token: process.env.UPSTASH_REDIS_TOKEN,
});
const qstashReceiver = new Receiver({
currentSigningKey: process.env.UPSTASH_QSTASH_CURRENT_SIGNING_KEY,
nextSigningKey: process.env.UPSTASH_QSTASH_CURRENT_NEXT_KEY,
});
const openaiApiClient = axios.create({
baseURL: "https://api.openai.com/v1",
headers: {
Authorization: "Bearer " + process.env.OPENAI_TOKEN,
"Content-Type": "application/json",
},
});
const server = express();
server.use("/generate", async (request, response, next) => {
// return next()
let validRequest = false;
try {
validRequest = await qstashReceiver.verify({
signature: request.headers["upstash-signature"],
body: "",
});
} catch (e) {}
if (!validRequest) return response.status(403).end("Forbidden");
return next();
});
server.post("/generate", async (_request, response) => {
const today = new Date();
const month = today.toLocaleString("default", { month: "long" });
let day = today.toLocaleString("default", { day: "numeric" });
day = day == 1 ? "1st" : day == 2 ? "2nd" : day == "3" ? "3rd" : day + "th";
const { data } = await openaiApiClient.post("/chat/completions", {
model: "gpt-3.5-turbo",
messages: [
{
role: "system",
content: "You are a comedian that tells short history jokes.",
},
{
role: "user",
content: `Please tell me a joke for ${month} the ${day}.`,
},
],
});
const joke = JSON.stringify({
date: month + " the " + day,
text: data.choices[0].message.content,
});
await redisClient.lpush("jokes", joke);
response.end();
});
server.get("/", async (_request, response) => {
let html = "<h1>History Jokes for Every Day</h1>";
const jokes = await redisClient.lrange("jokes", 0, -1);
html +=
"<ul>" +
jokes
.map(({ date, text }) => `<li><b>${date}</b><br><pre>${text}</pre></li>`)
.join("") +
"</ul>";
response.setHeader("Content-Type", "text/html");
response.end(html);
});
server.listen(3000);
require("dotenv").config();
const axios = require("axios");
const express = require("express");
const { Redis } = require("@upstash/redis");
const { Receiver } = require("@upstash/qstash");
const redisClient = new Redis({
url: process.env.UPSTASH_REDIS_URL,
token: process.env.UPSTASH_REDIS_TOKEN,
});
const qstashReceiver = new Receiver({
currentSigningKey: process.env.UPSTASH_QSTASH_CURRENT_SIGNING_KEY,
nextSigningKey: process.env.UPSTASH_QSTASH_CURRENT_NEXT_KEY,
});
const openaiApiClient = axios.create({
baseURL: "https://api.openai.com/v1",
headers: {
Authorization: "Bearer " + process.env.OPENAI_TOKEN,
"Content-Type": "application/json",
},
});
const server = express();
server.use("/generate", async (request, response, next) => {
// return next()
let validRequest = false;
try {
validRequest = await qstashReceiver.verify({
signature: request.headers["upstash-signature"],
body: "",
});
} catch (e) {}
if (!validRequest) return response.status(403).end("Forbidden");
return next();
});
server.post("/generate", async (_request, response) => {
const today = new Date();
const month = today.toLocaleString("default", { month: "long" });
let day = today.toLocaleString("default", { day: "numeric" });
day = day == 1 ? "1st" : day == 2 ? "2nd" : day == "3" ? "3rd" : day + "th";
const { data } = await openaiApiClient.post("/chat/completions", {
model: "gpt-3.5-turbo",
messages: [
{
role: "system",
content: "You are a comedian that tells short history jokes.",
},
{
role: "user",
content: `Please tell me a joke for ${month} the ${day}.`,
},
],
});
const joke = JSON.stringify({
date: month + " the " + day,
text: data.choices[0].message.content,
});
await redisClient.lpush("jokes", joke);
response.end();
});
server.get("/", async (_request, response) => {
let html = "<h1>History Jokes for Every Day</h1>";
const jokes = await redisClient.lrange("jokes", 0, -1);
html +=
"<ul>" +
jokes
.map(({ date, text }) => `<li><b>${date}</b><br><pre>${text}</pre></li>`)
.join("") +
"</ul>";
response.setHeader("Content-Type", "text/html");
response.end(html);
});
server.listen(3000);
Let’s go through the important parts of this code.
We use the redisClient
to save our responses into Upstash Redis and load them for display.
The qstashReceiver
is responsible for verifying the daily calls from Upstash QStash; this way, we ensure that only QStash calls our /generate
endpoint.
We use Axios to create a client for the OpenAI API, so we don’t have to pass the baseUrl
and Authentication
header whenever we call the API.
Next, we create an Express middleware that uses the qstashReceiver
to verify all requests to the /generate
endpoint. Since we use QStash only to trigger the endpoint and don’t pass any values, we can use an empty string as the body
.
Note: The middleware allows all requests if you un-comment the line with "
// return next()
". You can use this to test the/generate
endpoint onlocalhost
.
The /generate
endpoint listens for POST
requests and creates and stores the actual jokes. It calculates the current month
and day
and uses it to create two prompts for the AI. One tells the AI to behave like a comedian, and the second asks to tell a history joke about the current month and day.
We call the OpenAI API with the prompts and save the result into a LIST
in Upstash Redis.
The /
endpoint listens for GET
requests and displays the saved jokes. It loads the LIST
from Upstash Redis, formats it as HTML list elements, and sends it to the client.
Deployment
Now that the server is set up, we must create the cloud resources and fill in the API credentials to access them.
Creating a Credential File
To store the credentials, create a .env
file with the following content:
OPENAI_TOKEN=""
UPSTASH_REDIS_REST_URL=""
UPSTASH_REDIS_REST_TOKEN=""
UPSTASH_QSTASH_CURRENT_SIGNING_KEY=""
UPSTASH_QSTASH_NEXT_SIGNING_KEY=""
OPENAI_TOKEN=""
UPSTASH_REDIS_REST_URL=""
UPSTASH_REDIS_REST_TOKEN=""
UPSTASH_QSTASH_CURRENT_SIGNING_KEY=""
UPSTASH_QSTASH_NEXT_SIGNING_KEY=""
In the next steps, we will fill each of these empty strings.
Creating an OpenAI API Token
Let’s start with the OpenAI API because we only need a key for the existing API; we don’t have to deploy anything.
Go to the OpenAI web console and click the “create new secret key” button. Give the key a name, click “Create secret key”, copy the new key, and paste it into the .env file as the value of OPENAI_TOKEN
.
Deploying a Recurring Request with QStash
To tell QStash that it should send a request to the /generate
endpoint once a day, you use the “Request Builder”. You find it in the Upstash console. Figure 1 shows the configuration details.
Figure 1: QStash Request Builder
Replace the <HOSTNAME>
with the domain that hosts your server. You need a publicly accessible hostname for QStash to work.
You find the singing keys QStash uses in the Upstash console in the “Request Builder” section. Click on the grey “Singing Keys” drop-down, copy each key and paste them in the right places of the .env
file.
Deploying an Upstash Redis Database
To create an Upstash Redis database, go to the Upstash console and click the “Create Database” button.
This will prompt you with a dialog to fill in the database configuration. Figure 2 shows the values you should use.
Figure 2: Upstash Redis Configuration
After the creation is completed, you can scroll down to the “REST API” section, which has two buttons, “UPSTASH_REDIS_REST_URL” and “UPSTASH_REDIS_REST_TOKEN”. Click each of these buttons to copy the corresponding credentials and paste them into the correct place in the .env
file.
Testing the Website
To test the website on your local machine, deactivate the QStash validation middleware by un-commenting the “// return next()
” line in the index.js
file.
Run the server with the following command:
$ node .
$ node .
If you open the / route in the browser, you should only see “History Jokes for Every Day” since no jokes have been generated.
You must send a POST
request to the /generate
endpoint to generate a joke. You can do so with this command:
$ curl -X POST http://localhost/generate
$ curl -X POST http://localhost/generate
If you refresh the browser page, you should see a joke like in Figure 3.
Figure 3: Joke website
This joke is now cached in Upstash Redis, so no OpenAI API requests will happen when someone visits the website. This gives you sub-second response times while saving you quite some money.
Summary
AI APIs are versatile tools that will transform the Internet as we know it, but right now, they’re slow and expensive. Caching is vital when using these APIs, and Upstash Redis is the easiest way to add it to your stack with just a few clicks.