Have you ever built an API endpoint that has to create an Excel file?

At first the idea seems simple. Fetch some records from MongoDB turn them into Excel format and send the file back to the client.

What happens when your database contains thousands or even hundreds of thousands of records?

Creating an Excel file is not always an operation. It can involve database queries, data transformation, memory consumption, file generation and uploading the file to cloud storage.

If we do all these operations inside a HTTP request we can easily create performance problems.

The API might take long to respond. Requests can time out. Users may experience responses and the application can struggle when many users request large exports at the same time.

I faced a challenge while working on a NestJS application. Of doing everything inside the HTTP request I moved the Excel generation process into a background job using BullMQ and Redis.

In this article I will explain how I approached this problem and how I structured the implementation and why background workers are useful for this king of task.

1. Understanding the Problem

Let us imagine that we have an application that manages alerts or tickets.

Users can filter alerts, select specific records and export the result into an Excel file.

For Example: A user might request an export containing 50,000 alert records.

A straight forward implementation could look something like this:

@Controller('alerts')
export class AlertsController {
 @Post('/export')
 async exportAlerts(@Body() filters: AlertExportDto) {
   const alerts = await this.alertService.findAlerts(filters);
   const excel = await this.generateExcel(alerts);
   return excel;
 }
}

There is nothing wrong with this approach for small datasets.

However as the amount of data grows the export process becomes more expensive.

Think about everything that happens during this request:

1. Fetching records from MongoDB.
2. Transforming database documents into Excel rows.
3. Generating the Excel file.
4. Saving the file.
5. Returning the generated file to the client.

The HTTP request stays open until all these steps are done.

This is not ideal when the process could take seconds or even minutes.

So I decided to separate the file generation process from the HTTP request.

2. The Approach - Background Job Processing

My solution was to use BullMQ with Redis.

The main idea is simple:

Instead of generating the Excel file immediately the API creates a job and puts it to a queue. A background worker picks up that job. Does the actual file generation. The API can respond immediately without waiting for the export to finish.

The architecture looks like this:

The important part here is that the API and the file generation process have different responsibilities.

The API handles the request and schedules the job. The background worker handles the heavy work.

3. Step One - Accept the Export Request

I started by keeping the controller as simple as possible.

The controller receives the export request and delegates the work to the service.

@Controller('alerts')
export class AlertsController {
 
  import { AlertsService } from './alerts.service';
  import {  AlertExportDto } from './dto';
  @Inject()
  private readonly alertsService: AlertsService;
  @Post('/:id/export')
  generate(
    @Param('id', ParseMongoIdPipe) monitorId: string,
    @Body() alertExportDto: AlertExportDto,
    @User() user: CurrentUser,
    @Res() res: Response,
  ) {
    return this.alertsService.generate(
      monitorId,
      alertExportDto,
      user,
      res,
    );
  }
}

Notice that the controller doesn't know anything about Excel generation.

It doesn't fetch thousands of records or process the file. It simply passes the request to the service. I prefer this approach because it keeps the controller focused on handling HTTP requests rather than implementing business logic.

4. Step Two - Check Whether the File Already Exists

Before creating an export job and I check whether the same export has already been generated.

This is useful because users might request the same report multiple times with identical filters.

Instead of generating the same file again I can reuse an existing completed export.

const alreadyProcessed =
  await this.alertExportHistoryModel.findOne({
    monitor: new Types.ObjectId(monitorId),
    filters: alertExportDto,
    status: ExportStatus.COMPLETED,
  });

If a valid generated file already exists in Azure Blob Storage and has not expired, I download and return that file.

I also maintain information about who downloaded the file and how many times it was downloaded.

This avoids unnecessary file generation and reduces repeated processing.

5. Step Three: Create an Export History Record

If there is not an existing completed export I create a new export history record.

const exportHistory =
  await this.alertExportHistoryModel.create({
    monitor: monitorId,
    status: ExportStatus.PROCESSING,
    name: alertExportDto.name,
    users: [{email: user.email, isItNew: true, accessGrantedAt: new Date(), downloadCount: 0}],
    fileType: alertExportDto.type,
    filters: alertQuery,
  });

Why do I create this record before starting the job?

Because I need to keep track of the export as it goes through its lifecycle.

For example:

• PROCESSING: The export request has been accepted and is being processed.
• COMPLETED: The Excel file has been generated and uploaded.
• FAILED: Something went wrong during processing.

The export history record also gives me a reference ID that I can pass to the background worker. This is especially helpful when the client wants to check the export status later.

6. Step Four : Add the Job to BullMQ

This is where the background processing starts. Instead of creating the Excel file inside the API request, I add a job to the queue.

this.alertQueue
  .add(ProcessName.BUILD_FILE, {
    monitorId,
    query: searchedQuery,
    userEmail: user.email,
    option: alertExportDto,
    exportHistoryId: exportHistory._id,
  })
  .catch((error) => this.logger.error('Error adding job to alertQueue:',error));
res.status(200).end('processing the alert export');

BullMQ uses Redis to manage jobs.

Once the job is added, a worker can pick it up and process it independently of the original HTTP request.

The client doesn't need to wait for the Excel file to be generated.

It can receive an immediate response indicating that the export is being processed.

This is one of the biggest advantages of background job processing.

The API request and the actual business operation no longer need to happen in the same request lifecycle.

7. Step Five: Process the Job Using a NestJS Worker

I separated the queue processing logic into two classes:

• AlertExportProcessor
• AlertExportWorker

The processor is responsible for receiving jobs from BullMQ and handling job lifecycle events.

@Processor(QueueName.ALERT_EXPORT)
export class AlertExportProcessor extends WorkerHost {
  @Inject()
  private alertExportWorker: AlertExportWorker;
  async process(job: Job<AlertExportQueue>): Promise<void> {
    await this.alertExportWorker.generate(
      job.data,
      job.data.exportHistoryId,
    );
  }
}

The @Processor() decorator registers the class as a BullMQ processor for the specific queue. When a job is ready, BullMQ runs the process() method.

I keep the processor simple. Its main responsibility is to delegate the actual work to AlertExportWorker.

This makes the implementation easier to maintain because queue lifecycle management and business logic are separated.

8. Step Six: Generate the Excel File

The actual Excel generation happens inside AlertExportWorker. First I retrieve the export history and monitor information. Then I build the MongoDB query depending on the export mode. so in my implementation, there are two supported modes:

Export all matching records

case Mode.ALL: {
  _query = {
    $and: [{ monitor: _monitor._id }, data.query],
  };
  break;
}

Export selected records

case Mode.SELECTION: {
  _query = {
    $and: [
      { monitor: _monitor._id }, { _id: { $in: data.option.alertIds } },
    ],
  };
  break;
}

After building the query, I initialize the Excel helper.

const _excel = new Excel()
  .setType(data.option.type)
  .setHeaders(data.option.columns || this.defaultColumns)
  .setPageRowLimit(
    parseInt(this.alertConfigService.perPageRowCount),
  )
  .build()
  .setTitle(_preset.name)
  .setAuthor(data.userEmail)
  .setSkipColumns(data.option.headers);

One important decision I made was to use a MongoDB cursor instead of loading all the records into memory at once.

const cursor = this.alertService.findAlerts(
  _query,
  data.option.columns || this.defaultColumns,
  data.option.max,
);

Then I iterate through the cursor:

for await (const doc of cursor) {
  const row = await this.finalizeAlertProperties(
    _.omit(doc.toJSON(), ['_id', 'monitor']),
    data.option.columns || this.defaultColumns,
  );
  _excel.appendData([row]);
}
await cursor.close();

Why use a cursor? Imagine loading 100,000 MongoDB documents into an array before generating the Excel file.

That could consume an amount of memory.

With a cursor I can iterate through the matching records progressively of materializing the entire result set in application memory.

I also transform the document properties before adding them to Excel.

For example I format date fields convert user information into values and truncate long descriptions to avoid exceeding Excels cell character limit.

This makes the exported data more useful to the person downloading the report.

9. Step Seven: Upload the Generated File to Azure Blob Storage

Once the Excel generation process is complete I upload the file to Azure Blob Storage.

_excel.setName(data.option.name);
_excel.setPath(this.alertConfigService.tempFilePath);
_excel.write();
const _fileName = _excel.getFileName();
await (
  await this.blobStorageService.init(
    this.azureConfigService.container,
    _fileName,
    this.alertConfigService.tempFilePath,
  )
).upload();

I do not need to send the generated Excel file through the original HTTP request. Instead I store the generated Excel file in the Azure Blob Storage and save its filename in the export history collection.

I also set and expiration date.

_exportHistory.set('blobName', _fileName);
_exportHistory.set(
  'expiredAt',
  new Date().getTime() + 7 * 24 * 60 * 60 * 1000,
);
return (await _exportHistory.save())._id;

In this implementation, generated exports are configured to expire after seven days, helping prevent generated files from being stored indefinitely.

10. Step Eight: Handle Job Lifecycle Events

Generating a file is only part of the implementation. I also want to know when a job starts, completes or fails. So I use BullMQ worker events.

When a job becomes active

@OnWorkerEvent('active')
onActive(job: Job<AlertExportQueue>) {
  this.alertExportWorker
    .updateExportHistory(
      job.data.exportHistoryId,
      ExportStatus.PROCESSING,
    )
    .catch((error) => {
      this.logger.error(
        'Error updating status to pending',
        error,
      );
    });
}

When a job start processing I update its export history status.

When a job completes

@OnWorkerEvent('completed')
onCompleted(job: Job<AlertExportQueue>) {
  this.clearNotifyTimer(job.data.exportHistoryId);
  this.alertExportWorker
    .updateExportHistory(
      job.data.exportHistoryId,
      ExportStatus.COMPLETED,
    )
    .then((history) => {
      this.eventEmitter.emit(FileEvent.FILE_CREATED, {
        user: job.data.userId,
        id: history._id,
      } as AlertExportEvent);
    });
}

After successful processing I update the status to COMPLETED and emit an event indicating that the file is reday.

When a job fails

@OnWorkerEvent('failed')
onFail(job: Job<AlertExportQueue>, err: Error) {
  this.clearNotifyTimer(job.data.exportHistoryId);
  this.logger.error(err);
  this.alertExportWorker
    .updateExportHistory(
      job.data.exportHistoryId,
      ExportStatus.FAILED,
    )
    .then((history) => {
      this.eventEmitter.emit(FileEvent.FILE_FAILED, {
        user: job.data.userId,
        id: history._id,
      } as AlertExportEvent);
    });
}

If something goes wrong during file generation I update the export status to FAILED and emit an event. This gives the application a way to communicate the result without keeping the original HTTP request open.

11. What About Downloading the File?

After file has been generated, the client can request it using a separate download endpoint.

@Post(':fileId/download')
downloadFile(
  @Param('fileId', ParseMongoIdPipe) id: string,
  @User() user: CurrentUser,
  @Res() res: Response,
) {
  return this.alertsService.download(id, user, res);
}

The download service fetch the generated file from Azure Blob Storage and I also maintain download information including user's email, download count and access time.

This means the export generation and download processes are completely separate.

The background worker generates the file once and authorized users can download the generated file when they need it.

12. What Did I Gain From This Approach?

Moving the export generation process into a background queue provided several architectural advantages.

1. Faster API responses

The API does not need to wait until the entire file has been generated.

2. Better separation of reponsibilities

The controller handles HTTP requests, the service handles business logic and the worker handles background processing.

3. Better handling of large datasets

Using a MongoDB cursor allows me to process records progressively rather than loading every matching document into memory.

4. Reusable generated files

Before generating a new export, I check whether an existing completed file can be reused.

5. Better visibility into background operations

By maintaining export history and job statuses, I can track whether an export is processing, completed, or failed.

6. Cloud-based file storage

Azure Blob Storage keeps generated files separate from the application server's HTTP response lifecycle.

13. Is This the Same as Node.js Worker Threads?

This is something worth understanding.

BullMQ background workers and Node.js worker_threads solve related but different problems.

Node.js worker threads are designed to execute JavaScript in separate threads. They are particularly useful for CPU-intensive operations that would otherwise block the main event loop.

BullMQ, on the other hand, is a job queue system built on Redis. It allows applications to schedule and process background jobs independently of HTTP requests.

In my implementation I'm using BullMQ's NestJS processor architecture.

I'm not directly creating threads using Node.js's worker_threads module.

If I needed to perform CPU intensive operations such as complex calculations, image processing or expensive data transformations I would also consider using worker threads or BullMQ's separate-process processing capabilities.

Choosing the right approach depends on the type of workload.

Conclusion

One thing I've learned from building backend applications is that not every operation needs to happen inside an HTTP request.

Some operations take time, consume resources, and don't need to block the client while they're running.

Generating large Excel files is a good example.

By combining NestJS, BullMQ, Redis, MongoDB cursors, and Azure Blob Storage, I was able to build a background export workflow that separates the request lifecycle from the actual file generation process.

For me, the biggest lesson isn't just about using a queue.

It's about designing backend systems where long-running operations can be handled independently, tracked properly, and made easier to maintain as the application grows.

If you're building a NestJS application that handles large reports, exports, or other long-running tasks, background job processing is definitely an approach worth considering.