Streaming CSV Data in Node.js Applications
csv-parser, csv-writer, and json2csv are essential utilities for handling Comma-Separated Values (CSV) in Node.js environments, but they solve different parts of the data pipeline. csv-parser is a high-performance, stream-based library designed exclusively for reading and parsing CSV data into JavaScript objects with minimal memory overhead. csv-writer (specifically csv-writer by txn2) focuses on the opposite direction, providing a clean interface to stream JavaScript objects or arrays into valid CSV files. json2csv is a versatile converter that transforms JSON data structures into CSV format, often used for one-off exports or report generation, though it has historically leaned towards buffering entire datasets in memory before output.
Npm Package Weekly Downloads Trend
Github Stars Ranking
Stat Detail
Streaming CSV Data: Parsing, Writing, and Converting in Node.js
Handling CSV data is a common requirement in backend development, whether you are ingesting user uploads, exporting database records, or migrating legacy data. The three libraries csv-parser, csv-writer, and json2csv address specific stages of this workflow. While they all deal with comma-separated values, their architectural approaches to memory management and data flow differ significantly. Let's break down how they handle real-world engineering challenges.
š„ Reading Data: Stream-First vs. Buffer-Based
When reading CSV files, the biggest risk in Node.js is running out of memory. If you load a 2GB file into an array, your process will crash. This is where the architectural difference between streaming and buffering matters most.
csv-parser is built strictly as a transform stream. It reads the file chunk by chunk, parses a line, emits an object, and immediately forgets that line. This allows you to process files of any size with a constant, tiny memory footprint.
// csv-parser: True streaming ingestion
const fs = require('fs');
const csv = require('csv-parser');
const results = [];
fs.createReadStream('large-data.csv')
.pipe(csv())
.on('data', (row) => {
// Process one row at a time
console.log(row);
})
.on('end', () => {
console.log('CSV file successfully processed');
});
json2csv is primarily a converter. While it has added stream support in newer versions (json2csv.Streamer), its most common usage pattern involves passing a full JSON array to the parser. If you use the standard parse method, it attempts to hold the data in memory, which can be dangerous for large datasets.
// json2csv: Typical buffer-based usage (Risky for large files)
const { Parser } = require('json2csv');
const data = [
{ name: 'John', age: 30 },
{ name: 'Jane', age: 25 }
];
try {
const parser = new Parser();
const csv = parser.parse(data); // Requires full data array in memory
console.log(csv);
} catch (err) {
console.error(err);
}
csv-writer does not read CSV files. It is strictly for writing. If you try to use it for parsing, you will find no API methods to do so. You must pair it with a reader like csv-parser if you need to read and then rewrite data.
š¤ Writing Data: Explicit Schemas vs. Auto-Detection
When generating CSV files, you often need to control the column order and headers precisely, especially if the downstream system is strict.
csv-writer forces you to define a schema (header configuration) upfront. This is a feature, not a bug. It ensures that your output CSV always has consistent columns, even if your input data is missing fields or has extra properties. It streams the output immediately.
// csv-writer: Explicit schema definition ensures consistency
const createObjectCsvWriter = require('csv-writer').createObjectCsvWriter;
const csvWriter = createObjectCsvWriter({
path: 'output.csv',
header: [
{ id: 'name', title: 'NAME' },
{ id: 'age', title: 'AGE' }
]
});
const records = [
{ name: 'Bob', age: 21 },
{ name: 'Alice', age: 34 }
];
// Writes headers first, then streams records
await csvWriter.writeRecords(records);
json2csv tries to be smart by auto-detecting fields from your JSON data. While convenient for quick scripts, this can lead to unpredictable column orders if your JSON objects have inconsistent keys. You can override this, but the default behavior is "guess the structure."
// json2csv: Auto-detects fields from data keys
const { Parser } = require('json2csv');
const data = [{ name: 'Bob', age: 21 }, { name: 'Alice', city: 'NY' }];
const parser = new Parser();
// Output columns might vary or include empty fields based on detection
const csv = parser.parse(data);
csv-parser cannot write data. It has no methods to generate CSV strings or files. Using it for writing is impossible.
ā” Memory Efficiency: The Critical Trade-off
The choice between these libraries often comes down to one question: "Do I have the whole dataset in memory already?"
If you are building an API endpoint that exports a database query result, you likely already have the data in an array. In this case, json2csv is the most direct tool. You pass the array, get a string, and send it as a response.
// json2csv: Perfect for in-memory API exports
app.get('/export', (req, res) => {
const data = await db.query('SELECT * FROM users'); // Already in memory
const csv = new Parser().parse(data);
res.header('Content-Type', 'text/csv');
res.send(csv);
});
However, if you are processing a file upload from a user that could be gigabytes in size, you must avoid holding the data in memory. Here, csv-parser (for reading) and csv-writer (for writing) are the only safe choices. They allow you to pipe data directly from the request stream to your logic, and then to a file stream, without ever storing the full dataset in RAM.
// csv-parser + csv-writer: Safe pipeline for massive files
const { createObjectCsvWriter } = require('csv-writer');
const csv = require('csv-parser');
const fs = require('fs');
const csvWriter = createObjectCsvWriter({
path: 'cleaned-data.csv',
header: [{ id: 'id', title: 'ID' }, { id: 'valid', title: 'VALID' }]
});
fs.createReadStream('upload.csv')
.pipe(csv())
.on('data', async (row) => {
// Process and write row immediately
await csvWriter.writeRecords([{ id: row.id, valid: row.status === 'ok' }]);
});
š ļø Handling Complex Data and Edge Cases
Real-world CSV data is messy. It contains quotes, newlines inside fields, and different delimiters.
csv-parser handles complex escaping and delimiters robustly because it is dedicated solely to parsing. You can easily configure it to handle tabs or custom separators.
// csv-parser: Handling custom delimiters
fs.createReadStream('data.tsv')
.pipe(csv({ separator: '\t' })) // Tab-separated values
.on('data', (row) => console.log(row));
json2csv excels at flattening nested JSON objects. If your data looks like { user: { name: 'John' } }, json2csv can flatten this to user.name automatically using its unwind or flatten options, which is a pain to do manually with streams.
// json2csv: Flattening nested structures
const parser = new Parser({
fields: ['user.name', 'user.email'],
flatten: true
});
const csv = parser.parse([{ user: { name: 'John', email: 'j@example.com' } }]);
csv-writer expects flat objects. If you pass nested objects, it will output [object Object] unless you manually flatten the data before passing it to the writer. It does not offer built-in transformation helpers.
š Summary Table
| Feature | csv-parser | csv-writer | json2csv |
|---|---|---|---|
| Primary Goal | Read/Parse CSV | Write/Generate CSV | Convert JSON to CSV |
| Memory Model | Streaming (Low Memory) | Streaming (Low Memory) | Buffering (High Memory)* |
| Schema Control | Auto-detects headers | Explicit schema required | Auto-detects or manual |
| Nested Data | Flattens to strings | Expects flat objects | Can flatten/unwind |
| Best For | Large file ingestion | Log exports, reports | API exports, small datasets |
*Note: json2csv supports streaming via json2csv.Streamer, but the common Parser API buffers data.
š” The Big Picture
These tools are not interchangeable; they are specialized components of a data pipeline.
csv-parser is your intake valve. Use it when data is coming in from the outside world and you need to process it safely without blowing up your server. It is the standard for robust, production-grade CSV ingestion.
csv-writer is your output nozzle. Use it when you need to generate clean, schema-compliant CSV files from your application data, especially when dealing with large volumes that require streaming.
json2csv is your translator. Use it when you already have JSON data in memory and need to quickly convert it to a CSV format for human consumption or simple downloads. It prioritizes developer convenience and feature richness (like flattening) over raw streaming performance.
Final Thought: For modern, scalable Node.js applications, prefer the streaming duo (csv-parser and csv-writer) for file operations. Reserve json2csv for specific conversion tasks where the data size is known to be small and manageable.
How to Choose: csv-parser vs csv-writer vs json2csv
- csv-parser:
Choose
csv-parserwhen you need to process large CSV files (hundreds of MBs or GBs) without crashing your server's memory. It is the ideal choice for ETL pipelines, data ingestion services, or any scenario where you must parse data row-by-row as it arrives via a stream. Avoid this if you need to write CSV files, as it is read-only. - csv-writer:
Choose
csv-writerwhen your primary goal is to generate CSV files from structured data in a streaming fashion. It is best suited for services that aggregate logs, export database query results, or create downloadable reports where memory efficiency is critical. It is not suitable for parsing existing CSV files. - json2csv:
Choose
json2csvwhen you have a complete JSON dataset in memory and need to convert it to CSV format for a quick export, email attachment, or simple file download. It is excellent for administrative dashboards or scripts where the dataset size is predictable and small enough to fit in RAM. Be cautious with very large datasets, as some configurations may require loading all data before writing.
Popular Comparisons
README for csv-parser
csv-parser
Streaming CSV parser that aims for maximum speed as well as compatibility with the csv-spectrum CSV acid test suite.
csv-parser can convert CSV into JSON at at rate of around 90,000 rows per
second. Performance varies with the data used; try bin/bench.js <your file>
to benchmark your data.
csv-parser can be used in the browser with browserify.
neat-csv can be used if a Promise
based interface to csv-parser is needed.
Note: This module requires Node v8.16.0 or higher.
Benchmarks
ā”ļø csv-parser is greased-lightning fast
ā npm run bench
Filename Rows Parsed Duration
backtick.csv 2 3.5ms
bad-data.csv 3 0.55ms
basic.csv 1 0.26ms
comma-in-quote.csv 1 0.29ms
comment.csv 2 0.40ms
empty-columns.csv 1 0.40ms
escape-quotes.csv 3 0.38ms
geojson.csv 3 0.46ms
large-dataset.csv 7268 73ms
newlines.csv 3 0.35ms
no-headers.csv 3 0.26ms
option-comment.csv 2 0.24ms
option-escape.csv 3 0.25ms
option-maxRowBytes.csv 4577 39ms
option-newline.csv 0 0.47ms
option-quote-escape.csv 3 0.33ms
option-quote-many.csv 3 0.38ms
option-quote.csv 2 0.22ms
quotes+newlines.csv 3 0.20ms
strict.csv 3 0.22ms
latin.csv 2 0.38ms
mac-newlines.csv 2 0.28ms
utf16-big.csv 2 0.33ms
utf16.csv 2 0.26ms
utf8.csv 2 0.24ms
Install
Using npm:
$ npm install csv-parser
Using yarn:
$ yarn add csv-parser
Usage
To use the module, create a readable stream to a desired CSV file, instantiate
csv, and pipe the stream to csv.
Suppose you have a CSV file data.csv which contains the data:
NAME,AGE
Daffy Duck,24
Bugs Bunny,22
It could then be parsed, and results shown like so:
const csv = require('csv-parser')
const fs = require('fs')
const results = [];
fs.createReadStream('data.csv')
.pipe(csv())
.on('data', (data) => results.push(data))
.on('end', () => {
console.log(results);
// [
// { NAME: 'Daffy Duck', AGE: '24' },
// { NAME: 'Bugs Bunny', AGE: '22' }
// ]
});
To specify options for csv, pass an object argument to the function. For
example:
csv({ separator: '\t' });
API
csv([options | headers])
Returns: Array[Object]
options
Type: Object
As an alternative to passing an options object, you may pass an Array[String]
which specifies the headers to use. For example:
csv(['Name', 'Age']);
If you need to specify options and headers, please use the the object notation
with the headers property as shown below.
escape
Type: String
Default: "
A single-character string used to specify the character used to escape strings in a CSV row.
headers
Type: Array[String] | Boolean
Specifies the headers to use. Headers define the property key for each value in
a CSV row. If no headers option is provided, csv-parser will use the first
line in a CSV file as the header specification.
If false, specifies that the first row in a data file does not contain
headers, and instructs the parser to use the column index as the key for each column.
Using headers: false with the same data.csv example from above would yield:
[
{ '0': 'Daffy Duck', '1': 24 },
{ '0': 'Bugs Bunny', '1': 22 }
]
Note: If using the headers for an operation on a file which contains headers on the first line, specify skipLines: 1 to skip over the row, or the headers row will appear as normal row data. Alternatively, use the mapHeaders option to manipulate existing headers in that scenario.
mapHeaders
Type: Function
A function that can be used to modify the values of each header. Return a String to modify the header. Return null to remove the header, and it's column, from the results.
csv({
mapHeaders: ({ header, index }) => header.toLowerCase()
})
Parameters
header String The current column header.
index Number The current column index.
mapValues
Type: Function
A function that can be used to modify the content of each column. The return value will replace the current column content.
csv({
mapValues: ({ header, index, value }) => value.toLowerCase()
})
Parameters
header String The current column header.
index Number The current column index.
value String The current column value (or content).
newline
Type: String
Default: \n
Specifies a single-character string to denote the end of a line in a CSV file.
quote
Type: String
Default: "
Specifies a single-character string to denote a quoted string.
raw
Type: Boolean
If true, instructs the parser not to decode UTF-8 strings.
separator
Type: String
Default: ,
Specifies a single-character string to use as the column separator for each row.
skipComments
Type: Boolean | String
Default: false
Instructs the parser to ignore lines which represent comments in a CSV file. Since there is no specification that dictates what a CSV comment looks like, comments should be considered non-standard. The "most common" character used to signify a comment in a CSV file is "#". If this option is set to true, lines which begin with # will be skipped. If a custom character is needed to denote a commented line, this option may be set to a string which represents the leading character(s) signifying a comment line.
skipLines
Type: Number
Default: 0
Specifies the number of lines at the beginning of a data file that the parser should skip over, prior to parsing headers.
maxRowBytes
Type: Number
Default: Number.MAX_SAFE_INTEGER
Maximum number of bytes per row. An error is thrown if a line exeeds this value. The default value is on 8 peta byte.
strict
Type: Boolean
Default: false
If true, instructs the parser that the number of columns in each row must match
the number of headers specified or throws an exception.
if false: the headers are mapped to the column index
less columns: any missing column in the middle will result in a wrong property mapping!
more columns: the aditional columns will create a "_"+index properties - eg. "_10":"value"
outputByteOffset
Type: Boolean
Default: false
If true, instructs the parser to emit each row with a byteOffset property.
The byteOffset represents the offset in bytes of the beginning of the parsed row in the original stream.
Will change the output format of stream to be { byteOffset, row }.
Events
The following events are emitted during parsing:
data
Emitted for each row of data parsed with the notable exception of the header row. Please see Usage for an example.
headers
Emitted after the header row is parsed. The first parameter of the event
callback is an Array[String] containing the header names.
fs.createReadStream('data.csv')
.pipe(csv())
.on('headers', (headers) => {
console.log(`First header: ${headers[0]}`)
})
Readable Stream Events
Events available on Node built-in
Readable Streams
are also emitted. The end event should be used to detect the end of parsing.
CLI
This module also provides a CLI which will convert CSV to newline-delimited JSON. The following CLI flags can be used to control how input is parsed:
Usage: csv-parser [filename?] [options]
--escape,-e Set the escape character (defaults to quote value)
--headers,-h Explicitly specify csv headers as a comma separated list
--help Show this help
--output,-o Set output file. Defaults to stdout
--quote,-q Set the quote character ('"' by default)
--remove Remove columns from output by header name
--separator,-s Set the separator character ("," by default)
--skipComments,-c Skip CSV comments that begin with '#'. Set a value to change the comment character.
--skipLines,-l Set the number of lines to skip to before parsing headers
--strict Require column length match headers length
--version,-v Print out the installed version
For example; to parse a TSV file:
cat data.tsv | csv-parser -s $'\t'
Encoding
Users may encounter issues with the encoding of a CSV file. Transcoding the source stream can be done neatly with a modules such as:
Or native iconv if part
of a pipeline.
Byte Order Marks
Some CSV files may be generated with, or contain a leading Byte Order Mark. This may cause issues parsing headers and/or data from your file. From Wikipedia:
The Unicode Standard permits the BOM in UTF-8, but does not require nor recommend its use. Byte order has no meaning in UTF-8.
To use this module with a file containing a BOM, please use a module like strip-bom-stream in your pipeline:
const fs = require('fs');
const csv = require('csv-parser');
const stripBom = require('strip-bom-stream');
fs.createReadStream('data.csv')
.pipe(stripBom())
.pipe(csv())
...
When using the CLI, the BOM can be removed by first running:
$ sed $'s/\xEF\xBB\xBF//g' data.csv