JSONL is a text format with one complete JSON object on each line. AI labs prefer it for training data because files can be read line by line, split, sampled and appended without loading everything. A valid file is UTF-8, has no blank lines, and uses the same keys in every record.
What JSONL is
JSONL stands for JSON Lines. Each line of the file is one JSON value, almost always an object, followed by a line break. There is no opening bracket around the file and no comma between records. The file extension is usually .jsonl, and some tools use .ndjson for the same idea.
The rules are few. The file is encoded as UTF-8. Each line is a complete, valid JSON object. Line breaks inside a string value are written as the escape sequence backslash n, never as a real line break, because a real line break ends the record. Blank lines are not allowed by strict parsers.
A record for a support exchange might hold four keys: an id, a timestamp, a channel and a text field. The next line holds the next record with the same four keys. Nothing links one line to another, which is the point of the format.
Why AI labs prefer it
Training pipelines process data as a stream of independent examples. JSONL matches that model. A program can read one line, use it and move on without holding the whole file in memory. A file of any size can be split at any line break into valid smaller files, and two files can be joined by appending one to the other.
A standard JSON array does not allow this. The parser has to read to the closing bracket before the file is valid, and one error anywhere can make the whole file unreadable. With JSONL, a bad line affects that line alone and can be reported by number.
Compared with CSV, JSONL handles nested structure and free text more safely. A conversation with a list of turns, or a document with commas, quotes and line breaks, fits naturally in a JSON object. In CSV the same content depends on quoting rules that tools apply inconsistently.
| Format | Strength | Weakness for training data |
|---|---|---|
| JSONL | Streamable; nested data; errors isolated to a line | Keys repeated on every line make files larger |
| JSON array | Widely supported; one valid document | Must be parsed whole; one error breaks the file |
| CSV | Compact; opens in a spreadsheet | No nesting; free text breaks on quotes and line breaks |
| Parquet | Compressed, typed columns; fast analytics | Binary; cannot be inspected or edited as text |
Chat formats
Conversational data is usually delivered in a chat layout. Each line is one conversation. The object has a key, commonly named messages, that holds an ordered list of turns. Each turn is an object with a role and a content field. Roles are typically system, user and assistant. Several model providers accept this layout for fine-tuning, though the exact field names and options differ, so confirm the buyer's schema before converting.
Company data needs mapping into that shape. In a support ticket, the customer is normally mapped to the user role and the agent to the assistant role. Internal notes do not fit either role. They can be kept in a separate metadata field or left out, depending on what the buyer wants.
Keep turn order and do not merge consecutive messages unless asked. Carry useful context in a metadata object beside the messages: a hashed conversation ID, timestamps, the channel, the product area and the outcome. The outcome is especially valuable, because it tells the buyer which conversations ended in a resolution.
Common validation errors
Most rejected JSONL deliveries fail for a small set of reasons. All of them are easy to find with a validator before the buyer does. The checker at /tools/jsonl-validator parses every line in the browser and reports the line number of each failure and of any record whose keys differ from the rest.
- A JSON array saved with a .jsonl extension: the file starts with a bracket and records are separated by commas
- Pretty-printed objects that span several lines instead of one
- A raw line break inside a string value that splits a record in two
- Trailing commas after the last key in an object
- Single quotes used in place of double quotes around keys or strings
- Unescaped double quotes or backslashes inside text
- A byte order mark at the start of the file, or an encoding other than UTF-8
- Blank lines between records or at the end of the file
- Inconsistent keys: a field present in some records and missing or renamed in others
- Inconsistent types: a field that is a string in some records and a number or null in others
- Values such as NaN or Infinity, which are not valid JSON
Converting from CSV and JSON
Most exports start as CSV or as a JSON array. The converter at /tools/csv-json-converter turns either into JSONL without uploading the file. Each CSV row becomes one object, with the column headers as keys.
Check three things after converting. First, types. CSV has no types, so numbers, dates and true or false values arrive as text unless converted. Agree with the buyer whether an ID should be a string or a number, and write dates in one format, preferably ISO 8601. Second, empty values. Decide whether an empty cell becomes an empty string, a null or an omitted key, and apply the rule everywhere. Third, key names. Use short, plain, consistent names with no spaces.
When a conversation is spread across many CSV rows, one row per message, the rows have to be grouped by conversation ID and sorted by time to build the messages list. That step needs a script or a query. A flat conversion will produce one object per message, which is valid JSONL but not a chat format.
Chunking long records
Chunking means splitting a long text into smaller pieces. It is needed when a record is longer than the buyer's models or tools can take in one pass, or when the data will be used for retrieval, where small passages work better than whole documents.
Split on natural boundaries. For documents, break at headings and paragraphs, not in the middle of a sentence. For conversations, break between turns and never inside one. For code, break between files or functions. Each chunk should make sense when read alone.
Every chunk needs to carry its origin: the ID of the parent record, its position in the sequence and, where useful, the heading it sat under. Some pipelines repeat a little text between neighboring chunks so that meaning is not lost at the cut. Ask the buyer before chunking at all. Many prefer to receive whole records and chunk them with their own tools, and chunks cannot be reliably rejoined once the boundaries are lost.
Counting tokens
Models read tokens, not words or characters. A token is a fragment of text, often a short word or part of a longer one. Buyers size and price text datasets in tokens because the count shows how much material a model will actually process.
The count depends on the tokenizer, and each model family has its own. The same file can produce somewhat different totals under two tokenizers, so any count given to a buyer should say which tokenizer was used or state that it is an estimate. The counter at /tools/token-counter gives an estimate for pasted text or a file.
Count the content fields, not the whole file. The JSON keys, brackets and quotation marks repeat on every line and add to the raw total without adding information. A buyer cares about the tokens in the text itself. Report tokens per record as well as the total, since a few very long records can distort an average.
A delivery checklist
Before a JSONL dataset leaves the building, run through these checks. Outrich Data asks for deliveries in this form, and most AI labs apply similar checks on arrival.
- Every line parses as one JSON object, with zero errors from the validator
- UTF-8 encoding, no byte order mark, no blank lines
- The same keys and types in every record, matching the agreed schema
- Dates in one format and IDs hashed consistently across files
- Personal data scrubbed from text fields, and a sample read by hand
- Files split to a size that is practical to transfer, each valid alone
- Record count and token estimate written into the data card
- A SHA-256 manifest listing every file