Skip to content
Outrich Data

Encoding and mojibake fixer

Drop a file or paste text and this tool detects the encoding, converts it to UTF-8 and repairs garbled characters such as é and ’. It also removes byte order marks, control and zero-width characters, mixed line endings and trailing spaces, and reports every fix. It runs entirely in your browser.

How to use it

  1. 01Drop a file or paste text. Files are read as bytes so the encoding can be detected.
  2. 02Switch individual fixes on or off. Converting smart quotes and dashes to ASCII is off by default.
  3. 03Read the count for each fix and the before and after view of changed lines.
  4. 04Download the cleaned file as UTF-8, or copy the text.

What it does not do

  • Encoding detection covers UTF-8, UTF-16 with a byte order mark, and Windows-1252 as the fallback. Other legacy encodings such as Shift JIS are not detected.
  • Mojibake repair reverses UTF-8 read as Windows-1252 or Latin-1. Characters that were replaced with a question mark or a box are already lost and cannot be restored.
  • Removing zero-width characters can change joined emoji and some Persian and Indic text. Switch that fix off for those files.
  • Files are read into memory, so the limit is 50 MB.

Questions

About the encoding and mojibake fixer.

Is my data uploaded anywhere?
No. The tool runs entirely in your browser. Your text and files are never sent to Outrich Data or anyone else, and nothing is stored after you close the tab. You can confirm it by turning off your internet connection after the page loads: the tool keeps working.
What is mojibake?
Garbled text produced when a file saved in one encoding is opened in another. The usual case is UTF-8 opened as Windows-1252, which turns é into é and a curly apostrophe into ’. The bytes are intact, so the damage can be reversed.
Why does encoding matter for AI training data?
Garbled characters become junk tokens. A model trained on them learns the noise, and a search index built on them fails to match the correct spelling. Buyers check for this, and a clean UTF-8 file avoids a round of rework.
What is a byte order mark?
An invisible marker some programs write at the start of a file. In a CSV it attaches to the first column name, so a column called id is read as a different name and lookups fail. Removing it is safe for UTF-8 files.
Is it really free?
Yes. There is no account, no limit on the number of uses and no watermark. Files up to 50 MB are supported.

More free tools

Personal data scrubber

Strip names, emails, phones, cards, keys and addresses from text and files.

Dataset profiler

Rows, columns, tokens, date span, duplicates and personal-data columns in one report.

Column hasher and remover

Drop sensitive columns or replace them with salted hashes that still join.

CSV, JSON and JSONL converter

Convert between CSV, TSV, JSON and JSONL without breaking quoted cells.

JSONL validator

Find the broken lines in a JSONL file and see which keys are missing.

Duplicate remover

Remove repeated rows, records or log lines, keeping the first of each.

Token counter

Size text the way AI labs do: tokens, words, characters and lines.

Data card generator

Write the one-page description every data buyer asks for.

Data rights pre-check

A tailored checklist of the contracts and questions to clear before a sale.

Slack export parser

Turn a Slack workspace export into clean thread JSONL with pseudonymized users.

Email thread parser

Convert .mbox and .eml files into threaded JSONL without quoted replies.

Ticket export parser

Map help desk exports to customer and agent turns in conversation JSONL.

Chat dataset format converter

Convert and validate OpenAI, ShareGPT, Alpaca and prompt/completion datasets.

Log parser

Turn web server, syslog, logfmt and JSON logs into structured JSONL or CSV.

HTML and document text extractor

Extract clean text from HTML, Markdown, XML and JSON files in batches.

Re-identification risk checker

Measure k-anonymity and l-diversity, then preview fixes that lower the risk.

Secrets scanner

Find API keys, tokens, passwords and private keys in a repo or export before you share it.

Delivery manifest builder

Create a SHA-256 checksum manifest for a dataset delivery, with one hash to compare.

Delivery verifier

Check received files against a manifest: match, changed, missing or extra.

File checksum calculator

SHA-256 of any file, with a compare box that turns green or red.

Train, validation and test splitter

Split a dataset into train, validation and test files without leaking related rows across splits.

Dataset sampler

Pull a random, stratified or first-N sample from a file, including files too large to open.

Schema inferrer and data dictionary

Infer field types and stats from a file and export JSON Schema, a data dictionary and SQL.

Text chunker

Split documents into overlapping chunks for retrieval or training, with heading context.

Dataset diff

Compare two versions of a dataset: added, removed and changed records, plus schema changes.

Column toolkit

Rename, split, merge, clean, filter and sort columns with a reusable pipeline of steps.

Cleaned it up? Sell it to Outrich.

Outrich Data buys company data directly. Check in five minutes whether yours qualifies, with no call.

esc