Quality and variety · Autessa Synthetic Data
De-duplication and the variety ledger
Autessa Synthetic Data removes near-duplicate rows across the whole job and keeps a variety ledger of what it has already made. The ledger is fed back into generation, so batch 40 does not repeat batch 1 and appended rows stay fresh.
The problem
Large synthetic sets tend to repeat themselves
Ask a model for 5,000 customers and you get the same twenty company names in different orders. Duplicates inflate the row count, skew tests and make demos look fake.
What you get
- Each row gets an identity signature from up to three key text columns, ignoring case and extra spaces.
- Rows that match an earlier signature in the job are dropped as duplicates.
- A running ledger of recent identities is included in later prompts so the model avoids them.
- Parallel batches share the ledger as each batch lands.
- Appends start from the job’s ledger, so new rows do not repeat earlier ones.
- Generation runs at a high-variety setting on purpose.
How it works
How does de-duplication and the variety ledger work?
Sign each row
Autessa builds a normalized signature from the row’s main text columns, preferring required ones.
Check the job
A row whose signature was already seen in the job is dropped and counted as rejected.
Update the ledger
Accepted signatures join the job’s ledger, which keeps the most recent 200.
Steer the next batch
The ledger goes into the next prompt so the model writes something new.
Every option
How variety is kept
| Option | What it does |
|---|---|
| Signature | Lowercase, spacing-collapsed text of up to three key text columns. |
| Scope | The whole job, including appends. |
| Ledger size | The 200 most recent identities are fed back into prompts. |
| Storage | The ledger is stored encrypted with your company key. |
| Limit | Two parallel batches finishing at the same moment can still produce one near-duplicate between them. |
Permissions
Who can do what?
| Who | Can |
|---|---|
| People in your company | Create datasets, see the company list of jobs, open any job and review, approve, append, cancel or discard it. |
Where AI does the work
How the generator handles this
The ledger is shown to the model as "already made, do not repeat", so variety is designed in rather than filtered out after the fact.
Generation runs on your company default model, and every row is checked against your columns before it is staged or written.
Your model and PII stripping →Connected
How de-duplication and the variety ledger connects to the rest of Autessa
FAQ
Questions about de-duplication and the variety ledger
Does it remove exact duplicates only?
No. Signatures ignore case and spacing, so "Brightline Dental" and "brightline dental" count as the same row.
What if my table has no text columns?
Rows without a usable text identity are not de-duplicated. Add a descriptive text column if variety matters.
Can I turn de-duplication off?
No. It is always on, because duplicate rows are almost never what a test or demo needs.
100% customizable, code and all
Make Autessa Synthetic Data fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours