Prerequisites
A Test Case with two files available to compare: uploaded as file variables, or produced earlier in the test (for example, a file captured by a previous step, referenced as
[DOWNLOADED_FILE_1]).Supported file types: PDF, DOCX, CSV, XML, JSON, TXT. The two files do not have to be the same type, so you can compare a CSV against a PDF.
Understanding the File Compare Step
The File Compare Step automatically checks one file against another and tells you, in plain language, whether their contents match. It is AI-powered: instead of a rigid byte-by-byte diff, it understands the documents and reports the differences a reviewer would actually care about.
Limitations: The comparison looks at content only. It does not flag differences in fonts, colours, spacing, or layout — for visual checks, use the Visual Compare Step. It supports PDF, DOCX, CSV, XML, JSON, and TXT; archive files (such as .zip) are not supported, so extract or convert the files first.
Why Use the File Compare Step
Confirm a downloaded invoice, export, or report matches an expected reference file.
Check that a generated document filled a template in correctly.
Catch changed, missing, or added values across formats, even when the two files are different types.
Adding a File Compare Step
Add a new step to your Test Case and choose File Compare. Then set:
File 1: the first file to compare (often your expected or reference file).
File 2: the second file (often the actual or generated file).
Guidance (optional): a short instruction describing how the two files relate and what "correct" means.
Tip: Guidance is the single biggest lever on accuracy. One clear sentence about how the files relate dramatically improves the result.
Writing Good Guidance
Guidance tells the comparison which of two styles to use:
Faithful comparison (default): both files hold concrete values. Every field is compared directly, and any value in File 2 that differs from, is missing from, or contradicts File 1 is reported as a mismatch. Use this for "does the actual file match the expected one."
Template to generate: one file is a template containing placeholders (such as
[CUSTOMER_NAME]) and the other is the document generated from it. Here, a placeholder correctly replaced by a real value is expected and correct, not a difference.
Examples of useful Guidance:
"File 1 is the expected order, File 2 is the downloaded invoice. The customer, item, quantity, and total should match."
"File 1 is a template, File 2 is the generated contract. Ignore the timestamp column."
"File 1 is the contract template, File 2 is the generated PDF.
[CLIENT_NAME]and[DATE]should be filled in correctly, not left as placeholders."
Reading the Results
When the step runs, you get an overall result and a breakdown:
Passed: no meaningful differences were found.
Failed: at least one mismatch was found.
Open the step to see each finding as its own sub-step, with the expected value, the actual value, and a short snippet of evidence from each file (including the page for PDF and DOCX). When the two files are identical, the result reads No differences found.
Troubleshooting / FAQ
"File has an unsupported format." See Limitations above for supported formats. Archive files (such as .zip) need to be extracted or converted first, then compare the files inside.
The result looks too lenient or too strict. Add Guidance describing the relationship between the files and what should match.
A value that should match is reported as different. Check that File 1 and File 2 are assigned the way your Guidance describes them (File 1 = expected or template, File 2 = actual or generated).