Prerequisites
A Test Case with two files available to compare, such as a file downloaded by an earlier step and referenced as
[DOWNLOADED_FILE_1].Both files within the formats and size limits listed under Limits and Supported Files below.
Understanding the File Compare Step
The File Compare Step automatically checks one file against another and tells you, in plain language, whether their contents match. It is AI-powered: instead of a rigid byte-by-byte diff, it understands the documents and reports the differences a reviewer would actually care about.
That distinction is what makes the step useful. A byte diff on two PDFs generated seconds apart fails on the timestamp in the footer and tells you nothing. It reads both documents and surfaces what actually matters: a total that does not add up, a placeholder that never got filled in, a row that went missing from an export.
Limitations: The comparison looks at content only. It does not flag differences in fonts, colours, spacing, or layout. For visual checks, use the Visual Compare Step.
Why Use the File Compare Step
Document output is where end-to-end tests usually stop looking. The test clicks Generate Invoice, sees a file land in the Downloads folder, and passes, without anyone ever checking what is inside it. The File Compare Step closes that gap.
Confirm a downloaded invoice, export, or report matches an expected reference file.
Check that a generated document filled a template in correctly, and that no placeholder leaked through.
Catch changed, missing, or added values across formats, even when the two files are different types.
Adding a File Compare Step
Add a new step to your Test Case and choose File Compare. Then set:
File 1: the first file to compare (often your expected or reference file).
File 2: the second file (often the actual or generated file).
Guidance (optional): a short instruction describing how the two files relate and what "correct" means.
All three fields accept test variables, so a file produced earlier in the run goes straight in as [DOWNLOADED_FILE_1], and Guidance can reference the values your test generated, such as [CUSTOMER_NAME]. You can also attach a file directly to File 1 or File 2 instead of referencing a variable.
Tip: Guidance is the single biggest lever on accuracy. One clear sentence about how the files relate dramatically improves the result.
Writing Good Guidance
Guidance does two jobs: it tells the comparison how the two files relate, and it defines what to check.
Choosing the comparison mode
Faithful comparison (default): both files hold concrete values. Every field is compared directly, and any value in File 2 that differs from, is missing from, or contradicts File 1 is reported as a mismatch. Use this for "does the actual file match the expected one." If the two files turn out to describe entirely different records, with different order numbers, customers or totals, that mismatch is reported too rather than quietly accepted.
Template to generate: one file is a template containing placeholders (such as
[CUSTOMER_NAME]or{d.customerName}) and the other is the document generated from it. Here, a placeholder correctly replaced by a real value is expected and correct, not a difference. A comparison in this mode fails when a placeholder leaks through literally, when a field that should have been filled is blank, or when a value contradicts what your Guidance says it should be.
Template mode is chosen automatically when either file visibly contains placeholders, so you do not have to name it, but saying so explicitly is still the most reliable option.
Defining what gets checked
Guidance also sets the scope of the comparison. Anything it asks about is checked and reported; a real difference it does not ask about is left out of the results entirely. Broad Guidance means a broad check, narrow Guidance means a narrow one.
Three runs over the same two product catalogues show the difference:
Guidance: "Check that File 2 matches File 1 exactly." → 3 mismatched. Every field is compared, so the changed price, the missing row, and the extra row are all reported.
Guidance: "Verify that the price of SKU-114 is the same in both files." → 1 mismatched. Only the price is checked; the missing row and the extra row are out of scope and are not reported.
Guidance: "Verify that the extension is the same." → 1 matched, so the step passes. Both files are CSV, which is all that was asked. The price difference is real, but out of scope.
The second and third runs, on that same pair of files:
If a difference you care about is missing from the results, widen your Guidance. If the results are noisy, narrow it. Guidance can also name fields to ignore, such as a timestamp column or an invoice number.
Examples
Checking a generated invoice against its purchase order
The most common pattern: the test drives the application, downloads the document it produced, and compares it against a reference. Because Guidance can carry a field mapping, the two documents do not have to use the same labels.
Test Case:
Navigate to the invoicing page.
Generate values into
[CUSTOMER_NAME],[ITEM_NAME],[QUANTITY], and[UNIT_PRICE].Type each value into the matching form field.
Click Generate Invoice. The generated PDF is captured as
[DOWNLOADED_FILE_1].Add a File Compare Step with File 1 set to
expected_order.pdf, File 2 set to[DOWNLOADED_FILE_1], and this Guidance: "File 1 is a purchase order and File 2 is the invoice generated from it. Compare the data in File 1 against File 2 using this field mapping: Customer[CUSTOMER_NAME]to Bill To, Line Item[ITEM_NAME]to Description, Quantity[QUANTITY]to Units, Unit Price[UNIT_PRICE]to Rate, Expected Total to Amount Due. Ignore document title, type, invoice number, order reference, and dates. Treat currency formatting as equal. Pass only if all mapped fields match."
The value here is in what the step catches that a UI assertion cannot: the invoice renders perfectly, the download succeeds, and the total is still wrong.
Catching a placeholder that never got filled
Set File 1 to quote_template.docx, File 2 to [DOWNLOADED_FILE_1], and the Guidance to "File 1 is a template containing placeholder fields; File 2 is the document generated from it. Check that every placeholder was replaced with a real value."
Result: the step fails with a single finding, Quote number placeholder, reporting that the quote number was never replaced in File 2.
Every other placeholder in the document was substituted correctly and is reported as passed, so the one broken field is easy to spot. Templates with conditional sections are handled the same way: exactly one branch should appear in the generated document, and a block that rendered no branch, or both, is reported as a failure.
Verifying a data export against reference data
Set File 1 to products_reference.csv, File 2 to [DOWNLOADED_FILE_1], and the Guidance to "Check that File 2 matches File 1 exactly." This catches the three ways an export goes wrong, each as its own finding:
Unit price for SKU-114: the unit price differs between the two files.
Missing SKU-117: the row is in File 1 but absent from File 2.
Extra SKU-999: File 2 carries a row that File 1 does not have.
Comparing across two different formats
Set File 1 to transactions.json, File 2 to transactions_export.csv, and the Guidance to "File 2 is a CSV export of the transactions listed in File 1. Check that every transaction matches."
Result: the step fails with one finding, TX-4471 amount, reporting that this transaction's amount differs between the two files.
A text diff cannot line up a JSON document against a CSV at all. The File Compare Step reads the transactions out of both and matches them by ID.
Reading the Results
When the step runs, you get an overall result and a breakdown:
A green step with a matched badge above the comparison: no meaningful differences were found within the scope your Guidance defined.
A red step with a mismatched badge: at least one mismatch was found.
The badge counts the checks, so it reads 3 mismatched when three findings failed and 1 matched when the files agree.
Open the step to see each finding as its own sub-step, with the expected value, the actual value, and a short snippet of evidence from each file (including the page for PDF and DOCX). A passing step lists no findings under it:
Selecting a finding opens both documents side by side, as File 1 and File 2, with the evidence highlighted in each, so you can see the mismatched value in its original context instead of taking the summary on trust.
Re-running a step whose files and Guidance have not changed returns the same result immediately. Editing either one runs a fresh comparison.
Limits and Supported Files
Supported formats: PDF, DOCX, CSV, XML, JSON, TXT. The two files do not have to be the same type, so you can compare a CSV against a PDF.
Maximum size: 20 MB per file.
Maximum length: 100 pages per PDF.
PDF files must contain selectable text. Scanned or image-only PDFs are not supported, because there is no OCR.
Password-protected or corrupted files cannot be read.
Archive files (such as .zip) are not supported; extract or convert them first, then compare the files inside.
Troubleshooting / FAQ
"File has an unsupported format." Convert the file to one of the supported formats, or extract an archive and compare the files inside.
"File appears to be image-only." The PDF has no extractable text, usually because it is a scan. Compare a text-based version instead.
"File is protected and cannot be read." Supply a copy without password protection.
"File exceeds the 20 MB size limit" or "supports PDF files up to 100 pages." Compare a smaller extract, or split the document and compare the relevant part.
A value that should match is reported as different. Check that File 1 and File 2 are assigned the way your Guidance describes them (File 1 = expected or template, File 2 = actual or generated).
A correctly filled placeholder is reported as a difference. State in the Guidance that File 1 is a template and File 2 is the document generated from it.
The step reports that the files describe different records. They do not correspond at all, so check which file the earlier step captured into File 2.