How to batch execute variations of your Test Case

This guide walks you through data-driven testing in Thunders: running the same test case multiple times with different data from CSV files, such as different URLs, browsers, credentials, or any variable your test uses.

Overview

With data-driven testing, you upload one or more CSV files as test assets. Each row in a CSV defines a unique test iteration. When you trigger a run from the Thunders Platform or CI, every row becomes its own test run, grouped under a single batch.

This guide covers the four supported execution paths:

Executing one test case with its own linked data file
Executing several test cases, each with its own linked data file
Executing selected test cases with one shared data file
Executing data-driven runs from CI

This is ideal for:

Testing the same flow across multiple browsers or devices
Running a login test with different user credentials
Validating a page across many URLs
Any scenario where you want to parameterize your test case

Step 1: Create Your CSV File

Create a CSV file where each row represents one test variation.

There are two types of columns:

Data Columns

These are columns whose names match the variable placeholders in your test case (without the [] brackets). The values override your test case variables with the highest priority.

For example, if your test step says:

Navigate to [ENVIRONMENT_URL] and log in with [USERNAME]

Your CSV should have columns named ENVIRONMENT_URL and USERNAME:

ENVIRONMENT_URL,USERNAME
https://staging.example.com,alice
https://production.example.com,bob

Native Columns (Browser/Environment Overrides)

Columns prefixed with _ override execution settings per row. Use these to vary the browser, resolution, environment, or other execution parameters across rows.

Column	Description	Example Values
`_rowname`	Display name for the row in results	Chrome Login, Mobile Safari
`_browserType`	Browser engine	Chromium, Chrome, Firefox, Safari
`_resolution`	Viewport size	1920x1080, 375x812
`_deviceType`	Device category	Desktop, Mobile, Tablet
`_deviceName`	Specific device	iPhone 14, Pixel 7
`_location`	Browser geolocation	ParisSelfHosted
`_locale`	Browser language	en-US, fr-FR, de-DE
`_darkTheme`	Dark mode	true, false
`_javascript`	JavaScript enabled	true, false
`_ignoreHttpsErrors`	Skip HTTPS validation	true, false
`_avoidDetection`	Stealth mode	true, false
`_deviceScaleFactor`	Pixel density	1, 2, 3
`_forcedColors`	Forced color mode	active, none
`_username`	HTTP auth username	admin
`_password`	HTTP auth password	secret123
`_proxy`	Proxy server	http://proxy:8080
`_highlightElements`	Highlight actions	true, false
`_environmentId`	Override environment	UUID of the environment
`_personaId`	Override persona	UUID of the persona
`_devicePoolId`	Override the device pool	UUID of the device pool
`_deviceSkuId`	Override the device SKU within the pool	UUID of the device SKU

Any other column starting with _ is rejected and the launch is refused.

Complete Example CSV

Here is a CSV that tests a login flow across 3 browsers with different users:

ENVIRONMENT_URL,USERNAME,PASSWORD,_rowname,_browserType,_resolution
https://app.example.com/login,alice,pass123,Chrome Desktop,Chrome,1920x1080
https://app.example.com/login,bob,pass456,Firefox Desktop,Firefox,1440x900
https://app.example.com/login,charlie,pass789,Safari Mobile,Safari,375x812

This produces 3 test runs:

Chrome Desktop: logs in as alice on Chrome at 1920x1080
Firefox Desktop: logs in as bob on Firefox at 1440x900
Safari Mobile: logs in as charlie on Safari at 375x812

Step 2: Upload the CSV as a Test Asset

Upload your CSV file to your project in the Thunders web app:

Open your Project
Go to the Test Assets tab
Click Upload and select your CSV file
Note the generated reference name displayed in the assets list (e.g., FILE_LOGINS_CSV)

The reference name follows the pattern: FILE__

The filename is uppercased
Hyphens, spaces, dots, and parentheses become underscores
Multiple consecutive underscores are collapsed to one

Examples:

logins.csv becomes FILE_LOGINS_CSV
test-data.csv becomes FILE_TEST_DATA_CSV
my users (v2).csv becomes FILE_MY_USERS_V2_CSV

Link the CSV to a test case

A test case can be linked to one of the project's CSVs. Set the link once and that test case always runs against that file, one run per row. Set it from either of these; both open the same picker: the project's CSV test assets, each with its row count, plus Upload new to add a CSV and assign it in one go.

From the test case: Open the test case and use the Data file field in the Properties panel.
From the Test Cases list: Use the Data file column to set, change or Clear the file on any row. This is the fast way to set up many test cases without opening each one.

Only .csv test assets can be assigned. A file blocks the launch when it has been deleted, has no data rows, has more than 1,000 rows, or when its row count cannot be read. The pill turns red in that state. A launch is then refused as a whole.

Step 3: Run the Batch

How you launch depends on where the file lives: linked to each test case, or chosen once for the whole launch.

3.1) One test case, with its own data file

Open the test case: Execute is a split button. The primary half always starts one run with no data, even when a file is linked. The chevron opens Execute with data file, which shows the run count and starts one run per row.

When a file is linked, the dropdown shows it as a pill; click the pill to select a different file without leaving the Execute control. It opens the same picker as the Properties panel and the Test Cases list, so whatever you select here becomes the test case's link.

With no file linked that row reads Select and opens the picker instead of launching.

Rows run in parallel up to your plan's limit (2 concurrent runs on Starter, 10 on paid plans), or the row count, whichever is smaller.

3.2) Several test cases, each with its own data file

Tick the test cases in the Test Cases list and click Execute, then leave Test data on Default. Every selected test case that has a linked data file runs once per row of its own file; the ones with no file run once with no data.

A Test Set works the same way on Default: every member test case that has a linked file runs once per row of its own file, and each one fans out within a single pass of the set.

The set's prerequisite is the exception. It runs once, even when it has a linked file. A prerequisite is a gate rather than a scenario: it exists to put the environment into the state the rest of the set needs, so repeating it per data row would redo that setup for nothing and multiply the size of the whole launch.

3.3) Use one shared data file for selected test cases

Select “From data file” in the “Test data” section when every selected test case should use the same CSV for this launch.

A file chosen here applies to every selected test case and supersedes any file those test cases remember.

3.4) From a CI pipeline

Pipelines launch through POST /api/ci/run, which supports both models. Pass testAssetReferences to apply one file to everything the call selects, the equivalent of 3.3:

{
  "ProjectId": "c8e34ec4-2464-43c7-8db8-1b3a47a22337",
  "TestCaseIds": [
    "a836fadc-377a-46fe-96b0-21f37c626bf9"
  ],
  "EnvironmentId": "eca24252-e566-40a8-b2b0-707b7efa85d8",
  "PersonaId": "70d0ac52-fe94-4d89-aba9-8ef28dd8c04c",
  "BrowserSettings": {
    "Location": "ParisSelfHosted",
    "browserType": "Chromium",
    "DeviceType": "Desktop",
    "Resolution": "1440x900"
  },
  "TestAssetReferences": ["FILE_LOGINS_CSV"]
}

The BrowserSettings in the request body serve as defaults. Any native column in the CSV (like _browserType) overrides the corresponding default for that specific row.

References can also be wrapped in brackets: [FILE_LOGINS_CSV].

To have every test case use its own linked file instead (the equivalent of 3.2, one CI call rather than one per test case), send applyRememberedDataFiles and no file reference:

{
  "projectId": "c8e34ec4-2464-43c7-8db8-1b3a47a22337",
  "testCaseIds": ["a836fadc-377a-46fe-96b0-21f37c626bf9"],
  "testSetIds": [],
  "environmentId": "eca24252-e566-40a8-b2b0-707b7efa85d8",
  "personaId": "70d0ac52-fe94-4d89-aba9-8ef28dd8c04c",
  "applyRememberedDataFiles": true
}

Things to keep in mind:

It is opt-in and off by default. A pipeline that does not send the flag behaves exactly as it does today, so no existing run suddenly multiplies.
It is ignored, silently, when the request already carries its own data. If the same call also passes testAssetReferences (or variables), that data wins: every selected test case runs against the file you passed, no linked file is consulted.
It covers loose test cases and test sets in the same request (testCaseIds and testSetIds). The 1,000-run ceiling and the whole-launch refusal apply to everything the call selects, taken together.

Headers

Header	Value
`Authorization`	`Bearer YOUR_THUNDER_TEST_TOKEN`
`X-MS-API-ROLE`	`M2M`
`Content-Type`	`application/json`

Step 4: Combine Multiple CSV Files (Optional)

You can reference multiple CSV files in a single request. Their rows are concatenated into one batch with continuous indexing.

"TestAssetReferences": 
["FILE_DESKTOP_BROWSERS_CSV", "FILE_MOBILE_DEVICES_CSV"]

If desktop-browsers.csv has 3 rows and mobile-devices.csv has 2 rows, the resulting batch will contain 5 test runs indexed 1 through 5.

This is useful for organizing your test data into logical groups (e.g., one CSV per device category, one per region, one per user role).

Step 5: Review Results

After execution, each row appears as a separate test run in the Test Runs tab. You can identify them by:

Batch Row Name: The _rowname column value (or a default like 1/3, 2/3, 3/3 if no _rowname was provided)
Batch Data Overrides: The variable values that were injected for that specific row

Each run also records the data file it came from. Open the run and its Properties panel shows a read-only Data file pill with the file name and row count, so you can always tell which dataset produced a result.

GitHub Actions Example

Here is a GitHub Actions workflow that triggers a batch of test variations. Add "applyRememberedDataFiles": true and drop TestAssetReferences to have each test case use its own linked file instead.

POST /api/ci/run returns a runId and the run continues asynchronously, so this step only starts it. To wait for the result and fetch the JUnit report, poll GET /api/ci/run/{runId}; the full pattern is in CI/CD.

name: Run Thunders Batch Tests

on:
  workflow_dispatch:

jobs:
  run-batch-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Run Batch Test Variations
        uses: fjogeleit/http-request-action@v1
        with:
          url: 'https://api.thunders.ai/api/ci/run'
          method: 'POST'
          customHeaders: >
            {
              "Authorization": "Bearer ${{ secrets.THUNDER_TEST_TOKEN }}",
              "X-MS-API-ROLE": "M2M",
              "Content-Type": "application/json"
            }
          data: >
            {
              "ProjectId": "c8e34ec4-2464-43c7-8db8-1b3a47a22337",
              "TestCaseIds": ["a836fadc-377a-46fe-96b0-21f37c626bf9"],
              "EnvironmentId": "eca24252-e566-40a8-b2b0-707b7efa85d8",
              "PersonaId": "70d0ac52-fe94-4d89-aba9-8ef28dd8c04c",
              "BrowserSettings": {
                "Location": "ParisSelfHosted",
                "browserType": "Chromium",
                "DeviceType": "Desktop",
                "Resolution": "1440x900"
              },
              "TestAssetReferences": ["FILE_LOGINS_CSV", "FILE_MOBILE_CSV"]
            }

Quick Reference

Concept	Detail
CSV upload	Test Assets tab in your project
Reference format	`FILE__` (uppercased, special chars become `_`)
API parameter	`testAssetReferences` (array of strings)
Native columns	Prefix with `_` to override browser/environment settings
Data columns	No prefix; override test case variable placeholders
Multiple CSVs	Pass multiple references; rows merge sequentially
Batch grouping	All runs share a `batchId`
Row naming	Use `_rowname` column or defaults to `index/total`