Skip to content

A/B tests on Meta Ads with AdPilot

Using AdPilotUpdated 17 Nov 20264 min read

Cover image: two simplified ad set cards, A and B, each with a half-filled share bar, a label bar above and an approval-gate tick below
Contents
  1. What an A/B test is here
  2. How to do it
  3. The proposal you see when approving
  4. An ad set can’t be in two active tests
  5. Undo
  6. Where to see the results
  7. What AdPilot doesn’t do yet
  8. Checklist

On the Campaigns screen, the “A/B test” button opens a panel for creating a test from the ad sets you’ve ticked. You can also ask Studio or Claude to do it through AdPilot. Either way, you get a proposal to approve on the web. This article covers what an A/B test is here, how to create one, what you approve, how to undo it, and what AdPilot doesn’t do yet.

1. What an A/B test is here

AdPilot creates a real A/B test on Meta between 2 and 5 ad sets that already exist in the same ad account. AdPilot doesn’t create new ad sets for the test; you pick existing ones.

Two ads in the same ad set are not an A/B test. AdPilot doesn’t call them one and doesn’t compare their results as if they were.

AdPilot reminds you to test one variable at a time: the ad sets should differ in only one thing.

2. How to do it

On the web: open the Campaigns screen, go to the “Ad sets” tab, tick 2 to 5 ad sets and click “A/B test”. The “Create A/B test” panel asks for the test name, each ad set’s share (%), and the start and end times in the ad account’s time zone; click “Build proposal”. Leave every share empty to split evenly. If your level is “View only”, the “A/B test” button is turned off and the screen says: “At the read-only level, you can’t create A/B tests in AdPilot — create them in Ads Manager.” To ask an AI instead:

  1. Open Studio on the ALFA web app, or ask Claude with AdPilot connected (how to connect). You need to be allowed to create proposals; proposals can still be created for an account that was just linked in read-only mode (roles and permissions).
  2. Say what you want. Example: “Create an A/B test for client X between ad set Y and ad set Z, running for 7 days from tomorrow”.
  3. The AI needs to know:
    • the ad account;
    • 2 to 5 ad sets (ask the AI to list the account’s ad sets if you don’t remember the names);
    • a name for the test, up to 100 characters;
    • each ad set’s share, if you don’t want equal shares;
    • the start time and the end time.
  4. The AI builds a proposal. The test is only created on Meta after the proposal is approved and applied.

AdPilot checks these rules when it builds the proposal, and says so in words if one is broken:

  • 2 to 5 ad sets, each one once.
  • Each ad set gets at least 10%, and the shares add up to 100% at most. If you give no shares, they’re equal.
  • The start time isn’t in the past; the end time is after the start time. Set the start far enough ahead to leave time to approve, because a proposal lasts 24 hours.
  • The ad account must belong to a Business Manager that AdPilot can read. If not, the AI says AdPilot couldn’t read the Business Manager of that account; check whether the account belongs to a Business Manager (what a Business Manager is).

3. The proposal you see when approving

The proposal is called “Proposal to create an A/B test” and is high risk: the organisation owner or an approver has to open the proposal page on the web and approve it; the AI can’t approve for you (how to approve).

The proposal page shows:

  • the test name, the ad sets and each one’s share, the start date and the end date;
  • the “Pre-check” section. Two warnings you may see; an approver can still go ahead once they’ve looked:
    • “The ad sets use different performance goals.” With a reminder to test one variable at a time.
    • “Ad set “…” is running.” With a reminder to review that ad set before approving.
  • the risk note: an A/B test uses the account’s existing ad sets; while the test is active, those ad sets can’t join another test; AdPilot can only delete a test that is at least 10 minutes from starting.

4. An ad set can’t be in two active tests

Meta doesn’t let an ad set be in two active tests at the same time. AdPilot can’t tell beforehand which ad set is already in a test. If one is, Meta refuses when the proposal is applied, and the proposal page shows Meta’s refusal. Then pick another ad set or wait for the other test to finish, and ask the AI to build a new proposal.

5. Undo

An applied proposal has an “Undo” block like any other. For an A/B test, undo deletes the test:

  • it only works while the test is at least 10 minutes from starting;
  • if the test has started or starts within 10 minutes, AdPilot doesn’t delete it; to stop it, go to Ads Manager;
  • once deleted, AdPilot can’t restore it;
  • undo is also high risk: the organisation owner or an approver clicks it on the web;
  • if the test has changed on Meta since the proposal was created, AdPilot doesn’t delete it and says which field changed.

6. Where to see the results

AdPilot doesn’t have a screen or a tool that reads an A/B test’s results yet, and doesn’t calculate them. To see the results or stop the test, go to Ads Manager.

7. What AdPilot doesn’t do yet

  • It doesn’t create new ad sets for a test; it only uses existing ones.
  • It doesn’t stop a running test.

Checklist

  • 2 to 5 existing ad sets in one account, differing in one thing only
  • Each share at least 10%, total at most 100%
  • Start time far enough ahead to approve; end time after the start
  • The approver reads “Pre-check” (different goals, running ad set) before clicking
  • To back out: undo at least 10 minutes before the start
  • See results on Meta, not in AdPilot

Still unsure? Message the support bot privately.