Overview

A protocol is a reusable template for experiment setup. It stores the metrics, randomization settings, duration, and statistical analysis plan that your team applies to a type of experiment.

Most teams run the same kinds of experiments repeatedly. A product team measures the same set of metrics for every feature rollout. A search team tests ranking changes with the same statistical method each time. Protocols capture these choices once. Experimenters then select a protocol instead of configuring each setting again.

Use protocols to:

• Reduce the number of decisions an experimenter makes when they create an experiment.
• Apply the same metrics and statistical method across a group of experiments.
• Give teams a set of approved experiment designs to choose from.

Prerequisites

Before you begin, make sure you have:

• Access to the right permissions. The product_analytics_experiments_read permission grants read access. The product_analytics_experiments_write permission grants access to create, edit, publish, archive, and delete protocols.
• A subject type to set the level at which Datadog randomizes experiments.
• (Optional) One or more experiment metrics to set the decision metrics for every experiment.

Create a protocol

To create a protocol:

1. Navigate to Experiments > Protocols in Datadog Product Analytics.
2. Click Create Protocol to open the dialog.
3. Enter a Protocol name. The name must be unique in your organization.
4. (Optional) Enter a Description that explains when to use the protocol.
5. Click Create Protocol.

Datadog creates the protocol as a draft and opens its setup page. Continue to Configure the protocol.

Configure the protocol

The setup page presents six numbered steps. Datadog saves your changes to a draft protocol as you work, so there is no separate save action.

The footer tracks your progress. It displays Required steps complete when the protocol is ready to publish. Until then, it names the steps that still need input, such as Complete steps 1 and 2.

Step 1: Protocol details

Set the identity of the protocol.

1. Review or edit the Protocol name. This field is required.
2. (Optional) Edit the Description.

The Owner field displays the user who created the protocol. This field is read-only.

Step 2: Decision metric defaults

Set the metrics that measure the outcome of experiments created from this protocol.

1. Use the Calculate metrics by dropdown to review or change the subject type. Datadog sets this field to your organization’s default subject type when it creates the protocol. The field cannot be empty.
2. (Optional) Under Primary metric, click Select a primary metric and choose the metric that measures the main outcome. Datadog uses this metric to determine whether an experiment succeeds.
3. (Optional) Under Secondary metrics, click Add secondary metrics to add guardrail metrics. Guardrail metrics check for unintended harm to other areas, such as performance, engagement, or revenue.

The subject type is the only required field in this step. You can publish a protocol that has no primary metric, and experimenters then choose the decision metrics on each experiment. Set a primary metric when you want every experiment created from the protocol to measure the same outcome.

A metric can be either the primary metric or a secondary metric, but not both.

If you change the subject type after you select metrics, Datadog opens the Change subject type? dialog. The dialog lists the metrics that are not compatible with the new subject type. Datadog removes those metrics from the protocol when you confirm.

Step 3: Additional metrics to add to every experiment

Add metrics that Datadog attaches to every experiment in addition to the decision metrics.

1. (Optional) Toggle on Set maximum number of metrics on the experiment to: and enter a positive whole number. Datadog blocks experimenters from exceeding this limit on experiments created from the protocol.
2. (Optional) Add a metric group:
   1. Click Add metric group to open the metric picker, then select the metrics to include. Datadog creates the group after you select the metrics.
   2. Enter a name for the group in the field that appears.
   3. To add more metrics to an existing group, click Add Metrics in that group.
3. (Optional) Repeat to create more groups.

Metric groups organize related metrics on the experiment results page.

Step 4: Randomization defaults

Set how Datadog splits traffic for experiments created from this protocol. The Environment and Targeting rules settings apply to Datadog Feature Flags only.

1. (Optional) Select an Environment from the dropdown. You must select an environment before you can configure targeting rules or traffic exposure.
2. (Optional) Under Targeting rules, add filters to limit the experiment audience by custom attributes, such as user role or subscription tier.
3. Under Traffic exposure, set the percentage of matching users to include in the experiment. You can also define rollout steps to ramp traffic up over time.
4. (Optional) Toggle on Require splitting traffic equally between variants. This records that experiments created from the protocol should use an equal variant split.

Step 5: Duration default

Set how long experiments created from this protocol run.

1. (Optional) Click Add a duration, then enter a number and select a unit. This value becomes the target duration of the experiment. To set a minimum experiment duration, you must first set this default duration.
2. Under Enforcements, set expectations for the duration of the experiment:
   • Toggle on Enforce a minimum experiment duration of, then enter a number and select days or weeks. Set a default duration first, because the minimum duration cannot exceed it.
   • Toggle on Require a duration to start the experiment to record that experiments created from the protocol need a duration.

Datadog locks Require a duration to start the experiment in the on position when you set a minimum duration, or when the confidence interval method requires a duration.

Step 6: Additional setting defaults

Set notifications, the statistical analysis plan, and the properties available for dimensional analysis.

Datadog pre-populates the statistical analysis plan from your organization’s default experiment analysis settings.

Notifications

Under Notifications, use the Recipients dropdown to select who receives notifications. After you add a recipient, the Notify when list appears with all triggers selected. Clear the triggers you do not want, or click Select all to restore them.

The available triggers are:

Datadog copies these notification settings to each experiment created from the protocol, and experimenters can then adjust the recipients.

To remove notifications from the protocol, clear all recipients or clear all triggers.

Statistical analysis plan

For guidance on choosing a method, see Analysis Methods.

1. Select a method from the Confidence interval method dropdown: Sequential, Fixed Sample, Sequential Hybrid, or Bayesian.
2. Select a percentage from the Confidence level dropdown.
3. Toggle Enable CUPED calculation on or off. CUPED uses pre-experiment data from each subject to reduce metric variance.
4. To control the family-wise error rate, toggle on Enable multiple testing correction. For details, see Multiple Testing Correction.
   • This setting is not available when you select the Bayesian method.

Dimensional analysis properties

Under Properties to compute for dimensional analysis, select the properties that experimenters use to filter and compare results. To add a property that is not in the list, type its name and click the option to add it as a custom property. For more information, see Break Out Metrics by Property.

Publish a protocol

Experimenters can select only published protocols. To publish a protocol:

1. Complete the required steps. The footer displays Required steps complete.
2. Click Publish.

Datadog checks that the metrics, subject type, environment, and other references in the protocol still exist. If a reference is missing, publishing fails and Datadog reports the problem.

After publishing, the protocol opens in a read-only view with an Overview tab and an Experiments tab. The Experiments tab lists the experiments that use the protocol.

Create an experiment from a protocol

To apply a protocol to a new experiment:

1. Navigate to Experiments > Experiment List and click Create Experiment.
2. Enter an Experiment name.
3. Under Protocol, select a protocol. The list holds published protocols only. Use the filter field to search by name, description, subject type, or primary metric.
   • To create an experiment without a protocol, select Start from scratch.
4. Review the summary panel, which displays the defaults the protocol applies.
5. Click Create Draft Experiment.

You can also start from the protocol itself. On a published protocol, open the Experiments tab and click Create Experiment from Protocol.

What the protocol applies

When you create an experiment from a protocol, Datadog applies these settings:

Datadog records which protocol an experiment uses. To find these experiments, use the Protocol filter on the experiment list.

Continue to Plan and Launch Experiments to add a feature flag and start the experiment.

Edit a published protocol

To change a published protocol:

1. Open the protocol from the Protocols page.
2. Click Edit.
3. Update the settings.
4. Click Save Changes, or click Cancel to discard your edits.

Your changes apply to experiments created after you save. Existing experiments keep the protocol settings they were created with.

Manage protocols

Open a protocol and use the actions menu. The available actions are Duplicate protocol, Archive protocol, and Delete protocol. The same actions appear as Duplicate, Archive, and Delete in the actions menu on each row of the Protocols list, where Archive appears for published protocols only.

Duplicate a protocol

Duplicating copies the settings into a new draft and opens it. Datadog assigns a default name to the copy; rename it in Step 1: Protocol details if you want a different name. Duplicating is useful when you need a variation of an approved design.

Archive a protocol

Archive a protocol to hide it from the protocol picker so that experimenters cannot select it for new experiments. Experiments that already use the protocol keep their settings and continue to run.

Note: Archiving cannot be undone.

Delete a protocol

Delete a protocol to remove it permanently. Datadog prompts you to confirm before deletion. Deleting frees its name for reuse.

You cannot delete a protocol that experiments reference. Archive it instead.

Further reading

Additional helpful documentation, links, and articles:

Plan and Launch ExperimentsDOCUMENTATION Create Experiment MetricsDOCUMENTATION Analysis MethodsDOCUMENTATION