Explainer: User Agent Image Replacement API

This proposal is an early design sketch by an AI team in Chrome to describe the problem below and solicit feedback on the proposed solution. It has not been approved to ship in Chrome.

• Jeremy Roman (Google LLC)

• explainers-by-googlers/ua-image-replacement (GitHub)

• Introduction
• User-Facing Problem
  • Goals
  • Non-goals
• Proposed Approach
  • Example
• Alternatives considered
  • CSS Pseudo-class only (:replaced-by-user-agent)
• Developer Opt-in and Opt-out
• Privacy & Security Considerations
• Accessibility Considerations

Modern browsers can provide capabilities to augment the browsing experience by modifying media in the page on behalf of the user. The advent of generative AI makes it more likely that browsers will add such features.

Sometimes, the replacement content added at the user request might not match other content and functionality in the page, which could confuse the user. Even if this cannot be completely avoided, if authors can observe when replacement happens they can adjust the document to mitigate confusion (e.g., by hiding or adjusting other content).

For example, a user browsing an e-commerce site with a generic product image (e.g., a model wearing a jacket) might wish to imagine themselves wearing the item. The user agent uses generative AI technology to produce that image and present it in place of the model image. The page improves the user experience by removing text referring to the model's dimensions and the garment size depicted, as it may not be correct in the replacement image.

When a user uses a browser feature that affects media in the page, it's sometimes confusing whether other page content is appropriate for the original media, replacement media, or both. While user agents should make an effort to make the distinction between UA- and author-provided content clear, users get the best experience when both coordinate to present a harmonious experience in which this confusion is kept to a minimum.

A Chrome feature which offers image replacement has heard feedback from multiple websites that this capability would enable them to both avoid a confusing user experience and improve their own analytics to understand how users of these new features interact with their site (for example, whether they are more likely to browse or make a purchase).

• Simplicity: An API which is simple, easy to adopt, and naturally degrades to existing behavior is going to have both a low barrier for adoption and minimize compatibility risk.
• Privacy: User agents should be empowered to limit the information available to the page in accordance with their own stance and user preferences. User agents can continue to withhold the fact that an image is replaced if this is sensitive, or surface only limited information about the nature of the replacement content, to reduce the exposure of additional information to the site.

• Replacement media access: This API does not intend to make replacement content directly available to the page (especially it may reflect user data or intent), and user agents are encouraged to replace images in a way that protects against this where feasible.
• Content-driven replacement: This API does not propose to allow authors to initiate such replacements of any kind.
• Enumeration of possible image replacement features: A future enhancement could provide information about what kind of replacement image is being used, but since this may have privacy and compatibility risk implications, it's left out of scope for now. It's likely that, if this information is added, it will be optional.

We propose extending HTMLImageElement with two new DOM events and one read-only boolean attribute.

1. uareplacestart event: Fired when the user agent begins substituting the image content.
2. uareplaceend event: Fired when the substitution finishes, fails, or is reverted.
3. replacedByUserAgent boolean property: A read-only attribute on HTMLImageElement indicating whether the element currently displays substitute content provided by the user agent.

When image replacement occurs, what changes to the document cause image replacement to end, and whether pages are notified or this information is withheld from the page is user agent (and even feature) specific behavior and not covered here.

When an e-commerce clothing page renders a product image, it often attaches interactive overlays (such as a zoom viewer which uses other content to render an enlarged version, and would confuse the user if the original image reappeared). When the user agent starts replacing the product image with a virtual try-on render of the user, the page can remove this control.

const imageEl = document.querySelector('img.product-view');
const zoomLens = document.querySelector('.zoom-lens');

// Listen for replacement start
imageEl.addEventListener('uareplacestart', () => {
  // Temporarily disable custom interactive zoom while substitution is in progress
  zoomLens.style.display = 'none';
});

// Listen for replacement end
imageEl.addEventListener('uareplaceend', () => {
  // Re-enable interactive zoom if replacement was reverted or canceled
  zoomLens.style.display = 'block';
});

// Check if the replacement happened before this script executed.
if (imageEl.replacedByUserAgent) {
  zoomLens.style.display = 'none';
}

Sites can adjust style by adding attributes, classes, etc. when these events fire.

A site can track which elements are replaced by using a capturing listener on the document for when replacements start, and then observing when replacement ends on each such element.

While would be useful for making style changes, it is not ergonomic to monitor from script for other changes. This is considered as a possible future enhancement.

While this explainer focuses on observation primitives (uareplacestart, uareplaceend, and replacedByUserAgent), it's reasonable for developers to prefer that such features not be used, or possibly be used, on their sites, because it is impractical to adapt the site when the user uses them without significantly degrading functionality or the user's experience. The browser might then not offer that feature, or offer a different experience (e.g., in a side panel or pop-up window). Deciding on a mechanism for developer hints is out of scope for this initial proposal, but is an area for potential future work.

Future work in this direction should consider this sort of case and might want to establish a registry of different kinds of transformations that sites could hint about. For example, it might look something like <html notransform="interior-decor-generation"> (or an element-scoped equivalent). Some effort would be required to make sure that vendors with similar features eventually supported interoperable tokens. This is somewhat analogous to the existing translate and autocomplete attributes.

If there is browser vendor and web developer interest, it would be useful at that time to also consider whether it would also be useful for hints to opt in to offering particular features, as a signal to browsers that the feature is likely to be particularly beneficial, especially for browsers that support transformations of this kind but wish to take a more cautious approach to applying them.

• Data Protection: User agents are encouraged to prevent sensitive replacement content from being accessed by the page without explicit user intent.
• Anti-Fingerprinting: uareplacestart and uareplaceend events only fire when an actual replacement operation is initiated by explicit user intent or user-configured browser policy. Web pages MUST NOT be able to probe for replacement capabilities by observing side-effects without the user using such a feature.

• Alt Text and Screen Readers: When a user agent replaces image visual content (e.g., text translation in image or virtual try-on), the UA may override the accessible description (alt text or ARIA attributes) so assistive technologies accurately convey the substitute content to visually impaired users.