DiffEdit: An Automated Diffusion-Based Framework for Reference-Driven Semantic Image Editing

DiffEdit: 레퍼런스 기반의 의미론적 이미지 편집을 위한 자동화된 디퓨전 기반 프레임워크

Abstract

The paper proposes Diffusion-Edit (DiffEdit), an advanced automated diffusion-based framework for reference-driven semantic image editing. It simplifies background removal and object replacement by automatically generating shape masks and using reference images as guidance. Traditional editing methods often require tedious manual adjustments or multiple specialized models. In contrast, DiffEdit leverages the generative capabilities of pre-trained diffusion models to create high-fidelity, context-aware edits within a single image. The framework automatically removes target regions (e.g., backgrounds or objects) by generating a precise shape mask from diffusion-driven discrepancy maps, without user annotation. It integrates reference images into the masked regions by applying cross-attention mechanisms and context-aware conditioning to ensure the inserted content is semantically and stylistically consistent with the original scene. Across our evaluation sets, DiffEdit attains the highest QS (editing quality), highest CLIP (reference alignment), and lowest FID (realism) among the compared methods, indicating consistent gains in realism, coherence, and controllability for object replacement. We designed the framework to extend to background correction and texture editing; we leave systematic validation of these tasks to future work. This method not only improves semantic image editing efficiency but also allows for adaptive, user-guided generative modeling.

1. Introduction

Photography is now embedded in everyday life, and many users wish to remove distracting backgrounds or objects to improve visual appeal. Beyond such cleanup, semantic image editing seeks to modify high‑level content while preserving realism. Prior efforts span GAN inversion/manipulation [1], [2], [3] and latent‑space factorization/tuning [4], [5], [6], but often remain confined to specific domains.

The advent of large‑scale text‑to‑image models—both autoregressive [10], [11] and diffusion‑based [12], [13], including guided/photorealistic variants [14], [15]—has enabled strong text‑guided editing [16], [17] and subject‑driven generation [18], [19]. However, text prompts can be ambiguous and frequently underspecify fine visual details, making precise control difficult, especially for novice users and non‑native speakers. This motivates image‑based references as a clearer, fine‑grained channel for editing intent.

This paper introduces a self‑supervised, reference‑driven diffusion framework in which semantic modifications are guided by a user‑provided or retrieved reference image. Crucially, our goal differs from image harmonization [20], [21], which primarily adjusts color/illumination during compositing; we tackle the harder problem of semantically transforming structure, pose, or viewpoint to maintain contextual coherence in the final image. To this end, we leverage a pre‑trained diffusion model and adapt it for reference‑conditioned editing [22], [23]. Collecting sufficient triplets (source, reference, ground truth) is impractical, and naïve self‑reference training tends to devolve into copy‑and‑paste behavior. We address this with three key innovations: (i) initialization from a strong text‑to‑image prior to avoid trivial solutions; (ii) retaining global embeddings while discarding fine‑grained spatial tokens during conditioning to preserve high‑level semantics and curb overfitting; and (iii) comprehensive augmentation of self‑reference data to bridge the train–test domain gap.

We further improve editability with two design decisions: (1) training with irregular random masks to emulate real user brush patterns, and (2) applying classifier‑free guidance to boost output quality and style fidelity to the exemplar [24]. Unlike image‑specific optimization approaches such as Null‑Text Inversion [36] and Imagic [19], our method performs edits without per‑image optimization, producing results in a single diffusion‑model pass—an advantage for real‑world use.

We summarize our contributions as follows:

• We propose an automated, reference‑driven framework for semantic image editing that offers fine‑grained control with simple user interaction.

• Our approach employs self‑supervised training that avoids large curated datasets by using the input itself as reference.

• User‑effort reduction: DiffEdit removes manual mask drawing via self‑guided mask inference, avoids prompt engineering via automatic captioning from the reference image, and eliminates per‑image optimization through a single‑pass diffusion pipeline with reference‑aware blending; users provide only source and reference images.

2. Background Study

2.1. Text-Guided Image Generation

Recent advances of denoising diffusion probabilistic models (DDPMs) have brought about in an entirely different phase of generative picture synthesis, allowing for the production of high-fidelity images from textual descriptions. Diffusion models, established by Ho et al. [22], continuously reduce noise into structured pictures via, which learn reversed of the linear noising process. Subsequent improvements, such as Rombach et al.'s Latent Diffusion Models (LDMs), increased computing efficiency by working in a compacted latent space while retaining visual quality [25]. Text-guided generation relies heavily on classifier-free guidance [24], which enables conditional synthesis without the need of additional classifiers by randomly deleting text inputs during training. Despite their expertise in picture production, applying these models for particular semantic editing where certain properties of an existing image must be changed however no relevant information is preserved remains an unresolved issue. Diffusion processes are naturally unpredictable global, complicating localized, context-aware adjustments. In this paper, we focus on semantic picture composition while taking the hard semantic inharmony for consideration as well.

2.2. Diffusion-Based Editing Approaches

Early attempts to reproduce diffusion models for editing sometimes compromised precision or efficiency. Recent works of SDEdit [14] uses stochastic differential equations to indicate changes by adding noise to an input picture and finally denoising. This is based on the destination text. While SDEdit seems wonderful over style transfer, and it sometimes modifies non-target regions because of global noise contamination and struggles with complicated semantic alterations (e.g., object replacement). A more recent work Prompt-to-Prompt (P2P) [17] provides tighter control by altering cross-attention patterns during DDIM inversion [37], allowing for localised modifications. However, P2P requires manual attention weight modification and fails for object substitutions (e.g., monkey change to dog) such as it relies on similarities in structure between the source and destination ideas of it. Inversion-based techniques, such as Null-Text Inversion [36] and Imagic [19], address these limitations by optimizing undetectable patterns or model weights on an image-by-image basis. Null-Text Inversion [36] optimizes null-text embeddings to match inversion progressions to target prompts, while Imagic [19] utilizes a two-step optimizing approach for real-image editing. Both solutions need large computer resources per image, which limits them to apply in real-world applications. In contrast, our DiffEdit framework automates the editing process, eliminating the need for manual intervention. It uses a reference image for smooth object replacement and works efficiently in a single step, without needing to adjust per image.

2.3. Mask-Guided Editing Techniques

Mask-based techniques segregate editable parts to reduce unintentional modifications. According to the works of GAN Dissection [35] and mask-guided attention models [34] employ external segmentation techniques, such as Panoptic FPN [33] or user-supplied masks to establish edit areas. The Blended Diffusion [16] combines text-driven transfer from relative masking by ignoring just specific locations and merging the results with the original picture. While enhancing localization, these approaches have serious flaws. Firstly, the manual mask generation is time-consuming and difficult for large of the pictures. Secondly, the automated techniques for segmentation [33] cannot be generalized well outside their training domains. Thirdly, all the boundary materials appear because of unclear blending, particularly when structural mismatches the situation between modified and remaining regions. As a result, mask reliance creates flexibility difficulties and reduces output coherence. Our DiffEdit framework solves this by automatically creating masks using the built-in features of a diffusion model. It uses a reference-driven inpainting process to make sure the mask blends smoothly and naturally.

2.4. Semantic Image Editing Reinvented

DiffEdit discusses the primary constraints of prior accomplishments by introducing multiple significant breakthroughs that together provide an alternative model for semantic picture editing. First, it provides a self-supervised masking technique that automatically detects sections that require to have modified by taking implementation of diffusion models' natural text sensitivity. This is accomplished by comparing each noise predictions from a pre-trained diffusion model dependent on a reference material (describing the original picture) and a query the material (describing the target modification). Besides, the difference between these predictions identifies editable zones that do not require masks, segmentation models, or user input, also eliminating the manual mask dependence that must be affect approaches such as Blended Diffusion [16] and GAN Dissection [35]. Second, unlike Null-Text Inversion [36] and Imagic [19], DiffEdit does not require any per-image optimization at inference. It must requires any adjusting for each picture. By dynamically exploiting the generative prior of pre-trained diffusion models, it provides a large computing savings while retaining zero-shot adaptivity. Third, the approach utilizes undetectable fixing to ensure context picture preservation. During the condition generation phase, the latent representation of the real photograph restricts the denoising process beyond the mask inferred. This guarantees of pixel-level integrity for untampered areas without SDEdit's global artifacts [14] or Prompt-to-Prompt's structural incoherence [17]. Our paper proposes that DiffEdit is a unique framework that combines automated mask inference, training-free execution, and context-preserving generation. This allows for complex picture editing tasks such as object replacement, feature modification. DiffEdit outperforms existing approaches for realism, textual alignment, and content retention.

3. Our Framework

We tackle exemplar‑based image editing that automatically integrates a reference image (retrieved from a database or provided by the user) into a source image so that the composite is perceptually convincing and photorealistic. Despite recent progress in text‑based image manipulation, expressing complex, multi‑attribute edits purely with language remains challenging. Images, by contrast, can convey editing intent more precisely—a picture is worth a thousand words.

Formally, let the source image be x_s∈ ℝ^{H x W × 3} where H and W denote height and width. The editable region is specified by a binary mask m ∈ {0, 1)^{H x W} that may be either a regular box or an arbitrary connected shape. By convention, m = 1 indicates pixels to be modified in x_s. Given a reference image x_r ∈ ℝ^{Hl x wl × 3} containing the desired object(s), our goal is to produce y = f(x_s, x_r, m). We require y to satisfy two criteria: (i) y ≈ x_s on pixels where m = 0 content preservation), and (ii) within m = 1, the inserted content matches the reference x_r while blending seamlessly with the surrounding context. This objective is challenging because it entails several non‑trivial subproblems. First, the model must localize and encode the target object in the reference image, capturing its shape and texture while suppressing background distractors. Second, it must synthesize a plausible transformation of that object (pose, scale, viewpoint, illumination) that is consistent with the source scene. Third, the model must inpaint the boundary region to ensure a smooth transition between edited and untouched areas. Finally, the reference image may be lower resolution than the target region; in practice we mitigate this by resizing inputs to a common resolution (512×512) rather than introducing a separate super‑resolution stage.

3.1. Self-Guided Mask Generation

Our solution employs self‑guided mask generation to automatically identify editable regions, reducing the need for manual annotation. First, we generate both a target prompt c_target from the reference x_r and a source prompt c_source from the source x_s using a A captioning model (e.g., BLIP); CLIP [29] is used only for CLIP‑Score; we then compute a noise‑difference heatmap across diffusion timesteps to obtain a binary self‑guided mask without supervision.

These prompts provide semantic guidance for subsequent steps: we add noise to the input image to obtain the latent z_t at diffusion step t. Next, we compute the absolute difference ∆ϵ between two noise predictions: one conditioned on c_target and the other conditioned on c_source. We aggregate ∆ϵ over T timesteps (averaging across t) to produce a heatmap that highlights regions responseive to the target modification. Thresholding this heatmap yields the self‑guided mask m ∈ {0, 1}^{H × W} that separates pixels to be modified. This procedure exploits the intrinsic sensitivity of diffusion models to conditioning to localize modifications without supervision. We need to aligned Adapting reference features

Q = ϕ_q(z_t) ∈ ℝ^{HW × d},

K = ϕ_k(f_r) ∈ ℝ^{H_rW_r × d}

, V = ϕ_v(f_r) ∈ ℝ^{H_rW_r × d}       (1)

(f_r) to z_t via across-attention block (1):

where ϕ_q, ϕ_k, ϕ_v are 1 × 1 linear projections with positional encoding. Q, K, V are derived from z_t and f_r. We compute (2):

\(\begin{align}A=\operatorname{softmax}\left(\frac{Q K^{T}}{\sqrt{d}}\right) \in \mathbb{R}^{H W \times H_{r} W_{r}}\end{align}\),

U = reshape (AV) ∈ ℝ^{C_u× H × W}       (2)

Attention weights A determine relevance and U realigns reference features f_r to match the spatial dimensions of z_t. The aligned feature is combined with the latent using a gated residual (3):

F(z_t, f_r) = g([z_t, U]) + γ. U       (3)

where g is a 1 × 1 convolution mapping [z_t, U] back to ℝ^{C × H × W} and γ is a learnable scalar.

3.2. Reference-Aware Latent Blending

To incorporate visual features from a reference images x_r while maintaining context, we utilize reference-aware neural blending. The diffusion model's encoder converts the reference picture to position characteristics f_r. During every denoising phase t, these characteristics are aligned with the noisy latent z_t using cross-attention. We also selectively merge f_r into z_t entirely inside the self-guided mask area M, resulting in an altered latent z^edit_t

z^edit_t = M ⊙ F(z_t, f_r) + (1 - M) ⊙ z_t       (4)

using operation (4):

F signifies feature alignment, while ⊙ represents element-wise multiplication. This spatially limited integration provides that reference properties are smoothly integrated while the background information continue uncovered.

3.3. Mask-Constrained Diffusion Sampling

We generate the final image using mask‑constrained diffusion sampling, a modified reverse process with region‑specific conditioning. Specifically, at each step t, we compute two noise predictions: one conditioned on c_target for masked regions and another conditioned on c_source for unmasked regions. Then, these predictions are integrated

ϵ_t = ϵ₀(z^edit_t, c_target, t) . M

+ ϵ₀(z_t, c_source, t) . (1 - M)       (5)

with the self-guided mask (5):

To balance edit fidelity and realism, ϵ_t is guided without using classifiers. The blended noise prediction then uses standard diffusion sampling with the aim of update z_t - 1. This dual-conditioning framework ensures coherent modifications narrowly within the M, while maintaining integrity in unaffected regions.

4. Experiments

4.1. Implementation

To facilitate real-world image editing, we construct our system using Stable Diffusion v1.5 [25], and utilize its universal generative priors. It also learns from the OpenImages V6 dataset [20], comprising 5,000 images with 5,000 bounding boxes for 600 object classes. Unless otherwise specified, we do not fine-tune per image at test time, we resized every input image to resolution 512×512. The model has two NVIDIA GeForce RTX 4090 GPUs, each 24GB VRAM, an NVIDIA driver x86_64 framework processor with 16 physical cores, and 128GB system RAM. To ensure high speed data, the experimental environment was set up with Python 3.10.16 and with CUDA acceleration on a generic Linux OS. It provides sufficient resources for optimal model optimization and it also ensures complete repeatability of the experimentation process.

Implementation details for captioning and mask inference. We resize inputs to 512×512. We average the noise‑difference heatmap over T diffusion timesteps, min–max normalize it, and binarize with threshold τ to obtain the self‑guided mask. Unless stated otherwise, we set the classifier‑free guidance scale to λ=8 (see Fig. 2). We will release code and configs for reproducibility.

4.2. Experiment Result

Our target is to combine a reference image into an original image, with the editing part matching the reference and the resulting outcome being image realistic. To assess these two features independently, we employ the three metrics listed below to analyze the created images. First, the FID [26] score is commonly utilized to assess created results. For FID (lower is better), we use standard Inception‑V3 features following [26]. For CLIP similarity (higher is better), we use the CLIP image encoder [29] to compute cosine similarity between the edited region and the reference image. We evaluate 3,500 generated images against the COCO test set. Second, Quality Score (QS) [28] evaluates the validity of each individual image. We need to take the average to assess the overall quality of the generated images. Third, use the CLIP score [29] to assess the resemblance between the altered region and the reference image. We scale these two images to 512 × 512, extract features using CLIP image encoder, and calculate the cosine similarity. Higher CLIP scores imply that the altered region is more comparable to the reference image.

Figure 1 shows DiffEdit's semantic editing process for object replacements. Given an input image of a monkey near the lake, our framework builds an autonomous certainty mask that correctly localizes the target item for removal without the needing from the user identifying them. The system combines this of the target item to conduct latent-space blending. The output clearly indicates full integration with the new item maintaining background coherence using DiffEdit's noise reinitialization process while matching the visual qualities of the reference.

Fig. 1. Using a reference and mask to remove an object from the background and subsequently replacing the removed object with the reference.

Ablating the classifier-free guidance scale (λ) shows that larger λ increases adherence to the reference (Figure 2). We use λ=8 by default.

Fig. 2. Shows the result of classifier-free guiding scale. A bigger increases the created part similarity to the reference.

4.3. Comparisons

Previous work at this level did not aim to perform semantic or local edits based on a reference image. Unlike our method being able to provide the same masks, prompts and reference inputs. Therefore, we selected four of the comparable methods to be the baselines for our method. 1) Blended Diffusion [16] utilizes the official implementation with CLIP guidance optimized for reference-driven methods that accept an example image as inpuxt, and text-only methods are reported separately. CLIP loss is calculated using both the text prompt and the reference image embeddings. 2) We slight modify the Blended Diffusion by we include a variant using only text prompts without needing reference image guidance. 3) Stable Diffusion [25], we use the standard inpainting procedure along with mask and reference image conditioning, bypassing intermediate rendering steps. 4) We compare directly against this harmonization method DCCF [31] as the baseline. Because DCCF expects a foreground mask and compositing, we first inpaint the target region with the unconditional LAMA model [32], then extract the foreground from the reference via a semantic mask, and finally composite it into the source image. Note that DCCF is an image harmonization method, not an end-to-end reference-based editing framework. To create a comparable baseline, we designed a pipeline where the target area is first inpainted using LAMA, and then the foreground from the reference image is composited using DCCF. While this multi-step process may not represent the optimal use of each individual component, it was a necessary adaptation to compare against a state-of-the-art harmonization technique within our editing task.

4.4. Qualitative

Figure 1 shows our qualitative evaluation of existing techniques that indicates important shortcomings. Text-guided Blended Diffusion places things at the correct area, however they are unrealistic and interfere with the source. Stable Diffusion increases realism however sacrificing commitment to the reference image, which is restricted by text's limited capacity to indicate the visual information. Image-guided Blended Diffusion fails to match the reference, potentially due to poor content preservation in neural gradient guiding. The image showing the unification results between source and reference is severely mismatched. It does not require adaptation of form, size, or posture to match the background. As demonstrated, our technique produces an image-realistic output for object replacement that closely matches the reference picture.

Table 1 shows the quantitative comparative results. Image‑based methods such as Blended Diffusion and DCCF tend to achieve high CLIP scores, indicating that conditioning on image information is preserved. However, they provide perceptually compared outcomes show of the high FID and low QS ratings score. In contrast, Stable Diffusion produces convincing that high-quality pictures as show by higher QS metrics. As summarized in Table 1, DiffEdit achieves FID 1.75 and QS 93.49, outperforming Blended Diffusion (image/text), DCCF, and Stable Diffusion across realism and perceptual quality; CLIP similarity is also highest (88.64, measured on edited‑region vs. reference), indicating strong reference alignment. These results indicate closer resemblance to the reference while maintaining high image quality, demonstrating robust preservation of conditional information.

Table 1. The result comparison with other methods. We measure the quality of the produced picture using FID and QS, as well as the semantic coherence with the reference picture using the CLIP score.

5. Limitations

While DiffEdit demonstrates strong performance in reference-driven object replacement, it has several limitations. First, the quality of the edit is highly dependent on the text prompt automatically generated by the CLIP-based captioning model. This model is responsible for the critical first step of generating a descriptive text prompt for the reference image. If this model fails to capture the precise semantic essence of the reference image for instance, misidentifying an object or missing key contextual details, the subsequent mask generation and diffusion process can be misguided from the commencement, leading to incorrect or incomplete edits. Second, our framework may struggle when there are extreme differences in lighting, view or scale between the source and reference images, which can lead to unrealistic blending. Finally, adjusting highly detailed repetitive textures remains a challenge, as our method prioritizes overall semantic coherence over pixel-perfect texture mapping. Future work will be required to overcome these challenges in order to make the approach more robust and generalizable.

6. Conclusion

The paper introduces diffusion-edit (DiffEdit) is a automated diffusion-based semantic image modifying framework that generates faithful, high-quality progress without requiring large labeled datasets or task-specific training. DiffEdit synthesizes a mask specifying parts to be modified based on reference images. It achieves the noise prediction behavior of diffusion models. The automated approach of the method provides comprehensive and consistent editing without altering irrelevant contents in the pictures. The experimental results also show that the DiffEdit surpasses state-of-the-art methods in qualitative realism and quantitative measurements. Furthermore, it comprehends high computing efficiency and scalability. This paper achieves an advance in the state-of-the-art in semantic picture editing and shows the effectiveness of pre-trained diffusion models for generation tasks.