Xunjian Yin / Research

TOMM 2026Multimodal learning · News captioning

EAMA: Entity-Aware Multimodal Alignment Based Approach for News Image Captioning

Junzhe Zhang, Huixuan Zhang, Xunjian Yin, Xiaojun Wan

Peking University

Paper arXiv

News captions need names and events that an image alone cannot reveal. EAMA teaches a multimodal LLM, InstructBLIP, to find the relevant sentences and entities in the article before writing the caption.

Result
GoodNews CIDEr: EAMA 87.70 vs 83.52 for Xu et al. (2024a), the previous best; on NYTimes800k, 87.00 vs 83.72.
New vs prior
Xu et al. (2024a) train a separate CLIP matcher to build confidence-aware prompts for LLaVA; EAMA trains the captioning MLLM itself to select entity-bearing sentences and entities.
Limitation
Named-entity precision stays below Xu et al. (2024a) on both datasets (GoodNews 27.58 vs 30.19; NYTimes800k 29.79 vs 32.38), although EAMA's entity recall is highest.
Entity-aware alignment prepares the model to extract its own supplementary context before generating a news caption.
Entity-aware alignment prepares the model to extract its own supplementary context before generating a news caption. Full size ↗
01 / The research question

Naming the scene requires evidence beyond the pixels

A generic caption can say that a person is speaking at a podium; a news caption must say who and at which event, and over 96% of captions in common news datasets contain entities (Tran et al., 2020). Many such facts are not visible, and the article also names people who are not pictured.

Multimodal large language models (MLLMs) rarely produce entities zero-shot and stay weak on them after official fine-tuning. InstructBLIP (7B), whose connector compresses the image into 32 tokens and leaves the most room for article text, performs best of those tested and becomes the backbone. Even so, truncating the article can drop the needed name, while the full article can bury it.

Closest prior work

Earlier news captioners either fill templates with entities extracted from the article or, like JoGANIC and ICECAP, read longer inputs or retrieve relevant sentences with a separately trained cross-modal module, leaving the caption model itself unchanged. Xu et al. (2024a) fine-tune CLIP to build confidence-aware prompts for LLaVA, and Qu et al. (2024) align face representations with names, which covers only people. EAMA trains the captioning MLLM itself, from caption data alone, to select entity-bearing sentences and entities.

02 / Inside the method

The auxiliary tasks teach two different kinds of relevance

Entity-Aware Sentence Selection turns caption data into an image–text matching task: article sentences that share a visual entity with the reference caption are positives, and same-article sentences without one are hard negatives, close in topic but differing in entities. Non-visual entities such as dates do not count. Unlike CLIP-style matchers trained on open-domain captions, this adds no separate training stage.

Entity Selection asks the model to list the article's entities that also appear in the caption, including non-visual ones. Labels for both tasks come from SpaCy entity matches against training captions, and both are trained jointly with captioning. At inference the same model extracts sentences and entities, adds them to its usual context within a 600-word limit, and writes the caption.

  1. 01

    Align around entities

    Train on entity-aware sentence selection and entity selection alongside news image captioning, using supervision derived from existing caption data.

  2. 02

    Extract relevant information

    At inference, the aligned model identifies useful sentences and entities from the article using the image as context.

  3. 03

    Generate with selected context

    Supplement the original textual input with the extracted information, then generate a caption with the same multimodal model.

03 / Reading the evidence

Selecting entity context beats reading more of the article

Measured · Tables 2 and 4, NYTimes800k

What each step adds, and how far the oracle is

On NYTimes800k, the ablation adds one component at a time to InstructBLIP: full fine-tuning, then each alignment task, then the self-supplemented input. Table 4 adds the alternatives the paper tested: reading more of the article, and oracle sentences and entities taken from the reference caption. Switch the metric to see which steps help the caption scores and which help the named entities.

Ablation ladder · Table 2

  1. InstructBLIP (OSFT)Official recipe: only the vision–language connector is fine-tuned
    75.95starting point
  2. Base+ full fine-tuning of the connector and the LLM
    82.70+6.75 vs OSFT
  3. Align (SENT + CAP)+ entity-aware sentence selection task
    83.60+0.90 vs Base
  4. Align (ENT + CAP)+ entity selection task
    84.28+1.58 vs Base
  5. Align (SENT + ENT + CAP)+ both alignment tasks
    84.45+1.75 vs Base
  6. EAMA+ self-supplemented input: the aligned model’s own selected sentences and entities
    87.00+2.55 vs Align (both tasks)

Reading more of the article · Table 4

  1. Base, full articleWhole article in training and inference (about twice the time and memory)
    83.17+0.47 vs Base
  2. Base, longer local contextUsual context extended to the supplemented length at inference
    81.75−0.95 vs Base
  3. Align, longer local contextAligned model, usual context extended to the same length
    83.50−0.95 vs Align (both tasks)

Oracle supplements from the reference caption · Table 4

  1. Base + oracle sentencesSentences sharing entities with the reference caption
    87.56+4.86 vs Base
  2. Base + oracle entitiesEntities in both the article and the reference caption
    90.44+7.74 vs Base
  3. Base + oracle sentences and entitiesUpper bound: both oracle supplements
    94.27+11.57 vs Base
Alignment tasks
+1.75
Self-supplemented input
+2.55
Still to the oracle
+7.27
Share of the Base-to-oracle gap EAMA closes
37%

Measured · Tables 2 and 4 NYTimes800k test results, as printed in arXiv v5. Computed each change against the row named next to it, and the gap shares. Bars run from that row’s value to this row’s value; oracle rows use the reference caption, which is not available at inference.

CIDEr: EAMA 87.00 against Base 82.70 and the oracle 94.27, closing 37% of the gap. The two alignment tasks together beat either task alone. Reading the full article changes Base by +0.47, against +4.30 for EAMA’s selected context.

Measured: all values from Tables 2 and 4 of arXiv 2402.19404v5 (NYTimes800k test set; BLEU-4, METEOR, ROUGE, CIDEr, and precision and recall of generated named entities). Computed: the change from the row named beside each value, and the share of the Base-to-oracle gap. Oracle rows use sentences and entities from the reference caption, which is unavailable at inference, so they are an upper bound rather than a method. The full-article model is trained and tested on the whole article, which the paper reports nearly doubles time and memory (Section 5.4). Single runs; no variance is reported.Tables 2 and 4 (arXiv v5) ↗

On NYTimes800k (Table 2), fully fine-tuning InstructBLIP beats the official connector-only recipe, 82.70 versus 75.95 CIDEr, the caption metric that weights rare words such as names most. Each alignment task helps alone, both together reach 84.45, and self-supplemented generation adds the largest step among EAMA's components. Reading the full article lifts the caption-only model only to 83.17, at nearly twice the cost.

Oracle sentences and entities from the reference caption reach 94.27 CIDEr, so selection has room to improve. EAMA recalls the most reference entities, including 28.21% of NYTimes800k entities unseen in training, but a larger share of its generated names miss the reference than for Xu et al. (2024a).

Table 1 (selected rows/columns), GoodNews test set. Caption metrics (BLEU-4, ROUGE, CIDEr) and precision (P) and recall (R) of generated named entities; higher is better. InstructBLIP (OSFT) is fine-tuned with the official scripts from a 7B checkpoint; EAMA's entity precision is below that of Xu et al. (2024a). Source ↗

MethodBLEU-4ROUGECIDErNamed entities PNamed entities R
JoGANIC6.8323.0561.2226.8722.05
NewsMEP8.3023.1763.9923.4323.24
Rule-driven8.1823.5671.5825.5123.68
Rajakumar Kalarani et al. (2023)7.1424.3072.3324.3720.09
Qu et al. (2024)8.6023.3871.9624.3025.54
Xu et al. (2024a)8.4926.2283.5230.1926.57
InstructBLIP (OSFT)9.5325.6178.0325.8927.33
EAMA10.0427.0687.7027.5828.92

Method proposed in the paper

Scope. EAMA is trained on 100K samples from each of GoodNews and NYTimes800k, both English, with InstructBLIP (7B) as its only backbone; the paper notes high GPU memory use and slower inference than earlier captioners. Read the study ↗

Further reading in the paperPaper · Sections 3–4 and Tables 1–4 ↗Published article ↗
Read the abstract

News image captioning requires model to generate an informative caption rich in entities, with the news image and the associated news article. Current MLLMs still bear limitations in handling entity information in news image captioning tasks. Besides, generating high-quality news image captions requires a trade-off between sufficiency and conciseness of textual input information. To explore the potential of MLLMs and address problems we discovered, we propose EAMA : an Entity-Aware Multimodal Alignment based approach for News Image Captioning. Our approach first aligns the MLLM with two extra alignment tasks: Entity-Aware Sentence Selection task and Entity Selection task, together with News Image Captioning task. The aligned MLLM will utilize the additional entity-related information extracted by itself to supplement the textual input while generating news image captions. Our approach achieves better results than all previous models on two mainstream news image captioning datasets.

Full paper ↗
Citation BibTeX 
@article{zhang2026eama,
  title = {EAMA: Entity-Aware Multimodal Alignment Based Approach for News Image Captioning},
  author = {Zhang, Junzhe and Zhang, Huixuan and Yin, Xunjian and Wan, Xiaojun},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  volume = {22},
  number = {5},
  pages = {1--19},
  year = {2026},
  doi = {10.1145/3801548},
  url = {https://doi.org/10.1145/3801548}
}