Презентация на тему «Joint Relevance and Freshness Learning From Clickthroughs for News Search»

Joint Relevance and Freshness Learning From Clickthroughs for News

Search

Hongning Wang+, Anlei Dong*, Lihong Li*, Yi Chang*, Evgeniy Gabrilovich* +CS@UIUC *Yahoo! Labs

Relevance v.s. Freshness

Relevance Topical relatedness Metric: tf*idf, BM25, Language Model Freshness Temporal closeness Metric: age, elapsed time Trade-off Serve for user’s information need

Freshness is Important for News Search

“Apple Company” @ Oct. 4, 2011

Freshness is Important for News Search

“Apple Company” @ Oct. 5, 2011

Understand User’s Information Need

User’s emphasis on relevance/freshness varies Breaking news queries Prefer latest news reports – freshness driven E.g., “apple company” Newsworthy queries Prefer high coverage and authority news reports – relevance driven E.g., “bin laden death”

Understand User’s Information Need

User’s emphasis on relevance/freshness varies

Assess User’s Information Need

Unsupervised integration [Efron 2011, Li 2003] Limited on timestamps Editor’s judgment [Dong 2010, Dai 2011] Expensive for timely annotation Inadequate to recover end-user’s information need

Manipulate Editor’s Annotation

Freshness-demoted relevance Rule-based hard demotion [Dong 2010] E.g., if the result is somewhat outdated, it should be demoted by one grade (e.g., from excellent to good)

User’s Judgment on Relevance and Freshness

User’s browsing behavior

Joint Relevance and Freshness Learning

JRFL: (Relevance, Freshness) -> Click

Query => trade-off

URL => relevance/freshness

Joint Relevance and Freshness Learning

Model formalization

Latent

Joint Relevance and Freshness Learning

Linear instantiation Associative property Relevance/Freshness model learning Query model learning

Joint Relevance and Freshness Learning

Coordinate descent for JRFL Randomly initialize , and set Repeat until converge Update Relevance/Freshness models: Update Query model: Return the final model

Convex programming

Temporal Features

URL freshness features Identify freshness from content analysis

Temporal Features

Query freshness features Capture latent preference

Experiment Results

Data sets Two months’ Yahoo! News Search sessions Normal bucket: top 10 positions Random bucket [Li 2011] Randomly shuffled top 4 positions Unbiased evaluation corpus Editor’s judgment: 1 day’s query log Preference pair selection [Joachims 2005] Click > Skip above Click > Skip next Ordered by Pearson’s value

Experiment Results

Data sets Statistics

Analysis of JRFL

Convergence Train/Test sets: 90k/60k preference pairs Varying initial query weight

(a) Object Function Value Update

Analysis of JRFL

Convergence Train/Test sets: 90k/60k preference pairs Varying initial query weight

(b) Pairwise Error Rate Update

Analysis of JRFL

Convergence Train/Test sets: 90k/60k preference pairs Varying initial query weight

(c) Query Weight Update

Analysis of JRFL

Feature weight learning

Analysis of JRFL

Relevance and Freshness Learning Baseline: GBRank trained on Dong et al.’s relevance/freshness annotation set Testing corpus: editor’s one day annotation set

Analysis of JRFL

Query weight analysis

Analysis of JRFL

Query weight analysis Query length differs in relevance/freshness driven queries significantly

Quantitative Comparison

Ranking performance Random bucket clicks

Quantitative Comparison

Ranking performance Normal clicks

Quantitative Comparison

Ranking performance Editorial annotations

Qualitative Comparison

CTR distribution revisit

Conclusions

Joint Relevance and Freshness Learning Query-specific preference Learning from query logs Temporal features Future work Personalized retrieval Broad spectral of user’s information need E.g., trustworthiness, opinion

References

[Efron 2011] M. Efron and G. Golovchinsky. Estimation methods for ranking recent information. In SIGIR, pages 495–504, 2011. [Li 2003] X. Li and W. Croft. Time-based language models. In CIKM, pages 469–475, 2003. [Dong 2010] A. Dong, Y. Chang, Z. Zheng, G. Mishne, J. Bai, R. Zhang, K. Buchner, C. Liao, and F. Diaz. Towards recency ranking in web search. In WSDM, pages 11–20, 2010. [Dai 2011] N. Dai, M. Shokouhi, and B. D. Davison. Learning to rank for freshness and relevance. In SIGIR, pages 95–104, 2011. [Li 2011] L. Li, W. Chu, J. Langford, and X. Wang. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of ACM WSDM '11, pages 297–306, 2011. [Joachims 2005] T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In SIGIR, pages 154–161, 2005.

Thank you

Q&A