Sources

Thunder and Phoenix source implementations for candidate retrieval in the Home Mixer pipeline

Sources are the first stage of the Home Mixer pipeline, responsible for retrieving candidate posts from various backend services. The Home Mixer implements two primary sources: Thunder (in-network) and Phoenix (out-of-network).

Overview

Sources implement the Source trait which provides an asynchronous get_candidates method that returns a list of PostCandidate objects. Each source can optionally implement an enable method to conditionally activate based on query parameters.

ThunderSource

The Thunder source retrieves in-network posts from users that the viewer follows. It queries the Thunder service’s AMP cluster to fetch posts from the user’s following list.

Implementation

pub struct ThunderSource {
    pub thunder_client: Arc<ThunderClient>,
}

Configuration

thunder_client · Arc<ThunderClient> · required

Client for communicating with the Thunder service clusters

cluster · ThunderCluster · default: "Amp"

The Thunder cluster to query (defaults to Amp cluster)

max_results · usize

Maximum number of posts to retrieve from Thunder service (configured via THUNDER_MAX_RESULTS)

Request Parameters

When querying Thunder, the source constructs a request with:

• user_id: The viewer’s user ID
• following_user_ids: List of user IDs the viewer follows
• max_results: Maximum posts to retrieve
• algorithm: Retrieval algorithm (defaults to “default”)
• exclude_tweet_ids: Posts to exclude from results

Output Fields

tweet_id · i64

The unique identifier of the post

author_id · u64

The user ID of the post author

in_reply_to_tweet_id · Option<u64>

If the post is a reply, the ID of the parent post

ancestors · Vec<u64>

List of ancestor post IDs in the conversation thread. Includes the immediate parent and conversation root if different.

served_type · ServedType

Set to ForYouInNetwork to indicate this candidate came from the user’s network

Code Example

home-mixer/sources/thunder_source.rs

let request = GetInNetworkPostsRequest {
    user_id: query.user_id as u64,
    following_user_ids: following_list.iter().map(|&id| id as u64).collect(),
    max_results: p::THUNDER_MAX_RESULTS,
    exclude_tweet_ids: vec![],
    algorithm: "default".to_string(),
    debug: false,
    is_video_request: false,
};

let response = client
    .get_in_network_posts(request)
    .await
    .map_err(|e| format!("ThunderSource: {}", e))?;

let candidates: Vec<PostCandidate> = response
    .into_inner()
    .posts
    .into_iter()
    .map(|post| PostCandidate {
        tweet_id: post.post_id,
        author_id: post.author_id as u64,
        in_reply_to_tweet_id: post.in_reply_to_post_id,
        ancestors,
        served_type: Some(pb::ServedType::ForYouInNetwork),
        ..Default::default()
    })
    .collect();

PhoenixSource

The Phoenix source retrieves out-of-network posts using the Phoenix retrieval service. It leverages the user’s action sequence to find relevant posts from outside their following graph.

Implementation

pub struct PhoenixSource {
    pub phoenix_retrieval_client: Arc<dyn PhoenixRetrievalClient + Send + Sync>,
}

Configuration

phoenix_retrieval_client · Arc<dyn PhoenixRetrievalClient> · required

Client for communicating with the Phoenix retrieval service

max_results · usize

Maximum number of posts to retrieve (configured via PHOENIX_MAX_RESULTS)

Conditional Activation

Phoenix source includes an enable method that checks query parameters:

home-mixer/sources/phoenix_source.rs

fn enable(&self, query: &ScoredPostsQuery) -> bool {
    !query.in_network_only
}

The source is disabled when in_network_only is true, ensuring only Thunder results are used.

Requirements

Phoenix source requires the user_action_sequence to be present in the query. This sequence captures the user’s recent interactions and is used by Phoenix to understand user preferences.

Output Fields

tweet_id · i64

The unique identifier of the post

author_id · u64

The user ID of the post author

in_reply_to_tweet_id · Option<u64>

If the post is a reply, the ID of the parent post

served_type · ServedType

Set to ForYouPhoenixRetrieval to indicate this candidate came from Phoenix

Code Example

home-mixer/sources/phoenix_source.rs

let sequence = query
    .user_action_sequence
    .as_ref()
    .ok_or_else(|| "PhoenixSource: missing user_action_sequence".to_string())?;

let response = self
    .phoenix_retrieval_client
    .retrieve(user_id, sequence.clone(), p::PHOENIX_MAX_RESULTS)
    .await
    .map_err(|e| format!("PhoenixSource: {}", e))?;

let candidates: Vec<PostCandidate> = response
    .top_k_candidates
    .into_iter()
    .flat_map(|scored_candidates| scored_candidates.candidates)
    .filter_map(|scored_candidate| scored_candidate.candidate)
    .map(|tweet_info| PostCandidate {
        tweet_id: tweet_info.tweet_id as i64,
        author_id: tweet_info.author_id,
        in_reply_to_tweet_id: Some(tweet_info.in_reply_to_tweet_id),
        served_type: Some(pb::ServedType::ForYouPhoenixRetrieval),
        ..Default::default()
    })
    .collect();

Source Pipeline Integration

Both sources are typically used together in the Phoenix candidate pipeline. Thunder provides in-network candidates while Phoenix provides out-of-network candidates, creating a diverse feed that balances familiar content with discovery.

Typical Flow

1. Thunder Source queries the user’s following graph
2. Phoenix Source (if enabled) retrieves out-of-network recommendations
3. Candidates from both sources are merged
4. Subsequent pipeline stages apply filters and scoring

Error Handling

Both sources return Result<Vec<PostCandidate>, String>, allowing the pipeline to handle failures gracefully. Common error scenarios include:

• No available service channels
• Network timeouts
• Missing required query fields
• Service-level errors

Related Components

• Filters - Filter candidates after source retrieval
• Scorers - Score candidates for ranking
• Phoenix Scorer - Scores Phoenix candidates using ML predictions