Overview
When consuming streaming data, maximizing connection time and receiving all matched data is a fundamental goal. This requires:- Taking advantage of redundant connections
- Automatically detecting disconnections
- Reconnecting quickly
- Having a plan for recovering lost data
Redundant connections
A redundant connection allows you to establish more than one simultaneous connection to a stream. This provides redundancy by connecting with two separate consumers, receiving the same data through both connections. Benefits:- Hot failover if one stream disconnects
- Protection if your primary server fails
- Continuous data delivery during reconnection
How to use
Simply connect to the same stream URL with a second client. Data will be sent through both connections.Redundant connections are available for Enterprise access. Filtered Stream allows up to two redundant connections for Enterprise projects. Check your specific endpoint documentation for connection limits.
Backfill
After detecting a disconnection, your system should track how long the disconnection lasted to determine the appropriate recovery method.For disconnections of 5 minutes or less
Use the backfill parameter when reconnecting to receive Posts matched during the disconnection period.| Endpoint | Parameter | Example |
|---|---|---|
| Filtered Stream | backfill_minutes | ?backfill_minutes=5 |
| Powerstream | backfillMinutes | ?backfillMinutes=5 |
Important considerations:
- Older Posts are generally delivered first, before newly matched Posts
- Posts are not deduplicated — if you were disconnected for 90 seconds but request 2 minutes of backfill, you’ll receive 30 seconds of duplicate Posts
- Your system should be tolerant of duplicates
- Backfill is available with Enterprise access
Recovery
For disconnections lasting longer than 5 minutes, use the Recovery feature to replay missed data from within the last 24 hours.How Recovery works
- Make a connection request with
start_timeandend_timeparameters - Recovery re-streams the specified time period
- Once complete, the connection disconnects
Parameters
| Parameter | Type | Description |
|---|---|---|
start_time | ISO 8601 date | Start time to recover from (UTC) |
end_time | ISO 8601 date | End time to recover to (UTC) |
Example request
Filtered Stream:Recovery limits:
- Available for Enterprise access
- Recovery window: up to 24 hours in the past
- Filtered Stream allows 2 concurrent recovery jobs
Alternative recovery: Search
If you don’t have access to backfill or recovery features, or if the disconnection exceeded 24 hours, you can use the Search Posts endpoint to request missed data.Recovery decision tree
Best practices
- Track disconnection duration — Your system should note when disconnections occur and how long they last
- Implement automatic recovery — Based on disconnection duration, automatically choose the appropriate recovery method
- Handle duplicates — Both backfill and recovery may deliver duplicate Posts; implement deduplication logic
- Use redundant connections — Prevent data loss by maintaining multiple connections when available
- Monitor recovery jobs — Track the status and completion of recovery operations