Solution

Delegate events with unrecoverable errors to another component for handling so that they do not block legitimate events from processing. Publish these errors as fault events, along with the affected events and the stream processor information, so that the events can be resubmitted when appropriate. Monitor and alert on these fault events so that the team can react in a timely manner. Provide the necessary utilities to resubmit the affected events from the data lake to the component that emitted the fault event.

Implement the stream processor logic so that it is tolerant of events that are received out of order and idempotent when it receives and processes events more than once. Optionally, leverage a traditional synchronous circuit breaker on resource calls so that faults can be produced immediately when a failure threshold is reached, and periodically test to see if the resource has healed. Leverage a functional reactive stream programming paradigm for implementing stream processors. It is a very natural fit for stream processing and its pull-based model provides for backpressure that helps control the speed of the processing flow so that resources do not get overwhelmed with too many calls.