20250314

• journaling
• conversations on SNS
• re-visited Designing Data-Intensive Applications Chapter 11
• made progress on reading Designing Data-Intensive Applications Chapter 12
• 30 minutes walk (I found out a nearby pub)
• called Santa Clara country for the electricity service
• chest press, lat pull down, kettle lift, arm curl, 30 minutes run

I thought I could get more things done, but I’m exhausted now. Humans tend to overvalue what they can do in the future.

The history of s/old/new/

Assuming the syntax has been there in the editors/programming languages since their invention, the syntax propagated from the “ed” editor.

ed (1971, Ken Thompson)

sed (1974, Lee E.McMahon)

vi (1976, Bill Joy)

Perl (1987, Larry Wall)

Designing Data-Intensive Applications - Chapter 11 - Stream Processing

Batch processing works well for fixed data sets, but real-world data often grows continuously. In these cases, stream processing is more suitable, as it handles unbounded data in real-time.

In a stream processing context, each record is referred to as an event and grouped into a topic. Systems that generate these events and send them as messages are called producers, and those that process them are called consumers. There is no fixed limit to the number of producers or consumers in a given system.

Sending messages (which contains events) directly from producers to consumers is risky if either side goes offline. To reduce this risk, a message broker (also known as a message queue) serves as an intermediary, handling the traffic between producers and consumers.

Because consumers can crash at any time, a mechanism is needed to ensure reliable message delivery. This mechanism, known as acknowledgements, works by having the consumer notify the message broker once it has successfully processed a message, allowing the broker to remove the message from the queue.

When multiple consumers subscribe to the same topic, there are two ways to deliver messages: load balancing and fan-out. In the load-balancing approach, each message goes to exactly one consumer, while in the fan-out approach, every consumer receives every message.

(Kaz: A typical pattern when developing a web application is to use a load-balancing messaging setup. In this approach, you place time-insensitive and/or computationally expensive tasks into a message queue (message broker), allowing one of the consumers to pick up each task and perform it asynchronously. A typical example in Django projects is the use of the Django web server as the producer, RabbitMQ as the message broker, and Celery processes as consumers.)

Some common use cases of stream processing include:

• Complex Event Processing (CEP): Detecting specific patterns involving multiple events in a stream.
  • Fraud Detection in Banking: detecting fraudulent activity.
• Stream Analystics: Measuring statistics over fixed time intervals.
  • Stock Market Ticker Analysis: calculating rolling averages of stock prices.
• Search on Streams: Searching for individual events based on predefined criteria.
  • Media Monitoring: subscribing to feeds of specific type of news articles.

TODO:

• Designing Data-Intensive Applications Chapter 12
• Prepare a docuemnt for the board meeting
• 2044E

index 20250313 20250315