Spark Streaming Programming Guide

參考,http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html html

Overview

SparkStreaming支持多種流輸入,like Kafka, Flume, Twitter, ZeroMQ or plain old TCP sockets,而且能夠在上面進行transform操做,最終數據存入HDFS,數據庫或dashboard
另外能夠把Spark’s in-built
machine learning algorithms, and graph processing algorithms用於spark streaming,這個比較有意思
SparkStreaming的原理,下面那幅圖很清晰,將stream數據離散化,提出的概念DStream,其實就是sequence of RDDs
java

Spark Streaming is an extension of the core Spark API that allows enables high-throughput, fault-tolerant stream processing of live data streams.
Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ or plain old TCP sockets and be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s in-built machine learning algorithms, and graph processing algorithms on data streams. node

image

Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches. git

image

Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. DStreams can be created either from input data stream from sources such as Kafka and Flume, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs. github

A Quick Example

Before we go into the details of how to write your own Spark Streaming program, let’s take a quick look at what a simple Spark Streaming program looks like.web


Initializing

StreamingContext是SparkStreaming的入口,就像SparkContext對於spark同樣
其實StreamingContext就是SparkContext的封裝,經過streamingContext.sparkContext能夠取得
參數除了batchDuration,其餘的都和SparkContext沒有區別
算法

To initialize a Spark Streaming program in Scala, a StreamingContext object has to be created, which is the main entry point of all Spark Streaming functionality.
A StreamingContext object can be created by using數據庫


The master parameter is a standard Spark cluster URL and can be 「local」 for local testing.
The appName is a name of your program, which will be shown on your cluster’s web UI.
The batchInterval is the size of the batches, as explained earlier.
Finally, the last two parameters are needed to deploy your code to a cluster if running in distributed mode, as described in the Spark programming guide.
Additionally, the underlying SparkContext can be accessed as streamingContext.sparkContext. express

The batch interval must be set based on the latency requirements of your application and available cluster resources. See the Performance Tuning section for more details. apache

DStreams

經過圖示很清晰的說明什麼是DStream,和基於DStream的transform是怎樣的?

Discretized Stream or DStream is the basic abstraction provided by Spark Streaming. It represents a continuous stream of data, either the input data stream received from source, or the processed data stream generated by transforming the input stream. Internally, it is represented by a continuous sequence of RDDs, which is Spark’s abstraction of an immutable, distributed dataset. Each RDD in a DStream contains data from a certain interval, as shown in the following figure.

image

Any operation applied on a DStream translates to operations on the underlying RDDs. For example, in the earlier example of converting a stream of lines to words, the flatmap operation is applied on each RDD in the lines DStream to generate the RDDs of the words DStream. This is shown the following figure.

image

These underlying RDD transformations are computed by the Spark engine. The DStream operations hide most of these details and provides the developer with higher-level API for convenience. These operations are discussed in detail in later sections.

Operations

DStream支持transformations and output,和Spark的action不太同樣

There are two kinds of DStream operations - transformations and output operations.
Similar to RDD transformations, DStream transformations operate on one or more DStreams to create new DStreams with transformed data.
After applying a sequence of transformations to the input streams, output operations need to called, which write data out to an external data sink, such as a filesystem or a database.

Transformations大部分都和RDD中的同樣,參考原文,重點說下底下的幾個,

UpdateStateByKey Operation

用流數據持續更新state,好比下面的例子,wordcount
只須要定義updateFunction,如何根據newValue更新現有的state

The updateStateByKey operation allows you to maintain arbitrary stateful computation, where you want to maintain some state data and continuously update it with new information.
To use this, you will have to do two steps.

  1. Define the state - The state can be of arbitrary data type.
  2. Define the state update function - Specify with a function how to update the state using the previous state and the new values from input stream.

Let’s illustrate this with an example. Say you want to maintain a running count of each word seen in a text data stream. Here, the running count is the state and it is an integer. We define the update function as


Transform Operation

Transform能夠在DStream上apply任意的RDD操做,尤爲有意思的是,能夠經過Transform使用Spark的MLlib和Graph算法

The transform operation (along with its variations like transformWith) allows arbitrary RDD-to-RDD functions to be applied on a DStream. It can be used to apply any RDD operation that is not exposed in the DStream API. For example, the functionality of joining every batch in a data stream with another dataset is not directly exposed in the DStream API. However, you can easily use transform to do this. This enables very powerful possibilities. For example, if you want to do real-time data cleaning by joining the input data stream with precomputed spam information (maybe generated with Spark as well) and then filtering based on it.

In fact, you can also use machine learning and graph computation algorithms in the transform method.

Window Operations

做爲streaming處理,固然須要支持典型的slide-windows based的方法
能夠看下面的圖,能夠將original中的slide-windows上的多個RDDs生成windowed DStream上的單個RDD
兩個重要的參數,windows length和slide inteval,下面的圖中分別是3和2

Finally, Spark Streaming also provides windowed computations, which allow you to apply transformations over a sliding window of data. This following figure illustrates this sliding window.

image

As shown in the figure, every time the window slides over a source DStream, the source RDDs that fall within the window are combined and operated upon to produce the RDDs of the windowed DStream. In this specific case, the operation is applied over last 3 time units of data, and slides by 2 time units. This shows that any window-based operation needs to specify two parameters.

  • window length - The duration of the window (3 in the figure)
  • slide interval - The interval at which the window-based operation is performed (2 in the figure).

These two parameters must be multiples of the batch interval of the source DStream (1 in the figure).

給個例子,統計過去30秒的word count,每次遞進10秒

Let’s illustrate the window operations with an example. Say, you want to extend the earlier example by generating word counts over last 30 seconds of data, every 10 seconds. To do this, we have to apply the reduceByKey operation on the pairs DStream of (word, 1) pairs over the last 30 seconds of data. This is done using the operation reduceByKeyAndWindow.

// Reduce last 30 seconds of data, every 10 seconds
val windowedWordCounts = pairs.reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))

Some of the common window-based operations are as follows. All of these operations take the said two parameters - windowLength and slideInterval.
image

Output Operations

When an output operator is called, it triggers the computation of a stream. Currently the following output operators are defined:

image

Persistence

DStream的persistence是經過persist之上的每一個RDD來實現的,並無什麼特別的地方
須要注意的是,window-based operations會自動將persist設爲true,由於slide-windows based算法中,每一個RDD都會被讀多遍
並且對於來自Kafka, Flume, sockets, etc.的數據,default的persist策略是,產生2個replicas並同時寫入兩個節點,便於fault-tolerance
Spark的fault-tolerance取決於replay,對於普通的Spark數據是從文件讀的,因此replay只是從新讀取數據而已,很簡單
但對於流數據,數據一旦流過就沒法replay,因此SparkStreaming必須在source採起2-replicas的策略

Similar to RDDs, DStreams also allow developers to persist the stream’s data in memory. That is, using persist() method on a DStream would automatically persist every RDD of that DStream in memory. This is useful if the data in the DStream will be computed multiple times (e.g., multiple operations on the same data). For window-based operations like reduceByWindow and reduceByKeyAndWindow and state-based operations like updateStateByKey, this is implicitly true. Hence, DStreams generated by window-based operations are automatically persisted in memory, without the developer calling persist().

For input streams that receive data over the network (such as, Kafka, Flume, sockets, etc.), the default persistence level is set to replicate the data to two nodes for fault-tolerance.

Note that, unlike RDDs, the default persistence level of DStreams keeps the data serialized in memory. This is further discussed in the Performance Tuning section. More information on different persistence levels can be found in Spark Programming Guide.

RDD Checkpointing

一樣DStream也支持checkpoint,即把RDD寫入HDFS中,固然cp會影響相應時間,因此要處理好這個balance,通常cp interval爲5 - 10 times of sliding interval,至少10秒
在DStream中cp主要用於stateful operation,好比前面說的wordcount,因爲streaming處理的特色,即便persist,也只可能保留一段時間的數據,因此若是不去作cp,crash後是沒法replay出過久之前的結果的,因此對於DStream而已CP的意義更爲重要(對於spark,cp只是節省時間)

A stateful operation is one which operates over multiple batches of data. This includes all window-based operations and the updateStateByKey operation. Since stateful operations have a dependency on previous batches of data, they continuously accumulate metadata over time. To clear this metadata, streaming supports periodic checkpointing by saving intermediate data to HDFS. Note that checkpointing also incurs the cost of saving to HDFS which may cause the corresponding batch to take longer to process. Hence, the interval of checkpointing needs to be set carefully. At small batch sizes (say 1 second), checkpointing every batch may significantly reduce operation throughput. Conversely, checkpointing too slowly causes the lineage and task sizes to grow which may have detrimental effects. Typically, a checkpoint interval of 5 - 10 times of sliding interval of a DStream is good setting to try.

To enable checkpointing, the developer has to provide the HDFS path to which RDD will be saved. This is done by using

ssc.checkpoint(hdfsPath) // assuming ssc is the StreamingContext or JavaStreamingContext

The interval of checkpointing of a DStream can be set by using

dstream.checkpoint(checkpointInterval)

For DStreams that must be checkpointed (that is, DStreams created by updateStateByKey and reduceByKeyAndWindow with inverse function), the checkpoint interval of the DStream is by default set to a multiple of the DStream’s sliding interval such that its at least 10 seconds.

Performance Tuning

在性能優化上,主要考慮兩點
1. 若是下降每一個batch的執行時間,增長併發度,減小沒必要要的serialization和下降task launch的overhead(減小tasksize和使用Standalone or coarse-grained Mesos mode )
2. 設置合適batch size,batch size越小確定越接近流處理,可是overhead越大,因此須要balance,建議是先在一個比較保守的size和比較小的流量下進行測試,而後若是系統穩定,在逐步減小size和加大流量

Getting the best performance of a Spark Streaming application on a cluster requires a bit of tuning. This section explains a number of the parameters and configurations that can tuned to improve the performance of you application. At a high level, you need to consider two things:

  1. Reducing the processing time of each batch of data by efficiently using cluster resources.
  2. Setting the right batch size such that the data processing can keep up with the data ingestion.

Reducing the Processing Time of each Batch

There are a number of optimizations that can be done in Spark to minimize the processing time of each batch. These have been discussed in detail in Tuning Guide. This section highlights some of the most important ones.

Level of Parallelism

Cluster resources maybe under-utilized if the number of parallel tasks used in any stage of the computation is not high enough. For example, for distributed reduce operations like reduceByKey and reduceByKeyAndWindow, the default number of parallel tasks is 8. You can pass the level of parallelism as an argument (see the PairDStreamFunctions documentation), or set the config property spark.default.parallelism to change the default.

Data Serialization

The overhead of data serialization can be significant, especially when sub-second batch sizes are to be achieved. There are two aspects to it.

  • Serialization of RDD data in Spark: Please refer to the detailed discussion on data serialization in the Tuning Guide. However, note that unlike Spark, by default RDDs are persisted as serialized byte arrays to minimize pauses related to GC.

  • Serialization of input data: To ingest external data into Spark, data received as bytes (say, from the network) needs to deserialized from bytes and re-serialized into Spark’s serialization format. Hence, the deserialization overhead of input data may be a bottleneck.

Task Launching Overheads

If the number of tasks launched per second is high (say, 50 or more per second), then the overhead of sending out tasks to the slaves maybe significant and will make it hard to achieve sub-second latencies. The overhead can be reduced by the following changes:

  • Task Serialization: Using Kryo serialization for serializing tasks can reduced the task sizes, and therefore reduce the time taken to send them to the slaves.

  • Execution mode: Running Spark in Standalone mode or coarse-grained Mesos mode leads to better task launch times than the fine-grained Mesos mode. Please refer to the Running on Mesos guide for more details.

These changes may reduce batch processing time by 100s of milliseconds, thus allowing sub-second batch size to be viable.

Setting the Right Batch Size

For a Spark Streaming application running on a cluster to be stable, the processing of the data streams must keep up with the rate of ingestion of the data streams. Depending on the type of computation, the batch size used may have significant impact on the rate of ingestion that can be sustained by the Spark Streaming application on a fixed cluster resources. For example, let us consider the earlier WordCountNetwork example. For a particular data rate, the system may be able to keep up with reporting word counts every 2 seconds (i.e., batch size of 2 seconds), but not every 500 milliseconds.

A good approach to figure out the right batch size for your application is to test it with a conservative batch size (say, 5-10 seconds) and a low data rate. To verify whether the system is able to keep up with data rate, you can check the value of the end-to-end delay experienced by each processed batch (either look for 「Total delay」 in Spark driver log4j logs, or use the StreamingListener interface). If the delay is maintained to be comparable to the batch size, then system is stable. Otherwise, if the delay is continuously increasing, it means that the system is unable to keep up and it therefore unstable. Once you have an idea of a stable configuration, you can try increasing the data rate and/or reducing the batch size. Note that momentary increase in the delay due to temporary data rate increases maybe fine as long as the delay reduces back to a low value (i.e., less than batch size).

24/7 Operation

Streaming操做,數據是無限的,因此必然須要cleanup metadata,好比generated RDD
能夠在SparkContext建立前,經過spark.cleaner.ttl 設置cleanup時間
注意cleanup時間必需要比任意window operation的窗口大小要長

By default, Spark does not forget any of the metadata (RDDs generated, stages processed, etc.). But for a Spark Streaming application to operate 24/7, it is necessary for Spark to do periodic cleanup of it metadata. This can be enabled by setting the configuration property spark.cleaner.ttl to the number of seconds you want any metadata to persist. For example, setting spark.cleaner.ttl to 600 would cause Spark periodically cleanup all metadata and persisted RDDs that are older than 10 minutes. Note, that this property needs to be set before the SparkContext is created.

This value is closely tied with any window operation that is being used. Any window operation would require the input data to be persisted in memory for at least the duration of the window. Hence it is necessary to set the delay to at least the value of the largest window operation used in the Spark Streaming application. If this delay is set too low, the application will throw an exception saying so.

Monitoring

Besides Spark’s in-built monitoring capabilities, the progress of a Spark Streaming program can also be monitored using the [StreamingListener] (streaming/index.html#org.apache.spark.scheduler.StreamingListener) interface, which allows you to get statistics of batch processing times, queueing delays, and total end-to-end delays. Note that this is still an experimental API and it is likely to be improved upon (i.e., more information reported) in the future.

Memory Tuning

Tuning the memory usage and GC behavior of Spark applications have been discussed in great detail in the Tuning Guide. It is recommended that you read that. In this section, we highlight a few customizations that are strongly recommended to minimize GC related pauses in Spark Streaming applications and achieving more consistent batch processing times.

  • Default persistence level of DStreams: Unlike RDDs, the default persistence level of DStreams serializes the data in memory (that is, StorageLevel.MEMORY_ONLY_SER for DStream compared to StorageLevel.MEMORY_ONLY for RDDs). Even though keeping the data serialized incurs higher serialization/deserialization overheads, it significantly reduces GC pauses.

  • Clearing persistent RDDs: By default, all persistent RDDs generated by Spark Streaming will be cleared from memory based on Spark’s in-built policy (LRU). If spark.cleaner.ttl is set, then persistent RDDs that are older than that value are periodically cleared. As mentioned earlier, this needs to be careful set based on operations used in the Spark Streaming program. However, a smarter unpersisting of RDDs can be enabled by setting the configuration property spark.streaming.unpersist to true. This makes the system to figure out which RDDs are not necessary to be kept around and unpersists them. This is likely to reduce the RDD memory usage of Spark, potentially improving GC behavior as well.

  • Concurrent garbage collector: Using the concurrent mark-and-sweep GC further minimizes the variability of GC pauses. Even though concurrent GC is known to reduce the overall processing throughput of the system, its use is still recommended to achieve more consistent batch processing times.

Fault-tolerance Properties

簡而言之,Spark的fault-tolerance是很容易作的,由於只有source data存在,任何RDD都是能夠被replay出來的
因此關鍵問題在於如何保證source data的fault-tolerance

In this section, we are going to discuss the behavior of Spark Streaming application in the event of a node failure. To understand this, let us remember the basic fault-tolerance properties of Spark’s RDDs.

  1. An RDD is an immutable, deterministically re-computable, distributed dataset. Each RDD remembers the lineage of deterministic operations that were used on a fault-tolerant input dataset to create it.
  2. If any partition of an RDD is lost due to a worker node failure, then that partition can be re-computed from the original fault-tolerant dataset using the lineage of operations.

Since all data transformations in Spark Streaming are based on RDD operations, as long as the input dataset is present, all intermediate data can recomputed. Keeping these properties in mind, we are going to discuss the failure semantics in more detail.

Failure of a Worker Node

對於worker node的失敗,只須要換個worker把RDD給replay出來就能夠了,前提就是要可以從新取到source data
因此對於用HDFS做爲input source的case,fault-tolerance是沒有問題的
關鍵是,以network做爲input source的case(好比,kafka,flume,socket),如何保證可以從新讀到source data?
作法是首先以2-replicas把source data放到兩個node的memory中,一個fail,能夠用另外一個replay
可是一種狀況下,仍是會丟數據,就是當network receiver所在的node fail的時候,那麼從network上讀到並尚未來得及完成replicas的數據就會丟失

另外須要注意的是,對於RDD的transform是能夠保證exactly-once semantics的,可是對於output operations就只能保證at-least once semantics
好比,某個RDD已經部分被寫到file中,這個時候fail,當worker replay出RDD的時候會從新寫一遍
因此若是要保證exactly-once,必須本身加上additional transactions-like mechanisms

There are two failure behaviors based on which input sources are used.

  1. Using HDFS files as input source - Since the data is reliably stored on HDFS, all data can re-computed and therefore no data will be lost due to any failure.
  2. Using any input source that receives data through a network - For network-based data sources like Kafka and Flume, the received input data is replicated in memory between nodes of the cluster (default replication factor is 2). So if a worker node fails, then the system can recompute the lost from the the left over copy of the input data. However, if the worker node where a network receiver was running fails, then a tiny bit of data may be lost, that is, the data received by the system but not yet replicated to other node(s). The receiver will be started on a different node and it will continue to receive data.

Since all data is modeled as RDDs with their lineage of deterministic operations, any recomputation always leads to the same result. As a result, all DStream transformations are guaranteed to have exactly-once semantics. That is, the final transformed result will be same even if there were was a worker node failure. However, output operations (like foreachRDD) have at-least once semantics, that is, the transformed data may get written to an external entity more than once in the event of a worker failure. While this is acceptable for saving to HDFS using the saveAs*Files operations (as the file will simply get over-written by the same data), additional transactions-like mechanisms may be necessary to achieve exactly-once semantics for output operations.

Failure of the Driver Node

對於Driver Node failure ,Spark是沒有作任何處理的,job會失敗
而對於SparkStreaming而言,這樣是沒法接受的,由於流數據你沒法從新讀出的,對於流處理必需要保證24/7
因此這裏作的更多,會按期把DStreams的metadata chechpoint到HDFS上,這樣當Driver Node failure,被restart後,能夠衝checkpoint文件中把狀態恢復過來,繼續執行
可是當前對於以network做爲input source的狀況下,數據仍是會丟失,因爲driver node的fail會致使sourcedata丟失,雖然你metadata是checkpoint了

To allows a streaming application to operate 24/7, Spark Streaming allows a streaming computation to be resumed even after the failure of the driver node. Spark Streaming periodically writes the metadata information of the DStreams setup through the StreamingContext to a HDFS directory (can be any Hadoop-compatible filesystem). This periodic checkpointing can be enabled by setting a the checkpoint directory using ssc.checkpoint(<checkpoint directory>) as described earlier. On failure of the driver node, the lost StreamingContext can be recovered from this information, and restarted.

To allow a Spark Streaming program to be recoverable, it must be written in a way such that it has the following behavior:

  1. When the program is being started for the first time, it will create a new StreamingContext, set up all the streams and then call start().
  2. When the program is being restarted after failure, it will re-create a StreamingContext from the checkpoint data in the checkpoint directory.
// Function to create and setup a new StreamingContext
def functionToCreateContext(): StreamingContext = {
    val ssc = new StreamingContext(...)   // new context
    val lines = ssc.socketTextStream(...) // create DStreams
    ...
    ssc.checkpoint(checkpointDirectory)   // set checkpoint directory
    ssc
}

// Get StreaminContext from checkpoint data or create a new one
val context = StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext _)

// Do additional setup on context that needs to be done,
// irrespective of whether it is being started or restarted
context. ...

// Start the context
context.start()
context.awaitTermination()

If the checkpointDirectory exists, then the context will be recreated from the checkpoint data. If the directory does not exist (i.e., running for the first time), then the function functionToCreateContext will be called to create a new context and set up the DStreams. See the Scala example RecoverableNetworkWordCount. This example appends the word counts of network data into a file.You can also explicitly create a StreamingContext from the checkpoint data and start the computation by using new StreamingContext(checkpointDirectory).

Note: If Spark Streaming and/or the Spark Streaming program is recompiled, you must create a new StreamingContext or JavaStreamingContext, not recreate from checkpoint data. This is because trying to load a context from checkpoint data may fail if the data was generated before recompilation of the classes. So, if you are using getOrCreate, then make sure that the checkpoint directory is explicitly deleted every time recompiled code needs to be launched.

This failure recovery can be done automatically using Spark’s standalone cluster mode, which allows any Spark application’s driver to be as well as, ensures automatic restart of the driver on failure (see supervise mode). This can be tested locally by launching the above example using the supervise mode in a local standalone cluster and killing the java process running the driver (will be shown as DriverWrapper when jps is run to show all active Java processes). The driver should be automatically restarted, and the word counts will cont

For other deployment environments like Mesos and Yarn, you have to restart the driver through other mechanisms.

Recovery Semantics

There are two different failure behaviors based on which input sources are used.

  1. Using HDFS files as input source - Since the data is reliably stored on HDFS, all data can re-computed and therefore no data will be lost due to any failure.
  2. Using any input source that receives data through a network - The received input data is replicated in memory to multiple nodes. Since, all the data in the Spark worker’s memory is lost when the Spark driver fails, the past input data will not be accessible and driver recovers. Hence, if stateful and window-based operations are used (like updateStateByKey, window, countByValueAndWindow, etc.), then the intermediate state will not be recovered completely.

In future releases, we will support full recoverability for all input sources. Note that for non-stateful transformations like map, count, and reduceByKey, with all input streams, the system, upon restarting, will continue to receive and process new data

相關文章
相關標籤/搜索