![]() These simple components measure and monitor the flow of fluid in free-flowing conditions. These are used to create failover paths for your sinks or load balance events across multiple sinks from a channel.Posted by Virtual Polymer Compounds, LLC on Ap10:30 am | Comments Off on Fiberglass Flumes for Flow Measurementįlumes are specially designed static structures or fittings that restrict the flow of fluid by narrowing the available passage area within a system. These are used to invoke a particular sink from the selected group of sinks. Multiplexing channel selectors − These decides the channel to send an event based on the address in the header of that event. There are two types of channel selectors −ĭefault channel selectors − These are also known as replicating channel selectors they replicates all the events in each channel. These are used to determine which channel is to be opted to transfer the data in case of multiple channels. Interceptors are used to alter/inspect flume events which are transferred between source and channel. In addition to this, we have a few more components that play a vital role in transferring the events from the data generator to the centralized stores. What we have discussed above are the primitive components of the agent. We have listed all the supported sources, sinks, channels in the Flume configuration chapter of this tutorial. Note − A flume agent can have multiple sources, sinks and channels. ![]() The destination of the sink might be another agent or the central stores. ![]() It consumes the data (events) from the channels and delivers it to the destination. SinkĪ sink stores the data into centralized stores like HBase and HDFS. These channels are fully transactional and they can work with any number of sources and sinks.Įxample − JDBC channel, File system channel, Memory channel, etc. It acts as a bridge between the sources and the sinks. ChannelĪ channel is a transient store which receives the events from the source and buffers them till they are consumed by sinks. SourceĪ source is the component of an Agent which receives data from the data generators and transfers it to one or more channels in the form of Flume events.Īpache Flume supports several types of sources and each source receives events from a specified data generator.Įxample − Avro source, Thrift source, twitter 1% source etc. Following diagram represents a Flume AgentĪs shown in the diagram a Flume Agent contains three main components namely, source, channel, and sink. It receives the data (events) from clients or other agents and forwards it to its next destination (sink or agent). A typical Flume event would have the following structure − Flume AgentĪn agent is an independent daemon process (JVM) in Flume. It contains a payload of byte array that is to be transported from the source to the destination accompanied by optional headers. ![]() Flume EventĪn event is the basic unit of the data transported inside Flume. Thereafter, a data collector (which is also an agent) collects the data from the agents which is aggregated and pushed into a centralized store such as HDFS or HBase. As shown in the illustration, data generators (such as Facebook, Twitter) generate data which gets collected by individual Flume agents running on them. The following illustration depicts the basic architecture of Flume. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |