Hire Top Hadoop developers with Ultragenius

• HDFS - HDFS stands for Hadoop Distributed File System. HDFS is responsible for storing massive amounts of datasets of unstructured and structured data. HDFS is comprised of two core components- 

• NameNode - NameNode manages the metadata. It keeps track of all the files in the file system and arranges the replicate blocks when the required data blocks managed by the DataNode are not available. NameNode is also called Master Node.
• DataNode - DataNode stores the actual data in HDFS as instructed by the NameNode. The DataNode performs read and write requests from the client’s file system.
  • YARN (Yet Another Resource Negotiator) - YARN is the widely used distributed operating system for managing data. The core components of YARN architecture include - 
  • Resource Manager - Resource Manager allocates the resources among all the applications. It includes two major components - Scheduler and Application Manager. Scheduler schedules the assigned tasks based on the available resources. It is not responsible for doing other tasks like monitoring and tracking the tasks. Scheduler does not guarantee to restart a process if it fails. Application Manager coordinates the process execution in an application and is responsible for accepting or rejecting the application when it is processed to the client. Application Manager guarantees to restart a process if it fails.
  • Nodes Manager - Nodes Manager launches and handles the node containers. It performs log management, monitors usage of resources, and also kills a container based on the instructions from the Resource Manager. 
  • Application Master - Application Master monitors the whole lifecycle of an application from requesting the required containers from the Resource Manager to executing specific operations on the obtained containers. 
  • Containers - Containers are the collection of physical resources like CPU cores, RAM, and disk on a single node. Container Launch Context (CLC) is the record that stores the environment variables, security tokens, dependencies, etc. CLC is responsible for invoking the containers.
• MapReduce - MapReduce uses the concept of distributed and parallel algorithms which makes Hadoop a very efficient technology to use. Map() sorts and filters the data, organizes it into a group, and generates a key-value pair of data. Reduce() aggregates the output generated by Map() and clusters it into smaller set of data.
•  

Apache Hive provides the facility of reading, writing,  and managing large data using SQL. Hive translates the HQL program into one or more Spark, Tez, or Java MapReduce processes. Then, it organizes the data into tabular format  for HDFS and run the processes to generate the result. 

These are the popular HDFS commands - 

• ls - ls command lists all the files.
• mkdir - mkdir creates a new directory.
• touchz - touchz creates an empty file.
• copyFromLocal - copies files/folders from local file system to Hadoop HDFS store,
• cat - cat command reads a file and print its content on the standard output.
• copyToLocal - copies files/folders from Hadoop store to local database system.
• du - du command is used to check the size of the file.
• text - HDFS prints the source file in text format.
• put - put command copies one or more than one sources from local file system to the destination file system.
• help - help shows the help for the specific command or for all commands if none of the command is specified.
• usage - usage command shows the help for a specific command.
• rmdir - removes an individual directory.
• expunge - makes the trash empty.
• mv - mv command transfers a file from source to destination. It can be used to move multiple files also when the destination is a directory.
• count - count command counts the number of files, directories, and the bytes under the paths matching to a specific file pattern.
• cp - cp command allows one to copy a file from source to destination. It supports copying multiple files in the case when the destination is a directory.

• Hadoop1 has less components and functionalities as compared to Hadoop2.
• Hadoop1 doesn't support non MapReducer model while Hadoop2 supports other distributed computing models as well like HBase, Spark, Message Passing Interface, Giraph, etc.
• Hadoop1 follows the concept of slots that can run only Map task or Reduce task only while Hadoop2 follows the containers concept in which it is possible to run generic tasks.
• There is  no windows support for Hadoop1 while Hadoop2 is supported in Windows also.

Inner bag is a kind of relation inside another bag. while an outer bag is a bag of tuples.

For example, (5,{(5,3,2),(5,4,4)}), this complete relation is an outer bag while {(5,3,2),(5,4,4)} is the inner bag.

Flume is a distributed system designed by Apache to store and transfer the large quantities of data from various web servers  to a centralized HDFS store like HBase for analysis.

Flume Event - Flume contains an array of byte payload which needs to be transported from source to destination with extra headers.

Flume Agent - Flume Agent receives the information from the client and transfers it to the destination (a sink or an agent). The flume agent has three main components - 

Source, Channel, and Sink. 

Source - A source receives the data from the data generators and moves the data to one or more channels in the form of Fluent events.

Channel - A channel acts as an intermediate that receives the data from the source and buffers it till it is consumed by all the sinks.

Sinks - A sink aggregates the data from the channel and stores it in the form of HBase, HDFS, Spark. 

The table data is stored in Apache Hive at location /user/house/warehouse in HDFS file system. Hadoop freelancers have to create these directories on HDFS file system before they use it. Here, you can find each and every directory of all the databases and all the sub-directories with the table name you make use of.