Query Surge is one of the solutions for Big Data testing. It ensures the quality of data quality and the shared data testing method that detects bad data while testing and provides an excellent view of the health of data. It makes sure that the data extracted from the sources stay intact on the target by examining and pinpointing the differences in the Big Data wherever necessary.
Hadoop has three common input formats:
Text Input Format – This is the default input format in Hadoop.
Sequence File Input Format – This input format is used to read files in a sequence.
Key-Value Input Format – This input format is used for plain text files (files broken into lines).
There does not exist any NameNode without data. If it is a NameNode then it should have some sort of data in it.
All the data nodes put together form a storage area i.e. the physical location of the data nodes is referred to as Rack in HDFS. The rack information i.e. the rack id of each data node is acquired by the NameNode. The process of selecting closer data nodes depending on the rack information is known as Rack Awareness.
The contents present in the file are divided into data block as soon as the client is ready to load the file into the hadoop cluster. After consulting with the NameNode, client allocates 3 data nodes for each data block. For each data block, there exists 2 copies in one rack and the third copy is present in another rack. This is generally referred to as the Replica Placement Policy.
Indexing process in HDFS depends on the block size. HDFS stores the last part of the data that further points to the address where the next part of data chunk is stored.
Virtualization is an essential stage in testing Big Data. The Latency of the virtual machine generates issues with timing. Management of images is not hassle-free too.
Rack Awareness is one of the popular big data interview questions. Rach awareness is an algorithm that identifies and selects DataNodes closer to the NameNode based on their rack information. It is applied to the NameNode to determine how data blocks and their replicas will be placed. During the installation process, the default assumption is that all nodes belong to the same rack.
Again, one of the most important big data interview questions. Here are six outlier detection methods:
* Extreme Value Analysis – This method determines the statistical tails of the data distribution. Statistical methods like ‘z-scores’ on univariate data are a perfect example of extreme value analysis.
* Probabilistic and Statistical Models – This method determines the ‘unlikely instances’ from a ‘probabilistic model’ of data. A good example is the optimization of Gaussian mixture models using ‘expectation-maximization’.
* Linear Models – This method models the data into lower dimensions. Proximity-based Models – In this approach, the data instances that are isolated from the data group are determined by Cluster, Density, or by the Nearest Neighbor Analysis.
* Information-Theoretic Models – This approach seeks to detect outliers as the bad data instances that increase the complexity of the dataset.
* High-Dimensional Outlier Detection – This method identifies the subspaces for the outliers according to the distance measures in higher dimensions.
Data science is a broad spectrum of activities involving analysis of Big Data, finding patterns, trends in data, interpreting statistical terms and predicting future trends. Big Data is just one part of Data Science. Though Data Science is a broad term and very important in the overall Business operations, it is nothing without Big Data.
The choice of language for a particular Big Data project depends on the kind of solution we want to develop. For example, if we want to do data manipulation, certain languages are good at the manipulation of data.
If we are looking for Big Data Analytics, we see another set of languages that should be preferred. As far as R and Python are concerned, both of these languages are preferred choices for Big Data. When we are looking into the visualization aspect of Big Data, R language is preferred as it is rich in tools and libraries related to graphics capabilities.
When we are into Big Data development, Model building, and testing, we choose Python.
R is more favourite among statisticians whereas developers prefer Python.
Next, we have Java as a popular language in the Big Data environment as the most preferred Big Data platform ‘Hadoop’ itself is written in java. There are other languages also popular such as Scala, SAS, and MATLAB.
There is also a community of Big Data people who prefer to use both R and Python. So we see that there are ways we can use a combination of both of these languages such as PypeR, PyRserve, rPython, rJython, PythonInR etc.
Thus, it is up to you to decide which one or a combination will be the best choice for your Big Data project.
Organizational Data, which is growing every data, ask for automation, for which the test of Big Data needs a highly skilled developer. Sadly, there are no tools capable of handling unpredictable issues that occur during the validation process. Lots of Focus on R&D is still going on.
The three modes are:
* Standalone mode – This is Hadoop’s default mode that uses the local file system for both input and output operations. The main purpose of the standalone mode is debugging. It does not support HDFS and also lacks custom configuration required for mapred-site.xml, core-site.xml, and hdfs-site.xml files.
* Pseudo-distributed mode – Also known as the single-node cluster, the pseudo-distributed mode includes both NameNode and DataNode within the same machine. In this mode, all the Hadoop daemons will run on a single node, and hence, the Master and Slave nodes are the same.
* Fully distributed mode – This mode is known as the multi-node cluster wherein multiple nodes function simultaneously to execute Hadoop jobs. Here, all the Hadoop daemons run on different nodes. So, the Master and Slave nodes run separately.
HDFS does not support modifications at arbitrary offsets in the file or multiple writers but files are written by a single writer in append only format i.e. writes to a file in HDFS are always made at the end of the file.
Replication factor is a property of HDFS that can be set accordingly for the entire cluster to adjust the number of times the blocks are to be replicated to ensure high data availability. For every block that is stored in HDFS, the cluster will have n-1 duplicated blocks. So, if the replication factor during the PUT operation is set to 1 instead of the default value 3, then it will have a single copy of data. Under these circumstances when the replication factor is set to 1 ,if the DataNode crashes under any circumstances, then only single copy of the data would be lost.
There are three main tombstone markers used for deletion in HBase. They are-
Family Delete Marker – For marking all the columns of a column family.
Version Delete Marker – For marking a single version of a single column.
Column Delete Marker – For marking all the versions of a single column.
There are three core methods of a reducer. They are-
setup() – This is used to configure different parameters like heap size, distributed cache and input data.
reduce() – A parameter that is called once per key with the concerned reduce task
cleanup() – Clears all temporary files and called only at the end of a reducer task.
Oozie, Ambari, Pig and Flume are the most common data management tools that work with Edge Nodes in Hadoop.
Edge nodes refer to the gateway nodes which act as an interface between Hadoop cluster and the external network. These nodes run client applications and cluster management tools and are used as staging areas as well. Enterprise-class storage capabilities are required for Edge Nodes, and a single edge node usually suffices for multiple Hadoop clusters.
A conventional way of a testing database does not need specialized environments due to its limited size whereas in the case of big data needs a specific testing environment.
HDFS indexes data blocks based on their sizes. The end of a data block points to the address of where the next chunk of data blocks get stored. The DataNodes store the blocks of data while NameNode stores these data blocks.
Listed in many Big Data Interview Questions and Answers, the best answer to this is –
Open-Source – Hadoop is an open-sourced platform. It allows the code to be rewritten or modified according to user and analytics requirements.
Scalability – Hadoop supports the addition of hardware resources to the new nodes.
Data Recovery – Hadoop follows replication which allows the recovery of data in the case of any failure.
Data Locality – This means that Hadoop moves the computation to the data and not the other way round. This way, the whole process speeds up.
Systems designed with multiple elements for processing a large amount of data needs to be tested with every single of these elements in isolation. E.g., how quickly the message is being consumed & indexed, MapReduce jobs, search, query performances, etc.
HDFS provides a distributed data copying facility through the DistCP from source to destination. If this data copying is within the hadoop cluster then it is referred to as inter cluster data copying. DistCP requires both source and destination to have a compatible or same version of hadoop.
It involves validating the rate at which map-reduce tasks are performed. It also consists of data testing, which can be processed in separation when the primary store is full of data sets. E.g., Map-Reduce tasks running on a specific HDFS.
The developer validates how fast the system is consuming the data from different sources. Testing involves the identification process of multiple messages that are being processed by a queue within a specific frame of time. It also consists of how fast the data gets into a particular data store, e.g., the rate of insertion into the Cassandra & Mongo database.
Performance testing consists of testing of the duration to complete the job, utilization of memory, the throughput of data, and parallel system metrics. Any failover test services aim to confirm that data is processed seamlessly in any case of data node failure. Performance Testing of Big Data primarily consists of two functions. First, is Data ingestion whereas the second is Data Processing
Commodity Hardware refers to inexpensive systems that do not have high availability or high quality. Commodity Hardware consists of RAM because there are specific services that need to be executed on RAM. Hadoop can be run on any commodity hardware and does not require any super computer s or high end hardware configuration to execute jobs.
Block - The minimum amount of data that can be read or written is generally referred to as a “block†in HDFS. The de
size of a block in HDFS is 64MB.
Block Scanner - Block Scanner tracks the list of blocks present on a DataNode and verifies them to find any kind of checksum errors. Block Scanners use a throttling mechanism to reserve disk bandwidth on the datanode.
i) Data Ingestion – The foremost step in deploying big data solutions is to extract data from different sources which could be an Enterprise Resource Planning System like SAP, any CRM like Salesforce or Siebel , RDBMS like MySQL or Oracle, or could be the log files, flat files, documents, images, social media feeds. This data needs to be stored in HDFS. Data can either be ingested through batch jobs that run every 15 minutes, once every night and so on or through streaming in real-time from 100 ms to 120 seconds.
ii) Data Storage – The subsequent step after ingesting data is to store it either in HDFS or NoSQL database like HBase. HBase storage works well for random read/write access whereas HDFS is optimized for sequential access.
iii) Data Processing – The ultimate step is to process the data using one of the processing frameworks like mapreduce, spark, pig, hive, etc.
The most common Input Formats defined in Hadoop are:
* Text Input Format- This is the default input format defined in Hadoop.
* Key Value Input Format- This input format is used for plain text files wherein the files are broken down into lines.
* Sequence File Input Format- This input format is used for reading files in sequence.
The best configuration for executing Hadoop jobs is dual core machines or dual processors with 4GB or 8GB RAM that use ECC memory. Hadoop highly benefits from using ECC memory though it is not low - end. ECC memory is recommended for running Hadoop because most of the Hadoop users have experienced various checksum errors by using non ECC memory. However, the hardware configuration also depends on the workflow requirements and can change accordingly.
This pattern of testing is to process a vast amount of data extremely resources intensive. That is why testing the architecture is vital for the success of any project on Big Data. A faulty planned system will lead to degradation of the performance, and the whole system might not meet the desired expectations of the organization. At least, failover and performance test services need a proper performance in any Hadoop environment.
Third and the last phase in the testing of bog data is the validation of output. Output files of the output are created & ready for being uploaded on EDW (warehouse at an enterprise level), or additional arrangements based on need. The third stage consists of the following activities.
* Assessing the rules for transformation whether they are applied correctly
* Assessing the integration of data and successful loading of the data into the specific HDFS.
* Assessing that the data is not corrupt by analyzing the downloaded data from HDFS & the source data uploaded.
MapReduce is the second phase of the validation process of Big Data testing. This stage involves the developer verifying the validation of the logic of business on every single systemic node and validating the data after executing on all the nodes, determining that:
* Proper Functioning, of Map-Reduce.
* Rules for Data segregation are being implemented.
* Pairing & Creation of Key-value.
* Correct Verification of data following the completion of Map Reduce.
The initial step in the validation, which engages in process verification. Data from a different source like social media, RDBMS, etc. are validated, so that accurate uploaded data to the system. We should then compare the data source with the uploaded data into HDFS to ensure that both of them match. Lastly, we should validate that the correct data has been pulled, and uploaded into a specific HDFS. There are many tools available, e.g., Talend, Datameer, which are mostly used for validation of data staging.
Along with processing capability, the quality of data is an essential factor while testing big data. Before testing, it is obligatory to ensure the data quality, which will be part of the examination of the database. It involves the inspection of various properties like conformity, perfection, repetition, reliability, validity, completeness of data, etc.
In Hadoop, engineers authenticate the processing of quantum of data used by the Hadoop cluster with supportive elements. Testing of Big data needs asks for extremely skilled professionals, as the handling is swift. Processing is three types namely Batch, Real-Time, & Interactive.
Hadoop distribution has a generic application programming interface for writing Map and Reduce jobs in any desired programming language like Python, Perl, Ruby, etc. This is referred to as Hadoop Streaming. Users can create and run jobs with any kind of shell scripts or executable as the Mapper or Reducers. The latest tool for Hadoop streaming is Spark.
This is one of the most important Big Data interview questions to help the interviewer gauge your knowledge of commands.
To start all the daemons:
./sbin/start-all.sh
To shut down all the daemons:
./sbin/stop-all.sh
The JPS command is used for testing the working of all the Hadoop daemons. It specifically tests daemons like NameNode, DataNode, ResourceManager, NodeManager and more.
(In any Big Data interview, you’re likely to find one question on JPS and its importance.)
Hadoop applications have wide range of technologies that provide great advantage in solving complex business problems.
Core components of a Hadoop application are-
1) Hadoop Common
2) HDFS
3) Hadoop MapReduce
4) YARN
Data Access Components are - Pig and Hive
Data Storage Component is - HBase
Data Integration Components are - Apache Flume, Sqoop, Chukwa
Data Management and Monitoring Components are - Ambari, Oozie and Zookeeper.
Data Serialization Components are - Thrift and Avro
Data Intelligence Components are - Apache Mahout and Drill.
FSCK stands for Filesystem Check. It is a command used to run a Hadoop summary report that describes the state of HDFS. It only checks for errors and does not correct them. This command can be executed on either the whole system or a subset of files.
This is yet another Big Data interview question you’re most likely to come across in any interview you sit for.
Commodity Hardware refers to the minimal hardware resources needed to run the Apache Hadoop framework. Any hardware that supports Hadoop’s minimum requirements is known as ‘Commodity Hardware.’
Data that can be stored in traditional database systems in the form of rows and columns, for example, the online purchase transactions can be referred to as Structured Data. Data that can be stored only partially in traditional database systems, for example, data in XML records can be referred to as semi-structured data. Unorganized and raw data that cannot be categorized as semi-structured or structured data is referred to as unstructured data. Facebook updates, tweets on Twitter, Reviews, weblogs, etc. are all examples of unstructured data.
Structured data: Schema-based data, Datastore in SQL, Postgresql databases etc
Semi-structured data : Json objects , json arrays, csv , txt ,xlsx files,web logs ,tweets etc
Unstructured data : Audio, Video files, etc
Big data analysis is helping businesses differentiate themselves – for example Walmart the world’s largest retailer in 2014 in terms of revenue - is using big data analytics to increase its sales through better predictive analytics, providing customized recommendations and launching new products based on customer preferences and needs. Walmart observed a significant 10% to 15% increase in online sales for $1 billion in incremental revenue. There are many more companies like Facebook, Twitter, LinkedIn, Pandora, JPMorgan Chase, Bank of America, etc. using big data analytics to boost their revenue.
Now that we’re in the zone of Hadoop, the next Big Data interview question you might face will revolve around the same.
The HDFS is Hadoop’s default storage unit and is responsible for storing different types of data in a distributed environment.
Big data analysis has become very important for the businesses. It helps businesses to differentiate themselves from others and increase the revenue. Through predictive analytics, big data analytics provides businesses customized recommendations and suggestions. Also, big data analytics enables businesses to launch new products depending on customer needs and preferences. These factors make businesses earn more revenue, and thus companies are using big data analytics. Companies may encounter a significant increase of 5-20% in revenue by implementing big data analytics. Some popular companies those are using big data analytics to increase their revenue is – Walmart, LinkedIn, Facebook, Twitter, Bank of America etc.
Hadoop is an open-source framework for storing, processing, and analyzing complex unstructured data sets for deriving insights and intelligence.
In the case of processing of the significant amount of data, performance, and functional testing is the primary key to performance. Testing is a validation of the data processing capability of the project and not the examination of the typical software features.
Big Data means a vast collection of structured and unstructured data, which is very expansive & is complicated to process by conventional database and software techniques. In many organizations, the volume of data is enormous, and it moves too fast in modern days and exceeds the current processing capacity. Compilation of databases that are not being processed by conventional computing techniques, efficiently. Testing involves specialized tools, frameworks, and methods to handle these massive amounts of datasets. Examination of Big data is meant to the creation of data and its storage, retrieving of data and analysis them which is significant regarding its volume and variety of speed.