Batch Processing in Mule | Batch Processing Example
Mule allows you to process messages in batches.It splits the large messages into individual records that are processed asynchronously within batch jobs.Within an application, you can initiate a batch job scope, which is a block of code that splits messages into individual records, performs actions upon each record, then reports on the results and potentially pushes the processed output to other systems or queues.
For example, you can use batch processing when :
1.Synchronizing data sets between business applications, such as syncing contacts between NetSuite and Salesforce.
2.Extracting, transforming and loading (ETL) information into a target system, such as uploading data from a flat file (CSV) to Hadoop.
3.Handling large quantities of incoming data from an API into a legacy system.
Batch processing has four stages :
Input Phase: This is an optional part of the batch job that can be used to retrieve the source data using any message source or inbound connector. It also comprises of message processors to transform the source data before it is ready for processing.
Load and Dispatch: In this phase, the payload generated in the Input phase or provided to the batch from the caller flow is turned into a collection of records. It also creates a job instance for processing records. The collection is then sent through the collection-splitter to queue individual records for processing.
Process: This is the required phase where the actual processing of every record occurs asynchronously.
Each record from the input queue is processed through the first step and sent back to the queue after processing of the first step completes.
Records that are processed in the first step are then passed through the second step and sent back to the queue after processing of the second step completes.
Mule continues this until all records are passed through each step.At the step level, you can also specify what type of records each step should accept.
Processing of records through the next step does not wait for the previous step to finish processing all records. Mule manages the state of each record while it moves back and forth between the queue and steps.
On Complete: In this final but optional phase, It holds information such as the number of records loaded, processed, failed, succeeded.
flow of batch processing :
Batch Execute : It is a mule component which accumulates records into chunks to process bulk upserts to external source or service.
In our example it calls the batch flow test_batchprocessBatch
Batch Execute configuration :
All loggers are configured with #[payload]
Batch Commit scope configuration :
URL : http://localhost:8085/api/batch Method : POST
Input :
Output :
Logger output :
XML project code :
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:quartz="http://www.mulesoft.org/schema/mule/quartz" xmlns:json="http://www.mulesoft.org/schema/mule/json" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:batch="http://www.mulesoft.org/schema/mule/batch" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/json http://www.mulesoft.org/schema/mule/json/current/mule-json.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd
http://www.mulesoft.org/schema/mule/quartz http://www.mulesoft.org/schema/mule/quartz/current/mule-quartz.xsd">
<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8085" basePath="/api" doc:name="HTTP Listener Configuration"/>
<quartz:connector name="Quartz" validateConnections="true" doc:name="Quartz"/>
<flow name="test_batchprocessFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/batch" doc:name="HTTP"/>
<byte-array-to-object-transformer returnClass="java.lang.Object" doc:name="Byte Array to Object"/>
<json:object-to-json-transformer doc:name="Object to JSON"/>
<json:json-to-object-transformer returnClass="java.util.ArrayList" doc:name="JSON to Object"/>
<batch:execute name="test_batchprocessBatch" doc:name="test_batchprocessBatch"/>
<json:object-to-json-transformer doc:name="Object to JSON"/>
</flow>
<batch:job name="test_batchprocessBatch">
<batch:process-records>
<batch:step name="Batch_Step">
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</batch:step>
<batch:step name="Batch_Step1">
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
<batch:commit size="2" doc:name="Batch Commit">
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</batch:commit>
</batch:step>
</batch:process-records>
<batch:on-complete>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</batch:on-complete>
</batch:job>
</mule>
For example, you can use batch processing when :
1.Synchronizing data sets between business applications, such as syncing contacts between NetSuite and Salesforce.
2.Extracting, transforming and loading (ETL) information into a target system, such as uploading data from a flat file (CSV) to Hadoop.
3.Handling large quantities of incoming data from an API into a legacy system.
Batch processing has four stages :
Input Phase: This is an optional part of the batch job that can be used to retrieve the source data using any message source or inbound connector. It also comprises of message processors to transform the source data before it is ready for processing.
Load and Dispatch: In this phase, the payload generated in the Input phase or provided to the batch from the caller flow is turned into a collection of records. It also creates a job instance for processing records. The collection is then sent through the collection-splitter to queue individual records for processing.
Process: This is the required phase where the actual processing of every record occurs asynchronously.
Each record from the input queue is processed through the first step and sent back to the queue after processing of the first step completes.
Records that are processed in the first step are then passed through the second step and sent back to the queue after processing of the second step completes.
Mule continues this until all records are passed through each step.At the step level, you can also specify what type of records each step should accept.
Processing of records through the next step does not wait for the previous step to finish processing all records. Mule manages the state of each record while it moves back and forth between the queue and steps.
On Complete: In this final but optional phase, It holds information such as the number of records loaded, processed, failed, succeeded.
flow of batch processing :
Batch Execute : It is a mule component which accumulates records into chunks to process bulk upserts to external source or service.
In our example it calls the batch flow test_batchprocessBatch
Batch Execute configuration :
All loggers are configured with #[payload]
Batch Commit scope configuration :
URL : http://localhost:8085/api/batch Method : POST
Input :
Output :
Logger output :
XML project code :
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:quartz="http://www.mulesoft.org/schema/mule/quartz" xmlns:json="http://www.mulesoft.org/schema/mule/json" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:batch="http://www.mulesoft.org/schema/mule/batch" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/json http://www.mulesoft.org/schema/mule/json/current/mule-json.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd
http://www.mulesoft.org/schema/mule/quartz http://www.mulesoft.org/schema/mule/quartz/current/mule-quartz.xsd">
<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8085" basePath="/api" doc:name="HTTP Listener Configuration"/>
<quartz:connector name="Quartz" validateConnections="true" doc:name="Quartz"/>
<flow name="test_batchprocessFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/batch" doc:name="HTTP"/>
<byte-array-to-object-transformer returnClass="java.lang.Object" doc:name="Byte Array to Object"/>
<json:object-to-json-transformer doc:name="Object to JSON"/>
<json:json-to-object-transformer returnClass="java.util.ArrayList" doc:name="JSON to Object"/>
<batch:execute name="test_batchprocessBatch" doc:name="test_batchprocessBatch"/>
<json:object-to-json-transformer doc:name="Object to JSON"/>
</flow>
<batch:job name="test_batchprocessBatch">
<batch:process-records>
<batch:step name="Batch_Step">
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</batch:step>
<batch:step name="Batch_Step1">
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
<batch:commit size="2" doc:name="Batch Commit">
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</batch:commit>
</batch:step>
</batch:process-records>
<batch:on-complete>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</batch:on-complete>
</batch:job>
</mule>
