How to use WSO2 EI File connector

In this article I’m going to show you how we can do various operations in the file system using WSO2 EI File connector. I would like to explain some of the file connector operations using the following use case.

As shown in the diagram it contains following scenarios

1. Copy single/multiple files in the Source directory to Stage directory
2. Read all the files one by one from Stage directory and stream the file to an external API
3. Once the API call is success Move the file to the Destination directory.

Using these scenarios I will explain how to use isFileExist, Copy, Search, Read and Move operations in File connector.

Before moving to the file connector operation first we need to download file connector from Connector store. You can download a version you like and please make sure to check the compatibility with the WSO2 EI you are using.

Then create an Integration studio Multi maven project and add fileconnector-connector-2.0.23.zip file (Here I’m using version 2.0.23). More information on adding file connector you can find in documentation.

File connector operations

isFileExist

First we check whether the source directory has any files, we can check for any file with any extension or a specific extension.

Ex:-

<fileconnector.isFileExist>
    <source>/mnt/file-connector-example/source</source>
    <filePattern>*.csv</filePattern> 
</fileconnector.isFileExist>

Here I have put the file pattern to only check .csv files in the directory we can put * If we do not need any specific file pattern, there are other parameters as well if we need to narrow it down. You can find more details on this operation in here.

This will return json with the attribute ‘fileExist’. Following property will give the result.

<property expression="json-eval($.fileExist)" name="isFileExist" scope="default" type="STRING"/>

Copy

Suppose we have multiple files in Source directory and we use Copy operation to copy all the files from source to stage directory (refer to the diagram).

<fileconnector.copy>
  <source>/mnt/file-connector-example/Source</source><destination>/mnt/file-connector-example/Stage</destination>
<filePattern>*.csv</filePattern><includeParentDirectory>false</includeParentDirectory
</fileconnector.copy>

Source and destination parameters are mandatory. We can specify the file pattern we need to copy. In this example we are copy all the files from source to destination

Note :- Please note that if we need to copy a specific file from source to destination, we can specify the full file path of the source file and destination file location.

Ex: - 
source - /mnt/file-connector-example/Source/file.csv
destination -   /mnt/file-connector-example/Stage/file.csv

Documentation for Copy operation.

After copy this will return response json with attribute “success”

<property expression="json-eval($.success)" name="isSuccess" scope="default" type="STRING"/>

Now we have copy all the files from “source” to “stage” directory.

Then we need to use read operation to read each file and stream to the API. In order to use read operation we need to set the absolute file path of each file.

Now in our use case it comes a little tricky since we have multiple files copied to the “Stage” directory but in order to set the absolute file path of each file, at this time we do not have the name of each file, we only have the path to the stage directory. Therefore how do we get the absolute file path of each file ? We can use Search operation to get the list of absolute file paths.

Search

Search operation returns the absolute file path of the list of files in the mentioned source directory.

<fileconnector.search>
<source>/mnt/file-connector-example/Stage</source><filePattern>*.csv</filePattern>. <recursiveSearch>false</recursiveSearch>
</fileconnector.search>

Search operations give the absolute file path of each file but we need to be careful here since if there is a single file this response the absolute file path as response not an array with a single value. If only there are multiple files it response array of file paths.

So how do we get to know if there is a single file or multiple files. I’ve used something like below to verify this.

<property expression="json-eval($.result.file)" name="file-search-response" scope="default" type="STRING"/>
<property expression="json-eval($.result.file[0])" name="file-search-response-first-element" scope="default" type="STRING"/>

If $.result.file[0] has a value it means response was an array, if not it is not an array only the single file.

Use a filter mediator to check this as below.

<filter description="Check file search multiple or single" regex="[^\n]+" source="get-property('file-search-response-first-element')">
<then>
  <iterate expression="//ns:file" id="filesIterate" sequential="true" xmlns:ns="http://org.wso2.esbconnectors.FileConnector" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
     <target>
        <sequence>
        </sequence>
     </target>
  </iterate>
</then>
<else>
</else>
</filter>

Now we have absolute file paths we can use Read operation to read and stream to API.

Read

We need to do a configuration change before using Read operation as below.

<EI_HOME>/repository/conf/axis2/axis2.xml file:
Add <messageFormatter contentType=”application/file”
 class=”org.wso2.carbon.relay.ExpandingMessageFormatter”/> under message formatters.
Add <messageBuilder contentType=”application/file”
 class=”org.apache.axis2.format.BinaryBuilder”/> under message builders.

Now we can use Read operation with the source as absolute file path and then call the API.

<fileconnector.read>
<source>/mnt/file-connector-example/Stage/file1.csv</source>
<contentType>application/file</contentType>
<filePattern>*.csv</filePattern>
<streaming>true</streaming>
</fileconnector.read>
<call>
<endpoint key="stream-http-ep"/>
</call>

More details on Read operation can find here.

Now after streaming to the API we can use the Move operation to move the file into the “Destination” directory.

Move

<fileconnector.move>
   <source>/mnt/file-connector-example/Stage/file1.csv</source>. <destination>/mnt/file-connector-example/destination/file1.csv</destination><includeParentDirectory>false</includeParentDirectory><filePattern>*.csv</filePattern>
</fileconnector.move>

We can use the Move operation to either move each file from Stage to Destination directory one by one or as all.

You can find the sequence for whole flow below.

References

WSO2 File Connector documentation

EI File connector operations