What am I missing in WSO2 API Manager 3.x datasource configuration?

5 min readDec 5, 2020

In this article I will be showing you how the datasources can be configured in WSO2 API Manager 3.x products because this approach is different than what we had in our 2.x family of products configuring the master-datasources.xml.

The main intention of this change was to allow the users to make changes in a single location reducing the hassle of having to deal with multiple configuration files. Therefore you need to edit only the deployment.toml file in <APIM_HOME>/repository/conf directory.

👇

At the end I will be providing you with a Bonus Tip 😍 which will help you do any configuration on your own without seeking help from someone else. 🥳

Main datasources that can be configured in WSO2 API Manager 3.x

WSO2AM_DB
WSO2_SHARED_DB
WSO2_CARBON_DB
WSO2USER_DB
WSO2CONFIG_DB
WSO2_MB_STORE_DB

Note : You might be wondering what happened to WSO2AM_STATS_DB. This datasource is no longer needed in WSO2 API Manager product as you should be configuring it in WSO2 API Manager Analytics instead.

Datasources configured by default

There are 2 datasource configurations that are already available in the deployment.toml file.

They are for WSO2AM_DB and WSO2_SHARED_DB.

WSO2AM_DB is one of the main datasources in WSO2 API Manager and is used for storing everything related to API Manager artifacts. This is configured by the following lines in the deployment.toml file in the product. See the default configuration.

[database.apim_db]
type = "h2"
url = "jdbc:h2:./repository/database/WSO2AM_DB;AUTO_SERVER=TRUE;DB_CLOSE_ON_EXIT=FALSE"
username = "wso2carbon"
password = "wso2carbon"

WSO2_SHARED_DB is the shared datasource for both user and registry related data. This is the next important datasource in WSO2 API Manager. See the default configuration of this datasource below.

[database.shared_db]
type = "h2"
url = "jdbc:h2:./repository/database/WSO2SHARED_DB;DB_CLOSE_ON_EXIT=FALSE"
username = "wso2carbon"
password = "wso2carbon"

Then comes the datasources that you might want to configure yourself depending on the need. Otherwise they are not needed to be specifically configured.

WSO2_CARBON_DB

You will not see this datasource configuration in the default settings of the deployment.toml file specifically in WSO2 API Manager 3.0.0, because it is sufficient to use the already configured embedded h2 datasource for this. But if you need to configure it to point to a datasource of your choice, use the following configs and add it(if not already available) to the <APIM_HOME>/repository/conf/deployment.toml file. Else edit the existing configuration.

[database.local]
type = "h2"
url = "jdbc:h2:./repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE;MVCC=TRUE"
username = "wso2carbon"
password = "wso2carbon"

The changes will be applied on server startup.

Note : But if you have multiple server nodes of WSO2 API Manager running, make sure that each node has its own WSO2_CARBON_DB in the case that you will change the embedded h2 database to one of your choice.

WSO2USER_DB

This is the datasource where user related data are stored (an optional datasource). If this datasource is not configured, all user related data will be stored in the WSO2_SHARED_DB.

So use this if you need a separate datasource for users. Then, only the registry or governance related data will be stored in the WSO2_SHARED_DB.

Add the following lines to the deployment.toml file in order to configure a separate WSO2USER_DB.

[database.user]
type = "h2"
url = "jdbc:h2:./repository/database/WSO2USER_DB;DB_CLOSE_ON_EXIT=FALSE"
username = "wso2carbon"
password = "wso2carbon"

WSO2CONFIG_DB

This is the datasource used for config registry (an optional datasource). If this datasource needs to be configured, add the following lines to the deployment.toml file and modify it with your data.

[database.config]
type = "h2"
url = "jdbc:h2:./repository/database/WSO2CONFIG_DB;DB_CLOSE_ON_EXIT=FALSE"
username = "wso2carbon"
password = "wso2carbon"

WSO2_MB_STORE_DB

This is the datasource used for the message broker database. You can use the embedded h2 database as the WSO2_MB_STORE_DB even in production. Hence it is not required to configure this database separately.

You can even add your own datasources into the product

You can use custom datasources if required with the extension capabilities that WSO2 API Manager provides.

Use the following approach when configuring one or more custom datasources.

Note : You can replace my_custom_db with your preferred name. Make sure you provide all necessary parameters to the configuration below such as the id, url, type, driver, username, password etc. The “id” can be one of your choice.

[datasource.my_custom_db]
type = "h2"
id = "CUSTOM_DB"
url = "jdbc:h2:./repository/database/MYCUSTOM_DB"
username = "wso2carbon"
password = "wso2carbon"
validationQuery = "SELECT 1"
driver = "org.h2.Driver"

If you have more than one custom datasource, repeatedly add the above configuration in the deployment.toml file and modify it with the configuration you need.

Now after doing all the above changes as needed, you should restart the server for the changes to apply. Then you can check the <APIM_HOME>/repository/conf/datasources/master-datasources.xml file content being updated with the configurations you have provided in the deployment.toml file.

Bonus Tip 😃

In <APIM_HOME>/repository/resources/conf/templates/repository/conf/datasources directory you will find a master-datasources.xml.j2 file. It is the file that is used to template the original master-datasources.xml file available in <APIM_HOME>/repository/conf directory.

If you look closely at master-datasources.xml.j2, you will see how a particular datasource is templated. Let me take the WSO2AM_DB datasource template to explain it to you.

<datasource>
    <name>WSO2AM_DB</name>
    <description>The datasource used for API Manager
database</description>
    <jndiConfig>
        <name>jdbc/WSO2AM_DB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>{{database.apim_db.url}}</url>
            <username>{{database.apim_db.username}}</username>
            <password>{{database.apim_db.password}}</password>
            <driverClassName>{{database.apim_db.driver}}
            </driverClassName>
            <validationQuery>{{database.apim_db.validationQuery}} 
            </validationQuery>
           {% for property_name,property_value in database.apim_db.pool_options.items() %}
                  <{{property_name}}>{{property_value}}</{{property_name}}>
           {% endfor %}
       </configuration>
   </definition>
</datasource>

Look at the above template and try to map it to the deployment.toml configuration below.

[database.apim_db]
type = "h2"
url = "jdbc:h2:./repository/database/WSO2AM_DB;AUTO_SERVER=TRUE;DB_CLOSE_ON_EXIT=FALSE"
username = "wso2carbon"
password = "wso2carbon"

{{database.apim_db.url}} = The url element under [databases.apim_db] configuration.
{{database.apim_db.username}} = The username element under [databases.apim_db] configuration.
{{database.apim_db.password}} = The password element under [databases.apim_db] configuration.

Then what about {{database.apim_db.driver}}…?

If we have not specified this element in the deployment.toml file, there is a place that we take the default configuration value for it.

It is…

<APIM_HOME>/repository/resources/conf/default.json

Following is the entry we can see in the default.json file.

"database.apim_db.driver": "org.h2.Driver",

Note : We shouldn’t be editing the default.json file. Instead, all configuration changes should be done in the deployment.toml file. The reason being, if you take a WUM update for the product, all changes that you have done in default.json will be replaced if it contains changes that has come through a fix. You don’t want this happening right? 🙂 So avoid changing the default.json.

Okay let’s come back…

Now look at this line in the WSO2AM_DB configuration in the master-datasources.xml.j2 file above.

{% for property_name,property_value in database.apim_db.pool_options.items() %}

What is this for loop for?

If you have any additional properties that should be configured for the datasource, you have to specify them in the deployment.toml file under the configuration [database.apim_db.pool_options] in the form of key value pairs. This is specific for the WSO2AM_DB. Other datasource configurations also have the same capability. See the sample configuration below.

[database.apim_db.pool_options]
minIdle = "10"
removeAbandoned = true
removeAbandonedTimeout = "180"
logAbandoned = true
maxActive = "150"

With this knowledge, you will be able to specify any property for a datasource that you need to configure in WOS2 API Manager by simply looking at the master-datasources.xml.j2 template file. 👏

😇 This can successfully be applied to any other configuration you see in WSO2 API Manager. You have to simply look at the relevant template file which can be found from the location <APIM_HOME>/repository/resources/conf/templates/repository/conf and add the required configuration in the deployment.toml file.

Cheers !