Using reserved/special characters in request URLs sent to WSO2 EI/APIM

With regard to API resources, when legacy clients that cannot encode request URLs, send API requests to the WSO2 API-Manager (APIM) or the Enterprise Integrator (EI) with the following special characters, the synapse engine does not capture the value and provides the error “Method not allowed for given API resource” or “Resource not found”.

‘:’, ‘/’, ‘?’, ‘#’, ‘[‘, ‘]’, ‘@’, ‘!’, ‘$’, ‘&’, ‘\’’, ‘(‘, ‘)’, ‘*’, ‘+’, ‘,’, ‘;’, ‘=’

This is a result of the findResource method (in the URITemplateBasedDispatcher class [1]) of the synapse engine checking for the above-reserved characters while mapping the request to the resource (in the SimpleStringExpression class [2] ).

This is actually the expected behaviour as per the following RFC [3] and the synapse engine expects these special characters in the percent-encoded form (URL encoding).

For example, let’s take the following API resource.

<api context="/testContext" name="testAPI" xmlns="http://ws.apache.org/ns/synapse"> <resource methods="GET" uri-template="/{testParam1}/{testParam2}">
...

The above API resource accepts two URI parameters: testParam1 and testParam2. If the client needs to send the following two values for the above parameters, it needs to conduct a URL encoding as follows.

testParam1 = 123:456
testParam2 = abc,xyz

The acceptable request with URL encoded parameters:

http://localhost:8280/testContext/123%3A456/abc%2Cxyz

And the following request with the plain values for the above parameters will fail to find the resource.

http://localhost:8290/testContext/123:456/abc,xyz

Solution 1: Using a wildcard pattern (uri-template=”/*”>) for the API resource should allow the API requests with special characters to pick the resource. But it will leave the API vulnerable for open requests. Therefore it’s expected that the API client conducts the encoding before sending the request to the APIM or EI servers.

However, there can be cases where legacy clients that send requests to the WSO2 servers cannot encode the request URIs.

Solution 2: The easiest approach to address the above case is adding a ‘+’ to the beginning of the variable name of the API resource as follows. It will capture the URI parameters which contain general delimiters that are not encoded.

<api context="/testContext" name="testAPI" xmlns="http://ws.apache.org/ns/synapse"> <resource methods="GET" uri-template="/{+testParam1}/{+testParam2}">
...

The above parameters received in the requests can be logged as follows.

<log level="custom">
            <property name="testParam1" expression="get-property('uri.var.testParam1')"/>
            <property name="testParam2" expression="get-property('uri.var.testParam2')"/>
            </log>

Still, there will be a vulnerability in the above approach because the above API resource accepts an unlimited number of URI parameters with the above tweak to the uri-template. In other words, even the following request will be accepted by the above resource.

http://localhost:8290/testContext/123:456/abc,xyz/test1/test2/.../testN

If there’s an extreme edge case where the request URL contains parameters such as % (which is used to indicate percent-encoding) a custom synapse handler to encode these values can be used. This will address the above case of limiting the number of URI parameters as well.

Solution 3: The following is a sample custom synapse handler [4] that can be used as a solution to conduct the encoding of configured characters (these can be configured through the deployment.toml file as synapse properties as explained in [5]) prior to dispatching the request to the synapse engine. This was tested using APIM-3.1.0 and the complete implementation including the instructions to enable the handler are available in [5].

public class RequestEncodingHandler extends AbstractSynapseHandler {
    private static final Log log = LogFactory.getLog(RequestEncodingHandler.class);
private static Properties synapseProperties = SynapsePropertiesLoader.loadSynapseProperties();
private static String escape_encode_chars_property = SynapsePropertiesLoader.loadSynapseProperties().
            getProperty("escape_encode_chars");
private static char[] escapedChars = escape_encode_chars_property.toCharArray();
protected boolean isEscaped(char ch) {
        for (char escapedChar : escapedChars) {
            if (ch == escapedChar) {
                return true;
            }
        }
        return false;
    }
public boolean handleRequestInFlow(MessageContext messageContext) {
if (log.isDebugEnabled()) {
            log.debug(this.getClass().getName() + " handleRequestInFlow ");
        }
org.apache.axis2.context.MessageContext axis2MessageContext
                = ((Axis2MessageContext) messageContext).getAxis2MessageContext();
        String rest_url_postfix = (String) axis2MessageContext.getProperty("REST_URL_POSTFIX");
int length = rest_url_postfix.length();
        StringBuffer buffer = new StringBuffer();
        for (int i = 0; i < length; i++) {
char ch = rest_url_postfix.charAt(i);
            if (isEscaped(ch)) {
                try {
                    buffer.append(URLEncoder.encode(String.valueOf(ch), "UTF-8"));
                } catch (UnsupportedEncodingException e) {
                    e.printStackTrace();
                }
            } else {
                buffer.append(ch);
            }
}
String encoded_rest_url_postfix = buffer.toString();
axis2MessageContext.setProperty("TransportInURL", "/" + encoded_rest_url_postfix);
return true;
    }
public boolean handleRequestOutFlow(MessageContext messageContext) {
        return true;
    }
public boolean handleResponseInFlow(MessageContext messageContext) {
        return true;
    }
public boolean handleResponseOutFlow(MessageContext messageContext) {
        return true;
    }
}

Notes:

1. As mentioned in [5] the “escape_encode_chars_property” is retrieved from a synapse property configured in the deployment.toml file to specify the special character/characters that will be included in the request URLs. And the handler will encode the URI fragments that include these character/characters.
2. Handlers or customization should be used only if it’s absolutely required to address a particular requirement. It’s extremely important that performance tests and capacity planning is done while including these sorts of customizations.

Solution 4: The most suitable approach depending on the deployment would be to use a load balancer or a reverse proxyfronting the WSO2 API-Manager/Enterprise Integrator servers and utilizing the load balancer or the reverse proxy server to conduct URL encoding prior to invoking the APIs from the WSO2 synapse engine.

Cheers.

[1]. https://github.com/wso2/wso2-synapse/blob/master/modules/core/src/main/java/org/apache/synapse/rest/dispatch/URITemplateBasedDispatcher.java

[2]. https://github.com/wso2/wso2-synapse/blob/master/modules/commons/src/main/java/org/apache/synapse/commons/templates/uri/parser/SimpleStringExpression.java

[3]. https://tools.ietf.org/html/rfc3986#page-12

[4]. https://ei.docs.wso2.com/en/latest/micro-integrator/develop/customizations/creating-synapse-handlers/

[5]. https://github.com/lkokila/RequestEncodingSynapseHandler

The HOTEL TRANSYLVANIA image to depict the special characters was extracted from [6]. :)

[6]. http://aselenatorsview.blogspot.com/2018/07/hotel-transylvania-3-summer-vacation.html