Chidi Okwudire IT Professional. ERP Enthusiast. NetSuite Certified (Administrator, SuiteCloud Developer II, and ERP Consultant). Passionate About Empowerment Through Knowledge Sharing. Always Eager to Learn.

Learn Pattern for Delivering Webhooks with Dynamic Payloads to NetSuite Using Celigo

3 min read

Celigo, like most iPaaS solutions, is strongly field-oriented. That means, given a (multi-field) input from the data source, Celigo can map each field to the corresponding field in the destination. For example, if you’re importing an Account from Salesforce (source) into a Vendor in NetSuite (destination), you can easily map fields across both systems (e.g. Salesforce Account Name -> NetSuite Vendor Name), possibly applying transformations as needed. This process is generally described as an ETL – Extract, Transform, Load. However, there are cases where you simply want to use Celigo to push data as-is from source to destination. This article explains one way of doing that.

Context

In a recent integration project, we resorted to using Celigo as a middleware to receive incoming webhooks from a third-party order management system and pass them to NetSuite. This was after we failed to establish a direct integration with NetSuite. (With a Suitelet, we received HTTP error code 500 and the vendor of the third-party application was hesitant to share the “raw” errors and server logs to enable us to debug the situation. A Restlet was not an option either because the third-party application was unable to deliver webhooks to authenticated endpoints which is a requirement when working with Restlets.) Since the client was already using Celigo for other integrations, it was a logical choice. Moreover, Celigo offers queuing facilities that come in handy in such situations.

Different events in the third-party system produced webhooks with event-specific payloads that all needed to be processed correctly in NetSuite. Therefore, we envisioned an architecture in which all incoming webhooks, regardless of the actual (dynamic) payload, will be pushed to a single custom record in NetSuite for further processing. As such, we needed a way to pass the payload from the incoming webhooks as-is to NetSuite.

For the sake of illustration, consider the following simplified payload. Notice the eventType field which determines the structure of the rest of the payload.

{
  "eventType": "1",
  "orderNumber": 12345,
  "customer": 54321,
  "status": "LOADED"
}

Challenge

When creating a Celigo flow with a Webhook Listener as the data source, we need to tell Celigo what the payload looks like. Celigo then makes each field available for mapping to fields on the NetSuite side. However, as stated, our payload is dynamic, and we want to pass the full payload to NetSuite instead of mapping individual fields. Thus, we needed to move away from the classical field-to-field mapping and figure out a way to pass the entire incoming payload into a single field in the NetSuite custom record.

Solution

After some interaction with Celigo’s support team, it became obvious that we needed a custom transformation to realize the desired outcome. Gladly, Celigo allows us to use JavaScript to customize integrations when necessary.

Here’s a step-by-step description of the process. Basic understanding of Celigo is assumed. As such, we don’t go into details on how to set up flows, add script files, etc. If you’re not grounded in these subjects, at the time of writing, Celigo offers a very robust training package that includes a test environment for free. Head over to Celigo University to learn more.

Step 1

In the Webhooks Listener, define your incoming data in a generic way:

Celigo webhook listener configuration illustrated

As illustrated above, we simply tell Celigo that we expect a JSON object with a single element called “payload” (you may call it anything you want provided you are consistent in subsequent steps).

Step 2

Define a simple input transformation in JavaScript using the transform hook.

Celigo webhook input transformation script

All we’re doing is wrapping the actual webhook data inside a dummy “payload” object so that Celigo can pick it up as per the mapping we defined.

Step 3

Map the placeholder “payload” field to whatever field you need on the NetSuite side (conveniently called Payload in our example).

Note that Celigo might show payload.payload on the source side as it tries to be smart and update the data structure based on the transformation we defined. However, that’s not what we want. Simply type in payload as the source field as shown below.

Celigo webhook NetSuite mapping

Be sure to specify the data type in your mapping as “String”!

By default, no data type is explicitly selected and during my tests, the data was not sent to NetSuite until I explicitly set the field type to “String” as illustrated below. It’s unclear if this is a bug in Celigo or expected behavior. In any case, the message is that you should be explicit about the data type.

Celigo mapping how to set data type to string

Due to the transformation in Step 2, the ‘Payload’ field on the NetSuite side will now contain the full payload received from the incoming webhook as desired!


That’s pretty much all there is to it. However, for the benefit of the reader who does not yet understand why this works, let me dive a little bit deeper. When this flow is run, the transformation will behave as follows:

Input before transformation:

{
  "eventType": "1",
  "orderNumber": 12345,
  "customer": 54321,
  "status": "LOADED"
}

Output after transformation:

{
  "payload": {
    "eventType": "1",
    "orderNumber": 12345,
    "customer": 54321,
    "status": "LOADED"
  }
}

In the mapping step, Celigo will now look for a field named payload in our transformed input data whose value will be the entire original webhook data, thanks to the transformation.

Note Regarding JSON Serialization

In an earlier implementation of the above solution, I recall getting an error (which I, regrettably, did not preserve) that I fixed by applying the jsonSerialize function in Celigo to achieve the desired results as illustrated below:

JSON serialize config illustrated

However, while preparing for this article, it worked without that step! Celigo is constantly evolving and I suspect Celigo has implemented logic to automatically stringify any incoming JSON. In any case, if you run into errors, remember jsonSerialize. Alternatively, you could simply add it right away as it makes sense to convert your JSON input to a string before transmitting it to NetSuite.


Hope you’ve found this article useful and this trick comes in handy the next time you need to get dynamic webhook data into NetSuite using Celigo.

NetSuite Insights is on a mission to raise the standards around NetSuite practices, one insight at a time. If that resonates with you, check out how you can become an author/collaborator here.

Also, consider subscribing to our no-nonsense email list to get these insights delivered to your inbox as soon as they’re published. Sometimes, ignorance is a choice. Choose wisely!

Chidi Okwudire IT Professional. ERP Enthusiast. NetSuite Certified (Administrator, SuiteCloud Developer II, and ERP Consultant). Passionate About Empowerment Through Knowledge Sharing. Always Eager to Learn.

Leave a Reply

Your email address will not be published. Required fields are marked *

You deserve to know as soon as we share a new NetSuite Insight.
We won't disturb you with anything else or share/sell your data.