Celigo, like most iPaaS solutions, is strongly field-oriented. That means, given a (multi-field) input from the data source, Celigo can map each field to the corresponding field in the destination. For example, if you’re importing an Account from Salesforce (source) into a Vendor in NetSuite (destination), you can easily map fields across both systems (e.g. Salesforce Account Name -> NetSuite Vendor Name), possibly applying transformations as needed. This process is generally described as an ETL – Extract, Transform, Load. However, there are cases where you simply want to use Celigo to push data as-is from source to destination. This article explains one way of doing that.
In a recent integration project, we resorted to using Celigo as a middleware to receive incoming webhooks from a third-party order management system and pass them to NetSuite. This was after we failed to establish a direct integration with NetSuite. (With a Suitelet, we received HTTP error code 500 and the vendor of the third-party application was hesitant to share the “raw” errors and server logs to enable us to debug the situation. A Restlet was not an option either because the third-party application was unable to deliver webhooks to authenticated endpoints which is a requirement when working with Restlets.) Since the client was already using Celigo for other integrations, it was a logical choice. Moreover, Celigo offers queuing facilities that come in handy in such situations.
Different events in the third-party system produced webhooks with event-specific payloads that all needed to be processed correctly in NetSuite. Therefore, we envisioned an architecture in which all incoming webhooks, regardless of the actual (dynamic) payload, will be pushed to a single custom record in NetSuite for further processing. As such, we needed a way to pass the payload from the incoming webhooks as-is to NetSuite.
For the sake of illustration, consider the following simplified payload. Notice the
eventType field which determines the structure of the rest of the payload.
When creating a Celigo flow with a Webhook Listener as the data source, we need to tell Celigo what the payload looks like. Celigo then makes each field available for mapping to fields on the NetSuite side. However, as stated, our payload is dynamic, and we want to pass the full payload to NetSuite instead of mapping individual fields. Thus, we needed to move away from the classical field-to-field mapping and figure out a way to pass the entire incoming payload into a single field in the NetSuite custom record.
Here’s a step-by-step description of the process. Basic understanding of Celigo is assumed. As such, we don’t go into details on how to set up flows, add script files, etc. If you’re not grounded in these subjects, at the time of writing, Celigo offers a very robust training package that includes a test environment for free. Head over to Celigo University to learn more.
In the Webhooks Listener, define your incoming data in a generic way:
As illustrated above, we simply tell Celigo that we expect a JSON object with a single element called “payload” (you may call it anything you want provided you are consistent in subsequent steps).
All we’re doing is wrapping the actual webhook data inside a dummy “payload” object so that Celigo can pick it up as per the mapping we defined.
Map the placeholder “payload” field to whatever field you need on the NetSuite side (conveniently called
Payload in our example).
Note that Celigo might show
payload.payloadon the source side as it tries to be smart and update the data structure based on the transformation we defined. However, that’s not what we want. Simply type in
payloadas the source field as shown below.
By default, no data type is explicitly selected and during my tests, the data was not sent to NetSuite until I explicitly set the field type to “String” as illustrated below. It’s unclear if this is a bug in Celigo or expected behavior. In any case, the message is that you should be explicit about the data type.
Due to the transformation in Step 2, the ‘Payload’ field on the NetSuite side will now contain the full payload received from the incoming webhook as desired!
That’s pretty much all there is to it. However, for the benefit of the reader who does not yet understand why this works, let me dive a little bit deeper. When this flow is run, the transformation will behave as follows:
Input before transformation:
Output after transformation:
In the mapping step, Celigo will now look for a field named
payload in our transformed input data whose value will be the entire original webhook data, thanks to the transformation.
Note Regarding JSON Serialization
In an earlier implementation of the above solution, I recall getting an error (which I, regrettably, did not preserve) that I fixed by applying the
jsonSerialize function in Celigo to achieve the desired results as illustrated below:
However, while preparing for this article, it worked without that step! Celigo is constantly evolving and I suspect Celigo has implemented logic to automatically stringify any incoming JSON. In any case, if you run into errors, remember
jsonSerialize. Alternatively, you could simply add it right away as it makes sense to convert your JSON input to a string before transmitting it to NetSuite.
Hope you’ve found this article useful and this trick comes in handy the next time you need to get dynamic webhook data into NetSuite using Celigo.
NetSuite Insights is on a mission to raise the standards around NetSuite practices, one insight at a time. If that resonates with you, check out how you can become an author/collaborator here.
Also, consider subscribing to our no-nonsense email list to get these insights delivered to your inbox as soon as they’re published. Sometimes, ignorance is a choice. Choose wisely!