Skip to content

Commit

Permalink
Merge branch 'develop' into chore.cache-dns-resolve-ips
Browse files Browse the repository at this point in the history
  • Loading branch information
Jayachand authored Apr 25, 2024
2 parents 14d3922 + 999b83e commit c1b34c4
Show file tree
Hide file tree
Showing 54 changed files with 5,877 additions and 8,085 deletions.
2 changes: 1 addition & 1 deletion .nvmrc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
18.19.1
18.20.1
44 changes: 44 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,50 @@

All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.

## [1.63.0](https://github.com/rudderlabs/rudder-transformer/compare/v1.62.2...v1.63.0) (2024-04-25)


### Features

* remove redundant data from traits in hubspot ([#3310](https://github.com/rudderlabs/rudder-transformer/issues/3310)) ([4b21f13](https://github.com/rudderlabs/rudder-transformer/commit/4b21f1353d3d9a431a0d5446d019f66a543b977b))

### [1.62.2](https://github.com/rudderlabs/rudder-transformer/compare/v1.62.1...v1.62.2) (2024-04-18)


### Bug Fixes

* twitter_ads logger ([#3295](https://github.com/rudderlabs/rudder-transformer/issues/3295)) ([e92b052](https://github.com/rudderlabs/rudder-transformer/commit/e92b052e03182deb41b20b3ec3741306afa50380))

### [1.62.1](https://github.com/rudderlabs/rudder-transformer/compare/v1.62.0...v1.62.1) (2024-04-18)


### Bug Fixes

* revert mixpanel deprecate /track ([#3291](https://github.com/rudderlabs/rudder-transformer/issues/3291)) ([ec068b4](https://github.com/rudderlabs/rudder-transformer/commit/ec068b49bd4a5652a762c60a8257c883e4709d1a))

## [1.62.0](https://github.com/rudderlabs/rudder-transformer/compare/v1.61.1...v1.62.0) (2024-04-15)


### Features

* do away myaxios ([#3222](https://github.com/rudderlabs/rudder-transformer/issues/3222)) ([9214594](https://github.com/rudderlabs/rudder-transformer/commit/9214594bab2c86a4ae6f75e12531f778490cf127))
* for reddit adding currency and value for addToCart, viewConent event as well ([#3239](https://github.com/rudderlabs/rudder-transformer/issues/3239)) ([ad235e7](https://github.com/rudderlabs/rudder-transformer/commit/ad235e785bf6039e11231a915be098130b25ec3b))
* logger upgrade in services, dest, source ([#3228](https://github.com/rudderlabs/rudder-transformer/issues/3228)) ([c204113](https://github.com/rudderlabs/rudder-transformer/commit/c204113eab37a782f217488d0d626a8d6df345d3))
* rakuten: adding a default value for tr ([#3240](https://github.com/rudderlabs/rudder-transformer/issues/3240)) ([3748f24](https://github.com/rudderlabs/rudder-transformer/commit/3748f24e21634fc74c5e5b3761551c64c8e69942))


### Bug Fixes

* adding check for reserved key words in extract custom fields ([#3264](https://github.com/rudderlabs/rudder-transformer/issues/3264)) ([3399c47](https://github.com/rudderlabs/rudder-transformer/commit/3399c47fdce1b3d19e29306ca3c5692a2fbc30fb))
* deployment file paths ([#3216](https://github.com/rudderlabs/rudder-transformer/issues/3216)) ([808727d](https://github.com/rudderlabs/rudder-transformer/commit/808727de17e400ed102a843ab3b30f81f8900f24))
* email mappings ([#3247](https://github.com/rudderlabs/rudder-transformer/issues/3247)) ([791cbf5](https://github.com/rudderlabs/rudder-transformer/commit/791cbf55fc6940af4e3208212b82c891c6618fc3))
* fixed userId mapping, now mapping to uid instead of id ([#3262](https://github.com/rudderlabs/rudder-transformer/issues/3262)) ([9c6b251](https://github.com/rudderlabs/rudder-transformer/commit/9c6b251a6c784cc391f27e846a008fbe2901e2c8))
* hs bugsnag error ([#3252](https://github.com/rudderlabs/rudder-transformer/issues/3252)) ([9daf1c9](https://github.com/rudderlabs/rudder-transformer/commit/9daf1c989258bd410d5780c1b11c4f6df9654af5))
* hubspot: search for contact using secondary prop ([#3258](https://github.com/rudderlabs/rudder-transformer/issues/3258)) ([0b57204](https://github.com/rudderlabs/rudder-transformer/commit/0b5720446693efe1fd0ccdfc141bd7f21b2c32ae))
* impact: support custom product mapping ([#3249](https://github.com/rudderlabs/rudder-transformer/issues/3249)) ([cb8ff2f](https://github.com/rudderlabs/rudder-transformer/commit/cb8ff2fb943c49df4ac083bd179d9674b40eb602))
* marketo bulk ignore null while checking data type mismatch ([#3263](https://github.com/rudderlabs/rudder-transformer/issues/3263)) ([6e3274b](https://github.com/rudderlabs/rudder-transformer/commit/6e3274bfba9e7838d1f81d845a070427b67e75f5))
* shopify: send 500 for identifier call in case of failure ([#3235](https://github.com/rudderlabs/rudder-transformer/issues/3235)) ([8eb4c4e](https://github.com/rudderlabs/rudder-transformer/commit/8eb4c4e9b8daebbaeb1d12ff0c17915fe19c2b50))

### [1.61.1](https://github.com/rudderlabs/rudder-transformer/compare/v1.61.0...v1.61.1) (2024-04-03)

## [1.61.0](https://github.com/rudderlabs/rudder-transformer/compare/v1.60.0...v1.61.0) (2024-04-02)
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1.4
FROM node:18.19.1-alpine3.18 AS base
FROM node:18.20.1-alpine3.18 AS base
ENV HUSKY 0

RUN apk update
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "rudder-transformer",
"version": "1.61.1",
"version": "1.63.0",
"description": "",
"homepage": "https://github.com/rudderlabs/rudder-transformer#readme",
"bugs": {
Expand Down
22 changes: 21 additions & 1 deletion src/cdk/v2/destinations/bluecore/config.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
const { getMappingConfig } = require('../../../../v0/util');

const BASE_URL = 'https://api.bluecore.com/api/track/mobile/v1';
const BASE_URL = 'https://api.bluecore.app/api/track/mobile/v1';

const CONFIG_CATEGORIES = {
IDENTIFY: {
Expand Down Expand Up @@ -46,11 +46,31 @@ const EVENT_NAME_MAPPING = [

const BLUECORE_EXCLUSION_FIELDS = ['query', 'order_id', 'total'];

const IDENTIFY_EXCLUSION_LIST = [
'name',
'firstName',
'first_name',
'firstname',
'lastName',
'last_name',
'lastname',
'email',
'age',
'sex',
'address',
'action',
'event',
];

const TRACK_EXCLUSION_LIST = [...IDENTIFY_EXCLUSION_LIST, 'query', 'order_id', 'total', 'products'];

const MAPPING_CONFIG = getMappingConfig(CONFIG_CATEGORIES, __dirname);
module.exports = {
CONFIG_CATEGORIES,
MAPPING_CONFIG,
EVENT_NAME_MAPPING,
BASE_URL,
BLUECORE_EXCLUSION_FIELDS,
IDENTIFY_EXCLUSION_LIST,
TRACK_EXCLUSION_LIST,
};
6 changes: 3 additions & 3 deletions src/cdk/v2/destinations/bluecore/procWorkflow.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ steps:
condition: $.outputs.messageType === {{$.EventType.IDENTIFY}}
template: |
const payload = $.constructProperties(.message);
payload.token = .destination.Config.bluecoreNamespace;
payload.properties.token = .destination.Config.bluecoreNamespace;
$.verifyPayload(payload, .message);
payload.event = payload.event ?? 'customer_patch';
payload.properties.distinct_id = $.populateAccurateDistinctId(payload, .message);
Expand All @@ -50,7 +50,7 @@ steps:
const temporaryProductArray = newPayload.properties.products ?? $.createProductForStandardEcommEvent(^.message, eventName);
newPayload.properties.products = $.normalizeProductArray(temporaryProductArray);
newPayload.event = eventName;
newPayload.token = ^.destination.Config.bluecoreNamespace;
newPayload.properties.token = ^.destination.Config.bluecoreNamespace;
$.verifyPayload(newPayload, ^.message);
$.removeUndefinedNullValuesAndEmptyObjectArray(newPayload)
)[];
Expand All @@ -61,7 +61,7 @@ steps:
const response = $.defaultRequestConfig();
response.body.JSON = .;
response.method = "POST";
response.endpoint = "https://api.bluecore.com/api/track/mobile/v1";
response.endpoint = "https://api.bluecore.app/api/track/mobile/v1";
response.headers = {
"Content-Type": "application/json"
};
Expand Down
45 changes: 43 additions & 2 deletions src/cdk/v2/destinations/bluecore/utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@ const {
validateEventName,
constructPayload,
getDestinationExternalID,
extractCustomFields,
} = require('../../../../v0/util');
const { CommonUtils } = require('../../../../util/common');
const { EVENT_NAME_MAPPING } = require('./config');
const { EVENT_NAME_MAPPING, IDENTIFY_EXCLUSION_LIST, TRACK_EXCLUSION_LIST } = require('./config');
const { EventType } = require('../../../../constants');
const { MAPPING_CONFIG, CONFIG_CATEGORIES } = require('./config');

Expand Down Expand Up @@ -167,6 +168,41 @@ const normalizeProductArray = (products) => {
return finalProductArray;
};

const mapCustomProperties = (message) => {
let customerProperties;
const customProperties = { properties: {} };
const messageType = message.type.toUpperCase();
switch (messageType) {
case 'IDENTIFY':
customerProperties = extractCustomFields(
message,
{},
['traits', 'context.traits'],
IDENTIFY_EXCLUSION_LIST,
);
customProperties.properties.customer = customerProperties;
break;
case 'TRACK':
customerProperties = extractCustomFields(
message,
{},
['traits', 'context.traits'],
IDENTIFY_EXCLUSION_LIST,
);
customProperties.properties = extractCustomFields(
message,
{},
['properties'],
TRACK_EXCLUSION_LIST,
);
customProperties.properties.customer = customerProperties;
break;
default:
break;
}
return customProperties;
};

/**
* Constructs properties based on the given message.
*
Expand All @@ -178,7 +214,12 @@ const constructProperties = (message) => {
const commonPayload = constructPayload(message, MAPPING_CONFIG[commonCategory.name]);
const category = CONFIG_CATEGORIES[message.type.toUpperCase()];
const typeSpecificPayload = constructPayload(message, MAPPING_CONFIG[category.name]);
const finalPayload = lodash.merge(commonPayload, typeSpecificPayload);
const typeSpecificCustomProperties = mapCustomProperties(message);
const finalPayload = lodash.merge(
commonPayload,
typeSpecificPayload,
typeSpecificCustomProperties,
);
return finalPayload;
};

Expand Down
2 changes: 1 addition & 1 deletion src/controllers/delivery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ const NON_DETERMINABLE = 'Non-determinable';

export class DeliveryController {
public static async deliverToDestination(ctx: Context) {
logger.info('Native(Delivery):: Request to transformer::', ctx.request.body);
logger.debug('Native(Delivery):: Request to transformer::', ctx.request.body);
let deliveryResponse: DeliveryV0Response;
const requestMetadata = MiscService.getRequestMetadata(ctx);
const deliveryRequest = ctx.request.body as ProxyV0Request;
Expand Down
2 changes: 1 addition & 1 deletion src/controllers/destination.ts
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ export class DestinationController {
}

public static batchProcess(ctx: Context) {
logger.info('Native(Process-Transform-Batch):: Requst to transformer::', ctx.request.body);
logger.debug('Native(Process-Transform-Batch):: Requst to transformer::', ctx.request.body);
const startTime = new Date();
const requestMetadata = MiscService.getRequestMetadata(ctx);
const routerRequest = ctx.request.body as RouterTransformationRequest;
Expand Down
2 changes: 1 addition & 1 deletion src/util/eventValidation.js
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ async function validate(event) {
trackingPlanId,
trackingPlanVersion,
event.message.type,
event.message.event,
event.message.type === 'track' ? event.message.event : '',
workspaceId,
);

Expand Down
36 changes: 25 additions & 11 deletions src/util/openfaas/faasApi.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@ const axios = require('axios');
const { RespStatusError, RetryRequestError } = require('../utils');

const OPENFAAS_GATEWAY_URL = process.env.OPENFAAS_GATEWAY_URL || 'http://localhost:8080';
const OPENFAAS_GATEWAY_USERNAME = process.env.OPENFAAS_GATEWAY_USERNAME || '';
const OPENFAAS_GATEWAY_PASSWORD = process.env.OPENFAAS_GATEWAY_PASSWORD || '';

const basicAuth = {
username: OPENFAAS_GATEWAY_USERNAME,
password: OPENFAAS_GATEWAY_PASSWORD,
};

const parseAxiosError = (error) => {
if (error.response) {
Expand All @@ -21,7 +28,7 @@ const deleteFunction = async (functionName) =>
new Promise((resolve, reject) => {
const url = `${OPENFAAS_GATEWAY_URL}/system/functions`;
axios
.delete(url, { data: { functionName } })
.delete(url, { data: { functionName }, auth: basicAuth })
.then(() => resolve())
.catch((err) => reject(parseAxiosError(err)));
});
Expand All @@ -30,7 +37,7 @@ const getFunction = async (functionName) =>
new Promise((resolve, reject) => {
const url = `${OPENFAAS_GATEWAY_URL}/system/function/${functionName}`;
axios
.get(url)
.get(url, { auth: basicAuth })
.then((resp) => resolve(resp.data))
.catch((err) => reject(parseAxiosError(err)));
});
Expand All @@ -39,7 +46,7 @@ const getFunctionList = async () =>
new Promise((resolve, reject) => {
const url = `${OPENFAAS_GATEWAY_URL}/system/functions`;
axios
.get(url)
.get(url, { auth: basicAuth })
.then((resp) => resolve(resp.data))
.catch((err) => reject(parseAxiosError(err)));
});
Expand All @@ -48,29 +55,36 @@ const invokeFunction = async (functionName, payload) =>
new Promise((resolve, reject) => {
const url = `${OPENFAAS_GATEWAY_URL}/function/${functionName}`;
axios
.post(url, payload)
.post(url, payload, { auth: basicAuth })
.then((resp) => resolve(resp.data))
.catch((err) => reject(parseAxiosError(err)));
});

const checkFunctionHealth = async (functionName) =>
new Promise((resolve, reject) => {
const checkFunctionHealth = async (functionName) => {
return new Promise((resolve, reject) => {
const url = `${OPENFAAS_GATEWAY_URL}/function/${functionName}`;
axios
.get(url, {
headers: { 'X-REQUEST-TYPE': 'HEALTH-CHECK' },
})
.get(
url,
{
headers: { 'X-REQUEST-TYPE': 'HEALTH-CHECK' },
},
{ auth: basicAuth },
)
.then((resp) => resolve(resp))
.catch((err) => reject(parseAxiosError(err)));
});
};

const deployFunction = async (payload) =>
new Promise((resolve, reject) => {
const url = `${OPENFAAS_GATEWAY_URL}/system/functions`;
axios
.post(url, payload)
.post(url, payload, { auth: basicAuth })
.then((resp) => resolve(resp.data))
.catch((err) => reject(parseAxiosError(err)));
.catch((err) => {
reject(parseAxiosError(err));
});
});

module.exports = {
Expand Down
12 changes: 11 additions & 1 deletion src/util/openfaas/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ const stats = require('../stats');
const { getMetadata, getTransformationMetadata } = require('../../v0/util');
const { HTTP_STATUS_CODES } = require('../../v0/util/constant');

const FAAS_SCALE_TYPE = process.env.FAAS_SCALE_TYPE || 'capacity';
const FAAS_SCALE_TARGET = process.env.FAAS_SCALE_TARGET || '4';
const FAAS_SCALE_TARGET_PROPORTION = process.env.FAAS_SCALE_TARGET_PROPORTION || '0.70';
const FAAS_SCALE_ZERO = process.env.FAAS_SCALE_ZERO || 'false';
const FAAS_SCALE_ZERO_DURATION = process.env.FAAS_SCALE_ZERO_DURATION || '15m';
const FAAS_BASE_IMG = process.env.FAAS_BASE_IMG || 'rudderlabs/openfaas-flask:main';
const FAAS_MAX_PODS_IN_TEXT = process.env.FAAS_MAX_PODS_IN_TEXT || '40';
const FAAS_MIN_PODS_IN_TEXT = process.env.FAAS_MIN_PODS_IN_TEXT || '1';
Expand Down Expand Up @@ -125,7 +130,7 @@ const deployFaasFunction = async (
trMetadata = {},
) => {
try {
logger.debug('[Faas] Deploying a faas function');
logger.debug(`[Faas] Deploying a faas function: ${functionName}`);
let envProcess = 'python index.py';

const lvidsString = libraryVersionIDs.join(',');
Expand All @@ -150,6 +155,11 @@ const deployFaasFunction = async (
'parent-component': 'openfaas',
'com.openfaas.scale.max': FAAS_MAX_PODS_IN_TEXT,
'com.openfaas.scale.min': FAAS_MIN_PODS_IN_TEXT,
'com.openfaas.scale.zero': FAAS_SCALE_ZERO,
'com.openfaas.scale.zero-duration': FAAS_SCALE_ZERO_DURATION,
'com.openfaas.scale.target': FAAS_SCALE_TARGET,
'com.openfaas.scale.target-proportion': FAAS_SCALE_TARGET_PROPORTION,
'com.openfaas.scale.type': FAAS_SCALE_TYPE,
transformationId: trMetadata.transformationId,
workspaceId: trMetadata.workspaceId,
team: 'data-management',
Expand Down
5 changes: 5 additions & 0 deletions src/util/trackingPlan.js
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,11 @@ async function getEventSchema(tpId, tpVersion, eventType, eventName, workspaceId
let eventSchema;
const tp = await getTrackingPlan(tpId, tpVersion, workspaceId);

if (Object.hasOwn(tp, 'events')) {
const ev = tp.events.find((e) => e.name === eventName && e.eventType === eventType);
return ev?.rules;
}

if (eventType !== 'track') {
if (Object.prototype.hasOwnProperty.call(tp.rules, eventType)) {
eventSchema = tp.rules[eventType];
Expand Down
6 changes: 5 additions & 1 deletion src/v0/destinations/awin/utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,11 @@ const trackProduct = (properties, advertiserId, commissionParts) => {
productsArray.forEach((product) => {
const productPayloadNew = {
advertiserId,
orderReference: properties.order_id || properties.orderId,
orderReference:
properties.order_id ||
properties.orderId ||
properties.orderReference ||
properties.order_reference,
productId: product.product_id || product.productId,
productName: product.name,
productItemPrice: product.price,
Expand Down
Loading

0 comments on commit c1b34c4

Please sign in to comment.