Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache the processed data #59

Open
wants to merge 5 commits into
base: stable
Choose a base branch
from
Open

Cache the processed data #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
62ea440
add cache
flolopdel Apr 27, 2022
74bf79b
Update DataApiInterceptors.cfc
flolopdel Apr 29, 2022
ccbdcb6
check for the object which does not use cache
flolopdel May 3, 2022
ca09568
double check in the cache
flolopdel May 3, 2022
5d9b472
ignore vscode
flolopdel May 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@
npm-debug.log
/tests/server*.json
/modules
.vscode/settings.json
20 changes: 19 additions & 1 deletion interceptors/DataApiInterceptors.cfc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ component extends="coldbox.system.Interceptor" {
property name="dataApiQueueService" inject="delayedInjector:dataApiQueueService";
property name="dataApiConfigurationService" inject="delayedInjector:dataApiConfigurationService";
property name="interceptorService" inject="coldbox:InterceptorService";
property name="queryCache" inject="cachebox:DefaultQueryCache";

variables._applicationLoaded = false;

Expand Down Expand Up @@ -165,4 +166,21 @@ component extends="coldbox.system.Interceptor" {

dataApiQueueService.queueInsert( argumentCollection=interceptData );
}
}



public void function onCreateSelectDataCacheKey( event, interceptData ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels dangerous / misplaced. This interception point could easily be fired off 1000's of times in any given request (i.e. some bad uncached admin request with lots of little queries). From my understanding, you want to use a 'processed' cache of the select data results, only when the select data result hasn't changed.

I think a more accurate / clean approach here would be:

  1. Create a dedicated API result cache
  2. Do your selectData as normal
  3. Generate a cache key that is API Endpoint + a hash of the db query result

Then just use your dedicated API cache with this cache key. If the data changes in the DB, that cache entry will no longer get looked up.

Does that make sense? Here is an example of an extension registering its own cachebox cache using afterConfigurationLoad coldbox interception point: https://github.com/pixl8/preside-ext-s3-storage-provider/blob/stable/interceptors/S3StorageProviderInterceptors.cfc

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @DominicWatson !

I understand you.

The answer to the question 2 Do your selectData as normal Is the reason I add the interceptor onCreateSelectDataCacheKey To check if the data from the selectData has been cached or not.

  • If it was cached then get the cache specific for api result
  • If not, processed and save cached

What do you propose to do this without use the intercelpor onCreateSelectDataCacheKey ?

It is true onCreateSelectDataCacheKey is fired a lot of times but it is also true I did not do any important action there more than set true or false some RequestContext

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To check if the data from the selectData has been cached or not.

This does not essentially matter. All that matters is that you have a version in cache that matches the result of the query. If the query is cached and the API result is cached then super-great. If the query is not cached but the result has not changed and we still have an API result cache, then we can still use this API result cache.

if ( !_applicationLoaded ) return;

var objectName = interceptData.objectName ?: "";
event.setValue( name="cacheKey", value=interceptData.cacheKey ?: "", private=true );
event.setValue( name="getFromCache", value=false, private=true );
event.setValue( name="#objectName#_useCache", value=true, private=true );

var cachedResult = queryCache.get( interceptData.cacheKey );
if ( !IsNull( local.cachedResult ) ) {
event.setValue( name="getFromCache", value=true, private=true );
}
}

}
25 changes: 21 additions & 4 deletions services/DataApiService.cfc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,12 +279,29 @@ component {
}

var records = dao.selectData( argumentCollection=args );
var processed = [];

for( var record in records ) {
processed.append( _processFields( record, fieldSettings ) );
var event = $getRequestContext();
var cacheKey = event.getValue( name="cacheKey", default ="", private=true );
var useCache = event.getValue( name="#objectName#_useCache", default =false, private=true );
var cachedProcessed= queryCache.get( "dataApi_#cacheKey#" );

var existsCache = false;
if ( !IsNull( local.cachedProcessed ) ) {
existsCache = true
}

var getFromCache = useCache && existsCache && event.getValue( name="getFromCache", default =false, private=true );

if(getFromCache){
var processed = local.cachedProcessed;
}else{
var processed = [];
for( var record in records ) {
processed.append( _processFields( record, fieldSettings ) );
}
$announceInterception( "postDataApiSelectData#namespace#", { selectDataArgs=args, entity=arguments.entity, data=processed } );
queryCache.set( "dataApi_#cacheKey#", processed);
}
$announceInterception( "postDataApiSelectData#namespace#", { selectDataArgs=args, entity=arguments.entity, data=processed } );

return processed;
}
Expand Down