Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle segment gran changes #1

Closed
wants to merge 6 commits into from
Closed

Handle segment gran changes #1

wants to merge 6 commits into from

Conversation

pjain1
Copy link
Owner

@pjain1 pjain1 commented Oct 19, 2015

With this PR tranquility can handle segment granularity changes without losing data and preventing hung up tasks.

  • If the granularity in decreased then new tasks with changed segment granularity will be not be spawned till the time existing task can handle the incoming events. For example, if at 10:12 the gran is changed from 5 Minutes to 1 Minute then till 10:15 no new tasks will be spawned.
  • If the gran is increased then a partial beam (task having segment gran < the required segment gran) will be created to handle the events that falls in the interval not handled by currently active tasks for the new segment interval. For example, if the change is from 5 to 15 Minutes at 10:03 then a new task having interval 10:10 - 10:15 will be spawned to handle the incoming events.

During the time new events come in for tasks spawned to handle segment granularity changes, tranquility will not perform smart things like warming up beams, performing strict checking on interval while constructing beam object from the dictionary etc.

@@ -155,6 +180,14 @@ class ClusteredBeam[EventType: Timestamper, InnerBeamType <: Beam[EventType]](
log.info("Writing new beam data to[%s]: %s", dataPath, new String(newMetaBytes))
curator.setData().forPath(dataPath, newMetaBytes)
}
zkMetadataCache = newMeta

log.debug(zkMetadataCache.latestCloseTime.toString)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that you are doing iteration here only to log debug, it's probably best to wrap this in a if (log.isDebugEnabled())

@pjain1
Copy link
Owner Author

pjain1 commented Nov 17, 2015

new PR #2

@pjain1 pjain1 closed this Nov 17, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants