Skip to content
This repository has been archived by the owner on Jun 7, 2024. It is now read-only.

Aruha 473 check size of events #515

Merged
merged 41 commits into from
Jan 23, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
9dabfa2
ARUHA-473 Reject batches with at least one event that's too large
Dec 15, 2016
cc50d0c
ARUHA-473 Move maximum event size to nakadi settings
Dec 15, 2016
761d455
ARUHA-473 Unit tests for max event size
Dec 15, 2016
146d6a0
ARUHA-473 Acceptance test for max event size
Dec 16, 2016
8849782
ARUHA-473 Fix checkstyle
Dec 16, 2016
a061392
ARUHA-473 Move size validation after validation, remove VALIDATING_SI…
Dec 16, 2016
0a791da
ARUHA-473 Move size validation inside validation
Dec 16, 2016
df5b675
ARUHA-473 Fix bug in test wrt event publishing step
Dec 16, 2016
e261cb8
ARUHA-473 Refactor string building
Dec 16, 2016
217be7f
ARUHA-473 Fix log string
Dec 16, 2016
ddd7a4f
ARUHA-473 Set capacity for StringBuilder
Dec 16, 2016
957938b
ARUHA-473 If an event is too large, that event is marked FAILED. Othe…
Dec 16, 2016
49c1846
ARUHA-473 Remove redundant Exception
Dec 16, 2016
cc15d1a
ARUHA-473 Explicit charset in getBytes()
Dec 20, 2016
3f9bf87
ARUHA-473 Refactor event size computation
Dec 23, 2016
6f7a450
ARUHA-473 Explicitely set Kafka consumers' fetch.message.max.bytes pr…
Dec 27, 2016
f315836
ARUHA-473 Add curly braces
Dec 28, 2016
8871dea
ARUHA-473 Explicit event size limit in swagger file
Dec 28, 2016
f5e6367
ARUHA-473 BatchItem constructor for String event, adding event size
Dec 30, 2016
d5e6f3d
ARUHA-473 Test BatchItem size with multi-byte characters
Dec 30, 2016
12253ed
ARUHA-473 Fix test style
Dec 30, 2016
30da84a
ARUHA-473 BatchFactory method to process batches as Strings
Dec 30, 2016
7287c40
ARUHA-473 Use String when publishing events to improve event size che…
Jan 3, 2017
d8af912
Merge branch 'master' into ARUHA-473-check-size-of-events
Jan 3, 2017
0fbaa46
ARUHA-473 Pass batch as String to publisher and fix checkstyle violat…
Jan 3, 2017
20aed22
ARUHA-473 Don't parse input to JSON in EventPublishingController
Jan 4, 2017
b9c36f8
ARUHA-473 Remove acceptance test
Jan 5, 2017
33bbc03
ARUHA-473 Remove unused private method
Jan 5, 2017
f6fcfe9
ARUHA-473 Error message for events too large includes the size of the…
Jan 5, 2017
f90e477
ARUHA-473 Fix tests' expected error messages for events too large
Jan 5, 2017
43a337a
Merge branch 'master' into ARUHA-473-check-size-of-events
Jan 5, 2017
bf90d6e
ARUHA-473 Fix test
Jan 5, 2017
0330145
ARUHA-473 Change max event size in swagger file
Jan 6, 2017
4cc0bcd
ARUHA-473 Fix bug when a batch is an array with only spaces.
Jan 6, 2017
30db7f5
ARUHA-473 Set max event size to 999,000 bytes by default
Jan 6, 2017
781db1e
ARUHA-473 Accept valid batches surrounded with tabs, carriage returns…
Jan 10, 2017
ffd729a
ARUHA-473 Refactoring
Jan 10, 2017
e3a4562
Merge branch 'master' into ARUHA-473-check-size-of-events
Jan 18, 2017
58cb450
ARUHA-473 Remove kafka max bytes property
Jan 18, 2017
3f2a2ac
ARUHA-473 Finish removing kafka max bytes property
Jan 19, 2017
57e67d3
Merge branch 'master' into ARUHA-473-check-size-of-events
Jan 23, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions src/main/java/org/zalando/nakadi/domain/BatchFactory.java
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,35 @@ public static List<BatchItem> from(final String events) {
int brackets = 0;
boolean insideQuote = false;
boolean escaped = false;
if ((!events.startsWith("[")) || (!events.endsWith("]"))) {
throw new JSONException("Array must be surrounded with square brackets");
int start = 0;
final int length = events.length();
int end = length - 1;

while ((events.charAt(start) == ' '
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this condition could be extracted to it's own method, something like isBlankCharacter.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or as per the test name isEmptyCharacter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, will extract

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

|| events.charAt(start) == '\t'
|| events.charAt(start) == '\n'
|| events.charAt(start) == '\r')
&& start < end) {
start++;
}
while ((events.charAt(end) == ' '
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this condition is extracted to its own method, then this should be refactored to reuse it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

|| events.charAt(end) == '\t'
|| events.charAt(end) == '\n'
|| events.charAt(end) == '\r')
&& end > start) {
end--;
}
if (!(events.charAt(start) == '[')) {
throw new JSONException(String.format("Unexpected character %s in position %d, expected '['",
events.charAt(start), start));
}
for (int i = 1; i < events.length() - 1; i++) {
start++;
if (!(events.charAt(end) == ']')) {
throw new JSONException(String.format("Unexpected character %s in position %d, expected ']'",
events.charAt(end), end));
}

for (int i = start; i < end; i++) {
if (!escaped && events.charAt(i) == '"') {
if (insideQuote) {
insideQuote = false;
Expand Down
7 changes: 7 additions & 0 deletions src/test/java/org/zalando/nakadi/domain/BatchFactoryTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,13 @@ public void testSpacesBetweenEvents() {
assertEquals("{\"name\":\"MyOtherEvent\"}", batch.get(1).getEvent().toString());
}

@Test
public void testEmptyCharactersAroundArray() {
final String events = "\t [{\"name\":\"MyEvent\"},{\"name\":\"MyOtherEvent\"}]\n\n";
final List<BatchItem> batch = BatchFactory.from(events);
assertEquals(2, batch.size());
}

@Test
public void testGarbageBetweenEvents() {
final String events = "[{\"name\":\"MyEvent\"},atb#{\"name\":\"MyOtherEvent\"}]";
Expand Down