Skip to content

Commit

Permalink
More compact protocol 32-bit fixes
Browse files Browse the repository at this point in the history
Summary:
malanka wants to send 6GB Glean fact batches to the server, which currently fails with Thrift deserialization errors:

```
W0118 11:49:04.386687  5031 SendQueue.hs:255] Thread killed: ApplicationException {applicationException_message = "user error (incomplete input)", applicationException_type = ApplicationExceptionType_Unknown}
DB write failure: ApplicationException {applicationException_message = "user error (incomplete input)", applicationException_type = ApplicationExceptionType_Unknown}
```

The spec for the compact protocol doesn't prescribe a bit-size for these fields, as far as I can tell:

https://www.internalfb.com/intern/staticdocs/thrift/docs/features/serialization/protocols#compact-protocol

Reviewed By: simonmar

Differential Revision: D52903378

fbshipit-source-id: fde0017779ced80a7911a99cc63f2dffefb26406
  • Loading branch information
Pepe Iborra authored and facebook-github-bot committed Jan 19, 2024
1 parent d71c8eb commit f833f52
Showing 1 changed file with 5 additions and 8 deletions.
13 changes: 5 additions & 8 deletions lib/Thrift/Protocol/Compact.hs
Original file line number Diff line number Diff line change
Expand Up @@ -183,13 +183,13 @@ instance Protocol Compact where
byte <- anyWord8
let _ty = byte .&. 0x0F
len <- case byte `shiftR` 4 of
0x0F -> parseVarint32
0x0F -> parseVarint
size -> pure $ fromIntegral size
ps <- replicateM (fromIntegral len) p
return (fromIntegral len, ps)

parseMap _ pk pv _ = do
len <- parseVarint32
len <- parseVarint
if len == 0
then pure []
else do
Expand Down Expand Up @@ -226,18 +226,18 @@ instance Protocol Compact where
parseSkip _ TI64 _ = void parseCompactI64
parseSkip _ TDOUBLE _ = P.skipN 8
parseSkip _ TSTRING _ =
void $ parseVarint32 >>= P.skipN . fromIntegral
void $ parseVarint >>= P.skipN . fromIntegral
parseSkip proxy TLIST _ = do
byte <- anyWord8
let ty = byte .&. 0x0F
len <- case byte `shiftR` 4 of
0x0F -> parseVarint32
0x0F -> parseVarint
size -> pure $ fromIntegral size
void $ replicateM (fromIntegral len) (parseSkip proxy ty Nothing)

parseSkip proxy TSET _ = parseSkip proxy TLIST Nothing
parseSkip proxy TMAP _ = do
len <- parseVarint32
len <- parseVarint
if len == 0
then pure ()
else do
Expand Down Expand Up @@ -381,6 +381,3 @@ parseVarint = go 0 0
if not (testBit w 7)
then return newVal
else go newVal (n + 7)

parseVarint32 :: Parser Int32
parseVarint32 = fromIntegral <$> parseVarint

0 comments on commit f833f52

Please sign in to comment.