T - data type T for RLEpublic abstract class RleEncoder<T extends Comparable<T>> extends Encoder
rle-bit-packing-hybrid: <length> <bitwidth> <encoded-data>
length := length of the <bitwidth> <encoded-data> in bytes stored as 4 bytes little endian
bitwidth := bitwidth for all encoded data in <encoded-data>
encoded-data := <run>*
run := <bit-packed-run> | <rle-run>
bit-packed-run := <bit-packed-header> <lastBitPackedNum> <bit-packed-values>
bit-packed-header := varint-encode(<bit-pack-count> << 1 | 1)
lastBitPackedNum := the number of useful value in last bit-pack may be less than 8, so
lastBitPackedNum indicates how many values are useful
bit-packed-values := bit packed
rle-run := <rle-header> <repeated-value>
rle-header := varint-encode( (number of times repeated) << 1)
repeated-value := value that is repeated, using a fixed-width of round-up-to-next-byte(bit-width)
.| Modifier and Type | Field and Description |
|---|---|
protected int |
bitPackedGroupCount
the number of group which using bit packing, it is saved in header.
|
protected int |
bitWidth
the bit width used for bit-packing and rle.
|
protected T[] |
bufferedValues
array to buffer values temporarily.
|
protected ByteArrayOutputStream |
byteCache
output stream to buffer
<bitwidth> <encoded-data>. |
protected List<byte[]> |
bytesBuffer
we will write all bytes using bit-packing to OutputStream once.
|
protected TSFileConfig |
config |
protected boolean |
isBitPackRun
flag which indicate encoding mode false -- rle true -- bit-packing.
|
protected boolean |
isBitWidthSaved |
protected int |
numBufferedValues
the number of buffered value in array.
|
protected T |
preValue
previous value written, used to detect repeated values.
|
protected int |
repeatCount
for a given value now buffered, how many times it occurs.
|
protected List<T> |
values
we save all value in a list and calculate its bitwidth.
|
MAX_POINT_NUMBER, MAX_STRING_LENGTH| Modifier | Constructor and Description |
|---|---|
protected |
RleEncoder()
constructor.
|
| Modifier and Type | Method and Description |
|---|---|
protected abstract void |
clearBuffer()
clean all useless value in bufferedValues and set 0.
|
protected abstract void |
convertBuffer() |
void |
encode(BigDecimal value,
ByteArrayOutputStream out) |
void |
encode(Binary value,
ByteArrayOutputStream out) |
void |
encode(boolean value,
ByteArrayOutputStream out) |
void |
encode(double value,
ByteArrayOutputStream out) |
void |
encode(float value,
ByteArrayOutputStream out) |
void |
encode(int value,
ByteArrayOutputStream out) |
void |
encode(long value,
ByteArrayOutputStream out) |
void |
encode(short value,
ByteArrayOutputStream out) |
protected void |
encodeValue(T value)
Encode T value using rle or bit-packing.
|
protected void |
endPreviousBitPackedRun(int lastBitPackedNum)
End a bit-packing run write all bit-packing group to OutputStream bit-packing format:
[header][lastBitPackedNum][bit-packing group]+ [bit-packing group]+ are saved in List<byte[]>
bytesBuffer . |
void |
flush(ByteArrayOutputStream out)
Write all values buffered in cache to OutputStream.
|
protected void |
reset() |
void |
writeOrAppendBitPackedRun()
Start a bit-packing run transform values to bytes and buffer them in cache.
|
protected abstract void |
writeRleRun()
Write bytes to OutputStream using rle.
|
getMaxByteSize, getOneItemMaxSize, getType, setTypeprotected List<T extends Comparable<T>> values
protected int bitWidth
protected int repeatCount
protected int bitPackedGroupCount
protected int numBufferedValues
protected List<byte[]> bytesBuffer
protected boolean isBitPackRun
protected T extends Comparable<T> preValue
protected T extends Comparable<T>[] bufferedValues
protected boolean isBitWidthSaved
protected ByteArrayOutputStream byteCache
<bitwidth> <encoded-data>.protected TSFileConfig config
protected void reset()
public void flush(ByteArrayOutputStream out) throws IOException
flush in class Encoderout - - byteArrayOutputStreamIOException - cannot flush to OutputStreamprotected abstract void writeRleRun()
throws IOException
[header][value] header: (repeated
value) << 1IOException - cannot write RLE runpublic void writeOrAppendBitPackedRun()
protected void endPreviousBitPackedRun(int lastBitPackedNum)
[header][lastBitPackedNum][bit-packing group]+ [bit-packing group]+ are saved in List<byte[]>
bytesBuffer .lastBitPackedNum - - in last bit-packing group, it may have useful values less than 8.
This param indicates how many values are usefulprotected void encodeValue(T value)
value - - value to encodeprotected abstract void clearBuffer()
protected abstract void convertBuffer()
public void encode(boolean value,
ByteArrayOutputStream out)
public void encode(short value,
ByteArrayOutputStream out)
public void encode(int value,
ByteArrayOutputStream out)
public void encode(long value,
ByteArrayOutputStream out)
public void encode(float value,
ByteArrayOutputStream out)
public void encode(double value,
ByteArrayOutputStream out)
public void encode(Binary value, ByteArrayOutputStream out)
public void encode(BigDecimal value, ByteArrayOutputStream out)
Copyright © 2022 The Apache Software Foundation. All rights reserved.