CharsetToolkit (gephi 0.9.1 API)

java.lang.Object
- org.gephi.utils.CharsetToolkit

public class CharsetToolkit
extends Object

Utility class to guess the encoding of a given text file.

Unicode files encoded in UTF-16 (low or big endian) or UTF-8 files with a Byte Order Marker are correctly discovered. For UTF-8 files with no BOM, if the buffer is wide enough, the charset should also be discovered.

A byte buffer of 4KB is usually sufficient to be able to guess the encoding.

Usage:

 // guess the encoding
 Charset guessedCharset = CharsetToolkit.guessEncoding(file, 4096);

 // create a reader with the correct charset
 CharsetToolkit toolkit = new CharsetToolkit(file);
 BufferedReader reader = toolkit.getReader();

 // read the file content
 String line;
 while ((line = br.readLine())!= null)
 {
     System.out.println(line);
 }

Author:: Guillaume Laforge

Field Summary

Fields
Modifier and Type	Field and Description
`private byte[]`	`buffer`
`private static int`	`BUFFER_SIZE`
`private Charset`	`charset`
`private Charset`	`defaultCharset`
`private static byte[]`	`EMPTY_BYTE_ARRAY`
`private boolean`	`enforce8Bit`
`private PushbackInputStream`	`input`

Constructor Summary

Constructors
Constructor and Description

CharsetToolkit(InputStream stream)
Constructor of the CharsetToolkit utility class.

Constructors
Constructor and Description
`CharsetToolkit(InputStream stream)` Constructor of the `CharsetToolkit` utility class.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`static Charset[]`	`getAvailableCharsets()` Retrieves all the available `Charset`s on the platform, among which the default `charset`.
`Charset`	`getCharset()`
`Charset`	`getDefaultCharset()` Retrieves the default Charset
`static Charset`	`getDefaultSystemCharset()` Retrieve the default charset of the system.
`boolean`	`getEnforce8Bit()` Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.
`BufferedReader`	`getReader()` Gets a `BufferedReader` (indeed a `LineNumberReader`) from the `File` specified in the constructor of `CharsetToolkit` using the charset discovered by the method `guessEncoding()`.
`private Charset`	`guessEncoding()` Guess the encoding of the provided buffer.
`boolean`	`hasUTF16BEBom()` Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2).
`boolean`	`hasUTF16LEBom()` Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le).
`boolean`	`hasUTF8Bom()` Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors).
`private static boolean`	`isContinuationChar(byte b)` If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character;
`private static boolean`	`isFiveBytesSequence(byte b)` If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character.
`private static boolean`	`isFourBytesSequence(byte b)` If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character.
`private static boolean`	`isSixBytesSequence(byte b)` If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character.
`private static boolean`	`isThreeBytesSequence(byte b)` If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character.
`private static boolean`	`isTwoBytesSequence(byte b)` If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character.
`void`	`setDefaultCharset(Charset defaultCharset)` Defines the default `Charset` used in case the buffer represents an 8-bit `Charset`.
`void`	`setEnforce8Bit(boolean enforce)` If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - BUFFER_SIZE
```
private static final int BUFFER_SIZE
```
    See Also:
    
    Constant Field Values
  - EMPTY_BYTE_ARRAY
```
private static final byte[] EMPTY_BYTE_ARRAY
```
  - buffer
```
private byte[] buffer
```
  - defaultCharset
```
private Charset defaultCharset
```
  - charset
```
private Charset charset
```
  - enforce8Bit
```
private boolean enforce8Bit
```
  - input
```
private final PushbackInputStream input
```
- Constructor Detail
  - CharsetToolkit
```
public CharsetToolkit(InputStream stream)
               throws IOException
```
    Constructor of the CharsetToolkit utility class.
    
    Parameters:
    
    stream - of which we want to know the encoding.
    
    Throws:
    
    IOException - if an io error occurs
- Method Detail
  - setDefaultCharset
```
public void setDefaultCharset(Charset defaultCharset)
```
    Defines the default Charset used in case the buffer represents an 8-bit Charset.
    
    Parameters:
    
    defaultCharset - the default Charset to be returned by guessEncoding() if an 8-bit Charset is encountered.
  - getCharset
```
public Charset getCharset()
```
  - setEnforce8Bit
```
public void setEnforce8Bit(boolean enforce)
```
    If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII. It might be a file without any special character in the range 128-255, but that may be or become a file encoded with the default charset rather than US-ASCII.
    
    Parameters:
    
    enforce - a boolean specifying the use or not of US-ASCII.
  - getEnforce8Bit
```
public boolean getEnforce8Bit()
```
    Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.
    
    Returns:
    
    a boolean representing the flag of use of US-ASCII.
  - getDefaultCharset
```
public Charset getDefaultCharset()
```
    Retrieves the default Charset
    
    Returns:
    
    charset
  - guessEncoding
```
private Charset guessEncoding()
```
    Guess the encoding of the provided buffer.
    If Byte Order Markers are encountered at the beginning of the buffer, we immidiately return the charset implied by this BOM. Otherwise, the file would not be a human readable text file.
    
    If there is no BOM, this method tries to discern whether the file is UTF-8 or not. If it is not UTF-8, we assume the encoding is the default system encoding (of course, it might be any 8-bit charset, but usually, an 8-bit charset is the default one).
    
    It is possible to discern UTF-8 thanks to the pattern of characters with a multi-byte sequence.
```
 UCS-4 range (hex.)        UTF-8 octet sequence (binary)
 0000 0000-0000 007F       0xxxxxxx
 0000 0080-0000 07FF       110xxxxx 10xxxxxx
 0000 0800-0000 FFFF       1110xxxx 10xxxxxx 10xxxxxx
 0001 0000-001F FFFF       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
 0020 0000-03FF FFFF       111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
 0400 0000-7FFF FFFF       1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
 
```
    With UTF-8, 0xFE and 0xFF never appear.
    Returns:
    
    the Charset recognized.
  - isContinuationChar
```
private static boolean isContinuationChar(byte b)
```
    If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character;
    
    Parameters:
    
    b - a byte.
    
    Returns:
    
    true if it's a continuation char.
  - isTwoBytesSequence
```
private static boolean isTwoBytesSequence(byte b)
```
    If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character.
    
    Parameters:
    
    b - a byte.
    
    Returns:
    
    true if it's the first byte of a two-bytes sequence.
  - isThreeBytesSequence
```
private static boolean isThreeBytesSequence(byte b)
```
    If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character.
    
    Parameters:
    
    b - a byte.
    
    Returns:
    
    true if it's the first byte of a three-bytes sequence.
  - isFourBytesSequence
```
private static boolean isFourBytesSequence(byte b)
```
    If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character.
    
    Parameters:
    
    b - a byte.
    
    Returns:
    
    true if it's the first byte of a four-bytes sequence.
  - isFiveBytesSequence
```
private static boolean isFiveBytesSequence(byte b)
```
    If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character.
    
    Parameters:
    
    b - a byte.
    
    Returns:
    
    true if it's the first byte of a five-bytes sequence.
  - isSixBytesSequence
```
private static boolean isSixBytesSequence(byte b)
```
    If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character.
    
    Parameters:
    
    b - a byte.
    
    Returns:
    
    true if it's the first byte of a six-bytes sequence.
  - getDefaultSystemCharset
```
public static Charset getDefaultSystemCharset()
```
    Retrieve the default charset of the system.
    
    Returns:
    
    the default Charset.
  - hasUTF8Bom
```
public boolean hasUTF8Bom()
```
    Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors).
    
    Returns:
    
    true if the buffer has a BOM for UTF8.
  - hasUTF16LEBom
```
public boolean hasUTF16LEBom()
```
    Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le).
    
    Returns:
    
    true if the buffer has a BOM for UTF-16 Low Endian.
  - hasUTF16BEBom
```
public boolean hasUTF16BEBom()
```
    Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2).
    
    Returns:
    
    true if the buffer has a BOM for UTF-16 Big Endian.
  - getReader
```
public BufferedReader getReader()
```
    Gets a BufferedReader (indeed a LineNumberReader) from the File specified in the constructor of CharsetToolkit using the charset discovered by the method guessEncoding().
    
    Returns:
    
    a BufferedReader
  - getAvailableCharsets
```
public static Charset[] getAvailableCharsets()
```
    Retrieves all the available Charsets on the platform, among which the default charset.
    
    Returns:
    
    an array of Charsets.

Class CharsetToolkit

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

BUFFER_SIZE

EMPTY_BYTE_ARRAY

buffer

defaultCharset

charset

enforce8Bit

input

Constructor Detail

CharsetToolkit

Method Detail

setDefaultCharset

getCharset

setEnforce8Bit

getEnforce8Bit

getDefaultCharset

guessEncoding

isContinuationChar

isTwoBytesSequence

isThreeBytesSequence

isFourBytesSequence

isFiveBytesSequence

isSixBytesSequence

getDefaultSystemCharset

hasUTF8Bom

hasUTF16LEBom

hasUTF16BEBom

getReader

getAvailableCharsets