Data types serve as the cornerstone of all data-related endeavours. They are the very essence that defines the nature and behaviour of data within any system.
Understanding data types ensures data integrity, optimises performance, fosters interoperability, and facilitates effective manipulation and analysis. From efficient memory usage to error-free operations, data storage to seamless data exchange, every aspect of data-driven endeavours relies on a deep understanding of data types.
Here are the most common data types in Apache Spark:
| Data Type | Description | Example |
|---|---|---|
| Binary | Represents binary data. | bytes([0x48, 0x65, 0x6C, 0x6C, 0x6F]) |
| Boolean | Represents boolean values (True or False). | TRUE or FALSE |
| Byte | Represents byte values (-128 to 127). | 42 |
| Date | Represents date values. | date(2024, 3, 18) |
| Decimal | Represents fixed precision decimal numbers. | Decimal(‘3.141592653589793238’) |
| Double | Represents double-precision floating-point numbers. | 3.14 |
| Float | Represents single-precision floating-point numbers. | 3.14 |
| Integer | Represents integer numbers. i.e. a signed 32-bit integer | 35 |
| Long | Represents long integer numbers. i.e. a signed 64-bit integer | 12345 |
| Null | Represents null values. | None |
| Array | Represents a collection of elements of the same type | [1, 2, 3, 4, 5] |
| Map | Represents key-value pairs. | {“key1”: “value1”, “key2”: “value2”} |
| Short | Represents short integer numbers. i.e. A signed 16-bit integer. | 30 |
| String | Represents text strings. | “Hello!” |
| Char | Represents character data. | “Jason” |
| Varchar | Represents variable-length character data. | “Smith” |
| Struct | Represents a structure with multiple fields. | struct_value = StructType([ StructField(“name”, StringType(), nullable=False), StructField(“age”, IntegerType(), nullable=True) ]) |
| Timestamp | Represents timestamp values. | ‘2024-03-01 12:00:00’ |
| DayTimeInterval | Represents intervals in days and seconds. | 1 day 3 hours 30 minutes |
| YearMonthInterval | Represents intervals in years and months. | 2 years 6 months |