Enhancing Data Lake Performance: Amazon S3 Now Supports Compaction for Apache Avro and ORC Formats in Apache Iceberg Tables
We are delighted to share exciting news from Amazon Web Services (AWS) that marks a significant advancement in the efficiency and performance of data lakes built on Amazon S3. As of July 15, 2025, Amazon S3 now offers native support for the compaction of data stored in Apache Avro and Apache ORC formats within Apache Iceberg tables. This new capability is poised to streamline data management, reduce operational overhead, and unlock even greater value from your data lake investments.
For those who leverage Apache Iceberg as their table format, this update is particularly noteworthy. Iceberg is renowned for its ability to bring reliable transactional capabilities and schema evolution to large analytic datasets. However, as data is ingested and modified, data files within tables can become numerous and small, a common phenomenon known as “small file problem.” This can lead to performance degradation during data read operations, as query engines often struggle with the overhead associated with opening and processing a large volume of small files.
The newly introduced compaction feature in Amazon S3 directly addresses this challenge. By intelligently consolidating smaller data files into larger, more optimized ones, this functionality significantly improves query performance. Specifically, for tables utilizing the popular Apache Avro and Apache ORC data formats, users can now benefit from this built-in compaction process.
What does this mean for you?
- Improved Query Performance: Larger data files reduce the I/O overhead for query engines, leading to faster data retrieval and analysis. This translates directly into quicker insights and a more responsive data analytics experience.
- Reduced Storage Costs: While not a direct cost reduction, improved efficiency means less time spent scanning unnecessary metadata and fewer open file handles, which can indirectly contribute to lower operational costs.
- Simplified Data Management: The automated nature of compaction reduces the manual effort previously required to manage and optimize data files within Iceberg tables. This frees up valuable time for data engineers and analysts to focus on higher-value tasks.
- Enhanced Scalability: As your data lake grows, maintaining optimal performance becomes increasingly crucial. This compaction support ensures that your Iceberg tables on S3 can continue to scale efficiently without performance bottlenecks.
How does it work?
While the specifics of the underlying implementation are managed by AWS, the core principle is straightforward: AWS services interacting with Iceberg tables on S3 can now intelligently identify and merge smaller data files into larger ones. This process is designed to be efficient and non-disruptive to ongoing data operations. The integration ensures that this optimization happens seamlessly within the S3 environment, further simplifying the user experience.
This development underscores AWS’s commitment to providing robust and performant solutions for modern data warehousing and analytics. By empowering users with native compaction capabilities for widely adopted formats like Avro and ORC within the context of Apache Iceberg, AWS is making it easier than ever to build and manage high-performing data lakes.
We encourage you to explore this new feature and experience the enhanced performance and simplified management it offers for your Apache Iceberg tables on Amazon S3. This is a significant step forward in optimizing data lake operations and unlocking the full potential of your data.
Amazon S3 now supports compaction of Apache Avro and ORC formats for Apache Iceberg tables
AI has delivered the news.
The answer to the following question is obtained from Google Gemini.
Amazon published ‘Amazon S3 now supports compaction of Apache Avro and ORC formats for Apache Iceberg tables’ at 2025-07-15 17:58. Please write a detailed article about this news in a polite tone with relevant information. Please reply in English with the article only.