Fastmail Exporter

A high-performance Go application to export emails from Fastmail accounts via IMAP with clean, modular architecture.

Features

• Multiple Export Formats: mbox, EML, JSON, and NDJSON (streaming)
• Zstandard Compression: Optional high-performance compression for all formats
• High Performance: Concurrent email processing with configurable worker pools
• Clean Architecture: Separation of concerns with modular design
• Flexible Configuration: YAML-based configuration with sensible defaults
• Comprehensive Logging: Configurable log levels for debugging and monitoring
• Advanced Attachment Handling: Multiple attachment handling modes (inline, separate files, skip) with size limits
• Timestamped Exports: Optional timestamp prefixes or timestamped export folders
• Date Range Filtering: Export emails within specific time periods
• Folder Selection: Export specific folders or all folders

Installation

Prerequisites

• Go 1.21 or higher
• Fastmail account with IMAP access
• App-specific password (recommended for security)

Build from Source

git clone <repository-url>
cd fastmail_exporter
go mod tidy
go build -o bin/fastmail-exporter cmd/fastmail-exporter/main.go

Configuration

1. Copy the example configuration:

cp config.example.yaml config.yaml

2. Edit config.yaml with your Fastmail credentials:

imap:
  server: "imap.fastmail.com"
  port: 993
  username: "your-email@fastmail.com"
  password: "your-app-password"  # Use app-specific password
  use_tls: true

export:
  format: "mbox"  # Options: mbox, eml, json, ndjson
  output_path: "./exported_emails"
  include_attachments: true
  max_concurrent: 5
  
  # Attachment settings
  attachment_handling: "separate_files"  # Options: inline, separate_files, skip
  max_attachment_size: 10485760         # 10MB in bytes (0 = no limit)
  attachment_directory: ""              # Custom path, defaults to "attachments" subfolder
  
  # Compression settings
  compression: true         # Enable zstandard compression
  compression_level: 3      # Compression level 1-22 (3 is default)
  
  # Timestamp settings
  timestamp: "folder"      # Options: none, prefix, folder
                           # none: no timestamp
                           # prefix: add timestamp to filename
                           # folder: timestamped export folder (recommended)
  
  # Optional: Specific folders to export
  folders:
    - "INBOX"
    - "Sent"
    - "Archive"

log_level: "info"

Usage

Basic Export

./bin/fastmail-exporter -config config.yaml

Command Line Options

• -config: Path to configuration file (default: config.yaml)

Recent Features

Advanced Attachment Handling

The exporter now provides flexible attachment handling options:

• inline: Embed attachments directly in export files (default for MBOX/EML)
• separate_files: Save attachments as separate files (recommended for JSON/NDJSON)
• skip: Exclude attachments entirely to save space

Configure size limits and custom attachment directories for better organization.

Timestamped Exports

Organize your exports with automatic timestamping:

• folder (recommended): Creates timestamped folders like 20240208_143000/
• prefix: Adds timestamp prefixes to filenames like 20240208_143000_INBOX.mbox
• none: No timestamp (default behavior)

Export Formats

MBOX Format

Standard Unix mailbox format, compatible with most email clients.

EML Format

Individual email files in standard email format, one file per email.

JSON Format

Structured JSON format with complete email metadata and content, stored as a JSON array.

NDJSON Format

Newline-delimited JSON format ideal for streaming and processing large datasets. Each email is a separate JSON object on its own line, making it perfect for:

• Streaming processing with tools like jq, grep, or custom scripts
• Import into databases or analytics tools
• Line-by-line processing without loading entire file into memory

Compression

The exporter supports optional zstandard (zstd) compression for all export formats:

Benefits

• High compression ratio: 60-80% space savings typical for email data
• Fast compression/decompression: Optimized for performance
• Industry standard: Widely supported compression format

Configuration

export:
  compression: true         # Enable compression
  compression_level: 3      # Level 1-22 (1=fast, 22=best compression, 3=balanced)

File Extensions

When compression is enabled, files get .zst extensions:

• INBOX.mbox → INBOX.mbox.zst
• INBOX.ndjson → INBOX.ndjson.zst
• Individual EML files: email_123.eml.zst

Example Usage

# Decompress files
zstd -d INBOX.mbox.zst

# Stream NDJSON without decompression
zstd -dc INBOX.ndjson.zst | jq '.Subject' | head -10

Architecture

The application follows clean architecture principles:

• cmd/: Application entry points
• internal/: Private application logic
  • client/: IMAP client implementation
  • config/: Configuration management
  • exporter/: Export orchestration and file writers
• pkg/: Public packages
  • models/: Data structures
  • logger/: Logging utilities

Performance Features

• Concurrent Processing: Configurable worker pools for parallel email fetching
• Memory Efficient: Streaming email processing to handle large mailboxes
• Connection Pooling: Efficient IMAP connection management
• Progress Monitoring: Real-time logging of export progress

Security

• App-Specific Passwords: Recommended for Fastmail authentication
• TLS Encryption: Secure IMAP connections
• No Credential Storage: Configuration-based authentication only

Troubleshooting

Common Issues

1. Authentication Failed: Use app-specific passwords instead of your main password
2. Connection Timeout: Check firewall settings for port 993 (IMAPS)
3. Permission Denied: Ensure output directory is writable

Fastmail Setup

1. Log into Fastmail
2. Go to Settings → Password & Security
3. Create an app-specific password for this tool
4. Use the app-specific password in your configuration

Contributing

1. Follow Go best practices and formatting (go fmt)
2. Add tests for new functionality
3. Update documentation for new features
4. Ensure clean, modular architecture

Testing

The project includes comprehensive test coverage for core functionality:

# Run all tests
go test ./...

# Run tests with coverage
go test -cover ./...

# Run tests with verbose output
go test -v ./...

# Run specific package tests
go test ./internal/config
go test ./internal/exporter
go test ./internal/client
go test ./pkg/models

Test Coverage

• Configuration Tests (internal/config/config_test.go): Validates YAML parsing, defaults, and validation
• Writer Tests (internal/exporter/writer_test.go): Tests all export formats, compression, and timestamp handling
• IMAP Client Tests (internal/client/imap_test.go): Tests email parsing, attachment handling, and encoding
• Model Tests (pkg/models/email_test.go): Validates data structures and constants

License

MIT License - see LICENSE file for details.