Qsv
Blazing-fast Data-Wrangling toolkit
Last verified:
What is Qsv?
qsv is a blazing-fast, command-line data-wrangling toolkit built with Rust and Polars to slice, dice, and analyze CSV data in seconds. Designed for very large datasets (even gigabytes with millions of rows), it provides 50+ composable commands for everyday data tasks without requiring specialized programming skills. The tool handles messy CSV files from disparate sources and is available on Linux, macOS, and Windows.
Key features include: joining and merging CSVs (join, joinp), running Polars SQL queries directly on CSV files (sqlp), computing summary statistics and frequency tables (stats, frequency), transforming columns with string/date/math/currency operations (apply), geocoding locations, fetching data from web services (fetch/fetchpost), converting between formats (Excel, JSON, Parquet, PostgreSQL, SQLite), removing duplicates, sorting arbitrarily large files, indexing for instant random access, and built-in support for Luau and Python scripting for complex transformations. It also supports AI-powered data description via describegpt using OpenAI-compatible LLMs.
qsv is ideal for data analysts, data engineers, data scientists, CSV/API users, government data publishers (CKAN integration), and anyone who needs to wrangle tabular data at scale from the command line. It's used for one-off data cleaning tasks as well as mission-critical data pipelines. The companion desktop application qsv pro offers a graphical interface for the same functionality.
Qsv pricing
Pricing model: Freemium
qsv CLI is free and open-source (BSD-3 license). Download prebuilt binaries, install via package managers (Homebrew, AUR, MacPorts, Nix, Scoop, Conda-forge), or compile from source. qsv pro is a paid desktop application with a free tier; paid features require purchasing a license key from store.dathere.com. License is USD pricing, 1 device at a time per key. Additional license keys needed for multiple simultaneous devices. Free plan includes basic data viewing and transformation; paid plan unlocks full feature set including Workflow, API access, CKAN upload/download, and advanced recipes.
Qsv pros
- Blazing fast performance built on Rust and Polars
- Pleasingly parallel and CPU-accelerated processing
- Handles arbitrarily large datasets without running out of memory
- 50+ specialized commands for comprehensive data wrangling
- No programming required for most data tasks
- Built-in SQL support via sqlp command (Polars SQL)
- Automatic indexing for instant random access to CSV rows
- Multithreaded commands that speed up without index
- Supports multiple file formats: Excel, JSON, Parquet, ODS, PostgreSQL, SQLite
- Built-in Luau DSL for complex data pipelines
- Python expression support via py command
- AI-powered data description with describegpt using LLMs
- Geocoding capability with 360,000 records/second speed
- HTTP fetch/fetchpost with rate limiting and JSON query support
- Automatic Snappy compression/decompression
- Sniff command detects CSV metadata with 99.55% accuracy
- Diff command compares two CSVs in under 600ms for 1M rows
- Schema inference outputs JSON Schema Validation format
- Validate command checks CSV against JSON Schema blazingly fast
- Cross-platform: Linux, macOS, Windows support
- Prebuilt binaries with CPU optimizations (AVX2, AVX-512, NEON)
- Self-update mechanism with signature verification
- qsvlite variant is only 13% the size of full qsv
- CKAN integration for data publishing workflows
Qsv cons
- Command-line interface has steep learning curve for non-technical users
- Windows Defender may show false-positive warnings on some binaries
- Musl prebuilt binaries do not include Luau feature
- Python feature requires separate binary variant (qsvpy) per Python version
- Not strictly RFC 4180 compliant to handle real-world messy CSVs
- Requires UTF-8 encoded input (no other encodings natively)
- Some commands load entire CSV into memory (memory intensive)
- Older CPUs may get
- errors without portable variant
- GUI features in qsv pro are limited compared to CLI
- qsv pro paid features require license key purchase
- No official multi-language support (English-only messages)
- Cannot process files larger than RAM without extdedup/extsort
- Regex engine lacks look-around and backreferences
- Windows PowerShell redirects output as UTF16-LE by default
- Some features gated behind feature flags during compilation
- No built-in visualization/charting capabilities
- Biased toward CSV as lingua franca (other formats secondary)
- Homebrew installation enables only apply and luau features
- No native support for look-around regex patterns
- Experimental features may be outdated or non-functional
Frequently asked questions about Qsv
What is qsv and what does it do?
qsv is a blazing-fast, parallel, CPU-accelerated data-wrangling toolkit built with Rust and Polars. It provides 50+ commands to slice, dice, analyze, filter, transform, join, validate, and convert CSV and spreadsheet data from the command line. It's designed to handle very large datasets without specialized programming skills.
Is qsv free to use?
Yes, the qsv CLI tool is free and open-source (BSD-3 license). You can download prebuilt binaries for Linux, macOS, and Windows, install via package managers, or compile from source. qsv pro has a free tier with basic features, but paid features require a license key purchase.
What file formats does qsv support?
qsv natively recognizes UTF-8 CSV, SSV, and TSV files. It also supports Excel (.xls, .xlsx, .xlsm, .xlsb), OpenDocument Spreadsheet (.ods), JSON/JSONL, Parquet, Apache Arrow IPC, Avro, and can directly populate PostgreSQL and SQLite databases. The to command converts CSV to Excel, ODS, Parquet, and Data Package formats.
How fast is qsv compared to other tools?
qsv is designed for maximum speed. Examples: counting rows in a 15GB, 27M row NYC 311 dataset takes 11.87 seconds without an index, instantaneous with an index. Diff compares two CSVs with 1M rows x 9 columns in under 600ms. Geocoding runs at 360,000 records/second. Snappy compresses 15GB in 5.77 seconds at 2.58 GB/sec.
Do I need to program to use qsv?
No, you don't need to program to wrangle data at scale with qsv. Most commands work with sensible defaults and require no coding. However, for complex transformations, qsv has built-in support for Luau (recommended, faster, no external dependencies) and Python expressions.
What is the difference between qsv and qsv pro?
qsv is the command-line toolkit with 50+ commands. qsv pro is a desktop application with a graphical user interface based on qsv CLI, offering drag-and-drop workflow, interactive data tables, Polars SQL queries, node-based pipeline builder, API access, CKAN integration, and Toolbox scripts. qsv pro has a free tier; advanced features require a paid license.
How do I install qsv?
Install via: (1) Download prebuilt binaries from GitHub releases for your platform, (2) Package managers: brew install qsv (macOS/Linux), yay -S qsv (Arch), scoop install qsv (Windows), conda install conda-forge::qsv, (3) Compile from source with cargo install qsv --locked --features all_features, or (4) Use the Debian/Ubuntu repository maintained by datHere.
Can qsv handle files larger than my RAM?
Yes, many qsv commands are streaming and use constant memory. For commands requiring full file load, extdedup removes duplicates from arbitrarily large files using memory-mapped on-disk hash tables, and extsort sorts arbitrarily large files using multithreaded external merge sort. joinp can also process files larger than RAM.
What is theLuau feature in qsv?
Luau is qsv's Domain-Specific Language (DSL) for data-wrangling. The luau command lets you create computed columns, filter rows, compute aggregations, and build complex data pipelines by executing Luau expressions/scripts. It's faster than Python, more memory-efficient, has no external dependencies, and includes qsv-specific helper functions for production pipelines.
How do I run SQL queries on CSV files with qsv?
Use the sqlp command to run Polars SQL queries against one or several CSVs, Parquet, JSONL, and Arrow files. sqlp converts SQL queries to blazing-fast Polars LazyFrame expressions and can process larger-than-memory CSV files. Results can be saved as CSV, JSON, JSONL, Parquet, Apache Arrow IPC, or Avro.