LangSmith
LangSmith is an AI agent and LLM observability platform with tracing, monitoring, and insights for debugging and cost tracking.
Last verified:
What is LangSmith?
LangSmith is the framework-agnostic agent engineering platform for observing, evaluating, and deploying AI agents and LLM applications. It provides complete visibility into agent behavior through tracing, real-time monitoring, and automated insights to debug failures, pinchpoint latency/cost issues, and track performance metrics.
Key features include agent tracing with native support for popular frameworks and OpenTelemetry, monitoring dashboards with cost/latency/error tracking and online evals, Insights for automatic trace clustering and failure mode detection, SmithDB for purpose-built agent observability with sub-second performance, Evaluation with LLM-as-judge/code-based/multi-turn evaluators, Deployment with human-in-the-loop approvals and durable runtime, and Fleet for no-code agent creation for non-technical teams.
LangSmith is designed for AI/LLM application developers, agent engineering teams, data scientists building RAG pipelines, subject-matter experts reviewing agent outputs, and non-technical teams needing automated task agents. It works with any LLM framework including OpenAI SDK, Anthropic SDK, Vercel AI SDK, LlamaIndex, or custom implementations, not just LangChain.
LangSmith pricing
Pricing model: Freemium
LangSmith offers three plans: Developer ($0/seat/month, pay-as-you-go) includes 5k base traces/month, tracing, online/offline evals, Prompt Hub/Playground/Canvas, annotation queues, monitoring/alerting, unlimited Fleet agents, 50 Fleet runs/month, community support, 1 seat. Plus ($39/seat/month, pay-as-you-go) includes 10k base traces/month, 1 free dev-sized deployment, email support, unlimited Fleet agents, 500 Fleet runs/month, unlimited seats, 3 workspaces, LangSmith Engine access, LangSmith Sandboxes access. Enterprise (custom pricing) includes all Plus features plus hybrid/self-hosted hosting, custom SSO/RBAC, access to deployed engineering team, support SLA, team trainings, architectural guidance, custom seats/workspaces, custom packages for Fleet/Engine/Sandboxes. Base traces cost $2.50 per 1k (14-day retention), extended traces cost $5.00 per 1k (400-day retention). Additional dev deployments: $0.005/run, production uptime: $0.0036/min, dev uptime: $0.0007/min. Engine: $1.50/LCU. Sandboxes: CPU $0.0576/vCPU-hr, Memory $0.0185/GiB-hr, Storage $0.000123/GiB-hr. Model costs billed separately by provider. Startup plan available with discounted rates.
LangSmith pros
- Framework-agnostic - works with any LLM framework or agent stack
- Native tracing for popular agent frameworks and OpenTelemetry support
- SDKs for Python, TypeScript, Go, and Java
- Sub-second query performance across millions of traces with SmithDB
- Real-time monitoring with cost tracking and latency metrics (P50, P99)
- Online LLM-as-judge and code-based evals on production traces
- Unsupervised topic clustering for automatic pattern detection
- Built-in AI assistant Polly for debugging large traces
- Human feedback through annotation queues with domain experts
- Calibrate LLM judges to match human preferences
- Side-by-side comparison view for agent version benchmarking
- CI/CD integration with pytest, Vitest, and GitHub workflows
- No-code Fleet agents for non-technical team members
- Self-hosted deployment option for data residency requirements
- Async callback handler ensures no latency added to applications
- Enterprise security with SSO/SAML, SCIM, RBAC/ABAC, audit logs
- Parksmith Engine automatically diagnoses failures and recommends fixes
- LangSmith Sandboxes for isolated agent-generated code execution
- Prompt Hub and Playground with auto-improving prompts via Polly
- Exactly-once execution for durable agent runtime
LangSmith cons
- Pay-as-you-go model can be unpredictable for high trace volumes
- Base traces have only 14-day retention period
- Engine usage is metered at $1.50 per LCU which can be expensive
- Developer plan limited to 1 seat maximum
- Sandboxes not available for self-hosted deployments yet
- Fleet self-hosting only available in Beta
- Plus plan requires payment for additional deployment runs beyond 1 free
- Monitoring alerts require webhook or PagerDuty setup
- Insights feature still in beta
- Enterprise plan requires annual invoice billing
Frequently asked questions about LangSmith
Why do teams need an LLM observability platform?
Teams need an LLM observability platform to understand how their AI applications behave in production. LLM observability platforms provide visibility into RAG pipelines, AI agent decisions, track model performance metrics like cost and latency, and help debug complex failures and hallucinations by showing the complete execution trace from end-to-end.
What frameworks and libraries does LangSmith work with?
LangSmith works with any LLM framework. You can trace applications built with OpenAI SDK, Anthropic SDK, Vercel AI SDK, LlamaIndex, or custom implementations, not just LangChain. OpenTelemetry support connects to existing pipelines.
What kind of evaluators does LangSmith support?
LangSmith's evaluation framework supports multiple evaluator types: human evaluation through annotation queues, heuristic checks like validating outputs or checking if code compiles, LLM-as-judge evaluators that score against criteria you define, and pairwise comparisons. You can also write custom evaluators in Python or TypeScript with any business logic you need, from correctness and ground truth matching to hallucination detection and guardrails validation.
What is the difference between offline and online evaluation?
Offline evaluation runs against curated datasets during development to catch regressions before deployment. They act as unit tests for your LLM application. Online evaluation scores real-world production traffic in real-time to detect quality drift. LangSmith supports both as part of an end-to-end evaluation lifecycle.
Can I use LangSmith Observability without LangSmith Evaluation?
Yes. Observability and Evaluation work well together but don't require each other. Start with tracing and monitoring, then add evals when ready. For all plan types, you'll get access to both and only pay for what you use.
I can't have data leave my environment. Can I self-host LangSmith?
Yes. LangSmith offers managed cloud, bring-your-own-cloud (BYOC), and self-hosted options for teams with data residency requirements. When using LangSmith hosted at smith.langchain.com, data is stored in GCP us-central-1. If you're on the Enterprise plan, we can deliver LangSmith to run on your Kubernetes cluster in AWS, GCP, or Azure so that data never leaves your environment.
Will LangSmith add latency to my application?
No. The LangSmith SDK uses an async callback handler that sends traces to a distributed collector. Your application performance is never impacted. If LangSmith experiences an incident, your agent keeps running normally.
Will you train on the data that I send LangSmith?
We will not train on your data, and you own all rights to your data. See LangSmith Terms of Service for more information.
How does LangSmith evaluate AI agents and multi-turn workflows?
Agent evaluation in LangSmith captures the full trajectory of steps, tool calls, and reasoning your agent took. Define evaluators that score intermediate decisions and agent behavior to debug complex agent workflows and pinpoint where things went wrong. Conversation threads, tools, sub-agent delegation, and memory are first-class concepts throughout the platform.
How much does LangSmith cost?
LangSmith has a free tier for development and small-scale production. The Developer plan is $0/seat/month with 5k base traces/month included. The Plus plan is $39/seat/month with 10k base traces/month included. Paid plans scale with trace volume. Enterprise plan has custom pricing. See the pricing page for details on pay-as-you-go rates for traces, deployments, Fleet runs, Engine LCUs, and Sandboxes, or contact sales for enterprise pricing.