T3 Code · Performance over time
Every metric, over time.
Agents
Range
120 metrics, Linux x64, 50 agents
No metric matches.
Upstream nightlyPerformance forkNoise bandRange of attempts green = better, red = worse, beyond the noise band
The catalogue as a table: every metric, its unit, direction, band and latest value at 50 agents
| Metric | Unit | Better | Band | From | Platforms | Lane | Latest | Verdict |
|---|---|---|---|---|---|---|---|---|
| Client-side latency and throughput | ||||||||
Turns completed in the windowturns_completed | n | higher | 22.4% | 17 Oct | Linux | daily | 188 | within noise |
Agents served in the windowagents_served | n | higher | 11.9% | 17 Oct | Linux | daily | 50 | within noise |
Turn end to end p50turn_p50_ms | ms | lower | 73.7% | 17 Oct | Linux | daily | 14,630 ms | noise band too wide for a verdict |
Turn end to end p95turn_p95_ms | ms | lower | 57.9% | 17 Oct | Linux | daily | 30,090 ms | noise band too wide for a verdict |
Dispatch p50dispatch_p50_ms | ms | lower | 64.3% | 17 Oct | Linux | daily | 536 ms | noise band too wide for a verdict |
Dispatch p95dispatch_p95_ms | ms | lower | 45.5% | 17 Oct | Linux | daily | 1,063 ms | noise band too wide for a verdict |
First streamed item p50first_item_p50_ms | ms | lower | 75.9% | 17 Oct | Linux | daily | 4,977 ms | noise band too wide for a verdict |
Push latency p95push_p95_ms | ms | context | 159.6% | 17 Oct | Linux | daily | 1,160 ms | shown for context, no verdict |
Thread switch: bounded open p50switch_open_p50_ms | ms | lower | 58.5% | 17 Oct | Linux | daily | 308 ms | noise band too wide for a verdict |
Thread switch: stream synchronised p95switch_sync_p95_ms | ms | lower | 255.6% | 17 Oct | Linux | daily | 71.9 ms | noise band too wide for a verdict |
Shell stream synchronised p95shell_sync_p95_ms | ms | context | – | 17 Oct | Linux | daily | – | no records |
MCP thread read poll p95poll_thread_read_p95_ms | ms | context | 50.8% | 17 Oct | Linux | daily | 407 ms | shown for context, no verdict |
MCP project list poll p95poll_project_list_p95_ms | ms | context | 108.4% | 17 Oct | Linux | daily | 266 ms | shown for context, no verdict |
HTTP bounded read poll p95poll_http_bounded_p95_ms | ms | context | 21.4% | 17 Oct | Linux | daily | 566 ms | shown for context, no verdict |
Session MCP call p95session_mcp_p95_ms | ms | context | 577.6% | 17 Oct | Linux | daily | 413 ms | shown for context, no verdict |
Thread events per client per secondthread_events_per_s | /s | context | 195.1% | 17 Oct | Linux | daily | 0.26 /s | shown for context, no verdict |
Shell events per client per secondshell_events_per_s | /s | context | 11.7% | 17 Oct | Linux | daily | 16.2 /s | shown for context, no verdict |
Failed callsclient_errors | n | lower | – | 17 Oct | Linux | daily | 0 | within noise |
Calls still running after the grace periodclient_unfinished | n | lower | – | 17 Oct | Linux | daily | 0 | within noise |
| Server CPU and event loop | ||||||||
Main thread busymain_busy_pct | % of one core | lower | 22.3% | 17 Oct | Linux | daily | 68.4% | within noise |
Main thread user CPUmain_user_pct | % of one core | lower | 24.6% | 17 Oct | Linux | daily | 25.9% | within noise |
Main thread system CPUmain_sys_pct | % of one core | lower | 19.3% | 17 Oct | Linux | daily | 42.7% | within noise |
Process CPU, all threadsproc_cpu_pct | % of one core | lower | 22.2% | 17 Oct | Linux | daily | 102% | within noise |
Worker threads CPUworkers_cpu_pct | % of one core | lower | 25.2% | 17 Oct | Linux | daily | 27.4% | within noise |
Child processes CPUchildren_cpu_pct | % of one core | lower | 32.9% | 17 Oct | Linux | daily | 99.7% | within noise |
Child processes alivechildren_n | n | context | 13.0% | 17 Oct | Linux | daily | 104 | shown for context, no verdict |
Event loop utilisation, meanelu | 0–1 | lower | 25.8% | 20 Oct | Linux | daily | 0.95 | within noise |
Event loop utilisation, maxelu_max | 0–1 | lower | 38.1% | 20 Oct | Linux | daily | 1 | within noise |
Event loop delay p50 (preload histogram)eld_p50_ms | ms | lower | 23.7% | 17 Oct | Linux | daily | 13.2 ms | within noise |
Event loop delay p99 (preload histogram)eld_p99_ms | ms | lower | 24.2% | 17 Oct | Linux | daily | 22.5 ms | within noise |
Event loop delay max (preload histogram)eld_max_ms | ms | lower | 86.9% | 17 Oct | Linux | daily | 62.2 ms | within noise |
Event loop delay max (server gauge)server_eld_max_ms | ms | lower | 204.7% | 20 Oct | Linux | daily | 21 ms | within noise |
GC pause per secondgc_ms_per_s | ms/s | lower | 18.9% | 17 Oct | Linux | daily | 12.0 ms/s | within noise |
Seconds with a stall of 40 ms or morestall_share_40ms | % | lower | 140.4% | 17 Oct | Linux | daily | 2.01% | within noise |
Seconds with a stall of 100 ms or morestall_share_100ms | % | lower | 398.2% | 17 Oct | Linux | daily | 0% | within noise |
Seconds with a stall of 500 ms or morestall_share_500ms | % | lower | – | 17 Oct | Linux | daily | 0% | no noise band yet, no verdict |
Main thread 1 s busy p95main_busy_1s_p95 | % | lower | 445.7% | 17 Oct | Linux | daily | 76.1% | within noise |
Main thread run-queue waitmain_runq_wait_ms_per_s | ms/s | lower | 52.3% | 17 Oct | Linux | daily | 0.082 ms/s | within noise |
Main thread minor faultsmain_minflt_per_s | /s | lower | 13.5% | 17 Oct | Linux | daily | 29,390 /s | within noise |
Main thread voluntary context switchesmain_vcs_per_s | /s | lower | 36.1% | 17 Oct | Linux | daily | 211 /s | within noise |
Main thread involuntary context switchesmain_nvcs_per_s | /s | lower | 57.1% | 17 Oct | Linux | daily | 3.26 /s | within noise |
Main thread read syscallsmain_syscr_per_s | /s | lower | 26.6% | 17 Oct | Linux | daily | 840 /s | within noise |
Main thread write syscallsmain_syscw_per_s | /s | lower | 20.1% | 17 Oct | Linux | daily | 765 /s | within noise |
Process read syscallsproc_syscr_per_s | /s | lower | 27.1% | 17 Oct | Linux | daily | 45,630 /s | within noise |
Process CPU per served agentcpu_per_session_pct | % per agent | lower | 20.7% | 17 Oct | Linux | daily | 0.10% | within noise |
| Memory | ||||||||
Peak RSSrss_peak_mib | MiB | lower | 48.4% | 17 Oct | Linux | daily | 1,115 MiB | within noise |
Peak V8 heap usedheap_peak_mib | MiB | lower | 72.2% | 20 Oct | Linux | daily | 605 MiB | regressed beyond noise |
Heap after the last GCheap_floor_mib | MiB | lower | – | 20 Oct | Linux | daily | – | no records |
Memory of every T3 process (PSS)pss_total_mib | MiB | lower | – | 17 Oct | Linux | weekly | – | no records |
Memory of every T3 process (USS)uss_total_mib | MiB | lower | – | 17 Oct | Linux | weekly | – | no records |
RSS slope over 2 hrss_slope_mib_per_h | MiB/h | lower | – | 17 Oct | Linux | weekly | – | no records |
Heap slope over 2 hheap_slope_mib_per_h | MiB/h | lower | – | 20 Oct | Linux | weekly | – | no records |
WAL size, maxwal_max_mib | MiB | lower | 22.5% | 17 Oct | Linux | daily | 41.4 MiB | within noise |
WAL size, medianwal_median_mib | MiB | lower | 22.5% | 17 Oct | Linux | daily | 41.4 MiB | within noise |
| SQLite | ||||||||
Reader transaction, meanreader_tx_avg_ms | ms | lower | 22.4% | 17 Oct | Linux | daily | 34.9 ms | within noise |
Reader transaction p95reader_tx_p95_ms | ms | lower | 19.8% | 17 Oct | Linux | daily | 80.9 ms | within noise |
Reader lock wait, meanreader_tx_wait_avg_ms | ms | lower | 30.6% | 17 Oct | Linux | daily | 4.51 ms | within noise |
Reader lock wait p95reader_tx_wait_p95_ms | ms | lower | 29.7% | 17 Oct | Linux | daily | 29.0 ms | within noise |
Reader statement p95reader_stmt_p95_ms | ms | lower | 37.6% | 17 Oct | Linux | daily | 40.2 ms | within noise |
Writer transaction, meanwriter_tx_avg_ms | ms | lower | 29.5% | 17 Oct | Linux | daily | 212 ms | regressed beyond noise |
Writer transaction p95writer_tx_p95_ms | ms | lower | 21.8% | 17 Oct | Linux | daily | 578 ms | regressed beyond noise |
Writer lock wait p95writer_tx_wait_p95_ms | ms | lower | 106.2% | 17 Oct | Linux | daily | 493 ms | regressed beyond noise |
Reader transactions per secondreader_tx_per_s | /s | context | 19.6% | 17 Oct | Linux | daily | 49.8 /s | shown for context, no verdict |
Writer transactions per secondwriter_tx_per_s | /s | context | 31.7% | 17 Oct | Linux | daily | 11.6 /s | shown for context, no verdict |
Reader slots busy, mean sharereader_pool_occupancy | 0–1 | lower | – | 17 Oct | Linux | daily | – | no records |
Free reader while a request waitedreader_free_while_waiting_s | s | lower | – | 17 Oct | Linux | daily | – | no records |
SQL statement time per secondsql_time_ms_per_s | ms/s | lower | 42.2% | 17 Oct | Linux | daily | 935 ms/s | regressed beyond noise |
| Server spans | ||||||||
Thread shell read p95get_thread_shell_p95_ms | ms | lower | 35.6% | 17 Oct | Linux | daily | 68 ms | within noise |
Thread snapshot window p95snapshot_window_p95_ms | ms | lower | 41.7% | 17 Oct | Linux | daily | 489 ms | within noise |
MCP request authentication p95mcp_auth_p95_ms | ms | lower | 77.1% | 17 Oct | Linux | daily | 52.2 ms | within noise |
Provider turn start p50turn_start_p50_ms | ms | lower | 49.2% | 17 Oct | Linux | daily | 850 ms | within noise |
Run update handling p95handle_run_update_p95_ms | ms | lower | 36.7% | 17 Oct | Linux | daily | 153 ms | within noise |
Subagent finalisation per child event p50finalize_subagent_p50_ms | ms | lower | – | 17 Oct | Linux | later | – | no records |
Effect queue wait p50effect_queue_wait_p50_ms | ms | lower | – | 17 Oct | Linux | daily | – | no records |
Effect queue wait p90effect_queue_wait_p90_ms | ms | lower | – | 17 Oct | Linux | daily | – | no records |
Items per provider commitevents_per_commit | n | higher | – | 17 Oct | Linux | later | – | no records |
Spans in the windowspans_in_window | n | context | 17.6% | 17 Oct | Linux | daily | 314,600 | shown for context, no verdict |
Trace lines droppedtrace_dropped | n | lower | – | 17 Oct | Linux | daily | – | no records |
| Git and process creation | ||||||||
Git commands per turngit_commands_per_turn | n | lower | – | 17 Oct | Linux | daily | – | no records |
Git time per turngit_ms_per_turn | ms | lower | – | 17 Oct | Linux | daily | – | no records |
Process creations per turnforks_per_turn | n | lower | – | 17 Oct | Linux | weekly | – | no records |
Main-thread gap after descendant polls, growthps_gap_ms_per_h | ms/h | lower | – | 17 Oct | Linux | weekly | – | no records |
| Startup, migrations and background work | ||||||||
Start to listening, first startlistening_first_ms | ms | lower | 18.6% | 17 Oct | Linux | record | 48,800 ms | regressed beyond noise |
Start to listening, warm startlistening_warm_ms | ms | lower | 24.1% | 17 Oct | Linux | record | 1,738 ms | within noise |
Start to port accept, first startaccept_first_ms | ms | lower | 20.6% | 17 Oct | Linux | record | 1,205 ms | within noise |
Start to port accept, warm startaccept_warm_ms | ms | lower | 23.2% | 17 Oct | Linux | record | 1,211 ms | within noise |
Start to database ready, first startdb_ready_first_ms | ms | lower | 18.4% | 17 Oct | Linux | record | 48,600 ms | regressed beyond noise |
Start to database ready, warm startdb_ready_warm_ms | ms | lower | 21.3% | 17 Oct | Linux | record | 1,537 ms | within noise |
Migrations on the era fixture, chain totalfirst_boot_migrations_ms | ms | lower | 10.7% | 17 Oct | Linux | record | 47,520 ms | regressed beyond noise |
Per-migration secondsmigration_s | s | lower | – | 17 Oct | Linux | record | – | no records |
Peak extra disk during migrationsmigration_extra_disk_mib | MiB | lower | 0.0% | 17 Oct | Linux | record | 459 MiB | regressed beyond noise |
SIGTERM to exitshutdown_s | s | lower | 193.4% | 17 Oct | Linux | daily | 6.55 s | within noise |
HTTP probe max during compactioncompaction_probe_max_ms | ms | lower | – | 17 Oct | Linux | weekly | – | no records |
Prune ratecompaction_pages_per_min | pages/min | context | – | 17 Oct | Linux | weekly | – | no records |
Full snapshot GET of the largest threadsnapshot_full_get_ms | ms | lower | 3.8% | 17 Oct | Linux | record | 21,210 ms | within noise |
Full snapshot size of the largest threadsnapshot_full_get_bytes | bytes | lower | 0.0% | 17 Oct | Linux | record | 506,900,000 bytes | within noise |
Idle CPU, one idle agentidle_cpu_pct | % | lower | – | 17 Oct | Linux | weekly | – | no records |
Idle wakeups, one idle agentidle_wakeups_per_s | /s | lower | – | 17 Oct | Linux | weekly | – | no records |
Memory after 30 min idleidle_pss_30min_mib | MiB | lower | – | 17 Oct | Linux | weekly | – | no records |
First turn after a cold startfirst_turn_after_start_ms | ms | lower | 3.1% | 17 Oct | Linux | record | 7,654 ms | within noise |
| Queue and delivery | ||||||||
Queued run start p50queue_start_p50_s | s | lower | – | 20 Oct | Linux | release | – | no records |
Queued run start p90queue_start_p90_s | s | lower | – | 20 Oct | Linux | release | – | no records |
Queued run start maxqueue_start_max_s | s | lower | – | 20 Oct | Linux | release | – | no records |
Run update consumer backlog, maxconsumer_backlog_max | n | lower | – | 20 Oct | Linux | release | – | no records |
Run update consumer drain timeconsumer_drain_s | s | lower | – | 20 Oct | Linux | release | – | no records |
Delivery latency by run state p50delivery_by_state_p50_s | s | lower | – | – | Linux | later | – | no records |
| Logs and validity | ||||||||
Live event buffer full warningsbuffer_full_warnings | n | lower | 0.0% | 17 Oct | Linux | daily | 0 | within noise |
Warning log linesserver_warn_lines | n | lower | 66.7% | 17 Oct | Linux | daily | 2 | within noise |
Error log linesserver_error_lines | n | lower | – | 17 Oct | Linux | daily | 0 | within noise |
Stream resynchronisationsstream_resyncs | n | lower | – | – | Linux | daily | – | no records |
Valid attemptsattempts_valid | n | context | – | 17 Oct | Linux | daily | 3 | shown for context, no verdict |
Attemptsattempts_total | n | context | – | 17 Oct | Linux | daily | 3 | shown for context, no verdict |
Host busyhost_busy_pct | % | context | 27.9% | 17 Oct | Linux | daily | 13.6% | shown for context, no verdict |
Host stealhost_steal_pct | % | context | – | 17 Oct | Linux | daily | 0% | shown for context, no verdict |
Host I/O waithost_iowait_pct | % | context | 16.7% | 17 Oct | Linux | daily | 0.025% | shown for context, no verdict |
Contaminated attemptscontaminated | n | context | – | 17 Oct | Linux | daily | 0 | shown for context, no verdict |
| Fork builds only | ||||||||
Effect queue wait p95 (span attribute)fork_effect_queue_wait_p95_ms | ms | lower | – | – | Linux | daily | – | no records |
Effect worker run p95fork_run_effect_p95_ms | ms | lower | – | – | Linux | daily | – | no records |
Projection decode p95fork_decode_projection_p95_ms | ms | lower | – | – | Linux | daily | – | no records |
Effect admission changesfork_admission_changes | n | context | – | – | Linux | daily | – | no records |