T3 Code · Performance over time

Every metric, over time.

Agents Range

120 metrics, Linux x64, 50 agents

Upstream nightlyPerformance forkNoise bandRange of attempts green = better, red = worse, beyond the noise band
The catalogue as a table: every metric, its unit, direction, band and latest value at 50 agents
MetricUnitBetterBandFromPlatformsLaneLatestVerdict
Client-side latency and throughput
Turns completed in the window
turns_completed
nhigher22.4%17 OctLinuxdaily188within noise
Agents served in the window
agents_served
nhigher11.9%17 OctLinuxdaily50within noise
Turn end to end p50
turn_p50_ms
mslower73.7%17 OctLinuxdaily14,630 msnoise band too wide for a verdict
Turn end to end p95
turn_p95_ms
mslower57.9%17 OctLinuxdaily30,090 msnoise band too wide for a verdict
Dispatch p50
dispatch_p50_ms
mslower64.3%17 OctLinuxdaily536 msnoise band too wide for a verdict
Dispatch p95
dispatch_p95_ms
mslower45.5%17 OctLinuxdaily1,063 msnoise band too wide for a verdict
First streamed item p50
first_item_p50_ms
mslower75.9%17 OctLinuxdaily4,977 msnoise band too wide for a verdict
Push latency p95
push_p95_ms
mscontext159.6%17 OctLinuxdaily1,160 msshown for context, no verdict
Thread switch: bounded open p50
switch_open_p50_ms
mslower58.5%17 OctLinuxdaily308 msnoise band too wide for a verdict
Thread switch: stream synchronised p95
switch_sync_p95_ms
mslower255.6%17 OctLinuxdaily71.9 msnoise band too wide for a verdict
Shell stream synchronised p95
shell_sync_p95_ms
mscontext–17 OctLinuxdaily–no records
MCP thread read poll p95
poll_thread_read_p95_ms
mscontext50.8%17 OctLinuxdaily407 msshown for context, no verdict
MCP project list poll p95
poll_project_list_p95_ms
mscontext108.4%17 OctLinuxdaily266 msshown for context, no verdict
HTTP bounded read poll p95
poll_http_bounded_p95_ms
mscontext21.4%17 OctLinuxdaily566 msshown for context, no verdict
Session MCP call p95
session_mcp_p95_ms
mscontext577.6%17 OctLinuxdaily413 msshown for context, no verdict
Thread events per client per second
thread_events_per_s
/scontext195.1%17 OctLinuxdaily0.26 /sshown for context, no verdict
Shell events per client per second
shell_events_per_s
/scontext11.7%17 OctLinuxdaily16.2 /sshown for context, no verdict
Failed calls
client_errors
nlower–17 OctLinuxdaily0within noise
Calls still running after the grace period
client_unfinished
nlower–17 OctLinuxdaily0within noise
Server CPU and event loop
Main thread busy
main_busy_pct
% of one corelower22.3%17 OctLinuxdaily68.4%within noise
Main thread user CPU
main_user_pct
% of one corelower24.6%17 OctLinuxdaily25.9%within noise
Main thread system CPU
main_sys_pct
% of one corelower19.3%17 OctLinuxdaily42.7%within noise
Process CPU, all threads
proc_cpu_pct
% of one corelower22.2%17 OctLinuxdaily102%within noise
Worker threads CPU
workers_cpu_pct
% of one corelower25.2%17 OctLinuxdaily27.4%within noise
Child processes CPU
children_cpu_pct
% of one corelower32.9%17 OctLinuxdaily99.7%within noise
Child processes alive
children_n
ncontext13.0%17 OctLinuxdaily104shown for context, no verdict
Event loop utilisation, mean
elu
0–1lower25.8%20 OctLinuxdaily0.95within noise
Event loop utilisation, max
elu_max
0–1lower38.1%20 OctLinuxdaily1within noise
Event loop delay p50 (preload histogram)
eld_p50_ms
mslower23.7%17 OctLinuxdaily13.2 mswithin noise
Event loop delay p99 (preload histogram)
eld_p99_ms
mslower24.2%17 OctLinuxdaily22.5 mswithin noise
Event loop delay max (preload histogram)
eld_max_ms
mslower86.9%17 OctLinuxdaily62.2 mswithin noise
Event loop delay max (server gauge)
server_eld_max_ms
mslower204.7%20 OctLinuxdaily21 mswithin noise
GC pause per second
gc_ms_per_s
ms/slower18.9%17 OctLinuxdaily12.0 ms/swithin noise
Seconds with a stall of 40 ms or more
stall_share_40ms
%lower140.4%17 OctLinuxdaily2.01%within noise
Seconds with a stall of 100 ms or more
stall_share_100ms
%lower398.2%17 OctLinuxdaily0%within noise
Seconds with a stall of 500 ms or more
stall_share_500ms
%lower–17 OctLinuxdaily0%no noise band yet, no verdict
Main thread 1 s busy p95
main_busy_1s_p95
%lower445.7%17 OctLinuxdaily76.1%within noise
Main thread run-queue wait
main_runq_wait_ms_per_s
ms/slower52.3%17 OctLinuxdaily0.082 ms/swithin noise
Main thread minor faults
main_minflt_per_s
/slower13.5%17 OctLinuxdaily29,390 /swithin noise
Main thread voluntary context switches
main_vcs_per_s
/slower36.1%17 OctLinuxdaily211 /swithin noise
Main thread involuntary context switches
main_nvcs_per_s
/slower57.1%17 OctLinuxdaily3.26 /swithin noise
Main thread read syscalls
main_syscr_per_s
/slower26.6%17 OctLinuxdaily840 /swithin noise
Main thread write syscalls
main_syscw_per_s
/slower20.1%17 OctLinuxdaily765 /swithin noise
Process read syscalls
proc_syscr_per_s
/slower27.1%17 OctLinuxdaily45,630 /swithin noise
Process CPU per served agent
cpu_per_session_pct
% per agentlower20.7%17 OctLinuxdaily0.10%within noise
Memory
Peak RSS
rss_peak_mib
MiBlower48.4%17 OctLinuxdaily1,115 MiBwithin noise
Peak V8 heap used
heap_peak_mib
MiBlower72.2%20 OctLinuxdaily605 MiBregressed beyond noise
Heap after the last GC
heap_floor_mib
MiBlower–20 OctLinuxdaily–no records
Memory of every T3 process (PSS)
pss_total_mib
MiBlower–17 OctLinuxweekly–no records
Memory of every T3 process (USS)
uss_total_mib
MiBlower–17 OctLinuxweekly–no records
RSS slope over 2 h
rss_slope_mib_per_h
MiB/hlower–17 OctLinuxweekly–no records
Heap slope over 2 h
heap_slope_mib_per_h
MiB/hlower–20 OctLinuxweekly–no records
WAL size, max
wal_max_mib
MiBlower22.5%17 OctLinuxdaily41.4 MiBwithin noise
WAL size, median
wal_median_mib
MiBlower22.5%17 OctLinuxdaily41.4 MiBwithin noise
SQLite
Reader transaction, mean
reader_tx_avg_ms
mslower22.4%17 OctLinuxdaily34.9 mswithin noise
Reader transaction p95
reader_tx_p95_ms
mslower19.8%17 OctLinuxdaily80.9 mswithin noise
Reader lock wait, mean
reader_tx_wait_avg_ms
mslower30.6%17 OctLinuxdaily4.51 mswithin noise
Reader lock wait p95
reader_tx_wait_p95_ms
mslower29.7%17 OctLinuxdaily29.0 mswithin noise
Reader statement p95
reader_stmt_p95_ms
mslower37.6%17 OctLinuxdaily40.2 mswithin noise
Writer transaction, mean
writer_tx_avg_ms
mslower29.5%17 OctLinuxdaily212 msregressed beyond noise
Writer transaction p95
writer_tx_p95_ms
mslower21.8%17 OctLinuxdaily578 msregressed beyond noise
Writer lock wait p95
writer_tx_wait_p95_ms
mslower106.2%17 OctLinuxdaily493 msregressed beyond noise
Reader transactions per second
reader_tx_per_s
/scontext19.6%17 OctLinuxdaily49.8 /sshown for context, no verdict
Writer transactions per second
writer_tx_per_s
/scontext31.7%17 OctLinuxdaily11.6 /sshown for context, no verdict
Reader slots busy, mean share
reader_pool_occupancy
0–1lower–17 OctLinuxdaily–no records
Free reader while a request waited
reader_free_while_waiting_s
slower–17 OctLinuxdaily–no records
SQL statement time per second
sql_time_ms_per_s
ms/slower42.2%17 OctLinuxdaily935 ms/sregressed beyond noise
Server spans
Thread shell read p95
get_thread_shell_p95_ms
mslower35.6%17 OctLinuxdaily68 mswithin noise
Thread snapshot window p95
snapshot_window_p95_ms
mslower41.7%17 OctLinuxdaily489 mswithin noise
MCP request authentication p95
mcp_auth_p95_ms
mslower77.1%17 OctLinuxdaily52.2 mswithin noise
Provider turn start p50
turn_start_p50_ms
mslower49.2%17 OctLinuxdaily850 mswithin noise
Run update handling p95
handle_run_update_p95_ms
mslower36.7%17 OctLinuxdaily153 mswithin noise
Subagent finalisation per child event p50
finalize_subagent_p50_ms
mslower–17 OctLinuxlater–no records
Effect queue wait p50
effect_queue_wait_p50_ms
mslower–17 OctLinuxdaily–no records
Effect queue wait p90
effect_queue_wait_p90_ms
mslower–17 OctLinuxdaily–no records
Items per provider commit
events_per_commit
nhigher–17 OctLinuxlater–no records
Spans in the window
spans_in_window
ncontext17.6%17 OctLinuxdaily314,600shown for context, no verdict
Trace lines dropped
trace_dropped
nlower–17 OctLinuxdaily–no records
Git and process creation
Git commands per turn
git_commands_per_turn
nlower–17 OctLinuxdaily–no records
Git time per turn
git_ms_per_turn
mslower–17 OctLinuxdaily–no records
Process creations per turn
forks_per_turn
nlower–17 OctLinuxweekly–no records
Main-thread gap after descendant polls, growth
ps_gap_ms_per_h
ms/hlower–17 OctLinuxweekly–no records
Startup, migrations and background work
Start to listening, first start
listening_first_ms
mslower18.6%17 OctLinuxrecord48,800 msregressed beyond noise
Start to listening, warm start
listening_warm_ms
mslower24.1%17 OctLinuxrecord1,738 mswithin noise
Start to port accept, first start
accept_first_ms
mslower20.6%17 OctLinuxrecord1,205 mswithin noise
Start to port accept, warm start
accept_warm_ms
mslower23.2%17 OctLinuxrecord1,211 mswithin noise
Start to database ready, first start
db_ready_first_ms
mslower18.4%17 OctLinuxrecord48,600 msregressed beyond noise
Start to database ready, warm start
db_ready_warm_ms
mslower21.3%17 OctLinuxrecord1,537 mswithin noise
Migrations on the era fixture, chain total
first_boot_migrations_ms
mslower10.7%17 OctLinuxrecord47,520 msregressed beyond noise
Per-migration seconds
migration_s
slower–17 OctLinuxrecord–no records
Peak extra disk during migrations
migration_extra_disk_mib
MiBlower0.0%17 OctLinuxrecord459 MiBregressed beyond noise
SIGTERM to exit
shutdown_s
slower193.4%17 OctLinuxdaily6.55 swithin noise
HTTP probe max during compaction
compaction_probe_max_ms
mslower–17 OctLinuxweekly–no records
Prune rate
compaction_pages_per_min
pages/mincontext–17 OctLinuxweekly–no records
Full snapshot GET of the largest thread
snapshot_full_get_ms
mslower3.8%17 OctLinuxrecord21,210 mswithin noise
Full snapshot size of the largest thread
snapshot_full_get_bytes
byteslower0.0%17 OctLinuxrecord506,900,000 byteswithin noise
Idle CPU, one idle agent
idle_cpu_pct
%lower–17 OctLinuxweekly–no records
Idle wakeups, one idle agent
idle_wakeups_per_s
/slower–17 OctLinuxweekly–no records
Memory after 30 min idle
idle_pss_30min_mib
MiBlower–17 OctLinuxweekly–no records
First turn after a cold start
first_turn_after_start_ms
mslower3.1%17 OctLinuxrecord7,654 mswithin noise
Queue and delivery
Queued run start p50
queue_start_p50_s
slower–20 OctLinuxrelease–no records
Queued run start p90
queue_start_p90_s
slower–20 OctLinuxrelease–no records
Queued run start max
queue_start_max_s
slower–20 OctLinuxrelease–no records
Run update consumer backlog, max
consumer_backlog_max
nlower–20 OctLinuxrelease–no records
Run update consumer drain time
consumer_drain_s
slower–20 OctLinuxrelease–no records
Delivery latency by run state p50
delivery_by_state_p50_s
slower––Linuxlater–no records
Logs and validity
Live event buffer full warnings
buffer_full_warnings
nlower0.0%17 OctLinuxdaily0within noise
Warning log lines
server_warn_lines
nlower66.7%17 OctLinuxdaily2within noise
Error log lines
server_error_lines
nlower–17 OctLinuxdaily0within noise
Stream resynchronisations
stream_resyncs
nlower––Linuxdaily–no records
Valid attempts
attempts_valid
ncontext–17 OctLinuxdaily3shown for context, no verdict
Attempts
attempts_total
ncontext–17 OctLinuxdaily3shown for context, no verdict
Host busy
host_busy_pct
%context27.9%17 OctLinuxdaily13.6%shown for context, no verdict
Host steal
host_steal_pct
%context–17 OctLinuxdaily0%shown for context, no verdict
Host I/O wait
host_iowait_pct
%context16.7%17 OctLinuxdaily0.025%shown for context, no verdict
Contaminated attempts
contaminated
ncontext–17 OctLinuxdaily0shown for context, no verdict
Fork builds only
Effect queue wait p95 (span attribute)
fork_effect_queue_wait_p95_ms
mslower––Linuxdaily–no records
Effect worker run p95
fork_run_effect_p95_ms
mslower––Linuxdaily–no records
Projection decode p95
fork_decode_projection_p95_ms
mslower––Linuxdaily–no records
Effect admission changes
fork_admission_changes
ncontext––Linuxdaily–no records