zed/crates/collab/k8s/collab.template.yml
Conrad Irwin 6ff69faf37
Start to send data to Snowflake too (#20698)
This PR adds support for sending telemetry events to AWS Kinesis.

In our AWS account we now have three new things:
* The [Kinesis data
stream](https://us-east-1.console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/details/zed-telemetry/monitoring)
that we will actually write to.
* A [Firehose for
Axiom](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-axiom/monitoring)
that sends events from that stream to Axiom for ad-hoc queries over
recent data.
* A [Firehose for
Snowflake](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-snowflake/monitoring)
that sends events from that stream to Snowflake for long-term retention.
This Firehose also backs up data into an S3 bucket in case we want to
change how the system works in the future.

In a follow-up PR, we'll add support for ad-hoc telemetry events; and
slowly move away from the current Clickhouse defined schemas; though we
won't move off click house until we have what we need in Snowflake.

Co-Authored-By: Nathan <nathan@zed.dev>

Release Notes:

- N/A
2024-11-15 12:58:00 -07:00

263 lines
7.8 KiB
YAML

---
apiVersion: v1
kind: Namespace
metadata:
name: ${ZED_KUBE_NAMESPACE}
---
kind: Service
apiVersion: v1
metadata:
namespace: ${ZED_KUBE_NAMESPACE}
name: ${ZED_SERVICE_NAME}
annotations:
service.beta.kubernetes.io/do-loadbalancer-name: "${ZED_SERVICE_NAME}-${ZED_KUBE_NAMESPACE}"
service.beta.kubernetes.io/do-loadbalancer-size-unit: "${ZED_LOAD_BALANCER_SIZE_UNIT}"
service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
service.beta.kubernetes.io/do-loadbalancer-certificate-id: ${ZED_DO_CERTIFICATE_ID}
service.beta.kubernetes.io/do-loadbalancer-disable-lets-encrypt-dns-records: "true"
spec:
type: LoadBalancer
selector:
app: ${ZED_SERVICE_NAME}
ports:
- name: web
protocol: TCP
port: 443
targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: ${ZED_KUBE_NAMESPACE}
name: ${ZED_SERVICE_NAME}
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: ${ZED_SERVICE_NAME}
template:
metadata:
labels:
app: ${ZED_SERVICE_NAME}
spec:
containers:
- name: ${ZED_SERVICE_NAME}
image: "${ZED_IMAGE_ID}"
args:
- serve
- ${ZED_SERVICE_NAME}
ports:
- containerPort: 8080
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 1
periodSeconds: 1
startupProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 1
periodSeconds: 1
failureThreshold: 15
env:
- name: HTTP_PORT
value: "8080"
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: database
key: url
- name: DATABASE_MAX_CONNECTIONS
value: "${DATABASE_MAX_CONNECTIONS}"
- name: API_TOKEN
valueFrom:
secretKeyRef:
name: api
key: token
- name: LLM_API_SECRET
valueFrom:
secretKeyRef:
name: llm-token
key: secret
- name: LLM_DATABASE_URL
valueFrom:
secretKeyRef:
name: llm-database
key: url
- name: LLM_DATABASE_MAX_CONNECTIONS
value: "${LLM_DATABASE_MAX_CONNECTIONS}"
- name: ZED_CLIENT_CHECKSUM_SEED
valueFrom:
secretKeyRef:
name: zed-client
key: checksum-seed
- name: LIVE_KIT_SERVER
valueFrom:
secretKeyRef:
name: livekit
key: server
- name: LIVE_KIT_KEY
valueFrom:
secretKeyRef:
name: livekit
key: key
- name: LIVE_KIT_SECRET
valueFrom:
secretKeyRef:
name: livekit
key: secret
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
name: openai
key: api_key
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: anthropic
key: api_key
- name: ANTHROPIC_STAFF_API_KEY
valueFrom:
secretKeyRef:
name: anthropic
key: staff_api_key
- name: LLM_CLOSED_BETA_MODEL_NAME
valueFrom:
secretKeyRef:
name: llm-closed-beta
key: model_name
- name: GOOGLE_AI_API_KEY
valueFrom:
secretKeyRef:
name: google-ai
key: api_key
- name: BLOB_STORE_ACCESS_KEY
valueFrom:
secretKeyRef:
name: blob-store
key: access_key
- name: BLOB_STORE_SECRET_KEY
valueFrom:
secretKeyRef:
name: blob-store
key: secret_key
- name: BLOB_STORE_URL
valueFrom:
secretKeyRef:
name: blob-store
key: url
- name: BLOB_STORE_REGION
valueFrom:
secretKeyRef:
name: blob-store
key: region
- name: BLOB_STORE_BUCKET
valueFrom:
secretKeyRef:
name: blob-store
key: bucket
- name: KINESIS_ACCESS_KEY
valueFrom:
secretKeyRef:
name: kinesis
key: access_key
- name: KINESIS_SECRET_KEY
valueFrom:
secretKeyRef:
name: kinesis
key: secret_key
- name: KINESIS_STREAM
valueFrom:
secretKeyRef:
name: kinesis
key: stream
- name: KINESIS_REGION
valueFrom:
secretKeyRef:
name: kinesis
key: region
- name: BLOB_STORE_BUCKET
valueFrom:
secretKeyRef:
name: blob-store
key: bucket
- name: CLICKHOUSE_URL
valueFrom:
secretKeyRef:
name: clickhouse
key: url
- name: CLICKHOUSE_USER
valueFrom:
secretKeyRef:
name: clickhouse
key: user
- name: CLICKHOUSE_PASSWORD
valueFrom:
secretKeyRef:
name: clickhouse
key: password
- name: CLICKHOUSE_DATABASE
valueFrom:
secretKeyRef:
name: clickhouse
key: database
- name: SLACK_PANICS_WEBHOOK
valueFrom:
secretKeyRef:
name: slack
key: panics_webhook
- name: STRIPE_API_KEY
valueFrom:
secretKeyRef:
name: stripe
key: api_key
optional: true
- name: COMPLETE_WITH_LANGUAGE_MODEL_RATE_LIMIT_PER_HOUR
value: "1000"
- name: SUPERMAVEN_ADMIN_API_KEY
valueFrom:
secretKeyRef:
name: supermaven
key: api_key
- name: USER_BACKFILLER_GITHUB_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: user-backfiller
key: github_access_token
optional: true
- name: INVITE_LINK_PREFIX
value: ${INVITE_LINK_PREFIX}
- name: RUST_BACKTRACE
value: "1"
- name: RUST_LOG
value: ${RUST_LOG}
- name: LOG_JSON
value: "true"
- name: ZED_ENVIRONMENT
value: ${ZED_ENVIRONMENT}
- name: AUTO_JOIN_CHANNEL_ID
value: "${AUTO_JOIN_CHANNEL_ID}"
securityContext:
capabilities:
# FIXME - Switch to the more restrictive `PERFMON` capability.
# This capability isn't yet available in a stable version of Debian.
add: ["SYS_ADMIN"]
terminationGracePeriodSeconds: 10