跳到主要内容

Architecture

Overview​

Truxt Components

graph TB subgraph "Truxt Platform" UI[Config UI<br/>Web Interface] API[API Server<br/>Main Interface] Runner[Runner<br/>Task Execution] DB[(Database<br/>MySQL/PostgreSQL)] subgraph "Plugins" P1[GitHub Plugin] P2[GitLab Plugin] P3[Jira Plugin] P4[Jenkins Plugin] P5[Other Plugins...] end subgraph "Data Sources" DS1[GitHub] DS2[GitLab] DS3[Jira] DS4[Jenkins] DS5[Other Tools...] end subgraph "Dashboards" Grafana[Grafana<br/>Visualization] Custom[Custom BI Tools] end end UI --> API API --> Runner Runner --> DB API --> DB P1 --> DS1 P2 --> DS2 P3 --> DS3 P4 --> DS4 P5 --> DS5 Runner --> P1 Runner --> P2 Runner --> P3 Runner --> P4 Runner --> P5 DB --> Grafana DB --> Custom style UI fill:#e1f5fe style API fill:#f3e5f5 style Runner fill:#fff3e0 style DB fill:#e8f5e8 style Grafana fill:#fce4ec

A Truxt installation typically consists of the following components:

  • Config UI: A handy user interface to create, trigger, and debug Blueprints. A Blueprint specifies the where (data connection), what (data scope), how (transformation rule), and when (sync frequency) of a data pipeline.
  • API Server: The main programmatic interface of Truxt.
  • Runner: The runner does all the heavy-lifting for executing tasks. In the default Truxt installation, it runs within the API Server, but Truxt provides a temporal-based runner (beta) for production environments.
  • Database: The database stores both Truxt's metadata and user data collected by data pipelines. Truxt supports MySQL and PostgreSQL as of v0.11.
  • Plugins: Plugins enable Truxt to collect and analyze dev data from any DevOps tools with an accessible API. Truxt community is actively adding plugins for popular DevOps tools, but if your preferred tool is not covered yet, feel free to open a GitHub issue to let us know or check out our doc on how to build a new plugin by yourself.
  • Dashboards: Dashboards deliver data and insights to Truxt users. A dashboard is simply a collection of SQL queries along with corresponding visualization configurations. Truxt's official dashboard tool is Grafana and pre-built dashboards are shipped in Grafana's JSON format. Users are welcome to swap for their own choice of dashboard/BI tool if desired.

Dataflow​

graph LR subgraph "Data Sources" DS[DevOps Tools<br/>APIs] end subgraph "Raw Layer" Raw[(Raw Data<br/>JSON Format)] end subgraph "Tool Layer" Tool[(Tool-Specific<br/>Relational Schema)] end subgraph "Domain Layer" Domain[(Domain Model<br/>Unified Schema)] end subgraph "Analytics" Metrics[Metrics & KPIs] Dashboards[Dashboards] Reports[Reports] end DS -->|API Calls| Raw Raw -->|Extract & Transform| Tool Tool -->|Normalize & Unify| Domain Domain --> Metrics Domain --> Dashboards Domain --> Reports style Raw fill:#ffebee style Tool fill:#e3f2fd style Domain fill:#e8f5e8 style Metrics fill:#fff3e0

A typical plugin's dataflow is illustrated below:

  1. The Raw layer stores the API responses from data sources (DevOps tools) in JSON. This saves developers' time if the raw data is to be transformed differently later on. Please note that communicating with data sources' APIs is usually the most time-consuming step.
  2. The Tool layer extracts raw data from JSONs into a relational schema that's easier to consume by analytical tasks. Each DevOps tool would have a schema that's tailored to its data structure, hence the name, the Tool layer.
  3. The Domain layer attempts to build a layer of abstraction on top of the Tool layer so that analytics logics can be re-used across different tools. For example, GitHub's Pull Request (PR) and GitLab's Merge Request (MR) are similar entities. They each have their own table name and schema in the Tool layer, but they're consolidated into a single entity in the Domain layer, so that developers only need to implement metrics like Cycle Time and Code Review Rounds once against the domain layer schema.

Principles​

  1. Extensible: Truxt's plugin system allows users to integrate with any DevOps tool. Truxt also provides a dbt plugin that enables users to define their own data transformation and analysis workflows.
  2. Portable: Truxt has a modular design and provides multiple options for each module. Users of different setups can freely choose the right configuration for themselves.
  3. Robust: Truxt provides an SDK to help plugins efficiently and reliably collect data from data sources while respecting their API rate limits and constraints.