skills/catalyst-by-zoho/references/devops-deep-dive.md

# DevOps Deep-Dive Reference

## When to use this file
Load this file when the user asks about: Application Alerts configuration, APM traces, Catalyst
logs (access/application), log levels, pushing logs from code, metrics, GitHub integration,
automation testing, test cases, test suites, test plans, test variables, or DevOps limitations.

External docs: https://docs.catalyst.zoho.com/en/devops/

---

## Application Alerts

Automatic email notifications when Catalyst components encounter failures, exceptions, or
timeouts.

### Supported Components and Event Conditions

**Cron (Job Scheduling):**
- Job execution failure
- Job timeout
- Code exception

**Event Listener (Signals targets):**
- Event delivery failure
- Target invocation timeout
- Code exception in event function

**Logs (Log-based alerts):**
- Trigger alerts based on log patterns or error counts
- Configure criteria to match specific log messages or error levels
- Useful for detecting application-specific failure patterns

### Alert Criteria
- **Component**: Select which component to monitor
- **Condition**: The event that triggers the alert (failure, exception, timeout, log match)
- **Frequency**: How often alerts are sent — per occurrence, or batched (e.g., hourly digest)
- **Recipients**: Email addresses to notify (max 10 recipients per alert)

### Limits

| Environment | Max Alerts |
|-------------|-----------|
| Development | 5 |
| Production | 20 |

- Max **10 recipients** per alert rule
- Alerts are environment-specific (dev alerts don't fire in prod and vice versa)

---

## Application Performance Monitoring (APM)

In-depth performance analytics for function executions.

### Supported Function Types and Languages

| Function Type | Java | Node.js | Python |
|---------------|------|---------|--------|
| Basic I/O | Yes | Yes | **No** |
| Advanced I/O | Yes | Yes | **No** |
| Event Function | Yes | Yes | **No** |
| Cron Function | Yes | Yes | **No** |
| Browser Logic | Yes | Yes | **No** |
| Job Function | Yes | Yes | **No** |

**Important: APM is NOT available for Python functions.** Python functions are monitored via
Logs only. APM traces and performance breakdowns are limited to Java and Node.js.

### Features

**Invocations:**
- Total invocation count over selected time range
- Success vs. failure breakdown
- Invocations per function

**Response Times:**
- Average, P50, P95, P99 response times
- Response time trend over time
- Breakdown by function

**Top 100 Slowest Executions:**
- List of the 100 slowest function executions in the selected time range
- Each entry shows: function name, execution time, timestamp, status
- Click into any execution for full trace details

**Traces:**
- Detailed execution trace for individual function invocations
- Shows time spent in each phase: initialization, execution, SDK calls, external calls
- Breakdown of internal Catalyst SDK operations (DataStore queries, Cache lookups, etc.)

### SDK vs API Call Tracking

APM automatically tracks Catalyst SDK calls made within your function (DataStore, Cache,
FileStore, etc.). However, external API calls (HTTP requests to third-party services) are
**not automatically traced** — they appear as part of the total execution time but are not
broken down individually. To get visibility into external call performance, add manual
logging with timestamps.

### Data Center Availability
APM is available in all Catalyst data centers. However, trace data retention and granularity
may vary. Check the Catalyst status page for your data center's current APM capabilities.

---

## Logs

Two categories of logs in Catalyst:

### Access Logs
Server-level logs for HTTP requests to your Catalyst project.

**Fields:**
- Request timestamp
- HTTP method and URL path
- Response status code
- Response time (ms)
- Client IP address
- User agent
- Request ID

### Application Logs
Logs generated by your code via console.log/System.out/print statements.

**Fields:**
- Timestamp
- Log level (mapped from your code — see mapping below)
- Function name
- Message content
- Request ID (correlate with access logs)
- Execution ID

### Pushing Logs from Code

**Java — Basic I/O Function:**
```java
import java.util.logging.Logger;
import java.util.logging.Level;

public class MyFunction implements ZCFunction {
    private static final Logger LOGGER = Logger.getLogger(MyFunction.class.getName());

    public void runner(CatalystApp catalystApp, Context context, BasicIO basicIO) {
        LOGGER.info("Processing request");
        LOGGER.warning("Potential issue detected");
        LOGGER.severe("Critical error occurred");

        // For structured logging
        LOGGER.info("{\"action\":\"createUser\",\"status\":\"success\",\"userId\":\"12345\"}");

        context.close();
    }
}
```

**Java — Advanced I/O / Other Types:**
```java
import java.util.logging.Logger;
import java.util.logging.Level;

public class MyAdvancedFunction implements ZCFunction {
    private static final Logger LOGGER = Logger.getLogger(MyAdvancedFunction.class.getName());

    public void runner(CatalystApp catalystApp, Context context, HttpRequest request, HttpResponse response) {
        LOGGER.info("Handling " + request.getMethod() + " " + request.getRequestURI());
        LOGGER.fine("Debug details: " + request.getParameterMap());
        LOGGER.severe("Error: " + exception.getMessage());
    }
}
```

**Node.js — Basic I/O Function:**
```javascript
const catalyst = require('zcatalyst-sdk-node');
module.exports = async (context, basicIO) => {
  const catalystApp = catalyst.initialize(context);
  console.log('Processing request');           // INFO
  console.warn('Potential issue detected');     // WARNING
  console.error('Critical error occurred');     // ERROR
  console.debug('Debug details');              // DEBUG (may not appear in prod)

  // Structured logging (recommended)
  console.log(JSON.stringify({
    action: 'createUser',
    status: 'success',
    userId: '12345',
    requestId: context.getRequestId()
  }));

  context.close();
};
```

**Node.js — Advanced I/O / Other Types:**
```javascript
module.exports = async (catalystApp, context, req, res) => {
  console.log(`Handling ${req.method} ${req.url}`);
  console.error(`Error: ${error.message}`);

  // Same console methods work in all function types
};
```

**Python — Basic I/O Function:**
```python
import logging

logger = logging.getLogger(__name__)

def handler(catalyst_app, context, basic_io):
    logger.info("Processing request")
    logger.warning("Potential issue detected")
    logger.error("Critical error occurred")
    logger.debug("Debug details")

    # Structured logging
    import json
    logger.info(json.dumps({
        "action": "createUser",
        "status": "success",
        "userId": "12345"
    }))

    context.close()
```

**Python — Advanced I/O / Other Types:**
```python
import logging

logger = logging.getLogger(__name__)

def handler(catalyst_app, context, request, response):
    logger.info(f"Handling {request.method} {request.path}")
    logger.error(f"Error: {str(exception)}")
```

### Log Levels Mapping

| Code Statement | Catalyst Log Level |
|---------------|-------------------|
| Java `LOGGER.info()` | INFO |
| Java `LOGGER.warning()` | WARNING |
| Java `LOGGER.severe()` | ERROR |
| Java `LOGGER.fine()` | DEBUG |
| Node.js `console.log()` | INFO |
| Node.js `console.warn()` | WARNING |
| Node.js `console.error()` | ERROR |
| Node.js `console.debug()` | DEBUG |
| Python `logger.info()` | INFO |
| Python `logger.warning()` | WARNING |
| Python `logger.error()` | ERROR |
| Python `logger.debug()` | DEBUG |

### Filtering Logs
- Filter by: function name, log level, time range, status code, request ID
- Full-text search across log messages
- Auto-filter when navigating from a specific function or Circuit execution

### Log Retention

| Environment | Retention Period |
|-------------|-----------------|
| Development | 7 days |
| Production | 14 days |

### Log Message Limit
- Max **1,500 characters** per log message
- Messages exceeding this limit are truncated

---

## Metrics

Aggregated performance and usage metrics for key Catalyst components.

### Monitored Components

| Component | Metrics Tracked |
|-----------|----------------|
| **DataStore** | Query count, query latency, row operations, storage usage |
| **Cache** | Hit rate, miss rate, operations count, memory usage |
| **Cron** | Execution count, success/failure rate, average duration |
| **Files (FileStore)** | Upload/download count, storage usage, bandwidth |
| **API** | Request count, response times, error rates, status code distribution |

Metrics are available in the console under DevOps → Metrics, with configurable time ranges
and component filters.

---

## GitHub Integration

Connect GitHub repositories to your Catalyst project for streamlined deployment workflows.

### Features
- Link GitHub repositories to Catalyst functions
- Auto-deploy functions on push to configured branches
- Branch-based environment mapping (e.g., `main` → production, `develop` → development)
- Commit-level deployment tracking

**Note:** GitHub Integration is the predecessor to **Pipelines**, which provides a more
complete CI/CD solution with multi-stage workflows, testing, and artifact management. For
new projects, consider using Pipelines instead.

---

## Automation Testing

Built-in API testing framework for validating Catalyst function endpoints.

### Hierarchy

```
Modules
  └── Test Cases
        └── Test Suites
              └── Test Plans
```

- **Modules**: Logical groupings of test cases (e.g., "User Management", "Orders")
- **Test Cases**: Individual API endpoint tests
- **Test Suites**: Collections of test cases executed together
- **Test Plans**: Scheduled or on-demand execution of test suites

### Test Cases

Each test case defines an HTTP request and its expected response.

**Supported HTTP methods:** GET, POST, PUT, PATCH, DELETE

**Assertions (4 types):**

| Assertion Type | Description | Example |
|----------------|-------------|---------|
| **Status Code** | Assert the HTTP response status code | Status code equals 200 |
| **Body** | Assert values in the response body (JSON path) | `$.user.name` equals "John" |
| **Headers** | Assert response header values | `Content-Type` contains "application/json" |
| **Response Time** | Assert the response completes within a time limit | Response time less than 500ms |

### Test Suites

Group test cases for batch execution.

**Execution modes:**
- **Sequential**: Test cases run one after another in defined order. A failure can optionally
  stop the remaining tests.
- **Parallel**: Test cases run simultaneously for faster execution. Order is not guaranteed.

**Performance reporting:**
- Total execution time
- Pass/fail count and percentage
- Individual test case results with response details
- Response time distribution across the suite

### Test Plans

Schedule test suite execution.

**Scheduling options:**

| Schedule Type | Description |
|---------------|-------------|
| **Nightly** | Runs every night at a configured time |
| **Day** | Runs on specific days of the week at a configured time |
| **Custom** | Custom cron expression for flexible scheduling |
| **Starts Now** | Execute immediately (one-time, on-demand) |

**Notifications:**
- Email notifications on test plan completion
- Configurable recipients
- Summary includes: pass/fail counts, duration, link to detailed results

### Variables

Three scopes for test variables, each with different syntax and visibility.

| Scope | Syntax | Description |
|-------|--------|-------------|
| **Global** | `{{$var_name}}` | Available across all test cases and suites. Set in project settings. |
| **Environment** | Environment-specific | Different values per environment (dev/prod). Same variable name, different values. |
| **Local / Request** | `{{var_name}}` | Scoped to a single test case or extracted from a previous response in a suite. |

**Bulk add:** Import multiple variables at once via CSV or JSON in the console.

Variables are used in request URLs, headers, body, and assertions. For example:
```
GET {{$base_url}}/api/users/{{user_id}}
Authorization: Bearer {{$auth_token}}
```

### Results

**All Runs View:**
- History of all test plan executions
- Filter by date range, status, test plan name
- Summary stats per run

**Individual Run Details:**
- Detailed results for each test case in the run
- Request sent, response received, assertion results
- Pass/fail status per assertion
- Response time per test case

**Re-run Failed:**
- Re-execute only the failed test cases from a previous run
- Useful for validating fixes without re-running the entire suite

---

## Limitations

### Data Center Availability
- Automation Testing is available in all Catalyst data centers, but some advanced features
  may have limited availability. Check documentation for your data center.

### Environment Restrictions
- Automation Testing is available in the **development environment only**
- Tests cannot be run against production endpoints from the testing framework
- Use test plans with appropriate environment variables for staging validation

### Supported Endpoints
- Only Catalyst function endpoints (Basic I/O, Advanced I/O, AppSail) are supported as
  test targets
- External URLs are not supported as test case targets
- WebSocket endpoints are not supported