Parsing GitLab logs with `jq`

DETAILS: Tier: Free, Premium, Ultimate Offering: GitLab Self-Managed

We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, but if they are not available you can still quickly parse GitLab logs in JSON format using jq.

NOTE: Specifically for summarizing error events and basic usage statistics, the GitLab Support Team provides the specialised fast-stats tool.

What is JQ?

As noted in its manual, jq is a command-line JSON processor. The following examples include use cases targeted for parsing GitLab log files.

Parsing Logs

The examples listed below address their respective log files by their relative Linux package installation paths and default filenames. Find the respective full paths in the GitLab logs sections.

Compressed logs

When log files are rotated, they are renamed in Unix timestamp format and compressed with gzip. The resulting file name looks like @40000000624492fa18da6f34.s. These files must be handled differently before parsing, than the more recent log files:

To uncompress the file, use gunzip -S .s @40000000624492fa18da6f34.s, replacing the filename with your compressed log file's name.
To read or pipe the file directly, use zcat or zless.
To search file contents, use zgrep.

General Commands

Pipe colorized `jq` output into `less`

jq . <FILE> -C | less -R

Search for a term and pretty-print all matching lines

grep <TERM> <FILE> | jq .

Skip invalid lines of JSON

jq -cR 'fromjson?' file.json | jq <COMMAND>

By default jq errors out when it encounters a line that is not valid JSON. This skips over all invalid lines and parses the rest.

Print a JSON log's time range

cat log.json | (head -1; tail -1) | jq '.time'

Use zcat if the file has been rotated and compressed:

zcat @400000006026b71d1a7af804.s | (head -1; tail -1) | jq '.time'

zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time'

Get activity for correlation ID across multiple JSON logs in chronological order

grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)'  | less -R

Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log`

Find all requests with a 5XX status code

jq 'select(.status >= 500)' <FILE>

Top 10 slowest requests

jq -s 'sort_by(-.duration_s) | limit(10; .[])' <FILE>

Find and pretty print all requests related to a project

grep <PROJECT_NAME> <FILE> | jq .

Find all requests with a total duration > 5 seconds

jq 'select(.duration_s > 5000)' <FILE>

Find all project requests with more than 5 Gitaly calls

grep <TERM> <FILE> | jq .
```0

#### Find all requests with a Gitaly duration > 10 seconds

```shell
grep <TERM> <FILE> | jq .
```1

#### Find all requests with a queue duration > 10 seconds

```shell
grep <TERM> <FILE> | jq .
```2

#### Top 10 requests by # of Gitaly calls

```shell
grep <TERM> <FILE> | jq .
```3

#### Output a specific time range

```shell
grep <TERM> <FILE> | jq .
```4

### Parsing `gitlab-rails/production_json.log`

#### Print the top three controller methods by request volume and their three longest durations

```shell
grep <TERM> <FILE> | jq .
```5

**Example output**

```shell
grep <TERM> <FILE> | jq .
```6

### Parsing `gitlab-rails/api_json.log`

#### Print top three routes with request count and their three longest durations

```shell
grep <TERM> <FILE> | jq .
```7

**Example output**

```shell
grep <TERM> <FILE> | jq .
```8

#### Print top API user agents

```shell
grep <TERM> <FILE> | jq .
```9

**Example output**:

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```0

This example shows a custom tool or script causing an unexpectedly high number of requests.
User agents in this situation can be specialized [third-party clients](../../api/rest/third_party_clients.md),
or general tools like `curl`.

You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics for those users or bots.

### Parsing `gitlab-rails/importer.log`

To troubleshoot [project imports](../../administration/raketasks/project_import_export.md) or
[migrations](../../user/project/import/index.md), run this command:

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```1

For common issues, see [troubleshooting](../../administration/raketasks/import_export_rake_tasks_troubleshooting.md).

### Parsing `gitlab-workhorse/current`

#### Print top Workhorse user agents

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```2

Similar to the [API `ua` example](#print-top-api-user-agents),
many unexpected user agents in this output indicate unoptimized scripts.
Expected user agents include `gitlab-runner`, `GitLab-Shell`, and browsers.

The performance impact of runners checking for new jobs can be reduced by increasing
[the `check_interval` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section),
for example.

### Parsing `gitlab-rails/geo.log`

#### Find most common Geo sync errors

If [the `geo:status` Rake task](../geo/replication/troubleshooting/common.md#sync-status-rake-task)
repeatedly reports that some items never reach 100%,
the following command helps to focus on the most common errors.

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```3

Refer to our [Geo troubleshooting page](../geo/replication/troubleshooting/index.md)
for advice about specific error messages.

### Parsing `gitaly/current`

Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md).

#### Find all Gitaly requests sent from web UI

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```4

#### Find all failed Gitaly requests

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```5

#### Find all requests that took longer than 30 seconds

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```6

#### Print top ten projects by request volume and their three longest durations

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```7

**Example output**

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```8

#### Types of user and project activity overview

```shell
jq -cR 'fromjson?' file.json | jq <COMMAND>
```9

#### Find all projects affected by a fatal Git problem

```shell
cat log.json | (head -1; tail -1) | jq '.time'
```0

### Parsing `gitlab-shell/gitlab-shell.log`

For investigating Git calls through SSH.

Find the top 20 calls by project and user:

```shell
cat log.json | (head -1; tail -1) | jq '.time'
```1

Find the top 20 calls by project, user, and command:

```shell
cat log.json | (head -1; tail -1) | jq '.time'
```2

Parsing GitLab logs with jq