JSON and YAML Data Manipulation with jq and yq

Overview

This lab focuses on mastering jq and yq, powerful tools for parsing and manipulating JSON and YAML data from files or APIs. You'll learn practical techniques such as filtering nested keys, iterating over arrays, extracting data from APIs, automating the parsing of configuration files, updating values, and converting between JSON and YAML formats—a critical skill set for scripting and DevOps tasks like Kubernetes management.

Inside this lab

You will explore the fundamental and advanced functionalities of jq and yq, apply them to real-world use cases, and automate repetitive tasks working with JSON and YAML files. The lab also highlights scenarios such as handling API responses with curl, managing Docker compose files, and extracting data from Kubernetes manifests. By the end of the lab, you will gain proficiency in working with structured data formats efficiently.

Key Skills

• Parsing JSON and YAML files
• Extracting nested data from JSON
• Working with arrays in JSON
• Combining curl with jq for API data processing
• Automating configuration file manipulation using shell scripts and jq
• Reading, extracting, modifying, and converting YAML files with yq
• Converting between JSON and YAML formats

Who this lab is for

This lab is ideal for programmers, DevOps engineers, and system administrators who need to work with JSON and YAML data in automation workflows, scripting, or managing infrastructure configurations.

Contents:

1. Introduction to jq - Basic Parsing and Filtering:

   • Learn the foundational features of jq, including parsing JSON files and extracting specific key values from nested and top-level structures.

2. Advanced jq Usage - Working with Arrays:

   • Master the ability to iterate over arrays and retrieve specific elements in JSON files using jq. Gain insights into practical use cases, such as processing lists of data.

3. Combining jq with curl to Parse API Responses:

   • Integrate jq with curl to fetch data from APIs, parse JSON responses, and extract specific fields for processing or analysis. Ideal for handling dynamic data from web services like GitHub APIs.

4. Automating Tasks with jq in Shell Scripts:

   • Automate JSON parsing and data extraction using jq in shell scripts. Learn to read configuration files, extract database credentials, and streamline workflows efficiently.

5. Introduction to yq - Basic Syntax and Reading YAML Files:

   • Explore the syntax and capabilities of yq for working with YAML files. Extract specific fields like service names or environment variables from YAML structures.

6. Modifying YAML Files with yq:

   • Understand how to modify YAML files in your terminal using yq. Perform in-place edits to tags, such as updating Docker images in a Kubernetes or Docker Compose YAML configuration file.

7. Format Conversion with yq - YAML to JSON and JSON to YAML:

   • Learn how to convert YAML to JSON and vice versa using yq. Ideal for workflows requiring compatibility across data formats, such as Kubernetes manifests and API interactions.

Technologies Covered

• jq: A lightweight command-line JSON processor.
• yq: A similar tool to jq, but for YAML manipulation.
• Shell Scripting: Automating tasks and workflows using scripts.

Why Learn jq and yq

These tools are essential for developers and IT professionals who work with structured data formats. Skills taught in this lab are directly applicable to tasks like API response parsing, configuration management, container orchestration, infrastructure-as-code, and CI/CD pipelines.

By the end of the lab, you'll have hands-on experience in automating workflows and managing structured data files using industry-standard tools.