mirror of
https://github.com/Fechin/reference.git
synced 2026-11-20 21:23:11 +03:00
Merge remote-tracking branch 'origin/main'
This commit is contained in:
commit
414d56468e
10 files changed
+1539
-381
No files matched your search
@@ -0,0 +1,464 @@
|
||||
---
|
||||
title: Antigravity CLI (agy)
|
||||
date: 2026-09-22 12:00:00
|
||||
background: bg-[#1a73e8]
|
||||
tags:
|
||||
- agy
|
||||
- antigravity
|
||||
- ai
|
||||
- cli
|
||||
- google
|
||||
- coding
|
||||
- agent
|
||||
categories:
|
||||
- AI
|
||||
intro: |
|
||||
[Antigravity CLI (agy)](https://antigravity.google/docs/cli) is Google's AI-first terminal development assistant and agentic coding CLI. It reads codebases, executes edits, manages background tasks, orchestrates subagents, and automates developer workflows.
|
||||
plugins:
|
||||
- copyCode
|
||||
---
|
||||
|
||||
## Getting Started {.cols-3}
|
||||
|
||||
### Quick Start {.row-span-2}
|
||||
|
||||
```bash
|
||||
# Launch interactive session in current repository
|
||||
$ agy
|
||||
|
||||
# Resume most recent session
|
||||
$ agy -c
|
||||
$ agy --continue
|
||||
|
||||
# Resume a specific session by conversation ID
|
||||
$ agy --conversation <conversation-id>
|
||||
|
||||
# Run non-interactive prompt and print output
|
||||
$ agy -p "explain the architecture of this repo"
|
||||
$ agy --print "fix lint errors across src/"
|
||||
|
||||
# Run prompt and continue interactively
|
||||
$ agy -i "refactor auth middleware"
|
||||
$ agy --prompt-interactive "add unit tests"
|
||||
|
||||
# Pipe stdin directly to agy
|
||||
$ git diff | agy -p "write a git commit message"
|
||||
$ cat logs.txt | agy -p "identify crash trace"
|
||||
|
||||
# Multi-directory workspace
|
||||
$ agy --add-dir ../backend --add-dir ../frontend
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
| Method | Description |
|
||||
| --- | --- |
|
||||
| **Google Account** | Run `agy`, follow browser login prompts on first run |
|
||||
| **Vertex AI (ADC)** | `gcloud auth application-default login` |
|
||||
| **Service Account** | Set `GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa.json` |
|
||||
| **Project Selection** | `agy --project <project-id>` or `GOOGLE_CLOUD_PROJECT` |
|
||||
|
||||
### Installation Methods
|
||||
|
||||
```bash
|
||||
# Verify installation
|
||||
$ agy --help
|
||||
|
||||
# Configure environment PATH and shell settings
|
||||
$ agy install
|
||||
|
||||
# Update CLI to latest release
|
||||
$ agy update
|
||||
|
||||
# View release notes & changelog
|
||||
$ agy changelog
|
||||
```
|
||||
|
||||
### Execution Modes
|
||||
|
||||
| Mode | Command Flag | Description |
|
||||
| --- | --- | --- |
|
||||
| **Default** | *(default)* | Prompts for confirmation before modifying files or running commands |
|
||||
| **Accept Edits** | `--mode accept-edits` | Automatically applies edits and writes files without prompting |
|
||||
| **Plan Mode** | `--mode plan` | Formulates step-by-step implementation plans before touching code |
|
||||
| **Sandbox** | `--sandbox` | Restricts shell execution inside an isolated terminal sandbox |
|
||||
| **Unrestricted** | `--dangerously-skip-permissions` | Auto-approves all tool permission prompts |
|
||||
|
||||
### Key Concepts
|
||||
|
||||
- **Agentic Loop**: Dynamically inspects files, runs shell commands, views diffs, and self-corrects based on tool results.
|
||||
- **Progressive Disclosure**: Skills and contextual rules are loaded on demand rather than cluttering token context.
|
||||
- **Asynchronous Tasks**: Background commands and multi-agent runs execute concurrently without blocking turns.
|
||||
|
||||
---
|
||||
|
||||
## CLI Options {.cols-2}
|
||||
|
||||
### Core Flags {.row-span-2}
|
||||
|
||||
| Flag | Short | Description |
|
||||
| --- | --- | --- |
|
||||
| `--print <prompt>` | `-p` | Run prompt non-interactively and print response |
|
||||
| `--prompt-interactive <prompt>` | `-i` | Run initial prompt and remain interactive |
|
||||
| `--continue` | `-c` | Resume the most recent conversation |
|
||||
| `--conversation <id>` | | Resume specific conversation by ID |
|
||||
| `--model <model>` | | Specify model for the session |
|
||||
| `--effort <level>` | | Set reasoning effort (`low`, `medium`, `high`) |
|
||||
| `--mode <mode>` | | Agent mode (`accept-edits`, `plan`) |
|
||||
| `--sandbox` | | Enable terminal sandbox restrictions |
|
||||
| `--dangerously-skip-permissions` | | Auto-approve all tool permissions |
|
||||
| `--add-dir <path>` | | Add directory to workspace (repeatable) |
|
||||
| `--project <id>` | | Google Cloud / Vertex project ID |
|
||||
| `--new-project` | | Create a new project for this session |
|
||||
| `--output-format <format>` | | Output format (`text`, `json`, `stream-json`) |
|
||||
| `--input-format <format>` | | Input format (`text`, `stream-json`) |
|
||||
| `--json-schema <schema>` | | Enforce structured JSON schema for outputs |
|
||||
| `--print-timeout <duration>` | | Timeout limit for print mode (default `0s`) |
|
||||
| `--log-file <path>` | | Override CLI log file path |
|
||||
| `--remote-control` | | Connect to remote control background daemon |
|
||||
| `--disable-slash-commands` | | Disable slash command expansion in print mode |
|
||||
|
||||
### Subcommands
|
||||
|
||||
```bash
|
||||
# Manage MCP servers
|
||||
$ agy mcp [list|add|remove|enable|disable]
|
||||
|
||||
# Manage plugins
|
||||
$ agy plugin [list|install|uninstall|enable|disable|import|validate]
|
||||
|
||||
# List supported AI models
|
||||
$ agy models
|
||||
|
||||
# List available agents
|
||||
$ agy agents
|
||||
|
||||
# Manage remote control background daemon
|
||||
$ agy remote-control [start|status|stop]
|
||||
|
||||
# Serve microphone input to a CLI on another host
|
||||
$ agy mic-serve
|
||||
|
||||
# Update agy CLI
|
||||
$ agy update
|
||||
```
|
||||
|
||||
### Headless & CI/CD Formats
|
||||
|
||||
```bash
|
||||
# Plain text output
|
||||
$ agy -p "check repo for vulnerabilities" --output-format text
|
||||
|
||||
# JSON output for tooling pipelines
|
||||
$ agy -p "extract exported API endpoints" --output-format json
|
||||
|
||||
# NDJSON streaming for real-time consumers
|
||||
$ agy -p "generate OpenAPI spec" --output-format stream-json
|
||||
|
||||
# Enforce structured output via JSON Schema
|
||||
$ agy -p "list todos" --json-schema schema.json --output-format json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Model Selection & Reasoning {.cols-2}
|
||||
|
||||
### Models & Reasoning Effort {.row-span-2}
|
||||
|
||||
```bash
|
||||
# Check all available models
|
||||
$ agy models
|
||||
|
||||
# Run with Gemini 3.8 Flash with high reasoning (fast & deep)
|
||||
$ agy --model gemini-3.8-flash-high
|
||||
|
||||
# Run with Gemini 3.1 Pro for complex architectures
|
||||
$ agy --model gemini-3.1-pro-high
|
||||
|
||||
# Run with Claude Sonnet 4.6 (Thinking)
|
||||
$ agy --model claude-sonnet-4-6
|
||||
|
||||
# Specify reasoning effort level
|
||||
$ agy --effort high
|
||||
$ agy --effort medium
|
||||
$ agy --effort low
|
||||
```
|
||||
|
||||
### Model Reference
|
||||
|
||||
| Model Identifier | Display Name | Recommended Use |
|
||||
| --- | --- | --- |
|
||||
| `gemini-3.8-flash-high` | Gemini 3.8 Flash (High) | Daily coding, rapid agent turns, default |
|
||||
| `gemini-3.8-flash-medium` | Gemini 3.8 Flash (Medium) | Balanced speed and thinking tokens |
|
||||
| `gemini-3.8-flash-low` | Gemini 3.8 Flash (Low) | Fastest latency, lightweight queries |
|
||||
| `gemini-3.7-flash-high` | Gemini 3.7 Flash (High) | Previous generation high reasoning |
|
||||
| `gemini-3.1-pro-high` | Gemini 3.1 Pro (High) | Massive refactors, cross-repo design |
|
||||
| `claude-sonnet-4-6` | Claude Sonnet 4.6 | Thinking-enabled coding assistant |
|
||||
| `claude-opus-4-6-thinking` | Claude Opus 4.6 | Deep complex multi-step reasoning |
|
||||
| `gpt-oss-120b-medium` | GPT-OSS 120B | Open-weights reasoning tier |
|
||||
|
||||
---
|
||||
|
||||
## Interactive Slash Commands {.cols-3}
|
||||
|
||||
### Slash Commands `/` {.row-span-3}
|
||||
|
||||
| Slash Command | Purpose |
|
||||
| --- | --- |
|
||||
| `/help` | Show available slash commands and keybindings |
|
||||
| `/plan` | Formulate a multi-step design & plan before execution |
|
||||
| `/goal` | Autonomous mode: runs until goal is completed |
|
||||
| `/boost` | Deep thinking, multi-angle exploration, rigorous checks |
|
||||
| `/browser` | Web automation, page inspection, and live testing |
|
||||
| `/grill-me` | Interactive interview to settle design decisions |
|
||||
| `/schedule` | Set a one-off timer or recurring cron schedule |
|
||||
| `/teamwork-preview` | Autonomous multi-agent collaboration |
|
||||
| `/learn` | Persist developer preferences and lessons |
|
||||
| `/exit` | Exit the CLI session |
|
||||
| `/quit` | Alias for `/exit` |
|
||||
|
||||
### Goal & Boost Modes
|
||||
|
||||
```bash
|
||||
# Inside interactive agy:
|
||||
> /goal Implement end-to-end tests for billing API and verify they pass
|
||||
|
||||
# Boost mode for mission-critical changes:
|
||||
> /boost Review security implications of session token rotation
|
||||
```
|
||||
|
||||
### Interactive Interviews
|
||||
|
||||
```bash
|
||||
# Challenge your assumptions before writing code
|
||||
> /grill-me I want to migrate our database from MongoDB to Postgres
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Keyboard Shortcuts {.cols-3}
|
||||
|
||||
### General Controls {.row-span-2}
|
||||
|
||||
| Keybinding | Action |
|
||||
| --- | --- |
|
||||
| `Ctrl+D Ctrl+D` | Exit agy CLI session |
|
||||
| `Ctrl+C` | Cancel currently running agent turn or streaming response |
|
||||
| `Enter` | Submit prompt / send message |
|
||||
| `Shift+Enter` | Insert a newline into multiline input |
|
||||
| `Alt+Enter` | Insert a newline into multiline input |
|
||||
|
||||
### Input Navigation
|
||||
|
||||
| Keybinding | Action |
|
||||
| --- | --- |
|
||||
| `Up` / `Down` | Browse prompt history |
|
||||
| `Ctrl+A` | Move cursor to beginning of line |
|
||||
| `Ctrl+E` | Move cursor to end of line |
|
||||
| `Ctrl+U` | Delete from cursor to start of line |
|
||||
| `Ctrl+K` | Delete from cursor to end of line |
|
||||
| `Ctrl+L` | Clear terminal screen |
|
||||
|
||||
### Agent Control
|
||||
|
||||
| Action | How |
|
||||
| --- | --- |
|
||||
| **Accept Edit** | Press `Enter` or click Approve on tool prompt |
|
||||
| **Reject Edit** | Press `Esc` or type feedback |
|
||||
| **Interrupt** | `Ctrl+C` halts tool call immediately |
|
||||
|
||||
---
|
||||
|
||||
## Subagents & Delegation {.cols-2}
|
||||
|
||||
### Subagent Architecture {.row-span-2}
|
||||
|
||||
Antigravity CLI can invoke specialized subagents that execute concurrently with their own contexts and workspaces:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Parent[agy CLI Main Agent] --> Sub1[research Subagent]
|
||||
Parent --> Sub2[self Subagent]
|
||||
Parent --> Sub3[Custom Subagents]
|
||||
Sub1 -->|Insights| Parent
|
||||
Sub2 -->|Code Diffs| Parent
|
||||
```
|
||||
|
||||
- **Isolated Context**: Subagents keep the main turn's context window clean and concise.
|
||||
- **Parallel Work**: Background subagents report results automatically without polling.
|
||||
- **Workspace Modes**:
|
||||
- `inherit`: Shares current project workspace.
|
||||
- `branch`: Creates an isolated git worktree branch.
|
||||
- `share`: Uses shared repository cache without copying storage.
|
||||
|
||||
### Built-in Subagent Types
|
||||
|
||||
| Subagent | Scope | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `research` | Read-only | Web search, codebase exploration, documentation analysis |
|
||||
| `self` | Full tools | Inherits parent tools to run parallel branch edits |
|
||||
| `custom` | Configurable | Defined dynamically via `define_subagent` |
|
||||
|
||||
---
|
||||
|
||||
## MCP Servers (Model Context Protocol) {.cols-2}
|
||||
|
||||
### Managing MCP Servers {.row-span-2}
|
||||
|
||||
Antigravity natively integrates MCP servers to connect your agent to external databases, APIs, and devtools:
|
||||
|
||||
```bash
|
||||
# List configured MCP servers
|
||||
$ agy mcp list
|
||||
|
||||
# Add local process (stdio) MCP server
|
||||
$ agy mcp add postgres npx -y @modelcontextprotocol/server-postgres "postgresql://localhost/db"
|
||||
|
||||
# Add HTTP / SSE MCP server
|
||||
$ agy mcp add weather https://api.weather-mcp.internal/sse
|
||||
|
||||
# Disable an MCP server temporarily
|
||||
$ agy mcp disable postgres
|
||||
|
||||
# Re-enable an MCP server
|
||||
$ agy mcp enable postgres
|
||||
|
||||
# Remove an MCP server
|
||||
$ agy mcp remove postgres
|
||||
```
|
||||
|
||||
### Configuration & Tool Loading
|
||||
|
||||
Servers are stored in `~/.gemini/antigravity-cli/mcp/<serverName>/`:
|
||||
|
||||
- **Eager Tools**: Automatically loaded into native agent tool registry.
|
||||
- **Lazy Tools**: Registered on-demand via schemas (`<toolName>.json`) to conserve prompt tokens.
|
||||
- **Instructions**: Optional `instructions.md` guide agent when and how to call tools.
|
||||
|
||||
---
|
||||
|
||||
## Customization: Rules & Skills {.cols-3}
|
||||
|
||||
### Hierarchical Rules {.row-span-2}
|
||||
|
||||
Rules guide coding standards and instructions automatically:
|
||||
|
||||
- **Project Root**: `GEMINI.md`, `AGENTS.md`
|
||||
- **Scoped Rules**: `.agents/rules/*.md` or `.gemini/rules/`
|
||||
- **Hierarchical Discovery**: Traverses from active file up to git root.
|
||||
|
||||
```markdown
|
||||
<!-- GEMINI.md example -->
|
||||
# Project Guidelines
|
||||
- Always use TypeScript strict mode.
|
||||
- Write tests with Vitest in tests/ directory.
|
||||
```
|
||||
|
||||
### Agent Skills
|
||||
|
||||
Skills teach reusable workflows and procedures:
|
||||
|
||||
- Located in `.agents/skills/<name>/SKILL.md` or `~/.gemini/config/skills/`.
|
||||
- Loaded dynamically via **Progressive Disclosure** (only activated when relevant).
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: deploy-preview
|
||||
description: Deploy branch to preview environment and run smoke tests
|
||||
---
|
||||
# Runbook steps here...
|
||||
```
|
||||
|
||||
### Precedence Order
|
||||
|
||||
1. Workspace Project (`.agents/`)
|
||||
2. Workspace Declared (`skills.json`, `plugins.json`)
|
||||
3. Global User Config (`~/.gemini/config/`)
|
||||
4. Built-in Applications & Skills
|
||||
5. Global Declared Config
|
||||
|
||||
---
|
||||
|
||||
## Plugins & Extensions {.cols-2}
|
||||
|
||||
### Managing Plugins {.row-span-2}
|
||||
|
||||
Plugins bundle skills, rules, hooks, and MCP servers into installable units:
|
||||
|
||||
```bash
|
||||
# List all installed plugins
|
||||
$ agy plugin list
|
||||
|
||||
# Install from registry or marketplace
|
||||
$ agy plugin install security-auditor@marketplace
|
||||
|
||||
# Install local plugin package
|
||||
$ agy plugin install ./my-plugin
|
||||
|
||||
# Import plugins from Claude or Gemini configs
|
||||
$ agy plugin import gemini
|
||||
$ agy plugin import claude
|
||||
|
||||
# Validate plugin manifest and schema
|
||||
$ agy plugin validate ./my-plugin
|
||||
|
||||
# Enable / Disable a plugin
|
||||
$ agy plugin enable security-auditor
|
||||
$ agy plugin disable security-auditor
|
||||
|
||||
# Uninstall plugin
|
||||
$ agy plugin uninstall security-auditor
|
||||
```
|
||||
|
||||
### Plugin Structure
|
||||
|
||||
```
|
||||
my-plugin/
|
||||
├── plugin.json # Manifest (name, version, entrypoints)
|
||||
├── skills/ # Custom agent skills
|
||||
│ └── audit/SKILL.md
|
||||
├── rules/ # Custom rules and guidelines
|
||||
│ └── security.md
|
||||
└── mcp_config.json # Bundled MCP server definitions
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration & Brain Storage {.cols-2}
|
||||
|
||||
### Configuration Locations {.row-span-2}
|
||||
|
||||
| Scope | Path | Purpose |
|
||||
| --- | --- | --- |
|
||||
| **CLI Settings** | `~/.gemini/antigravity-cli/settings.json` | Global CLI preferences, default model, reasoning |
|
||||
| **Global Config** | `~/.gemini/config/` | User-wide skills, rules, and plugins |
|
||||
| **Workspace Config** | `.agents/` or `.gemini/` | Project-scoped rules and agent configurations |
|
||||
| **Sessions & Brain** | `~/.gemini/antigravity-cli/brain/<session-id>/` | Transcripts, tasks, and generated artifacts |
|
||||
|
||||
### Example `settings.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "gemini-3.8-flash-high",
|
||||
"effort": "high",
|
||||
"mode": "accept-edits",
|
||||
"telemetry": false
|
||||
}
|
||||
```
|
||||
|
||||
### Transcripts & Artifacts
|
||||
|
||||
- Transcripts stored in `brain/<id>/.system_generated/logs/transcript.jsonl`.
|
||||
- Markdown artifacts and diff reports stored in `brain/<id>/`.
|
||||
|
||||
---
|
||||
|
||||
## Also see {.cols-1}
|
||||
|
||||
- [Official Antigravity Documentation](https://antigravity.google/docs) _(antigravity.google)_
|
||||
- [CLI Reference Guide](https://antigravity.google/docs/cli/reference) _(antigravity.google)_
|
||||
- [CLI Features & Subagents](https://antigravity.google/docs/cli/features) _(antigravity.google)_
|
||||
- [Skills Documentation](https://antigravity.google/docs/skills) _(antigravity.google)_
|
||||
- [Rules & Guidelines Guide](https://antigravity.google/docs/rules-workflows) _(antigravity.google)_
|
||||
- [Model Context Protocol (MCP)](https://antigravity.google/docs/mcp) _(antigravity.google)_
|
||||
- [Antigravity Python SDK](https://github.com/google-antigravity/antigravity-sdk-python) _(github.com)_
|
||||
@@ -50,9 +50,11 @@ NAME = "John" # => Error (about space)
|
||||
### Comments
|
||||
|
||||
```bash
|
||||
# This is an inline Bash comment.
|
||||
# This is a signle-line (inline) Bash comment.
|
||||
```
|
||||
|
||||
Single-line comments in Bash start with `#` and continue to the end of the line.
|
||||
|
||||
```bash
|
||||
: '
|
||||
This is a
|
||||
@@ -61,7 +63,9 @@ in bash
|
||||
'
|
||||
```
|
||||
|
||||
Multi-line comments use `:'` to open and `'` to close
|
||||
Multi-line comments use `:'` to open and `'` to close.
|
||||
|
||||
> **Note:** Bash does not have true multi-line comments. This works by passing a multi-line string to the no-op builtin `:` (colon), which ignores its argument.
|
||||
|
||||
### Arguments {.row-span-2}
|
||||
|
||||
|
||||
@@ -1220,13 +1220,11 @@ void main() {
|
||||
|
||||
```c{8}
|
||||
#include <stdio.h>
|
||||
#include <conio.h>
|
||||
|
||||
void main() {
|
||||
FILE *fp;
|
||||
char c;
|
||||
|
||||
clrscr();
|
||||
|
||||
fp = fopen("myfile.txt", "r");
|
||||
|
||||
@@ -1235,7 +1233,6 @@ void main() {
|
||||
}
|
||||
fclose(fp);
|
||||
|
||||
getch();
|
||||
}
|
||||
```
|
||||
|
||||
@@ -1243,18 +1240,15 @@ void main() {
|
||||
|
||||
```c {8}
|
||||
#include<stdio.h>
|
||||
#include<conio.h>
|
||||
|
||||
void main() {
|
||||
FILE *fp;
|
||||
|
||||
clrscr();
|
||||
|
||||
fp = fopen("myfile2.txt","w");
|
||||
fputs("hello c programming",fp);
|
||||
fclose(fp);
|
||||
|
||||
getch();
|
||||
}
|
||||
```
|
||||
|
||||
@@ -1262,19 +1256,16 @@ void main() {
|
||||
|
||||
```c {10}
|
||||
#include<stdio.h>
|
||||
#include<conio.h>
|
||||
|
||||
void main() {
|
||||
FILE *fp;
|
||||
char text[300];
|
||||
|
||||
clrscr();
|
||||
|
||||
fp = fopen("myfile2.txt", "r");
|
||||
printf("%s", fgets(text, 200, fp));
|
||||
fclose(fp);
|
||||
|
||||
getch();
|
||||
}
|
||||
```
|
||||
|
||||
@@ -1303,13 +1294,11 @@ Set the file pointer to the given position
|
||||
|
||||
```c{11}
|
||||
#include <stdio.h>
|
||||
#include <conio.h>
|
||||
|
||||
void main() {
|
||||
FILE *fp;
|
||||
char c;
|
||||
|
||||
clrscr();
|
||||
|
||||
fp = fopen("file.txt", "r");
|
||||
|
||||
@@ -1324,7 +1313,6 @@ void main() {
|
||||
}
|
||||
fclose(fp);
|
||||
|
||||
getch();
|
||||
}
|
||||
// output
|
||||
// Hello World! Hello World!
|
||||
@@ -1334,13 +1322,11 @@ void main() {
|
||||
|
||||
```c{11}
|
||||
#include <stdio.h>
|
||||
#include <conio.h>
|
||||
|
||||
void main () {
|
||||
FILE *fp;
|
||||
int length;
|
||||
|
||||
clrscr();
|
||||
|
||||
fp = fopen("file.txt", "r");
|
||||
|
||||
@@ -1350,7 +1336,6 @@ void main () {
|
||||
|
||||
printf("File size: %d bytes", length);
|
||||
|
||||
getch();
|
||||
}
|
||||
// output
|
||||
// file size: 18 bytes
|
||||
|
||||
@@ -908,14 +908,19 @@ Provide key details about French Revolution
|
||||
Explain history of Byzantine Empire
|
||||
```
|
||||
|
||||
Science Reseearch
|
||||
Science Research
|
||||
|
||||
```{.wrap}
|
||||
Explain quantum computing in simple terms
|
||||
```
|
||||
Verified University Papers
|
||||
|
||||
```{.wrap}
|
||||
Explain rocket science like i'm five
|
||||
Get me top university research papers with verified direct links
|
||||
|
||||
|
||||
```{.wrap}
|
||||
Explain rocket science like I'm five
|
||||
```
|
||||
|
||||
### SEO
|
||||
|
||||
@@ -153,13 +153,13 @@ $ git branch -av
|
||||
Switch to my_branch, and update working directory
|
||||
|
||||
```shell script
|
||||
$ git checkout my_branch
|
||||
$ git switch my_branch
|
||||
```
|
||||
|
||||
Create a new branch called new_branch
|
||||
|
||||
```shell script
|
||||
$ git checkout -b new_branch
|
||||
$ git switch -c new_branch
|
||||
```
|
||||
|
||||
Delete the branch called my_branch
|
||||
|
||||
+306
-175
@@ -1,38 +1,58 @@
|
||||
---
|
||||
title: 'Matplotlib'
|
||||
date: 06-06-2025
|
||||
title: Matplotlib
|
||||
date: 2026-07-19
|
||||
background: bg-[#1a073e]
|
||||
tags:
|
||||
- 'data visualization'
|
||||
- 'matplotlib'
|
||||
- 'plotting'
|
||||
- data visualization
|
||||
- matplotlib
|
||||
- plotting
|
||||
categories:
|
||||
- 'Python'
|
||||
- Python
|
||||
intro: |
|
||||
[Matplotlib](https://matplotlib.org/) is a comprehensive library for creating static, animated, and interactive plots in Python.
|
||||
This cheat sheet provides a quick reference from basic to advanced usage, covering essential features for data science, machine learning, and scientific computing.
|
||||
[Matplotlib](https://matplotlib.org/) is a comprehensive library for creating static, animated, and interactive plots in Python.
|
||||
This cheat sheet provides a production-grade reference for matplotlib ≥3.8 with emphasis on publication-quality figures, OOP interface, and performance optimization.
|
||||
plugins:
|
||||
- copyCode
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## Getting Started {.cols-2}
|
||||
|
||||
### Importing
|
||||
### Import Convention
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt # Core plotting library
|
||||
import numpy as np # For numerical operations
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
|
||||
# For high-quality figures
|
||||
plt.rcParams['figure.dpi'] = 120
|
||||
plt.rcParams['savefig.dpi'] = 300
|
||||
plt.rcParams['font.size'] = 10
|
||||
plt.rcParams['axes.labelsize'] = 12
|
||||
plt.rcParams['axes.titlesize'] = 14
|
||||
plt.rcParams['legend.fontsize'] = 10
|
||||
```
|
||||
|
||||
### Two Interfaces
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Pyplot (procedural) - quick | `plt.plot(x, y); plt.show()` |
|
||||
| OOP (explicit) - production | `fig, ax = plt.subplots(); ax.plot(x, y); fig.savefig('fig.pdf')` |
|
||||
| Create figure and axes | `fig, ax = plt.subplots(figsize=(8, 6))` |
|
||||
| Create figure with grid | `fig, axes = plt.subplots(2, 2, figsize=(10, 8))` |
|
||||
| Get current figure/axes | `fig = plt.gcf(); ax = plt.gca()` |
|
||||
|
||||
### Basic Plot
|
||||
|
||||
```python
|
||||
x = np.linspace(0, 10, 100) # 100 points between 0 and 10
|
||||
y = np.sin(x) # Sine function values
|
||||
plt.plot(x, y) # Create a line plot
|
||||
plt.show() # Display the plot
|
||||
x = np.linspace(0, 10, 100)
|
||||
y = np.sin(x)
|
||||
fig, ax = plt.subplots()
|
||||
ax.plot(x, y)
|
||||
ax.set_title("Sine Wave")
|
||||
ax.set_xlabel("x")
|
||||
ax.set_ylabel("sin(x)")
|
||||
plt.show()
|
||||
```
|
||||
|
||||
---
|
||||
@@ -41,226 +61,337 @@ plt.show() # Display the plot
|
||||
|
||||
### Line Plot
|
||||
|
||||
```python
|
||||
plt.plot(x, y) # Line plot of y vs x
|
||||
plt.title("Sine Wave") # Set title
|
||||
plt.xlabel("x-axis") # Label x-axis
|
||||
plt.ylabel("y-axis") # Label y-axis
|
||||
plt.grid(True) # Show gridlines
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Simple line plot | `ax.plot(x, y)` |
|
||||
| With custom style | `ax.plot(x, y, 'r--o', linewidth=2, markersize=4)` |
|
||||
| Multiple series | `ax.plot(x, y1, label='sin'); ax.plot(x, y2, label='cos'); ax.legend()` |
|
||||
| Step plot | `ax.step(x, y, where='mid')` |
|
||||
| Stacked lines | `ax.stackplot(x, y1, y2, labels=['A','B'])` |
|
||||
| Fill between | `ax.fill_between(x, y1, y2, alpha=0.3)` |
|
||||
|
||||
### Scatter Plot
|
||||
|
||||
```python
|
||||
plt.scatter(x, y) # Scatter plot
|
||||
plt.title("Scatter Plot")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Basic scatter | `ax.scatter(x, y)` |
|
||||
| Scatter with sizes | `ax.scatter(x, y, s=sizes, c=colors, alpha=0.6, cmap='viridis')` |
|
||||
| Scatter with categorical colors | `ax.scatter(x, y, c=labels, cmap='Set1', edgecolors='black')` |
|
||||
| 3D scatter | `ax.scatter3D(x, y, z, c=z, cmap='viridis')` |
|
||||
|
||||
### Bar Plot
|
||||
### Bar Charts
|
||||
|
||||
```python
|
||||
categories = ['A', 'B', 'C']
|
||||
values = [10, 20, 15]
|
||||
plt.bar(categories, values) # Vertical bar chart
|
||||
plt.title("Bar Chart")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Vertical bar | `ax.bar(categories, values, width=0.8, color='steelblue')` |
|
||||
| Horizontal bar | `ax.barh(categories, values, height=0.8)` |
|
||||
| Grouped bar | `ax.bar(x - width/2, y1, width, label='A'); ax.bar(x + width/2, y2, width, label='B')` |
|
||||
| Stacked bar | `ax.bar(x, y1, label='A'); ax.bar(x, y2, bottom=y1, label='B')` |
|
||||
| Error bars | `ax.bar(x, y, yerr=errors, capsize=5)` |
|
||||
|
||||
### Horizontal Bar Plot
|
||||
### Histograms
|
||||
|
||||
```python
|
||||
plt.barh(categories, values) # Horizontal bar chart
|
||||
plt.title("Horizontal Bar Chart")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Basic histogram | `ax.hist(data, bins=30, edgecolor='black')` |
|
||||
| Multiple histograms | `ax.hist([data1, data2], bins=30, label=['A','B'], alpha=0.6)` |
|
||||
| Density (normalized) | `ax.hist(data, bins=30, density=True, histtype='step')` |
|
||||
| Cumulative | `ax.hist(data, bins=30, cumulative=True, histtype='step')` |
|
||||
| 2D histogram | `ax.hist2d(x, y, bins=30, cmap='viridis')` |
|
||||
|
||||
### Histogram
|
||||
### Pie Charts
|
||||
|
||||
```python
|
||||
data = np.random.randn(1000) # Random normal distribution
|
||||
plt.hist(data, bins=30) # Histogram with 30 bins
|
||||
plt.title("Histogram")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Basic pie | `ax.pie(sizes, labels=labels, autopct='%1.1f%%')` |
|
||||
| Exploded pie | `ax.pie(sizes, labels=labels, explode=(0, 0.1, 0, 0), shadow=True)` |
|
||||
| Donut chart | `ax.pie(sizes, labels=labels, wedgeprops=dict(width=0.3))` |
|
||||
| Custom colors | `ax.pie(sizes, labels=labels, colors=plt.cm.Set3(range(len(sizes))))` |
|
||||
|
||||
### Pie Chart
|
||||
### Box Plots
|
||||
|
||||
```python
|
||||
sizes = [25, 35, 20, 20]
|
||||
labels = ['A', 'B', 'C', 'D']
|
||||
plt.pie(sizes, labels=labels, autopct='%1.1f%%') # Pie chart with % labels
|
||||
plt.title("Pie Chart")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Basic box plot | `ax.boxplot(data)` |
|
||||
| Grouped box plots | `ax.boxplot([data1, data2, data3], labels=['A','B','C'])` |
|
||||
| With notches | `ax.boxplot(data, notch=True, patch_artist=True)` |
|
||||
| Violin plot | `ax.violinplot(data, showmeans=True, showmedians=True)` |
|
||||
|
||||
---
|
||||
|
||||
## Customization {.row-span-2}
|
||||
|
||||
| Feature | Code Example | Description |
|
||||
| --------------- | ------------------------------------------------------------------------------------ | ------------------------------------------- |
|
||||
| Title | `plt.title("Title")` | Set the title of the plot |
|
||||
| X/Y Labels | `plt.xlabel("X")`, `plt.ylabel("Y")` | Label the axes |
|
||||
| Grid | `plt.grid(True)` | Show grid |
|
||||
| Legend | `plt.legend(["line1"])` | Add legend |
|
||||
| Line Style | `plt.plot(x, y, linestyle='--')` | Dashed line |
|
||||
| Color | `plt.plot(x, y, color='green')` | Set line color |
|
||||
| Marker | `plt.plot(x, y, marker='o')` | Show markers on points |
|
||||
| Axis Limits | `plt.xlim(0, 10)`, `plt.ylim(-1, 1)` | Set axis range |
|
||||
| Tick Labels | `plt.xticks([...])`, `plt.yticks([...])` | Customize tick positions |
|
||||
| Text Annotation | `plt.text(5, 0, "Midpoint")` | Add text at a specific coordinate |
|
||||
| Arrow | `plt.annotate("Peak", xy=(7, 1), xytext=(6, 1.5), arrowprops=dict(arrowstyle="->"))` | Add annotation arrow |
|
||||
| Style Sheets | `plt.style.use('ggplot')` | Use predefined styles like `seaborn`, `bmh` |
|
||||
### Figure Elements
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Title | `ax.set_title("Title", fontsize=14, fontweight='bold')` |
|
||||
| Axis labels | `ax.set_xlabel("X Label", fontsize=12); ax.set_ylabel("Y Label")` |
|
||||
| Axis limits | `ax.set_xlim(0, 10); ax.set_ylim(-1.5, 1.5)` |
|
||||
| Grid | `ax.grid(True, linestyle='--', alpha=0.7)` |
|
||||
| Legend | `ax.legend(loc='best', frameon=True, fancybox=True, shadow=True)` |
|
||||
| Legend outside | `ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left')` |
|
||||
|
||||
### Line & Marker Styles
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Linestyles | `'-'`, `'--'`, `'-.'`, `':'` |
|
||||
| Markers | `'.'`, `'o'`, `'v'`, `'^'`, `'s'`, `'*'`, `'+'`, `'x'` |
|
||||
| Color specification | `'r'`, `'#FF5733'`, `(0.2, 0.4, 0.6)`, `'red'` |
|
||||
| Line width | `linewidth=2` or `lw=2` |
|
||||
| Marker size | `markersize=6` or `ms=6` |
|
||||
| Transparency | `alpha=0.5` |
|
||||
|
||||
### Text & Annotations
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Add text | `ax.text(x, y, "Text", fontsize=10, ha='center', va='bottom')` |
|
||||
| Annotate with arrow | `ax.annotate("Peak", xy=(7, 1), xytext=(6, 1.5), arrowprops=dict(arrowstyle='->', color='red'))` |
|
||||
| Math expressions | `ax.text(5, 0.5, r'$E = mc^2$', fontsize=14)` |
|
||||
| Formula in title | `ax.set_title(r'$\sin(x) + \cos(x)$')` |
|
||||
|
||||
### Ticks & Scales
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Custom ticks | `ax.set_xticks([0, 2, 4, 6, 8, 10])` |
|
||||
| Custom tick labels | `ax.set_xticklabels(['zero', 'two', 'four', 'six', 'eight', 'ten'])` |
|
||||
| Rotate ticks | `plt.xticks(rotation=45)` |
|
||||
| Log scale | `ax.set_xscale('log'); ax.set_yscale('log')` |
|
||||
| Symlog | `ax.set_xscale('symlog', linthresh=0.01)` |
|
||||
| Twin axes | `ax2 = ax.twinx(); ax2.plot(x, y2, 'r-')` |
|
||||
|
||||
### Colormaps
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Sequential | `'viridis'`, `'plasma'`, `'inferno'`, `'magma'`, `'cividis'` |
|
||||
| Diverging | `'RdBu'`, `'RdBu_r'`, `'coolwarm'`, `'seismic'` |
|
||||
| Qualitative | `'Set1'`, `'Set2'`, `'Set3'`, `'tab10'`, `'tab20'` |
|
||||
| Reversed | `'viridis_r'` |
|
||||
| Use colormap | `plt.cm.viridis(np.linspace(0, 1, 10))` |
|
||||
|
||||
### Stylesheets
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Default | `plt.style.use('default')` |
|
||||
| Seaborn | `plt.style.use('seaborn-v0_8')` |
|
||||
| FiveThirtyEight | `plt.style.use('fivethirtyeight')` |
|
||||
| GGPlot | `plt.style.use('ggplot')` |
|
||||
| Dark background | `plt.style.use('dark_background')` |
|
||||
| Publication | `plt.style.use('bmh')` |
|
||||
| Available styles | `print(plt.style.available)` |
|
||||
|
||||
---
|
||||
|
||||
## Subplots & Layouts
|
||||
## Subplots & Layouts {.row-span-2}
|
||||
|
||||
### Multiple Subplots
|
||||
### Creating Subplots
|
||||
|
||||
```python
|
||||
fig, axs = plt.subplots(2, 2) # Create 2x2 grid of subplots
|
||||
axs[0, 0].plot(x, y) # Top-left subplot
|
||||
axs[0, 1].scatter(x, y) # Top-right subplot
|
||||
axs[1, 0].bar(categories, values) # Bottom-left subplot
|
||||
axs[1, 1].hist(data) # Bottom-right subplot
|
||||
plt.tight_layout() # Adjust spacing to prevent overlap
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| 2x2 grid | `fig, axes = plt.subplots(2, 2, figsize=(10, 8))` |
|
||||
| Access subplot | `axes[0, 0].plot(x, y)` |
|
||||
| Flatten axes array | `axes_flat = axes.flatten()` |
|
||||
| Share x-axis | `fig, axes = plt.subplots(2, 1, sharex=True)` |
|
||||
| Share y-axis | `fig, axes = plt.subplots(2, 1, sharey=True)` |
|
||||
| Add subplot manually | `ax = fig.add_subplot(2, 2, 1)` |
|
||||
| GridSpec | `gs = fig.add_gridspec(2, 2, width_ratios=[1, 2], height_ratios=[1, 2])` |
|
||||
| Span multiple cells | `ax = fig.add_subplot(gs[0, :])` |
|
||||
|
||||
### Figure Size
|
||||
### Layout Adjustments
|
||||
|
||||
```python
|
||||
plt.figure(figsize=(10, 5)) # Set figure size (width, height in inches)
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Automatic spacing | `plt.tight_layout(pad=1.5)` |
|
||||
| Fine-tune spacing | `plt.subplots_adjust(left=0.1, right=0.9, top=0.9, bottom=0.1, wspace=0.3, hspace=0.3)` |
|
||||
| Constrained layout | `fig, axes = plt.subplots(constrained_layout=True)` |
|
||||
| Colorbar | `fig.colorbar(im, ax=axes, orientation='vertical', fraction=0.02, pad=0.02)` |
|
||||
|
||||
### Figure Size & DPI
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Set figure size | `fig = plt.figure(figsize=(8, 6))` |
|
||||
| Set DPI | `fig.set_dpi(150)` |
|
||||
| Auto-figure size | `fig.set_size_inches(8, 6)` |
|
||||
| Save with DPI | `fig.savefig('fig.png', dpi=300, bbox_inches='tight')` |
|
||||
|
||||
---
|
||||
|
||||
## Advanced Visualizations {.cols-2}
|
||||
|
||||
### Heatmap
|
||||
### Image & Heatmap
|
||||
|
||||
```python
|
||||
data = np.random.rand(10, 10) # Random 10x10 matrix
|
||||
plt.imshow(data, cmap='hot', interpolation='nearest') # Display as image
|
||||
plt.colorbar() # Show color scale
|
||||
plt.title("Heatmap")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Display image | `ax.imshow(image, cmap='gray', interpolation='bilinear')` |
|
||||
| Heatmap | `im = ax.imshow(matrix, cmap='hot', aspect='auto')` |
|
||||
| Colorbar | `fig.colorbar(im, ax=ax, label='Intensity')` |
|
||||
| Hide axes | `ax.axis('off')` |
|
||||
| Aspect ratio | `ax.set_aspect('equal')` |
|
||||
|
||||
### Contour Plot
|
||||
### Contour & Surface
|
||||
|
||||
```python
|
||||
X, Y = np.meshgrid(x, x)
|
||||
Z = np.sin(X) * np.cos(Y)
|
||||
plt.contour(X, Y, Z) # Contour lines
|
||||
plt.title("Contour Plot")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Contour lines | `contour = ax.contour(X, Y, Z, levels=20, cmap='viridis')` |
|
||||
| Filled contour | `ax.contourf(X, Y, Z, levels=20, cmap='viridis')` |
|
||||
| Contour labels | `ax.clabel(contour, inline=True, fontsize=8)` |
|
||||
| 3D surface | `ax.plot_surface(X, Y, Z, cmap='viridis', alpha=0.8)` |
|
||||
| 3D wireframe | `ax.plot_wireframe(X, Y, Z, color='black', alpha=0.3)` |
|
||||
|
||||
### 3D Plot
|
||||
### 3D Plots
|
||||
|
||||
```python
|
||||
from mpl_toolkits.mplot3d import Axes3D
|
||||
fig = plt.figure()
|
||||
ax = fig.add_subplot(111, projection='3d') # 3D subplot
|
||||
ax.plot3D(x, y, np.cos(x)) # 3D line
|
||||
plt.title("3D Plot")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Setup 3D axes | `from mpl_toolkits.mplot3d import Axes3D; fig = plt.figure(); ax = fig.add_subplot(111, projection='3d')` |
|
||||
| 3D line | `ax.plot3D(x, y, z)` |
|
||||
| 3D scatter | `ax.scatter3D(x, y, z, c=z, cmap='viridis')` |
|
||||
| 3D bar | `ax.bar3d(x, y, z, dx, dy, dz)` |
|
||||
| Set 3D view | `ax.view_init(elev=30, azim=45)` |
|
||||
|
||||
### Specialized Plots
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Error bars | `ax.errorbar(x, y, yerr=errors, xerr=x_errors, fmt='o', capsize=5)` |
|
||||
| Stem plot | `ax.stem(x, y)` |
|
||||
| Event plot | `ax.eventplot(data, lineoffsets=1, linelengths=0.5)` |
|
||||
| Stream plot | `ax.streamplot(X, Y, U, V, density=1.5)` |
|
||||
| Quiver (vector field) | `ax.quiver(x, y, u, v, scale=0.5)` |
|
||||
| Polar plot | `ax = fig.add_subplot(111, projection='polar'); ax.plot(theta, r)` |
|
||||
|
||||
---
|
||||
|
||||
## Working with Images
|
||||
|
||||
```python
|
||||
import matplotlib.image as mpimg
|
||||
img = mpimg.imread('image.jpg') # Load image
|
||||
plt.imshow(img) # Display image
|
||||
plt.axis('off') # Hide axes
|
||||
plt.title("Image Display")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Load image | `import matplotlib.image as mpimg; img = mpimg.imread('image.jpg')` |
|
||||
| Display image | `ax.imshow(img)` |
|
||||
| Save image | `mpimg.imsave('output.jpg', img, dpi=300)` |
|
||||
| Resize with interpolation | `ax.imshow(img, interpolation='bicubic')` |
|
||||
| RGB channels | `red = img[:,:,0]; green = img[:,:,1]; blue = img[:,:,2]` |
|
||||
|
||||
---
|
||||
|
||||
## For Machine Learning
|
||||
|
||||
### Plotting Loss vs Epoch
|
||||
### Training Curves
|
||||
|
||||
```python
|
||||
epochs = range(1, 11)
|
||||
loss = [0.9, 0.7, 0.5, 0.4, 0.3, 0.25, 0.2, 0.18, 0.15, 0.13]
|
||||
plt.plot(epochs, loss)
|
||||
plt.title("Training Loss")
|
||||
plt.xlabel("Epoch")
|
||||
plt.ylabel("Loss")
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Loss vs epochs | `ax.plot(epochs, train_loss, label='Train'); ax.plot(epochs, val_loss, label='Validation')` |
|
||||
| Learning rate schedule | `ax.semilogy(epochs, learning_rates)` |
|
||||
| Accuracy | `ax.plot(epochs, train_acc, 'b-', label='Train'); ax.plot(epochs, val_acc, 'r-', label='Val')` |
|
||||
| Multiple metrics | `ax1 = ax.twinx(); ax1.plot(epochs, loss, 'r-')` |
|
||||
|
||||
### Confusion Matrix (Heatmap)
|
||||
### Confusion Matrix
|
||||
|
||||
```python
|
||||
from sklearn.metrics import confusion_matrix
|
||||
import seaborn as sns
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Confusion matrix | `from sklearn.metrics import confusion_matrix; cm = confusion_matrix(y_true, y_pred)` |
|
||||
| Heatmap | `im = ax.imshow(cm, cmap='Blues', interpolation='nearest')` |
|
||||
| Annotate values | `for i in range(cm.shape[0]): for j in range(cm.shape[1]): ax.text(j, i, cm[i, j], ha='center', va='center')` |
|
||||
| Normalized | `cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]` |
|
||||
|
||||
y_true = [0, 1, 2, 2, 0]
|
||||
y_pred = [0, 0, 2, 2, 1]
|
||||
cm = confusion_matrix(y_true, y_pred) # Create confusion matrix
|
||||
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues') # Heatmap visualization
|
||||
plt.title("Confusion Matrix")
|
||||
plt.show()
|
||||
```
|
||||
### ROC & Precision-Recall
|
||||
|
||||
### ROC Curve
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| ROC curve | `from sklearn.metrics import roc_curve, auc; fpr, tpr, _ = roc_curve(y_true, y_scores); auc_val = auc(fpr, tpr)` |
|
||||
| ROC plot | `ax.plot(fpr, tpr, label=f'AUC = {auc_val:.2f}'); ax.plot([0,1], [0,1], 'k--')` |
|
||||
| Precision-Recall | `from sklearn.metrics import precision_recall_curve; precision, recall, _ = precision_recall_curve(y_true, y_scores)` |
|
||||
| PR plot | `ax.plot(recall, precision, label='PR Curve')` |
|
||||
|
||||
```python
|
||||
from sklearn.metrics import roc_curve, auc
|
||||
### Feature Analysis
|
||||
|
||||
fpr, tpr, _ = roc_curve([0, 0, 1, 1], [0.1, 0.4, 0.35, 0.8]) # Compute ROC
|
||||
roc_auc = auc(fpr, tpr) # Area under curve
|
||||
plt.plot(fpr, tpr, label=f'AUC = {roc_auc:.2f}')
|
||||
plt.plot([0, 1], [0, 1], 'k--') # Diagonal line
|
||||
plt.xlabel("False Positive Rate")
|
||||
plt.ylabel("True Positive Rate")
|
||||
plt.title("ROC Curve")
|
||||
plt.legend()
|
||||
plt.show()
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Feature importance | `ax.barh(feature_names, importances, color='steelblue')` |
|
||||
| Correlation matrix | `im = ax.imshow(corr_matrix, cmap='RdBu_r', vmin=-1, vmax=1)` |
|
||||
| Pairwise scatter | `import seaborn as sns; sns.pairplot(df)` |
|
||||
| Parallel coordinates | `from pandas.plotting import parallel_coordinates; parallel_coordinates(df, 'class')` |
|
||||
|
||||
---
|
||||
|
||||
## Saving Plots
|
||||
## Animations
|
||||
|
||||
```python
|
||||
plt.savefig("figure.png", dpi=300, bbox_inches='tight') # Save plot to file
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Basic animation | `from matplotlib.animation import FuncAnimation; anim = FuncAnimation(fig, update, frames=100, interval=50)` |
|
||||
| Save animation | `anim.save('animation.gif', writer='pillow', fps=20)` |
|
||||
| Update function | `def update(frame): ax.clear(); ax.plot(x[:frame], y[:frame]); return ax,` |
|
||||
| Interactive animation | `plt.ion(); fig, ax = plt.subplots(); ax.plot(x, y); plt.pause(0.01); plt.ioff()` |
|
||||
|
||||
---
|
||||
|
||||
## Saving & Exporting
|
||||
|
||||
### File Formats
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| PNG (high DPI) | `fig.savefig('figure.png', dpi=300, bbox_inches='tight')` |
|
||||
| PDF (vector) | `fig.savefig('figure.pdf', bbox_inches='tight')` |
|
||||
| SVG (vector) | `fig.savefig('figure.svg', bbox_inches='tight')` |
|
||||
| EPS (vector) | `fig.savefig('figure.eps', bbox_inches='tight')` |
|
||||
| Transparent background | `fig.savefig('figure.png', transparent=True)` |
|
||||
| With metadata | `fig.savefig('figure.pdf', metadata={'Author': 'Name', 'Title': 'Plot'})` |
|
||||
|
||||
### Publication Quality
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| High-res for paper | `plt.rcParams['figure.dpi'] = 300` |
|
||||
| Fonts for publication | `plt.rcParams['font.family'] = 'serif'; plt.rcParams['font.serif'] = ['Times New Roman']` |
|
||||
| Multiple figures | `fig.savefig('figure.pdf', bbox_inches='tight', pad_inches=0.05)` |
|
||||
| Size constraints | `fig = plt.figure(figsize=(3.4, 2.4))` (single column) |
|
||||
|
||||
---
|
||||
|
||||
## Show & Clear
|
||||
|
||||
```python
|
||||
plt.show() # Show plot window
|
||||
plt.clf() # Clear current figure (useful when plotting in loops)
|
||||
plt.close() # Close figure window (useful in scripts or GUI apps)
|
||||
```
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Display plot | `plt.show()` |
|
||||
| Clear current figure | `plt.clf()` |
|
||||
| Clear current axes | `plt.cla()` |
|
||||
| Close all figures | `plt.close('all')` |
|
||||
| Close specific figure | `plt.close(fig)` |
|
||||
| Suppress display | `plt.ioff()` |
|
||||
|
||||
---
|
||||
|
||||
## More Useful Functions
|
||||
## Performance & Memory
|
||||
|
||||
| Function | Use Case |
|
||||
| ------------------------------ | ------------------------------- |
|
||||
| `plt.fill_between(x, y1, y2)` | Fill area between curves |
|
||||
| `plt.axhline(y=value)` | Draw horizontal line at y |
|
||||
| `plt.axvline(x=value)` | Draw vertical line at x |
|
||||
| `plt.errorbar(x, y, yerr=...)` | Plot with error bars |
|
||||
| `plt.twinx()` | Create secondary y-axis |
|
||||
| `plt.subplots_adjust(...)` | Manually adjust subplot spacing |
|
||||
| `plt.gca()` | Get current axes |
|
||||
| `plt.gcf()` | Get current figure |
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Fast plotting (AGG backend) | `import matplotlib; matplotlib.use('Agg')` |
|
||||
| Interactive backend | `import matplotlib; matplotlib.use('TkAgg')` |
|
||||
| Reduce memory | `ax.plot(x[::10], y[::10])` (downsample) |
|
||||
| Efficient scatter | `ax.scatter(x, y, s=1)` (small markers) |
|
||||
| Disable antialiasing | `ax.plot(x, y, antialiased=False)` |
|
||||
| Batch plotting | `fig, axes = plt.subplots(2, 2); for ax in axes.flat: ax.plot(x, y)` |
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls & Best Practices
|
||||
|
||||
| Pitfall | Solution |
|
||||
|---------|----------|
|
||||
| Mixing pyplot and OOP | Stick to one interface; prefer OOP for production |
|
||||
| Overlapping legends | Place legend outside: `ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left')` |
|
||||
| Small fonts in publication | Set `plt.rcParams['font.size'] = 12` globally |
|
||||
| Slow rendering | Downsample data or use `agg` backend |
|
||||
| Figures cut off | Use `bbox_inches='tight'` in savefig |
|
||||
| Memory leaks in loops | Use `plt.clf()` or `plt.close()` after each iteration |
|
||||
| Inconsistent colormaps | Use same colormap for comparable visualizations |
|
||||
| Missing axis labels | Always label for reproducibility |
|
||||
| 3D plot overlapping | Adjust view with `ax.view_init()` |
|
||||
|
||||
---
|
||||
+443
-112
@@ -1,148 +1,479 @@
|
||||
|
||||
---
|
||||
title: Numpy
|
||||
date: 2020-12-14 18:28:43
|
||||
title: NumPy
|
||||
date: 2026-07-19
|
||||
background: bg-[#4f6fc3]
|
||||
tags:
|
||||
- scientific
|
||||
- computing
|
||||
- scientific computing
|
||||
- numerical analysis
|
||||
- array processing
|
||||
categories:
|
||||
- Python
|
||||
intro: |
|
||||
[NumPy](https://numpy.org/) is the fundamental package for scientific computing with Python. This cheat sheet is a quick reference for NumPy beginners.
|
||||
[NumPy](https://numpy.org/) is the fundamental package for scientific computing with Python. This cheat sheet provides a production-grade reference for NumPy ≥2.0 with emphasis on performance, memory efficiency, and advanced array operations.
|
||||
plugins:
|
||||
- copyCode
|
||||
---
|
||||
|
||||
## Getting Started {.cols-2}
|
||||
|
||||
### Introduction
|
||||
|
||||
You’ll also need to import numpy to get started:
|
||||
### Import Convention
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
# For performance-critical code
|
||||
np.set_printoptions(precision=4, suppress=True, linewidth=120)
|
||||
|
||||
# Enable error handling for floating point
|
||||
np.seterr(divide='raise', over='raise', under='ignore', invalid='raise')
|
||||
```
|
||||
|
||||
### Importing/exporting
|
||||
### Array Creation
|
||||
|
||||
| - | - |
|
||||
| ------------------------------------------ | --------------------- |
|
||||
| `np.loadtxt('file.txt')` | From a text file |
|
||||
| `np.genfromtxt('file.csv',delimiter=',')` | From a CSV file |
|
||||
| `np.savetxt('file.txt',arr,delimiter=' ')` | Writes to a text file |
|
||||
| `np.savetxt('file.csv',arr,delimiter=',')` | Writes to a CSV file |
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| From Python list | `np.array([1, 2, 3])` |
|
||||
| From Python list of lists | `np.array([(1, 2, 3), (4, 5, 6)])` |
|
||||
| Zeros array | `np.zeros((3, 4), dtype=np.float64)` |
|
||||
| Ones array | `np.ones((2, 3), dtype=np.int64)` |
|
||||
| Identity matrix | `np.eye(5, dtype=np.float32)` |
|
||||
| Diagonal array | `np.diag([1, 2, 3])` |
|
||||
| Full array | `np.full((2, 3), 8, dtype=np.float64)` |
|
||||
| Empty (uninitialized) | `np.empty((3, 3), dtype=np.float64)` |
|
||||
| Evenly spaced range | `np.linspace(0, 100, 6, dtype=np.float64)` |
|
||||
| Arithmetic range | `np.arange(0, 10, 3)` |
|
||||
| Log-spaced | `np.logspace(0, 3, 4, base=10)` |
|
||||
| From function | `np.fromfunction(lambda i, j: i + j, (3, 3))` |
|
||||
| From string | `np.fromstring('1 2 3', dtype=int, sep=' ')` |
|
||||
|
||||
### Creating Arrays {.row-span-2}
|
||||
### Random Arrays
|
||||
|
||||
| - | - |
|
||||
| --------------------------------- | ----------------------------------------------------------------- |
|
||||
| `np.array([1,2,3])` | One dimensional array |
|
||||
| `np.array([(1,2,3),(4,5,6)])` | Two dimensional array |
|
||||
| `np.zeros(3)` | 1D array of length 3 all values 0 |
|
||||
| `np.ones((3,4))` | 3x4 array with all values 1 |
|
||||
| `np.eye(5)` | 5x5 array of 0 with 1 on diagonal (Identity matrix) |
|
||||
| `np.linspace(0,100,6)` | Array of 6 evenly divided values from 0 to 100 |
|
||||
| `np.arange(0,10,3)` | Array of values from 0 to less than 10 with step 3 (eg [0,3,6,9]) |
|
||||
| `np.full((2,3),8)` | 2x3 array with all values 8 |
|
||||
| `np.random.rand(4,5)` | 4x5 array of random floats between 0–1 |
|
||||
| `np.random.rand(6,7)*100` | 6x7 array of random floats between 0–100 |
|
||||
| `np.random.randint(5,size=(2,3))` | 2x3 array with random ints between 0–4 |
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Uniform [0,1) | `np.random.rand(4, 5)` |
|
||||
| Uniform [low, high) | `np.random.uniform(0, 100, (4, 5))` |
|
||||
| Standard normal | `np.random.randn(4, 5)` |
|
||||
| Normal (mean, std) | `np.random.normal(0, 1, (4, 5))` |
|
||||
| Integers [low, high) | `np.random.randint(0, 10, (2, 3))` |
|
||||
| Random choice | `np.random.choice([1, 2, 3, 4, 5], size=3, replace=False)` |
|
||||
| Shuffle in-place | `np.random.shuffle(arr)` |
|
||||
| Permutation | `np.random.permutation(10)` |
|
||||
| Random seed | `np.random.seed(42)` |
|
||||
| Generator (new API) | `rng = np.random.default_rng(seed=42); rng.random((3, 3))` |
|
||||
|
||||
### Inspecting Properties
|
||||
---
|
||||
|
||||
| - | - |
|
||||
| ------------------- | ---------------------------------------- |
|
||||
| `arr.size` | Returns number of elements in arr |
|
||||
| `arr.shape` | Returns dimensions of arr (rows,columns) |
|
||||
| `arr.dtype` | Returns type of elements in arr |
|
||||
| `arr.astype(dtype)` | Convert arr elements to type dtype |
|
||||
| `arr.tolist()` | Convert arr to a Python list |
|
||||
| `np.info(np.eye)` | View documentation for np.eye |
|
||||
## Array Properties {.row-span-2}
|
||||
|
||||
### Copying/sorting/reshaping
|
||||
### Attributes
|
||||
|
||||
| - | - |
|
||||
| --------------------- | ------------------------------------------------------- |
|
||||
| `np.copy(arr)` | Copies arr to new memory |
|
||||
| `arr.view(dtype)` | Creates view of arr elements with type dtype |
|
||||
| `arr.sort()` | Sorts arr |
|
||||
| `arr.sort(axis=0)` | Sorts specific axis of arr |
|
||||
| `two_d_arr.flatten()` | Flattens 2D array two_d_arr to 1D |
|
||||
| `arr.T` | Transposes arr (rows become columns and vice versa) |
|
||||
| `arr.reshape(3,4)` | Reshapes arr to 3 rows, 4 columns without changing data |
|
||||
| `arr.resize((5,6))` | Changes arr shape to 5x6 and fills new values with 0 |
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Shape (dimensions) | `arr.shape` |
|
||||
| Number of elements | `arr.size` |
|
||||
| Number of dimensions | `arr.ndim` |
|
||||
| Data type | `arr.dtype` |
|
||||
| Item size (bytes) | `arr.itemsize` |
|
||||
| Total memory (bytes) | `arr.nbytes` |
|
||||
| Strides (memory layout) | `arr.strides` |
|
||||
| Base (view check) | `arr.base` is not None |
|
||||
| C-contiguous | `arr.flags['C_CONTIGUOUS']` |
|
||||
| F-contiguous | `arr.flags['F_CONTIGUOUS']` |
|
||||
|
||||
### Adding/removing Elements
|
||||
### Type Conversion
|
||||
|
||||
| - | - |
|
||||
| ------------------------- | -------------------------------------- |
|
||||
| `np.append(arr,values)` | Appends values to end of arr |
|
||||
| `np.insert(arr,2,values)` | Inserts values into arr before index 2 |
|
||||
| `np.delete(arr,3,axis=0)` | Deletes row on index 3 of arr |
|
||||
| `np.delete(arr,4,axis=1)` | Deletes column on index 4 of arr |
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Convert dtype | `arr.astype(np.float32)` |
|
||||
| To Python list | `arr.tolist()` |
|
||||
| To bytes | `arr.tobytes()` |
|
||||
| To file | `arr.tofile('data.bin')` |
|
||||
| To string | `np.array2string(arr, precision=2)` |
|
||||
| View as different dtype | `arr.view(np.int32)` |
|
||||
|
||||
### Combining/splitting
|
||||
---
|
||||
|
||||
| - | - |
|
||||
| ------------------------------------ | ---------------------------------------- |
|
||||
| `np.concatenate((arr1,arr2),axis=0)` | Adds arr2 as rows to the end of arr1 |
|
||||
| `np.concatenate((arr1,arr2),axis=1)` | Adds arr2 as columns to end of arr1 |
|
||||
| `np.split(arr,3)` | Splits arr into 3 sub-arrays |
|
||||
| `np.hsplit(arr,5)` | Splits arr horizontally on the 5th index |
|
||||
## Indexing & Slicing {.row-span-2}
|
||||
|
||||
### Indexing/slicing/subsetting
|
||||
### Basic Indexing
|
||||
|
||||
| - | - |
|
||||
| --------------------- | ------------------------------------------------------------------------- |
|
||||
| `arr[5]` | Returns the element at index 5 |
|
||||
| `arr[2,5]` | Returns the 2D array element on index [2][5] |
|
||||
| `arr[1]=4` | Assigns array element on index 1 the value 4 |
|
||||
| `arr[1,3]=10` | Assigns array element on index [1][3] the value 10 |
|
||||
| `arr[0:3]` | Returns the elements at indices 0,1,2 (On a 2D array: returns rows 0,1,2) |
|
||||
| `arr[0:3,4]` | Returns the elements on rows 0,1,2 at column 4 |
|
||||
| `arr[:2]` | Returns the elements at indices 0,1 (On a 2D array: returns rows 0,1) |
|
||||
| `arr[:,1]` | Returns the elements at index 1 on all rows |
|
||||
| `arr<5` | Returns an array with boolean values |
|
||||
| `(arr1<3) & (arr2>5)` | Returns an array with boolean values |
|
||||
| `~arr` | Inverts a boolean array |
|
||||
| `arr[arr<5]` | Returns array elements smaller than 5 |
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Single element | `arr[5]` |
|
||||
| 2D element | `arr[2, 5]` |
|
||||
| Single assignment | `arr[1] = 4` |
|
||||
| 2D assignment | `arr[1, 3] = 10` |
|
||||
| Slice (rows 0,1,2) | `arr[0:3]` |
|
||||
| Slice (rows 0,1,2, col 4) | `arr[0:3, 4]` |
|
||||
| First two rows | `arr[:2]` |
|
||||
| All rows, column 1 | `arr[:, 1]` |
|
||||
| Row 0, all columns | `arr[0, :]` |
|
||||
| Step slice | `arr[0:10:2]` |
|
||||
| Negative indexing | `arr[-1]`, `arr[-3:]` |
|
||||
| Ellipsis (multiple dims) | `arr[..., 1]` |
|
||||
|
||||
### Vector Math
|
||||
### Boolean Indexing
|
||||
|
||||
| - | - |
|
||||
| --------------------------- | ----------------------------------------------------------- |
|
||||
| `np.add(arr1,arr2)` | Elementwise add arr2 to arr1 |
|
||||
| `np.subtract(arr1,arr2)` | Elementwise subtract arr2 from arr1 |
|
||||
| `np.multiply(arr1,arr2)` | Elementwise multiply arr1 by arr2 |
|
||||
| `np.divide(arr1,arr2)` | Elementwise divide arr1 by arr2 |
|
||||
| `np.power(arr1,arr2)` | Elementwise raise arr1 raised to the power of arr2 |
|
||||
| `np.array_equal(arr1,arr2)` | Returns True if the arrays have the same elements and shape |
|
||||
| `np.sqrt(arr)` | Square root of each element in the array |
|
||||
| `np.sin(arr)` | Sine of each element in the array |
|
||||
| `np.log(arr)` | Natural log of each element in the array |
|
||||
| `np.abs(arr)` | Absolute value of each element in the array |
|
||||
| `np.ceil(arr)` | Rounds up to the nearest int |
|
||||
| `np.floor(arr)` | Rounds down to the nearest int |
|
||||
| `np.round(arr)` | Rounds to the nearest int |
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Boolean mask | `arr < 5` |
|
||||
| Multiple conditions | `(arr1 < 3) & (arr2 > 5)` |
|
||||
| OR condition | `(arr1 < 3) \| (arr2 > 5)` |
|
||||
| Invert | `~arr` |
|
||||
| Filter with condition | `arr[arr < 5]` |
|
||||
| Where (select) | `np.where(arr < 5, arr, 0)` |
|
||||
| Where (indices) | `np.where(arr < 5)` |
|
||||
| Extract | `np.extract(arr < 5, arr)` |
|
||||
|
||||
### Scalar Math
|
||||
### Fancy Indexing
|
||||
|
||||
| - | - |
|
||||
| -------------------- | -------------------------------------------------------------------- |
|
||||
| `np.add(arr,1)` | Add 1 to each array element |
|
||||
| `np.subtract(arr,2)` | Subtract 2 from each array element |
|
||||
| `np.multiply(arr,3)` | Multiply each array element by 3 |
|
||||
| `np.divide(arr,4)` | Divide each array element by 4 (returns np.nan for division by zero) |
|
||||
| `np.power(arr,5)` | Raise each array element to the 5th power |
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Integer list | `arr[[1, 3, 5]]` |
|
||||
| 2D integer list | `arr[[0, 1], [0, 1]]` |
|
||||
| Using np.ix_ | `arr[np.ix_([0, 2], [0, 2])]` |
|
||||
| Select with conditions | `arr[np.where(arr > 0)]` |
|
||||
| Clip values | `np.clip(arr, 0, 1)` |
|
||||
| Put values | `np.put(arr, [0, 2], [10, 20])` |
|
||||
| Take values | `np.take(arr, [0, 2])` |
|
||||
|
||||
### Statistics
|
||||
### Advanced Indexing
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Index as array | `arr[[0, 1, 2], [0, 1, 0]]` |
|
||||
| Index with masks | `arr[arr > 0.5] = 1` |
|
||||
| Complex condition | `arr[(arr > 0.2) & (arr < 0.8)]` |
|
||||
| Choose from arrays | `np.choose(selector, [arr1, arr2])` |
|
||||
| Diagonal selection | `np.diagonal(arr, offset=0)` |
|
||||
| Trace | `np.trace(arr)` |
|
||||
|
||||
---
|
||||
|
||||
## Shape Operations {.row-span-2}
|
||||
|
||||
### Reshaping
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Reshape (new view if possible) | `arr.reshape(3, 4)` |
|
||||
| Resize (in-place, may fill) | `arr.resize((5, 6))` |
|
||||
| Flatten (C-order) | `arr.flatten()` |
|
||||
| Flatten (F-order) | `arr.flatten('F')` |
|
||||
| Ravel (view when possible) | `arr.ravel()` |
|
||||
| Transpose | `arr.T` |
|
||||
| Transpose with axes | `arr.transpose(1, 0, 2)` |
|
||||
| Swap axes | `arr.swapaxes(0, 1)` |
|
||||
| Move axis | `np.moveaxis(arr, 0, -1)` |
|
||||
| Expand dims | `np.expand_dims(arr, axis=0)` |
|
||||
| Squeeze (remove size 1 dims) | `np.squeeze(arr)` |
|
||||
| Broadcast to shape | `np.broadcast_to(arr, (3, 5, 5))` |
|
||||
|
||||
### Adding/Removing Dimensions
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| New axis (view) | `arr[np.newaxis, :]` |
|
||||
| New axis (at end) | `arr[:, np.newaxis]` |
|
||||
| Expand dims | `np.expand_dims(arr, 1)` |
|
||||
| Remove singleton dims | `arr.squeeze(axis=0)` |
|
||||
| Concatenate | `np.concatenate([arr1, arr2], axis=0)` |
|
||||
| Stack (new axis) | `np.stack([arr1, arr2], axis=0)` |
|
||||
| Vertical stack | `np.vstack([arr1, arr2])` |
|
||||
| Horizontal stack | `np.hstack([arr1, arr2])` |
|
||||
| Column stack | `np.column_stack([arr1, arr2])` |
|
||||
| Tile (repeat) | `np.tile(arr, (2, 3))` |
|
||||
| Repeat each element | `np.repeat(arr, 3, axis=0)` |
|
||||
|
||||
### Splitting
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Split into N equal parts | `np.split(arr, 3)` |
|
||||
| Split by indices | `np.split(arr, [2, 5])` |
|
||||
| Horizontal split | `np.hsplit(arr, 5)` |
|
||||
| Vertical split | `np.vsplit(arr, 3)` |
|
||||
| Depth split (3D) | `np.dsplit(arr, 3)` |
|
||||
| Array split (unequal) | `np.array_split(arr, 3)` |
|
||||
|
||||
---
|
||||
|
||||
## Element-wise Operations
|
||||
|
||||
### Arithmetic
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Addition | `np.add(arr1, arr2)` or `arr1 + arr2` |
|
||||
| Subtraction | `np.subtract(arr1, arr2)` or `arr1 - arr2` |
|
||||
| Multiplication | `np.multiply(arr1, arr2)` or `arr1 * arr2` |
|
||||
| Division | `np.divide(arr1, arr2)` or `arr1 / arr2` |
|
||||
| Floor division | `np.floor_divide(arr1, arr2)` or `arr1 // arr2` |
|
||||
| Power | `np.power(arr1, arr2)` or `arr1 ** arr2` |
|
||||
| Modulo | `np.mod(arr1, arr2)` or `arr1 % arr2` |
|
||||
| Negative | `np.negative(arr)` or `-arr` |
|
||||
| Absolute | `np.abs(arr)` |
|
||||
| Sign | `np.sign(arr)` |
|
||||
|
||||
### Mathematical Functions
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Square root | `np.sqrt(arr)` |
|
||||
| Exponential | `np.exp(arr)` |
|
||||
| Natural log | `np.log(arr)` |
|
||||
| Log base 10 | `np.log10(arr)` |
|
||||
| Log base 2 | `np.log2(arr)` |
|
||||
| Sin/Cos/Tan | `np.sin(arr)`, `np.cos(arr)`, `np.tan(arr)` |
|
||||
| Arcsin/Arccos/Arctan | `np.arcsin(arr)`, `np.arccos(arr)`, `np.arctan(arr)` |
|
||||
| Hyperbolic | `np.sinh(arr)`, `np.cosh(arr)`, `np.tanh(arr)` |
|
||||
| Rounding | `np.round(arr, decimals=2)` |
|
||||
| Ceiling | `np.ceil(arr)` |
|
||||
| Floor | `np.floor(arr)` |
|
||||
| Truncate | `np.trunc(arr)` |
|
||||
| Degrees to radians | `np.radians(arr)` |
|
||||
| Radians to degrees | `np.degrees(arr)` |
|
||||
|
||||
### Comparison & Logic
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Equal | `np.equal(arr1, arr2)` or `arr1 == arr2` |
|
||||
| Not equal | `np.not_equal(arr1, arr2)` or `arr1 != arr2` |
|
||||
| Less than | `np.less(arr1, arr2)` or `arr1 < arr2` |
|
||||
| Greater than | `np.greater(arr1, arr2)` or `arr1 > arr2` |
|
||||
| Less/equal | `np.less_equal(arr1, arr2)` or `arr1 <= arr2` |
|
||||
| Greater/equal | `np.greater_equal(arr1, arr2)` or `arr1 >= arr2` |
|
||||
| Logical AND | `np.logical_and(arr1, arr2)` |
|
||||
| Logical OR | `np.logical_or(arr1, arr2)` |
|
||||
| Logical NOT | `np.logical_not(arr)` |
|
||||
| Logical XOR | `np.logical_xor(arr1, arr2)` |
|
||||
| Is finite | `np.isfinite(arr)` |
|
||||
| Is nan | `np.isnan(arr)` |
|
||||
| Is inf | `np.isinf(arr)` |
|
||||
|
||||
---
|
||||
|
||||
## Aggregation & Statistics
|
||||
|
||||
### Reductions
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Sum (all) | `np.sum(arr)` |
|
||||
| Sum (axis) | `np.sum(arr, axis=0)` |
|
||||
| Mean | `np.mean(arr)` |
|
||||
| Mean (axis) | `np.mean(arr, axis=1)` |
|
||||
| Median | `np.median(arr)` |
|
||||
| Min/Max | `np.min(arr)`, `np.max(arr)` |
|
||||
| Min/Max (axis) | `np.min(arr, axis=0)`, `np.max(arr, axis=1)` |
|
||||
| Standard deviation | `np.std(arr)` |
|
||||
| Variance | `np.var(arr)` |
|
||||
| Percentile | `np.percentile(arr, 75)` |
|
||||
| Quantile | `np.quantile(arr, 0.75)` |
|
||||
| Cumulative sum | `np.cumsum(arr)` |
|
||||
| Cumulative product | `np.cumprod(arr)` |
|
||||
| Dot product | `np.dot(arr1, arr2)` |
|
||||
| Inner product | `np.inner(arr1, arr2)` |
|
||||
| Outer product | `np.outer(arr1, arr2)` |
|
||||
|
||||
### Statistical Functions
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Correlation coefficient | `np.corrcoef(arr1, arr2)` |
|
||||
| Covariance | `np.cov(arr1, arr2)` |
|
||||
| Cross-correlation | `np.correlate(arr1, arr2, mode='full')` |
|
||||
| Convolution | `np.convolve(arr1, arr2, mode='full')` |
|
||||
| Histogram | `np.histogram(arr, bins=10)` |
|
||||
| Bincount | `np.bincount(arr.astype(int))` |
|
||||
| Unique values | `np.unique(arr)` |
|
||||
| Unique with counts | `np.unique(arr, return_counts=True)` |
|
||||
| Count non-zero | `np.count_nonzero(arr)` |
|
||||
| Any/All | `np.any(arr > 0)`, `np.all(arr > 0)` |
|
||||
|
||||
---
|
||||
|
||||
## Linear Algebra
|
||||
|
||||
### Matrix Operations
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Matrix multiplication | `np.matmul(A, B)` or `A @ B` |
|
||||
| Dot product | `np.dot(A, B)` |
|
||||
| Inner product | `np.inner(A, B)` |
|
||||
| Outer product | `np.outer(A, B)` |
|
||||
| Trace | `np.trace(A)` |
|
||||
| Determinant | `np.linalg.det(A)` |
|
||||
| Rank | `np.linalg.matrix_rank(A)` |
|
||||
| Condition number | `np.linalg.cond(A)` |
|
||||
|
||||
### Matrix Decompositions
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Inverse | `np.linalg.inv(A)` |
|
||||
| Pseudo-inverse | `np.linalg.pinv(A)` |
|
||||
| Eigenvalues/vectors | `np.linalg.eig(A)` |
|
||||
| Eigenvalues (only) | `np.linalg.eigvals(A)` |
|
||||
| SVD | `np.linalg.svd(A)` |
|
||||
| QR decomposition | `np.linalg.qr(A)` |
|
||||
| LU decomposition | `np.linalg.lu(A)` |
|
||||
| Cholesky decomposition | `np.linalg.cholesky(A)` |
|
||||
|
||||
### Solving Systems
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Solve linear system | `np.linalg.solve(A, b)` |
|
||||
| Least squares | `np.linalg.lstsq(A, b)` |
|
||||
| Norm (Euclidean) | `np.linalg.norm(A)` |
|
||||
| Norm (specific order) | `np.linalg.norm(A, ord=1)` |
|
||||
| Singular values | `np.linalg.svdvals(A)` |
|
||||
|
||||
---
|
||||
|
||||
## Broadcasting & Vectorization
|
||||
|
||||
### Broadcasting Rules
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Scalar + array | `arr + 1` |
|
||||
| 1D + 2D | `arr2d + arr1d` |
|
||||
| Matching trailing dims | `arr2d + arr1d.reshape(-1, 1)` |
|
||||
| Check broadcast | `np.broadcast_shapes((2,3), (3,))` |
|
||||
| Explicit broadcast | `np.broadcast_to(arr1d, (3, 4))` |
|
||||
| Broadcasting in ufunc | `np.add(arr2d, arr1d)` |
|
||||
|
||||
### Vectorization Patterns
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Vectorized addition | `arr1 + arr2` (instead of loop) |
|
||||
| Vectorized condition | `np.where(cond, arr1, arr2)` |
|
||||
| Vectorized selection | `arr[mask]` |
|
||||
| Vectorized accumulation | `np.cumsum(arr)` |
|
||||
| Element-wise operations | `np.sqrt(arr)` |
|
||||
| Reduce operations | `np.sum(arr)` |
|
||||
| Universal functions | `np.exp(arr)`, `np.log(arr)` |
|
||||
|
||||
---
|
||||
|
||||
## Memory & Performance
|
||||
|
||||
### Memory Management
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| View vs copy | `arr_view = arr[:]` (view); `arr_copy = arr.copy()` (copy) |
|
||||
| Check ownership | `arr.base is None` (own memory) |
|
||||
| Share memory check | `np.shares_memory(arr1, arr2)` |
|
||||
| Memory alignment | `np.array(arr, copy=False, order='C')` |
|
||||
| Memory layout | `np.ascontiguousarray(arr)` |
|
||||
| Fortran order | `np.asfortranarray(arr)` |
|
||||
| Byteswap | `arr.byteswap(inplace=True)` |
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Preallocate | `np.empty((N,))` |
|
||||
| Use local variables | `np_add = np.add` |
|
||||
| Avoid Python loops | Use NumPy vectorized operations |
|
||||
| Out-of-place operations | `np.add(arr1, arr2, out=result)` |
|
||||
| Use in-place | `arr += 1` (less memory) |
|
||||
| Reduce allocations | Reuse arrays with `out=` parameter |
|
||||
| Use float32 if possible | `np.float32` instead of `np.float64` |
|
||||
| Use memory-mapped files | `np.memmap('data.dat', dtype='float32', mode='r', shape=(1000, 1000))` |
|
||||
|
||||
---
|
||||
|
||||
## Input/Output {.cols-2}
|
||||
|
||||
### Text Files
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Load text | `np.loadtxt('file.txt', delimiter=' ')` |
|
||||
| Load CSV | `np.genfromtxt('file.csv', delimiter=',', skip_header=1)` |
|
||||
| Load with missing values | `np.genfromtxt('file.csv', delimiter=',', filling_values=0)` |
|
||||
| Save text | `np.savetxt('file.txt', arr, delimiter=' ', fmt='%.4f')` |
|
||||
| Save CSV | `np.savetxt('file.csv', arr, delimiter=',', fmt='%.4f')` |
|
||||
|
||||
### Binary Files
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Load .npy | `np.load('file.npy')` |
|
||||
| Save .npy | `np.save('file.npy', arr)` |
|
||||
| Load .npz (multiple arrays) | `data = np.load('file.npz'); data['arr1']` |
|
||||
| Save .npz | `np.savez('file.npz', arr1=arr1, arr2=arr2)` |
|
||||
| Save compressed .npz | `np.savez_compressed('file.npz', arr1=arr1, arr2=arr2)` |
|
||||
| Binary to file | `arr.tofile('data.bin')` |
|
||||
| Read from file | `np.fromfile('data.bin', dtype=np.float32)` |
|
||||
| Memory-mapped | `arr = np.memmap('data.dat', dtype='float32', mode='r', shape=(100, 100))` |
|
||||
|
||||
### Data Types
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Integer types | `np.int8`, `np.int16`, `np.int32`, `np.int64` |
|
||||
| Unsigned ints | `np.uint8`, `np.uint16`, `np.uint32`, `np.uint64` |
|
||||
| Floating types | `np.float16`, `np.float32`, `np.float64`, `np.float128` |
|
||||
| Complex types | `np.complex64`, `np.complex128` |
|
||||
| Boolean | `np.bool_` |
|
||||
| Object | `np.object_` |
|
||||
| String | `np.string_`, `np.unicode_` |
|
||||
| Datetime | `np.datetime64`, `np.timedelta64` |
|
||||
| Structured dtypes | `np.dtype([('name', 'U10'), ('age', 'i4')])` |
|
||||
|
||||
---
|
||||
|
||||
## Structured Arrays
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Create structured | `np.array([(1, 'Alice'), (2, 'Bob')], dtype=[('id','i4'), ('name','U10')])` |
|
||||
| Access by field | `arr['name']` |
|
||||
| Access by position | `arr[0]` |
|
||||
| Multiple fields | `arr[['id', 'name']]` |
|
||||
| Complex dtype | `np.dtype({'names': ['id','name'], 'formats': ['i4','U10']})` |
|
||||
| Recarray (by attribute) | `arr = np.rec.array([(1,'Alice'),(2,'Bob')], dtype=[('id','i4'), ('name','U10')]); arr.name` |
|
||||
|
||||
---
|
||||
|
||||
## Datetime & Timedelta
|
||||
|
||||
| Description | Code |
|
||||
|-------------|------|
|
||||
| Create datetime | `np.datetime64('2026-07-01')` |
|
||||
| Array of datetimes | `np.arange('2026-01', '2026-07', dtype='datetime64[M]')` |
|
||||
| Timedelta | `np.timedelta64(1, 'D')` |
|
||||
| Arithmetic | `np.datetime64('2026-07-01') - np.datetime64('2026-06-01')` |
|
||||
| Extract components | `dt.astype('datetime64[Y]')` (year) |
|
||||
| Day of week | `(dt.astype('datetime64[D]') - np.datetime64('1970-01-01')).astype('int64') % 7` |
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls & Best Practices
|
||||
|
||||
| Pitfall | Solution |
|
||||
|---------|----------|
|
||||
| Copy vs view confusion | Use `arr.copy()` explicitly for independent copy |
|
||||
| Mutation of views | Check with `arr.base is not None` |
|
||||
| Python loops over arrays | Use vectorized operations instead of loops |
|
||||
| Using int64 unnecessarily | Use `np.float32` or `np.int32` for memory efficiency |
|
||||
| Mutable default arguments | Avoid `arr=[]` in functions; use `None` |
|
||||
| Not specifying dtype | Specify `dtype` for precision control |
|
||||
| Using `np.append` in loops | Preallocate array for better performance |
|
||||
| Shape mismatch in broadcasting | Check shapes with `arr.shape` |
|
||||
| Floating point comparison | Use `np.isclose()` for comparison of floats |
|
||||
| Memory leak in large arrays | Use `del arr` or `arr = None` after use |
|
||||
| Inconsistent random seeding | Use `np.random.seed(42)` for reproducibility |
|
||||
| Using old random API | Use new `np.random.default_rng()` API |
|
||||
|
||||
---
|
||||
|
||||
| - | - |
|
||||
| --------------------- | ----------------------------------------------- |
|
||||
| `np.mean(arr,axis=0)` | Returns mean along specific axis |
|
||||
| `arr.sum()` | Returns sum of arr |
|
||||
| `arr.min()` | Returns minimum value of arr |
|
||||
| `arr.max(axis=0)` | Returns maximum value of specific axis |
|
||||
| `np.var(arr)` | Returns the variance of array |
|
||||
| `np.std(arr,axis=1)` | Returns the standard deviation of specific axis |
|
||||
| `arr.corrcoef()` | Returns correlation coefficient of array |
|
||||
+297
-73
@@ -1,6 +1,7 @@
|
||||
|
||||
---
|
||||
title: Pandas
|
||||
date: 2024-07-04
|
||||
date: 2026-07-19
|
||||
background: bg-[#110750]
|
||||
tags:
|
||||
- data analysis
|
||||
@@ -8,129 +9,352 @@ tags:
|
||||
categories:
|
||||
- Python
|
||||
intro: |
|
||||
[Pandas](https://pandas.pydata.org/) is a powerful data analysis and manipulation library for Python. This cheat sheet is a quick reference for Pandas beginners.
|
||||
[Pandas](https://pandas.pydata.org/) is a powerful data analysis and manipulation library for Python. This cheat sheet provides a comprehensive reference for modern pandas (≥2.2) with emphasis on performance, nullable dtypes, and PyArrow integration.
|
||||
plugins:
|
||||
- copyCode
|
||||
---
|
||||
|
||||
## Getting Started {.cols-2}
|
||||
|
||||
### Introduction
|
||||
|
||||
You’ll need to import pandas to get started:
|
||||
### Import Convention
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
|
||||
# Enable future features (Pandas 2.2+)
|
||||
pd.options.future.infer_string = True # Arrow-backed strings
|
||||
pd.options.mode.copy_on_write = True # Lazy copying, memory efficient
|
||||
pd.set_option('display.max_columns', None) # Full display in notebooks
|
||||
|
||||
# Optional: suppress performance warnings
|
||||
import warnings
|
||||
warnings.filterwarnings('ignore', category=pd.errors.PerformanceWarning)
|
||||
```
|
||||
|
||||
### Creating DataFrames
|
||||
|
||||
| - | - |
|
||||
| --------------------------------------------------------- | --------------------------- |
|
||||
| `pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})` | From a dictionary |
|
||||
| `pd.DataFrame(data=[{'a': 1, 'b': 2}, {'a': 3, 'b': 4}])` | From a list of dictionaries |
|
||||
| `pd.read_csv('file.csv')` | From a CSV file |
|
||||
| `pd.read_excel('file.xlsx')` | From an Excel file |
|
||||
| From a dictionary | `pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})` |
|
||||
| From a list of dictionaries | `pd.DataFrame([{'a': 1, 'b': 2}, {'a': 3, 'b': 4}])` |
|
||||
| From a CSV file (optimized) | `pd.read_csv('file.csv', engine='pyarrow', dtype_backend='pyarrow')` |
|
||||
| From a Parquet file | `pd.read_parquet('file.parquet', engine='pyarrow')` |
|
||||
| From an Excel file | `pd.read_excel('file.xlsx', engine='openpyxl')` |
|
||||
| From a SQL query | `pd.read_sql('SELECT * FROM table', con, index_col='id', parse_dates=['date'])` |
|
||||
| From a dictionary of arrays | `pd.DataFrame.from_dict(data, orient='index')` |
|
||||
| From a list of tuples | `pd.DataFrame.from_records([(1,2),(3,4)], columns=['a','b'])` |
|
||||
| Empty DataFrame with columns | `pd.DataFrame(columns=['col1','col2'])` |
|
||||
| From a JSON file | `pd.read_json('file.json', orient='records', lines=True)` |
|
||||
| From clipboard | `pd.read_clipboard()` |
|
||||
|
||||
### Inspecting Data {.row-span-2}
|
||||
|
||||
| - | - |
|
||||
| --------------- | -------------------------- |
|
||||
| `df.head()` | First 5 rows |
|
||||
| `df.tail()` | Last 5 rows |
|
||||
| `df.shape` | Number of rows and columns |
|
||||
| `df.info()` | Info on DataFrame |
|
||||
| `df.describe()` | Summary statistics |
|
||||
| `df.columns` | Column names |
|
||||
| `df.index` | Index |
|
||||
| `df.dtypes` | Data types of columns |
|
||||
| First n rows (default 5) | `df.head(10)` |
|
||||
| Last n rows (default 5) | `df.tail(10)` |
|
||||
| Number of rows and columns | `df.shape` |
|
||||
| Detailed info (memory, dtypes, nulls) | `df.info(show_counts=True, memory_usage='deep')` |
|
||||
| Summary statistics for numeric columns | `df.describe(percentiles=[.25,.5,.75], include='all')` |
|
||||
| Column names as list | `df.columns.tolist()` |
|
||||
| Index object | `df.index` |
|
||||
| Data types of columns | `df.dtypes` |
|
||||
| Memory usage per column | `df.memory_usage(deep=True)` |
|
||||
| Value counts with proportions | `df['col'].value_counts(dropna=False, normalize=True)` |
|
||||
| Unique values in column | `df['col'].unique()` |
|
||||
| Number of unique values | `df['col'].nunique(dropna=False)` |
|
||||
| Random sample of rows | `df.sample(n=5, random_state=42)` |
|
||||
| Check if DataFrame is empty | `df.empty` |
|
||||
|
||||
### Selecting Data
|
||||
|
||||
| - | - |
|
||||
| ---------------------- | ------------------------------ |
|
||||
| `df['col1']` | Select column |
|
||||
| `df[['col1', 'col2']]` | Select multiple columns |
|
||||
| `df.loc[0]` | Select row by index |
|
||||
| `df.loc[:, 'col1']` | Select all rows for 'col1' |
|
||||
| `df.iloc[0]` | Select row by position |
|
||||
| `df.iloc[0, 1]` | Select specific value |
|
||||
| `df[df['col1'] > 2]` | Select rows based on condition |
|
||||
| Select a single column (as Series) | `df['col']` or `df.col` |
|
||||
| Select multiple columns | `df[['col1', 'col2']]` |
|
||||
| Select rows by label index | `df.loc[5:10]` |
|
||||
| Select rows by integer position | `df.iloc[0:5]` |
|
||||
| Select a specific cell by label | `df.at[5, 'col']` |
|
||||
| Select a specific cell by position | `df.iat[0, 1]` |
|
||||
| Filter rows by condition | `df[df['col'] > 2]` |
|
||||
| Filter using query string (fast) | `df.query('col > 2 and col2 == "foo"')` |
|
||||
| Filter with multiple conditions | `df[(df['col']>2) & (df['col2']=='bar')]` |
|
||||
| Select columns by data type | `df.select_dtypes(include=['number', 'bool'])` |
|
||||
| Select first row | `df.iloc[0]` |
|
||||
| Select last row | `df.iloc[-1]` |
|
||||
| Select random row | `df.sample(1)` |
|
||||
|
||||
### Data Cleaning
|
||||
|
||||
| - | - |
|
||||
| --------------------------------------------- | ------------------------------------ |
|
||||
| `df.dropna()` | Drop rows with any missing values |
|
||||
| `df.dropna(axis=1)` | Drop columns with any missing values |
|
||||
| `df.fillna(0)` | Replace missing values with 0 |
|
||||
| `df.drop_duplicates()` | Drop duplicate rows |
|
||||
| `df.rename(columns={'old_name': 'new_name'})` | Rename columns |
|
||||
| `df.astype('int')` | Change data type |
|
||||
| Drop rows with any missing values | `df.dropna(how='any', axis=0)` |
|
||||
| Drop rows where all values are missing | `df.dropna(how='all', axis=0)` |
|
||||
| Drop columns with any missing values | `df.dropna(axis=1, how='any')` |
|
||||
| Drop columns where all values are missing | `df.dropna(axis=1, how='all')` |
|
||||
| Fill missing values with a scalar | `df.fillna(0, inplace=False)` |
|
||||
| Forward fill missing values | `df.ffill(axis=0)` |
|
||||
| Backward fill missing values | `df.bfill(axis=0)` |
|
||||
| Interpolate missing values | `df.interpolate(method='linear', limit_direction='both')` |
|
||||
| Replace values using dict or regex | `df.replace({'old': 'new'}, regex=True)` |
|
||||
| Rename columns | `df.rename(columns={'old':'new'})` |
|
||||
| Rename index labels | `df.rename(index={0:'first'})` |
|
||||
| Drop duplicate rows | `df.drop_duplicates(subset=['col'], keep='first')` |
|
||||
| Change column data type (nullable int) | `df = df.astype({'col': 'Int64'})` |
|
||||
| Convert to Arrow-backed string dtype | `df['col'] = df['col'].astype('string[pyarrow]')` |
|
||||
| Drop columns with variance below threshold | `df.loc[:, df.std() > threshold]` |
|
||||
| Clip values to a range | `df['col'].clip(lower=0, upper=100)` |
|
||||
|
||||
### Adding/Removing Data {.row-span-2}
|
||||
|
||||
| - | - |
|
||||
| -------------------------------------- | ------------------------------- |
|
||||
| `df['col3'] = df['col1'] + df['col2']` | Add new column |
|
||||
| `df.drop('col1', axis=1)` | Drop column |
|
||||
| `df.append(new_row)` | Add new row |
|
||||
| `df.insert(2, 'new_col', new_data)` | Insert new column at position 2 |
|
||||
| Add a new column by assignment | `df['new'] = df['a'] + df['b']` |
|
||||
| Add multiple new columns | `df.assign(new1 = df.a + df.b, new2 = df.a - df.b)` |
|
||||
| Insert column at a specific position | `df.insert(2, 'new_col', values)` |
|
||||
| Remove and return a column | `df.pop('col')` |
|
||||
| Drop a column | `df.drop('col', axis=1, inplace=False)` |
|
||||
| Drop a row by index label | `df.drop(0, axis=0)` |
|
||||
| Add a single row | `df.loc[len(df)] = [1,2,3]` |
|
||||
| Concatenate DataFrames row-wise | `pd.concat([df1, df2], ignore_index=True)` |
|
||||
| Concatenate DataFrames column-wise | `pd.concat([df1, df2], axis=1)` |
|
||||
| Concatenate with group keys | `pd.concat([df1, df2], keys=['A','B'])` |
|
||||
| **Note:** `df.append()` is deprecated. Use `pd.concat()` instead. |
|
||||
|
||||
### Combining Data
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| `pd.concat([df1, df2])` | Concatenate rows |
|
||||
| `pd.concat([df1, df2], axis=1)` | Concatenate columns |
|
||||
| `pd.merge(df1, df2, on='key')` | Merge DataFrames on key |
|
||||
| `pd.merge(df1, df2, left_on='key1', right_on='key2')` | Merge on different keys |
|
||||
| `df1.join(df2, lsuffix='_left', rsuffix='_right')` | Join DataFrames |
|
||||
| Merge two DataFrames on a key (inner join) | `pd.merge(df1, df2, on='key')` |
|
||||
| Merge two DataFrames (outer join) | `pd.merge(df1, df2, on='key', how='outer')` |
|
||||
| Merge two DataFrames (left/right join) | `pd.merge(df1, df2, on='key', how='left')` |
|
||||
| Merge on different key columns | `pd.merge(df1, df2, left_on='key1', right_on='key2')` |
|
||||
| Merge with validation and metadata | `pd.merge(df1, df2, on='key', validate='1:m', suffixes=('_l','_r'), indicator=True)` |
|
||||
| Join DataFrames on index | `df1.join(df2, how='inner', lsuffix='_l', rsuffix='_r')` |
|
||||
| Cartesian product (cross join) | `df1.join(df2, how='cross')` |
|
||||
| Fill missing values from another DataFrame | `df1.combine_first(df2)` |
|
||||
|
||||
## Advanced Indexing
|
||||
|
||||
### MultiIndex / Hierarchical Indexing
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| Create a MultiIndex from tuples | `pd.MultiIndex.from_tuples([('A',1), ('A',2), ('B',1), ('B',2)])` |
|
||||
| Set multiple columns as index | `df.set_index(['col1', 'col2'])` |
|
||||
| Select all rows at first level | `df.loc[('A', slice(None)), 'x']` |
|
||||
| Cross-section selection | `df.xs('A', level=0)` |
|
||||
| Get values of a specific index level | `df.index.get_level_values(0)` |
|
||||
| Pivot a level to columns (unstack) | `df.unstack(level=0)` |
|
||||
| Compress columns to index (stack) | `df.stack()` |
|
||||
| Swap levels of MultiIndex | `df.swaplevel()` |
|
||||
| Sort by a specific index level | `df.sort_index(level=1)` |
|
||||
| Reorder index levels | `df.reorder_levels([1,0])` |
|
||||
|
||||
### Categorical Data
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| Create an ordered categorical column | `df['cat'] = pd.Categorical(df['col'], categories=['low','med','high'], ordered=True)` |
|
||||
| Get integer codes for categories | `df['cat'].cat.codes` |
|
||||
| Reorder categories | `df['cat'].cat.reorder_categories(['high','med','low'])` |
|
||||
| Add new categories | `df['cat'].cat.add_categories(['very_high'])` |
|
||||
| Remove unused categories | `df['cat'].cat.remove_unused_categories()` |
|
||||
| Group by categorical (skip unobserved) | `df.groupby('cat', observed=True).mean()` |
|
||||
|
||||
## Time Series Operations
|
||||
|
||||
### Date Handling
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| Convert string column to datetime (with format) | `df['date'] = pd.to_datetime(df['date'], format='%Y-%m-%d')` |
|
||||
| Extract year from datetime | `df['year'] = df['date'].dt.year` |
|
||||
| Extract month from datetime | `df['month'] = df['date'].dt.month` |
|
||||
| Extract day from datetime | `df['day'] = df['date'].dt.day` |
|
||||
| Extract day of week (0=Monday) | `df['dayofweek'] = df['date'].dt.dayofweek` |
|
||||
| Extract quarter | `df['quarter'] = df['date'].dt.quarter` |
|
||||
| Check if date is weekend | `df['is_weekend'] = df['date'].dt.dayofweek >= 5` |
|
||||
| Set datetime column as index | `df.set_index('date', inplace=True)` |
|
||||
| Generate a date range | `pd.date_range('2026-01-01', periods=100, freq='D')` |
|
||||
| Add a date offset to each date | `df['date'] + pd.DateOffset(months=1)` |
|
||||
|
||||
### Resampling & Frequency Conversion
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| Downsample to monthly frequency (aggregate) | `df.resample('M').mean()` |
|
||||
| Upsample to daily frequency (forward fill) | `df.resample('D').ffill(limit=2)` |
|
||||
| Custom aggregation during resampling | `df.resample('QE-DEC').agg(['sum', 'mean'])` |
|
||||
| Resample with custom label and closed side | `df.resample('W-MON', label='right', closed='right')` |
|
||||
| Business quarter frequency | `df.resample('BQ').mean()` |
|
||||
| Custom business days (skip holidays) | `from pandas.tseries.offsets import CustomBusinessDay`<br>`offset = CustomBusinessDay(weekmask='Mon Tue Wed Thu Fri', holidays=['2026-07-04'])`<br>`df.resample(offset)` |
|
||||
| Change frequency without aggregation (asfreq) | `df.asfreq('D', method='ffill', limit=2)` |
|
||||
| Shift values forward/backward | `df['shifted'] = df['value'].shift(1)` |
|
||||
| Create lag/lead columns | `df['lag_3'] = df['value'].shift(3)` |
|
||||
|
||||
## Transformation Operations
|
||||
|
||||
### Reshaping
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| Pivot without aggregation (exact reshape) | `df.pivot(index='date', columns='variable', values='value')` |
|
||||
| Pivot with aggregation (pivot table) | `pd.pivot_table(df, values='value', index='date', columns='var', aggfunc='mean', fill_value=0, margins=True)` |
|
||||
| Melt from wide to long format | `df.melt(id_vars=['date'], value_vars=['a','b'], var_name='var', value_name='val')` |
|
||||
| Stack/unstack MultiIndex | `df.stack(future_stack=True)` / `df.unstack()` |
|
||||
| Cross-tabulation with normalization | `pd.crosstab(df['col1'], df['col2'], normalize='columns', margins=True, margins_name='Total')` |
|
||||
| Explode list-like column into rows | `df.explode('list_column', ignore_index=True)` |
|
||||
|
||||
### Aggregating Data
|
||||
|
||||
| - | - |
|
||||
| -------------------------------------- | ------------------------- |
|
||||
| `df['col1'].sum()` | Sum of values in column |
|
||||
| `df['col1'].mean()` | Mean of values in column |
|
||||
| `df['col1'].count()` | Count of values in column |
|
||||
| `df['col1'].min()` | Minimum value in column |
|
||||
| `df['col1'].max()` | Maximum value in column |
|
||||
| `df['col1'].std()` | Standard deviation |
|
||||
| `df['col1'].var()` | Variance |
|
||||
| `df.groupby('col1').sum()` | Group by and sum |
|
||||
| `df.groupby('col1').mean()` | Group by and mean |
|
||||
| `df.groupby(['col1', 'col2']).count()` | Group by multiple columns |
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| Sum of values | `df['col'].sum(skipna=True)` |
|
||||
| Mean of values | `df['col'].mean()` |
|
||||
| Count of non-NA values | `df['col'].count()` |
|
||||
| Minimum value | `df['col'].min()` |
|
||||
| Maximum value | `df['col'].max()` |
|
||||
| Standard deviation (sample) | `df['col'].std(ddof=1)` |
|
||||
| Variance (sample) | `df['col'].var(ddof=1)` |
|
||||
| Quantile (e.g., 75th percentile) | `df['col'].quantile(0.75)` |
|
||||
| Group by a column and sum | `df.groupby('col1')['col2'].sum()` |
|
||||
| Group by and aggregate multiple columns | `df.groupby('col1').agg({'col2': ['sum','mean'], 'col3': 'max'})` |
|
||||
| Named aggregation (new style) | `df.groupby('group').agg(avg_value=('value','mean'), max_date=('date','max'), count=('value','count'))` |
|
||||
| Custom aggregation function | `df.groupby('col1').agg(lambda x: x.max() - x.min())` |
|
||||
|
||||
### GroupBy Advanced
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| Transform: broadcast result to original rows | `df['mean_per_group'] = df.groupby('group')['value'].transform('mean')` |
|
||||
| Filter groups based on condition | `df.groupby('group').filter(lambda x: x['value'].sum() > 10)` |
|
||||
| Apply custom function to each group | `df.groupby('group').apply(lambda g: g.sort_values('date').head(2), include_groups=False)` |
|
||||
| Pipe transformation after groupby | `df.groupby('group').pipe(lambda g: g['value'].transform(lambda x: (x - x.mean()) / x.std()))` |
|
||||
| Groupby with observed categoricals | `df.groupby('category', observed=True).mean()` |
|
||||
|
||||
### Window Functions
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------------------- | ----------------------- |
|
||||
| Rolling mean with window size 5 | `df['col'].rolling(window=5, min_periods=3, center=True).mean()` |
|
||||
| Multiple aggregations on rolling window | `df['col'].rolling(3).agg(['sum','mean', lambda x: x.max() - x.min()])` |
|
||||
| Rolling with custom weights | `df['col'].rolling(window=10, win_type='gaussian', weights=lambda x: np.exp(-x**2)).mean()` |
|
||||
| Expanding statistics (all data up to current point) | `df['col'].expanding(min_periods=2).std()` |
|
||||
| Exponentially weighted moving average | `df['col'].ewm(span=10, adjust=True, ignore_na=False).mean()` |
|
||||
| Exponentially weighted variance | `df['col'].ewm(alpha=0.3, min_periods=5).var()` |
|
||||
| Apply custom function to rolling window | `df['col'].rolling(3).apply(lambda x: x.sum(), raw=True)` |
|
||||
|
||||
### Applying Functions {.row-span-2}
|
||||
|
||||
| - | - |
|
||||
| ------------------------------------ | ------------------------------------ |
|
||||
| `df.apply(np.sqrt)` | Apply function to all values |
|
||||
| `df['col1'].apply(lambda x: x ** 2)` | Apply function to column |
|
||||
| `df.applymap(str)` | Apply function to DataFrame elements |
|
||||
| `df['col1'].map({'a': 1, 'b': 2})` | Map values |
|
||||
| `df['col1'].replace('a', 1)` | Replace values |
|
||||
| Apply function element-wise to a column | `df['col'].apply(lambda x: x**2)` |
|
||||
| Apply function row-wise | `df.apply(lambda row: row['a'] + row['b'], axis=1)` |
|
||||
| Apply function element-wise to entire DataFrame | `df.map(lambda x: x.upper() if isinstance(x,str) else x)` (**Note:** `applymap` is deprecated) |
|
||||
| Map values using a dictionary | `df['col'].map({'a':1,'b':2}, na_action='ignore')` |
|
||||
| Vectorized string operations | `df['col'].str.upper()`, `df['col'].str.contains('pattern')` |
|
||||
| Vectorized datetime operations | `df['date'].dt.year`, `df['date'].dt.dayofweek` |
|
||||
| Fast arithmetic using eval (numexpr) | `df.eval('c = a * b + 2 * d', engine='numexpr')` |
|
||||
| Fast filtering using query (numexpr) | `df.query('a > b & c.str.contains("foo")')` |
|
||||
| Chain operations with pipe | `df.pipe(lambda d: d.assign(ratio=d.a/d.b)).pipe(...)` |
|
||||
|
||||
### Handling Dates
|
||||
### String Operations
|
||||
|
||||
| - | - |
|
||||
| ----------------------------------------- | ------------------- |
|
||||
| `df['date'] = pd.to_datetime(df['date'])` | Convert to datetime |
|
||||
| `df['year'] = df['date'].dt.year` | Extract year |
|
||||
| `df['month'] = df['date'].dt.month` | Extract month |
|
||||
| `df['day'] = df['date'].dt.day` | Extract day |
|
||||
| `df.set_index('date', inplace=True)` | Set date as index |
|
||||
| - | - |
|
||||
| ------------------------------------ | ------------------------------------ |
|
||||
| Convert to uppercase/lowercase | `df['col'].str.upper()`, `df['col'].str.lower()` |
|
||||
| Check if string contains pattern | `df['col'].str.contains('pattern', regex=True, na=False)` |
|
||||
| Split string into columns | `df['col'].str.split(',', expand=True, n=1)` |
|
||||
| Replace using regular expression | `df['col'].str.replace(r'\s+', '_', regex=True)` |
|
||||
| Extract capture groups into columns | `df['col'].str.extract(r'(\d+)-(\w+)')` |
|
||||
| Extract all matches | `df['col'].str.extractall(r'(\d+)')` |
|
||||
| Find all matches (returns list) | `df['col'].str.findall(r'\d+')` |
|
||||
| Concatenate strings with separator | `df['col'].str.cat(sep='|')` |
|
||||
| Pad strings to fixed width | `df['col'].str.pad(width=10, side='left', fillchar='0')` |
|
||||
| Extract substring | `df['col'].str.slice(start=0, stop=5)` |
|
||||
| Get string length | `df['col'].str.len()` |
|
||||
| Check start/end of string | `df['col'].str.startswith('prefix')`, `df['col'].str.endswith('.csv')` |
|
||||
| Strip whitespace | `df['col'].str.strip()`, `df['col'].str.lstrip()`, `df['col'].str.rstrip()` |
|
||||
|
||||
### Input/Output
|
||||
## Performance & Memory Optimization
|
||||
|
||||
### Performance Settings
|
||||
|
||||
| - | - |
|
||||
| ------------------------------------ | ------------------------------------ |
|
||||
| Enable copy-on-write (Pandas 2.0+) | `pd.options.mode.copy_on_write = True` |
|
||||
| Fast arithmetic with eval | `df.eval('c = a * b + 2 * d', engine='numexpr')` |
|
||||
| Fast filtering with query | `df.query('a > b and c == "foo"', engine='numexpr')` |
|
||||
| Read large CSV in chunks | `for chunk in pd.read_csv('large.csv', chunksize=100000, dtype_backend='pyarrow'): process(chunk)` |
|
||||
| Convert to numpy array (avoid copy) | `arr = df[['a','b']].to_numpy(copy=False)` |
|
||||
| Memory map large files | `df = pd.read_csv('large.csv', memory_map=True)` |
|
||||
| Use PyArrow backend for CSV | `pd.read_csv('file.csv', engine='pyarrow', dtype_backend='pyarrow')` |
|
||||
|
||||
### Memory Optimization
|
||||
|
||||
| - | - |
|
||||
| ------------------------------------ | ------------------------------------ |
|
||||
| Downcast integer columns | `def downcast(df):`<br>` for col in df.select_dtypes(include='integer'):`<br>` df[col] = pd.to_numeric(df[col], downcast='unsigned')`<br>` for col in df.select_dtypes(include='float'):`<br>` df[col] = pd.to_numeric(df[col], downcast='float')`<br>` return df` |
|
||||
| Convert to categorical if low cardinality | `for col in df.select_dtypes(include='object'):`<br>` if df[col].nunique() / len(df) < 0.05:`<br>` df[col] = df[col].astype('category')` |
|
||||
| Convert all to Arrow dtypes | `df = df.convert_dtypes(dtype_backend='pyarrow')` |
|
||||
| Use sparse arrays for zeros | `df['sparse'] = pd.arrays.SparseArray(df['col'], fill_value=0)` |
|
||||
| Diagnose memory usage | `df.memory_usage(deep=True)` |
|
||||
|
||||
## Input/Output {.cols-2}
|
||||
|
||||
| - | - |
|
||||
| ---------------------------------------------- | ------------------------------------ |
|
||||
| `df.to_csv('file.csv')` | Save DataFrame to CSV |
|
||||
| `df = pd.read_csv('file.csv')` | Load DataFrame from CSV |
|
||||
| `df.to_excel('file.xlsx')` | Save DataFrame to Excel |
|
||||
| `df = pd.read_excel('file.xlsx')` | Load DataFrame from Excel |
|
||||
| `from sqlalchemy import create_engine` | Import SQLAlchemy for SQL operations |
|
||||
| `engine = create_engine('sqlite:///:memory:')` | Create SQL engine |
|
||||
| `df.to_sql('table_name', engine)` | Save to SQL table |
|
||||
| `df = pd.read_sql('table_name', engine)` | Load from SQL table |
|
||||
| Save to CSV (optimized) | `df.to_csv('file.csv', index=False, encoding='utf-8')` |
|
||||
| Load from CSV (optimized) | `df = pd.read_csv('file.csv', engine='pyarrow', dtype_backend='pyarrow')` |
|
||||
| Save to Parquet (fast, compressed) | `df.to_parquet('file.parquet', engine='pyarrow')` |
|
||||
| Load from Parquet | `df = pd.read_parquet('file.parquet', engine='pyarrow')` |
|
||||
| Save to Excel | `df.to_excel('file.xlsx', sheet_name='Sheet1', engine='openpyxl')` |
|
||||
| Load from Excel | `df = pd.read_excel('file.xlsx', sheet_name=0, engine='openpyxl')` |
|
||||
| Save to JSON (line-delimited) | `df.to_json('file.json', orient='records', lines=True)` |
|
||||
| Load from JSON (line-delimited) | `df = pd.read_json('file.json', orient='records', lines=True)` |
|
||||
| Save to SQL table | `df.to_sql('table', con, if_exists='replace', index=False)` |
|
||||
| Load from SQL query | `df = pd.read_sql('SELECT * FROM table', con, index_col='id', parse_dates=['date'])` |
|
||||
| Copy to/from clipboard | `df.to_clipboard()` / `pd.read_clipboard()` |
|
||||
| Save to/from pickle (fast but unsafe) | `df.to_pickle('file.pkl')` / `pd.read_pickle('file.pkl')` |
|
||||
| Save to/from Feather (fast) | `df.to_feather('file.feather')` / `pd.read_feather('file.feather')` |
|
||||
| Save to/from HDF5 | `df.to_hdf('file.h5', key='data', mode='w')` / `pd.read_hdf('file.h5', key='data')` |
|
||||
|
||||
## Common Pitfalls & Best Practices
|
||||
|
||||
| Pitfall | Solution |
|
||||
|---------|----------|
|
||||
| Chained indexing (creates views) | Use `df.loc[df.a > 0, 'b']` instead of `df[df.a > 0]['b']` |
|
||||
| In-place mutation of large DataFrames | Use `inplace=False` (default) for method chaining; avoid mutation unless necessary |
|
||||
| Slicing without copy | Use `df.copy()` when you need an independent slice |
|
||||
| Slow datetime parsing | Always specify `format` in `pd.to_datetime()` for 10x speedup |
|
||||
| Memory bloat from object dtype | Use `categorical` dtype for low cardinality columns (<5% unique) |
|
||||
| Ambiguous merge relationships | Specify `validate` in merge ('1:1', '1:m', etc.) to catch errors |
|
||||
| Groupby with unused categories | Use `observed=True` to skip unobserved categories in groupby |
|
||||
| Slow loops with `apply` | Use vectorized operations when possible (`df['a'] + df['b']` vs `apply`) |
|
||||
| Deprecated `append()` | Use `pd.concat([df, other])` instead |
|
||||
| Deprecated `applymap()` | Use `df.map()` instead (Pandas 2.2+) |
|
||||
| Missing `pd.NA` handling | Check for `pd.NA` when using nullable dtypes |
|
||||
| Large CSV without optimization | Enable copy-on-write and use PyArrow backend |
|
||||
|
||||
## Deprecations & Version Notes
|
||||
|
||||
| Version | Deprecated/Removed | Replacement |
|
||||
|---------|-------------------|-------------|
|
||||
| 1.4.0 | `df.append()` | `pd.concat([df, other])` |
|
||||
| 2.2.0 | `df.applymap()` | `df.map()` |
|
||||
| 2.0.0 | `df.astype('int')` with NA | Use nullable `'Int64'` |
|
||||
| 2.0.0 | Inferred string dtype | Use `pd.options.future.infer_string = True` |
|
||||
| 2.0.0 | `pd.DataFrame.to_csv()` with `compression` | Use `compression='gzip'` explicitly |
|
||||
|
||||
|
||||
## Official Documentation & Resources
|
||||
|
||||
- [pandas.pydata.org](https://pandas.pydata.org/)
|
||||
- [pandas GitHub](https://github.com/pandas-dev/pandas)
|
||||
- [PyArrow integration](https://arrow.apache.org/docs/python/pandas.html)
|
||||
- [API Reference](https://pandas.pydata.org/docs/reference/index.html)
|
||||
- [User Guide](https://pandas.pydata.org/docs/user_guide/index.html)
|
||||
```
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
<svg width="1em" height="1em" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M12 2L1.5 20.5H22.5L12 2Z" fill="#1A73E8"/>
|
||||
<path d="M12 6.5L4.8 19H19.2L12 6.5Z" fill="#FFFFFF" fill-opacity="0.25"/>
|
||||
<path d="M12 9L7.5 17H16.5L12 9Z" fill="#FFFFFF"/>
|
||||
<circle cx="12" cy="14" r="1.5" fill="#1A73E8"/>
|
||||
<path d="M12 2V12.5" stroke="#FFFFFF" stroke-width="1.2" stroke-linecap="round"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 425 B |
@@ -0,0 +1,7 @@
|
||||
<svg width="1em" height="1em" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M12 2L1.5 20.5H22.5L12 2Z" fill="#1A73E8"/>
|
||||
<path d="M12 6.5L4.8 19H19.2L12 6.5Z" fill="#FFFFFF" fill-opacity="0.25"/>
|
||||
<path d="M12 9L7.5 17H16.5L12 9Z" fill="#FFFFFF"/>
|
||||
<circle cx="12" cy="14" r="1.5" fill="#1A73E8"/>
|
||||
<path d="M12 2V12.5" stroke="#FFFFFF" stroke-width="1.2" stroke-linecap="round"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 425 B |
Reference in new issue
Block a user