Merge remote-tracking branch 'origin/main'

This commit is contained in:
Fechin committed 2026-11-18 09:48:28 +08:00
commit 414d56468e
10 files changed
+1539 -381

No files matched your search

+464
View File
@@ -0,0 +1,464 @@
---
title: Antigravity CLI (agy)
date: 2026-09-22 12:00:00
background: bg-[#1a73e8]
tags:
- agy
- antigravity
- ai
- cli
- google
- coding
- agent
categories:
- AI
intro: |
[Antigravity CLI (agy)](https://antigravity.google/docs/cli) is Google's AI-first terminal development assistant and agentic coding CLI. It reads codebases, executes edits, manages background tasks, orchestrates subagents, and automates developer workflows.
plugins:
- copyCode
---
## Getting Started {.cols-3}
### Quick Start {.row-span-2}
```bash
# Launch interactive session in current repository
$ agy
# Resume most recent session
$ agy -c
$ agy --continue
# Resume a specific session by conversation ID
$ agy --conversation <conversation-id>
# Run non-interactive prompt and print output
$ agy -p "explain the architecture of this repo"
$ agy --print "fix lint errors across src/"
# Run prompt and continue interactively
$ agy -i "refactor auth middleware"
$ agy --prompt-interactive "add unit tests"
# Pipe stdin directly to agy
$ git diff | agy -p "write a git commit message"
$ cat logs.txt | agy -p "identify crash trace"
# Multi-directory workspace
$ agy --add-dir ../backend --add-dir ../frontend
```
### Authentication
| Method | Description |
| --- | --- |
| **Google Account** | Run `agy`, follow browser login prompts on first run |
| **Vertex AI (ADC)** | `gcloud auth application-default login` |
| **Service Account** | Set `GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa.json` |
| **Project Selection** | `agy --project <project-id>` or `GOOGLE_CLOUD_PROJECT` |
### Installation Methods
```bash
# Verify installation
$ agy --help
# Configure environment PATH and shell settings
$ agy install
# Update CLI to latest release
$ agy update
# View release notes & changelog
$ agy changelog
```
### Execution Modes
| Mode | Command Flag | Description |
| --- | --- | --- |
| **Default** | *(default)* | Prompts for confirmation before modifying files or running commands |
| **Accept Edits** | `--mode accept-edits` | Automatically applies edits and writes files without prompting |
| **Plan Mode** | `--mode plan` | Formulates step-by-step implementation plans before touching code |
| **Sandbox** | `--sandbox` | Restricts shell execution inside an isolated terminal sandbox |
| **Unrestricted** | `--dangerously-skip-permissions` | Auto-approves all tool permission prompts |
### Key Concepts
- **Agentic Loop**: Dynamically inspects files, runs shell commands, views diffs, and self-corrects based on tool results.
- **Progressive Disclosure**: Skills and contextual rules are loaded on demand rather than cluttering token context.
- **Asynchronous Tasks**: Background commands and multi-agent runs execute concurrently without blocking turns.
---
## CLI Options {.cols-2}
### Core Flags {.row-span-2}
| Flag | Short | Description |
| --- | --- | --- |
| `--print <prompt>` | `-p` | Run prompt non-interactively and print response |
| `--prompt-interactive <prompt>` | `-i` | Run initial prompt and remain interactive |
| `--continue` | `-c` | Resume the most recent conversation |
| `--conversation <id>` | | Resume specific conversation by ID |
| `--model <model>` | | Specify model for the session |
| `--effort <level>` | | Set reasoning effort (`low`, `medium`, `high`) |
| `--mode <mode>` | | Agent mode (`accept-edits`, `plan`) |
| `--sandbox` | | Enable terminal sandbox restrictions |
| `--dangerously-skip-permissions` | | Auto-approve all tool permissions |
| `--add-dir <path>` | | Add directory to workspace (repeatable) |
| `--project <id>` | | Google Cloud / Vertex project ID |
| `--new-project` | | Create a new project for this session |
| `--output-format <format>` | | Output format (`text`, `json`, `stream-json`) |
| `--input-format <format>` | | Input format (`text`, `stream-json`) |
| `--json-schema <schema>` | | Enforce structured JSON schema for outputs |
| `--print-timeout <duration>` | | Timeout limit for print mode (default `0s`) |
| `--log-file <path>` | | Override CLI log file path |
| `--remote-control` | | Connect to remote control background daemon |
| `--disable-slash-commands` | | Disable slash command expansion in print mode |
### Subcommands
```bash
# Manage MCP servers
$ agy mcp [list|add|remove|enable|disable]
# Manage plugins
$ agy plugin [list|install|uninstall|enable|disable|import|validate]
# List supported AI models
$ agy models
# List available agents
$ agy agents
# Manage remote control background daemon
$ agy remote-control [start|status|stop]
# Serve microphone input to a CLI on another host
$ agy mic-serve
# Update agy CLI
$ agy update
```
### Headless & CI/CD Formats
```bash
# Plain text output
$ agy -p "check repo for vulnerabilities" --output-format text
# JSON output for tooling pipelines
$ agy -p "extract exported API endpoints" --output-format json
# NDJSON streaming for real-time consumers
$ agy -p "generate OpenAPI spec" --output-format stream-json
# Enforce structured output via JSON Schema
$ agy -p "list todos" --json-schema schema.json --output-format json
```
---
## Model Selection & Reasoning {.cols-2}
### Models & Reasoning Effort {.row-span-2}
```bash
# Check all available models
$ agy models
# Run with Gemini 3.8 Flash with high reasoning (fast & deep)
$ agy --model gemini-3.8-flash-high
# Run with Gemini 3.1 Pro for complex architectures
$ agy --model gemini-3.1-pro-high
# Run with Claude Sonnet 4.6 (Thinking)
$ agy --model claude-sonnet-4-6
# Specify reasoning effort level
$ agy --effort high
$ agy --effort medium
$ agy --effort low
```
### Model Reference
| Model Identifier | Display Name | Recommended Use |
| --- | --- | --- |
| `gemini-3.8-flash-high` | Gemini 3.8 Flash (High) | Daily coding, rapid agent turns, default |
| `gemini-3.8-flash-medium` | Gemini 3.8 Flash (Medium) | Balanced speed and thinking tokens |
| `gemini-3.8-flash-low` | Gemini 3.8 Flash (Low) | Fastest latency, lightweight queries |
| `gemini-3.7-flash-high` | Gemini 3.7 Flash (High) | Previous generation high reasoning |
| `gemini-3.1-pro-high` | Gemini 3.1 Pro (High) | Massive refactors, cross-repo design |
| `claude-sonnet-4-6` | Claude Sonnet 4.6 | Thinking-enabled coding assistant |
| `claude-opus-4-6-thinking` | Claude Opus 4.6 | Deep complex multi-step reasoning |
| `gpt-oss-120b-medium` | GPT-OSS 120B | Open-weights reasoning tier |
---
## Interactive Slash Commands {.cols-3}
### Slash Commands `/` {.row-span-3}
| Slash Command | Purpose |
| --- | --- |
| `/help` | Show available slash commands and keybindings |
| `/plan` | Formulate a multi-step design & plan before execution |
| `/goal` | Autonomous mode: runs until goal is completed |
| `/boost` | Deep thinking, multi-angle exploration, rigorous checks |
| `/browser` | Web automation, page inspection, and live testing |
| `/grill-me` | Interactive interview to settle design decisions |
| `/schedule` | Set a one-off timer or recurring cron schedule |
| `/teamwork-preview` | Autonomous multi-agent collaboration |
| `/learn` | Persist developer preferences and lessons |
| `/exit` | Exit the CLI session |
| `/quit` | Alias for `/exit` |
### Goal & Boost Modes
```bash
# Inside interactive agy:
> /goal Implement end-to-end tests for billing API and verify they pass
# Boost mode for mission-critical changes:
> /boost Review security implications of session token rotation
```
### Interactive Interviews
```bash
# Challenge your assumptions before writing code
> /grill-me I want to migrate our database from MongoDB to Postgres
```
---
## Keyboard Shortcuts {.cols-3}
### General Controls {.row-span-2}
| Keybinding | Action |
| --- | --- |
| `Ctrl+D Ctrl+D` | Exit agy CLI session |
| `Ctrl+C` | Cancel currently running agent turn or streaming response |
| `Enter` | Submit prompt / send message |
| `Shift+Enter` | Insert a newline into multiline input |
| `Alt+Enter` | Insert a newline into multiline input |
### Input Navigation
| Keybinding | Action |
| --- | --- |
| `Up` / `Down` | Browse prompt history |
| `Ctrl+A` | Move cursor to beginning of line |
| `Ctrl+E` | Move cursor to end of line |
| `Ctrl+U` | Delete from cursor to start of line |
| `Ctrl+K` | Delete from cursor to end of line |
| `Ctrl+L` | Clear terminal screen |
### Agent Control
| Action | How |
| --- | --- |
| **Accept Edit** | Press `Enter` or click Approve on tool prompt |
| **Reject Edit** | Press `Esc` or type feedback |
| **Interrupt** | `Ctrl+C` halts tool call immediately |
---
## Subagents & Delegation {.cols-2}
### Subagent Architecture {.row-span-2}
Antigravity CLI can invoke specialized subagents that execute concurrently with their own contexts and workspaces:
```mermaid
flowchart TD
Parent[agy CLI Main Agent] --> Sub1[research Subagent]
Parent --> Sub2[self Subagent]
Parent --> Sub3[Custom Subagents]
Sub1 -->|Insights| Parent
Sub2 -->|Code Diffs| Parent
```
- **Isolated Context**: Subagents keep the main turn's context window clean and concise.
- **Parallel Work**: Background subagents report results automatically without polling.
- **Workspace Modes**:
- `inherit`: Shares current project workspace.
- `branch`: Creates an isolated git worktree branch.
- `share`: Uses shared repository cache without copying storage.
### Built-in Subagent Types
| Subagent | Scope | Purpose |
| --- | --- | --- |
| `research` | Read-only | Web search, codebase exploration, documentation analysis |
| `self` | Full tools | Inherits parent tools to run parallel branch edits |
| `custom` | Configurable | Defined dynamically via `define_subagent` |
---
## MCP Servers (Model Context Protocol) {.cols-2}
### Managing MCP Servers {.row-span-2}
Antigravity natively integrates MCP servers to connect your agent to external databases, APIs, and devtools:
```bash
# List configured MCP servers
$ agy mcp list
# Add local process (stdio) MCP server
$ agy mcp add postgres npx -y @modelcontextprotocol/server-postgres "postgresql://localhost/db"
# Add HTTP / SSE MCP server
$ agy mcp add weather https://api.weather-mcp.internal/sse
# Disable an MCP server temporarily
$ agy mcp disable postgres
# Re-enable an MCP server
$ agy mcp enable postgres
# Remove an MCP server
$ agy mcp remove postgres
```
### Configuration & Tool Loading
Servers are stored in `~/.gemini/antigravity-cli/mcp/<serverName>/`:
- **Eager Tools**: Automatically loaded into native agent tool registry.
- **Lazy Tools**: Registered on-demand via schemas (`<toolName>.json`) to conserve prompt tokens.
- **Instructions**: Optional `instructions.md` guide agent when and how to call tools.
---
## Customization: Rules & Skills {.cols-3}
### Hierarchical Rules {.row-span-2}
Rules guide coding standards and instructions automatically:
- **Project Root**: `GEMINI.md`, `AGENTS.md`
- **Scoped Rules**: `.agents/rules/*.md` or `.gemini/rules/`
- **Hierarchical Discovery**: Traverses from active file up to git root.
```markdown
<!-- GEMINI.md example -->
# Project Guidelines
- Always use TypeScript strict mode.
- Write tests with Vitest in tests/ directory.
```
### Agent Skills
Skills teach reusable workflows and procedures:
- Located in `.agents/skills/<name>/SKILL.md` or `~/.gemini/config/skills/`.
- Loaded dynamically via **Progressive Disclosure** (only activated when relevant).
```yaml
---
name: deploy-preview
description: Deploy branch to preview environment and run smoke tests
---
# Runbook steps here...
```
### Precedence Order
1. Workspace Project (`.agents/`)
2. Workspace Declared (`skills.json`, `plugins.json`)
3. Global User Config (`~/.gemini/config/`)
4. Built-in Applications & Skills
5. Global Declared Config
---
## Plugins & Extensions {.cols-2}
### Managing Plugins {.row-span-2}
Plugins bundle skills, rules, hooks, and MCP servers into installable units:
```bash
# List all installed plugins
$ agy plugin list
# Install from registry or marketplace
$ agy plugin install security-auditor@marketplace
# Install local plugin package
$ agy plugin install ./my-plugin
# Import plugins from Claude or Gemini configs
$ agy plugin import gemini
$ agy plugin import claude
# Validate plugin manifest and schema
$ agy plugin validate ./my-plugin
# Enable / Disable a plugin
$ agy plugin enable security-auditor
$ agy plugin disable security-auditor
# Uninstall plugin
$ agy plugin uninstall security-auditor
```
### Plugin Structure
```
my-plugin/
├── plugin.json # Manifest (name, version, entrypoints)
├── skills/ # Custom agent skills
│ └── audit/SKILL.md
├── rules/ # Custom rules and guidelines
│ └── security.md
└── mcp_config.json # Bundled MCP server definitions
```
---
## Configuration & Brain Storage {.cols-2}
### Configuration Locations {.row-span-2}
| Scope | Path | Purpose |
| --- | --- | --- |
| **CLI Settings** | `~/.gemini/antigravity-cli/settings.json` | Global CLI preferences, default model, reasoning |
| **Global Config** | `~/.gemini/config/` | User-wide skills, rules, and plugins |
| **Workspace Config** | `.agents/` or `.gemini/` | Project-scoped rules and agent configurations |
| **Sessions & Brain** | `~/.gemini/antigravity-cli/brain/<session-id>/` | Transcripts, tasks, and generated artifacts |
### Example `settings.json`
```json
{
"model": "gemini-3.8-flash-high",
"effort": "high",
"mode": "accept-edits",
"telemetry": false
}
```
### Transcripts & Artifacts
- Transcripts stored in `brain/<id>/.system_generated/logs/transcript.jsonl`.
- Markdown artifacts and diff reports stored in `brain/<id>/`.
---
## Also see {.cols-1}
- [Official Antigravity Documentation](https://antigravity.google/docs) _(antigravity.google)_
- [CLI Reference Guide](https://antigravity.google/docs/cli/reference) _(antigravity.google)_
- [CLI Features & Subagents](https://antigravity.google/docs/cli/features) _(antigravity.google)_
- [Skills Documentation](https://antigravity.google/docs/skills) _(antigravity.google)_
- [Rules & Guidelines Guide](https://antigravity.google/docs/rules-workflows) _(antigravity.google)_
- [Model Context Protocol (MCP)](https://antigravity.google/docs/mcp) _(antigravity.google)_
- [Antigravity Python SDK](https://github.com/google-antigravity/antigravity-sdk-python) _(github.com)_
+6 -2
View File
@@ -50,9 +50,11 @@ NAME = "John" # => Error (about space)
### Comments
```bash
# This is an inline Bash comment.
# This is a signle-line (inline) Bash comment.
```
Single-line comments in Bash start with `#` and continue to the end of the line.
```bash
: '
This is a
@@ -61,7 +63,9 @@ in bash
'
```
Multi-line comments use `:'` to open and `'` to close
Multi-line comments use `:'` to open and `'` to close.
> **Note:** Bash does not have true multi-line comments. This works by passing a multi-line string to the no-op builtin `:` (colon), which ignores its argument.
### Arguments {.row-span-2}
-15
View File
@@ -1220,13 +1220,11 @@ void main() {
```c{8}
#include <stdio.h>
#include <conio.h>
void main() {
FILE *fp;
char c;
clrscr();
fp = fopen("myfile.txt", "r");
@@ -1235,7 +1233,6 @@ void main() {
}
fclose(fp);
getch();
}
```
@@ -1243,18 +1240,15 @@ void main() {
```c {8}
#include<stdio.h>
#include<conio.h>
void main() {
FILE *fp;
clrscr();
fp = fopen("myfile2.txt","w");
fputs("hello c programming",fp);
fclose(fp);
getch();
}
```
@@ -1262,19 +1256,16 @@ void main() {
```c {10}
#include<stdio.h>
#include<conio.h>
void main() {
FILE *fp;
char text[300];
clrscr();
fp = fopen("myfile2.txt", "r");
printf("%s", fgets(text, 200, fp));
fclose(fp);
getch();
}
```
@@ -1303,13 +1294,11 @@ Set the file pointer to the given position
```c{11}
#include <stdio.h>
#include <conio.h>
void main() {
FILE *fp;
char c;
clrscr();
fp = fopen("file.txt", "r");
@@ -1324,7 +1313,6 @@ void main() {
}
fclose(fp);
getch();
}
// output
// Hello World! Hello World!
@@ -1334,13 +1322,11 @@ void main() {
```c{11}
#include <stdio.h>
#include <conio.h>
void main () {
FILE *fp;
int length;
clrscr();
fp = fopen("file.txt", "r");
@@ -1350,7 +1336,6 @@ void main () {
printf("File size: %d bytes", length);
getch();
}
// output
// file size: 18 bytes
+7 -2
View File
@@ -908,14 +908,19 @@ Provide key details about French Revolution
Explain history of Byzantine Empire
```
Science Reseearch
Science Research
```{.wrap}
Explain quantum computing in simple terms
```
Verified University Papers
```{.wrap}
Explain rocket science like i'm five
Get me top university research papers with verified direct links
```{.wrap}
Explain rocket science like I'm five
```
### SEO
+2 -2
View File
@@ -153,13 +153,13 @@ $ git branch -av
Switch to my_branch, and update working directory
```shell script
$ git checkout my_branch
$ git switch my_branch
```
Create a new branch called new_branch
```shell script
$ git checkout -b new_branch
$ git switch -c new_branch
```
Delete the branch called my_branch
+306 -175
View File
@@ -1,38 +1,58 @@
---
title: 'Matplotlib'
date: 06-06-2025
title: Matplotlib
date: 2026-07-19
background: bg-[#1a073e]
tags:
- 'data visualization'
- 'matplotlib'
- 'plotting'
- data visualization
- matplotlib
- plotting
categories:
- 'Python'
- Python
intro: |
[Matplotlib](https://matplotlib.org/) is a comprehensive library for creating static, animated, and interactive plots in Python.
This cheat sheet provides a quick reference from basic to advanced usage, covering essential features for data science, machine learning, and scientific computing.
[Matplotlib](https://matplotlib.org/) is a comprehensive library for creating static, animated, and interactive plots in Python.
This cheat sheet provides a production-grade reference for matplotlib ≥3.8 with emphasis on publication-quality figures, OOP interface, and performance optimization.
plugins:
- copyCode
---
---
## Getting Started {.cols-2}
### Importing
### Import Convention
```python
import matplotlib.pyplot as plt # Core plotting library
import numpy as np # For numerical operations
import matplotlib.pyplot as plt
import numpy as np
# For high-quality figures
plt.rcParams['figure.dpi'] = 120
plt.rcParams['savefig.dpi'] = 300
plt.rcParams['font.size'] = 10
plt.rcParams['axes.labelsize'] = 12
plt.rcParams['axes.titlesize'] = 14
plt.rcParams['legend.fontsize'] = 10
```
### Two Interfaces
| Description | Code |
|-------------|------|
| Pyplot (procedural) - quick | `plt.plot(x, y); plt.show()` |
| OOP (explicit) - production | `fig, ax = plt.subplots(); ax.plot(x, y); fig.savefig('fig.pdf')` |
| Create figure and axes | `fig, ax = plt.subplots(figsize=(8, 6))` |
| Create figure with grid | `fig, axes = plt.subplots(2, 2, figsize=(10, 8))` |
| Get current figure/axes | `fig = plt.gcf(); ax = plt.gca()` |
### Basic Plot
```python
x = np.linspace(0, 10, 100) # 100 points between 0 and 10
y = np.sin(x) # Sine function values
plt.plot(x, y) # Create a line plot
plt.show() # Display the plot
x = np.linspace(0, 10, 100)
y = np.sin(x)
fig, ax = plt.subplots()
ax.plot(x, y)
ax.set_title("Sine Wave")
ax.set_xlabel("x")
ax.set_ylabel("sin(x)")
plt.show()
```
---
@@ -41,226 +61,337 @@ plt.show() # Display the plot
### Line Plot
```python
plt.plot(x, y) # Line plot of y vs x
plt.title("Sine Wave") # Set title
plt.xlabel("x-axis") # Label x-axis
plt.ylabel("y-axis") # Label y-axis
plt.grid(True) # Show gridlines
plt.show()
```
| Description | Code |
|-------------|------|
| Simple line plot | `ax.plot(x, y)` |
| With custom style | `ax.plot(x, y, 'r--o', linewidth=2, markersize=4)` |
| Multiple series | `ax.plot(x, y1, label='sin'); ax.plot(x, y2, label='cos'); ax.legend()` |
| Step plot | `ax.step(x, y, where='mid')` |
| Stacked lines | `ax.stackplot(x, y1, y2, labels=['A','B'])` |
| Fill between | `ax.fill_between(x, y1, y2, alpha=0.3)` |
### Scatter Plot
```python
plt.scatter(x, y) # Scatter plot
plt.title("Scatter Plot")
plt.show()
```
| Description | Code |
|-------------|------|
| Basic scatter | `ax.scatter(x, y)` |
| Scatter with sizes | `ax.scatter(x, y, s=sizes, c=colors, alpha=0.6, cmap='viridis')` |
| Scatter with categorical colors | `ax.scatter(x, y, c=labels, cmap='Set1', edgecolors='black')` |
| 3D scatter | `ax.scatter3D(x, y, z, c=z, cmap='viridis')` |
### Bar Plot
### Bar Charts
```python
categories = ['A', 'B', 'C']
values = [10, 20, 15]
plt.bar(categories, values) # Vertical bar chart
plt.title("Bar Chart")
plt.show()
```
| Description | Code |
|-------------|------|
| Vertical bar | `ax.bar(categories, values, width=0.8, color='steelblue')` |
| Horizontal bar | `ax.barh(categories, values, height=0.8)` |
| Grouped bar | `ax.bar(x - width/2, y1, width, label='A'); ax.bar(x + width/2, y2, width, label='B')` |
| Stacked bar | `ax.bar(x, y1, label='A'); ax.bar(x, y2, bottom=y1, label='B')` |
| Error bars | `ax.bar(x, y, yerr=errors, capsize=5)` |
### Horizontal Bar Plot
### Histograms
```python
plt.barh(categories, values) # Horizontal bar chart
plt.title("Horizontal Bar Chart")
plt.show()
```
| Description | Code |
|-------------|------|
| Basic histogram | `ax.hist(data, bins=30, edgecolor='black')` |
| Multiple histograms | `ax.hist([data1, data2], bins=30, label=['A','B'], alpha=0.6)` |
| Density (normalized) | `ax.hist(data, bins=30, density=True, histtype='step')` |
| Cumulative | `ax.hist(data, bins=30, cumulative=True, histtype='step')` |
| 2D histogram | `ax.hist2d(x, y, bins=30, cmap='viridis')` |
### Histogram
### Pie Charts
```python
data = np.random.randn(1000) # Random normal distribution
plt.hist(data, bins=30) # Histogram with 30 bins
plt.title("Histogram")
plt.show()
```
| Description | Code |
|-------------|------|
| Basic pie | `ax.pie(sizes, labels=labels, autopct='%1.1f%%')` |
| Exploded pie | `ax.pie(sizes, labels=labels, explode=(0, 0.1, 0, 0), shadow=True)` |
| Donut chart | `ax.pie(sizes, labels=labels, wedgeprops=dict(width=0.3))` |
| Custom colors | `ax.pie(sizes, labels=labels, colors=plt.cm.Set3(range(len(sizes))))` |
### Pie Chart
### Box Plots
```python
sizes = [25, 35, 20, 20]
labels = ['A', 'B', 'C', 'D']
plt.pie(sizes, labels=labels, autopct='%1.1f%%') # Pie chart with % labels
plt.title("Pie Chart")
plt.show()
```
| Description | Code |
|-------------|------|
| Basic box plot | `ax.boxplot(data)` |
| Grouped box plots | `ax.boxplot([data1, data2, data3], labels=['A','B','C'])` |
| With notches | `ax.boxplot(data, notch=True, patch_artist=True)` |
| Violin plot | `ax.violinplot(data, showmeans=True, showmedians=True)` |
---
## Customization {.row-span-2}
| Feature | Code Example | Description |
| --------------- | ------------------------------------------------------------------------------------ | ------------------------------------------- |
| Title | `plt.title("Title")` | Set the title of the plot |
| X/Y Labels | `plt.xlabel("X")`, `plt.ylabel("Y")` | Label the axes |
| Grid | `plt.grid(True)` | Show grid |
| Legend | `plt.legend(["line1"])` | Add legend |
| Line Style | `plt.plot(x, y, linestyle='--')` | Dashed line |
| Color | `plt.plot(x, y, color='green')` | Set line color |
| Marker | `plt.plot(x, y, marker='o')` | Show markers on points |
| Axis Limits | `plt.xlim(0, 10)`, `plt.ylim(-1, 1)` | Set axis range |
| Tick Labels | `plt.xticks([...])`, `plt.yticks([...])` | Customize tick positions |
| Text Annotation | `plt.text(5, 0, "Midpoint")` | Add text at a specific coordinate |
| Arrow | `plt.annotate("Peak", xy=(7, 1), xytext=(6, 1.5), arrowprops=dict(arrowstyle="->"))` | Add annotation arrow |
| Style Sheets | `plt.style.use('ggplot')` | Use predefined styles like `seaborn`, `bmh` |
### Figure Elements
| Description | Code |
|-------------|------|
| Title | `ax.set_title("Title", fontsize=14, fontweight='bold')` |
| Axis labels | `ax.set_xlabel("X Label", fontsize=12); ax.set_ylabel("Y Label")` |
| Axis limits | `ax.set_xlim(0, 10); ax.set_ylim(-1.5, 1.5)` |
| Grid | `ax.grid(True, linestyle='--', alpha=0.7)` |
| Legend | `ax.legend(loc='best', frameon=True, fancybox=True, shadow=True)` |
| Legend outside | `ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left')` |
### Line & Marker Styles
| Description | Code |
|-------------|------|
| Linestyles | `'-'`, `'--'`, `'-.'`, `':'` |
| Markers | `'.'`, `'o'`, `'v'`, `'^'`, `'s'`, `'*'`, `'+'`, `'x'` |
| Color specification | `'r'`, `'#FF5733'`, `(0.2, 0.4, 0.6)`, `'red'` |
| Line width | `linewidth=2` or `lw=2` |
| Marker size | `markersize=6` or `ms=6` |
| Transparency | `alpha=0.5` |
### Text & Annotations
| Description | Code |
|-------------|------|
| Add text | `ax.text(x, y, "Text", fontsize=10, ha='center', va='bottom')` |
| Annotate with arrow | `ax.annotate("Peak", xy=(7, 1), xytext=(6, 1.5), arrowprops=dict(arrowstyle='->', color='red'))` |
| Math expressions | `ax.text(5, 0.5, r'$E = mc^2$', fontsize=14)` |
| Formula in title | `ax.set_title(r'$\sin(x) + \cos(x)$')` |
### Ticks & Scales
| Description | Code |
|-------------|------|
| Custom ticks | `ax.set_xticks([0, 2, 4, 6, 8, 10])` |
| Custom tick labels | `ax.set_xticklabels(['zero', 'two', 'four', 'six', 'eight', 'ten'])` |
| Rotate ticks | `plt.xticks(rotation=45)` |
| Log scale | `ax.set_xscale('log'); ax.set_yscale('log')` |
| Symlog | `ax.set_xscale('symlog', linthresh=0.01)` |
| Twin axes | `ax2 = ax.twinx(); ax2.plot(x, y2, 'r-')` |
### Colormaps
| Description | Code |
|-------------|------|
| Sequential | `'viridis'`, `'plasma'`, `'inferno'`, `'magma'`, `'cividis'` |
| Diverging | `'RdBu'`, `'RdBu_r'`, `'coolwarm'`, `'seismic'` |
| Qualitative | `'Set1'`, `'Set2'`, `'Set3'`, `'tab10'`, `'tab20'` |
| Reversed | `'viridis_r'` |
| Use colormap | `plt.cm.viridis(np.linspace(0, 1, 10))` |
### Stylesheets
| Description | Code |
|-------------|------|
| Default | `plt.style.use('default')` |
| Seaborn | `plt.style.use('seaborn-v0_8')` |
| FiveThirtyEight | `plt.style.use('fivethirtyeight')` |
| GGPlot | `plt.style.use('ggplot')` |
| Dark background | `plt.style.use('dark_background')` |
| Publication | `plt.style.use('bmh')` |
| Available styles | `print(plt.style.available)` |
---
## Subplots & Layouts
## Subplots & Layouts {.row-span-2}
### Multiple Subplots
### Creating Subplots
```python
fig, axs = plt.subplots(2, 2) # Create 2x2 grid of subplots
axs[0, 0].plot(x, y) # Top-left subplot
axs[0, 1].scatter(x, y) # Top-right subplot
axs[1, 0].bar(categories, values) # Bottom-left subplot
axs[1, 1].hist(data) # Bottom-right subplot
plt.tight_layout() # Adjust spacing to prevent overlap
plt.show()
```
| Description | Code |
|-------------|------|
| 2x2 grid | `fig, axes = plt.subplots(2, 2, figsize=(10, 8))` |
| Access subplot | `axes[0, 0].plot(x, y)` |
| Flatten axes array | `axes_flat = axes.flatten()` |
| Share x-axis | `fig, axes = plt.subplots(2, 1, sharex=True)` |
| Share y-axis | `fig, axes = plt.subplots(2, 1, sharey=True)` |
| Add subplot manually | `ax = fig.add_subplot(2, 2, 1)` |
| GridSpec | `gs = fig.add_gridspec(2, 2, width_ratios=[1, 2], height_ratios=[1, 2])` |
| Span multiple cells | `ax = fig.add_subplot(gs[0, :])` |
### Figure Size
### Layout Adjustments
```python
plt.figure(figsize=(10, 5)) # Set figure size (width, height in inches)
```
| Description | Code |
|-------------|------|
| Automatic spacing | `plt.tight_layout(pad=1.5)` |
| Fine-tune spacing | `plt.subplots_adjust(left=0.1, right=0.9, top=0.9, bottom=0.1, wspace=0.3, hspace=0.3)` |
| Constrained layout | `fig, axes = plt.subplots(constrained_layout=True)` |
| Colorbar | `fig.colorbar(im, ax=axes, orientation='vertical', fraction=0.02, pad=0.02)` |
### Figure Size & DPI
| Description | Code |
|-------------|------|
| Set figure size | `fig = plt.figure(figsize=(8, 6))` |
| Set DPI | `fig.set_dpi(150)` |
| Auto-figure size | `fig.set_size_inches(8, 6)` |
| Save with DPI | `fig.savefig('fig.png', dpi=300, bbox_inches='tight')` |
---
## Advanced Visualizations {.cols-2}
### Heatmap
### Image & Heatmap
```python
data = np.random.rand(10, 10) # Random 10x10 matrix
plt.imshow(data, cmap='hot', interpolation='nearest') # Display as image
plt.colorbar() # Show color scale
plt.title("Heatmap")
plt.show()
```
| Description | Code |
|-------------|------|
| Display image | `ax.imshow(image, cmap='gray', interpolation='bilinear')` |
| Heatmap | `im = ax.imshow(matrix, cmap='hot', aspect='auto')` |
| Colorbar | `fig.colorbar(im, ax=ax, label='Intensity')` |
| Hide axes | `ax.axis('off')` |
| Aspect ratio | `ax.set_aspect('equal')` |
### Contour Plot
### Contour & Surface
```python
X, Y = np.meshgrid(x, x)
Z = np.sin(X) * np.cos(Y)
plt.contour(X, Y, Z) # Contour lines
plt.title("Contour Plot")
plt.show()
```
| Description | Code |
|-------------|------|
| Contour lines | `contour = ax.contour(X, Y, Z, levels=20, cmap='viridis')` |
| Filled contour | `ax.contourf(X, Y, Z, levels=20, cmap='viridis')` |
| Contour labels | `ax.clabel(contour, inline=True, fontsize=8)` |
| 3D surface | `ax.plot_surface(X, Y, Z, cmap='viridis', alpha=0.8)` |
| 3D wireframe | `ax.plot_wireframe(X, Y, Z, color='black', alpha=0.3)` |
### 3D Plot
### 3D Plots
```python
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d') # 3D subplot
ax.plot3D(x, y, np.cos(x)) # 3D line
plt.title("3D Plot")
plt.show()
```
| Description | Code |
|-------------|------|
| Setup 3D axes | `from mpl_toolkits.mplot3d import Axes3D; fig = plt.figure(); ax = fig.add_subplot(111, projection='3d')` |
| 3D line | `ax.plot3D(x, y, z)` |
| 3D scatter | `ax.scatter3D(x, y, z, c=z, cmap='viridis')` |
| 3D bar | `ax.bar3d(x, y, z, dx, dy, dz)` |
| Set 3D view | `ax.view_init(elev=30, azim=45)` |
### Specialized Plots
| Description | Code |
|-------------|------|
| Error bars | `ax.errorbar(x, y, yerr=errors, xerr=x_errors, fmt='o', capsize=5)` |
| Stem plot | `ax.stem(x, y)` |
| Event plot | `ax.eventplot(data, lineoffsets=1, linelengths=0.5)` |
| Stream plot | `ax.streamplot(X, Y, U, V, density=1.5)` |
| Quiver (vector field) | `ax.quiver(x, y, u, v, scale=0.5)` |
| Polar plot | `ax = fig.add_subplot(111, projection='polar'); ax.plot(theta, r)` |
---
## Working with Images
```python
import matplotlib.image as mpimg
img = mpimg.imread('image.jpg') # Load image
plt.imshow(img) # Display image
plt.axis('off') # Hide axes
plt.title("Image Display")
plt.show()
```
| Description | Code |
|-------------|------|
| Load image | `import matplotlib.image as mpimg; img = mpimg.imread('image.jpg')` |
| Display image | `ax.imshow(img)` |
| Save image | `mpimg.imsave('output.jpg', img, dpi=300)` |
| Resize with interpolation | `ax.imshow(img, interpolation='bicubic')` |
| RGB channels | `red = img[:,:,0]; green = img[:,:,1]; blue = img[:,:,2]` |
---
## For Machine Learning
### Plotting Loss vs Epoch
### Training Curves
```python
epochs = range(1, 11)
loss = [0.9, 0.7, 0.5, 0.4, 0.3, 0.25, 0.2, 0.18, 0.15, 0.13]
plt.plot(epochs, loss)
plt.title("Training Loss")
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.show()
```
| Description | Code |
|-------------|------|
| Loss vs epochs | `ax.plot(epochs, train_loss, label='Train'); ax.plot(epochs, val_loss, label='Validation')` |
| Learning rate schedule | `ax.semilogy(epochs, learning_rates)` |
| Accuracy | `ax.plot(epochs, train_acc, 'b-', label='Train'); ax.plot(epochs, val_acc, 'r-', label='Val')` |
| Multiple metrics | `ax1 = ax.twinx(); ax1.plot(epochs, loss, 'r-')` |
### Confusion Matrix (Heatmap)
### Confusion Matrix
```python
from sklearn.metrics import confusion_matrix
import seaborn as sns
| Description | Code |
|-------------|------|
| Confusion matrix | `from sklearn.metrics import confusion_matrix; cm = confusion_matrix(y_true, y_pred)` |
| Heatmap | `im = ax.imshow(cm, cmap='Blues', interpolation='nearest')` |
| Annotate values | `for i in range(cm.shape[0]): for j in range(cm.shape[1]): ax.text(j, i, cm[i, j], ha='center', va='center')` |
| Normalized | `cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]` |
y_true = [0, 1, 2, 2, 0]
y_pred = [0, 0, 2, 2, 1]
cm = confusion_matrix(y_true, y_pred) # Create confusion matrix
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues') # Heatmap visualization
plt.title("Confusion Matrix")
plt.show()
```
### ROC & Precision-Recall
### ROC Curve
| Description | Code |
|-------------|------|
| ROC curve | `from sklearn.metrics import roc_curve, auc; fpr, tpr, _ = roc_curve(y_true, y_scores); auc_val = auc(fpr, tpr)` |
| ROC plot | `ax.plot(fpr, tpr, label=f'AUC = {auc_val:.2f}'); ax.plot([0,1], [0,1], 'k--')` |
| Precision-Recall | `from sklearn.metrics import precision_recall_curve; precision, recall, _ = precision_recall_curve(y_true, y_scores)` |
| PR plot | `ax.plot(recall, precision, label='PR Curve')` |
```python
from sklearn.metrics import roc_curve, auc
### Feature Analysis
fpr, tpr, _ = roc_curve([0, 0, 1, 1], [0.1, 0.4, 0.35, 0.8]) # Compute ROC
roc_auc = auc(fpr, tpr) # Area under curve
plt.plot(fpr, tpr, label=f'AUC = {roc_auc:.2f}')
plt.plot([0, 1], [0, 1], 'k--') # Diagonal line
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve")
plt.legend()
plt.show()
```
| Description | Code |
|-------------|------|
| Feature importance | `ax.barh(feature_names, importances, color='steelblue')` |
| Correlation matrix | `im = ax.imshow(corr_matrix, cmap='RdBu_r', vmin=-1, vmax=1)` |
| Pairwise scatter | `import seaborn as sns; sns.pairplot(df)` |
| Parallel coordinates | `from pandas.plotting import parallel_coordinates; parallel_coordinates(df, 'class')` |
---
## Saving Plots
## Animations
```python
plt.savefig("figure.png", dpi=300, bbox_inches='tight') # Save plot to file
```
| Description | Code |
|-------------|------|
| Basic animation | `from matplotlib.animation import FuncAnimation; anim = FuncAnimation(fig, update, frames=100, interval=50)` |
| Save animation | `anim.save('animation.gif', writer='pillow', fps=20)` |
| Update function | `def update(frame): ax.clear(); ax.plot(x[:frame], y[:frame]); return ax,` |
| Interactive animation | `plt.ion(); fig, ax = plt.subplots(); ax.plot(x, y); plt.pause(0.01); plt.ioff()` |
---
## Saving & Exporting
### File Formats
| Description | Code |
|-------------|------|
| PNG (high DPI) | `fig.savefig('figure.png', dpi=300, bbox_inches='tight')` |
| PDF (vector) | `fig.savefig('figure.pdf', bbox_inches='tight')` |
| SVG (vector) | `fig.savefig('figure.svg', bbox_inches='tight')` |
| EPS (vector) | `fig.savefig('figure.eps', bbox_inches='tight')` |
| Transparent background | `fig.savefig('figure.png', transparent=True)` |
| With metadata | `fig.savefig('figure.pdf', metadata={'Author': 'Name', 'Title': 'Plot'})` |
### Publication Quality
| Description | Code |
|-------------|------|
| High-res for paper | `plt.rcParams['figure.dpi'] = 300` |
| Fonts for publication | `plt.rcParams['font.family'] = 'serif'; plt.rcParams['font.serif'] = ['Times New Roman']` |
| Multiple figures | `fig.savefig('figure.pdf', bbox_inches='tight', pad_inches=0.05)` |
| Size constraints | `fig = plt.figure(figsize=(3.4, 2.4))` (single column) |
---
## Show & Clear
```python
plt.show() # Show plot window
plt.clf() # Clear current figure (useful when plotting in loops)
plt.close() # Close figure window (useful in scripts or GUI apps)
```
| Description | Code |
|-------------|------|
| Display plot | `plt.show()` |
| Clear current figure | `plt.clf()` |
| Clear current axes | `plt.cla()` |
| Close all figures | `plt.close('all')` |
| Close specific figure | `plt.close(fig)` |
| Suppress display | `plt.ioff()` |
---
## More Useful Functions
## Performance & Memory
| Function | Use Case |
| ------------------------------ | ------------------------------- |
| `plt.fill_between(x, y1, y2)` | Fill area between curves |
| `plt.axhline(y=value)` | Draw horizontal line at y |
| `plt.axvline(x=value)` | Draw vertical line at x |
| `plt.errorbar(x, y, yerr=...)` | Plot with error bars |
| `plt.twinx()` | Create secondary y-axis |
| `plt.subplots_adjust(...)` | Manually adjust subplot spacing |
| `plt.gca()` | Get current axes |
| `plt.gcf()` | Get current figure |
| Description | Code |
|-------------|------|
| Fast plotting (AGG backend) | `import matplotlib; matplotlib.use('Agg')` |
| Interactive backend | `import matplotlib; matplotlib.use('TkAgg')` |
| Reduce memory | `ax.plot(x[::10], y[::10])` (downsample) |
| Efficient scatter | `ax.scatter(x, y, s=1)` (small markers) |
| Disable antialiasing | `ax.plot(x, y, antialiased=False)` |
| Batch plotting | `fig, axes = plt.subplots(2, 2); for ax in axes.flat: ax.plot(x, y)` |
---
## Common Pitfalls & Best Practices
| Pitfall | Solution |
|---------|----------|
| Mixing pyplot and OOP | Stick to one interface; prefer OOP for production |
| Overlapping legends | Place legend outside: `ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left')` |
| Small fonts in publication | Set `plt.rcParams['font.size'] = 12` globally |
| Slow rendering | Downsample data or use `agg` backend |
| Figures cut off | Use `bbox_inches='tight'` in savefig |
| Memory leaks in loops | Use `plt.clf()` or `plt.close()` after each iteration |
| Inconsistent colormaps | Use same colormap for comparable visualizations |
| Missing axis labels | Always label for reproducibility |
| 3D plot overlapping | Adjust view with `ax.view_init()` |
---
+443 -112
View File
@@ -1,148 +1,479 @@
---
title: Numpy
date: 2020-12-14 18:28:43
title: NumPy
date: 2026-07-19
background: bg-[#4f6fc3]
tags:
- scientific
- computing
- scientific computing
- numerical analysis
- array processing
categories:
- Python
intro: |
[NumPy](https://numpy.org/) is the fundamental package for scientific computing with Python. This cheat sheet is a quick reference for NumPy beginners.
[NumPy](https://numpy.org/) is the fundamental package for scientific computing with Python. This cheat sheet provides a production-grade reference for NumPy ≥2.0 with emphasis on performance, memory efficiency, and advanced array operations.
plugins:
- copyCode
---
## Getting Started {.cols-2}
### Introduction
You’ll also need to import numpy to get started:
### Import Convention
```python
import numpy as np
# For performance-critical code
np.set_printoptions(precision=4, suppress=True, linewidth=120)
# Enable error handling for floating point
np.seterr(divide='raise', over='raise', under='ignore', invalid='raise')
```
### Importing/exporting
### Array Creation
| - | - |
| ------------------------------------------ | --------------------- |
| `np.loadtxt('file.txt')` | From a text file |
| `np.genfromtxt('file.csv',delimiter=',')` | From a CSV file |
| `np.savetxt('file.txt',arr,delimiter=' ')` | Writes to a text file |
| `np.savetxt('file.csv',arr,delimiter=',')` | Writes to a CSV file |
| Description | Code |
|-------------|------|
| From Python list | `np.array([1, 2, 3])` |
| From Python list of lists | `np.array([(1, 2, 3), (4, 5, 6)])` |
| Zeros array | `np.zeros((3, 4), dtype=np.float64)` |
| Ones array | `np.ones((2, 3), dtype=np.int64)` |
| Identity matrix | `np.eye(5, dtype=np.float32)` |
| Diagonal array | `np.diag([1, 2, 3])` |
| Full array | `np.full((2, 3), 8, dtype=np.float64)` |
| Empty (uninitialized) | `np.empty((3, 3), dtype=np.float64)` |
| Evenly spaced range | `np.linspace(0, 100, 6, dtype=np.float64)` |
| Arithmetic range | `np.arange(0, 10, 3)` |
| Log-spaced | `np.logspace(0, 3, 4, base=10)` |
| From function | `np.fromfunction(lambda i, j: i + j, (3, 3))` |
| From string | `np.fromstring('1 2 3', dtype=int, sep=' ')` |
### Creating Arrays {.row-span-2}
### Random Arrays
| - | - |
| --------------------------------- | ----------------------------------------------------------------- |
| `np.array([1,2,3])` | One dimensional array |
| `np.array([(1,2,3),(4,5,6)])` | Two dimensional array |
| `np.zeros(3)` | 1D array of length 3 all values 0 |
| `np.ones((3,4))` | 3x4 array with all values 1 |
| `np.eye(5)` | 5x5 array of 0 with 1 on diagonal (Identity matrix) |
| `np.linspace(0,100,6)` | Array of 6 evenly divided values from 0 to 100 |
| `np.arange(0,10,3)` | Array of values from 0 to less than 10 with step 3 (eg [0,3,6,9]) |
| `np.full((2,3),8)` | 2x3 array with all values 8 |
| `np.random.rand(4,5)` | 4x5 array of random floats between 0–1 |
| `np.random.rand(6,7)*100` | 6x7 array of random floats between 0–100 |
| `np.random.randint(5,size=(2,3))` | 2x3 array with random ints between 0–4 |
| Description | Code |
|-------------|------|
| Uniform [0,1) | `np.random.rand(4, 5)` |
| Uniform [low, high) | `np.random.uniform(0, 100, (4, 5))` |
| Standard normal | `np.random.randn(4, 5)` |
| Normal (mean, std) | `np.random.normal(0, 1, (4, 5))` |
| Integers [low, high) | `np.random.randint(0, 10, (2, 3))` |
| Random choice | `np.random.choice([1, 2, 3, 4, 5], size=3, replace=False)` |
| Shuffle in-place | `np.random.shuffle(arr)` |
| Permutation | `np.random.permutation(10)` |
| Random seed | `np.random.seed(42)` |
| Generator (new API) | `rng = np.random.default_rng(seed=42); rng.random((3, 3))` |
### Inspecting Properties
---
| - | - |
| ------------------- | ---------------------------------------- |
| `arr.size` | Returns number of elements in arr |
| `arr.shape` | Returns dimensions of arr (rows,columns) |
| `arr.dtype` | Returns type of elements in arr |
| `arr.astype(dtype)` | Convert arr elements to type dtype |
| `arr.tolist()` | Convert arr to a Python list |
| `np.info(np.eye)` | View documentation for np.eye |
## Array Properties {.row-span-2}
### Copying/sorting/reshaping
### Attributes
| - | - |
| --------------------- | ------------------------------------------------------- |
| `np.copy(arr)` | Copies arr to new memory |
| `arr.view(dtype)` | Creates view of arr elements with type dtype |
| `arr.sort()` | Sorts arr |
| `arr.sort(axis=0)` | Sorts specific axis of arr |
| `two_d_arr.flatten()` | Flattens 2D array two_d_arr to 1D |
| `arr.T` | Transposes arr (rows become columns and vice versa) |
| `arr.reshape(3,4)` | Reshapes arr to 3 rows, 4 columns without changing data |
| `arr.resize((5,6))` | Changes arr shape to 5x6 and fills new values with 0 |
| Description | Code |
|-------------|------|
| Shape (dimensions) | `arr.shape` |
| Number of elements | `arr.size` |
| Number of dimensions | `arr.ndim` |
| Data type | `arr.dtype` |
| Item size (bytes) | `arr.itemsize` |
| Total memory (bytes) | `arr.nbytes` |
| Strides (memory layout) | `arr.strides` |
| Base (view check) | `arr.base` is not None |
| C-contiguous | `arr.flags['C_CONTIGUOUS']` |
| F-contiguous | `arr.flags['F_CONTIGUOUS']` |
### Adding/removing Elements
### Type Conversion
| - | - |
| ------------------------- | -------------------------------------- |
| `np.append(arr,values)` | Appends values to end of arr |
| `np.insert(arr,2,values)` | Inserts values into arr before index 2 |
| `np.delete(arr,3,axis=0)` | Deletes row on index 3 of arr |
| `np.delete(arr,4,axis=1)` | Deletes column on index 4 of arr |
| Description | Code |
|-------------|------|
| Convert dtype | `arr.astype(np.float32)` |
| To Python list | `arr.tolist()` |
| To bytes | `arr.tobytes()` |
| To file | `arr.tofile('data.bin')` |
| To string | `np.array2string(arr, precision=2)` |
| View as different dtype | `arr.view(np.int32)` |
### Combining/splitting
---
| - | - |
| ------------------------------------ | ---------------------------------------- |
| `np.concatenate((arr1,arr2),axis=0)` | Adds arr2 as rows to the end of arr1 |
| `np.concatenate((arr1,arr2),axis=1)` | Adds arr2 as columns to end of arr1 |
| `np.split(arr,3)` | Splits arr into 3 sub-arrays |
| `np.hsplit(arr,5)` | Splits arr horizontally on the 5th index |
## Indexing & Slicing {.row-span-2}
### Indexing/slicing/subsetting
### Basic Indexing
| - | - |
| --------------------- | ------------------------------------------------------------------------- |
| `arr[5]` | Returns the element at index 5 |
| `arr[2,5]` | Returns the 2D array element on index [2][5] |
| `arr[1]=4` | Assigns array element on index 1 the value 4 |
| `arr[1,3]=10` | Assigns array element on index [1][3] the value 10 |
| `arr[0:3]` | Returns the elements at indices 0,1,2 (On a 2D array: returns rows 0,1,2) |
| `arr[0:3,4]` | Returns the elements on rows 0,1,2 at column 4 |
| `arr[:2]` | Returns the elements at indices 0,1 (On a 2D array: returns rows 0,1) |
| `arr[:,1]` | Returns the elements at index 1 on all rows |
| `arr<5` | Returns an array with boolean values |
| `(arr1<3) & (arr2>5)` | Returns an array with boolean values |
| `~arr` | Inverts a boolean array |
| `arr[arr<5]` | Returns array elements smaller than 5 |
| Description | Code |
|-------------|------|
| Single element | `arr[5]` |
| 2D element | `arr[2, 5]` |
| Single assignment | `arr[1] = 4` |
| 2D assignment | `arr[1, 3] = 10` |
| Slice (rows 0,1,2) | `arr[0:3]` |
| Slice (rows 0,1,2, col 4) | `arr[0:3, 4]` |
| First two rows | `arr[:2]` |
| All rows, column 1 | `arr[:, 1]` |
| Row 0, all columns | `arr[0, :]` |
| Step slice | `arr[0:10:2]` |
| Negative indexing | `arr[-1]`, `arr[-3:]` |
| Ellipsis (multiple dims) | `arr[..., 1]` |
### Vector Math
### Boolean Indexing
| - | - |
| --------------------------- | ----------------------------------------------------------- |
| `np.add(arr1,arr2)` | Elementwise add arr2 to arr1 |
| `np.subtract(arr1,arr2)` | Elementwise subtract arr2 from arr1 |
| `np.multiply(arr1,arr2)` | Elementwise multiply arr1 by arr2 |
| `np.divide(arr1,arr2)` | Elementwise divide arr1 by arr2 |
| `np.power(arr1,arr2)` | Elementwise raise arr1 raised to the power of arr2 |
| `np.array_equal(arr1,arr2)` | Returns True if the arrays have the same elements and shape |
| `np.sqrt(arr)` | Square root of each element in the array |
| `np.sin(arr)` | Sine of each element in the array |
| `np.log(arr)` | Natural log of each element in the array |
| `np.abs(arr)` | Absolute value of each element in the array |
| `np.ceil(arr)` | Rounds up to the nearest int |
| `np.floor(arr)` | Rounds down to the nearest int |
| `np.round(arr)` | Rounds to the nearest int |
| Description | Code |
|-------------|------|
| Boolean mask | `arr < 5` |
| Multiple conditions | `(arr1 < 3) & (arr2 > 5)` |
| OR condition | `(arr1 < 3) \| (arr2 > 5)` |
| Invert | `~arr` |
| Filter with condition | `arr[arr < 5]` |
| Where (select) | `np.where(arr < 5, arr, 0)` |
| Where (indices) | `np.where(arr < 5)` |
| Extract | `np.extract(arr < 5, arr)` |
### Scalar Math
### Fancy Indexing
| - | - |
| -------------------- | -------------------------------------------------------------------- |
| `np.add(arr,1)` | Add 1 to each array element |
| `np.subtract(arr,2)` | Subtract 2 from each array element |
| `np.multiply(arr,3)` | Multiply each array element by 3 |
| `np.divide(arr,4)` | Divide each array element by 4 (returns np.nan for division by zero) |
| `np.power(arr,5)` | Raise each array element to the 5th power |
| Description | Code |
|-------------|------|
| Integer list | `arr[[1, 3, 5]]` |
| 2D integer list | `arr[[0, 1], [0, 1]]` |
| Using np.ix_ | `arr[np.ix_([0, 2], [0, 2])]` |
| Select with conditions | `arr[np.where(arr > 0)]` |
| Clip values | `np.clip(arr, 0, 1)` |
| Put values | `np.put(arr, [0, 2], [10, 20])` |
| Take values | `np.take(arr, [0, 2])` |
### Statistics
### Advanced Indexing
| Description | Code |
|-------------|------|
| Index as array | `arr[[0, 1, 2], [0, 1, 0]]` |
| Index with masks | `arr[arr > 0.5] = 1` |
| Complex condition | `arr[(arr > 0.2) & (arr < 0.8)]` |
| Choose from arrays | `np.choose(selector, [arr1, arr2])` |
| Diagonal selection | `np.diagonal(arr, offset=0)` |
| Trace | `np.trace(arr)` |
---
## Shape Operations {.row-span-2}
### Reshaping
| Description | Code |
|-------------|------|
| Reshape (new view if possible) | `arr.reshape(3, 4)` |
| Resize (in-place, may fill) | `arr.resize((5, 6))` |
| Flatten (C-order) | `arr.flatten()` |
| Flatten (F-order) | `arr.flatten('F')` |
| Ravel (view when possible) | `arr.ravel()` |
| Transpose | `arr.T` |
| Transpose with axes | `arr.transpose(1, 0, 2)` |
| Swap axes | `arr.swapaxes(0, 1)` |
| Move axis | `np.moveaxis(arr, 0, -1)` |
| Expand dims | `np.expand_dims(arr, axis=0)` |
| Squeeze (remove size 1 dims) | `np.squeeze(arr)` |
| Broadcast to shape | `np.broadcast_to(arr, (3, 5, 5))` |
### Adding/Removing Dimensions
| Description | Code |
|-------------|------|
| New axis (view) | `arr[np.newaxis, :]` |
| New axis (at end) | `arr[:, np.newaxis]` |
| Expand dims | `np.expand_dims(arr, 1)` |
| Remove singleton dims | `arr.squeeze(axis=0)` |
| Concatenate | `np.concatenate([arr1, arr2], axis=0)` |
| Stack (new axis) | `np.stack([arr1, arr2], axis=0)` |
| Vertical stack | `np.vstack([arr1, arr2])` |
| Horizontal stack | `np.hstack([arr1, arr2])` |
| Column stack | `np.column_stack([arr1, arr2])` |
| Tile (repeat) | `np.tile(arr, (2, 3))` |
| Repeat each element | `np.repeat(arr, 3, axis=0)` |
### Splitting
| Description | Code |
|-------------|------|
| Split into N equal parts | `np.split(arr, 3)` |
| Split by indices | `np.split(arr, [2, 5])` |
| Horizontal split | `np.hsplit(arr, 5)` |
| Vertical split | `np.vsplit(arr, 3)` |
| Depth split (3D) | `np.dsplit(arr, 3)` |
| Array split (unequal) | `np.array_split(arr, 3)` |
---
## Element-wise Operations
### Arithmetic
| Description | Code |
|-------------|------|
| Addition | `np.add(arr1, arr2)` or `arr1 + arr2` |
| Subtraction | `np.subtract(arr1, arr2)` or `arr1 - arr2` |
| Multiplication | `np.multiply(arr1, arr2)` or `arr1 * arr2` |
| Division | `np.divide(arr1, arr2)` or `arr1 / arr2` |
| Floor division | `np.floor_divide(arr1, arr2)` or `arr1 // arr2` |
| Power | `np.power(arr1, arr2)` or `arr1 ** arr2` |
| Modulo | `np.mod(arr1, arr2)` or `arr1 % arr2` |
| Negative | `np.negative(arr)` or `-arr` |
| Absolute | `np.abs(arr)` |
| Sign | `np.sign(arr)` |
### Mathematical Functions
| Description | Code |
|-------------|------|
| Square root | `np.sqrt(arr)` |
| Exponential | `np.exp(arr)` |
| Natural log | `np.log(arr)` |
| Log base 10 | `np.log10(arr)` |
| Log base 2 | `np.log2(arr)` |
| Sin/Cos/Tan | `np.sin(arr)`, `np.cos(arr)`, `np.tan(arr)` |
| Arcsin/Arccos/Arctan | `np.arcsin(arr)`, `np.arccos(arr)`, `np.arctan(arr)` |
| Hyperbolic | `np.sinh(arr)`, `np.cosh(arr)`, `np.tanh(arr)` |
| Rounding | `np.round(arr, decimals=2)` |
| Ceiling | `np.ceil(arr)` |
| Floor | `np.floor(arr)` |
| Truncate | `np.trunc(arr)` |
| Degrees to radians | `np.radians(arr)` |
| Radians to degrees | `np.degrees(arr)` |
### Comparison & Logic
| Description | Code |
|-------------|------|
| Equal | `np.equal(arr1, arr2)` or `arr1 == arr2` |
| Not equal | `np.not_equal(arr1, arr2)` or `arr1 != arr2` |
| Less than | `np.less(arr1, arr2)` or `arr1 < arr2` |
| Greater than | `np.greater(arr1, arr2)` or `arr1 > arr2` |
| Less/equal | `np.less_equal(arr1, arr2)` or `arr1 <= arr2` |
| Greater/equal | `np.greater_equal(arr1, arr2)` or `arr1 >= arr2` |
| Logical AND | `np.logical_and(arr1, arr2)` |
| Logical OR | `np.logical_or(arr1, arr2)` |
| Logical NOT | `np.logical_not(arr)` |
| Logical XOR | `np.logical_xor(arr1, arr2)` |
| Is finite | `np.isfinite(arr)` |
| Is nan | `np.isnan(arr)` |
| Is inf | `np.isinf(arr)` |
---
## Aggregation & Statistics
### Reductions
| Description | Code |
|-------------|------|
| Sum (all) | `np.sum(arr)` |
| Sum (axis) | `np.sum(arr, axis=0)` |
| Mean | `np.mean(arr)` |
| Mean (axis) | `np.mean(arr, axis=1)` |
| Median | `np.median(arr)` |
| Min/Max | `np.min(arr)`, `np.max(arr)` |
| Min/Max (axis) | `np.min(arr, axis=0)`, `np.max(arr, axis=1)` |
| Standard deviation | `np.std(arr)` |
| Variance | `np.var(arr)` |
| Percentile | `np.percentile(arr, 75)` |
| Quantile | `np.quantile(arr, 0.75)` |
| Cumulative sum | `np.cumsum(arr)` |
| Cumulative product | `np.cumprod(arr)` |
| Dot product | `np.dot(arr1, arr2)` |
| Inner product | `np.inner(arr1, arr2)` |
| Outer product | `np.outer(arr1, arr2)` |
### Statistical Functions
| Description | Code |
|-------------|------|
| Correlation coefficient | `np.corrcoef(arr1, arr2)` |
| Covariance | `np.cov(arr1, arr2)` |
| Cross-correlation | `np.correlate(arr1, arr2, mode='full')` |
| Convolution | `np.convolve(arr1, arr2, mode='full')` |
| Histogram | `np.histogram(arr, bins=10)` |
| Bincount | `np.bincount(arr.astype(int))` |
| Unique values | `np.unique(arr)` |
| Unique with counts | `np.unique(arr, return_counts=True)` |
| Count non-zero | `np.count_nonzero(arr)` |
| Any/All | `np.any(arr > 0)`, `np.all(arr > 0)` |
---
## Linear Algebra
### Matrix Operations
| Description | Code |
|-------------|------|
| Matrix multiplication | `np.matmul(A, B)` or `A @ B` |
| Dot product | `np.dot(A, B)` |
| Inner product | `np.inner(A, B)` |
| Outer product | `np.outer(A, B)` |
| Trace | `np.trace(A)` |
| Determinant | `np.linalg.det(A)` |
| Rank | `np.linalg.matrix_rank(A)` |
| Condition number | `np.linalg.cond(A)` |
### Matrix Decompositions
| Description | Code |
|-------------|------|
| Inverse | `np.linalg.inv(A)` |
| Pseudo-inverse | `np.linalg.pinv(A)` |
| Eigenvalues/vectors | `np.linalg.eig(A)` |
| Eigenvalues (only) | `np.linalg.eigvals(A)` |
| SVD | `np.linalg.svd(A)` |
| QR decomposition | `np.linalg.qr(A)` |
| LU decomposition | `np.linalg.lu(A)` |
| Cholesky decomposition | `np.linalg.cholesky(A)` |
### Solving Systems
| Description | Code |
|-------------|------|
| Solve linear system | `np.linalg.solve(A, b)` |
| Least squares | `np.linalg.lstsq(A, b)` |
| Norm (Euclidean) | `np.linalg.norm(A)` |
| Norm (specific order) | `np.linalg.norm(A, ord=1)` |
| Singular values | `np.linalg.svdvals(A)` |
---
## Broadcasting & Vectorization
### Broadcasting Rules
| Description | Code |
|-------------|------|
| Scalar + array | `arr + 1` |
| 1D + 2D | `arr2d + arr1d` |
| Matching trailing dims | `arr2d + arr1d.reshape(-1, 1)` |
| Check broadcast | `np.broadcast_shapes((2,3), (3,))` |
| Explicit broadcast | `np.broadcast_to(arr1d, (3, 4))` |
| Broadcasting in ufunc | `np.add(arr2d, arr1d)` |
### Vectorization Patterns
| Description | Code |
|-------------|------|
| Vectorized addition | `arr1 + arr2` (instead of loop) |
| Vectorized condition | `np.where(cond, arr1, arr2)` |
| Vectorized selection | `arr[mask]` |
| Vectorized accumulation | `np.cumsum(arr)` |
| Element-wise operations | `np.sqrt(arr)` |
| Reduce operations | `np.sum(arr)` |
| Universal functions | `np.exp(arr)`, `np.log(arr)` |
---
## Memory & Performance
### Memory Management
| Description | Code |
|-------------|------|
| View vs copy | `arr_view = arr[:]` (view); `arr_copy = arr.copy()` (copy) |
| Check ownership | `arr.base is None` (own memory) |
| Share memory check | `np.shares_memory(arr1, arr2)` |
| Memory alignment | `np.array(arr, copy=False, order='C')` |
| Memory layout | `np.ascontiguousarray(arr)` |
| Fortran order | `np.asfortranarray(arr)` |
| Byteswap | `arr.byteswap(inplace=True)` |
### Performance Optimization
| Description | Code |
|-------------|------|
| Preallocate | `np.empty((N,))` |
| Use local variables | `np_add = np.add` |
| Avoid Python loops | Use NumPy vectorized operations |
| Out-of-place operations | `np.add(arr1, arr2, out=result)` |
| Use in-place | `arr += 1` (less memory) |
| Reduce allocations | Reuse arrays with `out=` parameter |
| Use float32 if possible | `np.float32` instead of `np.float64` |
| Use memory-mapped files | `np.memmap('data.dat', dtype='float32', mode='r', shape=(1000, 1000))` |
---
## Input/Output {.cols-2}
### Text Files
| Description | Code |
|-------------|------|
| Load text | `np.loadtxt('file.txt', delimiter=' ')` |
| Load CSV | `np.genfromtxt('file.csv', delimiter=',', skip_header=1)` |
| Load with missing values | `np.genfromtxt('file.csv', delimiter=',', filling_values=0)` |
| Save text | `np.savetxt('file.txt', arr, delimiter=' ', fmt='%.4f')` |
| Save CSV | `np.savetxt('file.csv', arr, delimiter=',', fmt='%.4f')` |
### Binary Files
| Description | Code |
|-------------|------|
| Load .npy | `np.load('file.npy')` |
| Save .npy | `np.save('file.npy', arr)` |
| Load .npz (multiple arrays) | `data = np.load('file.npz'); data['arr1']` |
| Save .npz | `np.savez('file.npz', arr1=arr1, arr2=arr2)` |
| Save compressed .npz | `np.savez_compressed('file.npz', arr1=arr1, arr2=arr2)` |
| Binary to file | `arr.tofile('data.bin')` |
| Read from file | `np.fromfile('data.bin', dtype=np.float32)` |
| Memory-mapped | `arr = np.memmap('data.dat', dtype='float32', mode='r', shape=(100, 100))` |
### Data Types
| Description | Code |
|-------------|------|
| Integer types | `np.int8`, `np.int16`, `np.int32`, `np.int64` |
| Unsigned ints | `np.uint8`, `np.uint16`, `np.uint32`, `np.uint64` |
| Floating types | `np.float16`, `np.float32`, `np.float64`, `np.float128` |
| Complex types | `np.complex64`, `np.complex128` |
| Boolean | `np.bool_` |
| Object | `np.object_` |
| String | `np.string_`, `np.unicode_` |
| Datetime | `np.datetime64`, `np.timedelta64` |
| Structured dtypes | `np.dtype([('name', 'U10'), ('age', 'i4')])` |
---
## Structured Arrays
| Description | Code |
|-------------|------|
| Create structured | `np.array([(1, 'Alice'), (2, 'Bob')], dtype=[('id','i4'), ('name','U10')])` |
| Access by field | `arr['name']` |
| Access by position | `arr[0]` |
| Multiple fields | `arr[['id', 'name']]` |
| Complex dtype | `np.dtype({'names': ['id','name'], 'formats': ['i4','U10']})` |
| Recarray (by attribute) | `arr = np.rec.array([(1,'Alice'),(2,'Bob')], dtype=[('id','i4'), ('name','U10')]); arr.name` |
---
## Datetime & Timedelta
| Description | Code |
|-------------|------|
| Create datetime | `np.datetime64('2026-07-01')` |
| Array of datetimes | `np.arange('2026-01', '2026-07', dtype='datetime64[M]')` |
| Timedelta | `np.timedelta64(1, 'D')` |
| Arithmetic | `np.datetime64('2026-07-01') - np.datetime64('2026-06-01')` |
| Extract components | `dt.astype('datetime64[Y]')` (year) |
| Day of week | `(dt.astype('datetime64[D]') - np.datetime64('1970-01-01')).astype('int64') % 7` |
---
## Common Pitfalls & Best Practices
| Pitfall | Solution |
|---------|----------|
| Copy vs view confusion | Use `arr.copy()` explicitly for independent copy |
| Mutation of views | Check with `arr.base is not None` |
| Python loops over arrays | Use vectorized operations instead of loops |
| Using int64 unnecessarily | Use `np.float32` or `np.int32` for memory efficiency |
| Mutable default arguments | Avoid `arr=[]` in functions; use `None` |
| Not specifying dtype | Specify `dtype` for precision control |
| Using `np.append` in loops | Preallocate array for better performance |
| Shape mismatch in broadcasting | Check shapes with `arr.shape` |
| Floating point comparison | Use `np.isclose()` for comparison of floats |
| Memory leak in large arrays | Use `del arr` or `arr = None` after use |
| Inconsistent random seeding | Use `np.random.seed(42)` for reproducibility |
| Using old random API | Use new `np.random.default_rng()` API |
---
| - | - |
| --------------------- | ----------------------------------------------- |
| `np.mean(arr,axis=0)` | Returns mean along specific axis |
| `arr.sum()` | Returns sum of arr |
| `arr.min()` | Returns minimum value of arr |
| `arr.max(axis=0)` | Returns maximum value of specific axis |
| `np.var(arr)` | Returns the variance of array |
| `np.std(arr,axis=1)` | Returns the standard deviation of specific axis |
| `arr.corrcoef()` | Returns correlation coefficient of array |
+297 -73
View File
@@ -1,6 +1,7 @@
---
title: Pandas
date: 2024-07-04
date: 2026-07-19
background: bg-[#110750]
tags:
- data analysis
@@ -8,129 +9,352 @@ tags:
categories:
- Python
intro: |
[Pandas](https://pandas.pydata.org/) is a powerful data analysis and manipulation library for Python. This cheat sheet is a quick reference for Pandas beginners.
[Pandas](https://pandas.pydata.org/) is a powerful data analysis and manipulation library for Python. This cheat sheet provides a comprehensive reference for modern pandas (≥2.2) with emphasis on performance, nullable dtypes, and PyArrow integration.
plugins:
- copyCode
---
## Getting Started {.cols-2}
### Introduction
You’ll need to import pandas to get started:
### Import Convention
```python
import pandas as pd
import numpy as np
# Enable future features (Pandas 2.2+)
pd.options.future.infer_string = True # Arrow-backed strings
pd.options.mode.copy_on_write = True # Lazy copying, memory efficient
pd.set_option('display.max_columns', None) # Full display in notebooks
# Optional: suppress performance warnings
import warnings
warnings.filterwarnings('ignore', category=pd.errors.PerformanceWarning)
```
### Creating DataFrames
| - | - |
| --------------------------------------------------------- | --------------------------- |
| `pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})` | From a dictionary |
| `pd.DataFrame(data=[{'a': 1, 'b': 2}, {'a': 3, 'b': 4}])` | From a list of dictionaries |
| `pd.read_csv('file.csv')` | From a CSV file |
| `pd.read_excel('file.xlsx')` | From an Excel file |
| From a dictionary | `pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})` |
| From a list of dictionaries | `pd.DataFrame([{'a': 1, 'b': 2}, {'a': 3, 'b': 4}])` |
| From a CSV file (optimized) | `pd.read_csv('file.csv', engine='pyarrow', dtype_backend='pyarrow')` |
| From a Parquet file | `pd.read_parquet('file.parquet', engine='pyarrow')` |
| From an Excel file | `pd.read_excel('file.xlsx', engine='openpyxl')` |
| From a SQL query | `pd.read_sql('SELECT * FROM table', con, index_col='id', parse_dates=['date'])` |
| From a dictionary of arrays | `pd.DataFrame.from_dict(data, orient='index')` |
| From a list of tuples | `pd.DataFrame.from_records([(1,2),(3,4)], columns=['a','b'])` |
| Empty DataFrame with columns | `pd.DataFrame(columns=['col1','col2'])` |
| From a JSON file | `pd.read_json('file.json', orient='records', lines=True)` |
| From clipboard | `pd.read_clipboard()` |
### Inspecting Data {.row-span-2}
| - | - |
| --------------- | -------------------------- |
| `df.head()` | First 5 rows |
| `df.tail()` | Last 5 rows |
| `df.shape` | Number of rows and columns |
| `df.info()` | Info on DataFrame |
| `df.describe()` | Summary statistics |
| `df.columns` | Column names |
| `df.index` | Index |
| `df.dtypes` | Data types of columns |
| First n rows (default 5) | `df.head(10)` |
| Last n rows (default 5) | `df.tail(10)` |
| Number of rows and columns | `df.shape` |
| Detailed info (memory, dtypes, nulls) | `df.info(show_counts=True, memory_usage='deep')` |
| Summary statistics for numeric columns | `df.describe(percentiles=[.25,.5,.75], include='all')` |
| Column names as list | `df.columns.tolist()` |
| Index object | `df.index` |
| Data types of columns | `df.dtypes` |
| Memory usage per column | `df.memory_usage(deep=True)` |
| Value counts with proportions | `df['col'].value_counts(dropna=False, normalize=True)` |
| Unique values in column | `df['col'].unique()` |
| Number of unique values | `df['col'].nunique(dropna=False)` |
| Random sample of rows | `df.sample(n=5, random_state=42)` |
| Check if DataFrame is empty | `df.empty` |
### Selecting Data
| - | - |
| ---------------------- | ------------------------------ |
| `df['col1']` | Select column |
| `df[['col1', 'col2']]` | Select multiple columns |
| `df.loc[0]` | Select row by index |
| `df.loc[:, 'col1']` | Select all rows for 'col1' |
| `df.iloc[0]` | Select row by position |
| `df.iloc[0, 1]` | Select specific value |
| `df[df['col1'] > 2]` | Select rows based on condition |
| Select a single column (as Series) | `df['col']` or `df.col` |
| Select multiple columns | `df[['col1', 'col2']]` |
| Select rows by label index | `df.loc[5:10]` |
| Select rows by integer position | `df.iloc[0:5]` |
| Select a specific cell by label | `df.at[5, 'col']` |
| Select a specific cell by position | `df.iat[0, 1]` |
| Filter rows by condition | `df[df['col'] > 2]` |
| Filter using query string (fast) | `df.query('col > 2 and col2 == "foo"')` |
| Filter with multiple conditions | `df[(df['col']>2) & (df['col2']=='bar')]` |
| Select columns by data type | `df.select_dtypes(include=['number', 'bool'])` |
| Select first row | `df.iloc[0]` |
| Select last row | `df.iloc[-1]` |
| Select random row | `df.sample(1)` |
### Data Cleaning
| - | - |
| --------------------------------------------- | ------------------------------------ |
| `df.dropna()` | Drop rows with any missing values |
| `df.dropna(axis=1)` | Drop columns with any missing values |
| `df.fillna(0)` | Replace missing values with 0 |
| `df.drop_duplicates()` | Drop duplicate rows |
| `df.rename(columns={'old_name': 'new_name'})` | Rename columns |
| `df.astype('int')` | Change data type |
| Drop rows with any missing values | `df.dropna(how='any', axis=0)` |
| Drop rows where all values are missing | `df.dropna(how='all', axis=0)` |
| Drop columns with any missing values | `df.dropna(axis=1, how='any')` |
| Drop columns where all values are missing | `df.dropna(axis=1, how='all')` |
| Fill missing values with a scalar | `df.fillna(0, inplace=False)` |
| Forward fill missing values | `df.ffill(axis=0)` |
| Backward fill missing values | `df.bfill(axis=0)` |
| Interpolate missing values | `df.interpolate(method='linear', limit_direction='both')` |
| Replace values using dict or regex | `df.replace({'old': 'new'}, regex=True)` |
| Rename columns | `df.rename(columns={'old':'new'})` |
| Rename index labels | `df.rename(index={0:'first'})` |
| Drop duplicate rows | `df.drop_duplicates(subset=['col'], keep='first')` |
| Change column data type (nullable int) | `df = df.astype({'col': 'Int64'})` |
| Convert to Arrow-backed string dtype | `df['col'] = df['col'].astype('string[pyarrow]')` |
| Drop columns with variance below threshold | `df.loc[:, df.std() > threshold]` |
| Clip values to a range | `df['col'].clip(lower=0, upper=100)` |
### Adding/Removing Data {.row-span-2}
| - | - |
| -------------------------------------- | ------------------------------- |
| `df['col3'] = df['col1'] + df['col2']` | Add new column |
| `df.drop('col1', axis=1)` | Drop column |
| `df.append(new_row)` | Add new row |
| `df.insert(2, 'new_col', new_data)` | Insert new column at position 2 |
| Add a new column by assignment | `df['new'] = df['a'] + df['b']` |
| Add multiple new columns | `df.assign(new1 = df.a + df.b, new2 = df.a - df.b)` |
| Insert column at a specific position | `df.insert(2, 'new_col', values)` |
| Remove and return a column | `df.pop('col')` |
| Drop a column | `df.drop('col', axis=1, inplace=False)` |
| Drop a row by index label | `df.drop(0, axis=0)` |
| Add a single row | `df.loc[len(df)] = [1,2,3]` |
| Concatenate DataFrames row-wise | `pd.concat([df1, df2], ignore_index=True)` |
| Concatenate DataFrames column-wise | `pd.concat([df1, df2], axis=1)` |
| Concatenate with group keys | `pd.concat([df1, df2], keys=['A','B'])` |
| **Note:** `df.append()` is deprecated. Use `pd.concat()` instead. |
### Combining Data
| - | - |
| ----------------------------------------------------- | ----------------------- |
| `pd.concat([df1, df2])` | Concatenate rows |
| `pd.concat([df1, df2], axis=1)` | Concatenate columns |
| `pd.merge(df1, df2, on='key')` | Merge DataFrames on key |
| `pd.merge(df1, df2, left_on='key1', right_on='key2')` | Merge on different keys |
| `df1.join(df2, lsuffix='_left', rsuffix='_right')` | Join DataFrames |
| Merge two DataFrames on a key (inner join) | `pd.merge(df1, df2, on='key')` |
| Merge two DataFrames (outer join) | `pd.merge(df1, df2, on='key', how='outer')` |
| Merge two DataFrames (left/right join) | `pd.merge(df1, df2, on='key', how='left')` |
| Merge on different key columns | `pd.merge(df1, df2, left_on='key1', right_on='key2')` |
| Merge with validation and metadata | `pd.merge(df1, df2, on='key', validate='1:m', suffixes=('_l','_r'), indicator=True)` |
| Join DataFrames on index | `df1.join(df2, how='inner', lsuffix='_l', rsuffix='_r')` |
| Cartesian product (cross join) | `df1.join(df2, how='cross')` |
| Fill missing values from another DataFrame | `df1.combine_first(df2)` |
## Advanced Indexing
### MultiIndex / Hierarchical Indexing
| - | - |
| ----------------------------------------------------- | ----------------------- |
| Create a MultiIndex from tuples | `pd.MultiIndex.from_tuples([('A',1), ('A',2), ('B',1), ('B',2)])` |
| Set multiple columns as index | `df.set_index(['col1', 'col2'])` |
| Select all rows at first level | `df.loc[('A', slice(None)), 'x']` |
| Cross-section selection | `df.xs('A', level=0)` |
| Get values of a specific index level | `df.index.get_level_values(0)` |
| Pivot a level to columns (unstack) | `df.unstack(level=0)` |
| Compress columns to index (stack) | `df.stack()` |
| Swap levels of MultiIndex | `df.swaplevel()` |
| Sort by a specific index level | `df.sort_index(level=1)` |
| Reorder index levels | `df.reorder_levels([1,0])` |
### Categorical Data
| - | - |
| ----------------------------------------------------- | ----------------------- |
| Create an ordered categorical column | `df['cat'] = pd.Categorical(df['col'], categories=['low','med','high'], ordered=True)` |
| Get integer codes for categories | `df['cat'].cat.codes` |
| Reorder categories | `df['cat'].cat.reorder_categories(['high','med','low'])` |
| Add new categories | `df['cat'].cat.add_categories(['very_high'])` |
| Remove unused categories | `df['cat'].cat.remove_unused_categories()` |
| Group by categorical (skip unobserved) | `df.groupby('cat', observed=True).mean()` |
## Time Series Operations
### Date Handling
| - | - |
| ----------------------------------------------------- | ----------------------- |
| Convert string column to datetime (with format) | `df['date'] = pd.to_datetime(df['date'], format='%Y-%m-%d')` |
| Extract year from datetime | `df['year'] = df['date'].dt.year` |
| Extract month from datetime | `df['month'] = df['date'].dt.month` |
| Extract day from datetime | `df['day'] = df['date'].dt.day` |
| Extract day of week (0=Monday) | `df['dayofweek'] = df['date'].dt.dayofweek` |
| Extract quarter | `df['quarter'] = df['date'].dt.quarter` |
| Check if date is weekend | `df['is_weekend'] = df['date'].dt.dayofweek >= 5` |
| Set datetime column as index | `df.set_index('date', inplace=True)` |
| Generate a date range | `pd.date_range('2026-01-01', periods=100, freq='D')` |
| Add a date offset to each date | `df['date'] + pd.DateOffset(months=1)` |
### Resampling & Frequency Conversion
| - | - |
| ----------------------------------------------------- | ----------------------- |
| Downsample to monthly frequency (aggregate) | `df.resample('M').mean()` |
| Upsample to daily frequency (forward fill) | `df.resample('D').ffill(limit=2)` |
| Custom aggregation during resampling | `df.resample('QE-DEC').agg(['sum', 'mean'])` |
| Resample with custom label and closed side | `df.resample('W-MON', label='right', closed='right')` |
| Business quarter frequency | `df.resample('BQ').mean()` |
| Custom business days (skip holidays) | `from pandas.tseries.offsets import CustomBusinessDay`<br>`offset = CustomBusinessDay(weekmask='Mon Tue Wed Thu Fri', holidays=['2026-07-04'])`<br>`df.resample(offset)` |
| Change frequency without aggregation (asfreq) | `df.asfreq('D', method='ffill', limit=2)` |
| Shift values forward/backward | `df['shifted'] = df['value'].shift(1)` |
| Create lag/lead columns | `df['lag_3'] = df['value'].shift(3)` |
## Transformation Operations
### Reshaping
| - | - |
| ----------------------------------------------------- | ----------------------- |
| Pivot without aggregation (exact reshape) | `df.pivot(index='date', columns='variable', values='value')` |
| Pivot with aggregation (pivot table) | `pd.pivot_table(df, values='value', index='date', columns='var', aggfunc='mean', fill_value=0, margins=True)` |
| Melt from wide to long format | `df.melt(id_vars=['date'], value_vars=['a','b'], var_name='var', value_name='val')` |
| Stack/unstack MultiIndex | `df.stack(future_stack=True)` / `df.unstack()` |
| Cross-tabulation with normalization | `pd.crosstab(df['col1'], df['col2'], normalize='columns', margins=True, margins_name='Total')` |
| Explode list-like column into rows | `df.explode('list_column', ignore_index=True)` |
### Aggregating Data
| - | - |
| -------------------------------------- | ------------------------- |
| `df['col1'].sum()` | Sum of values in column |
| `df['col1'].mean()` | Mean of values in column |
| `df['col1'].count()` | Count of values in column |
| `df['col1'].min()` | Minimum value in column |
| `df['col1'].max()` | Maximum value in column |
| `df['col1'].std()` | Standard deviation |
| `df['col1'].var()` | Variance |
| `df.groupby('col1').sum()` | Group by and sum |
| `df.groupby('col1').mean()` | Group by and mean |
| `df.groupby(['col1', 'col2']).count()` | Group by multiple columns |
| - | - |
| ----------------------------------------------------- | ----------------------- |
| Sum of values | `df['col'].sum(skipna=True)` |
| Mean of values | `df['col'].mean()` |
| Count of non-NA values | `df['col'].count()` |
| Minimum value | `df['col'].min()` |
| Maximum value | `df['col'].max()` |
| Standard deviation (sample) | `df['col'].std(ddof=1)` |
| Variance (sample) | `df['col'].var(ddof=1)` |
| Quantile (e.g., 75th percentile) | `df['col'].quantile(0.75)` |
| Group by a column and sum | `df.groupby('col1')['col2'].sum()` |
| Group by and aggregate multiple columns | `df.groupby('col1').agg({'col2': ['sum','mean'], 'col3': 'max'})` |
| Named aggregation (new style) | `df.groupby('group').agg(avg_value=('value','mean'), max_date=('date','max'), count=('value','count'))` |
| Custom aggregation function | `df.groupby('col1').agg(lambda x: x.max() - x.min())` |
### GroupBy Advanced
| - | - |
| ----------------------------------------------------- | ----------------------- |
| Transform: broadcast result to original rows | `df['mean_per_group'] = df.groupby('group')['value'].transform('mean')` |
| Filter groups based on condition | `df.groupby('group').filter(lambda x: x['value'].sum() > 10)` |
| Apply custom function to each group | `df.groupby('group').apply(lambda g: g.sort_values('date').head(2), include_groups=False)` |
| Pipe transformation after groupby | `df.groupby('group').pipe(lambda g: g['value'].transform(lambda x: (x - x.mean()) / x.std()))` |
| Groupby with observed categoricals | `df.groupby('category', observed=True).mean()` |
### Window Functions
| - | - |
| ----------------------------------------------------- | ----------------------- |
| Rolling mean with window size 5 | `df['col'].rolling(window=5, min_periods=3, center=True).mean()` |
| Multiple aggregations on rolling window | `df['col'].rolling(3).agg(['sum','mean', lambda x: x.max() - x.min()])` |
| Rolling with custom weights | `df['col'].rolling(window=10, win_type='gaussian', weights=lambda x: np.exp(-x**2)).mean()` |
| Expanding statistics (all data up to current point) | `df['col'].expanding(min_periods=2).std()` |
| Exponentially weighted moving average | `df['col'].ewm(span=10, adjust=True, ignore_na=False).mean()` |
| Exponentially weighted variance | `df['col'].ewm(alpha=0.3, min_periods=5).var()` |
| Apply custom function to rolling window | `df['col'].rolling(3).apply(lambda x: x.sum(), raw=True)` |
### Applying Functions {.row-span-2}
| - | - |
| ------------------------------------ | ------------------------------------ |
| `df.apply(np.sqrt)` | Apply function to all values |
| `df['col1'].apply(lambda x: x ** 2)` | Apply function to column |
| `df.applymap(str)` | Apply function to DataFrame elements |
| `df['col1'].map({'a': 1, 'b': 2})` | Map values |
| `df['col1'].replace('a', 1)` | Replace values |
| Apply function element-wise to a column | `df['col'].apply(lambda x: x**2)` |
| Apply function row-wise | `df.apply(lambda row: row['a'] + row['b'], axis=1)` |
| Apply function element-wise to entire DataFrame | `df.map(lambda x: x.upper() if isinstance(x,str) else x)` (**Note:** `applymap` is deprecated) |
| Map values using a dictionary | `df['col'].map({'a':1,'b':2}, na_action='ignore')` |
| Vectorized string operations | `df['col'].str.upper()`, `df['col'].str.contains('pattern')` |
| Vectorized datetime operations | `df['date'].dt.year`, `df['date'].dt.dayofweek` |
| Fast arithmetic using eval (numexpr) | `df.eval('c = a * b + 2 * d', engine='numexpr')` |
| Fast filtering using query (numexpr) | `df.query('a > b & c.str.contains("foo")')` |
| Chain operations with pipe | `df.pipe(lambda d: d.assign(ratio=d.a/d.b)).pipe(...)` |
### Handling Dates
### String Operations
| - | - |
| ----------------------------------------- | ------------------- |
| `df['date'] = pd.to_datetime(df['date'])` | Convert to datetime |
| `df['year'] = df['date'].dt.year` | Extract year |
| `df['month'] = df['date'].dt.month` | Extract month |
| `df['day'] = df['date'].dt.day` | Extract day |
| `df.set_index('date', inplace=True)` | Set date as index |
| - | - |
| ------------------------------------ | ------------------------------------ |
| Convert to uppercase/lowercase | `df['col'].str.upper()`, `df['col'].str.lower()` |
| Check if string contains pattern | `df['col'].str.contains('pattern', regex=True, na=False)` |
| Split string into columns | `df['col'].str.split(',', expand=True, n=1)` |
| Replace using regular expression | `df['col'].str.replace(r'\s+', '_', regex=True)` |
| Extract capture groups into columns | `df['col'].str.extract(r'(\d+)-(\w+)')` |
| Extract all matches | `df['col'].str.extractall(r'(\d+)')` |
| Find all matches (returns list) | `df['col'].str.findall(r'\d+')` |
| Concatenate strings with separator | `df['col'].str.cat(sep='|')` |
| Pad strings to fixed width | `df['col'].str.pad(width=10, side='left', fillchar='0')` |
| Extract substring | `df['col'].str.slice(start=0, stop=5)` |
| Get string length | `df['col'].str.len()` |
| Check start/end of string | `df['col'].str.startswith('prefix')`, `df['col'].str.endswith('.csv')` |
| Strip whitespace | `df['col'].str.strip()`, `df['col'].str.lstrip()`, `df['col'].str.rstrip()` |
### Input/Output
## Performance & Memory Optimization
### Performance Settings
| - | - |
| ------------------------------------ | ------------------------------------ |
| Enable copy-on-write (Pandas 2.0+) | `pd.options.mode.copy_on_write = True` |
| Fast arithmetic with eval | `df.eval('c = a * b + 2 * d', engine='numexpr')` |
| Fast filtering with query | `df.query('a > b and c == "foo"', engine='numexpr')` |
| Read large CSV in chunks | `for chunk in pd.read_csv('large.csv', chunksize=100000, dtype_backend='pyarrow'): process(chunk)` |
| Convert to numpy array (avoid copy) | `arr = df[['a','b']].to_numpy(copy=False)` |
| Memory map large files | `df = pd.read_csv('large.csv', memory_map=True)` |
| Use PyArrow backend for CSV | `pd.read_csv('file.csv', engine='pyarrow', dtype_backend='pyarrow')` |
### Memory Optimization
| - | - |
| ------------------------------------ | ------------------------------------ |
| Downcast integer columns | `def downcast(df):`<br>` for col in df.select_dtypes(include='integer'):`<br>` df[col] = pd.to_numeric(df[col], downcast='unsigned')`<br>` for col in df.select_dtypes(include='float'):`<br>` df[col] = pd.to_numeric(df[col], downcast='float')`<br>` return df` |
| Convert to categorical if low cardinality | `for col in df.select_dtypes(include='object'):`<br>` if df[col].nunique() / len(df) < 0.05:`<br>` df[col] = df[col].astype('category')` |
| Convert all to Arrow dtypes | `df = df.convert_dtypes(dtype_backend='pyarrow')` |
| Use sparse arrays for zeros | `df['sparse'] = pd.arrays.SparseArray(df['col'], fill_value=0)` |
| Diagnose memory usage | `df.memory_usage(deep=True)` |
## Input/Output {.cols-2}
| - | - |
| ---------------------------------------------- | ------------------------------------ |
| `df.to_csv('file.csv')` | Save DataFrame to CSV |
| `df = pd.read_csv('file.csv')` | Load DataFrame from CSV |
| `df.to_excel('file.xlsx')` | Save DataFrame to Excel |
| `df = pd.read_excel('file.xlsx')` | Load DataFrame from Excel |
| `from sqlalchemy import create_engine` | Import SQLAlchemy for SQL operations |
| `engine = create_engine('sqlite:///:memory:')` | Create SQL engine |
| `df.to_sql('table_name', engine)` | Save to SQL table |
| `df = pd.read_sql('table_name', engine)` | Load from SQL table |
| Save to CSV (optimized) | `df.to_csv('file.csv', index=False, encoding='utf-8')` |
| Load from CSV (optimized) | `df = pd.read_csv('file.csv', engine='pyarrow', dtype_backend='pyarrow')` |
| Save to Parquet (fast, compressed) | `df.to_parquet('file.parquet', engine='pyarrow')` |
| Load from Parquet | `df = pd.read_parquet('file.parquet', engine='pyarrow')` |
| Save to Excel | `df.to_excel('file.xlsx', sheet_name='Sheet1', engine='openpyxl')` |
| Load from Excel | `df = pd.read_excel('file.xlsx', sheet_name=0, engine='openpyxl')` |
| Save to JSON (line-delimited) | `df.to_json('file.json', orient='records', lines=True)` |
| Load from JSON (line-delimited) | `df = pd.read_json('file.json', orient='records', lines=True)` |
| Save to SQL table | `df.to_sql('table', con, if_exists='replace', index=False)` |
| Load from SQL query | `df = pd.read_sql('SELECT * FROM table', con, index_col='id', parse_dates=['date'])` |
| Copy to/from clipboard | `df.to_clipboard()` / `pd.read_clipboard()` |
| Save to/from pickle (fast but unsafe) | `df.to_pickle('file.pkl')` / `pd.read_pickle('file.pkl')` |
| Save to/from Feather (fast) | `df.to_feather('file.feather')` / `pd.read_feather('file.feather')` |
| Save to/from HDF5 | `df.to_hdf('file.h5', key='data', mode='w')` / `pd.read_hdf('file.h5', key='data')` |
## Common Pitfalls & Best Practices
| Pitfall | Solution |
|---------|----------|
| Chained indexing (creates views) | Use `df.loc[df.a > 0, 'b']` instead of `df[df.a > 0]['b']` |
| In-place mutation of large DataFrames | Use `inplace=False` (default) for method chaining; avoid mutation unless necessary |
| Slicing without copy | Use `df.copy()` when you need an independent slice |
| Slow datetime parsing | Always specify `format` in `pd.to_datetime()` for 10x speedup |
| Memory bloat from object dtype | Use `categorical` dtype for low cardinality columns (<5% unique) |
| Ambiguous merge relationships | Specify `validate` in merge ('1:1', '1:m', etc.) to catch errors |
| Groupby with unused categories | Use `observed=True` to skip unobserved categories in groupby |
| Slow loops with `apply` | Use vectorized operations when possible (`df['a'] + df['b']` vs `apply`) |
| Deprecated `append()` | Use `pd.concat([df, other])` instead |
| Deprecated `applymap()` | Use `df.map()` instead (Pandas 2.2+) |
| Missing `pd.NA` handling | Check for `pd.NA` when using nullable dtypes |
| Large CSV without optimization | Enable copy-on-write and use PyArrow backend |
## Deprecations & Version Notes
| Version | Deprecated/Removed | Replacement |
|---------|-------------------|-------------|
| 1.4.0 | `df.append()` | `pd.concat([df, other])` |
| 2.2.0 | `df.applymap()` | `df.map()` |
| 2.0.0 | `df.astype('int')` with NA | Use nullable `'Int64'` |
| 2.0.0 | Inferred string dtype | Use `pd.options.future.infer_string = True` |
| 2.0.0 | `pd.DataFrame.to_csv()` with `compression` | Use `compression='gzip'` explicitly |
## Official Documentation & Resources
- [pandas.pydata.org](https://pandas.pydata.org/)
- [pandas GitHub](https://github.com/pandas-dev/pandas)
- [PyArrow integration](https://arrow.apache.org/docs/python/pandas.html)
- [API Reference](https://pandas.pydata.org/docs/reference/index.html)
- [User Guide](https://pandas.pydata.org/docs/user_guide/index.html)
```
+7
View File
@@ -0,0 +1,7 @@
<svg width="1em" height="1em" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M12 2L1.5 20.5H22.5L12 2Z" fill="#1A73E8"/>
<path d="M12 6.5L4.8 19H19.2L12 6.5Z" fill="#FFFFFF" fill-opacity="0.25"/>
<path d="M12 9L7.5 17H16.5L12 9Z" fill="#FFFFFF"/>
<circle cx="12" cy="14" r="1.5" fill="#1A73E8"/>
<path d="M12 2V12.5" stroke="#FFFFFF" stroke-width="1.2" stroke-linecap="round"/>
</svg>

After

Width:  |  Height:  |  Size: 425 B

+7
View File
@@ -0,0 +1,7 @@
<svg width="1em" height="1em" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M12 2L1.5 20.5H22.5L12 2Z" fill="#1A73E8"/>
<path d="M12 6.5L4.8 19H19.2L12 6.5Z" fill="#FFFFFF" fill-opacity="0.25"/>
<path d="M12 9L7.5 17H16.5L12 9Z" fill="#FFFFFF"/>
<circle cx="12" cy="14" r="1.5" fill="#1A73E8"/>
<path d="M12 2V12.5" stroke="#FFFFFF" stroke-width="1.2" stroke-linecap="round"/>
</svg>

After

Width:  |  Height:  |  Size: 425 B