27 Commits
Author SHA1 Message Date
Tom Moor 6d82287a08 fix: Do not download relative attachment paths in API imports (#14650) 2026-11-18 22:36:26 -05:00
Tom Moor 587c211f57 fix: Do not overwrite canceled or finished imports from import tasks (#14578) 2026-11-13 08:40:05 -05:00
Tom Moor 6fd7eb7cf0 chore: Share import state lists from the Import model (#14575) 2026-11-13 08:05:13 -05:00
Tom Moor 631114129e perf: Lock only the target row in locked queries with joins (#14564) 2026-11-12 21:48:14 -05:00
Tom Moor f95761dddf perf: Use FOR NO KEY UPDATE where locked rows are only updated (#14559)
* perf: Use FOR NO KEY UPDATE where locked rows are only updated

Rows that other tables reference by foreign key are now locked with
FOR NO KEY UPDATE when the transaction only updates or soft-deletes
them, so concurrent inserts that reference the row are not blocked.

* test: Collaborative save does not wait for foreign key inserts
2026-11-12 18:54:37 -05:00
Tom Moor fb2f0e76d1 fix: Prevent Notion imports looping forever (#14456)
* fix: Prevent Notion import looping on self-referencing pages

* fix: Stop import tasks once the import has finished
2026-11-03 08:27:14 -05:00
mehmet turac f9ae2dcd6d fix: Imported attachment keys end in "null" when no name is set (#13750)
uploadAttachments reads the name off alt for images and title for video
and attachment nodes. Both come through as null when the markup doesn't
set one, and String(null) is "null" - 4 chars, so the name.length !== 0
check passes and the node.type.name fallback never runs. getKey then
sticks "null" on the end of the key: uploads/<userId>/<uuid>/null.

Fall back to the node type before the String(). Left the length check in,
it still catches a whitespace-only name. Only hits importers on the
default shouldUploadAttachmentsPerPage(); markdown import opts out.
2026-09-19 10:51:53 -04:00
Tom Moor f852339c57 fix: Remove source uploads after imports (#13757)
* fix: Remove source uploads after imports

* fix: Defer import upload cleanup

* fix: Initialize import cleanup state
2026-09-16 23:58:03 +00:00
Tom Moor 5a9fb0024f fix: Immediately stop retrying malformed imports 2026-07-25 12:38:57 -04:00
6a14ac8ae6 feat: Slab importer (#12859)
* Add Slab importer backed by the Markdown import pipeline

Slab exports a zip of Markdown documents that is structurally identical to
the Outline Markdown export, with one difference: images are referenced as
remote signed URLs rather than files inside the archive.

Rather than build a parallel importer, this reuses the Markdown task and
processor and teaches the shared import pipeline to download remote images:

- Enable per-page attachment upload for the Markdown task. The base
  APIImportTask already downloads remote image/video/attachment URLs and
  rewrites them to internal redirect URLs; the Markdown task previously
  opted out because its attachments live in the zip.
- Make the base upload step skip URLs that are already internal, so local
  zip attachments (resolved to redirect URLs during rewriteMarkdown and
  uploaded from the archive in onAllTasksCompleted) pass through untouched
  while remote URLs are fetched and re-hosted. Also guards the URL rewrite
  against nodes not present in the download map.
- Introduce IntegrationService.Slab as a distinct, importable service that
  routes through the Markdown task/processor, so imports are tracked and
  labelled as "Slab" with no duplicated import logic.
- Wire the create API (schema + route), the import settings UI (Slab card
  and dialog), and a placeholder logo asset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o

* Add shell SlabAPIImportTask as a Slab-specific seam

Slab imports previously ran through MarkdownAPIImportTask directly. Introduce
a thin SlabAPIImportTask subclass so Slab gets its own task name (for
scheduling, tracing, and retries) and a dedicated place to override
behavior as Slab exports diverge from the generic Markdown shape.

The subclass inherits all conversion/attachment/persistence logic and only
overrides scheduleNextTask to keep the whole import chain on the Slab class.
MarkdownImportsProcessor now selects the task implementation by the import's
service when scheduling the first task.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o

* Unwrap single root "slab" directory in Slab imports

Slab exports wrap the entire workspace in a single top-level directory named
"slab". Without handling this, the import produces one "slab" collection
containing everything, instead of mapping each workspace area to its own
collection.

Add a `resolveCollectionRootNodes` seam on the Markdown bootstrap phase
(default: pass entries through unchanged) and override it in SlabAPIImportTask
to descend into a lone, case-insensitive "slab" root directory so its child
directories become collections. Paths are left intact, so the attachment
manifest, completion re-walk, and internal-link resolution stay consistent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o

* Use the filename, not the first heading, as the title for Slab imports

In Slab a document's leading heading is real content, not its title — the
filename is authoritative. Add an `extractTitle` option to
DocumentConverter.convert (default true, preserving existing callers) that,
when false, skips lifting a leading H1 into the title and leaves it in the
body. Expose it through a `shouldExtractTitleFromHeading` seam on the
Markdown task and override it in SlabAPIImportTask. With no extracted title,
the page falls back to the filename-derived title.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o

* Update logo

* Narrow MarkdownAPIImportTask generic back to Markdown

Slab imports now run through SlabAPIImportTask, so the base Markdown task no
longer handles the Slab service directly and its generic can return to
Markdown-only. SlabAPIImportTask keeps a Markdown | Slab union on its
scheduleNextTask override, which is required so the override stays a valid
(contravariant) override of the base's Markdown-typed parameter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o

* Optimised images with calibre/image-actions

* reorder

* Move Slab importer into its own plugin

Restructure the Slab importer to live in plugins/slab, mirroring the Notion
plugin, instead of being threaded through the core Markdown importer.

- plugins/slab/server registers SlabImportsProcessor (Hook.Processor) and
  SlabAPIImportTask (Hook.Task). The processor extends MarkdownImportsProcessor
  to claim the Slab service and schedule the Slab task; the task keeps its
  Slab-specific overrides (remote image download, root "slab" dir unwrap,
  filename-as-title).
- plugins/slab/client registers the Slab import card (Hook.Imports) and owns
  the import dialog.
- Revert MarkdownImportsProcessor to Markdown-only and drop the hardcoded Slab
  card from the Import settings screen.

The Slab service enum, import schema, and create route remain in core, as the
Notion plugin's equivalents do.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o

* Generalize DropToImport to accept an importable service

DropToImport previously branched on `format` to map MarkdownZip/JSON/Slab
to specific imports.create calls, with a dead collections.import fallback.
Replace this with a single `service` prop and one generic imports.create
call. Core no longer hardcodes any service name (notably the Slab plugin
service), and the Markdown, JSON, and Slab dialogs each pass their service.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-07-10 19:05:08 -04:00
Tom Moor 66eee9007f chore: Upgrade to Typescript 6 (#12737)
* chore: Upgrade TypeScript to 6.0
2026-06-18 17:34:05 -04:00
Tom Moor 09e99ac98d fix: Graceful exit when import is canceled beneath import task (#12497) 2026-05-27 22:37:54 -04:00
Tom MoorandClaude Opus 4.7 82d7041b6b chore: Refactor Markdown importer to use new import pipeline (#12361)
* chore: Refactor Markdown importer to use new import pipeline

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 14:10:15 -04:00
Tom Moor 0139b91b5d chore: Replace lodash with es-toolkit (#12281)
* chore: Replace lodash with es-toolkit

Migrate all direct lodash imports to es-toolkit/compat for a smaller,
faster, lodash-compatible utility library. Transitive lodash usage from
other packages remains unchanged.

* fix: Restore isPlainObject semantics in CanCan policy

The lodash migration aliased `isObject` to `lodash/isPlainObject` and
the codemod incorrectly mapped the local name to es-toolkit's `isObject`,
which also returns true for arrays and functions. This caused condition
objects in policy definitions to be skipped, breaking authorization
checks across the codebase.

* fix: Restore unicode-aware length counting in validators

es-toolkit/compat's size() returns string.length, while lodash's _.size()
counts unicode code points. Switch to [...value].length to preserve the
previous behavior so multi-byte characters like emoji count as one.
2026-05-06 21:03:47 -04:00
Tom MoorandClaude Opus 4.6 793804cd0d feat: Strip comments from presentation mode (#11860)
* feat: Strip comment marks from documents in presentation mode

Move removeMarks to shared ProsemirrorHelper and use it to strip comment
marks before rendering slides. Make server ProsemirrorHelper extend the
shared class to eliminate duplication and remove SharedProsemirrorHelper
imports from server code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: Use Set for mark lookup and cloneDeep for browser compat

Use a Set for O(1) mark lookups in removeMarks traversal. Replace
structuredClone with lodash/cloneDeep to support older browsers
that lack the native API.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: Add tests for ProsemirrorHelper.removeMarks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-23 20:21:43 -04:00
Tom Moor 0722b42613 fix: Potential task queue saturation in Notion importer (#11428)
* fix: Potential task queue saturation in Notion import

* Reduces concurrent Notion API pressure from 3× the recursive call depth down to 1
2026-02-12 21:56:00 -05:00
Tom Moor 00fb4d1af7 chore: Update node style imports (#11277)
- crypto → node:crypto
  - fs → node:fs
  - fs/promises → node:fs/promises
  - path → node:path
  - http → node:http
  - https → node:https
  - stream → node:stream
  - buffer → node:buffer
  - url → node:url
  - os → node:os
  - net → node:net
  - dns → node:dns
  - events → node:events
  - readline → node:readline
  - querystring → node:querystring
  - util → node:util
2026-01-26 20:51:50 -05:00
Tom Moor a3b2615edf chore: Remove future public bucket usage (#10977)
* No longer upload avatars to public bucket

* Public redirect

* tests

* test

* test
2025-12-24 20:24:58 -05:00
Tom Moor bf45e97641 chore: Enforce type import consistency (#10968)
* Update types

* fix circular dep

* type imports

* lint type imports and --fix
2025-12-19 23:07:02 -05:00
Tom Moor 42959d66db chore: Add cron task partitioning (#10736)
* wip

* Implementation complete

* tidying

* test

* Address feedback

* Remove duplicative retry logic from UpdateDocumentsPopularityScoreTask.
Now that we're split across many runs this is not neccessary

* Refactor to subclass, config to instance

* Refactor BaseTask to named export

* fix: Missing partition

* tsc

* Feedback
2025-11-27 16:57:52 +01:00
Alex 76701e35ec fix: replace uuid package with standard module (#10318) 2025-10-10 17:06:51 -04:00
codegen-sh[bot]andTom Moor 879c568a2c Upgrade Prettier to v3.6.2 (#9500)
* Upgrade Prettier to v3.6.2 and eslint-plugin-prettier to v5.5.1

- Upgraded prettier from ^2.8.8 to ^3.6.2 (latest version)
- Upgraded eslint-plugin-prettier from ^4.2.1 to ^5.5.1 for compatibility
- Applied automatic formatting changes from new Prettier version
- All existing ESLint and Prettier configurations remain compatible

* Applied automatic fixes

* Trigger CI

---------

Co-authored-by: codegen-sh[bot] <131295404+codegen-sh[bot]@users.noreply.github.com>
Co-authored-by: Tom Moor <tom@getoutline.com>
2025-06-28 10:22:28 -04:00
Tom Moor 75f173c6ff Add option to publish but not persist events (#9448)
* Add option to publish but not persist events

* tsc
2025-06-14 10:46:01 -04:00
codegen-sh[bot]andTom Moor a226889143 Update task scheduling to use instance method (#9092)
* Update task scheduling to use instance method

* Delete update_task_schedule.sh

* Applied automatic fixes

* tsc

---------

Co-authored-by: codegen-sh[bot] <131295404+codegen-sh[bot]@users.noreply.github.com>
Co-authored-by: Tom Moor <tom@getoutline.com>
2025-04-29 06:47:51 -04:00
66e4ec32ed Fix: Handle Notion database not found errors gracefully (#8860)
* Fix: Handle Notion database not found errors gracefully

* Fix: Use Logger.warn instead of console.log in Notion import task

* Applied automatic fixes

* Touch to trigger actions

* Fix: Implement additional improvements for Notion import error handling

* Applied automatic fixes

* Change to trigger CI

* Fix TypeScript error: Add type assertion for filtered parsedPages

---------

Co-authored-by: codegen-sh[bot] <131295404+codegen-sh[bot]@users.noreply.github.com>
Co-authored-by: Tom Moor <tom@getoutline.com>
Co-authored-by: Codegen <codegen@example.com>
2025-04-06 09:04:23 -07:00
Hemachandar dcb7b86df8 Store import error in DB (#8811) 2025-03-29 06:08:07 -07:00
Hemachandar 6e98568e5b API importer for Notion (#8710) 2025-03-23 12:19:13 -07:00