mirror of
https://github.com/outline/outline.git
synced 2026-11-22 14:01:41 +03:00
* feat: Personal collections backend Adds an owner to collections. A personal collection is private, one per user per team, created lazily through collections.personal, and only its owner can see or manage it. Admin bypasses in policies, lists, exports, events, webhooks and file operations exclude the personal collections of other users. Turning off the PersonalDocs preference makes the collection read-only. * test: Access matrix for personal collections Checks that collection and document endpoints, list and search results, MCP tools, collaboration and websocket rooms agree with the read policy for the owner, collaborators, members, admins, guests and other teams. * feat: Personal collection in the sidebar Adds a Personal sidebar section that renders the personal collection with the collection document tree, creates the collection on the first new document, and keeps it out of the Collections section. Adds the workspace setting, the personal sidebar context, and hides collection sharing and membership UI for personal collections. * fix: Drag and drop in shared with me trees Anchors the reorder drop cursor inside the row, and prefetches documents on hover so that nested rows that were never opened can be dragged. * feat: Personal collection owner for collaborators, no collection page Presents the owner of the personal collection on documents, so that collaborators see the owner in the share popover. Root rows in the Personal section show the default document icon. A personal collection has no page: the scene shows not found, and the breadcrumb, command bar and delete redirect no longer link to it. * fix: Personal sidebar indent and breadcrumb items without a link Children of rows at depth 0 skip a level so that they start beyond the row icon. Breadcrumb items without a destination, such as the personal and deleted collection, are shown as plain text. * test: Reduce personal collection tests to list paths Endpoints that authorize a single model are already covered by the policy tests. Keeps the lists and searches, which filter in queries, and the presenter. * test: Move personal collection list tests into collections tests * feat: Create documents in the personal collection by alias documents.create, documents.import and the MCP create_document tool accept the collectionId "personal", which refers to the personal collection of the user and creates it when it does not exist yet. * refactor: Remove collections.personal route The sidebar creates its first personal document with the personal collectionId alias, so documents.create is the only path that creates the personal collection. * feat: Personal collection name, starring, templates and default Personal collections cannot be starred, renamed or hold templates, and the PersonalDocs preference is on by default. The app shows collections through a displayName getter, which is the translated "Personal" for a personal collection. * fix: Use the same policy for starring and unstarring collections * fix: No pins or viewer insights for personal documents * revert: Show personal collection events in the audit log Admins with audit access should see events in personal collections, even when they cannot open the documents. * fix: Load collection owner in share scopes * fix: Keep personal collections out of workspace exports and webhooks Workspace exports skip all personal collections, while an explicit export of a personal collection stays available to its owner. Events in personal collections, including attachments and documents moved after queueing, are never delivered to workspace webhooks. Personal collections cannot be renamed or made public through any path, cannot hold pins, and are not offered as mentions. * Change icon to avatar * fix: Personal documents cannot be unpublished * feat: Personal collection context for MCP Collections are presented with an isPersonal flag, the server instructions explain the personal collection, and move_document and update_document accept the personal collectionId alias. * fix: Keep moved personal exports private and remove pins on move Document exports record the owner of the personal collection they came from, so they stay private after the document moves. Moving documents into a personal collection removes their pins, including pins to home. * fix: Only create the personal collection in MCP when it is used
322 lines
9.5 KiB
TypeScript
322 lines
9.5 KiB
TypeScript
import { Readable } from "node:stream";
|
|
import fs from "fs-extra";
|
|
import { truncate } from "es-toolkit/compat";
|
|
import { Op } from "sequelize";
|
|
import type { WhereOptions } from "sequelize";
|
|
import type { ZipFile } from "yazl";
|
|
import { toError } from "@shared/utils/error";
|
|
import type { NavigationNode } from "@shared/types";
|
|
import { FileOperationState, NotificationEventType } from "@shared/types";
|
|
import { bytesToHumanReadable } from "@shared/utils/files";
|
|
import ExportFailureEmail from "@server/emails/templates/ExportFailureEmail";
|
|
import ExportSuccessEmail from "@server/emails/templates/ExportSuccessEmail";
|
|
import env from "@server/env";
|
|
import { ValidationError } from "@server/errors";
|
|
import Logger from "@server/logging/Logger";
|
|
import {
|
|
Attachment,
|
|
Collection,
|
|
Document,
|
|
Event,
|
|
FileOperation,
|
|
Team,
|
|
User,
|
|
} from "@server/models";
|
|
import fileOperationPresenter from "@server/presenters/fileOperation";
|
|
import { sequelizeReadOnly } from "@server/storage/database";
|
|
import FileStorage from "@server/storage/files";
|
|
import { BaseTask, TaskPriority } from "./base/BaseTask";
|
|
|
|
type Props = {
|
|
fileOperationId: string;
|
|
};
|
|
|
|
export default abstract class ExportTask extends BaseTask<Props> {
|
|
protected jobId({ fileOperationId }: Props) {
|
|
return `export:${fileOperationId}`;
|
|
}
|
|
|
|
/**
|
|
* Transforms the data to be exported, uploads, and notifies user.
|
|
*
|
|
* @param props The props
|
|
*/
|
|
public async perform({ fileOperationId }: Props) {
|
|
Logger.info("task", `ExportTask fetching data for ${fileOperationId}`);
|
|
const fileOperation = await FileOperation.findByPk(fileOperationId, {
|
|
rejectOnEmpty: true,
|
|
});
|
|
const [team, user] = await Promise.all([
|
|
Team.findByPk(fileOperation.teamId, { rejectOnEmpty: true }),
|
|
User.findByPk(fileOperation.userId, { rejectOnEmpty: true }),
|
|
]);
|
|
|
|
let filePath: string | undefined;
|
|
let readStream: fs.ReadStream | undefined;
|
|
|
|
try {
|
|
Logger.info("task", `ExportTask processing data for ${fileOperationId}`, {
|
|
options: fileOperation.options,
|
|
});
|
|
|
|
await this.updateFileOperation(fileOperation, {
|
|
state: FileOperationState.Creating,
|
|
});
|
|
|
|
filePath = await this.loadDataAndExport(fileOperation, user);
|
|
|
|
Logger.info("task", `ExportTask uploading data for ${fileOperationId}`);
|
|
|
|
await this.updateFileOperation(fileOperation, {
|
|
state: FileOperationState.Uploading,
|
|
});
|
|
|
|
const stat = await fs.stat(filePath);
|
|
readStream = fs.createReadStream(filePath);
|
|
const url = await FileStorage.store({
|
|
body: readStream,
|
|
contentLength: stat.size,
|
|
contentType: "application/zip",
|
|
key: fileOperation.key,
|
|
acl: "private",
|
|
});
|
|
|
|
await this.updateFileOperation(fileOperation, {
|
|
size: stat.size,
|
|
state: FileOperationState.Complete,
|
|
url,
|
|
});
|
|
|
|
if (user.subscribedToEventType(NotificationEventType.ExportCompleted)) {
|
|
await new ExportSuccessEmail({
|
|
to: user.email,
|
|
language: user.language,
|
|
userId: user.id,
|
|
id: fileOperation.id,
|
|
teamUrl: team.url,
|
|
teamId: team.id,
|
|
}).schedule();
|
|
}
|
|
} catch (error) {
|
|
await this.updateFileOperation(fileOperation, {
|
|
state: FileOperationState.Error,
|
|
error: toError(error),
|
|
});
|
|
if (user.subscribedToEventType(NotificationEventType.ExportCompleted)) {
|
|
await new ExportFailureEmail({
|
|
to: user.email,
|
|
language: user.language,
|
|
userId: user.id,
|
|
teamUrl: team.url,
|
|
teamId: team.id,
|
|
}).schedule();
|
|
}
|
|
throw error;
|
|
} finally {
|
|
// Destroy the read stream first to release the file handle before deletion
|
|
if (readStream) {
|
|
readStream.destroy();
|
|
}
|
|
if (filePath) {
|
|
void fs.unlink(filePath).catch((error) => {
|
|
Logger.error(`Failed to delete temporary file ${filePath}`, error);
|
|
});
|
|
}
|
|
}
|
|
}
|
|
|
|
public async loadDataAndExport(
|
|
fileOperation: FileOperation,
|
|
user: User
|
|
): Promise<string> {
|
|
if (fileOperation.documentId) {
|
|
const document = await sequelizeReadOnly.transaction((transaction) =>
|
|
Document.findByPk(fileOperation.documentId!, {
|
|
include: {
|
|
model: Collection.scope("withDocumentStructure"),
|
|
as: "collection",
|
|
},
|
|
rejectOnEmpty: true,
|
|
transaction,
|
|
})
|
|
);
|
|
|
|
const documentStructure = document.collection?.getDocumentTree(
|
|
document.id
|
|
);
|
|
|
|
if (!documentStructure) {
|
|
throw new Error("Document not found in collection tree");
|
|
}
|
|
|
|
return this.exportDocument(
|
|
document,
|
|
documentStructure.children ?? [],
|
|
fileOperation.options?.includeAttachments ?? true
|
|
);
|
|
}
|
|
|
|
// ensure attachment size is within limits
|
|
if (
|
|
!fileOperation.collectionId &&
|
|
fileOperation.options?.includeAttachments &&
|
|
env.MAXIMUM_EXPORT_SIZE
|
|
) {
|
|
const totalAttachmentsSize = await Attachment.getTotalSizeForTeam(
|
|
sequelizeReadOnly,
|
|
user.teamId
|
|
);
|
|
|
|
if (totalAttachmentsSize > env.MAXIMUM_EXPORT_SIZE) {
|
|
throw ValidationError(
|
|
`${bytesToHumanReadable(
|
|
totalAttachmentsSize
|
|
)} of attachments in workspace is larger than maximum export size of ${bytesToHumanReadable(
|
|
env.MAXIMUM_EXPORT_SIZE
|
|
)}.`
|
|
);
|
|
}
|
|
}
|
|
|
|
const where: WhereOptions<Collection> = {
|
|
teamId: user.teamId,
|
|
};
|
|
|
|
if (!fileOperation.options?.includePrivate) {
|
|
where.permission = {
|
|
[Op.ne]: null,
|
|
};
|
|
}
|
|
|
|
if (fileOperation.collectionId) {
|
|
where.id = fileOperation.collectionId;
|
|
// Only the owner can export a personal collection.
|
|
where.ownerId = { [Op.or]: [{ [Op.is]: null }, { [Op.eq]: user.id }] };
|
|
} else {
|
|
where.archivedAt = {
|
|
[Op.eq]: null,
|
|
};
|
|
// A workspace export never includes personal collections.
|
|
where.ownerId = { [Op.is]: null };
|
|
}
|
|
|
|
const collections = await sequelizeReadOnly.transaction((transaction) =>
|
|
Collection.scope("withDocumentStructure").findAll({
|
|
where,
|
|
transaction,
|
|
})
|
|
);
|
|
|
|
return this.exportCollections(collections, fileOperation);
|
|
}
|
|
|
|
/**
|
|
* Transform the data in all of the passed collections into a single Buffer.
|
|
*
|
|
* @param collections The collections to export
|
|
* @returns A promise that resolves to a temporary file path
|
|
*/
|
|
protected abstract exportCollections(
|
|
collections: Collection[],
|
|
fileOperation: FileOperation
|
|
): Promise<string>;
|
|
|
|
/**
|
|
* Transform the data in the document into a single Buffer.
|
|
*
|
|
* @param document The document to export
|
|
* @param documentStructure Structure of document's children
|
|
* @param includeAttachments Whether to include attachments in the export
|
|
* @returns A promise that resolves to a temporary file path
|
|
*/
|
|
protected abstract exportDocument(
|
|
document: Document,
|
|
documentStructure: NavigationNode[],
|
|
includeAttachments: boolean
|
|
): Promise<string>;
|
|
|
|
/**
|
|
* Add an attachment to the archive, streaming its contents from storage as
|
|
* the archive is written so that the file is never held in memory whole.
|
|
*
|
|
* A read that fails, or is interrupted part way through, fails the export —
|
|
* a silently truncated archive is worse than none, as it looks complete.
|
|
* An attachment that is simply absent from storage is written as an empty
|
|
* entry, since a row outliving its file must not make export impossible.
|
|
*
|
|
* @param zip The archive to add the attachment to
|
|
* @param attachment The attachment to add
|
|
* @param pathInZip The path to store the attachment at within the archive
|
|
*/
|
|
protected addAttachmentToArchive(
|
|
zip: ZipFile,
|
|
attachment: Attachment,
|
|
pathInZip: string
|
|
) {
|
|
zip.addReadStreamLazy(
|
|
pathInZip,
|
|
{ mtime: attachment.updatedAt },
|
|
(callback) => {
|
|
attachment.stream.then(
|
|
(stream) => {
|
|
if (!stream) {
|
|
Logger.warn(`Attachment is missing from storage`, {
|
|
attachmentId: attachment.id,
|
|
teamId: attachment.teamId,
|
|
});
|
|
return callback(null, Readable.from([]));
|
|
}
|
|
|
|
// yazl does not watch the streams it is handed, so an interrupted
|
|
// read would otherwise stall the archive rather than fail it.
|
|
stream.on("error", (err) => zip.emit("error", toError(err)));
|
|
return callback(null, stream);
|
|
},
|
|
(err) => callback(toError(err), Readable.from([]))
|
|
);
|
|
}
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Update the state of the underlying FileOperation in the database and send
|
|
* an event to the client.
|
|
*
|
|
* @param fileOperation The FileOperation to update
|
|
*/
|
|
private async updateFileOperation(
|
|
fileOperation: FileOperation,
|
|
options: Omit<Partial<FileOperation>, "error"> & { error?: Error }
|
|
) {
|
|
await fileOperation.update(
|
|
{
|
|
...options,
|
|
error: options.error
|
|
? truncate(options.error.message, { length: 255 })
|
|
: undefined,
|
|
},
|
|
{
|
|
hooks: false,
|
|
}
|
|
);
|
|
|
|
await Event.schedule({
|
|
name: "fileOperations.update",
|
|
modelId: fileOperation.id,
|
|
teamId: fileOperation.teamId,
|
|
actorId: fileOperation.userId,
|
|
data: fileOperationPresenter(fileOperation),
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Job options such as priority and retry strategy, as defined by Bull.
|
|
*/
|
|
public get options() {
|
|
return {
|
|
priority: TaskPriority.Background,
|
|
attempts: 1,
|
|
};
|
|
}
|
|
}
|