CN-JS-HuiBai/Media-Coding-Web
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: CN-JS-HuiBai:df5999c338f17cfee76a3ba2a2ee588d9e42269f
Branches Tags
CN-JS-HuiBai:Rockchip CN-JS-HuiBai:Universal
..
compare: CN-JS-HuiBai:Universal
Branches Tags
CN-JS-HuiBai:Rockchip CN-JS-HuiBai:Universal

1 Commits
df5999c338 ... Universal

Author SHA1 Message Date
CN-JS-HuiBai
2ca8958a1e 修改为对通用计算的支持模块 2026-04-04 13:02:50 +08:00
5 changed files with 272 additions and 374 deletions
Expand all files Collapse all files

66
README.md
View File

@@ -1,57 +1,27 @@
# Media Coding Web - Configuration & Run Guide
Media Coding Web is a high-performance video transcoding and streaming web application. It retrieves source media files from AWS S3 or MinIO, transcodes them on the fly utilizing hardware-accelerated video encoders, and streams them efficiently to the browser using HLS (m3u8).
To properly test and run this project, you will need to prepare your environment:
## Features
1. **Install Node.js & FFmpeg**:
- Ensure Node.js (v18+) is installed.
- Install **FFmpeg** on your system and make sure it is available in your PATH environment variable. The Node.js library `fluent-ffmpeg` requires it.
- Note: The platform supports Intel QSV, Nvidia NVENC, and VAAPI hardware encoders. Ensure your FFmpeg build includes support for `h264_qsv` / `hevc_qsv` / `h264_nvenc` / `hevc_nvenc` / `h264_vaapi` / `hevc_vaapi` if you wish to use them.
- The service always uses Jellyfin FFmpeg by default: `/usr/lib/jellyfin-ffmpeg/ffmpeg` and `/usr/lib/jellyfin-ffmpeg/ffprobe`. You can override them with `JELLYFIN_FFMPEG_PATH` and `JELLYFIN_FFPROBE_PATH`.
- **High-Performance HLS Streaming:** Streams video using HLS for fast startup, localized buffering, native browser video controls, and frame-accurate seeking.
- **Hardware-Accelerated Encoding:** Detects and seamlessly supports Intel QSV, NVIDIA NVENC, and VAAPI hardware encoders natively via FFmpeg.
- **AWS S3 / MinIO Integration:** Securely downloads media from S3-compatible endpoints to a dedicated local cache directory before processing.
- **Persistent Sessions:** Utilizes Valkey (Redis) for stateful session management, eliminating repetitive login actions.
- **Real-Time Progress:** Displays live feedback for both background S3 downloading and FFmpeg transcoding, communicated to the web application via WebSockets.
- **Modern User Interface:** Features a sleek and highly responsive user interface complete with built-in light and dark mode switching.
- **Centralized Log Storage:** Persists application activity and debugging events safely within the project's `/logs/` directory utilizing Winston loggers.
- **Unified Configuration:** A single `.env` file powers the entire web application settings.
2. **AWS S3 / MinIO Configuration**:
- Modify the `.env` file (copy from `.env.example`).
- Add your `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, and `S3_BUCKET_NAME`.
- If using **MinIO**, ensure you set the `S3_ENDPOINT` (e.g., `http://127.0.0.1:9000`) and set `S3_FORCE_PATH_STYLE=true`.
- Ensure your bucket has `.mp4` video files.
## Prerequisites
To properly run and deploy this project, your environment should contain:
1. **Node.js**: Version 18.x or newer is required to natively support all APIs.
2. **FFmpeg with Hardware Support**:
- Ensure you configure an FFmpeg instance on your system.
- For maximal performance, the application defaults to searching for `jellyfin-ffmpeg` located at `/usr/lib/jellyfin-ffmpeg/ffmpeg`. Feel free to point it at any other binary containing QSV or NVENC support by modifying the `JELLYFIN_FFMPEG_PATH` path inside `.env`.
3. **Valkey (or Redis)**: An active Redis-compatible cache server string is required for user sessions.
4. **AWS S3 / MinIO Account**: An object storage endpoint containing your encoded `.mp4` media catalog.
## Configuration & Installation
1. **Install Dependencies**:
Navigate into the project root directory and execute:
3. **Install Dependencies & Start**:
```bash
npm install
npm start
```
2. **Establish Environment Variables**:
Create a pristine copy of the `.env.example` structure:
```bash
cp .env.example .env
```
Open `.env` using your preferred editor to tailor your environment:
- **Valkey/Redis Settings**: Configure `VALKEY_URL`.
- **S3 Connectivity**: Insert your `S3_BUCKET_ADDRESS` and `AWS_REGION`. For MinIO specifically, fill the `S3_ENDPOINT` key (e.g. `http://127.0.0.1:9000`) and verify `S3_FORCE_PATH_STYLE=true`.
## Running the Application
To launch the web interface and application server, simply run:
```bash
npm start
```
*(You may also run `npm run dev` to execute the local environment.)*
## Verifying Setup
1. Open a browser and load `http://localhost:3000` (or your defined `PORT`).
2. Log into the system using your relevant S3 authorization if using local user mode.
3. The server will communicate directly to your S3 bucket and load the application's root dashboard.
4. Upon clicking any available video media, the server will rapidly download the required chunks to the local cache, start compiling HLS indexing data using native hardware encoding, and pipe an advanced native video stream back into your browser.
4. **Test Delivery**:
- Open your browser to `http://localhost:3000`.
- The application should display available `.mp4` items from your S3 bucket.
- Click one video, and the Node server will begin to read the S3 stream and pipe it to FFmpeg, transcoding it into HLS segments inside the `/public/hls/` directory.
- The frontend polls via `/api/status`, and once the index playlist is available, HLS playback starts!

40
public/css/style.css
View File

@@ -105,7 +105,6 @@ header h1 {
header h1 span {
background: linear-gradient(135deg, #3b82f6, #8b5cf6);
-webkit-background-clip: text;
background-clip: text;
-webkit-text-fill-color: transparent;
}
@@ -137,37 +136,35 @@ header p {
display: flex;
align-items: center;
justify-content: center;
background: transparent;
background: radial-gradient(circle at top left, rgba(249, 250, 252, 0.95), rgba(255, 255, 255, 0.98));
z-index: 50;
padding: 2rem;
}
.login-card {
width: min(480px, 100%);
background: var(--panel-bg);
backdrop-filter: blur(16px);
-webkit-backdrop-filter: blur(16px);
border: 1px solid var(--panel-border);
background: #ffffff;
border: 1px solid rgba(148, 163, 184, 0.3);
border-radius: 24px;
padding: 2rem;
box-shadow: 0 20px 40px rgba(148, 163, 184, 0.15);
box-shadow: 0 20px 60px rgba(15, 23, 42, 0.12);
}
.login-card h2 {
margin: 0 0 0.75rem;
font-size: 1.75rem;
color: var(--text-primary);
color: #111827;
}
.login-card p {
color: var(--text-secondary);
color: #6b7280;
margin: 0 0 1.5rem;
}
.login-card label {
display: block;
margin-bottom: 1rem;
color: var(--text-secondary);
color: #4b5563;
font-size: 0.95rem;
}
@@ -178,17 +175,17 @@ header p {
.login-card input {
width: 100%;
border: 1px solid var(--panel-border);
background: rgba(255, 255, 255, 0.08);
color: var(--text-primary);
border: 1px solid rgba(148, 163, 184, 0.5);
background: #f8fafc;
color: #111827;
border-radius: 12px;
padding: 0.85rem 1rem;
outline: none;
}
.login-card input:focus {
border-color: var(--accent);
box-shadow: 0 0 0 4px var(--accent-glow);
border-color: #2563eb;
box-shadow: 0 0 0 4px rgba(59, 130, 246, 0.12);
}
.login-card .primary-btn {
@@ -303,6 +300,13 @@ header p {
margin-bottom: 0.75rem;
}
.section-actions-row {
display: block;
margin-bottom: 1.5rem;
padding-top: 1rem;
border-top: 1px solid var(--panel-border);
}
.section-actions {
display: flex;
flex-wrap: nowrap;
@@ -311,12 +315,6 @@ header p {
justify-content: flex-start;
}
.banner-actions {
display: flex;
align-items: center;
gap: 0.5rem;
}
.action-btn {
border: none;
background: #2563eb;

20
public/index.html
View File

@@ -42,14 +42,12 @@
<label>页面主题：
<select id="theme-selector">
<option value="light">白天模式</option>
<option value="dark">黑夜模式</option>
<option value="dark">深夜模式</option>
</select>
</label>
<div class="banner-actions">
<button id="refresh-btn" class="action-btn" title="刷新列表">刷新列表</button>
</div>
<div class="user-info">
<button id="logout-btn" class="action-btn danger">退出登录</button>
<span id="current-username"></span>
<button id="logout-btn" class="action-btn">退出登录</button>
</div>
</div>
</div>
@@ -57,7 +55,13 @@
<main class="dashboard">
<section class="video-list-section glass-panel">
<div class="section-header">
<h2 id="video-list-header">Available Videos</h2>
<h2>Available Videos</h2>
</div>
<div class="section-actions-row">
<div class="section-actions">
<button id="refresh-btn" class="action-btn" title="刷新列表">刷新列表</button>
<button id="clear-download-cache-btn" class="action-btn" title="清空下载缓存">清空下载缓存</button>
</div>
</div>
<div class="bucket-panel">
<label for="bucket-select">Select Bucket</label>
@@ -139,12 +143,8 @@
<div id="now-playing" class="now-playing hidden">
<h3>Now Playing</h3>
<p id="current-video-title">video.mp4</p>
<div style="display: flex; gap: 0.5rem; margin-top: 1rem; align-items: center; flex-wrap: wrap;">
<button id="transcode-btn" class="play-btn hidden">Start Transcode</button>
<button id="stop-transcode-btn" class="play-btn stop-btn hidden">Stop Transcode</button>
<button id="clear-playing-download-cache-btn" class="action-btn danger hidden">清空下载缓存</button>
<button id="clear-playing-transcode-cache-btn" class="action-btn danger hidden">清空转码缓存</button>
</div>
</div>
</section>
</main>

113
public/js/main.js
View File

@@ -2,6 +2,7 @@ document.addEventListener('DOMContentLoaded', () => {
const videoListEl = document.getElementById('video-list');
const loadingSpinner = document.getElementById('loading-spinner');
const refreshBtn = document.getElementById('refresh-btn');
const clearDownloadCacheBtn = document.getElementById('clear-download-cache-btn');
const bucketSelect = document.getElementById('bucket-select');
const loginScreen = document.getElementById('login-screen');
const appContainer = document.getElementById('app-container');
@@ -18,10 +19,8 @@ document.addEventListener('DOMContentLoaded', () => {
const currentVideoTitle = document.getElementById('current-video-title');
const transcodeBtn = document.getElementById('transcode-btn');
const stopTranscodeBtn = document.getElementById('stop-transcode-btn');
const clearPlayingDownloadBtn = document.getElementById('clear-playing-download-cache-btn');
const clearPlayingTranscodeBtn = document.getElementById('clear-playing-transcode-cache-btn');
const themeSelector = document.getElementById('theme-selector');
const videoListHeader = document.getElementById('video-list-header');
const currentUsername = document.getElementById('current-username');
const logoutBtn = document.getElementById('logout-btn');
const topBannerTitle = document.getElementById('top-banner-title');
const playBtn = document.getElementById('play-btn');
@@ -141,7 +140,7 @@ document.addEventListener('DOMContentLoaded', () => {
const buildHlsPlaylistUrl = () => {
const decoder = 'auto';
const encoder = encoderSelect?.value || 'h264_rkmpp';
const encoder = encoderSelect?.value || 'h264_qsv';
let streamUrl = `/api/hls/playlist.m3u8?bucket=${encodeURIComponent(selectedBucket)}&key=${encodeURIComponent(selectedKey)}&decoder=${encodeURIComponent(decoder)}&encoder=${encodeURIComponent(encoder)}`;
const sessionId = localStorage.getItem('sessionId');
if (sessionId) {
@@ -544,7 +543,7 @@ document.addEventListener('DOMContentLoaded', () => {
if (topBannerTitle) topBannerTitle.textContent = title;
topBanner.classList.remove('hidden');
document.title = title;
populateSelect(encoderSelect, data.videoEncoders || [], data.defaultVideoEncoder || 'h264_rkmpp');
populateSelect(encoderSelect, data.videoEncoders || [], data.defaultVideoEncoder || 'h264_qsv');
} catch (err) {
console.error('Config load failed:', err);
}
@@ -587,7 +586,7 @@ document.addEventListener('DOMContentLoaded', () => {
localStorage.setItem('sessionId', sessionId);
if (username) {
localStorage.setItem('username', username);
if (videoListHeader) videoListHeader.textContent = `${username} Available Videos`;
if (currentUsername) currentUsername.textContent = username;
}
};
@@ -675,7 +674,26 @@ document.addEventListener('DOMContentLoaded', () => {
}
};
const clearDownloadCache = async () => {
if (!clearDownloadCacheBtn) return;
clearDownloadCacheBtn.disabled = true;
clearDownloadCacheBtn.textContent = '清空中...';
try {
const res = await fetch('/api/clear-download-cache', { method: 'POST' });
if (!res.ok) {
const data = await res.json().catch(() => ({}));
throw new Error(data.error || '清空下载缓存失败');
}
alert('下载缓存已清空');
} catch (err) {
console.error('Clear download cache failed:', err);
alert(`清空下载缓存失败: ${err.message}`);
} finally {
clearDownloadCacheBtn.disabled = false;
clearDownloadCacheBtn.textContent = '清空下载缓存';
}
};
if (transcodeBtn) {
transcodeBtn.addEventListener('click', () => {
@@ -715,8 +733,7 @@ document.addEventListener('DOMContentLoaded', () => {
// Build a tree structure from S3 keys
const tree = {};
data.videos.forEach(vid => {
const key = vid.key;
data.videos.forEach(key => {
const parts = key.split('/');
let current = tree;
parts.forEach((part, index) => {
@@ -725,18 +742,15 @@ document.addEventListener('DOMContentLoaded', () => {
}
if (index === parts.length - 1) {
current[part].__file = key;
current[part].__hasTranscodeCache = vid.hasTranscodeCache;
current[part].__hasDownloadCache = vid.hasDownloadCache;
}
current = current[part];
});
});
const createFileItem = (name, key, hasTranscodeCache, hasDownloadCache) => {
const createFileItem = (name, key) => {
const li = document.createElement('li');
li.className = 'video-item file-item';
const ext = name.split('.').pop().toUpperCase();
li.innerHTML = `
<div class="video-icon">
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m22 8-6 4 6 4V8Z"/><rect width="14" height="12" x="2" y="6" rx="2" ry="2"/></svg>
@@ -748,17 +762,16 @@ document.addEventListener('DOMContentLoaded', () => {
`;
li.addEventListener('click', (e) => {
e.stopPropagation();
selectVideo(key, li, hasTranscodeCache, hasDownloadCache);
selectVideo(key, li);
});
return li;
};
const renderTree = (node, container) => {
for (const [name, value] of Object.entries(node)) {
if (name === '__file' || name === '__hasTranscodeCache' || name === '__hasDownloadCache') continue;
if (name === '__file') continue;
const childKeys = Object.keys(value).filter(k => k !== '__file' && k !== '__hasTranscodeCache' && k !== '__hasDownloadCache');
const childKeys = Object.keys(value).filter(key => key !== '__file');
const hasChildren = childKeys.length > 0;
const isFile = typeof value.__file === 'string';
@@ -789,13 +802,13 @@ document.addEventListener('DOMContentLoaded', () => {
li.appendChild(folderHeader);
if (isFile) {
subListContainer.appendChild(createFileItem(name, value.__file, value.__hasTranscodeCache, value.__hasDownloadCache));
subListContainer.appendChild(createFileItem(name, value.__file));
}
renderTree(value, subListContainer);
li.appendChild(subListContainer);
container.appendChild(li);
} else if (isFile) {
container.appendChild(createFileItem(name, value.__file, value.__hasTranscodeCache, value.__hasDownloadCache));
container.appendChild(createFileItem(name, value.__file));
}
}
};
@@ -808,7 +821,7 @@ document.addEventListener('DOMContentLoaded', () => {
};
// Handle video selection
const selectVideo = (key, listItemNode, hasTranscodeCache = false, hasDownloadCache = false) => {
const selectVideo = (key, listItemNode) => {
document.querySelectorAll('.video-item').forEach(n => n.classList.remove('active'));
listItemNode.classList.add('active');
@@ -842,16 +855,6 @@ document.addEventListener('DOMContentLoaded', () => {
stopTranscodeBtn.disabled = false;
stopTranscodeBtn.textContent = 'Stop Transcode';
}
if (clearPlayingDownloadBtn) {
if (hasDownloadCache) clearPlayingDownloadBtn.classList.remove('hidden');
else clearPlayingDownloadBtn.classList.add('hidden');
}
if (clearPlayingTranscodeBtn) {
if (hasTranscodeCache) clearPlayingTranscodeBtn.classList.remove('hidden');
else clearPlayingTranscodeBtn.classList.add('hidden');
}
if (playBtn) {
playBtn.classList.add('hidden');
}
@@ -1096,56 +1099,12 @@ document.addEventListener('DOMContentLoaded', () => {
// Bind events
refreshBtn.addEventListener('click', () => fetchVideos(selectedBucket));
if (clearDownloadCacheBtn) {
clearDownloadCacheBtn.addEventListener('click', clearDownloadCache);
}
if (stopTranscodeBtn) {
stopTranscodeBtn.addEventListener('click', stopTranscode);
}
if (clearPlayingDownloadBtn) {
clearPlayingDownloadBtn.addEventListener('click', async () => {
if (!currentVideoKey) return;
clearPlayingDownloadBtn.disabled = true;
clearPlayingDownloadBtn.textContent = '清空中...';
try {
const res = await fetch('/api/clear-video-download-cache', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ bucket: selectedBucket, key: currentVideoKey })
});
if (!res.ok) throw new Error('清空失败');
clearPlayingDownloadBtn.classList.add('hidden');
fetchVideos(selectedBucket);
} catch (err) {
alert('清空下载缓存失败: ' + err.message);
} finally {
clearPlayingDownloadBtn.disabled = false;
clearPlayingDownloadBtn.textContent = '清空下载缓存';
}
});
}
if (clearPlayingTranscodeBtn) {
clearPlayingTranscodeBtn.addEventListener('click', async () => {
if (!currentVideoKey) return;
clearPlayingTranscodeBtn.disabled = true;
clearPlayingTranscodeBtn.textContent = '清空中...';
try {
const res = await fetch('/api/clear-video-transcode-cache', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ bucket: selectedBucket, key: currentVideoKey })
});
if (!res.ok) throw new Error('清空失败');
clearPlayingTranscodeBtn.classList.add('hidden');
fetchVideos(selectedBucket);
} catch (err) {
alert('清空转码缓存失败: ' + err.message);
} finally {
clearPlayingTranscodeBtn.disabled = false;
clearPlayingTranscodeBtn.textContent = '清空转码缓存';
}
});
}
if (loginBtn) {
loginBtn.addEventListener('click', login);
}
@@ -1165,7 +1124,7 @@ document.addEventListener('DOMContentLoaded', () => {
method: 'POST',
headers: { 'X-Session-ID': sessionId }
});
} catch(e){}
} catch (e) { }
}
clearSessionAuth();
location.reload();

381
server.js
View File

@@ -7,7 +7,7 @@ const path = require('path');
const http = require('http');
const WebSocket = require('ws');
const ffmpeg = require('fluent-ffmpeg');
const { S3Client, ListBucketsCommand, ListObjectsV2Command, GetObjectCommand, HeadObjectCommand } = require('@aws-sdk/client-s3');
const { S3Client, ListBucketsCommand, ListObjectsV2Command, GetObjectCommand } = require('@aws-sdk/client-s3');
const crypto = require('crypto');
const Redis = require('ioredis');
@@ -59,10 +59,10 @@ const logger = winston.createLogger({
]
});
console.log = function(...args) { logger.info(util.format(...args)); };
console.info = function(...args) { logger.info(util.format(...args)); };
console.warn = function(...args) { logger.warn(util.format(...args)); };
console.error = function(...args) { logger.error(util.format(...args)); };
console.log = function (...args) { logger.info(util.format(...args)); };
console.info = function (...args) { logger.info(util.format(...args)); };
console.warn = function (...args) { logger.warn(util.format(...args)); };
console.error = function (...args) { logger.error(util.format(...args)); };
process.on('uncaughtException', (err) => {
logger.error(`Uncaught Exception: ${err.message}`, err);
@@ -142,11 +142,14 @@ const createS3Client = (credentials) => {
const progressMap = {};
const transcodeProcesses = new Map();
const wsSubscriptions = new Map();
const activeDownloads = new Map();
const AVAILABLE_VIDEO_ENCODERS = [
{ value: 'h264_rkmpp', label: 'H.264(RKMPP HighSpeed)' },
{ value: 'hevc_rkmpp', label: 'H.265(RKMPP HighSpeed)' },
{ value: 'h264_qsv', label: 'H.264(Intel QSV)' },
{ value: 'hevc_qsv', label: 'H.265(Intel QSV)' },
{ value: 'h264_nvenc', label: 'H.264(Nvidia NVENC)' },
{ value: 'hevc_nvenc', label: 'H.265(Nvidia NVENC)' },
{ value: 'h264_vaapi', label: 'H.264(VAAPI)' },
{ value: 'hevc_vaapi', label: 'H.265(VAAPI)' },
{ value: 'libx264', label: 'H.264(Software Slow)' },
{ value: 'libx265', label: 'H.265(Software Slow)' }
];
@@ -197,35 +200,14 @@ const createFfmpegOptions = (encoderName) => {
options.push('-preset', 'fast', '-global_quality', '23');
} else if (/_vaapi$/.test(encoderName)) {
options.push('-qp', '23');
} else if (/_rkmpp$/.test(encoderName)) {
options.push('-qp_init', '23', '-pix_fmt', 'nv12');
}
return options;
};
const isRkmppCodec = (codecName) => /_rkmpp$/.test(codecName);
const isVaapiCodec = (codecName) => /_vaapi$/.test(codecName);
const availableEncoderValues = new Set(AVAILABLE_VIDEO_ENCODERS.map((item) => item.value));
const availableDecoderValues = new Set(AVAILABLE_VIDEO_DECODERS.map((item) => item.value));
const getRkmppDecoderName = (metadata) => {
const videoStream = (metadata?.streams || []).find((stream) => stream.codec_type === 'video');
const codecName = (videoStream?.codec_name || '').toLowerCase();
const decoderMap = {
av1: 'av1_rkmpp',
h263: 'h263_rkmpp',
h264: 'h264_rkmpp',
hevc: 'hevc_rkmpp',
mjpeg: 'mjpeg_rkmpp',
mpeg1video: 'mpeg1_rkmpp',
mpeg2video: 'mpeg2_rkmpp',
mpeg4: 'mpeg4_rkmpp',
vp8: 'vp8_rkmpp',
vp9: 'vp9_rkmpp'
};
return decoderMap[codecName] || null;
};
const parseFpsValue = (fpsText) => {
if (typeof fpsText !== 'string' || !fpsText.trim()) {
return 0;
@@ -264,8 +246,6 @@ const getSeekFriendlyOutputOptions = (encoderName, metadata) => {
options.push('-forced-idr', '1', '-force_key_frames', 'expr:gte(t,n_forced*2)');
} else if (/_qsv$/.test(encoderName)) {
options.push('-idr_interval', '1');
} else if (/_rkmpp$/.test(encoderName)) {
options.push('-force_key_frames', 'expr:gte(t,n_forced*2)');
}
return options;
@@ -273,7 +253,7 @@ const getSeekFriendlyOutputOptions = (encoderName, metadata) => {
const shouldRetryWithSoftware = (message) => {
if (!message) return false;
return /Cannot load libcuda\.so\.1|Could not open encoder before EOF|Error while opening encoder|Operation not permitted|Invalid argument|mpp_create|rkmpp/i.test(message);
return /Cannot load libcuda\.so\.1|Could not open encoder before EOF|Error while opening encoder|Operation not permitted|Invalid argument/i.test(message);
};
const probeFile = (filePath) => {
@@ -387,131 +367,28 @@ wss.on('connection', (ws) => {
ws.on('close', () => removeWsClient(ws));
});
const ensureS3Downloaded = async (s3Client, bucket, key, tmpInputPath, progressKey, streamSessionId) => {
if (activeDownloads.has(progressKey)) {
const clearDownloadCache = () => {
const tmpDir = CACHE_DIR;
try {
await activeDownloads.get(progressKey);
} catch (err) {
// Ignore error and retry if previous failed
if (!fs.existsSync(tmpDir)) return;
const files = fs.readdirSync(tmpDir);
for (const file of files) {
if (file.startsWith('s3-input-') && file.endsWith('.tmp')) {
const filePath = path.join(tmpDir, file);
fs.rmSync(filePath, { force: true });
} else if (file.startsWith('hls-')) {
const filePath = path.join(tmpDir, file);
fs.rmSync(filePath, { recursive: true, force: true });
}
}
let shouldDownload = true;
let s3Metadata = null;
if (fs.existsSync(tmpInputPath)) {
const stats = fs.statSync(tmpInputPath);
const localSize = stats.size;
try {
const headCommand = new HeadObjectCommand({ Bucket: bucket, Key: key });
s3Metadata = await s3Client.send(headCommand);
if (s3Metadata.ContentLength === localSize) {
shouldDownload = false;
console.log(`[Cache] Verified ${key}: Local size matches S3 (${localSize} bytes).`);
} else {
console.log(`[Cache] Mismatch for ${key}: Local ${localSize} vs S3 ${s3Metadata.ContentLength}. Re-downloading.`);
}
} catch (err) {
console.error(`[Cache] Failed to verify S3 metadata for ${key}:`, err.message);
}
}
if (!shouldDownload) {
progressMap[progressKey] = {
status: 'downloaded',
percent: 100,
downloadedBytes: s3Metadata.ContentLength,
totalBytes: s3Metadata.ContentLength,
streamSessionId,
details: 'Source cached locally and verified against S3...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
return;
}
const downloadPromise = (async () => {
const command = new GetObjectCommand({ Bucket: bucket, Key: key });
const response = await s3Client.send(command);
const totalBytes = response.ContentLength || 0;
const s3Stream = response.Body;
let downloadedBytes = 0;
const downloadingPath = tmpInputPath + '.downloading';
progressMap[progressKey] = {
status: 'downloading',
percent: 0,
downloadedBytes: 0,
totalBytes,
streamSessionId,
details: 'Downloading full source...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
await new Promise((resolve, reject) => {
const writeStream = fs.createWriteStream(downloadingPath);
s3Stream.on('data', (chunk) => {
downloadedBytes += chunk.length;
const percent = totalBytes ? Math.min(100, Math.round((downloadedBytes / totalBytes) * 100)) : 0;
const downloadState = {
status: 'downloading',
percent,
downloadedBytes,
totalBytes,
streamSessionId,
details: totalBytes ? `Downloading source ${percent}%` : 'Downloading source...',
mp4Url: null
};
progressMap[progressKey] = downloadState;
broadcastWs(progressKey, { type: 'progress', key, progress: downloadState });
});
s3Stream.on('error', reject);
writeStream.on('error', reject);
writeStream.on('finish', () => {
try {
fs.renameSync(downloadingPath, tmpInputPath);
resolve();
} catch (e) {
reject(e);
}
});
s3Stream.pipe(writeStream);
});
progressMap[progressKey] = {
status: 'downloaded',
percent: 100,
downloadedBytes,
totalBytes,
streamSessionId,
details: 'Source download complete...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
})();
activeDownloads.set(progressKey, downloadPromise);
try {
await downloadPromise;
} catch (err) {
if (fs.existsSync(tmpInputPath + '.downloading')) {
fs.rmSync(tmpInputPath + '.downloading', { force: true });
}
console.error('Failed to clear download cache:', err);
throw err;
} finally {
activeDownloads.delete(progressKey);
}
};
app.get('/api/buckets', async (req, res) => {
try {
const auth = await extractS3Credentials(req);
@@ -561,17 +438,6 @@ app.get('/api/videos', async (req, res) => {
if (!key) return false;
const lowerKey = key.toLowerCase();
return videoExtensions.some(ext => lowerKey.endsWith(ext));
})
.map(key => {
const safeBucket = bucket.replace(/[^a-z0-9]/gi, '_');
const safeKeySegments = key.split('/').map(segment => segment.replace(/[^a-z0-9]/gi, '_'));
const hlsDir = path.join(CACHE_DIR, `hls-${safeBucket}-${safeKeySegments.join('-')}`);
const tmpInputPath = path.join(CACHE_DIR, `s3-input-${safeBucket}-${safeKeySegments.join('-')}.tmp`);
return {
key: key,
hasTranscodeCache: fs.existsSync(hlsDir),
hasDownloadCache: fs.existsSync(tmpInputPath)
};
});
res.json({ videos });
@@ -614,49 +480,19 @@ app.get('/api/config', (req, res) => {
title,
ffmpegPath: JELLYFIN_FFMPEG_PATH,
ffprobePath: JELLYFIN_FFPROBE_PATH,
defaultVideoEncoder: 'h264_rkmpp',
defaultVideoEncoder: 'h264_qsv',
defaultVideoDecoder: 'auto',
videoEncoders: AVAILABLE_VIDEO_ENCODERS,
videoDecoders: AVAILABLE_VIDEO_DECODERS
});
});
app.post('/api/clear-video-transcode-cache', async (req, res) => {
app.post('/api/clear-download-cache', (req, res) => {
try {
const { bucket, key } = req.body;
if (!bucket || !key) {
return res.status(400).json({ error: 'Bucket and key are required' });
}
const safeBucket = bucket.replace(/[^a-z0-9]/gi, '_');
const safeKeySegments = key.split('/').map(segment => segment.replace(/[^a-z0-9]/gi, '_'));
const hlsDir = path.join(CACHE_DIR, `hls-${safeBucket}-${safeKeySegments.join('-')}`);
if (fs.existsSync(hlsDir)) {
fs.rmSync(hlsDir, { recursive: true, force: true });
}
res.json({ message: 'Transcode cache cleared for video' });
clearDownloadCache();
res.json({ message: 'Download cache cleared' });
} catch (error) {
console.error('Error clearing video transcode cache:', error);
res.status(500).json({ error: 'Failed to clear transcode cache', detail: error.message });
}
});
app.post('/api/clear-video-download-cache', async (req, res) => {
try {
const { bucket, key } = req.body;
if (!bucket || !key) {
return res.status(400).json({ error: 'Bucket and key are required' });
}
const safeBucket = bucket.replace(/[^a-z0-9]/gi, '_');
const safeKeySegments = key.split('/').map(segment => segment.replace(/[^a-z0-9]/gi, '_'));
const tmpInputPath = path.join(CACHE_DIR, `s3-input-${safeBucket}-${safeKeySegments.join('-')}.tmp`);
if (fs.existsSync(tmpInputPath)) {
fs.rmSync(tmpInputPath, { force: true });
}
res.json({ message: 'Download cache cleared for video' });
} catch (error) {
console.error('Error clearing video download cache:', error);
console.error('Error clearing download cache:', error);
res.status(500).json({ error: 'Failed to clear download cache', detail: error.message });
}
});
@@ -720,15 +556,85 @@ app.get('/api/hls/playlist.m3u8', async (req, res) => {
const auth = await extractS3Credentials(req);
const s3Client = createS3Client(auth);
let totalBytes = 0;
const progressKey = getProgressKey(key);
const streamSessionId = createStreamSessionId();
let downloadedBytes = 0;
if (!fs.existsSync(tmpInputPath)) {
try {
await ensureS3Downloaded(s3Client, bucket, key, tmpInputPath, progressKey, streamSessionId);
const command = new GetObjectCommand({ Bucket: bucket, Key: key });
const response = await s3Client.send(command);
totalBytes = response.ContentLength;
const s3Stream = response.Body;
progressMap[progressKey] = {
status: 'downloading',
percent: 0,
downloadedBytes: 0,
totalBytes,
streamSessionId,
details: 'Downloading full source before streaming...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
await new Promise((resolve, reject) => {
const writeStream = fs.createWriteStream(tmpInputPath);
s3Stream.on('data', (chunk) => {
downloadedBytes += chunk.length;
const percent = totalBytes ? Math.min(100, Math.round((downloadedBytes / totalBytes) * 100)) : 0;
const downloadState = {
status: 'downloading',
percent,
downloadedBytes,
totalBytes,
streamSessionId,
details: totalBytes ? `Downloading source ${percent}%` : 'Downloading source...',
mp4Url: null
};
progressMap[progressKey] = downloadState;
broadcastWs(progressKey, { type: 'progress', key, progress: downloadState });
});
s3Stream.on('error', reject);
writeStream.on('error', reject);
writeStream.on('finish', resolve);
s3Stream.pipe(writeStream);
});
progressMap[progressKey] = {
status: 'downloaded',
percent: 100,
downloadedBytes,
totalBytes,
streamSessionId,
details: 'Source download complete, parsing for HLS...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
} catch (err) {
console.error('S3 Download Failed:', err);
return res.status(500).send('S3 Download Failed');
}
} else {
const stats = fs.statSync(tmpInputPath);
totalBytes = stats.size;
downloadedBytes = totalBytes;
progressMap[progressKey] = {
status: 'downloaded',
percent: 100,
downloadedBytes,
totalBytes,
streamSessionId,
details: 'Source already downloaded locally, parsing for HLS...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
}
let duration = 0;
try {
@@ -746,7 +652,7 @@ app.get('/api/hls/playlist.m3u8', async (req, res) => {
if (i === totalSegments - 1 && duration % HLS_SEGMENT_TIME !== 0) {
segDur = (duration % HLS_SEGMENT_TIME) || HLS_SEGMENT_TIME;
}
m3u8 += `#EXTINF:${segDur.toFixed(6)},\nsegment.ts?bucket=${encodeURIComponent(bucket)}&key=${encodeURIComponent(key)}&seg=${i}&encoder=${req.query.encoder || 'h264_rkmpp'}&decoder=${req.query.decoder || 'auto'}\n`;
m3u8 += `#EXTINF:${segDur.toFixed(6)},\nsegment.ts?bucket=${encodeURIComponent(bucket)}&key=${encodeURIComponent(key)}&seg=${i}&encoder=${req.query.encoder || 'h264_qsv'}&decoder=${req.query.decoder || 'auto'}\n`;
}
m3u8 += `#EXT-X-ENDLIST\n`;
@@ -778,7 +684,7 @@ app.get('/api/hls/segment.ts', async (req, res) => {
const bucket = req.query.bucket;
const key = req.query.key;
const seg = parseInt(req.query.seg || '0');
const requestedEncoder = req.query.encoder || 'h264_rkmpp';
const requestedEncoder = req.query.encoder || 'h264_qsv';
const requestedDecoder = req.query.decoder || 'auto';
if (!bucket || !key || isNaN(seg)) return res.status(400).send('Bad Request');
@@ -826,7 +732,7 @@ app.get('/api/hls/segment.ts', async (req, res) => {
let sourceMetadata = null;
try { sourceMetadata = await probeFile(tmpInputPath); } catch (e) { }
const encoderName = availableEncoderValues.has(requestedEncoder) ? requestedEncoder : 'h264_rkmpp';
const encoderName = availableEncoderValues.has(requestedEncoder) ? requestedEncoder : 'h264_qsv';
const decoderName = availableDecoderValues.has(requestedDecoder) ? requestedDecoder : 'auto';
const m3u8Path = path.join(hlsDir, `temp.m3u8`);
@@ -840,7 +746,7 @@ app.get('/api/hls/segment.ts', async (req, res) => {
if (isVaapiCodec(encoderName)) {
ffmpegCommand.inputOptions(['-vaapi_device', '/dev/dri/renderD128']).videoFilters('format=nv12,hwupload');
}
const resolvedDecoderName = decoderName === 'auto' && isRkmppCodec(encoderName) ? getRkmppDecoderName(sourceMetadata) : decoderName;
const resolvedDecoderName = decoderName;
if (resolvedDecoderName && resolvedDecoderName !== 'auto') ffmpegCommand.inputOptions(['-c:v', resolvedDecoderName]);
const segmentFilename = path.join(hlsDir, `segment_%d.ts`);
@@ -916,7 +822,7 @@ app.get('/api/stream', async (req, res) => {
const bucket = req.query.bucket;
const key = req.query.key;
const requestedDecoder = typeof req.query.decoder === 'string' ? req.query.decoder.trim() : 'auto';
const requestedEncoder = typeof req.query.encoder === 'string' ? req.query.encoder.trim() : 'h264_rkmpp';
const requestedEncoder = typeof req.query.encoder === 'string' ? req.query.encoder.trim() : 'h264_qsv';
const startSeconds = parseFloat(req.query.ss) || 0;
const streamSessionId = typeof req.query.streamSessionId === 'string' && req.query.streamSessionId.trim()
? req.query.streamSessionId.trim()
@@ -929,7 +835,7 @@ app.get('/api/stream', async (req, res) => {
return res.status(400).json({ error: 'Video key is required' });
}
const videoEncoder = availableEncoderValues.has(requestedEncoder) ? requestedEncoder : 'h264_rkmpp';
const videoEncoder = availableEncoderValues.has(requestedEncoder) ? requestedEncoder : 'h264_qsv';
const requestedVideoDecoder = availableDecoderValues.has(requestedDecoder) ? requestedDecoder : 'auto';
const safeKeySegments = key.split('/').map(segment => segment.replace(/[^a-zA-Z0-9_\-]/g, '_'));
@@ -956,7 +862,74 @@ app.get('/api/stream', async (req, res) => {
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
}
await ensureS3Downloaded(s3Client, bucket, key, tmpInputPath, progressKey, streamSessionId);
let totalBytes = 0;
let downloadedBytes = 0;
if (!cacheExists) {
const command = new GetObjectCommand({ Bucket: bucket, Key: key });
const response = await s3Client.send(command);
const s3Stream = response.Body;
totalBytes = response.ContentLength || 0;
progressMap[progressKey] = {
status: 'downloading',
percent: 0,
downloadedBytes: 0,
totalBytes,
streamSessionId,
details: 'Downloading full source before streaming...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
await new Promise((resolve, reject) => {
const writeStream = fs.createWriteStream(tmpInputPath);
s3Stream.on('data', (chunk) => {
downloadedBytes += chunk.length;
const percent = totalBytes ? Math.min(100, Math.round(downloadedBytes / totalBytes * 100)) : 0;
const downloadState = {
status: 'downloading',
percent,
downloadedBytes,
totalBytes,
streamSessionId,
details: totalBytes ? `Downloading source ${percent}%` : 'Downloading source...',
mp4Url: null
};
progressMap[progressKey] = downloadState;
broadcastWs(progressKey, { type: 'progress', key, progress: downloadState });
});
s3Stream.on('error', reject);
writeStream.on('error', reject);
writeStream.on('finish', resolve);
s3Stream.pipe(writeStream);
});
progressMap[progressKey] = {
status: 'downloaded',
percent: 100,
downloadedBytes,
totalBytes,
streamSessionId,
details: 'Source download complete, starting real-time transcode...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
} else {
const stats = fs.statSync(tmpInputPath);
totalBytes = stats.size;
downloadedBytes = totalBytes;
progressMap[progressKey] = {
status: 'downloaded',
percent: 100,
downloadedBytes,
totalBytes,
streamSessionId,
details: 'Source already downloaded locally, starting real-time transcode...',
mp4Url: null
};
broadcastWs(progressKey, { type: 'progress', key, progress: progressMap[progressKey] });
}
// Probe file for duration and broadcast to clients
let sourceMetadata = null;
@@ -1003,9 +976,7 @@ app.get('/api/stream', async (req, res) => {
.videoFilters('format=nv12,hwupload');
}
const resolvedDecoderName = decoderName === 'auto' && isRkmppCodec(encoderName)
? getRkmppDecoderName(sourceMetadata)
: decoderName;
const resolvedDecoderName = decoderName;
if (resolvedDecoderName && resolvedDecoderName !== 'auto') {
ffmpegCommand.inputOptions(['-c:v', resolvedDecoderName]);